Peter Hedenskog (@soulislove) works in the performance team at the Wikimedia Foundation. Peter is one of the creators of the Open Source sitespeed.io web performance tools.
Image by andrekheren
My main focus the last months has been setting up a Wikimedia performance mobile device lab. We started with Android devices and plan to add iOS when Apple pushed our First Contentful Paint time contribution by Noam Rosenthal in Safari and fix a WebDriver issue. You can read all about our setup at our documentation page at Wikitech
Our goal is to use the performance device lab to find performance regressions on Wikipedia and get us a better feeling for mobile users performance. We do collect RUM data from our mobile users but RUM doesn’t give us the details that we can get from synthetic testing.
Running synthetic tests on a server vs running them on a mobile phone is different. I wanted to share what I’ve learned so far.
Phones
To set up a performance device lab you need phones. I bought all my phones second hand. At the moment we use Moto G4 and Moto G5. The Moto G4 because it’s been the recommendation for testing since Alex Russels investigation blog post. Even though it could be that it is slower than your average phone, it is good because it will help you find regressions. Using too slow phones, makes the metrics unstable. I’ve been testing with a Huawei Y360 and there are so many variations in the metrics so it’s hard to use.
We also use Moto G5 to follow how Mozilla runs their performance tests (copying/learning from the best). If you have rooted a Moto G5 you can tune the performance of the CPU and GPU to try to get more stable metrics and that’s what we try to do.
USB connection/cord
Some of the Android phones come with a bad USB cable. Change them. I cannot recommend any specific brand but you should find chords that click into your phone (they make a click-sound when they are inserted in your phone correctly). They tend to stick in the device and never lose the connection.
Root vs non rooted devices
Rooting your device is good because it gives you full access to do whatever you want on your phone and you can do what Mozilla does to set up your phone for stable metrics. You have full access to the shell in the phone and that is nice. Especially if you do not have physical access to the phone.
However it’s always a little scary to root your device. It could go wrong and your phone is broken. I have a broken rooted phone that continuously restarts, restarts and restarts until it runs out of battery.
Battery temperature
If your mobile phone gets too hot, the CPU frequency will change and then your performance metrics will change. You want to keep your phone cold. But what is the temperature limit?
I asked Annie Sullivan of the Chrome metrics team for help and she told me they use 35 Celsius for battery and 38 Celsius for the CPU.
We only measure the battery temperature because that is easier. When we run our tests, the temperature increases, so after our tests we let the phones rest for a while before start to run the next test.
The temperature in the room is important. One day some time ago our phones increased their highest temperature by 10 Celsius. I started to investigate the phones: Was there something else running on the phones? After a while I could see that the temperature didn’t come down after the rest period. I increased the rest time but no luck. The problem was caused by our hosting company that had added phones from other users near to our phones, so that the default rack temperature was higher.
Connectivity
There’s two ways of thinking about connectivity for your tests: Either you use a sim card and run your tests with current connection 3g/4g/5g. This is good because you will get a feeling of what your users are seeing. Running continuous tests will let you see any fluctuations. But your tests will not be stable so it will be hard to use for finding regressions.
We run two types of tests: One using WebPageReplay to replay Wikipedia to easier find frontend regressions and one where our hosting has set up a solution where we throttle the connectivity to look like 3g/4g.
Helpful tools/resources
Running an Android device lab, you need to learn to use ADB (Android Debug Bridge). ADB helps you run commands on your phone and is your swiss army knife when you are working with your phone. I use this list to remember all the ADB commands.
If you are working remotely with your device lab (you are not in the same room as the phones) my number one go tool is scrcpy (it is Open Source!). It gives you access to the phone’s display and you can control your device. This is super helpful when you are remotely debugging problems. If your tests stop working, access the phone using scrcpy to find out why.
Hosted phones vs hosting yourself
I really recommend hosting the phones yourself or at least do that during the setup. You often need physical access to the device and there are a lot of things that can go wrong.
If you use hosted devices I can recommend setting up a test script that iterates over your phones checking that they are online, have an internet connection and the temperature of the battery. You can use our script as a start and modify it. That kind of script helps me find problems in the environment.
Summary
I used to run one off tests on mobile and I think the device lab will help us a lot in finding performance regressions by continuously running our tests and give us a better feel for users mobile performance. I hope I can share more in the next year’s performance calendar when we had the lab up and running for a year.