Web Performance Calendar

If your “quest for fast” goes through device detection, you’ll want to pay attention to this.

Introduction

I’ll go out on a limb and make a wild guess: if you care about web performance, that’s because you want to deliver great UX to users. Assuming my guess is right, in the last two decades the performance of websites and apps on mobile devices has been a key aspect of getting it right across mobile devices as well.

I’m not here for a history lesson, yet a little summary is useful to bring everyone on the same page. There was a time – somewhere between the late ‘90s and 2010 – when drawing a distinction between the web and the mobile web made a lot of sense. In fact, we almost did not need to make a distinction. The difference was there for everyone to see: UX on (WAP) mobile phones and UX on “the wired web” were so far apart that accounting for different experiences on the different platforms was the norm for developers. Considering it one platform? An almost ridiculous proposition for everyone but the most visionary among us techies.

That was an age when delivering great UX was tantamount to understanding which devices people were using. That was the time when the noble art of (mobile) Device Detection was born. Companies needed a framework to understand whether HTTP requests were coming from a fully-fledged desktop browser OR a little Nokia phone with a numeric keypad and a tiny display. As a minimum, mobile devices would be directed to a specific m.* (mDot) WAP site written in XHTML (or even WML in the very beginning), but further optimization – that accounted for specific features of each device – was also normal thanks to tools like WURFL (a repository of mobile devices that applications could query in real time through an API). Desktop web browser would be directed to www.

Then 2010 came. Smartphones had arrived, for real this time. CSS Media Queries on mobile browsers became the norm and this made RWD – Responsive Web Design – the new standard for web development. Complex websites would columnify themselves on smartphones and tablets. From that moment, thinking of the mobile web as a separate entity from the web no longer made sense. That was the moment when the need for Device Detection started decreasing. It didn’t fade away, but certainly Device Detection became confined to more “niche” use cases (that was the time when WURFL went from being an open-source project to becoming a commercial product, coincidentally… or maybe not so coincidentally, but I digress).

Device Detection and Performance

Long story short, many developers who deal with performance today may not even think of device detection in those terms, and they’ll be happy with the level of device-awareness that Media Queries and JavaScript feature-detection delivers. Still, there are performance-focused companies out there that elect to push the envelope and optimize their service and content on a device-by-device basis. Here at ScientiaMobile we know this scenario well. Not only are “the WURFL guys”, but one of our main products – ImageEngine – is an ImageCDN: we are in charge of delivering the optimal version of images to apps and (responsive) web sites, depending on which device each user holds in their hands. The magic relies on our ability to detect the device, i.e. to analyze HTTP requests and determine in real-time the capabilities of that phone, tablet, smart-TV or whatever other funky device the guy who likes to browse the web with his wristwatch may hold. In other words, Device Detection is at the core of web performance in many cases.

Device Detection is changing. Google dixit.

Old timers may have detected (no pun intended) that I am probably talking about the HTTP User-Agent string when I talk about detecting mobile devices. And they would be right. That little string has been part of the HTTP specs since time immemorial and tech companies of all kinds have come to rely on it for web performance, for security, for serving relevant ads, for streaming video and for a plethora of other use cases.

You might expect that the User-Agent string is here to stay then, right? Wrong, unfortunately. Google has decided that they don’t want it and have already started killing it in Chrome and all Chromium-based browsers. Technically they don’t call it “killing the User-Agent”. Initially they called it freezing the UA string, then they moved to “UA reduction”. What this means in practice is that “a sad excuse for a user-agent string” will still be there, but it won’t disclose any information on the device and it will even lie obout the Android version the device is running.

You may wonder what the rationale behind this decision is, and I could use this space (and more!) to provide my opinion (In short: I think this is a textbook example of how a bully exploits its market dominance to hurt the rest of the ecosystem with a might makes right move), but let’s not go there. The official story is that the User-Agent leaked too much personal information and Google cares about users’ privacy so much that they just had to get rid of the UA string. Enough laughing for today.

What you may be more curious about is how developers will be able to tell the make and model of a device for the thousands of legit usages in the technology industry.

Enter Client-Hints

The alternative to UA strings are UA Client-Hints (UA-CH), i.e. an additional set of HTTP headers that will carry bits of information about a browser and device that were previously traceable in or through the UA string. The problem is that those headers won’t be automatically available for each request. Rather, they’ll need to be explicitly requested through a rather complex HTTP dance between browser and server. The browser will send a request without any device information, and the server that cares about it will say “I need that additional information, dear”. Hopefully, at that point, newer HTTPS requests will carry the UA-CH. From a WebPerf perspective this isn’t good. It sucks in fact. But there is more. Requesting UA-CH implies that developers need to proactively configure their servers to demand that the good data is sent. This is key if one’s business model depends on access to device information.

The Gory Details: what the UA string still is. And what it’s going to be.

This is what the UA string of a Pixel 5 looks like in December 2022:

Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4812.0 Mobile Safari/537.36

In a matter of weeks, as newer Android updates roll out in Q1 2023, this UA string will be replaced with:

Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Mobile Safari/537.36

The platform, platform version, the device model, the browser name and browser version are there today. They’ll soon be gone.

Not all client hints are created equal

There are actually two kinds of User-Agent Client Hints you can expect to see in your traffic – “low entropy” and “high entropy”. The former is the set of Client Hints you get for free with every request from a browser that supports User-Agent Client Hints. This kind of Client-Hints are not very “informative”. Which is why you probably want the second kind, the “high entropy” ones, as they provide all the bits and pieces for optimal device detection.

The terms “low entropy” and “high entropy” are used in the User-Agent Client Hint specification to describe two possible sets of headers you may receive. At ScientiaMobile we talk about “Basic” header quality and “Full” header quality respectively, as this is the key differentiator to what WURFL can do to perform high quality Device Detection.

While “Basic” User-Agent Client Hints are included with every request, the “Full” header quality must be specifically requested. This means that you need to configure your servers to request them. Here’s a table that explains what pieces of information you get with each set of User-Agent Client Hints.

If you don’t request the entire set of User-Agent Client Hints explicitly, the ones you are seeing in your logs are most likely the “low entropy” stuff. In practice, this means that you will only be able to tell the OS platform, a list of possible browsers that requested the page and Chromium’s idea of whether the client is a mobile device without further differentiation between tablets, desktops, Smart TVs, IOT devices and so on.

Here’s an example of what you are likely to see with just the “Basic” or “low entropy” set of User-Agent Client Hints:

User-Agent: Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Mobile Safari/537.36
Sec-Ch-Ua: "Chromium";v="106", "Microsoft Edge";v="106", "Not;A=Brand";v="99"
Sec-Ch-Ua-Platform: "Android"
Sec-Ch-Ua-Mobile: ?1

Note: If you are seeing “Full” header quality or “high entropy” User-Agent Client Hints in your logs, the “Basic” header quality or “low entropy” User-Agent Client Hints are also included in the request.

These set of “low entropy” User-Agent Client Hints tells us that:

1. The browser originating the request could be either Chromium or Microsoft Edge
2. That the originating device is considered a mobile device

More importantly, we know that this is an Android device, but not the version. The device make and model are unknown, and so is the exact version of the browser.

Please note that the “Not A Brand” token in the Sec-Ch-Ua aka the Brands header is intentional and is added according to the User-Agent Client Hints specification. This is called GREASE-ing, an allusion to how the Chromium team ensures cipher suite compatibility.

Now let’s put this in contrast with the complete “high entropy” User-Agent Client Hints. When you specifically request the “full” set of User-Agent Client Hint headers (and you must request them individually), you also receive the platform version, the full browser version, the model name, the architecture of the device and, depending on the platform, the “bitness” or the bit width of the device.

Here’s a specimen of a HTTP request that contains those high entropy Client-Hints:

User-Agent: Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Mobile Safari/537.36
Sec-Ch-Ua: "Chromium";v="106", "Microsoft Edge";v="106", "Not;A=Brand";v="99"
Sec-Ch-Ua-Full-Version-List: "Chromium";v="106.0.5249.65", "Microsoft Edge";v="106.0.5249.65", "Not;A=Brand";v="99.0.0.0"
Sec-Ch-Ua-Platform: "Android"
Sec-Ch-Ua-Mobile: ?1
Sec-Ch-Ua-Platform-Version: 13.0.0
Sec-Ch-Ua-Model: Pixel 6 Pro

We can immediately see that we get quite a lot of additional information, such as:

1. Full version for the browser that originated the request
2. Full platform/OS version
3. Device model name

In other words, all the good stuff is available again with high entropy Client Hints.

Interestingly, the actual browser name is never directly exposed via User-Agent Client Hints. Instead, the raw header data will always contain a comma separated list of browser brands.This is an intentional part of the User-Agent Client Hints specification and was introduced with the assumption that this would make it harder for developers to discriminate against minor browsers. Without debating that move here, it is important to note that the WURFL API is able to sort through this information and return the accurate browser name.

How to make sure that the good client-hints get sent

If your business depends on – or simply benefits from – accurate device detection, you should probably go out of your way to ensure access to the “Full” header quality User-Agent Client Hints. Without proactive action, Device Detection quality will suffer.

You should configure your application and HTTP servers to request those additional headers through the Accept-CH header. It will need to advertise which Client-Hints are needed and – of course – you want the good ones, i.e. all the “Full” header quality User-Agent Client Hints. We recommend this:

Accept-CH: sec-ch-ua-platform-version,sec-ch-ua-full-version,sec-ch-ua-full-version-list,sec-ch-ua-model,sec-ch-ua-arch,sec-ch-ua-bitness,sec-ch-ua-wow64

Additionally, you may want the server to signal that these high entropy Client Hints are critical for an optimal browsing experience on your website. You can do this by setting the Critical-CH response header as follows:

Critical-CH: sec-ch-ua-platform-version,sec-ch-ua-full-version,sec-ch-ua-full-version-list,sec-ch-ua-model,sec-ch-ua-arch,sec-ch-ua-bitness,sec-ch-ua-wow64

If you are using a third-party cloud-based solution, you will also need to delegate access to the User-Agent Client Hints to that party. You can do this by using a Permissions-Policy header. Here’s an example from one of our services (WURFL.js Lite), i.e. what we recommend to users of that service:

permissions-policy: ch-ua-platform-version=(self "https://wurfl.io"),ch-ua-full-version=(self "https://wurfl.io"),ch-ua-full-version-list=(self "https://wurfl.io"),ch-ua-model=(self "https://wurfl.io"),ch-ua-arch=(self "https://wurfl.io"),ch-ua-bitness=(self "https://wurfl.io"),ch-ua-wow64=(self "https://wurfl.io")

(More detailed information on this topic is available in ScientiaMobile’s “Request and Implement User-Agent Client Hints” technical documentation here ).

We realize that different web servers have different methods of setting request headers, so we have kept these instructions general enough to be server/platform agnostic. We also have Apache specific instructions on enabling User-Agent Client Hints support (here)