Web Performance Calendar

Every developer knows that when it comes to web performance and user experience, every image matters: images are the low-hanging fruit of performance optimization. The ability to apply visually lossless compression can benefit organizations by reducing page weight without compromising user experience, thus improving performance and driving greater audience engagement. There are many techniques available for determining which image to request, but you also have to create those “right images” to send. Delivering optimal images to all users across all device types and all browsers should be the goal of all web developers and performance services. (Accessibility should be another goal, but that’s a different blog!)

Defining the image(s) to include

There are several frontend methods for serving the right image, including media queries for background images and the <picture> and <source> elements and srcset attribute for foreground images.

Using CSS @media queries, you can define which background image to use based on viewport size and screen density. For example, you can send lowres.jpg to an old laptop and hires.jpeg to the iPad Pro:

header {
   background-image: url(img/hires_header.jpg);
}

@media only screen and (min-device-pixel-ratio: 2) and (min-width: 1024px) {
   header {
       background-image: url(img/hires_header.jpg);
   }
}

But what about foreground images?

The clown car technique for responsive images from early 2013 took advantage of the fact that SVG uses its own container width and height in CSS media queries rather than the browser itself as the viewport size. Fortunately, with the current state of browser support for <picture> and srcset, that hack and the picturefill polyfill are no longer needed.

The main issue with foreground images is that the browser has access to the viewport size and the screen density, but before loading the page the browser does not know the image size relative to the viewport or the source files’ dimensions. But the developer knows these dimensions and can include them.

Similar to @media queries, with <picture>, <source>, and the srcset attribute you can dictate which foreground image to use based on viewport size and screen density:

<picture>
   <source srcset="small_lowres.jpg, small_highres.jpg 2x" media="(max-width: 768px)">
   <source srcset="default.jpg, default_highres.jpg 2x">
   <source srcset="large_lowres.jpg, large_highres.jpg 2x" media="(min-width: 1024px)">
   <img src="default.jpg" alt="image descriptor">
</picture>

Note: Always include a default <img> within each <picture>, including an alt attribute describing the image within each <img>.

You can make a simple <img> element pick the best resource, without the <picture> parent and <source> siblings, by including a srcset attribute along with the sizes attribute.

<img src="default.jpg"
  srcset="large.jpg 1024w,
        medium.jpg 768w,
        default.jpg 420w"
  sizes="(min-width: 1024px) 1024px,
       (min-width: 768px) 90vw,
       100vw"
  alt="image descriptor" />

We can also provide different image media types with the type attribute:

<picture>
   <source srcset="photo.jxr" type="image/vnd.ms-photo">
   <source srcset="photo.jp2" type="image/jp2">
   <source srcset="photo.webp" type="image/webp">
   <img srcset="photo.jpg" alt="My beautiful face">
</picture>

In case the above is new to you, JPEG-XR and the old MIME type image/vnd.ms-photo are for Windows Media Photo, a proprietary Microsoft image format, supported in IE8+ and Microsoft Edge browsers. JPEG 2000‘s extension is jp2, and will be displayed in Safari browsers when the above markup is used. WebP is a lossless and lossy compression image format supported in Opera and Chrome. Firefox will use the default, be that PNG-A, SVG, GIF or JPEG, as in the code snippet above.

Firefox and IE8 need to be served the JPEG or PNG fallback. While both Safari and Firefox are experimenting with supporting WebP images, there is no indication that support is forthcoming, according to CanIUse.com.

There are some great online tutorials for media queries, <picture> and srcset if you want to dig deeper and implement any of these newer but well-supported features.

Almost unlimited configurations

One of the main issues with the code snippets in the section above is we’re only including a small subset of the possible assets we need in order to target every media type, size, and pixel density combination. While including each breakpoint, resolution and media type for each asset is certainly doable, it’s onerous — I wouldn’t want to have to write them out by hand. Fortunately, creating all the assets a site could possibly need to employ can be automated.

The optimal solution is writing the most optimal asset call server side, based on three of the four following criteria: viewport size, device pixel density, browser media type support, and the image size relative to the viewport. Why three of four? Because you can still take advantage of the browser’s understanding of sourcesets, but you don’t want to include every possibly needed combination — just a few.

If your DOM is written to the browser solely with client-side JavaScript, as we see with many React-based applications, you can query the user agent and script in the single, correct image call, but your users may be staring at a partially loaded, non-interactive screen while the code is parsed (screenshot). In the more common scenario, when you DON’T write the entire site client-side, rewriting an asset call with JavaScript isn’t optimal: the browser will download the original asset when the HTML is parsed, then download the second, optimized asset based on the updated DOM.

If you are employing progressive enhancement, which you definitely should be, you need to be very cognizant that the original image will be downloaded when the HTML for that image is parsed.

Unfortunately, Client Hints, the specification adding device pixel ratio and viewport width information to a set of HTTP request header fields, has only been implemented in Blink (Chrome and Opera). When you know the user agent, resolution and viewport size, you can automate the updating of all image calls, progressively enhancing the requests server side.  With Client Hints we can to confirm support for webP (Chrome and Opera support both client hints and webP), resolution and viewport size. At Instart Logic, we have a script called the Nanovisor that provides for similar goals to Client Hints, enabling us to send the optimized assets in the optimized image format even when Client Hints isn’t supported.

While UA sniffing is often looked down upon, you can use the user agent string from HTTP headers along with a look-up table to determine which media type to return based on those headers. You can rewrite the file extension, or you can respond to each image call with the “right” image type for each individual browser, but with the “wrong” extension. For example, if a request is made for foo.jpg, serve the best media type for the browser making the request, but call it foo.jpg no matter the media type. At Instart Logic we serve webP to Chrome and Opera, JPEG-XR to Edge, etc., without rewriting the image path. Instead, we simply employ the original file extension — in this case jpg — in the file name. We take advantage of the fact that all browsers render images of media types they understand, no matter the extension (or lack thereof) in the file name. In this way, you can keep the same markup, with no JavaScript DOM manipulations, while only needing to download a single image per image request.

If you are going to serve different media types based on browser support but with the same file extension, you likely have to serve the correct image server-side, as you’ll need to make use of a datastore to handle the various image versions all with the exact same file name. To be done client side, you would need to make the request for the right extension. (If you find a better or just different solution, please let me know).

Automating the asset creation process

Site visitors are best served by individually-tuned images for the individually tuned image declarations browsers now support. The previous section covered how to tell the browser to use an image of a specific size and type based on the browser features. However, we still need to create the images for all the needed sizes for all the image media types with the best compression level appropriate for each individual image. The question now is “how do we create all those images?”

What is needed is automation of image compression and resizing in the multiple formats supported by the various browsers, devices and operating systems, preferably in a way that can figure out the best compression ratio and levels for each individual image.

Optimizing images can be time consuming for non-static sites that may need to scale up to thousands, hundreds of thousands, or even millions of images, with the different sizes, resolutions and media types. Software like ImageMagick can be used to convert images into PNG, JPEG, JPEG-2000, GIF, WebP and almost any other image format via the command line:

convert myImg.jpg -quality 78 -define webp:lossless=true myImg.webp

The above command will save a converted copy of myImg.jpg in WebP format; the command encodes a lossless converted copy of the image file at a quality of 78%.

With ImageMagick, you can resize an image while converting it (or without converting it) with the -resize flag:

convert myImg.jpg -quality 78 -resize 50% myImg.webp

The above command creates a new file, converting the JPEG to WebP, saving it at 78% of the original quality and at half the height and width of the original, maintaining the original aspect ratio.

Via the command line, it is possible to create all the different images types and sizes you may need. Automating the type and size conversion of thousands of images is doable. What is less practical and not easily doable is determining the right compression quality for each image. This is a step that usually must be done manually.

At web-scale, taking the time to find the best quality settings for each individual image would take a large army of people. Most image transcoding tools and services reduce image file size by reducing the quality of images by a single compression ratio or level for all images. For example, when I had sites with just a few images I could take the time to individually compress images; the quality settings ranged from a low of about 35% to a high of 88%. For galleries and other projects with a plethora of images, I used Adobe Fireworks (I’m dating myself) to automate image compression and export, saving all JPEGs to a quality setting of 78%. Like my hokey efforts, most automated compression tools have a single quality setting, usually set to around 80%. My guess to compress to 78% was pseudo-random (it was slightly based on experience). There is no true “one size fits all” magic image quality value: no magic export compression level which works best for all images all the time.

The “right” compression depends on the content and usage of the image. Export quality levels generally depend upon details contained within an image. Depending on the content, you’ll generally set a different level for each image. Often the more detail contained in an image, the lower the quality: scenery may need a higher quality setting of over 90, whereas an image with intricate detail might look fine at level even lower than 50.

It is not impossible to automate per-image quality adjustments. Instart Logic’s web delivery platform uses computer vision, machine learning and image transcoding to enable content-aware image optimizations, automating the serving of images optimized by device resolution, browser capability, and network capacity and optimal image compression ratios, without sacrificing user experience. What we call “SmartVision” is an algorithmic approach, using machine learning to automate adaptive settings for individual images, compressing each image at the best possible compression ratio set, without anyone having to eyeball the image to figure out what that ratio is (with the exception of the first few images to calibrate the settings of the machine learning algorithm). The advanced computer vision algorithms “look at” the content of an image and intelligently maximizes compression levels without impacting user experience.  Parvez Ahammad explained his algorithms for SmartVision much better than I ever could back in 2014.

Browser’s have greatly improved over the past 6 years, as has the size of the average site download size. Images are the main culprit of this bloat, and modern image solutions are part of the solution. Taking the time to optimize your image assets is time well spent.