Get Support from the Top

If you’re lucky like me, your CEO is already on the web performance bus. It might even have been their idea to focus on performance, and you’ve been recruited to lead the charge. If this isn’t your situation, you have to start your evangelism at the top. You might start with the CEO, or perhaps COO or an SVP. The key is it has to be someone who is a leader across the different organizations within your company. The culture shift to focus on performance doesn’t happen in engineering alone. It has to happen across product management, marketing, sales, and all other parts of the company. You need to identify who the key leader or leadership team is and get them excited about web performance, and make them believe in the benefits it delivers.

Speak the Right Vocabulary

As an engineer you probably know how to sell to other engineers. “Optimization” makes a developer’s ears perk up. Speaking in terms of reduced regressions and fewer outages wins over folks in devops. But you also need to know how to speak across the organization both horizontally and vertically. The UX team likes hearing about better user metrics (longer sessions, more sessions per month). The folks in finance wants to hear about reduced operating costs in terms of hardware, power consumption, and data center bandwidth. Marketing and sales will light up hearing case studies about doubling unique users from search engine marketing as a result of a faster website. Make sure to use terms that resonate with your audience.

A key skill in evangelizing to upper management is knowing how to speak hierarchically – start with the high-level stats and drill down into the details if the need arises. I see many engineers who start with the details which many folks don’t have the time or background to follow. Start by showing a median and save the logarithmic scale charts in the “more slides” section.

Pick the Right Product

If you’ve convinced the senior execs to focus on performance, your next step is to pick a product to focus on. You want to pick a high visibility product, so that the wins are significant. But you don’t want to pick the company’s flagship product. It’s possible you might hit a few bumps on your first forays into adopting web performance. Also, you might have to alter the release cycle as you rollout metrics and start A/B testing. That’s harder to do with a product that’s the company’s cash cow. Start with a product that’s in the top 5 or 10, but not #1.

Pick the Right Team

The team you choose to work with is even more important than which product you choose. It all comes down to people, and if the team is too busy, has other priorities, or simply doesn’t believe in WPO (Web Performance Optimization) then you should move on to another team. You can always come back and revisit this team in the future.

I always have a kickoff meeting with a team that’s interested in working on performance, and I ask how many people they can dedicate to work on performance. I’m usually looking for at least two people full-time for 3 months. Sometimes teams think it’s sufficient to have someone spend 20% of their time working on performance, but this usually doesn’t have a positive outcome. If this is the company’s first engagement with web performance, you want to make sure to pick a team that has the mindset and resources to focus on the work ahead.

Pick the Right Task

It’s critical that the first performance optimization deployed has a significant impact. There’s nothing more frustrating than getting folks excited about WPO only to have their work show no improvement. For most websites it’s fairly straightforward to pick an optimization that will have a big impact. I’m reminded of Ismail Elshareef‘s case study about Edmunds.com getting 80% faster. He talks about how the first task they picked was making resources cacheable. After just a day of work they pushed the fix and cut their CDN traffic by 34%! This is the type of win you want to have right out of the gate – something that takes a small amount of work and makes a big improvement.

Start with Metrics

I’ve had several engagements where teams got so excited about the optimizations, they started deploying fixes before the metrics were in place. This is bad for two reasons. Without metrics you’re flying blind so you don’t know the actual impact of any fixes. But more importantly, it’s likely that the first fixes you deploy will have the biggest impact. If the metrics aren’t in place then you miss out on quantifying your best work! It’s best to establish the baseline when the site is at its worst. Occasionally teams don’t want to do this because they’re embarrassed by the slowness of the site. Just remind them how happy the execs will be to see a chart showing the site getting twice as fast.

Identify Your Replacement

Within the chosen team there needs to be someone who is aligned with you to take over your role. This is the person who keeps the team focused on performance after you’ve moved on to help the next team. He is the one who tracks the dashboards, identifies the changes that were deployed, analyzes the A/B test results, and prioritizes the next optimizations to work on.

It’s not scalable for you to be the only performance expert in the company. You want to build a virtual performance team that spans all the products in the company. At Yahoo! we called this team the SpeedFreaks. We had regular gatherings, a mailing list, etc. It was a great way to share lessons learned across the different teams and re-energize our excitement about making things faster.

Get Everyone on Board

Making and keeping the website fast requires everyone to be thinking about performance. It’s important to keep the entire company involved. There are several ways of doing this. One technique I see often is having realtime dashboards deployed at large gathering spots in the company. The Wall of Fame is another good chart. Eventually teams that are always at the bottom will start wondering what they have to do to get to the top. Getting time during the company all-hands meetings to review current performance and highlight some wins is good. Adding performance (speed) to the annual performance (HR) review form makes everyone think about their contributions in the past and plan on how they can contribute in the future.

Use Carrot over Stick

If you follow these tips it’s likely that you’ll start off having successful engagements evangelizing and deploying performance best practices at your company. After working with the choicest teams, however, it’s also likely that you’ll run into a team that just isn’t drinking the WPO kool-aid. This is more likely to happen at larger companies where it’s more challenging to create a cultural shift. If you can’t convince this team to apply the right amount of focus, one possible reaction is to bring in a senior exec to command them to make performance a priority and do the work. This might work in the short term, but will fail in the long term and might even set you further back from where you started.

Performance is a way of thinking. It requires vigilance. Anyone who has it forced upon them will likely not value performance and will instead look upon it as a nuisance that took them away from their desired focus. This person is now even harder to win over. It’s better to avoid taking the “stick” approach and instead use the “carrot” as motivation – t-shirts, bonuses, executive praise, shout outs at company all-hands, etc. No one enjoys the stick approach – the team doesn’t enjoy it and neither will you. Everyone comes away with negative memories. The carrot approach might not work in the short term, but it leaves the door open for a more positive re-engagement in the future.

Be Passionate

It’s likely you’re the “performance lead” within the company, or at least the one who cares the most about making performance a high priority. It’s not going to be easy to get everyone else on board. You can’t go about this half heartedly. You have to be passionate about it. John Rauser spoke (passionately) about this in Creating Cultural Change at Velocity 2010. He says you have to be excited, and relentless. I agree.

Creating a culture of performance at your company is about creating a culture of quality. This is especially true because best (and worst) practices propagate quickly at web companies. Code written for product A is reused by product B. And folks who worked on team A transfer over to team C. If product A is built in a high performance way, those best practices are carried forward by the code and team members. Unfortunately, bad practices spread just as easily.

Companies like Google, Etsy, and Betfair have gone so far as to publish their commitment to performance. This is a win for their customers and for their brand. It’s also a win for the performance community because these companies are more likely to share their best practices and case studies. If your company is focused on performance, please help the community by sharing your lessons learned. If your company doesn’t have a focus on performance, I hope these tips help you establish that WPO focus to create a website that has a better user experience, more traffic, greater revenue, and reduced operating expenses.

There are many ways to prefetch content, but here are 3 simple options.

DNS Prefetching

DNS is the protocol that converts human readable domains (mysite.com) into computer readable IPs (123.123.123.123). DNS resolution is generally pretty fast and measured in 100′s of milliseconds, but because it must happen before any request to the server can be made it can cause a cascade effect that has a real impact on the overall load time of a page. Often we know about several other domains that will need to be loaded for resources later in the page or user session, such as subdomains for static content (images.mydomain.com) or domains for 3rd party content. Some browsers support a meta tag that identifies these domains that need to be resolved so the browser can resolve them ahead of time. The tag to do this is pretty straight forward:

<link href="//my.domain.com" rel="dns-prefetch" />
<link href="http://my.domain.com/" rel="prefetch" /> <!– IE9+ –>

Adding this tag causes the browser to do the DNS resolution ahead of time, instead of waiting until a resource requires it later. This technique is probably most valuable to preload DNS for content on other pages on your site that visitors are likely to go to. This feature is supported in Chrome, Firefox, and IE9+.

Although shaving a few hundred milliseconds might seem trivial, in aggregate this can be a measurable gain. It’s also a safe optimization and easy to implement. I was curious to see how often this technique is used, so I crawled the top 100K Alexa sites. It turns out only 552 sites (0.55%) are currently using DNS prefetching. This is a cheap win, and something more sites should leverage.

Resource Prefetching

Images make up a large portion of the overall bytes of many major websites today. Often the overhead of making the requests and downloading images can have a significant performance impact. In many cases, though, the site developer knows when an image will be needed that won’t be detected early by the browser, such as an image loaded from an ajax request or other user action on the page. Resource prefetching is when you load an image, script, stylesheet, or other resource into the browser preemptively. This is most often done with images, but can be done with any type of resource that can be cached in the browser.

Of the three techniques I’m covering here, this is by far the oldest and the most used. Unfortunately I can’t give a concrete number about adoption because there are too many ways to implement this to detect in my Alexa crawl. Still, many sites don’t properly leverage this technique and even just preloading a few images can make a huge difference for the user experience.

Page Prefetching / Prerendering

Page prefetching is very similar to resource prefetching, except that we actually load the new page itself preemptively. This was first made available in Firefox. You can hint to the browser that a page (or an individual resource) should be prefetched by including the following tag:

<link rel="prefetch" href="/my-next-page.htm">

In the case of prerendering, the browser not only downloads the page, but also the necessary resources for that page. It also begins to render the page in memory (not visible to the user) so that when the request for the page is made it can appear nearly instantaneous to the user. Prerendering was first added in Chrome. You can hint that a page should be prerendered by including the following tag:

<link rel="prerender" href="http://mydomain.com/my-next-page.htm">

This technique is by far the most controversial and the riskiest of the three. Prerendering a page should only be done when there is a high confidence that the user will go to that page next. The most well known example of this is Google Search, which will prerender the first result of the page if the confidence is high enough. I found only 95 examples of this in my crawl of the Alexa Top 100k sites. Although this technique is clearly not for every use case, I think many more sites could leverage this to improve the user experience.

The Downsides

Prefetching in general is often a controversial topic. Many people argue that it is not efficient and leads to a waste in bandwidth. It also uses client resources unnecessarily (most notably on mobile devices). Also worth mentioning is that in some cases prefetching or prerendering of pages can have adverse effects on analytics and log tracking since there is no obvious way to discern a user visiting the page (and seeing it) or simply the browser prerendering without the user’s knowledge.

Conclusion

Despite all of these cautions, prefetching can be a huge win. The fastest request is always the one we never have to make and getting as much into the cache as possible is the best way to make that happen. By making these expensive requests when the user is not waiting on them, we can greatly improve the perceived performance of even the slowest sites on the slowest networks. If you’re not already doing so, it’s worth trying these techniques on your site. The results will vary, so be sure to use Real User Measurement (e.g. Torbit) to find out how much of an improvement prefetching makes for you.

Most of the focus today on improving mobile performance is on Layer 7 with front end optimizations (FEO). At CloudFlare, we’ve done significant work in this area with front end optimization technologies like Rocket Loader, Mirage, and Polish that dynamically modify web content to make it load quickly whatever device is being used. However, while FEO is important to make mobile fast, the unique characteristics of mobile networks also means we have to pay attention to the underlying performance of the technologies down at Layer 4 of the network stack.

This article is about the challenges mobile devices present, how the default TCP configuration is ill-suited for optimal mobile performance, and what you can do to improve performance for visitors connecting via mobile networks. Before diving into the details, a quick technical note. At CloudFlare, we’ve built most of our systems on top of a custom version of Linux so, while the underlying technologies can apply to other operating systems, the examples I’ll use are from Linux.

TCP Congestion Control

To understand the challenges of mobile network performance at Layer 4 of the networking stack you need to understand TCP Congestion Control. TCP Congestion Control is the gatekeeper that determines how to control the flow of packets from your server to your clients. Its goal is to prevent Internet congestion by detecting when congestion occurs and slowing down the rate data is transmitted. This helps ensure that the Internet is available to everyone, but can cause problems on mobile network when TCP mistakes mobile network problems for congestion.

TCP Congestion Control holds back the floodgates if it detects congestion (i.e. packet loss) on the remote end. A network is, inherently, a shared resource. The purpose of TCP Congestion Control was to ensure that every device on the network cooperates to not overwhelm its resource. On a wired network, if packet loss is detected it is a fairly reliable indicator that a port along the connection is overburdened. What is typically going on in these cases is that a memory buffer in a switch somewhere has filled beyond its capacity because packets are coming in faster than they can be sent out and data is being discarded. TCP Congestion Control on clients and servers is setup to “back off” in these cases in order to ensure that the network remains available for all its users.

But figuring out what packet loss means on a mobile network is a different matter. Radio networks are inherently susceptible to interference which results in packet loss. If packets are being dropped does that mean a switch is overburdened, like we can infer on a wired network? Or did someone travel from an under-subscribed wireless cell to an oversubscribed one? Or did someone just turn on a microwave? Or maybe it was just a random solar flare? Since it’s not as clear what packet loss means on a mobile network, it’s not clear what action a TCP Congestion Control algorithm should take.

A Series of Leaky Tubes

To optimize networks for lossy networks like those on mobile networks, it’s important to understand exactly how TCP Congestion Control algorithms are designed. While the high level concept makes sense, the details of TCP Congestion Control are not widely understood by most people working in the web performance industry. That said, it is an important core part of what makes the Internet reliable and the subject of very active research and development.

To understand how TCP Congestion Control algorithms work, imagine the following analogy. Think of your web server as your local water utility plant. You’ve built on a large network of pipes in your hometown and you need to guarantee that each pipe is as pressurized as possible for delivery, but you don’t want to burst the pipes. (Note: I recognize the late Senator Ted Stevens got a lot of flack for describing the Internet as a “series of tubes,” but the metaphor is surprisingly accurate.)

Your client, Crazy Arty, runs a local water bottling plant that connects to your pipe network. Crazy Arty’s infrastructure is built on old pipes that are leaky and brittle. For you to get water to them without bursting his pipes, you need to infer the capability of Crazy Arty’s system. If you don’t know in advance then you do a test — you send a known amount of water to the line and then measure the pressure. If the pressure is suddenly lost then you can infer that you broke a pipe. If not, then that level is likely safe and you can add more water pressure and repeat the test. You can iterate this test until you burst a pipe, see the drop in pressure, write down the maximum water volume, and going forward ensure you never exceed it.

Imagine, however, that there’s some exogenous factor that could decrease the pressure in the pipe without actually indicating a pipe had burst. What if, for example, Crazy Arty ran a pump that he only turned on randomly from time to time and you didn’t know about. If the only signal you have is observing a loss in pressure, you’d have no way of knowing whether you’d burst a pipe or if Crazy Arty had just plugged in the pump. The effect would be that you’d likely record a pressure level much less than the amount the pipes could actually withstand — leading to all your customers on the network potentially having lower water pressure than they should.

Optimizing for Congestion or Loss

If you’ve been following up to this point then you already know more about TCP Congestion Control than you would guess. The initial amount of water we talked about in TCP is known as the Initial Congestion Window (initcwnd) it is the initial number of packets in flight across the network. The congestion window (cwnd) either shrinks, grows, or stays the same depending on how many packets make it back and how fast (in ACK trains) they return after the initial burst. In essence, TCP Congestion Control is just like the water utility — measuring the pressure a network can withstand and then adjusting the volume in an attempt to maximize flow without bursting any pipes.

When a TCP connection is first established it attempts to ramp up the cwnd quickly. This phase of the connection, where TCP grows the cwnd rapidly, is called Slow Start. That’s a bit of a misnomer since it is generally an exponential growth function which is quite fast and aggressive. Just like when the water utility in the example above detects a drop in pressure it turns down the volume of water, when TCP detects packets are lost it reduces the size of the cwnd and delays the time before another burst of packets is delivered. The time between packet bursts is known as the Retransmission Timeout (RTO). The algorithm within TCP that controls these processes is called the Congestion Control Algorithm. There are many congestion control algorithms and clients and servers can use different strategies based in the characteristics of their networks. Most of Congestion Control Algorithms focus on optimizing for one type of network loss or another: congestive loss (like you see on wired networks) or random loss (like you see on mobile networks).

In the example above, a pipe bursting would be an indication of congestive loss. There was a physical limit to the pipes, it is exceeded, and the appropriate response is to back off. On the other hand, Crazy Arty’s pump is analogous to random loss. The capacity is still available on the network and only a temporary disturbance causes the water utility to see the pipes as overfull. The Internet started as a network of wired devices, and, as its name suggests, congestion control was largely designed to optimize for congestive loss. As a result, the default Congestion Control Algorithm in many operating systems is good for communicating wired networks but not as good for communicating with mobile networks.

A few Congestion Control algorithms try to bridge the gap by using the time of the delay in the “pressure increase” to “expected capacity” to figure out the cause of the loss. These are known as bandwidth estimation algorithms, and examples include Vegas, Veno and Westwood+. Unfortunately, all of these methods are reactive and reuse no information across two similar streams.

At companies that see a significant amount of network traffic, like CloudFlare or Google, it is possible to map the characteristics of the Internet’s networks and choose a specific congestion control algorithm in order to maximize performance for that network. Unfortunately, unless you are seeing the large amounts of traffic as we do and can record data on network performance, the ability to instrument your congestion control or build a “weather forecast” is usually impossible. Fortunately, there are still several things you can do to make your server more responsive to visitors even when they’re coming from lossy, mobile devices.

Compelling Reasons to Upgrade Your Kernel

The Linux network stack has been under extensive development to bring about some sensible defaults and mechanisms for dealing with the network topology of 2012. A mixed network of high bandwidth low latency and high bandwidth, high latency, lossy connections was never fully anticipated by the kernel developers of 2009 and if you check your server’s kernel version chances are its running a 2.6.32.x kernel from that era.

uname -a

There are a number of reasons that if you’re running an old kernel on your web server and want to increase web performance, especially for mobile devices, you should investigate upgrading. To begin, Linux 2.6.38 bumps the default initcwnd and initrwnd (inital receive window) from 3 to 10. This is an easy, big win. It allows for 14.2KB (vs 5.7KB) of data to be sent or received in the initial round trip before slow start grows the cwnd further. This is important for HTTP and SSL because it gives you more room to fit the header in the initial set of packets. If you are running an older kernel you may be able to run the following command on a bash shell (use caution) to set all of your routes’ initcwnd and initrwnd to 10. On average, this small change can be one of the biggest boosts when you’re trying to maximize web performance.

ip route | while read p; do ip route change $p initcwnd 10 initrwnd 10; done

Linux kernel 3.2 implements Proportional Rate Reduction (PRR). PRR decreases the time it takes for a lossy connection to recover its full speed, potentially improving HTTP response times by 3-10%. The benefits of PRR are significant for mobile networks. To understand why, it’s worth diving back into the details of how previous congestion control strategies interacted with loss.

Many congestion control algorithms halve the cwnd when a loss is detected. When multiple losses occur this can result in a case where the cwnd is lower than the slow start threshold. Unfortunately, the connection never goes through slow start again. The result is that a few network interruptions can result in TCP slowing to a crawl for all the connections in the session.

This is even more deadly when combined with tcp_no_metrics_save=0 sysctl setting on unpatched kernels before 3.2. This setting will save data on connections and attempt to use it to optimize the network. Unfortunately, this can actually make performance worse because TCP will apply the exception case to every new connection from a client within a window of a few minutes. In other words, in some cases, one person surfing your site from a mobile phone who has some random packet loss can reduce your server’s performance to this visitor even when their temporary loss has cleared.

If you expect your visitors to be coming from mobile, lossy connections and you cannot upgrade or patch your kernel I recommend setting tcp_no_metrics_save=1. If you’re comfortable doing some hacking, you can patch older kernels.

The good news is that Linux 3.2 implements PRR, which decreases the amount of time that a lossy connection will impact TCP performance. If you can upgrade, it may be one of the most significant things you can do in order to increase your web performance.

More Improvements Ahead

Linux 3.2 also has another important improvement with RFC2099bis. The initial Retransmission Timeout (initRTO) has been changed to 1s from 3s. If loss happens after sending the initcwnd two seconds waiting time are saved when trying to resend the data. With TCP streams being so short this can have a very noticeable improvement if a connection experiences loss at the beginning of the stream. Like the PRR patch this can also be applied (with modification) to older kernels if for some reason you cannot upgrade (here’s the patch).

Looking forward, Linux 3.3 has Byte Queue Limits when teamed with CoDel (controlled delay) in the 3.5 kernel helps fight the long standing issue of Bufferbloat by intelligently managing packet queues. Bufferbloat is when the queuing overhead on the network stack becomes backed up because its littered with stale data. Linux 3.3 has features to auto QoS important packets (SYN/DNS/ARP/etc.,) keep down buffer queues thereby reducing bufferbloat and improving latency on loaded servers.

Linux 3.5 implements TCP Early Retransmit with some safeguards for connections that have a small amount of packet reordering. This allows connections, under certain conditions, to trigger fast retransmit and bypass the costly Retransmission Timeout (RTO) mentioned earlier. By default it is enabled in the failsafe mode tcp_early_retrans=2. If for some reason you are sure your clients have loss but no reordering then you could set tcp_early_retrans=1 to save one quarter a RTT on recovery.

One of the most extensive changes to 3.6 that hasn’t got much press is the removal of the IPv4 routing cache. In a nutshell it was an extraneous caching layer in the kernel that mapped interfaces to routes to IPs and saved a lookup to the Forward Information Base (FIB). The FIB is a routing table within the network stack. The IPv4 routing cache was intended to eliminate a FIB lookup and increase performance. While a good idea in principle, unfortunately it provided a very small performance boost in less than 10% of connections. In the 3.2.x-3.5.x kernels it was extremely vulnerable to certain DDoS techniques so it has been removed.

Finally, one important setting you should check, regardless of the Linux kernel you are running: tcp_slow_start_after_idle. If you’re concerned about web performance, it has been proclaimed sysctl setting of the year. It can be enabled in almost any kernel. By default this is set to 1 which will aggressively reduce cwnd on idle connections and negatively impact any long lived connections such as SSL. The following command will set it to 0 and can significantly improve performance:

sysctl -w tcp_slow_start_after_idle=0

The Missing Congestion Control Algorithm

You may be curious as to why I haven’t made a recommendation as far as a quick and easy change of congestion control algorithms. Since Linux 2.6.19, the default congestion control algorithm in the Linux kernel is CUBIC, which is time based and optimized for high speed and high latency networks. Its killer feature, known as called Hybrid Slow Start (HyStart), allows it to safely exit slow start by measuring the ACK trains and not overshoot the cwnd. It can improve startup throughput by up to 200-300%.

While other Congestion Control Algorithms may seem like performance wins on connections experiencing high amounts of loss (>.1%) (e.g., TCP Westwood+ or Hybla), unfortunately these algorithms don’t include HyStart. The net effect is that, in our tests, they under perform CUBIC for general network performance. Unless a majority of your clients are on lossy connections, I recommend staying with CUBIC.

Of course the real answer here is to dynamically swap out congestion control algorithms based on historical data to better serve these edge cases. Unfortunately, that is difficult for the average web server unless you’re seeing a very high volume of traffic and are able to record and analyze network characteristics across multiple connections. The good news is that loss predictors and hybrid congestion control algorithms are continuing to mature, so maybe we will have an answer in an upcoming kernel.

Yet, when we look at the actual numbers “in the wild”, we see that few developers actually compress their images, and even for those that do, the results are not always ideal.

A few months ago, I downloaded 5.8 million images from Alexa’s top 200,000 sites. Using that image data, I’ll demonstrate how much data can be saved by properly compressing images.

Image Formats

I’m sure most of you know this by now, but here is a short overview of the image formats on the Web:

• GIF – Best suited for computer generated images with relatively few number of colors. It works by choosing a palette of up to 256 colors that best fits the image, creating a bitmap that represents the image using the palette’s color numbers, and then compressing that bitmap using a generic compression algorithm. The format supports animation and transparency, but not a full alpha channel.
• PNG – Best suited for computer generated images, but can represent more than 256 colors. The format has several subtypes. The subtype usually referred to as PNG8 is very similar to GIF, but uses a different compression algorithm. It does not support animation, but does support a full alpha channel. The subtypes referred to as PNG24 and PNG24α can represent the full RGB color space, with the latter also supporting a full alpha channel. The downside is that both PNG24 subtypes are represented as bitmaps to which a generic compression algorithm is applied. This is usually not ideal in terms of byte size.
• JPEG – Best suited for real life photos. It is not a bitmap based format, but represents the images by storing the frequency of color changes between different pixels, eliminating high frequencies that humans are likely not to notice anyway, and then compressing that. It is a lossy image format, which means a JPEG cannot be converted to the original bitmap image with perfect accuracy. For most uses on the Web, this is not a limitation.
• WebP – Best suited for both real life photos and computer generated images, since it can employ both lossy and lossless techniques. Based on the VP8 video codec, the WebP format uses predictive coding to achieve its high lossy compression rates and the latest entropy coding techniques to achieve better lossless results. It also supports a full alpha channel and animation.What’s the catch, then? The main issue is that WebP is not really part of the Web platform’s “official” formats since it is only supported by Chrome and Opera at the present. The lack of simple fallback mechanisms (both client and server side) poses a high barrier of entry for developers that want to use WebP today.

Here’s a look at the presence each format has on the Web today based on bytes.

Format Distribution

Some of you may say: “You forgot SVG!”. I didn’t. SVG comprise only 0.001% of the overall image data, so it didn’t make it into the format distribution table. Sad, but true.

Lossless Optimization

In my quest for finding image optimization opportunities, I first sought to find the savings that could be achieved without any compromise on quality. I ran lossless optimizations on JPEG and PNG using the jpegtran and pngcrush utilities, as well as conversion to lossless WebP. The results are below.

Overall with these lossless optimization techniques about 12.75% of image data can be saved. That is 101KB for an average page! If we use the lossless variant of WebP, we can save 18.2% of overall image data for browsers that support it, which is 144KB.

Lossy Optimization

Now let’s see what happens when we are willing to (slightly) compromise quality for the sake of data savings. I used the SSIM index in order to get an objective idea of the trade-off we make between quality and byte size. Basically, an SSIM score of 100% means identical images. Lower SSIM score means a bigger difference between the images.

JPEG

Using ImageMagick I compressed JPEGs several levels of quality. Then I applied the lossless optimizations that we saw above to them, in order to squeeze these images some more. I also compressed the images using imgmin which is a utility that deploys binary search to find the ideal quality level for each image. Finally, I ran JPEG to WebP conversion to see if the benefits match Google’s result of 30% data reduction.

WebP gives us compression levels close to “quality 30″ with “quality 75″ image quality. Another way to look at this is that WebP files are 37% smaller than the size of JPEGs with equivalent quality.

PNG24

I tried several lossy optimizations on these images: Kornel Lesiński‘s improved pngquant, conversion to JPEG using ImageMagick+jpegtran and conversion to WebP.

I’m not sure what’s more impressive here, pngquant’s 57.1% data reduction with practically zero quality loss, or JPEG’s and WebP’s results. Here again, the WebP files were 33% smaller than JPEG. Lossless WebP gave an extra 14.2% compression when applied to PNGs after pngquant. Note: I avoided converting PNGs smaller than 500 bytes to JPEG since this usually resulted in larger file sizes.

PNG24α

For PNGs with an alpha channel, I couldn’t use the above conversion to JPEG, since JPEG doesn’t have an alpha channel. Also, because of problems the original SSIM utility I used had with a full alpha channel, I used Kornel’s dssim utility instead.

Again, pngquant’s results are extremely impressive, providing files that are almost 3 times smaller with negligible quality loss. Lossless WebP gave an extra 15.8% compression on these pngquant results. Lossy WebP provides even better compression results with files that are 40% smaller than pngquant and almost 5 time smaller than the original PNGs, although it does that with slight visual quality loss.

Why Don’t Developers Compress Their Images?

While I have no evidence to support that theory, I suspect most developers don’t compress their images since there is no automated process in place. Depending on the workflow, there are a few options to automate image compression:

• Build time – For static images, adding image compression utilities to the build process can make sure that no static uncompressed images make it through.
• Upload time – For images that are dynamically added by the site’s users or administrators, the developers should find a way to add image compression utilities to the upload process. That may not always be easy (e.g. when working with legacy CMSs), but it is essential to avoid serving bloated images to users.
• Serving time – If neither of the previous options is feasible, there’s always the possibility to apply image compression before the images are served to the user. The open source option here is mod page speed‘s image optimization filters. Otherwise, commercial options are also available.

Each developer should choose the optimization options that fit his workflow best, but everyone should automate image optimization, otherwise there’s a strong chance it will not happen.

Conclusions

Even though every Web developer knows that images must be properly compressed, very few actually do that optimally, as we can see from the extra compression we can squeeze out of the Web’s images, with no or little compromise in terms of quality.

Even if developers only choose the truly lossless path, image data can be reduced by 12.75% or 100KB per page. Using lossless WebP turns that into 18.2% or 144KB for supporting browsers.

If every Web developer would employ maximal lossy and lossless techniques to compress his site’s images to the maximal extent, with practically non existent visual impact (i.e. imgmin for JPEG, pngquant for PNG24), the current average page size image data could be reduced by 37.8% or 300KB!

Willingness to apply more lossy techniques (but still maintain good visual quality), can result in 47.5% image data savings or 368KB.

Using WebP would increase the savings to 61% of image data or 483KB for browsers that support it.

That’s huge! Image compression is something that every one of us should add into our workflow, since it can save a large chunk of your site’s Web traffic. All the tools I used are free and open-source software. There are no excuses!

Bandwidth-wise, images are hogs. They are the largest average web site payload (62%), and they are most often the content bottleneck. When images arrive, they come tripping onto the page, pushing other elements around and triggering a clumsy repaint. They come “chop chop chop chop chop down” or you get nothing until suddenly “boom!” out of nowhere there it is. We all know what I’m talking about when I say “chop chop down” and “boom” and it makes us a little bit sick, because we sense how much time we’ve lost of our precious, short lives, waiting for pictures to download.

A missed opportunity

Photos are the main culprit when it comes to slow rendering. They are the most common type of image requested and on average weigh more. They are millions of colors and pixel depth is increasing. They are beautiful, and we don’t want to compromise on quality.

Web-optimized photos are jpegs, and jpegs come in two flavors: baseline and progressive. A baseline jpeg is a full-resolution top-to-bottom scan of the image, and a progressive jpeg is a series of scans of increasing quality. And that’s how they render; baseline jpegs paint top to bottom (“chop chop chop…”), and progressive jpegs quickly stake out their territory and refine (or at least that’s the idea).

Progressive jpegs are better because they are faster. Appearing faster is being faster, and perceived speed is more important that actual speed. Even if we are being greedy about what we are trying to deliver, progressive jpegs give us as much as possible as soon as possible. They assist us in our challenge of delivering big beautiful photos today.

Experimenting locally with a throttled bandwidth, an 80K progressive jpeg beats a 5K baseline jpeg (the same image, downsized) to the page in Firefox on Windows. This should blow your mind. Sure, the progressive jpeg’s first pass is low-resolution, but it contains as much information, or more, as the small image. And if you are zoomed out, perhaps on a mobile device, you will not notice it’s low-res. That’s responsive images working for us right now!

Basically, progressive jpegs are better. So what’s the most common type of jpeg online? You guessed it: baseline, and by a very wide margin. In a thousand-image sample, 92.6% are baseline.

No worries, we just need to declare progressive jpegs a best practice and get the rest of the world on-board with us. But in order to declare progressive jpeg a best practice, we need to be confident that it is. And to do so we need to first understand what browser support for this type of jpeg looks like today.

Reality Check #1

Progressive jpegs are displayed in all browsers, that’s not a worry. Our concern is how they render.

Behavior of progressive jpegs across browsers

These are disappointing results, but overall, market share and progressive rendering for progressive jpegs are trending upward. Support is currently at about 65% (Chrome + Firefox + IE9).

Unfortunately, the browsers that do not render progressive jpegs progressively render them all at once after download is complete, which makes them less progressive and slower than baseline jpegs. While baseline rendering is not as immediate and smooth as progressive rendering, at least it’s something while we wait, and the “chop chop” is a kind of progress indicator (a good thing). We can’t underestimate the reassurance we give users when they see something is happening.

By choosing progressive jpegs we are giving a majority of users an excellent experience and a minority — but a significant minority — a worse experience. But if we select baseline jpegs because it is a less poor experience in a minority of views, that’s a terrible compromise. We need to offer the best experience to our users, and look ahead.

Reality Check #2

You might ask “Aren’t progressive jpegs bigger than regular jpegs? Don’t we pay for the ‘layers’?” This is true for other types of interlaced images, but not jpegs. A progressive jpeg is usually a few kilobytes smaller than its baseline version. Plotting the savings of 10000 random baseline jpegs converted to progressive, Stoyan Stefanov discovered a valuable rule of thumb: files that are over 10K will generally be smaller using the progressive option.

It would be an easier sell if we could say progressive jpegs are always smaller, so always make progressive jpegs. Stoyan helps us out here. He says “One observation about the 10K rule is that when baseline is smaller, it’s smaller by a small margin. When progressive is smaller it’s usually a lot smaller. So it’s ok to say go 100% progressive and you’ll do better.”

That’s exactly what I wanted to hear! For all the baseline jpegs we’ve been serving, we’ve been missing opportunities in file size and perceived speed. Choosing the progressive option is win-win, and should always be the default. Then, after all jpegs are progressive, if we want to optimize further, it’s just a few bytes we’ll save and only on our smallest images.

The reason baseline jpegs are most common online, no doubt, is because image-optimization tools make them by default. However, all of the ones I looked at — Photoshop, Fireworks, ImageMagick, jpegtran — have a progressive option. Therefore, to serve progressive jpegs you’ll need to consciously modify your image optimization process.

I’d expect Smushit to translate baseline jpegs to progressive, and sure enough it does. (Smushit, btw, can be run from the command line and integrated into your image optimization process.)

How do you know if your jpegs are progressive? Here are a few ways to identify jpeg type:

1. ImageMagick — On the command line run: identify -verbose mystery.jpg | grep Interlace The output will either be “Interlace: JPEG” or “Interlace: None.”
2. Photoshop — Open file. Select File -> Save for Web & Devices. If it’s a progressive jpeg, the Progressive checkbox will be selected.
3. Any browser — Baselines jpegs will load top to bottom, and progressive jpegs will do something else. If the file loads too fast you may need to add bandwidth throttling. I use ipfw on my Mac.

Reality Check #3

According to this progressive jpeg FAQ, each progressive scan requires about the same amount of CPU as the entire baseline jpeg would take to render. This is not a concern for desktops but possibly for mobile devices.

The extra computation is a disadvantage but not a deal breaker. Delivering photos on small hardware is a challenge regardless. I know this because I’m writing a photo gallery application with infinite scrolling and it crashes on iPad. If you are handling a lot of images, you will have challenges on mobile anyway — different challenges.

As we’ve seen in the chart, Mobile Safari does not render progressive jpegs progressively anyway, and probably because they tax the CPU. But this is not a new image file format. Therefore, it wasn’t an option for browsers, even mobile browsers, to choose not to support progressive jpegs. Hopefully soon mobile browsers will leverage progressive rendering, but it makes sense why they currently don’t. It’s also a crying shame; we could really use the speed and file size savings progressive jpegs give us for mobile. When I said they are a kind of solution for responsive images right now, well, they would be, but aren’t yet.

Onward

In the last few days, Google got on-board with their Mod_Pagespeed service, making convert_jpeg_to_progressive a core filter. SPDY does as well, translating jpegs that are over 10K to progressive by default, following Stoyan’s rule of thumb. This will make browsers that support incremental display seem much faster. As you can see in the chart above that includes Google Chrome, so it makes sense that Google would make this choice. I’m not going to say that because “do-no-evil-make-the-web-faster” Google has selected progressive jpeg to be a best practice so should we. But it’s more data and validation. Most importantly, it shows that progressive jpeg — a format that has been in a kind of deep freeze for a decade — has staged a comeback.

And even though not all current browsers make use of progressive jpeg’s progressive rendering, the ones that do really benefit, and we get file size savings across the board. It’s our best option today and we should use it. Progressive jpegs are the future, not the past.

There are already a bunch of utilities and plugins out there which can detect possible SPOF vulnerabilities in a web application. The most notable ones being webpagetest.org, SPOF-O-Matic chrome plugin and YSlow 3PO extension. At eBay we wanted to detect SPOF at a very early stage, during the development cycle itself. This means an additional hook in our automated testing pipeline. The solution resulted in creating a simple tool which works on our test URLs and produces SPOF alerts. The tool is SPOFCheck.

SPOFCheck is a Command Line Interface (CLI) built in Node.js to detect possible Frontend SPOF for web pages. The output is generated in an XML format that can be consumed and reported by CI jobs. The tool is integrated with our secondary jobs, which run daily automation on a testing server where a development branch is deployed. In case of a SPOF alert, engineers are notified and they act on it accordingly. This process ensures that SPOFs are contained within the development cycle and do not sneak into staging or production.

The command line interface

SPOFCheck provides a simple command line interface and runs on Node.js

To install SPOFCheck run the following

$ npm install -g spofcheck

To run SPOFCheck, use the following format

spofcheck[options]*[urls]*Options
--help | -hDisplaysthisinformation.
--format=<format> | -f <format>             Indicatewhichformat[junit-xml | spof-xml | text]touseforoutput.
--outputdir=<dir> | -o <dir>                Outputsthespofresultstothisdirectory.
--rules=<rule[,rule]+> | -r <rule[,rule]+>  Indicatewhichrulestoinclude.
--print | -pOutputstheresultsinconsole,
                                              insteadofsavingtoafile.
--quiet | -qKeepstheconsoleclearfromlogging.

Example

$ spofcheck -f junit-xml -o /tests www.ebay.com www.amazon.com

Rules

SPOFCheck by default runs with 5 rules (checks). The rules are maintained in the rules.js file. New rules are easily added by pushing entries to the rules array or calling the spof API registerRules. The default rules come from Souders’s original list outlined below.

1. 3rdparty-scripts – Always load 3rd party external scripts asyncronously in a non-blocking pattern
2. application-js – Load application JS in a non-blocking pattern or towards the end of page
3. fontface-stylesheet – Try to inline @font-face style. Also make the font files compressed and cacheable
4. fontface-inline – Make sure the fonts files are compressed, cached and small in size
5. fontface-inline-precede-script-IE – Make sure inlined @font-face is not preceded by a SCRIPT tag, causes SPOF in IE

Output

SPOFCheck creates a file and writes results in one of these formats:

• junit-xml – a format most CI servers can parse (the default format)
• spof-xml – an XML format that can be consumed by other utilities
• text – a textual representation of the results

The format can be specified using the --format or -f option. For just printing results, i.e. no file creation, use the --print or -p option.

A SPOF Free World

Our primary goal was to eradicate frontend SPOF before it creeps in, and SPOFCheck is a step towards it. Go ahead, give SPOFCheck a try and integrate it with your CI environments. Let’s build a SPOF Free World .

Thanks to github projects spof-o-matic and 3po, a lot of the code logic has been re-used here. The design and packaging of the tool is based on csslint, thanks to Nicholas Zakas and Nicole Sullivan.

The definition of “interacting” will vary depending on your page. For some, this may be as simple as getting the text visible on the page, such that the user can begin consuming the information they requested (e.g. this page). For others, this may require wiring up dozens of JavaScript components to build up a JavaScript UI (e.g. Gmail). However, in both cases, there is one prerequisite: the user must be able to see the page, which is to say, the browser needs to render something to the screen.

So, with that in mind, what does it actually take to do a first content render in a modern browser?

DOM + CSSOM = Render Tree

The exact timing and behavior of the rendering pipeline will, of course, vary based on the parsing, layout and compositing pipelines of the browser. However, implementation differences aside, to get anything visible on the screen, all browsers must construct something resembling a “render tree”.

The parsing of the HTML document is what constructs the DOM. In parallel, there is an oft forgotten cousin, the CSSOM, which is constructed from the specified stylesheet rules and resources. The two are then combined to create the “render tree”, at which point the browser has enough information to perform a layout and paint something to the screen. So far, so good.

However, the diagram above shows an optimistic case: both the CSSOM and the DOM trees are shown as being constructed in parallel. This is where we must, unfortunately, introduce our favorite friend and foe – JavaScript.

• Synchronous JavaScript can issue a doc.write at any point; hence the DOM tree construction is blocked anytime a synchronous script is encountered
• JavaScript can query for a computed style of any object, which means it can also block on CSS

Instead of nice, parallel construction of the DOM and CSSOM objects shown in the earlier diagram, the two are now potentially intertwined: DOM construction can’t proceed until JavaScript is executed, and JavaScript can’t proceed until CSSOM is available. Yikes.

Depending on how this dependency graph is resolved on your pages, which is governed by how, and how many resources you include in that first “critical path” of the page load, the time to first render will vary accordingly. Can we get some metrics, or insights into this process? Turns out, yes we can!

Document Interactive & DOMContentLoaded

The HTML5 spec defines a well documented sequence of steps which the user agent must follow while constructing the page. Specifically, the end sequence captures two states, which can help answer our earlier question:

• The document is marked as “interactive” when the user agent stops parsing the document. Meaning, the DOM tree is ready.
• The user agent fires the DOMContentLoaded (DCL) event once any scripts marked with “defer have been executed, and there are no stylesheets that are blocking scripts. Meaning, the CSSOM is ready.

If no synchronous JavaScript is thrown into the mix, then the DOM and CSSOM construction can proceed in parallel. Things get more interesting once we introduce JavaScript into the picture.

If you add a script and tag it with “defer”, then you unblock the construction of the DOM: the document interactive state does not have to wait for execution of JavaScript. However, note that this same script will be executed before DCL is fired. Further, recall that JavaScript may query CSSOM, which means that the DCL event may be held until the CSSOM is ready, at which point the script will be executed. In short: we’ve unblocked the “document interactive” state, but we’re still potentially blocking DCL.

If you add a script and tag it with “async”, then you inherit similar behavior as above, but with one distinction: DCL does not have to wait for execution of async scripts!

The first important takeaway here is that by default, JavaScript will block DOM construction, which may block on CSSOM. Sync scripts are bad, but you already knew that. Marking scripts with “defer” and “async” makes an implicit promise to the document parser that you will not use doc.write, which in turn allows it to unblock DOM construction.

Second takeaways is: if at any point we must wait for JavaScript execution, then we will have to first wait for the CSSOM construction to finish. In other words, there is a hard dependency edge between JavaScript and CSS… Stylesheets at the top, scripts at the bottom? Now you know why.

Ok! This is all great in theory, but is this practical knowledge to help us optimize pages? Neither metric is a direct indicator of when the page will be painted, but monitoring either or both is a step in the right direction towards our ultimate goal of improving perceived performance.

Tracking the critical path of your page

If nothing else, monitoring “document interactive” will give you a good indicator of whether you are blocking DOM construction due to synchronous scripts. Sometimes, there is no way around this behavior, but this should be a known fact and a tradeoff, not an implicit “that’s how it works”.

The DCL event is also a critical milestone. Many popular libraries, such as JQuery, will begin executing their code once it fires. In other words, this is likely the first point at which your client code can begin interacting with the page, as well as provide meaningful feedback to the user. If you do your job right, then through the magic of progressive enhancement, you can get the skeleton of the page up, such that the user can begin interacting with the page while the browser continues to load the remaining assets. The IE team has an excellent example illustrating the difference between DCL and the window.onload events.

When does your DOMContentLoaded fire?

What you can measure, you can optimize. Even better, Navigation Timing spec already captures all the events we need: domInteractive, domContentLoadedEvent{Start,End}, and loadEvent{Start,End}. If you are already tracking the onload event already, then you might want to add the two events we’ve have covered here as well!

On that note, if you are using Google Analytics, then Christmas came early this year. The team recently added a new “DOM Timings” section. Guess which values it tracks? Yep.

Login into your GA account and head to “Content > Site Speed“. Once there, head to the “Performance” tab to see the timing histograms for all of your pages, or drill into the stats for a particular page. From there, you can track your document interactive, DCL, and onload events.

Just for fun, here is a side by side comparison of the DCL vs. onload histograms for my site:

The median time to DCL is under 1s, whereas the median for onload is ~1.5s. The relatively high DCL timing immediately tells me that there is likely a script that is blocking the construction of the DOM – something I should revisit. Having said that, the fact that there is a ~0.5s delta between DCL and onload tells me that I’m not forcing users to wait for all the assets to download before they can see some of the content.

When do your document interactive and DCL events fire?

In reality, there are many more things to consider for web applications when transferring web contents from Seoul to Buenos Aires. First of all, the line of optic fiber is definitely not straight. Thus the actual distance is much longer. Also there is 3 way handshake needed to establish a TCP connection between the two points. And then there can be multiple objects such as JavaScript, CSS and images to transfer in addition to HTML. Finally there are the actual time to process a request and provide a response in the web server.

According to some studies from Google, users consider anything more than 100ms to be a perceivable delay. i.e. if your web site does not respond within 100ms, it is considered to be slow. The interaction have to be smooth, just like flipping a magazine. However, as the numbers above have shown, it can be quite a challenge.

One of the more popular tricks to lower the latency for web applications is to use edge cache. The idea is to cache JavaScript, CSS and images in servers that are close to the user (i.e. the edge of the Internet). So the latency to access these resources will be much lower.

But what about the HTML page? Normally there are many information captured in a HTML page. Some (e.g. news article) are cacheable. Some (e.g. personal information) are not. That makes it hard to cache the entire HTML page. And that’s where Edge Side Include (ESI) can help.

ESI

So how can ESI help? We can take a look at the diagram below. Suppose we have a news article page with some user specific modules such as advertisement and recently read news. The edge cache will forward the user request for the article page to a special ESI service. The ESI service will then return a cacheable ESI document.The document will contain the content of the news article as well as one or more “ESI includes” for each of the user specific modules. The ESI includes are markup that instructs the edge cache to request these modules from origin servers. The edge cache will replace the ESI include markups with the actual contents of these requests before sending the final processed response back to user.

For subsequent requests from other users, the ESI document with the news article is already cached in the edge cache and the ESI includes will be processed for each user to request the corresponding personalized modules. The overall latency can be lower because much of the HTML page is coming from an edge cache server close to the user and only parts for the personalized module are coming directly from origin servers where the latency in between could be higher.

One can also use client side Ajax JavaScripts to fetch modules to display on the page. However, JavaScript may not always be supported (e.g. in cell phone) or enabled (e.g. user can disable it in browser) in all situations. Also this requires extra coding while the ESI solution returns a normal HTML page transparent to browser. Finally as a rule of thumb, we would always like to limit the number of HTTP connections from the client but the Ajax solution is actually creating more.

Performance Characteristics

There are two performance characteristics that we need to consider before we talk about how to start using ESI.

1. Concurrent Requests for ESI includes – When there are more than one ESI includes in the ESI document, we would want all these requests to be executed concurrently.
2. First Byte Flush – After the edge cache server executes the requests for the ESI includes, it should start flushing the ESI document to user till the first ESI include. Then when the request for that include is done with a response, the edge cache should flush the content of the include to user and the rest of the ESI document till the next ESI include is reached.

Where do I get started?

CDN vendors such as F5 and Akamai provide support for ESI. So if you are using services from these vendors, you can take advantage of that. Akamai even extended the original specification. You can read more about it on its web site.

If you aren’t using a CDN vendor, there are still other open source proxy software that support ESI, such as Varnish and Apache Traffic Server. Varnish supported a minimum subset of the ESI specification. However, error handling is missing. So when the ESI includes fail, you cannot use ESI markup to instruct Varnish to take alternative action, such as rendering an error message or fetch a different module. You can read more about the Varnish ESI support here. Apache Traffic Server curreently has an experimental ESI plugin available. It supports most of the specification and you can read more about it here.

In terms of the performance characteristics described above, Varnish supports first byte flush but not concurrent requests for ESI includes. And it is the opposite for Apache Traffic Server ESI plugin. Work is under way to provide first byte flush support for the Apache Traffic Server ESI plugin and it should be available soon.

Future & Conclusion

ESI can be a very powerful tool to lower latency for web sites and applications. However, it had been around for over 10 years without any change. And the markup is not powerful and expressive enough to support some modern use cases. e.g. If we want to render different modules for different device, we need to be able to use a custom function on the user agent request header to determine the device type and use different ESI includes for the modules. So an update on the specification is definitely needed and we should see some progress of it in 2013

For more information, you can also check out my presentation and the corresponding pdf on this topic at Velocity China 2012.

Performance is a vigilante task and as such one has to always keep an eye on the application performance monitoring. This is especially true after new releases when new features, bug fixes and other changes might unintentionally affect the application performance, eventually breaking end user experience.

Web Performance Optimization best practices should always be applied while developing an application. Whereas some tools might help identifying potential performance issues during the development cycle, it is a matter of where in the development cycle should WPO tools be run.

Worst case scenario: no instrumentation

In a development cycle without any instrumentation we have no idea how the application is performing for end users. Even worse we have no clue how good or bad the user experience is. In this scenario when a performance regression is introduced, the end user is the one having a bad experience and ultimately raising the red flag for performance. With luck some bad review will be published forcing us to reactively fix the issue and start over. This might last a few cycles until no one cares and then sadly the application is abandoned for good.

• Build the application
• Test to ensure nothing is broken
• Deploy to production
• Possible happy users and angry ones raising the red flag for performance
• Angry users write a bad review
• Reading the news, it’s time to improve performance and start over

Better case: RUM

Real User Measurement (RUM) is an essential piece of instrumentation for every web application. RUM gives the real status of what is going on for the user end. It provides valuable data such as bandwidth, page load times, etc. which allows monitoring and estimating what the end user experience is like. In the case of a performance regression, RUM tells when exactly the regression happens. Nevertheless, the end users are the one suffering with a bad experience. Reactively the issue should be fixed for a next cycle release.

• Build the application
• Test to ensure nothing is broken
• Deploy to production
• Possible happy and angry users
• Possible RUM raising the red flag for performance knowing end users get bad experience
• It’s time to improve performance and start over

YSlow

YSlow was initially developed to run manually in order to perform static performance analysis of the page, reporting any issues found based on a set of performance rules. There was some attempts at automation like hosting a real browser with YSlow installed as an extension and scheduling URLs to be loaded and analyzed.

Since 2011 YSlow has also been available from the command line for NodeJS using HAR files to perform the static analysis. As of early 2012 YSlow is also available for PhantomJS (Headless WebKit browser) which allows for static analysis of a URL loaded by PhantomJS and analyzed by YSlow reporting the results all via command line. YSlow for PhantomJS also provides two new output test formats: TAP and JUnit. Both techniques test all the rules based on a configurable threshold, producing an indication of exactly which tests pass.

Even better case: RUM + YSlow on CI

With the advent of YSlow for PhantomJS it becomes easy to integrate YSlow into the development cycle plugging it into the continous integration (CI) pipeline. If there is a performance regression, it breaks the build avoiding a potential performance regression from being pushed to production. This saves the users from ultimately getting a bad experience, as CI is the one raising the red flag for performance regressions. RUM will show no regression was introduced intentionally, however other causes might affect performance and RUM will notify something went wrong.

There is a comprehensive section on YSlow Wiki that explains how to plug YSlow + PhantomJS into Jenkins, but it’s also worth noting that the --threshold parameter is the ultimate way to configure the desired performance acceptance criteria for CI.

• Build the application
• Test to ensure nothing is broken
• Analyze build with YSlow to either pass or fail web performance acceptance criteria
• If it fails, proactively go back and fix the performance issue
• Once performance is fine, deploy to production
• Hopefully happy users only
• Keep monitoring RUM for performance issues
• It’s always time to improve performance and start over

Best case: RUM + YSlow on CI + WPT

For high performance applications in a well defined building cycle, YSlow scores might become stale always reporting A or B. This doesn’t tell much about smaller performance regressions and still might lead to some regression being pushed to the end user. It’s very important to keep monitoring RUM data in order to detect any unplanned variation. This is the most accurate info one can get about the user experience.

Once the YSlow score is satisfied, i.e., it doesn’t break the build, the next layer of proactive WPO is benchmarking the build in real browsers with a reasonable sample (the greater the better) to avoid variations. Use either the median or average of these runs. This should be compared to the current production baseline and within a certain threshold this should either pass or break the build to avoid fine performance regression.

To automate the benchmarking part, WebPagetest is a good fit and the WebPagetest API Wrapper can be used power NodeJS applications (more info on previous post).

• Build the application
• Test to ensure nothing is broken
• Analyze build with YSlow to either pass or fail web performance acceptance criteria
• If it fails, proactively go back and fix the performance issue
• Once YSlow is satisfied, it’s time to benchmark and compare against the current production baseline
• If it fails, proactively go back and fix the performance issue
• Once the two layers of performance prevention are satisfied, deploy to production
• Very likely lots of happy users only
• Keep monitoring RUM for other performance issues
• There’s always a few ms to squeeze in order to improve performance and start over

When comparing new builds (branches) to the production baseline, the ideal scenario is to have performance boxes as close as possible to production boxes (replica ideally) and performance WPT benchmarks in an isolated environment so test results are reproducible and don’t get too much deviation.

The last two cases: RUM + YSlow on CI vs RUM + YSlow on CI + WPT are quite similar to prevent performance regression whereas the latter gets more in-depth performance metrics to either pass or fail minimum performance acceptance criteria. RUM + YSlow on CI is analogous to going to the doctor for a routine checkup. The doctor asks a few questions and checks for heart beat amongst other superficial exams, eventually recommending lab exams. RUM + YSlow on CI + WPT on the other hand is analogous to going straight to the lab for a full body exam. It’s more invasive however more precise and would tell exactly what’s wrong.

Takeaway

Stop introducing performance regressions. Don’t let your end users be the ones raising the red flag for performance when you can proactively prevent regressions by simply plugging YSlow into the CI pipeline, and even better by benchmarking before releasing.

But where should we focus our attention when optimizing mobile web performance? We know that mobile networks have highly variable latency and bandwidth characteristics, and that in general mobile network latency is substantially higher than that on desktop connections. We also know that for modern networks, it is round trip time, not bandwidth, that is the dominating factor in page load time. Given this, to make the mobile web fast, our attention should be focused on minimizing the number of blocking round trips incurred before a web page can render its content to the device’s screen.

What blocks rendering a web page to the screen?

First, let’s look at the sequence of events that happens between the time a user initiates a page navigation and the time the browser can render that page to the screen. Round trips may be incurred for DNS resolution, TCP connection, and the request being sent to the server and the response being streamed back. Unfortunately, there’s not much developers can do to avoid these round trips. For repeat visitors, a longer DNS TTL can help, but TCP connection and request/response overhead will be incurred on every new navigation to a page (assuming there is no warm TCP connection ready to be reused by the client).

Once these initial round trips are incurred, the mobile device can begin parsing the HTML response. But the browser can’t paint content to the screen just yet. Before content in the HTML can be painted to the screen, the browser must construct the render tree to determine where the elements in the DOM will appear on screen. And before the render tree can be constructed, the DOM tree must be constructed. The DOM tree is constructed through a combination of parsing HTML and possibly JavaScript execution.

So what are the things that block parsing of HTML, DOM tree construction, and render tree construction? Most of the time, parsing, DOM tree, and render tree construction are very fast. However, there are a few antipatterns that can cause these processes to get blocked on the network.

Sources of delay during rendering: external JavaScript and CSS

The most significant source of delay during HTML parsing is external JavaScript. When a browser encounters a (non-async) external script during HTML parsing, it must halt parsing of subsequent HTML until that JavaScript is downloaded, parsed, and executed. This incurs additional round trips, which are especially expensive on mobile. If the script is loaded from a hostname other than the hostname the HTML was served from, additional round trips may be incurred for DNS resolution and TCP connection.

In addition, render tree construction gets blocked on stylesheets, so just as external JavaScript introduces delays during DOM tree construction, external stylesheets introduce delays during render tree construction.

In short, external JavaScript and CSS loaded early in the document (e.g. in the <head>) are performance killers, and they are especially expensive on mobile due to the higher round trip times associated with mobile networks.

Making mobile pages fast

To be fast, a mobile web page must include all of the content needed to render the above the fold region in the initial HTML payload without blocking on external JavaScript or CSS resources. Ideally, all the content needed to render the above the fold region should be in the first 15kB on the network (this is post-gzip-compression size; pre-gzip can be larger), since this is the size of the initial congestion window on modern Linux kernels. This does not mean simply inlining all of the JavaScript and CSS that used to be loaded externally. Instead, just the JavaScript and CSS needed to render the above the fold region should be inlined, and JavaScript or CSS needed to add additional functionality to the page should be loaded asynchronously. For instance, if we have a page like the following:

<html>
<head>
  <link rel="stylesheet" href="my.css">
  <script src="my.js"></script>
</head>
<body>
  <div class="main">
    Here is my content.
  </div>
  <div class="leftnav">
    Perhaps there is a left nav bar here.
  </div>
  ...
</body>
</html>

We need to identify the parts of my.js and my.css needed to render the initial content, inline those parts, and delay or async load the remaining JavaScript and CSS needed for the page. This may end up looking something like:

<html>
<head>
  <style>
  .main { ... }
  .leftnav { ... }
  /* ... any other styles needed for the initial render here ... */
  </style>
  <script>
  // Any script needed for initial render here.
  // Ideally, there should be no JS needed for the initial render
  </script>
</head>
<body>
  <div class="main">
    Here is my content.
  </div>
  <div class="leftnav">
    Perhaps there is a left nav bar here.
  </div>
  ...
  <!-- 
    NOTE: delay loading of script and stylesheet may best be done
     in an asynchronous callback such as `requestAnimationFrame` 
     rather than inline in HTML, since the callback will be invoked 
     after the browser has rendered the earlier HTML content to the screen.
   -->
  <link rel="stylesheet" href="my_leftover.css">
  <script src="my_leftover.js"></script>
</body>
</html>

The current state of the mobile web

A brief survey of mobile web pages shows that nearly all pages include blocking external JavaScript and/or CSS before any above the fold content is displayed. Exceptions include Google Maps, Google Search, Yahoo! News, and sites like http://www.yummly.com/ and Kayak. Unfortunately, some of these sites inline JavaScript and CSS that isn’t needed for the initial render, which unnecessarily delays the time it takes to render these pages.

Interestingly, browsing the mobile web with JavaScript disabled reveals that even though most pages load blocking external JavaScript in the head, few of these pages actually need that JavaScript to render their initial content. These pages would benefit from delay or async loading their JavaScript to get the JavaScript out of the critical path of the initial render of the page.

What else can block the initial render of a web page?

Blocking external JavaScript and CSS are the most common sources of delay on the web. However, there are other common sources of delay that mobile developers should be aware of. One source is HTTP redirects of the main HTML document. These redirects incur additional round trips, and if the redirect navigates to a different hostname (e.g. www.example.com to m.example.com), they add even greater delays due to DNS resolution and TCP connection times. A second source of delay is server backend time spent generating the HTML response. All of the time spent generating the initial HTML response will block rendering on the screen, so server backend time should be kept to a minimum.

Rendering in under one second

If we estimate 3G network round trip time at 250ms, we can compute the minimum estimated time between when a user initiates a web page navigation and when that page renders its above the fold content on the screen. Assuming no blocking external JavaScript or CSS, we incur three round trips for DNS, TCP, and request/response, for a total of 750ms, plus 100ms for backend time. This brings us to 850ms. As long as render-blocking JavaScript and CSS is inlined and the size of the initial HTML payload is kept to a minimum (e.g. under 15kB compressed), the time it takes to parse and render should be well under 100ms, bringing us in at 950ms, just under our one second target.

Summary

In summary, to make your mobile web page render in under one second, you should:

• keep server backend time to generate HTML to a minimum (under 100ms)
• avoid HTTP redirects for the main HTML resource
• avoid loading blocking external JavaScript and CSS before the initial render
• inline just the JavaScript and CSS needed for the initial render
• delay or async load any JavaScript and CSS not needed for the initial render
• keep HTML payload needed to render initial content to under 15kB compressed

If you are looking to improve the performance of your mobile web pages, give these optimizations a try.