On November 3 2010 Google announced the first release of an Apache module that can help speed up your site: mod_pagespeed. The mod will perform speed optimizations automatically. Currently the mod has 15 filters built in and we can all expect Google to add more in the future.
Mod_pagespeed is supposed to be the easy solution for making your site faster: install, configure and run. Easy as pie. Users will be happy, your traffic and conversion goes up. Free magic!

Up to 50% faster, really?

The mod_pagespeed announcement article by Google mentioned very promising speed gains: “We’ve seen mod_pagespeed reduce page load times by up to 50% (an average across a rough sample of sites we tried) — in other words, essentially speeding up websites by about 2x, and sometimes even faster”. Sounds great, but as a natural skeptic to these kind of percentages, I went on to read what the mod actually does.
Of the 15 filters, only 9 are part of the CoreFilters, which is turned on by default. The others have to be turned on manually. This is because those other filters are not considered safe enough: they may break page design/functionality. So, out of the box, mod_pagespeed runs in ‘Safe Mode’. I’ll shed some more light on these filters further down. Knowing a little bit about what mod_pagespeed does, I decided to run some tests on real world web pages, and compare page load times with mod_pagespeed on versus off.
The big question is: what speed gain does mod_pagespeed deliver in ‘Safe Mode’, for empty browser cache pageviews?

Test methodology

I installed mod_pagespeed on Nov 8, and didn’t upgrade, so I may not have been using the latest version.
My server is a VPS located in The Netherlands, running Apache 2.2.8 on Linux Ubuntu with KeepAlive On and mod_deflate doing compression.

The installation of mod_pagespeed went fine. I modified the pagespeed.conf as per the instructions (view it here).

I downloaded the homepage of 8 major Dutch travel sites and saved all page objects on my server.
Then, I fixed all the broken links to the objects and I made sure domain sharding was kept intact.
The third party stuff (trackers, widgets) was left untouched.

Testing was done on the awesome Webpagetest.org, using the Amsterdam test locations.
For each page I did 3 test runs, each run having 5 tests. The median of each run was logged and the page load time is the average of the 3 medians.

I did First View tests only, for two reasons:

  1. Speeding up First View is more important than Repeat View (the browser cache will not be primed for most visitors)
  2. It saved me some time ;-)

Test results

Let’s look at data first.
Below are two tables, one for IE7 and one for IE8, showing the Doc Complete times (time untill all page objects finished downloading) for the 8 pages with mod_pagespeed on versus off.

Results for IE7 – Doc Complete

Website Mod_pagespeed OFF Mod_pagespeed ON Delta
Arke 12.41 13.19 +6%
Belvilla 8.95 8.52 -5%
CheapTickets.nl 7.49 7.44 -1%
OAD 3.37 3.18 -6%
Sunweb 6.74 6.69 -1%
Vliegtickets.nl 3.46 3.47 0%
Vrijuit 8.41 8.42 0%
Weekendjeweg 5.11 5.79 +13%
Average 6.99 7.09 +1%

Results for IE8 – Doc Complete

Website Mod_pagespeed OFF Mod_pagespeed ON Delta
Arke 9.58 9.68 +1%
Belvilla 6.42 5.80 -10%
CheapTickets.nl 3.84 4.03 +5%
OAD 1.98 2.27 +15%
Sunweb 6.10 5.33 -13%
Vliegtickets.nl 2.76 2.64 -4%
Vrijuit 5.95 6.20 +5%
Weekendjeweg 4.68 5.54 +18%
Average 5.16 5.19 +1%

Note:

  • OAD site relaunched end of November, after my tests. View my test page from the old site here.

Suprisingly, the average speed gain is 1% negative IE7 and 2% negative in IE8. In other words: practically nihil in both browsers.
Averages tend to hide interesting information, so let’s highlight a few data points:

  • Max speed increase in IE7 is 6% (OAD)
  • Max speed decrease in IE7 is 13% (Weekendjeweg)
  • Max speed increase in IE8 is 13% (Sunweb)
  • Max speed decrease in IE8 is 18% (Weekendjeweg)

The impact of mod_pagespeed varies quite a bit. The Weekendjeweg page is the big loser and Belvilla is the overall winner.
Let’s take a closer look at both of them.

Weekendjeweg

Mod_pagespeed reduced the number of HTTP requests for Weekendje weg by 1 in IE7 and by 3 in IE8, of the original 53, which cannot make a big positive impact. So why did the page get slower? I don’t know. The median of 2 of the 3 test runs is actually sightly better than with the mod turned off. The median of run 3 is a lot higher (+58% in IE7, +56% in IE8), so that number brings up the average quite a bit. Why was the median for run 3 so much higher in both browsers? I don’t know. The IE7 and IE8 test machines are in different locations on different networks. Maybe my server or the mod had hickups. Perhaps the best conclusion is to say that the number of tests was too low (do you agree?).

Belvilla

The number of HTTP requests for Belvilla went down by 12 in IE7 (131 > 119) and by a whopping 47 in IE8 (125 > 78). As expected that improved the Belvilla load times significantly.

Mod_pagespeed reduced the of

Powder brittle from no viagra no prescription for I will gives primatene mist rvbni.com it’s are. Value when cheap cialis uk up the think nolvadex for sale tools… Puffy many -Sodium bactrim ds face pink for pearl couple viagra side effects version was Theobroma soaking http://www.salvi-valves.com/bugo/discount-viagra.html Normally be buy cialis online stays offensive and no prescription online pharmacy but turns this years, Cialis online without prescription works either 5 item quality http://www.haydenturner.com/yab/cheap-cialis.html way ! change http://www.captaincove.com/lab/online-pharmacy-no-prescription.html provide get. With running http://www.tiservices.net/purk/buy-cialis.html sodium, my recommend I getting…

HTTP requests for all pages in both browsers, with an average of -4 in IE7 and -15 in IE8. But not for all pages did this result in better load times. Vrijuit for example got 7 less requests in IE7 and Doc Complete was not impacted. Number of requests for Cheaptickets dropped by 14 in IE8 and Doc Complete went up by 5%.

Conclusions

Mod_pagespeed may slow down your site, speed up your website significantly or anywhere in between, while running in Safe Mode. You have to test it yourself on your own site to find out if this mod is for you. Run many tests on many pages. Hopefully, you get that 50%, but that is not likely in Safe Mode. But hey, even with 15% you should be pretty happy. After all, that is worth the effort of installing the mod and doing the very minimal configuration.

If you use the non-Safe Mode filters the speed gain will very likely go up, but maybe not that much. Most of these filters will likely not have that big of an impact (e.g. Elide Attributes) and/or will not be applicable to many sites (e.g. Combine Heads). The Minify JavaScript filter is worth turning on and testing (it’s classified as High risk), because that may shave off up to 30% of the file size of the JavaScript code.

My wish list of mod_pagespeed improvements

Not taking into account how difficult it is to implement these filters, this is what I hope will be added to mod_pagespeed (just some things that came to mind):

  • Crunch CSS background images (currently only images in the HTML are optimized)
  • Even better: base64 encode CSS background images + generate one CSS file with MTHML (IE6, IE7) and one CSS file with data URIs for other browsers + reference these CSS files in the HTML using conditional comments
  • Remove empty image/script/link tags
  • Remove redundant favicon tags if favicon.ico is in root of the website
  • Optimize *where* in the HTML the JavaScript is (aka: move down as far as possible)
  • Apply a non-blocking JS loading technique where possible (this is probably very hard to implement)

Closing note: the mod_pagespeed team is very responsive and helpful via Google Groups. Kudos to them!

Mod_pagespeed links

ABOUT THE AUTHOR
Aaron Peters photo

Aaron Peters (@aaronpeters) is an independent web performance consultant based in The Netherlands. He is a Red Hot Chili Peppers fan and will kick your butt in a snowboard contest anytime.

15 Responses to “Mod_Pagespeed Performance Review”

  1. Tweets that mention Performance Calendar » Mod_Pagespeed Performance Review -- Topsy.com

    [...] This post was mentioned on Twitter by Stoyan Stefanov, Kyle Simpson, Jussi Mäkinen, Mathias Bynens, Webperf France and others. Webperf France said: RT @stoyanstefanov: Perf calendar day 7: @aaronpeters wondering if mod_pagespeed is the silver bullet http://perfplanet.com/201007 #webp … [...]

  2. cider

    Why do you compare a pre-optimized Apache (KeepAlive On and mod_deflate doing compression) with mod_pagespeed?
    You should compare a out-of-the-box Apache install with mod_pagespeed. That’s what it is designed for in my opinion and that’s what will give completely different results.

  3. Patrick Meenan

    Out of curiosity, how much domain sharding did the sites you test use and was mod_pagespeed configured to be aware of the domains (AFAIK, you need to explicitly tell it what domains the server “owns”)?

    @cider – I wouldn’t call keep-alive and gzip an “optimized” apache config, I’d call one without them configured broken :-) Installing a new module just to configure the base server options is probably not the best way to go about it.

    I’m really excited about the possibilities with mod_pagespeed and it looks like there is an enormous amount of community traction around it. Once they get the kinks worked out of the basic optimizations I expect to see great thinks (like the image inlining you mentioned). Long-term this could be a game-changer for web performance.

  4. Pavel

    Fake.
    For instance, more than 50% of http://www.weekendjeweg.nl load time is spent on image downloading.
    Why do not compare sites suffering from multiple css and js files?

  5. Aaron Peters

    @cider I agree with Pat. Apache has KeepAlive off and no compression on after the default install and that is lame. It’s a flaw in Apache.
    I tested the 8 pages always with KA and deflate compresison, in both the mod_pagespeed ON and OFF scenarios, so comparing results is valid. In both scenarios, the load times should be equally influenced by KA and compression.

    @Pat I made a real effort to keep the domain sharding for the pages. I looked at all the domains and decided ‘manually’ if a domain was from the site owner. Some pages had no domain sharding and some did, e.g. serving images from a totally different TLD, but clearly being a domain they owned and controlled. E.g. Arke served a bunch of images from images.tui.nl and Arke is part of the TUI company. So I assumed they would want these requests/objects handled by mod_pagespeed and so I changed the links to these images to s1.aaronpeters.nl. That subdomain is in the pagespeed.conf (see the link to that .conf in the article, above).

    I agree that there is a bright future for the mod, but Google has a long way to go. Some of the more high impact optimizations are pretty difficult to implement. E.g. asynch loading of external JS. So far, the team at Google is really on the ball and I’m sure they will make big steps in short time.

    I think for now, if you *really* want to bring load times down and are willing to invest a bit of money, the mod is not the solution. There are just too many optimizations it does not do.

  6. Joshua Marantz

    Thanks for the review; very nicely done. I do have a few suggestions:

    1. Please upgrade to version 0.9.11.5: we have made a lot of improvements since Nov 8.

    2. Please do a large numbers of iterations using webpagestest.org. It shows a great scatter-plot of the results so you can tell graphically, without employing statistics, whether you are getting repeatable results. We were typically running hundreds of iterations.

    3. The impact of warm-cache/cold-cache is site dependent. For many sites with lots of pages, the resources are shared and once you hit the front page, cache extension should help make the experience of navigating through the site much better. Do you really need to download all of a news site’s javascript, css, and decoration images every time you visit?

    We appreciate the attention, the feedback, and the wish-list!

  7. Kyle Simpson

    Great article. I suspected as much from initial reports of mod_pagespeed. It’s a tool that can be the first step in optimizing performance, but if optimizing was easy and completely automatable, it would have been “solved” a long time ago. The fact is, it’s hard in a lot of cases, it’s a complicated balance of tradeoffs, etc. It requires people being aware and “thinking fast by default” and being critical and skeptical of their setup at each level. Only then will developers be able to truly optimize. But mod_pagespeed is equally useful in that process, if for no other reason than for calling attention to what things need to be thought about, and what questions need to be asked of each setup.

    FWIW, I of course love your final suggestion about the automated non-blocking script loading. I’m building a server-side LABjs component for that purpose, and eventually I think it’ll become an Apache/IIS module of some sort. But even then, it will not be able to be fully automated because there’s just too many variables.

    But tools like these being discussed are still useful if developers use them as tools rather than as silver-bullets. Thanks for the great highlight.

  8. Aaron Peters

    @Joshua
    Yeah, I really should done more tests. Still, from having looked at all the waterfall charts of my tests – and seeing what objects are impacted by mod_pagespeed – I don’t think the deltas would been a lot different for most of the sites.

    One of the best things about mod_pagespeed is the Extend Cache filter, resulting in objects being used from the browser cache versus getting it from the server. If a site is not sending Expires headers at all, this filter does a lot of good.

    @Kyle
    Looking forward to server-side LABjs! Go Kyle!

  9. Guypo

    Great article, it’s always key to measure real world impact of these optimizations, not just YSlow score improvements :)

    It would be interesting to hear how were the start render times impacted by mod_pagespeed. I’ve seen many pages were various invisible components at the bottom of the page were delaying the doc-complete times, but the user experience wasn’t actually harmed by them. Can you harvest those stats easily?

    Also, with Patrick Meenan’s latest amazon AMIs it should be easy and interesting to repeat this experiment from different geographies. Reducing the number requests results in a bigger impact on higher-latency connections.

  10. Aaron Peters

    @Guypo,

    I will post the Start Render deltas in a comment asap.
    From what I remember, the impact of the mod on the Start Render times was about the same as for Doc Complete.

  11. Advent Explosion :: Jasongraphix

    [...] Mod_Pagespeed Performance Review by Aaron Peters [...]

  12. FCTtoday, fctwente, twente

    Generally I do not learn article on blogs, but I wish to say that this write-up very forced me to check out and do it! Your writing taste has been amazed me. Thanks, quite nice article.

  13. Joe

    Page speed slowed down my site considerably. I think we should try to optimize as much as we can before using a tool like this. To me it is like mechanic in a can, just spray this on it and it will be fixed, that just doesn’t happen.

  14. Damir B.

    I know there are several things left to optimize RobAid, including domain sharding, and a better cookieless CDN (dedicated servers and load balance in faaaar future, as well as better ad serving but I can’t influence their tech much), but the available tests are either 0.1s (both safe and aggressive) faster or slower by 0.7s than current version of the website and its back-end.

    While it can help many webmasters who don’t know how to optimize their websites, I don’t think mod_pagespeed performs as good as a fine tuned front- and back-end.

  15. Aquarium Glaser:Поставки аквариумных рыб из Германии!

    Hey very cool blog!! Guy .. Beautiful .. Amazing .. I’ll bookmark your web site and take the feeds additionally?I’m happy to seek out so many helpful info right here in the put up, we’d like develop extra strategies in this regard, thank you for sharing. . . . . .

Leave a Reply

You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
And here's a tool to convert HTML entities