I thought I’d take the opportunity this year to give a little bit of visibility into how WebPagetest gathers the performance data from browsers. Other tools on windows use similar techniques but the information here may not be representative of how other tools work.

First off, it helps to understand the networking stack on Windows from a browser’s perspective:

Browser Network Diagram

It doesn’t matter what the browser is, if it runs on Windows the architecture pretty much HAS to look like the diagram above where all of the communications go through the Windows socket API’s (for that matter, just about any application that talks TCP/IP on Windows looks like the picture above).

Function Interception

The key to how WebPagetest works is its ability to intercept arbitrary function calls and inspect or alter the request or response before passing it on to the original implementation (or choosing not to pass it on at all). Luckily someone else did most of the heavy lifting and provided a nice open source library that can take care of the details for you but it basically works like this:

  • Find the target function in memory (trivial if it is exported from a dll)
  • Copy the first several bytes from the function (making sure to keep x86 instructions intact)
  • Overwrite the function entry with a jmp to the new function
  • Provide a replacement function that includes the bytes copied from the original function along with a jmp to the remaining code

It’s pretty hairy stuff and things tend to go VERY wrong if you aren’t extremely careful but with well-defined functions (like all of the Windows API’s) you can pretty much intercept anything you’d like.

One catch is that you can only redirect calls to code running in the same process as the original function which is fine if you wrote the code but doesn’t help a lot if you are trying to spy on software that you don’t control which leads us to…

Code Injection

Lucky for me, Windows provides several ways to inject arbitrary code into processes. There is a good overview of several different techniques here (and there are actually more ways to do it than that even but it covers the basics). Some of the techniques insert your code into every process but I wanted to be a lot more targeted and just instrument the specific browser instances that we are interested in so after a bunch of experimentation (and horrible failures) I ended up using the CreateRemoteThread/LoadLibrary technique which essentially lets you force any process to load an arbitrary dll and execute code in it (assuming you have the necessary rights).

Resulting Browser Architecture

Now that we can intercept arbitrary function calls, it just becomes a matter of identifying the “interesting” functions, preferably ones that are used by all of the browsers so you can re-use as much code as possible. In WebPagetest we intercept all of the Winsock calls that have to do with resolving host names, connecting sockets and reading or writing data:

Modified Browser Network Diagram

This gives us access to all of the network access from the browser and we essentially just keep track of what the browsers are doing. Other than having to decode the raw byte streams it is pretty straightforward and it gives us a consistent way to do the measurements across all browsers. SSL does add a bit of a wrinkle so we also intercept calls to the various SSL libraries that the browsers use so that we can see the unencrypted version of the data (which is a little more difficult for Chrome since the library is compiled into the Chrome code itself but luckily they make debug symbols available for every build so we can still find the code in memory).

The same technique is used to intercept drawing calls from the browser so we can tell when it paints to the screen (for the start render measurement).

Get the Code

Since WebPagetest is under a BSD license you are welcome to re-use any of the code for whatever purposes you’d like. The project lives on Google Code here: http://code.google.com/p/webpagetest/ and some of the more interesting files are:

Browser Advancements

Luckily, browsers are starting to expose more interesting information in standard ways and as the W3C Resource Timing spec advances you will be able to access a lot of this information directly from the browser through JavaScript (from your end users even!).

Patrick Meenan

Patrick Meenan (@patmeenan) created WebPagetest while working at AOL and now works at Google with the team that is working to make the web faster.

14 Responses to “WebPagetest internals”

  1. Billy Hoffman

    Great article Patrick and awesome tool. But don’t stop the nerding out just yet! I’d love more technical details on some of the supporting structure around WPT. Things like: extracting YSlow/Pagespeed scores from headless browsers. Generating the page load movies (AVIsynth?). The wiring with other logic/tools to test for performance issues (ImageMagick I imagine).

    Keep up the great work!

  2. Aaron Peters

    As a WPT power user and knowing quite a lot about it, even for me this post and the diagrams are a useful resource. Txs Pat!

  3. Ein zweites Adventst├╝rchen « F-LOG-GE

    [...] Adventskalender rund um Performance schrieb gestern über technische Interna der Seite WebPagetest. Auch in diesem Jahr gibt es wieder den [...]

  4. Andy Davies

    Thanks Pat, it’s really good to get an insight into what’s going on under the hood.

    WPT is one of the tools I couldn’t live without

  5. Samarjeet Mohanty

    Very nice article indeed. Wud be interesting to see Safari or Opera in there too .

  6. Philip Tellis

    Hey Pat, good insights here. Brought back memories of writing (prototype) virii for .com files back in the MS-DOS days.

    Question: What did you use to make the images?

  7. Patrick Meenan

    @Billy, I’ll put together some blog posts to keep the details flowing. AVISynth in particular is freakishly cool (wish there was a Linux alternative).

    Most of the other logic is hand-written c++ or php/GD (no ImageMagik) but there is also a pretty nice integration with the Page Speed c++ SDK.

  8. Patrick Meenan

    @Philip Which images?

    The screen and video captures are all done with native c++ code on Windows (using CxImage for support and libjpeg for compression).

    The waterfalls are hand drawn pixel by pixel in php with GD (well, mostly line by line).

    The Pie charts are done using jpgraph.

  9. Patrick Meenan

    @Philip err, or did you mean the images in the calendar post? :-) For those I used Powerpoint 2010 (old habit from management when I spent most of my days in powerpoint).

  10. Kiran

    Its good one Patrick.I have wondered for long as how page test does really work behind the scenes.Now its clear to me.

    Thanks and its very helpfull and a fantastic tool.

  11. Justin Dorfman

    Thanks Patrick for sharing this. I use WPT daily (M-F) and can’t thank you enough for all your hard work and dedication.

  12. Performance Calendar » I see HTTP

    [...] the iOS SDK doesn’t provide APIs to inspect the traffic. But then I saw what Patrick Meenan needs to do to make WebPagetest happen, so I guess a little hacking and method swizzling might be appropriate. Which would lower the [...]

  13. The 20 best web performance links of Q4

    [...] How WebPagetest works If you’ve ever wondered how exactly WebPagetest gathers performance data from the various browsers it simulates, this is great post from Pat Meenan in which he cracks the hood of WebPagetest and explains all that. [...]

  14. rajat raja

    Its good one Patrick.I have wondered for long as how page test does really work behind the scenes.Now its clear to me.

    Thanks and its very helpfull and a fantastic tool.

Leave a Reply

You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
And here's a tool to convert HTML entities