WebPagetest is a fantastic Web Performance project which ultimately became the straightforward standard tool for testing and comparing web site performance. It was first developed and opensourced by Patrick Meenan at AOL. Pat’s now at Google and works with a dedicated team on WebPagetest.

API

WebPagetest provides a RESTful API which exposes all UI functions, however the output varies between XML, JSON, and CSV.

NodeJS

In order to develop a NodeJS project that consumes the WebPagetest API from a private instance (or even the public one), one has to deal with several RESTful API endpoints and convert between different output formats, a not so pleasant task and hard to maintain.

What if we had a Swiss Army Knife tool that wraps the whole WebPagetest API and makes the output as well as the commands and options consistent? What if this tool could provide an easy to use command line with both short and long options? What if this tool could provide an asynchronous function callback for NodeJS applications? What if both command line and NodeJS modules also provided a WebPagetest API proxy listener so it also could provide consistent RESTful API endpoints?

‘Tis Xmas and so Santa is bringing WebPagetest API Wrapper, a WebPagetest Swiss Army Knife tool that provides everything listed above in a deadly simple manner.

Motivation

While I was working on an internal performance project at Twitter that would consume our private instance of WebPagetest API, I realized that instead of consuming the API directly from the application, it would be much better to abstract the whole API and decouple it into a separated NPM package so other projects could benefit from it as well as add some sugar features like command line and a RESTful API proxy.

First Things First: Getting it installed

From the Terminal assuming NodeJS is properly installed:

$ npm install webpagetest -g

The -g is required to make the command line available.

If for some reason you don’t feel like installing and would like to try it out first, you can play with the live API Console that is pointing to the public instance. An API key is required to run tests but any other command works without it.

Help

Once WebPagetest API Wrapper is installed you have the command line available in your Terminal and can get more info:

$ webpagetest --help

In the project source code, there is an exhaustive README file that describes each command and available options as well as a few examples for both command line and module. It’s highly recommended reading before proceeding.

WPT Server

The default WebPagetest API Wrapper server is the public instance (www.webpagetest.org) but can be overridden in the command line by:

  • setting -s, –server <server> option, e.g.: $ webpagetest -s my-wpt-server.com or
  • setting the environment variable WEBPAGETEST_SERVER, e.g.: export WEBPAGETEST_SERVER=my-wpt-server.com

As a NodeJS module, the default WPT server is also the public instance and can be overridden by specifying the first parameter of the constructor:

var WebPagetest = require('webpagetest');
 
var publicWPT = new WebPagetest();
 
var myWPT = new WebPagetest('my-wpt-server.com');

Even when a WPT server is specified it can still be overridden with any method by supplying the server option:

var wpt = new WebPagetest('my-wpt-server.com');
 
wpt.getLocations({server: 'another-wpt-server.com'}, function(err, data) {
  console.log(JSON.stringify(err || data, null, 2));
});

API Key

WebPagetest can be configured to require an API key in order to run tests. The public instance is configured as such to avoid test abuse. Contact the WPT administrator if a key is required (see the WebPagetest About page).

To specify the API key in the command line in order to run a test, set the -k, –key <api_key>:

$ webpagetest -k MY_API_KEY test http://twitter.com/marcelduran

As a NodeJS module, it can be set either as the second parameter in the constructor function or as an option in the runTest function:

var wpt = new WebPagetest('my-wpt-server.com', 'MY_API_KEY');
 
// run test on my-wpt-server.com with MY_API_KEY
wpt.runTest('twitter.com/marcelduran', function(err, data) {
  console.log(err || data);
});
 
// run test on my-wpt-server.com with ANOTHER_API_KEY
wpt.runTest('twitter.com', {key: 'ANOTHER_API_KEY'}, function(err, data) {
  console.log(err || data);
});

Command Line Examples

In the project README file there are some basic command line examples.

To run a simple test:

$ webpagetest test http://twitter.com/marcelduran

There’s no built in way to run a batch of URLs but it can be easily achieved with some basic shell scripting assuming urls.txt contains the URLs to be tested one URL per line:

$ for url in `cat urls.txt`; do webpagetest test $url; done

NodeJS Module Examples

All methods are asynchronous, i.e., they require a callback function that is executed once the WPT API response is received with either data or error. Unlike the command line, method names on the NodeJS module are a bit more verbose (e.g.: results vs getTestResults) for code clarity.

The following code is a contrived example that checks for the first available Chrome browser to test the first view of my Twitter profile page.  Then it polls the test status every 3 seconds and once done gets the results and prints the average page load speeds (yet another contrived performance metric):

var WebPagetest = require('webpagetest');
 
var wpt = new WebPagetest('my-wpt-server.com');
 
wpt.getLocations(function(err, data) {
  // get the first Chrome available
  var browser = data.response.data.location.filter(function(loc) {
    return loc.Browser === 'Chrome' && loc.PendingTests.Idle;
  })[0];
 
  if (!browser) {
    return;
  }
 
  // run test for my Twitter profile page on Chrome
  wpt.runTest('http://twitter.com/marcelduran', {
    location: browser.id,
    firstViewOnly: true
  }, function(err, data) {
 
    function checkStatus() {
      wpt.getTestStatus(data.data.testId, function (err, data) {
        if (!data.data.completeTime) {
          // polling status (every 3s)
          setTimeout(checkStatus, 3000);
        } else {
          // once test is complete, get results
          getResults(data.data.testId);
        }
      });
    }
 
    function getResults(testId) {
      wpt.getTestResults(testId, function (err, data) {
        var firstView = data.response.data.average.firstView,
            time = firstView.fullyLoaded / 1000,
            size = firstView.bytesIn / 1024,
            reqs = firstView.requests;
 
        // print page load average speeds
        console.log(Math.round(size / time) + ' KB/s');
        console.log(Math.round(reqs / time) + ' requests/s');
      });
    }
 
    checkStatus();
  });
});

WebPagetest API provides a pingback option which informs a given URL when the test is complete, so it allows us to have a non-polling solution:

var WebPagetest = require('webpagetest'),
    os          = require('os'),
    url         = require('url'),
    http        = require('http');
 
var wpt = new WebPagetest('my-wpt-server.com');
 
// local server to listen for test complete
var server = http.createServer(function (req, res) {
  var uri = url.parse(req.url, true);
 
  res.end();
 
  // get test results
  if (uri.pathname === '/testdone' && uri.query.id) {
    server.close(function() {
      wpt.getTestResults(uri.query.id, function (err, data) {
        // print page fully loaded time
        console.log(data.response.data.average.firstView.fullyLoaded);
      });
    });
  }
});
 
// run test for my Twitter profile page
wpt.runTest('http://twitter.com/marcelduran', {
  firstViewOnly: true,
  pingback: url.format({
    protocol: 'http',
    hostname: os.hostname(),
    port: 8080,
    pathname: '/testdone'
  })
}, function(err, data) {
  // listen for test complete (pingback)
  server.listen(8080);
});

* note: The pingback URL must be reachable from the WebPagetest server, my-wpt-server.com in the example above.

Scripting

WebPagetest provides a powerful TAB delimited scripting language. To avoid the error prone hassle of TABs vs spaces, WebPagetest API Wrapper provides a script builder function named scriptToString, e.g.:

var script = wpt.scriptToString([
  {logData: 0},
  {navigate: 'http://foo.com/login'},
  {logData: 1},
  {setValue: ['name=username', 'johndoe']},
  {setValue: ['name=password', '12345']},
  {submitForm: 'action=http://foo.com/main'},
  'waitForComplete'
]);
wpt.runTest(script, function (err, data) {
  console.log(err || data);
});

RESTful Proxy (Listener)

WebPagetest API Wrapper comes with a handy RESTful proxy (listener) that exposes WebPagetest API methods consistently. This means that all the benefits of methods, options and JSON output from WebPagetest API Wrapper can be easily reachable through RESTful endpoints.

From command line

Assuming you have a WebPagetest private instance at my-wpt-server.com and from your local machine (localhost with my-machine as hostname for the local network access) you run:

$ webpagetest listen --server my-wpt-server.com

This will turn your local machine into a WebPagetest API Wrapper RESTful proxy for my-wpt-server.com, so from any other machine in the network you can access it either via a browser or curl:

  • http://my-machine/help
  • http://my-machine/locations
  • http://my-machine/test/http%3A%2F%2Ftwitter.com%2Fmarcelduran?first=true

If –server or -s is not provided, WebPagetest API Wrapper first checks the environment variable WEBPAGETEST_SERVER and falls back to the public instance www.webpagetest.org

The project API Console demo is powered by WebPagetest API Wrapper RESTful proxy and is pointing to the WebPagetest public instance. You can access it at http://wpt.rs.af.cm, for example:

From NodeJS module

The method is called listen and has one optional port parameter (default 7791). As a matter of fact the API Console demo is a 3 line app:

var WebPageTest = require('webpagetest');
var wpt = new WebPageTest();
wpt.listen(3000);

For the one-liners:

require('webpagetest')().listen(3000);

Merry Xmas

I hope you enjoy this Swiss Army knife gift Santa brings to you and use it in your custom WebPagetest projects. It was first designed to reduce the effort of building NodeJS apps that consume the WebPagetest API. Lately with the addition of command line capabilities chances are you may end up using it more than the WebPagetest UI. Best of all it’s open source and contributions are always welcome. Fork it now and show Santa your gratitude for performance. :-) Merry Xmas!

[update 12-21-12 - replaced nodejitsu for appfog links]

ABOUT THE AUTHOR

Marcel Duran Photo

Marcel Duran is a performance front end engineer at Twitter and previously had been into web performance optimization on high traffic sites at Yahoo! Front Page and Search teams where he applied and researched web performance best practices making pages even faster. He was also the Front End Lead for Yahoo!’s Exceptional Performance Team dedicated to YSlow (now it's his personal open source project) and other performance tools development, researches and evangelism. Besides being a "speed freak", he's a (skate|snow)boarder and purple belt in Brazilian Jiu-Jitsu

13 Responses to “Xmas Gift: WebPagetest API Swiss Army Knife”

  1. Performance Calendar » Proactive Web Performance Optimization

    [...] For high performance applications in a well defined building cycle, YSlow scores might become stale always reporting A or B. This won’t tell much about fine performance regression and still might lead to some regression being lately pushed to the end user. It’s very important to keep monitoring RUM data in order to detect any unplanned variation, this is the most accurate info one can get from user experience. Once YSlow score is satisfied, i.e.: it doesn’t break the build, the next layer of proactive WPO is benchmarking the build in real browsers with a reasonable sample (the greater the better) to avoid variation then get either the median or average of these runs, this should be compared to the current production baseline and within a certain threshold this should either pass or break the build to avoid fine performance regression. To automate the benchmarking part, WebPagetest is a good fit and WebPagetest API Wrapper can be used power NodeJS applications (more info on previous post). [...]

  2. vanshaj

    Hi,

    With warm regards, Actually I want the browser detail when i put the location in parameter any one can please help me out.

    regards,
    Vanshaj Rai

  3. 行動電源推薦

    You are so interesting! I do not think I’ve study anything like this before. So good to find somebody having a few original thoughts on this topic. Seriously.. numerous thanks for starting this up. This website is something that is required around the internet, someone with a little bit of originality!

  4. 移動電源

    Totally amazing study wish everybody would bring up this problem much more, however whenever you have a facebook page I’ll accept you as a buddy!

  5. Katharina

    Nice blog here! Also your website loads up very fast!
    What host are you using? Can I get your affiliate
    link to your host? I wish my web site loaded up as quickly as yours lol

  6. Ervin

    It’s a shame you don’t have a donate button! I’d certainly donate to this superb blog!
    I suppose for now i’ll settle for bookmarking and adding your RSS feed to my Google account.
    I look forward to fresh updates and will share this website with my Facebook group.
    Talk soon!

  7. newport plastic bumper repair

    Hello! Someone in my Myspace group shared this site with
    us so I came to give it a look. I’m definitely loving
    the information. I’m bookmarking and will be tweeting this to my followers!
    Superb blog and wonderful design and style.

  8. http://Youtu.be/QdKvKpwGOkk

    These are the kinds of employment that individuals constantly ignore to do.
    Purely like changing your filter within their furnace.

    My web-site: Ductwork Cleaning (http://Youtu.be/QdKvKpwGOkk)

  9. Metal carts

    On the other hand, adopting the well-known well-being guidelines for an unique device generally decreases your risk of injury.

    Visit my weblog Metal carts

  10. home improvement Websites

    Greetings! Very helpful advice in this particular post! It is the little changes
    that produce the biggest changes. Many thanks for sharing!

    my web blog … home improvement Websites

  11. Hair Ties

    Hey i really liked the design. Thought your thing had been
    really cool.

    Also visit my web-site: Hair Ties

  12. starting a corporation

    I usually do not drop a lot of comments, but i did some searching and wound up here
    Performance Calendar Xmas Gift: WebPagetest API Swiss Army Knife.
    And I actually do have 2 questions for you if you tend not
    to mind. Is it just me or does it look like a few of these
    comments look like they are coming from brain dead folks?
    :-P And, if you are writing at additional social sites, I would like to follow anything fresh you have to post.
    Would you make a list of the complete urls of all your shared
    pages like your linkedin profile, Facebook page or twitter feed?

    Here is my page :: starting a corporation

  13. Pinterest.com

    Cottage – June 2011

    Also visit my weblog Camping in The Manistee National Forest (Pinterest.com)

Leave a Reply

You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
And here's a tool to convert HTML entities