Web Performance Calendar

WebPagetest is a fantastic Web Performance project which ultimately became the straightforward standard tool for testing and comparing web site performance. It was first developed and opensourced by Patrick Meenan at AOL. Pat’s now at Google and works with a dedicated team on WebPagetest.

API

WebPagetest provides a RESTful API which exposes all UI functions, however the output varies between XML, JSON, and CSV.

NodeJS

In order to develop a NodeJS project that consumes the WebPagetest API from a private instance (or even the public one), one has to deal with several RESTful API endpoints and convert between different output formats, a not so pleasant task and hard to maintain.

What if we had a Swiss Army Knife tool that wraps the whole WebPagetest API and makes the output as well as the commands and options consistent? What if this tool could provide an easy to use command line with both short and long options? What if this tool could provide an asynchronous function callback for NodeJS applications? What if both command line and NodeJS modules also provided a WebPagetest API proxy listener so it also could provide consistent RESTful API endpoints?

‘Tis Xmas and so Santa is bringing WebPagetest API Wrapper, a WebPagetest Swiss Army Knife tool that provides everything listed above in a deadly simple manner.

Motivation

While I was working on an internal performance project at Twitter that would consume our private instance of WebPagetest API, I realized that instead of consuming the API directly from the application, it would be much better to abstract the whole API and decouple it into a separated NPM package so other projects could benefit from it as well as add some sugar features like command line and a RESTful API proxy.

First Things First: Getting it installed

From the Terminal assuming NodeJS is properly installed:

$ npm install webpagetest -g

The -g is required to make the command line available.

If for some reason you don’t feel like installing and would like to try it out first, you can play with the live API Console that is pointing to the public instance. An API key is required to run tests but any other command works without it.

Help

Once WebPagetest API Wrapper is installed you have the command line available in your Terminal and can get more info:

$ webpagetest --help

In the project source code, there is an exhaustive README file that describes each command and available options as well as a few examples for both command line and module. It’s highly recommended reading before proceeding.

WPT Server

The default WebPagetest API Wrapper server is the public instance (www.webpagetest.org) but can be overridden in the command line by:

• setting -s, –server <server> option, e.g.: $ webpagetest -s my-wpt-server.com or
• setting the environment variable WEBPAGETEST_SERVER, e.g.: export WEBPAGETEST_SERVER=my-wpt-server.com

As a NodeJS module, the default WPT server is also the public instance and can be overridden by specifying the first parameter of the constructor:

var WebPagetest = require('webpagetest');
 
var publicWPT = new WebPagetest();
 
var myWPT = new WebPagetest('my-wpt-server.com');

Even when a WPT server is specified it can still be overridden with any method by supplying the server option:

var wpt = new WebPagetest('my-wpt-server.com');
 
wpt.getLocations({server: 'another-wpt-server.com'}, function(err, data) {
  console.log(JSON.stringify(err || data, null, 2));
});

API Key

WebPagetest can be configured to require an API key in order to run tests. The public instance is configured as such to avoid test abuse. Contact the WPT administrator if a key is required (see the WebPagetest About page).

To specify the API key in the command line in order to run a test, set the -k, –key <api_key>:

$ webpagetest -k MY_API_KEY test http://twitter.com/marcelduran

As a NodeJS module, it can be set either as the second parameter in the constructor function or as an option in the runTest function:

var wpt = new WebPagetest('my-wpt-server.com', 'MY_API_KEY');
 
// run test on my-wpt-server.com with MY_API_KEY
wpt.runTest('twitter.com/marcelduran', function(err, data) {
  console.log(err || data);
});
 
// run test on my-wpt-server.com with ANOTHER_API_KEY
wpt.runTest('twitter.com', {key: 'ANOTHER_API_KEY'}, function(err, data) {
  console.log(err || data);
});

Command Line Examples

In the project README file there are some basic command line examples.

To run a simple test:

$ webpagetest test http://twitter.com/marcelduran

There’s no built in way to run a batch of URLs but it can be easily achieved with some basic shell scripting assuming urls.txt contains the URLs to be tested one URL per line:

$ for url in `cat urls.txt`; do webpagetest test $url; done

NodeJS Module Examples

All methods are asynchronous, i.e., they require a callback function that is executed once the WPT API response is received with either data or error. Unlike the command line, method names on the NodeJS module are a bit more verbose (e.g.: results vs getTestResults) for code clarity.

The following code is a contrived example that checks for the first available Chrome browser to test the first view of my Twitter profile page.  Then it polls the test status every 3 seconds and once done gets the results and prints the average page load speeds (yet another contrived performance metric):

var WebPagetest = require('webpagetest');
 
var wpt = new WebPagetest('my-wpt-server.com');
 
wpt.getLocations(function(err, data) {
  // get the first Chrome available
  var browser = data.response.data.location.filter(function(loc) {
    return loc.Browser === 'Chrome' && loc.PendingTests.Idle;
  })[0];
 
  if (!browser) {
    return;
  }
 
  // run test for my Twitter profile page on Chrome
  wpt.runTest('http://twitter.com/marcelduran', {
    location: browser.id,
    firstViewOnly: true
  }, function(err, data) {
 
    function checkStatus() {
      wpt.getTestStatus(data.data.testId, function (err, data) {
        if (!data.data.completeTime) {
          // polling status (every 3s)
          setTimeout(checkStatus, 3000);
        } else {
          // once test is complete, get results
          getResults(data.data.testId);
        }
      });
    }
 
    function getResults(testId) {
      wpt.getTestResults(testId, function (err, data) {
        var firstView = data.response.data.average.firstView,
            time = firstView.fullyLoaded / 1000,
            size = firstView.bytesIn / 1024,
            reqs = firstView.requests;
 
        // print page load average speeds
        console.log(Math.round(size / time) + ' KB/s');
        console.log(Math.round(reqs / time) + ' requests/s');
      });
    }
 
    checkStatus();
  });
});

WebPagetest API provides a pingback option which informs a given URL when the test is complete, so it allows us to have a non-polling solution:

var WebPagetest = require('webpagetest'),
    os          = require('os'),
    url         = require('url'),
    http        = require('http');
 
var wpt = new WebPagetest('my-wpt-server.com');
 
// local server to listen for test complete
var server = http.createServer(function (req, res) {
  var uri = url.parse(req.url, true);
 
  res.end();
 
  // get test results
  if (uri.pathname === '/testdone' && uri.query.id) {
    server.close(function() {
      wpt.getTestResults(uri.query.id, function (err, data) {
        // print page fully loaded time
        console.log(data.response.data.average.firstView.fullyLoaded);
      });
    });
  }
});
 
// run test for my Twitter profile page
wpt.runTest('http://twitter.com/marcelduran', {
  firstViewOnly: true,
  pingback: url.format({
    protocol: 'http',
    hostname: os.hostname(),
    port: 8080,
    pathname: '/testdone'
  })
}, function(err, data) {
  // listen for test complete (pingback)
  server.listen(8080);
});

* note: The pingback URL must be reachable from the WebPagetest server, my-wpt-server.com in the example above.

Scripting

WebPagetest provides a powerful TAB delimited scripting language. To avoid the error prone hassle of TABs vs spaces, WebPagetest API Wrapper provides a script builder function named scriptToString, e.g.:

var script = wpt.scriptToString([
  {logData: 0},
  {navigate: 'http://foo.com/login'},
  {logData: 1},
  {setValue: ['name=username', 'johndoe']},
  {setValue: ['name=password', '12345']},
  {submitForm: 'action=http://foo.com/main'},
  'waitForComplete'
]);
wpt.runTest(script, function (err, data) {
  console.log(err || data);
});

RESTful Proxy (Listener)

WebPagetest API Wrapper comes with a handy RESTful proxy (listener) that exposes WebPagetest API methods consistently. This means that all the benefits of methods, options and JSON output from WebPagetest API Wrapper can be easily reachable through RESTful endpoints.

From command line

Assuming you have a WebPagetest private instance at my-wpt-server.com and from your local machine (localhost with my-machine as hostname for the local network access) you run:

$ webpagetest listen --server my-wpt-server.com

This will turn your local machine into a WebPagetest API Wrapper RESTful proxy for my-wpt-server.com, so from any other machine in the network you can access it either via a browser or curl:

• http://my-machine/help
• http://my-machine/locations
• http://my-machine/test/http%3A%2F%2Ftwitter.com%2Fmarcelduran?first=true

If –server or -s is not provided, WebPagetest API Wrapper first checks the environment variable WEBPAGETEST_SERVER and falls back to the public instance www.webpagetest.org

The project API Console demo is powered by WebPagetest API Wrapper RESTful proxy and is pointing to the WebPagetest public instance. You can access it at http://wpt.rs.af.cm, for example:

• http://wpt.rs.af.cm/help
• http://wpt.rs.af.cm/locations
• http://wpt.rs.af.cm/test/http%3A%2F%2Ftwitter.com%2Fmarcelduran?first=true
• http://wpt.rs.af.cm/results/121119_DJ_J0R

From NodeJS module

The method is called listen and has one optional port parameter (default 7791). As a matter of fact the API Console demo is a 3 line app:

var WebPageTest = require('webpagetest');
var wpt = new WebPageTest();
wpt.listen(3000);

For the one-liners:

require('webpagetest')().listen(3000);

Merry Xmas

I hope you enjoy this Swiss Army knife gift Santa brings to you and use it in your custom WebPagetest projects. It was first designed to reduce the effort of building NodeJS apps that consume the WebPagetest API. Lately with the addition of command line capabilities chances are you may end up using it more than the WebPagetest UI. Best of all it’s open source and contributions are always welcome. Fork it now and show Santa your gratitude for performance. 🙂 Merry Xmas!

[update 12-21-12 – replaced nodejitsu for appfog links]