Web Performance Calendar

There are a lot of reasons for running your own WebPagetest private instance, some of which include:

• Lots of automated testing using the API (or CI tools like the WebPagetest API wrapper or grunt-perfbudget that are built on top of the API).
• Control over the testing infrastructure for consistent results across test locations or SLA’s that are within your control (there are no SLA’s for the public WebPagetest instance).
• Integration with existing logs-processing infrastructure for consuming test results (splunk, etc).
• Testing of content not reachable from the public Internet.

Historically this required setting up a web server with the UI and Windows test machines as needed and usually resulted in at least several days of work and a bunch of questions about how to get the configuration working. Once it is up and running you would also have to keep it up to date, both with releases of the UI that came out every few months as well as agent updates that come out quite a bit more frequently. That’s a pretty significant barrier to entry, particularly if someone is trying to do a quick one-time test. Recently we released an AMI on Amazon’s EC2 that makes it trivial to start up full WebPagetest testing infrastructure.

Bringing up WebPagetest on EC2

The configuration of the server can be provided through user data (through the UI or API) and the flow to bringing up an instance is:

1. Log into AWS (Create an account if needed)
2. Generate an AWS API Key and Secret for the instance to use. This will usually mean creating a “user” with just the permissions that you want the instance to have and getting the AWS access key for that user. At a minimum the key needs to be able to create and terminate EC2 instances but it may also need S3 access if you want to persist the test results in S3.
3. Find the AMI for the WebPagetest server. The ID for the current server can be found on the documentation page.
4. Launch an instance of the server AMI on the appropriate instance size (micro works well for small deployments). Make sure to go through the configuration flow, open the advanced details and provide the configuration user data.
5. Wait for the instance to start (should just be a few minutes)
6. Profit! You now have a private WebPagetest instance that can run tests from any location where EC2 has a data center (test machines will be started and stopped automatically as needed).

The running costs will depend on the amount of testing and the server configuration but even keeping a test location busy 24×7 should run around $50 per month including the server and significantly less than that if the test machines are not needed all of the time.

The server will automatically update the web UI from github and the agent code from the public WebPagetest instance so you will also be guaranteed to always be up to date with no additional effort.

Configuring the server

The initial configuration for the server is done through the user data provided when the instance is started (once running you can modify the configuration directly as needed). Any of the server-side settings.ini settings can be specified in the user data as key/value pairs with one setting per line in the user data.

The basic server configuration assumes you will provide an access key and secret so that it can start and terminate test instances as needed. You will also want to seed the server with an API key that you know so you can run tests through the API and that is all that is needed to have a fully functioning server:

ec2_key=<access key>
ec2_secret=<secret>
api_key=MakeUpASecretAPIKeyToUse

• The server will be configured as a “headless” server which means you will only be able to run tests through the API (this can be overridden by specifying headless=0 if you want to expose the ability to run tests through the UI)
• The server will automatically populate the list of test locations based on the EC2 list in github (which will be updated automatically as AMI’s change or new data centers are opened)
• If a test is submitted for a location where no instance is running the server will start a medium instance automatically. It may take five to ten minutes for the instance to fully start on EC2 and start running tests (if you need faster responses on the first test there are config options below to keep one instance running).
• Periodically the server checks the running instances and if an instance is close to running for an hourly multiple and hasn’t seen work in the last fifteen minutes it will terminate it. The hourly interval is since EC2 instances are billed with an hourly granularity there is no benefit in shutting it down any sooner.
• Tests are kept locally on the instance storage (archiving tests to S3 or persistent disk can both be configured through the user data)

Persisting test results

You can choose to archive your test results into an S3 bucket which will make the tests available even if the server dies (though it would require starting a new instance) and it removes any storage constraints on keeping test results locally. There are a few relevant configuration parameters:

archive_s3_server=s3.amazonaws.com
archive_s3_key=<access key>
archive_s3_secret=<secret>
archive_s3_bucket=<bucket>
archive_days=1

The access key and secret can be the same as the ones used for scaling EC2 instances or keys unique to the S3 storage. The tests will be stored in S3 as soon as they complete and they will be removed from local disk after “archive_days” days (the tests will be restored transparently when accessed even after they are deleted locally). You can configure the retention policies on the S3 bucket to age-out tests as needed (say, automatically deleting tests older than 1 year for example). You can terminate the server instance after tests have run and any new server instances that are configured for the same bucket will be able to view all of the tests. That can be very helpful if you need to re-deploy the server (growing to a larger instance for example).

Tuning the number of instances

By default the server is configured to be conservative in order to keep costs down. It will shut down instances when they are not needed and it will only run one instance per test location. For doing large tests or getting faster responses on the first test you can change the behavior on a per-location basis by specifying a minimum and maximum number of instances:

EC2.us-east-1.min=1
EC2.us-east-1.max=10

This will keep 1 instance always running in the US-East location and scale up to 10 instances if the number of tests warrants it. It will start a new instance for every 100 pending tests (configurable through the EC2.ScaleFactor parameter).

Wrap-up

Hopefully the ease of deploying a private instance on EC2 lowers the bar sufficiently that people no longer see it as a daunting task and can spin them up as needed for projects they are working on. While the configuration is geared toward providing EC2 locations automatically as-needed, the server AMI is a full WebPagetest server that can also be used for more complicated environments (with custom locations, logs integration, etc).