Stefan Wintermeyer (@wintermeyer) is a book author, speaker and freelancer who is specialized in Ruby on Rails, Phoenix and WebPerformance. He is located in Germany but helps his clients all over the world. Find more about his work at wintermeyer-consulting.de and his strange hobby to improve vacation times at mehr-schulferien.de and feriendaten.de. Please do open your waterfall tab in the browser to analyze those pages.
We all use compression in our daily work. It is boring because it is baked into everything. We don’t have to think about it. Or do we? In this article I’d like to share some thoughts about compression which will change the way you use it in the WebPerformance environment.
The Setup
Let’s set the stage for this article: You are an up and coming webperfomance consulting. Happy for any new project. A new client hires you for a comical exaggerated setup. I do that exaggeration to make my points more clear. Your client booked a super bowl commercial and a) wants to provide a top notch webperformance for his/her users and he/she wants to be sure that his/her server can keep up with the traffic (there’s no budget to scale up the server). Millions of visits per minute are expected in a boost period. Your client is a cheapskate and there for uses a Raspberry Pi 4 as a server. The webserver serves one static HTML file which your client’s nephew who is learning HTML created. Your client tells you that he/she likes that page very much design and content wise. You are not allowed to change the appearance of it.
Here’s the file. Let’s call it index.html.
<!DOCTYPE html>
<html>
<head>
<title>Hello World!</title>
<link rel="stylesheet" href="https://unpkg.com/tailwindcss@^1.0/dist/tailwind.min.css" />
</head>
<body class="bg-gray-200 text-center">
<h1 class="font-bold text-blue-700 py-4 text-4xl">Hello World!</h1>
<div class="text-xl">
<p class="py-4 ">
Lorem ipsum dolor sit amet, ...
</p >
<p class=" py-4">
Sed in nisl mollis, dictum arcu non, ...
</p>
<p class="py-4">
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua, ...
</p>
<p class="py-4">
consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, se,
...
</p>
<p class="py-4">
At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est
Lorem ipsum dolor sit amet.
</p >
</div>
</body>
</html>
Here’s a screenshot:
Isn’t she a beauty!? But at least you don’t have to fiddle around with design problems. 😉
Webserver Configuration
Your client’s nephew installed nginx on a Debian Linux system. Not a bad choice. Because you are first reaching for the low hanging fruits you check for it’s configuration:
server {
gzip on;
gzip_types text/plain application/xml;
...
}
Looking good so fare. The webserver always compresses HTML files before delivering them to the webclient. Unfortunately it does that every time a GET / request hits the server. So that is going to be our first and lowest hanging fruit: There is no need to compress that file every time. We do it once and store the file in the filesystem.
$ gzip -k index.html $ ls -l total 16 -rw-r--r-- 1 stefan staff 907 9 Dez 08:36 index.html -rw-r--r-- 1 stefan staff 496 9 Dez 08:36 index.html.gz
You change the nginx config to make nginx check first for a compressed version in the filesystem and use that in case it exists:
server {
gzip on;
gzip_types text/plain application/xml;
gzip_static on;
...
}
With that you already made a big difference for that expected peak in traffic. The poor Raspberry Pi has to do less work and can deliver more pages. No compression on the server means faster delivery to the web client. Better webperformance.
The client’s nephew looks over your shoulder and is kind of a smart ass. He tells you that the command gzip -9 will result in an even smaller file. The -9 flag tells the gzip command to use the highest level of compression, which will compress the file as much as possible but may take longer to run than lower compression levels.
When one runs the gzip command without the -9 flag, it will use the default compression level, which is typically level 6. This level provides a good balance between compression ratio and speed, but it may not compress the file as much as using the -9 flag.
That’s worth a try so we do that:
$ gzip -kc -9 index.html > index-gzip9.html.gz $ ls -l total 24 -rw-r--r-- 1 stefan staff 496 9 Dez 08:48 index-gzip9.html.gz -rw-r--r-- 1 stefan staff 907 9 Dez 08:36 index.html -rw-r--r-- 1 stefan staff 496 9 Dez 08:36 index.html.gz
No difference! Why? Mainly because the original index.html file is too small to give the compression algorithm enough food to come up with a better compression ratio. With bigger files you will see a difference.
Try harder
But you don’t want to give up yet on the idea of a better compression. Isn’t there a better compression tool for gzip files? Yes, there is! It is called Zopfli (https://github.com/google/zopfli) and was created by Google developers. Let’s give it a shot:
$ mv index.html.gz index-gzip.html.gz $ zopfli index.html $ ls -l total 32 -rw-r--r-- 1 stefan staff 496 9 Dez 09:03 index-gzip.html.gz -rw-r--r-- 1 stefan staff 496 9 Dez 08:51 index-gzip9.html.gz -rw-r--r-- 1 stefan staff 907 9 Dez 09:04 index.html -rw-r--r-- 1 stefan staff 475 9 Dez 09:04 index.html.gz
That did reduce the filesize by about 5%. That is an easy win! For bigger files it also makes sense to use the --i50 (or a higher value) switch to force zopfli to try more often/harder.
Try even harder
Zopfli is a diminutive for the Swiss word Zöpfli which is a yeast braid. For this kind of software Google developers team names its projects after Swiss bakery products. And guess what, there is an other compression tool by them. It is called Brotli (https://github.com/google/brotli) which is the Swiss word for bread. Let’s try brotli:
$ brotli index.html $ ls -l total 40 -rw-r--r-- 1 stefan staff 496 9 Dez 09:03 index-gzip.html.gz -rw-r--r-- 1 stefan staff 496 9 Dez 08:51 index-gzip9.html.gz -rw-r--r-- 1 stefan staff 907 9 Dez 09:04 index.html -rw-r--r-- 1 stefan staff 399 9 Dez 09:04 index.html.br -rw-r--r-- 1 stefan staff 475 9 Dez 09:04 index.html.gz
It optimized about 15% better than zopfli. That is a big win! But it has a different file extension and you can not inflate it with gzip. Luckily all modern browsers can use brotli compressed files by default today (check on https://caniuse.com/brotli if you don’t believe me). But you have to tell nginx to use it too (https://github.com/google/ngx_brotli). We add the following code to our nginx configuration:
brotli on; brotli_comp_level 6; brotli_static on; brotli_types text/plain application/xml;
No changes to the look and feel of the page
Your client asked you not to change the content of the webpage. By that he/she ment the design and the text. But it is worth a try to optimize the HTML. That source code did look pretty messy. Because of that you rename it to messy.html and delete all the other files.
$ mv index.html messy.html $ rm index* $ brotli messy.html $ ls -l total 16 -rw-r--r-- 1 stefan staff 907 9 Dez 09:04 messy.html -rw-r--r-- 1 stefan staff 399 9 Dez 09:04 messy.html.br
Before we do anything manually we can use the command line tool tidy (https://www.html-tidy.org) to clean up the HTML code:
$ tidy -i --show-warnings false --wrap 0 messy.html | grep -v "Tidy" | grep -v "generator" | grep . > tidy.html $ brotli tidy.html $ ls -l total 32 -rw-r--r-- 1 stefan staff 0 9 Dez 12:34 grep -rw-r--r-- 1 stefan staff 907 9 Dez 09:04 messy.html -rw-r--r-- 1 stefan staff 399 9 Dez 09:04 messy.html.br -rw-r--r-- 1 stefan staff 905 9 Dez 12:40 tidy.html -rw-r--r-- 1 stefan staff 373 9 Dez 12:40 tidy.html.br
We saved another 7% compared to the vanilla messy.html.br. That is because the new HTML file is more compressible:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
<head>
<title>Hello World!</title>
<link rel="stylesheet" href="https://unpkg.com/tailwindcss@^1.0/dist/tailwind.min.css" type="text/css">
</head>
<body class="bg-gray-200 text-center">
<h1 class="font-bold text-blue-700 py-4 text-4xl">Hello World!</h1>
<div class="text-xl">
<p class="py-4">Lorem ipsum dolor sit amet, ...</p>
<p class=" py-4">Sed in nisl mollis, dictum arcu non, ...</p>
<p class="py-4">eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua, ...</p>
<p class="py-4">consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, se, ...</p>
<p class="py-4">At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.</p>
</div>
</body>
</html>
While we are looking at the HTML code … lets figure out if we can manually change it in a way that we don’t change the design of the page. We do the following changes:
- Remove the
text-centerfrom the body class and put it into every element below. - Remove the
<div class="text-xl">and put thetext-xlclass into every element below.
The result of that change:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"> <html> <head> <title>Hello World!</title> <link rel="stylesheet" href="https://unpkg.com/tailwindcss@^1.0/dist/tailwind.min.css" type="text/css"> </head> <body class="bg-gray-200"> <h1 class="font-bold text-blue-700 py-4 text-4xl text-center">Hello World!</h1> <p class="py-4 text-center text-xl">Lorem ipsum dolor sit amet, ...</p> <p class="py-4 text-center text-xl">Sed in nisl mollis, dictum arcu non, ...</p> <p class="py-4 text-center text-xl">eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua, ...</p> <p class="py-4 text-center text-xl">consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, se, ...</p> <p class="py-4 text-center text-xl">At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.</p> </body> </html>
We save this as manual-optimization.html and realize two things:
- The filesize of the HTML file got bigger!
- The filesize of the Brotli compressed version of that file is smaller than everything we saw yet. About 4%.
$ brotli manual-optimization.html $ ls -l total 48 -rw-r--r-- 1 stefan staff 961 9 Dez 12:52 manual-optimization.html -rw-r--r-- 1 stefan staff 360 9 Dez 12:52 manual-optimization.html.br -rw-r--r-- 1 stefan staff 907 9 Dez 09:04 messy.html -rw-r--r-- 1 stefan staff 399 9 Dez 09:04 messy.html.br -rw-r--r-- 1 stefan staff 905 9 Dez 12:40 tidy.html -rw-r--r-- 1 stefan staff 373 9 Dez 12:40 tidy.html.br
So it pays out to have a larger RAW HTML file in case the content of it is more repetitive and therefore easier to compress. We did not change the design or the content. Just the way we encode it into HTML.
In the world of extrem webperformance you really need to understand the used pipeline.
Are we done yet?
Until now everything we did improved the HTML code in some way or another. But since we want to get the best result for our client and therefor the smallest possible file size we have to dive deeper into the structure and logic of HTML. Do we need a nicely formated HTML source code? No, we don’t. Let’s get rid of all those unneeded spaces and carriage returns. Do we need to close all HTML tags? No, we don’t. Believe it or not. We don’t even need to close the <html> tag with a </html> at the end of our document. Webbrowsers are extremely good in understanding “untidy” HTML code. They simply don’t care.
With html-minified (https://www.npmjs.com/package/html-minifier) we have a tool which takes care of all that work for us:
$ html-minifier --collapse-whitespace --remove-comments --remove-optional-tags --remove-redundant-attributes --remove-script-type-attributes --remove-tag-whitespace --use-short-doctype --minify-css true --output html-minified.html manual-optimization.html $ brotli html-minified.html $ ls -l total 64 -rw-r--r-- 1 stefan staff 839 9 Dez 13:05 html-minified.html -rw-r--r-- 1 stefan staff 333 9 Dez 13:05 html-minified.html.br -rw-r--r-- 1 stefan staff 961 9 Dez 12:52 manual-optimization.html -rw-r--r-- 1 stefan staff 360 9 Dez 12:52 manual-optimization.html.br -rw-r--r-- 1 stefan staff 907 9 Dez 09:04 messy.html -rw-r--r-- 1 stefan staff 399 9 Dez 09:04 messy.html.br -rw-r--r-- 1 stefan staff 905 9 Dez 12:40 tidy.html -rw-r--r-- 1 stefan staff 373 9 Dez 12:40 tidy.html.br
WOW! We got down to 333 for the brotli compressed version of that file (html-minified.html.br). Our very first index.gz file had a file size of 496 which means that we saved 33%. That is a huge optimization!
The HTML doesn’t look pretty but it is standards-conforming and the computer doesn’t care:
<!doctypehtml><title>Hello World!</title><link rel="stylesheet"href="https://unpkg.com/tailwindcss@^1.0/dist/tailwind.min.css"type="text/css"><body class="bg-gray-200"><h1 class="font-bold text-blue-700 py-4 text-4xl text-center">Hello World!</h1><p class="py-4 text-center text-xl">Lorem ipsum dolor sit amet, ...<p class="py-4 text-center text-xl">Sed in nisl mollis, dictum arcu non, ...<p class="py-4 text-center text-xl">eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua, ...<p class="py-4 text-center text-xl">consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, se, ...<p class="py-4 text-center text-xl">At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
The resulting rendered webpage is still the same. But the payload decreased dramatically. Smaller files are faster delivered to the webbrowser. That results in better WebPerformance and as a byproduct less needed CPU power on the server side. I call this a win-win scenario.
Summary
We had a very minimal Hello World! example HTML file. The possibilities of optimization are bigger the bigger and more complex the HTML files are. So the effects you saw with our artificial example would become even bigger in real life HTML files.
Bonus Improvement
Go one step back and rethink our given problem. Can you think of one more improvement which would increase the webperformance and decrease the server load?
TCP Slowstart starts with an initial 14 KB payload in the first packet. You can not transport less in that first packet! Our initial index.html file had a file size which was below that 14 KB threshold. What does that mean? We would actually gain webperformance by not using any compression at all! Tell nginx to not compress that file (e.g. with the gzip_min_length software switch). By that the webbrowser doesn’t have to unzip the file which saves time and CPU power. Increasing the total webperformance.
But that is just because our example file is such an edge case. I just add this bonus improvement to point out the importance of understand the pipeline aspect. Never underestimate the improvement potential even of such a simple setup.