Many of us are facing the challenge of promoting the idea of speed of user experience within our organizations and crave any support data to help “sell” it to business stakeholders, product managers and engineers.

Real User Measurement (RUM) is a great tool to capture speed from actual users in the wild, measure their engagement and with one swift gesture point at an A/B test comparing speeds of two versions of the app or even use awesome “what-if” diagram to predict the impact the planned improvement.

Unfortunately, sometimes we just didn’t fund the purchase of a RUM tool, did not integrate it or simply don’t have A/B test set up and did not roll out new code to any of our user yet. This leaves us without data and therefore without a powerful convincer and visualization tool.

I was struggling with some of these problems on multiple occasions and at one point I realized that many conversations do not require actual user data.

  • some of these conversations were just about training team members who needed better grasp in statistics of web speed, distributions, percentiles and effect variability plays
  • some of them needed showing the reasoning behind prioritization of speed and visualization of known industry trends like conversion rate decay or increase in bounce rate as experiences slow down
  • and some other conversations were about supporting the very purchase of a RUM solution so business decisions could be made with real data

One day in early spring, frustration of luck of data overpowered lazy opensourcerer’s brain and I spend couple weekends coding up a visualization tool that lets you pretend you have the data simulate data and visualize it to drive your point across.

The tool is called “UX Speed Calculator” and you can use it at https://ux-speed-calculator.netlify.com/ and contribute to it on Github – it is open source and you can help add bells and whistles to satisfy your inner data geek.

UX Speed Calculator does a few useful things:

  • it calculates a log-normal distribution based on base speed and variability so you can simulate your page’s distribution and show how it is different from a normal, “bell curve” distribution many people assume by default
  • shows median, 90 and 95 percentiles so you can try and match your existing stats or simply educate engineers on what those numbers are
  • calculates bounce rate distribution and let’s you adjust the slope of degradation to your hearts desire to help explain to marketing teams the impact of speed on their marketing campaigns
  • calculates exponential decaying conversion rate distribution and helps you explain conversion poverty line and why running experiment by slowing down pages is not necessarily predictive of how much savings you’ll get by speeding them up the same amount
  • it also tries to shine light on the phenomenon on of low conversion / high bounce rates on the very left of the chart (near the zero) where despite the logic of “faster means more business” we often see unexplainable data. Many RUM experts believe it is due to the low numbers which do not produce statistically significant results, but my work with UX metrics and general lack of understanding of statistics always made it hard for me to accept that as the whole story. I postulate that is is caused by a high percentage of errors in that part of the chart and experiences that are measured as “fast” were actually incomplete, functionally broken experiences or outright error pages. So in the tool you can adjust decaying failure rate distribution. Charts also show theoretical bounce and conversion rate distributions as dotted lines and distributions that reflect error rates as solid lines
  • based on a combination of failure rates, bounce rates and conversion rates, charts also show color-coded bars for each of the populations to make it sleazier time relate to each pixel as a user who had different experience and help consumers of this visualization empathize with the data.
  • ohh yeah, I almost forgot – it also shows you the $$$ all these conversions amount to so those of your peers who don’t like math or can’t bear the cognitive load of charts can still translate the value of speed to something tangible. Feel free to substitute $ for €, £, ¥ or even ₽ if that works better for you.

If so many lines and labels confuse you and your stakeholders, feel free to click on the legends on the right to hide those that do not matter for your story.

And if you need to share a chart, just copy a URL and send it over, it updates as you tweak the knobs and dials so you can always jump to the right chart and don’t need to re-create perfection from scratch.

Hope this tool will help you with your creative story telling and you’ll be able to push the limits even further.

Happy #webperf-orming!

And remember, there are lies, damn lies and statistics! So don’t overuse this or other scientific data tools in your battles for the better within the fuzzy world of user experience!

ABOUT THE AUTHOR

Sergey Chernyshev

Sergey Chernyshev (@sergeyche) is a web performance enthusiast, open source hacker and web addict. He works as a principal engineer at Meetup and organizes New York Web Performance Meetup Group, local community of web performance geeks in New York and helps kick-start local groups about Web Performance around the world. Sergey teaches a live on-line course about performance with O'Reilly and often speaks on performance-related topics at various local New York events and global conferences, including Velocity and QCon.

20 Responses to “Visualizing Speed Without Real Data”

  1. Philip Tellis

    Hi Sergei,

    Regarding the fast bounces/low conversions, I’ve done some more research on the topic, and the reason has to do with the way we calculate the x-axis.

    The x-axis of this chart is typically set to one of two values…
    1. It can be the arithmetic mean of load times of all pages across the session or
    2. It can be the load time of the first page of the session.

    Neither of these is necessarily right. In the case of bounce rate, #2 is the correct measure as it allows you to compare like quantities. In other words, whether a user bounces or not is purely based on their experience on the first page of the session. Their experience on subsequent pages are moot as it’s too late to change the unbounced state.

    For conversions it gets a little trickier. Conversions typically don’t happen early in a session. It’s very rare to see the first or second page of a session convert, so perhaps it makes sense to ignore 1 & 2 page sessions from the calculation, however my research shows that this does not have an impact on the resulting chart.

    The second thing we notice is that the act of converting is a slow process. For example, purchasing something involves credit card processing which requires background calls to third party APIs. This process is slow, users expect this process to be slow, and users WILL NOT LEAVE at this point for fear of being double-billed. If we exclude only this page from the calculation, then we see a dramatic reduction in the number of fast-non converts, and entire converted sessions chart tends to move further to the left.

    The third thing that comes into play is various cognitive biases. In particular, serial-position bias, peak-end rule, negativity bias, and next-in-line effect have a part to play in how the overall experience has less to do than specific individual experiences over the course of the session.

    I should blog about this at some point.

  2. Sergey Chernyshev

    Hey Philip,

    Thanks a lot for sharing your research – it is one of the most mind-boggling things about RUM data and if anyone would have best answers, I knew it would be you and I really appreciate it!

    Not sure what to do about the calculator then. Should I remove the “error states” or should I just rename it somehow? Calculator did not mean to answer all the questions and shed light on all 1000 dimensions of data here (although it’s very tempting to add multimodal distributions, session lengths and whole bunch of other things) so I am not sure if simpler is better here (it already has two meanings for bounce rates which is hard to explain).

    And yes, we all want to see a blog post, a slide deck and a video with comic book-style illustrations! 😉

  3. Philip Tellis

    Oh I wouldn’t change anything here. It’s a very good tool and post. I just wanted to address an important point you brought up.

  4. Visualizing Speed Without Real Data – Binary Orders of Magnitude

    […] This article was originally posted as part of PerfPlanet Calendar on December 12th, […]

  5. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] - Print2Web

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  6. Liste de contrôle des performances frontales 2020 [PDF, Apple Pages, MS Word]

    […] auriez-vous besoin de plaider pour une solution RUM dans votre organisation, Sergey Chernyshev a construit un UX Speed ​​Calculator un outil open source qui vous aide à simuler des données et à les […]

  7. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] - The Branding Store | Logo Design, Web Design and E-commerce specialists.| Pembroke Pines, Florida.

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  8. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] - Hawkart Design Services

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  9. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] | All Pro Web Designs

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  10. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] | Cunningham Web Solutions

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  11. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] – AdvertisSEO

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  12. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] – Tech & Web News Daily

    […] or perhaps you'd need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  13. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] | Web Design News from Dubado

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  14. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] — Smashing Magazine – Dinezh.com

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  15. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] | Moore Design

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  16. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] - Web Design Tips

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  17. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] - Shawn DeWolfe Consulting

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  18. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] | IdjgieWeb

    […] or perhaps you'd need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  19. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] — Smashing Magazine – Digital Marketing Authority

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

  20. Front-End Performance Checklist 2020 [PDF, Apple Pages, MS Word] | Miden

    […] or perhaps you’d need to advocate for a RUM solution in your organization, Sergey Chernyshev has built a UX Speed Calculator, an open-source tool that helps you simulate data and visualize it to drive […]

Leave a Reply

You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>
And here's a tool to convert HTML entities