Introduction
Whether you use JMeter, k6, Locust, or Gatling for performance tests, it is easy to be overwhelmed when interpreting test results. Poor leverage of your performance test data can hinder adoption and eventually lead to a worsened user experience. In this article, I'll share some common problems when adopting performance tests and how to solve them.
Wide performance test surface area
Performance tests can quickly absorb a lot of your team's time. To avoid this you must approach the design of your test setup as you would with a typical software application designed to scale. Modularity and optimizing for ease of maintenance can go a long way in ensuring your performance test suite grows successfully with your codebase.
Aim to define a performance test for each operation your users depend on. These operations are typically APIs, but these practices can be extended to background tasks. It can be tempting to define tests that cover multiple operations required for a complicated flow, but this will become difficult to maintain and action upon. If your APIs require a complex setup process, be deliberate about separating the setup from the actual test that is being executed.
Interpreting results in isolation without searching for trends
Teams can fall in a bad habit of optimizing for "green" performance tests rather than taking time to interpret how the performance may impact your business. Once you have attained the result data, you should compare with previous executions to identify improvements and degradations.
During this analysis step, you have the opportunity to capture action items for improvements, escalate negative trends to other teams, tune your service objectives, and document context that may be useful to future test runners. All of these are critical to using your performance tests as insights that improve your user experience.
Focusing on averages and ignoring percentiles
Understanding percentiles is important when interpreting performance test results. The percentiles p50, p75, p90, p95, and p99 are commonly used for application performance monitoring. The 75th percentile value would be the latency of which 75% of the distribution is below.
Avoid making decisions based solely on the average or p50 latency values, since you will be ignoring the experience for a large portion of your users. Define multiple objectives at varying percentiles to ensure an overall great user experience is provided. You can start with three objectives at p50, p75, p95 and adjust from there.
Losing track of action items or not quantifying their impact
Time invested into finding action items to improve performance can be wasted if teams aren't held accountable to tackle them and measure their impact. Action items should be prioritized through normal engineering cycles and attached to the initial reports that spawned the task. When an action item is completed, run another suite of performance tests to quantify and document the impact.
This cycle of investigating performance, declaring action items, completing action items, and measuring impact is a powerful process to improve your service latencies.
Falling out of habit
Teams constantly need to balance priorities of product features and performance improvements. The process for improving and testing performance should be easy to allow engineering teams to budget for it in road maps.
Take strides to automate as much of the performance testing process as possible. Build a habit of generating recurring reports to collect metrics and investigating any operations at risk of degrading. All performance investigations should be documented and act as a valuable knowledge base for your team. Be able to quantify the value added by investing in performance tests and investigations.
Summary
When done right, performance tests and investigations drive a deeper understanding of your application, an improved user experience, and better collaboration across your teams. Follow these recommendations to ensure performance tests scale and grow alongside your codebase.
Thanks for reading! The team at Latency Lingo is hard at work to build a solution to help you solve these problems. Subscribe here to receive updates on our products.