The new Github Actions integration helps users automate performance tests in CI/CD workflows to catch performance degradation before their users.

Announcing The New<br/> Github Actions Integration

Latency Lingo is excited to announce the release of the new integration with Github Actions! Visit the [page on Github marketplace](https://github.com/marketplace/actions/latency-lingo-performance-test-coverage) to install it to your repository.

This integration allows software teams to automatically publish performance metrics from their CI/CD pipeline, making it easier to track the performance of their software over time and maintain a great user experience. Performance testing tools supported include [JMeter](https://jmeter.apache.org), [Gatling](https://gatling.io/), and [k6](https://k6.io/).

With this integration, users can easily introduce Latency Lingo as a step in their Github workflow. Each time code is pushed to the repository, your performance metrics can be published to the user's Latency Lingo account. These metrics can then be accessed by logging in to the account and navigating to the dashboard.

[Thresholds](https://latencylingo.com/blog/introducing-performance-test-thresholds) can also be set to fail builds if the performance metrics fall below a certain level. This ensures that software teams are alerted to any potential performance issues before their users experience them.

Additionally, users can add a Github shield to their README file to preview performance metrics from the latest build. This shield will display the run's P90 response time or number of thresholds passed.

![](/img/docs/github-badge-example.png)

Get started with this integration by [following the documentation](https://docs.latencylingo.com/docs/integrations/github-actions). Users simply need to sign up for a free Latency Lingo account and install the Github Action in a workflow.

I also recommend reading this post to see an example use case - [How to automate performance testing with JMeter and Github](https://latencylingo.com/blog/automate-performance-testing-with-jmeter).

This integration will make it even easier for software teams to monitor and improve the performance of their software using Latency Lingo. If you have any issues, please don't hesitate to [contact support](mailto:support@latencylingo.com) for assistance.

## Learn more

- [Latency Lingo Github integration](https://github.com/marketplace/actions/latency-lingo-performance-test-coverage)
- [Latency Lingo documentation](https://docs.latencylingo.com/docs/integrations/github-actions)
- [How to automate performance testing with JMeter and Github](https://latencylingo.com/blog/automate-performance-testing-with-jmeter)
- [Github workflow syntax](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)


Latency Lingo can now integrate with Gatling load tests. Get more out of your Gatling tests by using our analytics and automation tools.

Added Support for Gatling

Attention all Gatling users!

I'm excited to announce that Latency Lingo can now integrate with [Gatling](https://gatling.io) load tests. This means that you can use Latency Lingo to analyze and act upon the data collected from your Gatling tests, leading to better application performance decisions.

<img
  style={{ marginLeft: 'auto', marginRight: 'auto' }}
  src="/img/gatling-logo.png"
  alt="gatling"
/>

Gatling is a popular open-source load testing tool that allows users to simulate large numbers of users making requests to a system, helping to identify performance issues and bottlenecks. By integrating Latency Lingo with Gatling, you can easily automate performance testing, track performance trends over time, and make informed decisions about how to optimize application performance.

To get started with this integration, [view our documentation](https://docs.latencylingo.com/docs/integrations/gatling). Leverage Latency Lingo to get more out of your Gatling performance tests.

If you have any questions or need further assistance, don't hesitate to [contact support](mailto:support@latencylingo.com).


Latency Lingo thresholds is a new feature to help software teams monitor their performance tests and identify performance regressions.

Introducing Performance Test Thresholds

[Performance test thresholds](https://docs.latencylingo.com/docs/product/test-runs#thresholds-panel) are a new feature that is now available to all Latency Lingo users! This feature helps software teams monitor the health of their performance tests and identify performance regressions.

<BrowserWindow
  minHeight={0}
  url="https://latencylingo.com/test-runs/0fc77990-87ab-4803-8ffa-d12115875a60"
>
  <img
    style={{ marginTop: 0 }}
    src="/img/docs/test-run-overview.png"
    alt="Latency Lingo test run overview"
  />
  <img
    style={{ marginTop: 0 }}
    src="/img/docs/test-run-thresholds.png"
    alt="Latency Lingo test run thresholds"
  />
</BrowserWindow>

Performance testing is an essential part of the software development process. It helps teams ensure that their applications can handle the expected workload and provide a good user experience. However, manually analyzing and interpreting performance test results can be time-consuming and error-prone.

Latency Lingo solves this problem by providing a comprehensive, easy-to-use platform for analyzing performance test results and the new thresholds feature takes this to the next level.

Thresholds allow teams to define pass/fail conditions for their performance tests, based on metrics such as response time, error count, and throughput. This makes it easy for teams to ensure they are meeting their performance requirements and objectives.

Since Latency Lingo can integrate with any load test tool like JMeter, K6, and Gatling, you no longer need to rely on a custom process to programmatically evaluate the test results from open-source tools. Instead, you can set up your tests to automatically send results to Latency Lingo and then use thresholds to monitor the health of your tests and notify you in case of failure.

<BrowserWindow
  minHeight={0}
  url="https://latencylingo.com/test-runs/0fc77990-87ab-4803-8ffa-d12115875a60"
>
  <img
    style={{ marginTop: 0 }}
    src="/img/docs/test-run-thresholds-new.png"
    alt="Latency Lingo test run create thresholds"
  />
</BrowserWindow>

Thresholds are also designed to be easily integrated with your CI/CD test automation pipeline like Github Actions or Jenkins. With this feature, you can use Latency Lingo to help determine your build status based on the results of your automated performance tests. This allows you to catch performance issues early in the development process before they become a problem in production.

You can [visit the documentation](https://docs.latencylingo.com/docs/product/test-runs#thresholds-panel) to learn more. Sign up for free to try it out and please [share any feedback](mailto:support@latencylingo.com)!

Have any questions or want to hear more?

- [Send me an email](mailto:support@latencylingo.com)
- [Follow us on LinkedIn](https://www.linkedin.com/company/latencylingo/)
- [View more content](/blog)


Traffic spikes are a great opportunity to evaluate the performance and reliability of your website. Here we share how you can do this with Black Friday traffic.

How to use Black Friday Website Performance to Improve Site Reliability

Black Friday is one of the busiest shopping days of the year, and as such, it's a great opportunity to evaluate the performance and reliability of your website. By analyzing how your website performed on Black Friday, you can identify areas for improvement and take steps to make your site more reliable for the next time traffic reaches these heights.

If your website traffic is not impacted by Black Friday, you can still use these tips to analyze your site's performance and reliability during any traffic spike.

This analysis should include data from the week before and after Black Friday. This will give you a more complete picture of how your site performed during the traffic spike period.

**Run a wide scan of your performance**: Check your observability platform - eg. [Datadog](https://www.datadoghq.com/) or [Grafana](https://grafana.com/) - to track your site's performance on Black Friday using your existing dashboards. Look for any critical monitors that have fired off during this period. It can also be useful to check for any recently filed user complaints that mention poor performance or increased errors.

This scan is intended to help you identify any obvious problems, such as slow loading times or errors, that may have been affecting your site's performance. The objective is to find hotspots that we will treat as entry points for deeper analysis. It is also likely that you'll find blind spots in observability that you can address in the future.

**Analyze your website's traffic patterns**: Analyzing your website's request traffic volume on Black Friday can give you valuable insights into how well your site can handle high levels of traffic. Look for patterns and trends to help you identify any potential bottlenecks. Important metrics to guage include: request count, concurrent users count, and the ramp-up period for these concurrent users. These metrics will give you a better understanding of your scaling needs.

**Optimize your website's loading times**: Slow loading times can be a major issue for websites, especially during peak traffic periods like Black Friday. It is good to start with the endpoints your application is spending the most time processing. An example of calculating this would be to use the sum of response time for all requests to a given endpoint.

I recommend comparing slow endpoint performance with their performance during normal traffic periods. Some of these endpoints may have scaling issues where their performance degrades with volume and others may be consistently slow but only cause site issues at higher volume. Depending on the situation, you can optimize these endpoints by optimizing their code or increasing their resources.

Optimizations on the frontend can include things like optimizing images, minifying CSS and JavaScript, and reducing the number of HTTP requests your site makes. Backend optimizations include introducing caches, restructuring data query patterns, adding indices, lowering cost of data serialization, and service deployment configuration.

**Increase your website's capacity**: If your website is struggling to handle the high levels of traffic on Black Friday, you may need to increase its capacity. This can include things like upgrading your application deployment strategy to add more resources, introducing cache instances, adding database replicas, and resizing servers. The strategy here varies heavily depending on your application's architecture and the resources it uses.

Be careful with defaulting to this approach, since it can be expensive and may not be necessary. It is important to understand the root cause of the performance issues before making any changes. Adding resources can be a good way to mitigate performance issues, but it may not be a long-term solution.

**Test your website's reliability**: Testing your website's reliability is a great way to identify any potential issues that may arise during high traffic periods. A good strategy for this is to run performance tests like load tests and stress tests. These tests can help you identify any potential issues with your application's architecture, such as bottlenecks, and help you identify any potential issues with your application's code, such as memory leaks. [JMeter](https://jmeter.apache.org/), [Gatling](https://gatling.io/), [Locust](https://locust.io/), and [k6](https://k6.io/) are some of the most popular tools for these types of tests.

Based on your performance analysis, you should always ask yourself whether there are any tests that you can implement which would have caught issues before they became a problem. If so, you should consider implementing these tests in the future. There may also be constraints preventing you from running these tests effectively, such has testing environments with limited resources. This is a great time to raise these issues to your team and discuss how to address them.

**Remove observability blind spots**: During these exercises, it's common to find new metrics that indicate areas of concern or improvement. It is important to update your observability tools - dashboards and monitors - to include these new metrics. This will help you to better monitor your site's performance in the future.

## Closing Thoughts

By taking these steps, you can use Black Friday website performance to identify areas for improvement and make your site more reliable. This will help ensure that your website can handle the demands of peak traffic periods, and provide a better experience for your customers. Remember, Boxing Day is just around the corner!

I'm building [Latency Lingo](https://latencylingo.com) to help teams better leverage performance testing, so I'd encourage you to [sign up](https://latencylingo.com/signup) and try it out. Feedback is always appreciated!

Have any questions or requests for additional content?

- [Send me an email](mailto:support@latencylingo.com)
- [Follow us on LinkedIn](https://www.linkedin.com/company/latencylingo/)
- [View other content](/blog)
- [Reach out on Twitter](https://twitter.com/anthony_bobsin)
- [Subscribe for updates](/mailing-list)


A guide on performance test metrics instrumentation, tagging, collection, reporting, monitoring, and insights.

What You Need to Know About Performance Testing Metrics, Reporting, and Analytics

Performance testing is an umbrella term to describe different types of software testing that check how responsive a system is under various circumstances. While there are many different types of performance tests, they all share the same goal of isolating performance bottlenecks so they can be fixed. Load test tools are available to help execute these tests, but software engineering teams still need to develop their own processes to monitor the test results. To help, I've compiled this guide that covers what you need to know about performance testing metrics, reporting, and analytics.

## Performance testing metrics

To begin, let us identify some key metrics that are useful indicators when identifying performance bottlenecks. To explain these metrics, it's useful to start with the questions we aim to answer with performance tests and then work from there.

**How much request traffic is being sent?**

Metric: Request count. This tells us how many requests are being sent to the service from our load test runner tool.

**How many errors are there?**

Metric: Failure count and rate. This signal shows how many requests were unsuccessful, typically depicted by an error response status code. It is useful to break this down for known error types or status codes.

**How long did it take for the service to respond?**

Metric: Response times. This can include network overhead along with the time the service took to process our request and provide a response. For these metrics it is extremely useful to have access to various percentiles of the values. A common set of percentiles to monitor for response times are p50, p75, p90, p95, and p99. By capturing these different values you can better understand how performant your user experience is across the board. [This article by Dynatrace](https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/) helps explain why.

**How many concurrent users are involved in the test?**

Metric: Virtual user count. This is another indicator to show how much traffic we are simulating in our load test. This may seem similar to reqest count, but the difference is that virtual users are the threads or processes responsible for sending the requests. There should be correlation between these metrics, so it is useful to compare this metric with request count to see how the service is handling the increase in user volume.

**Do the servers have appropriate resources to handle the load?**

Metric: CPU and memory usage. This can be in reference to the client executing the load test or the servers accepting the test traffic. An overutilized client can act as a bottleneck and cause unintended test results. If your client is hitting a bottleneck despite healthy CPU and memory usage, you may need to [update your OS settings for resource limits, eg. ulimit](https://linuxhint.com/linux_ulimit_command/).

## Performance testing dimensions

Metrics are quantitative values that alone would represent an overall view of your systems. It is often not useful to analyze metrics across all your systems and environments. To better query and gain insights from these metrics, you can associate these metrics with descriptive attributes known as dimensions. By tagging each metric with these descriptive attributes it makes it a lot easier to navigate and make sense of this data.

Here are some common dimensions used to tag performance testing metrics:

- Hostname: The host where the metric is emitted.
- Environment: The environment where the metric is emitted.
- Test plan: The name of the test plan that is being executed.
- Operation name: The name of the HTTP operation (eg. HTTP method and URL) being executed.
- Status code: The HTTP response status code.
- Error type: Category of error received by the service.

## Where to collect metrics

These metrics should be collected on both the client and server sides. The client is the environment where the performance test is being executed. The server is the application being tested.

Some teams opt for just collecting metrics from the server side as part of their application performance monitoring, but this can cause discrepancies due to network latency, network failures, server request queueing, cached requests, overloaded test environments, and server middleware overhead.

To allow for correlation between metrics across the client and server sides, you can leverage HTTP headers and trace IDs. The strategy of how this works depends on how you store metrics, which is discussed in the next section.

### Where to store metrics

There are a few options when deciding where to store these metrics. If you prefer to self-host, you can use a combination of [InfluxDB](https://www.influxdata.com/) and [Grafana](https://grafana.com/). For managed solutions, I recommend [Datadog](https://www.datadoghq.com/), [NewRelic](https://newrelic.com/), and [Dynatrace](https://www.dynatrace.com/).

For your server-side integration, each of these observability platforms offers SDKs that you can use to publish metrics. Client-side metrics may be a bit more difficult depending on which tool you use to execute load tests. If you're using JMeter, you can install a custom backend listener that publishes metrics to these platforms.

## Test metric reporting

Each of the observability platforms from the previous section offers flexible reporting and visualization tools to build test results dashboards. That being said, these dashboards can be difficult to define and maintain without the appropriate domain expertise.

You can build a dashboard that highlights your business KPIs using the metrics and dimensions mentioned previously. A common approach is to build a shared dashboard that allows dimensions to be specified through variables. This allows your dashboard to be reusable across all tests, rather than building a dashboard for each test plan.

<BrowserWindow
  minHeight={0}
  url="https://latencylingo.com/test-runs/0fc77990-87ab-4803-8ffa-d12115875a60"
>
  <img
    style={{ marginTop: 0 }}
    src="/img/performance-metrics-reporting/dashboard.png"
    alt="Latency Lingo Dashboard Example"
  />
</BrowserWindow>

[Latency Lingo](https://latencylingo.com) is a simple, opinionated observability platform dedicated to web performance testing. It provides an out-of-the-box reporting toolset so that you don't need to design your own reporting infrastructure using domain expertise. The product is still very early, but please try it out and let me know your thoughts! If you're willing, I'd also love to chat with you about your performance testing problems to see how Latency Lingo can eventually help solve them. You can reach me [by email](mailto:anthony.bobsin.dev@gmail.com) and we'll schedule something.

## Metric analysis and interpretation

Dashboards are useful for ad-hoc analysis, but you cannot expect your team to always be monitoring these dashboards. In some conditions, your team may want to be automatically notified so that you can start an investigation. An example of this case is if your performance tests suddenly cause an increase in error rate or response times.

I recommend setting up automated monitors that check your error rate and response times to help evaluate system health. These monitors can be configured to evaluate metrics over a specific time interval and at a specific frequency. You can also set these monitors to integrate with your paging systems like [OpsGenie](https://www.atlassian.com/software/opsgenie) or [PagerDuty](https://www.pagerduty.com/) to be notified when further investigation is required.

When deciding the conditions for your monitors a common approach is to define service-level objectives for each of your HTTP operations. You can read more about this in [Google's book on site reliability engineering](https://sre.google/sre-book/service-level-objectives/).

Besides comparing with defined response times and errors thresholds, here are some other conditions that you may want to evaluate:

- Throughput limit hit: Virtual users are increasing while requests per second have plateaued.
- Failure rate anomaly: Noticed a short-term spike in failure rate.
- Response time anomaly: Noticed a short-term spike in response times.
- Test plan duration: Overall execution time of a test plan.
- Load test environment CPU usage: Acceptable CPU usage for load test tool.
- Load test environment memory usage: Acceptable memory usage for load test tool.
- Negative historical trend: Response times or error rates are negatively changing across recent tests.

Teams should also develop a habit of reviewing performance dashboards manually on a recurring basis to identify any unexpected changes or potential bottlenecks that didn't trigger monitor alerts. This is useful collaborative process to ensure your software maintains good performance and better understand your systems.

### Closing thoughts

To recap, these are the steps to analyzing performance testing results:

- Identify key metrics and dimensions.
- Emit these metrics from your load test runner and application to your observability platform.
- Build reporting dashboards for your metrics.
- Define automated monitors for unhealthy metric conditions.
- Analyze test runs to find bottlenecks and prioritize improvements.

I hope you've enjoyed this guide! I'm building [Latency Lingo](https://latencylingo.com) to help teams understand their performance test results, so I'd encourage you to sign up, try it out, and share any feedback.

If you're willing, I'd love to have a chat and learn about the performance testing problems you face. Please reach out and we can schedule something!

- [Send me an email](mailto:anthony.bobsin.dev@gmail.com)
- [Subscribe for updates](/mailing-list)
- [Reach out on Twitter](https://twitter.com/anthony_bobsin)
- [Follow on LinkedIn](https://www.linkedin.com/company/latencylingo/)


Learn how to automate your web performance testing with JMeter, Github Actions, and Latency Lingo.

How to automate performance testing with JMeter and Github Actions

Performance testing involves sending requests to an application to help find performance issues and confirm that it runs as expected. This is a difficult task and can be especially tedious for those who have not worked with test automation before.

In this article, we will demonstrate how you can automate performance tests with JMeter and Github Actions. You can find the source code for all examples in [this Github repo](https://github.com/AnthonyBobsin/automate-jmeter-example).

## JMeter and Performance Testing

[JMeter](https://jmeter.apache.org/) is a tool that allows you to write and run performance tests against your application. There are many other tools for generating load for testing purposes, but JMeter is one of the most mature. Similar alternatives include [K6](https://k6.io), [Gatling](https://gatling.io), and [Locust](https://locust.io).

JMeter provides both a graphical user interface and a command-line interface. It is common for teams use the GUI to edit their test plan and the CLI to execute the test plan. The CLI is especially convenient for invoking tests on an automation server.

Performance testing can apply to a large variety of systems. JMeter has tremendous support for a wide range of protocols and load testing strategies. This is what has popularized JMeter over the decades - the ability to handle any type of load test scenario.

## Set Up Your Environment

There are two environments we will need to setup to start running our automated tests.

- A testing or staging instance of the application being tested.
- An automation server where we can execute JMeter against the test application.

You may already have some testing or staging instance of the application from your manual testing efforts. Consider creating a dedicated instance for performance tests to avoid manual testers or other forms of automation from impacting the results.

For the automation server you have many services at your disposal to reduce the maintenance burden. I recommend [Github Actions](https://docs.github.com/en/actions), [Jenkins](https://www.jenkins.io/), [Circle CI](https://circleci.com/), or [Travis CI](https://www.travis-ci.com/). The main idea is to have some server that listens for code changes and can execute steps that depict whether the code change is safe or not.

Github Actions are simple to setup and will be used in this demo. We'll configure [PerfAction](https://github.com/QAInsights/PerfAction) on our Github repo to execute JMeter tests on new code changes.

## Create A Test Plan For Your Automated Tests

With your test environment set up, you can now create a test plan. This is how you communicate to JMeter what requests it should run and in what order. Please see [my other article](https://latencylingo.com/blog/web-performance-testing-overview) for an overview on building a test plan.

For the purpose of this demonstration, I created a basic test plan that walks through a checkout flow for an online storefront. You can view the test plan [here](https://github.com/AnthonyBobsin/automate-jmeter-example/blob/main/jmeter-files/test-plan.jmx).

![Test plan thread group](/img/automation-with-jmeter-blog/test-plan-1.png)
![Test plan sampler](/img/automation-with-jmeter-blog/test-plan-2.png)

Our test plan iterates five different endpoints to take the user from signing in to checking out. It will simulate 50 concurrent users repeating these checkout steps for 5 minutes with a 2 minute ramp-up period.

Your automation test plan should not be an extreme load test, but rather focused on a comfortable load that gives an accurate representation of performance. There are chances that this test plan could be executed concurrently for multiple changes and if the volume is too large it can impact your test environment.

Always be sure to share your test plan with team members for other opinions!

## Run a Test and Analyze Results

Before we can schedule our test plan for automation, you should manually test it on a safe environment and also ensure you have system observability through application metrics.

Instrumenting system and application metrics is a recommended prerequisite to adopting this level of testing. Teams typically use [Grafana](https://grafana.com/), [Datadog](https://www.datadoghq.com/), [Dynatrace](https://www.dynatrace.com/), or [Sentry](https://sentry.io) for this purpose. During manual tests, the tester is actively monitoring the results and can stop the tests if unintended impact is noticed. When these tests start running automatically, you cannot afford to have someone monitoring each time the tests are run.

To accommodate this lack of a tester monitoring the results, you can leverage automated monitors in your chosen observability platform. These monitors define conditions for when the environment is healthy and the team should get paged if the conditions are breached.

Here are some basic monitors as an example.

- 5xx error rate eg. 5xx percentage < 1%.
- 4xx error rate eg. 4xx percentage < 10%.
- Response time at a specific percentile eg. latency p75 < 5s.

These monitors are not exhaustive and I suggest you learn more about what metrics indicate poor performance for your specific application.

Setting up an observability platorm and monitors is out-of-scope for this guide, but there are a lot of good resources available. For example you can follow [this guide by QAInsights](https://qainsights.com/jmeter-integration-with-influxdb-2-0/) to start emitting JMeter metrics to Grafana using InfluxDB.

Latency Lingo can also help you set thresholds and collaborate on performance testing reports. [Thresholds](https://latencylingo.com/blog/introducing-performance-test-thresholds) are used to alert you when performance is outside of a healthy range, which is especially useful in CI/CD environments.

## Automation with CI

We are now ready to create our automation workflow that runs the test plan. Let's finish configuring [PerfAction](https://github.com/QAInsights/PerfAction) to execute our test plan and also use the [Latency Lingo Github Action](https://github.com/marketplace/actions/latency-lingo-performance-test-coverage) to analyze test results.

Github Actions are configured using YAML files within your Github repo's `.github/workflows` directory. Please refer to [their documentation](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions) for any details.

Github Actions revolve around jobs and steps. A job is a collection of steps that are executed in sequence. A step is a single command that is executed. Step commands can invoke other published actions or a manual bash command.

The snippet below is the actions file we'll use to run our test plan. You can view the source code [here](https://github.com/AnthonyBobsin/automate-jmeter-example/blob/main/.github/workflows/run-jmeter-tests.yml). We will step through this example to show how it works.

```yaml
on: [push]
jobs:
  Main:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run JMeter Tests
        uses: QAInsights/PerfAction@v3.1
        with:
          test-plan-path: jmeter-files/test-plan.jmx
          args: ''
      - name: Upload Results
        uses: actions/upload-artifact@v2
        with:
          name: jmeter-results
          path: result.jtl
      - name: Analyze Results with Latency Lingo
        uses: latency-lingo/github-action@v0.0.2
        with:
          api-key: ${{ secrets.LATENCY_LINGO_API_KEY }}
          file: result.jtl
          label: Checkout Flow Automated Test Plan
          format: jmeter
```

For our example, we have a single job called `Main` that has five steps. The `on` argument tells the workflow to run on each push to a remote branch. The `runs-on` argument tells the workflow to run on Ubuntu Linux.

Most of the complexity lives within the `steps` we define, so let's explain them one-by-one.

1. Pull the Github repo to make it available in future steps.

```yaml
- uses: actions/checkout@v3
```

2. Use [PerfAction](https://github.com/QAInsights/PerfAction) to run our JMeter test plan.

```yaml
- name: Run JMeter Tests
  uses: QAInsights/PerfAction@v3.1
  with:
    test-plan-path: jmeter-files/test-plan.jmx
    args: ''
```

3. Store the JMeter results as a Github artifact for future reference.

```yaml
- name: Upload Results
  uses: actions/upload-artifact@v2
  with:
    name: jmeter-results
    path: result.jtl
```

4. Finally, we'll publish our results to Latency Lingo. The CLI requires the API key found in your [account settings](https://latencylingo.com/account/api-access). We use [Github Actions secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) to store it safely. You can update the label to match your test scenario.

```yaml
- name: Analyze Results with Latency Lingo
  uses: latency-lingo/github-action@v0.0.2
  with:
    api-key: ${{ secrets.LATENCY_LINGO_API_KEY }}
    file: result.jtl
    label: Checkout Flow Automated Test Plan
    format: jmeter
```

Now whenever we push a commit, you should see that Github Actions will run all these steps automatically.

<BrowserWindow
  minHeight={0}
  url="https://github.com/AnthonyBobsin/automate-jmeter-example/runs/8006800406"
>
  <img
    style={{ marginTop: 0 }}
    src="/img/automation-with-jmeter-blog/github-action-result.png"
    alt="Github Action results"
  />
</BrowserWindow>

The output is a test report generated in Latency Lingo where we can view the test results. If you configured instrumentation and monitors, you can verify the data is populated there as well.

<BrowserWindow
  minHeight={0}
  url="https://latencylingo.com/test-runs/8b25212c-761d-41f8-ab05-8d9bb79d8533"
>
  <img
    style={{ marginTop: 0, marginBottom: 0 }}
    src="/img/automation-with-jmeter-blog/latency-lingo-result.png"
    alt="Latency Lingo report"
  />
  <img
    style={{ marginTop: 0 }}
    src="/img/automation-with-jmeter-blog/latency-lingo-result-2.png"
    alt="Latency Lingo report"
  />
</BrowserWindow>

## Closing Thoughts

Performance testing is essential for validating that your application runs as expected under various loads. It can be a difficult task for anyone who has not worked with test automation before. With the right tools and processes in place, it can be easy to leverage automated performance testing to improve your product quality.

You can [visit the documentation](https://docs.latencylingo.com/docs/intro) to learn more about how Latency Lingo can help your team performance test effectively. Sign up for free to try it out and please [share any feedback](mailto:support@latencylingo.com)!

Have any questions or want to hear more?

- [Send me an email](mailto:support@latencylingo.com)
- [Follow us on LinkedIn](https://www.linkedin.com/company/latencylingo/)
- [View more content](/blog)


Keep up with the industry on web performance testing. Testing site performance is easier than ever before! Learn what performance testing is, how it works, as well as some of its most common uses.

Web Performance Testing - What It Is, How It Works, And How You Can Use It

## Introduction

Web performance is a broad and ever-evolving topic. Keeping up with the industry can be challenging for individuals, small businesses and large corporations alike. Luckily, more and more software solutions are emerging that make testing your site’s performance easier than ever before. This article covers what performance testing is, how it works, as well as some of its most common uses.

## What is web performance testing?

Web performance testing is a process that software teams use to measure response times and error rates of their services under certain loads. Experimentation through performance tests provides a better understanding of your service's capacity for both backends and frontends. This deep understanding of your limits is crucial to providing a great user experience.

To provide an example, say an e-commerce company would like to test the amount of traffic that can occur on their website prior to a peak load event like Black Friday. They estimate that their peak traffic will include 100 active users per second, which occurs during the first 30 minutes of the day. An acceptable error rate over this duration is 0.01% - often referred to as [four nines](https://en.wikipedia.org/wiki/High_availability).

A performance tester will build a test plan to match these expectations with some buffer added. This plan would explain:

- What is being tested?
- How much traffic during the test? (100 active users per second)
- What is an acceptable error rate? (0.01%)
- How long should the test run? (30 minutes)?
- What type of traffic ramp-up to simulate? (immediate ramp-up)
- What are our objectives for response time?

Test plans are executed with a load test runner like [JMeter](https://jmeter.apache.org/), which will help send requests. Once the test is finished, the team will analyze the results, declare the test a success or failure, and decide the next steps.

This process of capturing web performance expectations, building test plans to simulate, executing the test plans, and analyzing test results is the practice of performance testing.

## Why use web performance testing?

There are many benefits of this practice that make it an essential tool in your arsenal when ensuring a good user experience. Here are a few:

- Receive metrics that are not available from live traffic. With just performance metrics from live traffic, you cannot accurately predict the result of new traffic behaviours. Traffic spikes are a lot more common with today's ability to go viral.
- The web is a dynamic environment, with changes to the user's experience causing unexpected behaviour. A performance test can help you understand the impact of these changes and what to expect in future releases, thus providing an accurate estimate of how well your website is performing over time.
- Develop a deeper understanding of how your systems interoperate and their limits. These learnings are valuable as you build upon the systems and push them to support new functions.
- You can simulate a wide variety of scenarios, including cold starts, certain failure increases, impact to dependencies, load spikes, and other situations that could occur during anomalous periods of time.

The importance of performance testing has overall increased as users become less patient and more dependent on websites and applications. The nature of organic traffic has become spikier due to viral events and engineering teams need to stay ahead of this traffic by experimenting in a safe environment before it breaks things in production.

## What can be tested in web performance tests?

Performance tests can be used to test and measure the performance of a website, mobile application, or backend API. The following are some of the scenarios that can be tested:

- Initial page load time of common site pages, eg. landing and product pages.
- How many users can a backend API handle at acceptable error rates and response times?
- How long does it take for users to walk through a common flow like checking out?
- When does a third-party service start degrading due to rate limit constraints?
- How much has a database query improved due to a newly added index?

Rather than tackling all this at once, we recommend starting from a top-down approach. Start with testing critical flows, then expand your tests to improve overall flow coverage, and lastly isolate APIs and other dependencies.

## How to build a test plan?

To explain your test scenarios, a plan must be drafted. The test plan can be formally documented or just implicitly captured in the load test runner you use. Let's discuss the steps to building this test plan that apply to either methods.

**Establish your performance baselines using metrics from existing traffic.** This would include existing traffic, response times at varying measurements (eg. average, p75, p90), and error rates. Application performance monitoring from a solution like [Datadog](https://www.datadoghq.com/) or [Grafana](https://grafana.com/) is necessary to begin extensive testing of your services.

**Decide on your test environment.** Tests should typically be run in a dedicated staging or test environment to avoid impacting your users in production. The environment chosen may affect the variables decided on later.

**Declare key performance indicators for your service performance.** The specific metrics you measure depend on your business requirements. Here are some questions you should evaluate:

- Which response time percentiles are important to you (p50 vs p99)?
- Does error rate or count matter more?
- How do performance expectations change with the time of day/week?

**Define performance objectives based on the baselines, expected traffic growth, and your key performance indicators previously identified.** This often includes scenarios that account for steady growth and also scenarios capturing anomalous spike traffic.

**Share your plan with stakeholders and get acceptance.** Collaboration in testing is often overlooked but crucial to sharing responsibility and ownership across your teams. Ask for feedback on your plan to engage others and improve ease of maintenance in the future.

Now that you understand how you'd like to test your system, let's learn how to execute those tests!

## How to run performance tests?

With your test plan ready, you'll need to decide on a load test runner to use if you don't have one already. The open-source solutions available are excellent with popular options including [JMeter](https://jmeter.apache.org/), [Gatling](https://gatling.io/), [Locust](https://locust.io/), or [K6](https://k6.io/). JMeter has historically been the most used and recommended.

Use the documentation from your load test runner to translate the test plan into a scenario you can simulate. This is written using a programming language or configuration files depending on the tool that you use. Some of these tools also accept your KPI metric objectives, whereas others will need you to confirm objectives are met on your own.

Here is an example test scenario defined in the JMeter GUI.

![JMeter thread group](/img/ecommerce-example-jmeter-thread-group.png 'JMeter thread group')

![JMeter HTTP request](/img/ecommerce-example-jmeter-http-request.png 'JMeter HTTP request')

![JMeter summary report](/img/ecommerce-example-jmeter-summary-report.png 'JMeter summary report')

Load test runners can be executed manually on your own machine to start. As complexity and volume of simulations increase you should run these tests on servers with more capacity. With dedicated performance test servers, teams can automate the scheduling of executions to align with new releases or on a recurring time schedule.

It is expected that a test runner will watch for any degradation to the environment while testing. To help with this you should already have monitors in-place to detect negative impact.

While running, the load test runner will generate an output that contains metrics for each request executed. Next we'll learn what to do with this output.

## **How to analyze performance test results?**

Once you’ve completed the process of running your performance tests, you need to analyze the results to decide the next steps.

First we will need an interface to view the result metrics. We'll look at three different interfaces here.

**Use interface provided by test runner**. Most test runners are able to generate an HTML file or some other report. The problem with these methods is that they are typically static reports, which are difficult to share with others and add your own context. You can view the [JMeter HTML dashboard](https://jmeter.apache.org/usermanual/generating-dashboard.html) as an example.

**How?** Convert the output CSV to JMeter's HTML dashboard.

```sh "Convert output CSV to HTML dashboard"
jmeter \
  -g ecommerce-demo-output.csv \
  -o ecommerce-demo-output
```

![JMeter HTML dashboard](/img/ecommerce-example-jmeter-html-dashboard.png 'JMeter HTML dashboard')

**Send metrics to an observability platform like Grafana**. This is a common approach, where teams send test result metrics to their existing observability platform. This has an added benefit of living alongside metrics from live traffic to encourage comparisons. Platforms like Datadog have a ton of features for you to use based on these custom metrics you emit like monitors and SLOs.

**How?** [Read here](https://qainsights.com/jmeter-integration-with-influxdb-2-0/) to set up a Grafana, InfluxDB, and JMeter integration.

![Grafana Metrics](/img/ecommerce-example-grafana.png 'Grafana Metrics')

**Send metrics to a dedicated testing analytics platform like [Latency Lingo](https://latencylingo.com)**. Even after emitting metrics to an observability platform, there are still some gaps in collaboration and automation that is useful for software testing. Dashboards and monitors need to be manually crafted with testing domain expertise. Analysis that requires collaboration will occur outside of the platform in some Google Sheet or Confluence document. Latency Lingo solves for this by providing a dedicated space to analyze and collaborate on your performance test results.

**How?** Publish the output CSV to Latency Lingo APIs using the [CLI](https://docs.latencylingo.com/docs/cli).

```sh "Publish output CSV to Latency Lingo APIs"
latency-lingo-cli \
  --file ~/Workspace/perf-test-workspace/jmeter-test/ecommerce-demo-output.csv \
  --label "Ecommerce Demo"
```

Once performance metrics are sent to Latency Lingo, you'll receive dashboards for all your tests, monitors to help catch degradation, and a way to document observations for team members to learn from.

[View the report from this demo](https://latencylingo.com/reports/4fa09797-7df9-4ad9-87b1-1bb484c640e0).

![Latency Lingo Report](/img/ecommerce-example-latency-lingo-pt-1.png 'Latency Lingo Report')
![Latency Lingo Report](/img/ecommerce-example-latency-lingo-pt-2.png 'Latency Lingo Report')

With an interface available, you can now compare the results with your test plan's objectives. These are some questions you may ask yourself:

- Have your objectives been met for response times, error rate, and request count?
- Do you have an acceptable buffer for anomalies that occur in live traffic?
- Do you see new errors that need to be addressed?
- Are there other key metrics that you found impactful but are not included in your KPIs?
- Are there anomalous spikes that are unexpected but hidden when viewing overall metrics?
- Does your understanding of system performance and limits match the behaviour witnessed?
- Is there noticeable issues in this environment that are not reflective of production?

As you analyze the metrics and answer these questions, you may notice some action items that need to be reported. Add these action items to your test report for future prioritization.

## Closing thoughts

Once analysis is complete, share your learnings with the team to see if others have the same observations. Software testing is a collaborative process that your team should learn from together.

Performance testing can be tedious and bog down velocity for your team if you allow it. Always think how to automate steps in the processes you develop and reduce the manual burden required. Automation is the key to scale and repeat these processes.

## **Where to learn more?**

Still have questions or have a request for additional content?

- [Reach out on Twitter](https://twitter.com/anthony_bobsin)
- [Send me an email](mailto:anthony.bobsin.dev@gmail.com)
- [Follow us on LinkedIn](https://www.linkedin.com/company/latencylingo/)
- [Subscribe for updates](/mailing-list)

[Latency Lingo](https://latencylingo.com) is a tool to help you analyze JMeter results, so I'd encourage you to try it out. I would really appreciate any feedback you have!

I've attached links below to documentation of the tools and processes discussed. Thanks for reading!

- [JMeter](https://jmeter.apache.org/)
- [Gatling](https://gatling.io/)
- [K6](https://k6.io/)
- [Locust](https://locust.io/)
- [Grafana](https://grafana.com/)
- [Datadog](https://www.datadoghq.com/)


Performance tests are critical to understanding your service health. The data generated from performance tests is extremely valuable, but they can be difficult to interpret. Let's discuss some common barriers to adopting performance tests and how to address them.

5 problems when adopting performance tests

### Introduction

Whether you use JMeter, k6, Locust, or Gatling for performance tests, it is easy to be overwhelmed when interpreting test results. Poor leverage of your performance test data can hinder adoption and eventually lead to a worsened user experience. In this article, I'll share some common problems when adopting performance tests and how to solve them.

### Wide performance test surface area

Performance tests can quickly absorb a lot of your team's time. To avoid this you must approach the design of your test setup as you would with a typical software application designed to scale. Modularity and optimizing for ease of maintenance can go a long way in ensuring your performance test suite grows successfully with your codebase.

Aim to define a performance test for each operation your users depend on. These operations are typically APIs, but these practices can be extended to background tasks. It can be tempting to define tests that cover multiple operations required for a complicated flow, but this will become difficult to maintain and action upon. If your APIs require a complex setup process, be deliberate about separating the setup from the actual test that is being executed.

### Interpreting results in isolation without searching for trends

Teams can fall in a bad habit of optimizing for "green" performance tests rather than taking time to interpret how the performance may impact your business. Once you have attained the result data, you should compare with previous executions to identify improvements and degradations.

During this analysis step, you have the opportunity to capture action items for improvements, escalate negative trends to other teams, tune your service objectives, and document context that may be useful to future test runners. All of these are critical to using your performance tests as insights that improve your user experience.

### Focusing on averages and ignoring percentiles

Understanding percentiles is important when interpreting performance test results. The percentiles p50, p75, p90, p95, and p99 are commonly used for application performance monitoring. The 75th percentile value would be the latency of which 75% of the distribution is below.

Avoid making decisions based solely on the average or p50 latency values, since you will be ignoring the experience for a large portion of your users. Define multiple objectives at varying percentiles to ensure an overall great user experience is provided. You can start with three objectives at p50, p75, p95 and adjust from there.

### Losing track of action items or not quantifying their impact

Time invested into finding action items to improve performance can be wasted if teams aren't held accountable to tackle them and measure their impact. Action items should be prioritized through normal engineering cycles and attached to the initial reports that spawned the task. When an action item is completed, run another suite of performance tests to quantify and document the impact.

This cycle of investigating performance, declaring action items, completing action items, and measuring impact is a powerful process to improve your service latencies.

### Falling out of habit

Teams constantly need to balance priorities of product features and performance improvements. The process for improving and testing performance should be easy to allow engineering teams to budget for it in road maps.

Take strides to automate as much of the performance testing process as possible. Build a habit of generating recurring reports to collect metrics and investigating any operations at risk of degrading. All performance investigations should be documented and act as a valuable knowledge base for your team. Be able to quantify the value added by investing in performance tests and investigations.

### Summary

When done right, performance tests and investigations drive a deeper understanding of your application, an improved user experience, and better collaboration across your teams. Follow these recommendations to ensure performance tests scale and grow alongside your codebase.

Thanks for reading! The team at Latency Lingo is hard at work to build a solution to help you solve these problems. [Subscribe here](/mailing-list) to receive updates on our products.


Blog

release2023-01-10

Announcing The New
Github Actions Integration

release2022-12-20

Added Support for Gatling

release2022-12-14

Introducing Performance Test Thresholds

tips2022-12-06

How to use Black Friday Website Performance to Improve Site Reliability

guide2022-09-04

What You Need to Know About Performance Testing Metrics, Reporting, and Analytics

guide2022-08-29

How to automate performance testing with JMeter and Github Actions

guide2022-07-20

Web Performance Testing - What It Is, How It Works, And How You Can Use It

guide2022-03-09

5 problems when adopting performance tests