This directory will contain tools and utilities that help us to measure the performance overhead introduced by the agent and to measure how this overhead changes over time.
The overhead tests here should be considered a "macro" benchmark. They serve to measure high-level overhead as perceived by the operator of a "typical" application. Tests are performed on a Java 11 distribution from Eclipse Temurin.
There is one dynamic test here called OverheadTests.
The @TestFactory method creates a test pass for each of the defined configurations.
Before the tests run, a single collector instance is started. Each test pass has one or more agents configured and those are tested in series.
For each agent defined in a configuration, the test runner (using testcontainers) will:
- create a fresh postgres instance and populate it with initial data.
- create a fresh instance of spring-petclinic-rest instrumented with the specified agent
- measure the time until the petclinic app is marked "healthy" and then write it to a file.
- if configured, perform a warmup phase. During the warmup phase, a bit of traffic is generated in order to get the application into a steady state (primarily helping facilitate jit compilations). Currently, we use a 30 second warmup time.
- start a JFR recording by running
jcmdinside the petclinic container - run the k6 test script with the configured number of iterations through the file and the configured number of concurrent virtual users (VUs).
- after k6 completes, petclinic is shut down
- after petclinic is shut down, postgres is shut down
And this repeats for every agent configured in each test configuration.
After all the tests are complete, the results are collected and committed back to the /results subdirectory as csv and summary text files.
For each test pass, we record the following metrics in order to compare agents and determine relative overhead.
| metric name | units | description |
|---|---|---|
| Startup time | ms | How long it takes for the spring app to report "healthy" |
| Total allocated mem | bytes | Across the life of the application |
| Heap (min) | bytes | Smallest observed heap size |
| Heap (max) | bytes | Largest observed heap size |
| Thread switch rate | # / s | Max observed thread context switch rate |
| GC time | ms | Total amount of time spent paused for garbage collection |
| Request mean | ms | Average time to handle a single web request (measured at the caller) |
| Request p95 | ms | 95th percentile time to handle a single web requ4st (measured at the caller) |
| Iteration mean | ms | average time to do a single pass through the k6 test script |
| Iteration p95 | ms | 95th percentile time to do a single pass through the k6 test script |
| Peak threads | # | Highest number of running threads in the VM, including agent threads |
| Network read mean | bits/s | Average network read rate |
| Network write mean | bits/s | Average network write rate |
| Average JVM user CPU | % | Average observed user CPU (range 0.0-1.0) |
| Max JVM user CPU | % | Max observed user CPU used (range 0.0-1.0) |
| Average machine tot. CPU | % | Average percentage of machine CPU used (range 0.0-1.0) |
| Total GC pause nanos | ns | JVM time spent paused due to GC |
| Run duration ms | ms | Duration of the test run, in ms |
Each config contains the following:
- name
- description
- list of agents (see below)
- maxRequestRate (optional, used to throttle traffic)
- concurrentConnections (number of concurrent virtual users [VUs])
- totalIterations - the number of passes to make through the k6 test script
- warmupSeconds - how long to wait before starting conducting measurements
Currently, we test:
- no agent versus latest released agent
- no agent versus latest snapshot
- latest release vs. latest snapshot
Additional configurations can be created by submitting a PR against the Configs class.
An agent is defined in code as a name, description, optional URL, and optional additional
arguments to be passed to the JVM (not including -javaagent:). New agents may be defined
by creating new instances of the Agent class. The AgentResolver is used to download
the relevant agent jar for an Agent definition.
The tests are run nightly via github actions. The results are collected and appended to
a csv file, which is committed back to the repo in the /results subdirectory.
The tests require docker to be running. Simply run OverheadTests in your IDE.
Alternatively, you can run the tests from the command line with gradle:
cd benchmark-overhead
./gradlew test
None yet. Help wanted! Our goal is to have the results and a rich UI running in the
gh-pages branch similar to earlier tools.
Please help us make this happen.