Skip to content
/ tango Public

Rust microbenchmarking harness based on paired-testing methodology

License

Notifications You must be signed in to change notification settings

bazhenov/tango

Repository files navigation

Tango.rs

Tango Bench Tango Bench

It used to be that benchmarking required a significant amount of time and numerous iterations to arrive at meaningful results, which was particularly arduous when trying to detect subtle changes, such as those within the range of a few percentage points.

Introducing Tango.rs, a novel benchmarking framework that employs paired benchmarking to assess code performance. This approach capitalizes on the fact that it's far more efficient to measure the performance difference between two simultaneously executing functions compared to two functions executed consecutively.

Features:

  • very high sensitivity to changes which allows to converge on results quicker than traditional (pointwise) approach. Often the fraction of a second is enough;
  • ability to compare different versions of the same code from different VCS commits (A/B-benchmarking);
  • macOS, Linux and Windows support;

1 second, 1 percent, 1 error

Compared to traditional pointwise benchmarking, paired benchmarking is significantly more sensitive to changes. This heightened sensitivity enables the early detection of statistically significant performance variations.

Tango is designed to have the capability to detect a 1% change in performance within just 1 second in at least 9 out of 10 test runs.

Prerequirements

  1. Rust and Cargo toolchain installed (Rust stable is supported on Linux/macOS, nightly is required for Windows)
  2. (Optional) cargo-export installed

Getting started

  1. Add cargo dependency and create new benchmark:

    [dev-dependencies]
    tango-bench = "0.5"
    
    [[bench<