test: Run tests faster with `nextest` #778

Hywan · 2022-06-20T09:28:19Z

cargo-nextest is a next-generation
test runner for Rust projects.

This patch installs and uses nextest to run our own tests.

Comparing cargo test and cargo nextest with hyperfine provides the
following results:

$ hyperfine 'cargo test --workspace' 'cargo nextest run --workspace && cargo test --doc'
Benchmark 1: cargo test --workspace
  Time (mean ± σ):     51.785 s ±  2.066 s    [User: 183.471 s, System: 10.563 s]
  Range (min … max):   49.151 s … 56.641 s    10 runs

Benchmark 2: cargo nextest run --workspace && cargo test --doc
  Time (mean ± σ):     44.556 s ±  0.894 s    [User: 192.213 s, System: 11.441 s]
  Range (min … max):   43.170 s … 45.762 s    10 runs

Benchmark 2 is 1.16 times faster than Benchmark 1

Real numbers are described in #778 (comment).

jplatte · 2022-06-20T09:34:22Z

nextest doesn't support doctests. We will have to run cargo test --doc as well (and add it to nextest's time in the comparison).

codecov · 2022-06-20T09:41:00Z

Codecov Report

Merging #778 (fdca461) into main (87639a4) will increase coverage by 0.02%.
The diff coverage is n/a.

❗ Current head fdca461 differs from pull request most recent head b0d51fd. Consider uploading reports for the commit b0d51fd to get more accurate results

@@            Coverage Diff             @@
##             main     #778      +/-   ##
==========================================
+ Coverage   77.37%   77.39%   +0.02%     
==========================================
  Files          94       94              
  Lines       13735    13730       -5     
==========================================
- Hits        10627    10626       -1     
+ Misses       3108     3104       -4

Impacted Files	Coverage Δ
crates/matrix-sdk-crypto/src/store/mod.rs	`53.33% <0.00%> (-1.63%)`	⬇️
crates/matrix-sdk-base/src/store/ambiguity_map.rs	`82.35% <0.00%> (ø)`
...tes/matrix-sdk-base/src/store/integration_tests.rs	`97.46% <0.00%> (ø)`
crates/matrix-sdk-base/src/store/mod.rs	`81.70% <0.00%> (+1.70%)`	⬆️
crates/matrix-sdk/src/client/builder.rs	`71.11% <0.00%> (+4.44%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87639a4...b0d51fd. Read the comment docs.

jplatte

Are the figures from the first commit description up-to-date now?

xtask/src/ci.rs

Hywan · 2022-06-20T18:02:37Z

We may be able to run cargo nextest in parallel of cargo test --doc with GNU parallel, or https://gitlab.redox-os.org/redox-os/parallel for having something cross-platform, but we should have prebuilt binaries, or cache, otherwise the compilation-time with cost us too much. Let's see with the current numbers.

Edit: xargs -P can also be useful in this case.

Hywan · 2022-06-20T18:29:34Z

With xargs -P, we can achieve the following results:

Benchmark 1: cargo test --workspace
  Time (mean ± σ):     51.785 s ±  2.066 s    [User: 183.471 s, System: 10.563 s]
  Range (min … max):   49.151 s … 56.641 s    10 runs

Benchmark 2: cargo nextest run --workspace && cargo test --doc
  Time (mean ± σ):     44.556 s ±  0.894 s    [User: 192.213 s, System: 11.441 s]
  Range (min … max):   43.170 s … 45.762 s    10 runs

Benchmark 3: echo "cargo nextest run --workspace\0cargo test --doc" | xargs -0 -P 0 -I@ sh -c @
  Time (mean ± σ):     34.339 s ±  4.357 s    [User: 192.472 s, System: 11.739 s]
  Range (min … max):   22.009 s … 36.422 s    10 runs

Benchmark 2 is 1.16 times faster than Benchmark 1
Benchmark 3 is 1.51 times faster than Benchmark 1

I'm not sure how it is possible to run such xargs command in a YAML file while having cross-compatibility, but let's try!

Hywan · 2022-06-20T18:58:19Z

Without xargs -P, here are the results for the “test” step of the CI workflow for some jobs (I won't do it exhaustively, time differs, it's just to provide an order of magnitude idea):

jobs	before	after	speed up factor
linux / stable	11m 21s	9m 13s	1.23
macOS / stable	15m 25s	10m 40s	1.45
linux / features-sso-login	9m 30s	5m 9s	1.84
linux / features-rustls-tls	5m 4s	2m 13s	2.29
linux / features-no-encryption	6m 16s	2m 14s	2.81
linux / features-no-sled	6m 51s	4m 9s	1.65
linux / features-no-encryption-and-sled	5m 10s	1m 51s	2.79

Just on those examples, running tests is 2.01 times faster on average.

Hywan · 2022-06-20T19:27:05Z

When running cargo nextest run and cargo test --doc in parallel, we have worst results on Github Actions, probably because the number of threads is limited. In that case, they are spending more time waiting on each other than truly running what we want. Let's keep things simple and let's use cargo-nextest for now.

jplatte

Removal of dead code is still not a separate commit afaict but otherwise looks good.

> [`cargo-nextest`](https://nexte.st/index.html) is a next-generation > test runner for Rust projects. This patch installs and uses `nextest` to run our own tests. Comparing `cargo test` and `cargo nextest` with hyperfine provides the following results: ```sh $ hyperfine 'cargo test --workspace' 'cargo nextest run --workspace && cargo test --doc' Benchmark 1: cargo test --workspace Time (mean ± σ): 51.785 s ± 2.066 s [User: 183.471 s, System: 10.563 s] Range (min … max): 49.151 s … 56.641 s 10 runs Benchmark 2: cargo nextest run --workspace && cargo test --doc Time (mean ± σ): 44.556 s ± 0.894 s [User: 192.213 s, System: 11.441 s] Range (min … max): 43.170 s … 45.762 s 10 runs ``` Benchmark 2 is 1.16 times faster than Benchmark 1.

`cargo-nextest` doesn't support doctests for now, so we must run them “manually” by running a separate `cargo test --doc` command.

This patch also changes the step's name from Clippy to Test.

Hywan · 2022-06-22T07:28:36Z

Removal of dead code is still not a separate commit afaict but otherwise looks good.

Done, d9475c1 :-).

Hywan added the enhancement New feature or request label Jun 20, 2022

Hywan force-pushed the test-nextest branch from 14c51fb to 749ad9f Compare June 20, 2022 12:06

jplatte reviewed Jun 20, 2022

View reviewed changes

xtask/src/ci.rs Show resolved Hide resolved

jplatte mentioned this pull request Jun 20, 2022

Slow to (re)compile #106

Closed

Hywan force-pushed the test-nextest branch from 749ad9f to fe5a486 Compare June 20, 2022 18:28

Hywan force-pushed the test-nextest branch from fdca461 to fe5a486 Compare June 20, 2022 19:26

jplatte approved these changes Jun 21, 2022

View reviewed changes

Hywan added 5 commits June 22, 2022 09:26

test(xtask): Remove xtask -- ci test as it is unused.

d9475c1

test: Run doctests manually.

eb33333

`cargo-nextest` doesn't support doctests for now, so we must run them “manually” by running a separate `cargo test --doc` command.

test: Add missing cargo-nextest installation.

3bfc68d

This patch also changes the step's name from Clippy to Test.

test: There is no doctest for matrix-sdk-crypto-ffi.

b0d51fd

Hywan force-pushed the test-nextest branch from fe5a486 to b0d51fd Compare June 22, 2022 07:27

Hywan enabled auto-merge June 22, 2022 07:28

Hywan merged commit 6ad323b into matrix-org:main Jun 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: Run tests faster with `nextest` #778

test: Run tests faster with `nextest` #778

Hywan commented Jun 20, 2022 •

edited

Loading

jplatte commented Jun 20, 2022

codecov bot commented Jun 20, 2022 •

edited

Loading

jplatte left a comment

Hywan commented Jun 20, 2022 •

edited

Loading

Hywan commented Jun 20, 2022

Hywan commented Jun 20, 2022

Hywan commented Jun 20, 2022

jplatte left a comment

Hywan commented Jun 22, 2022

test: Run tests faster with nextest #778

test: Run tests faster with nextest #778

Conversation

Hywan commented Jun 20, 2022 • edited Loading

jplatte commented Jun 20, 2022

codecov bot commented Jun 20, 2022 • edited Loading

Codecov Report

jplatte left a comment

Choose a reason for hiding this comment

Hywan commented Jun 20, 2022 • edited Loading

Hywan commented Jun 20, 2022

Hywan commented Jun 20, 2022

Hywan commented Jun 20, 2022

jplatte left a comment

Choose a reason for hiding this comment

Hywan commented Jun 22, 2022

test: Run tests faster with `nextest` #778

test: Run tests faster with `nextest` #778

Hywan commented Jun 20, 2022 •

edited

Loading

codecov bot commented Jun 20, 2022 •

edited

Loading

Hywan commented Jun 20, 2022 •

edited

Loading