Run the Vulkan Conformance Test Suite in parallel and robustly.
This is fork/rewrite of the mesa parallel deqp runner.
- Run tests in parallel
- Recover from crashes (e.g. failing asserts)
- Recover from timeouts
- Save results even on fatal errors
- Cross-platform
- Automatically retry failing tests
- Automatically bisect failures if they depend on a combination of tests
- Save a junit compatible xml file of the results
Create a testlist.txt
with one deqp test name per line. Run with
deqp-runner -t testlist.txt -- ./deqp-vk --deqp-caselist-file
A more complicated command:
deqp-runner -t testlist.txt \
--timeout 10 \
--jobs 2 \
--shuffle \
--no-progress \
--start 50 \
-- ./deqp-vk \
--deqp-log-flush=disable \
--deqp-log-images=disable \
--deqp-log-shader-sources=disable \
--deqp-surface-width=256 \
--deqp-surface-height=256 \
--deqp-surface-type=pbuffer \
--deqp-gl-config-name=rgba8888d24s8ms0 \
--deqp-caselist-file
deqp-runner
produces a number of output files to collect results and help finding driver issues.
summary.csv
contains the result for every test.summary.xml
contains failures and flakes in the junit xml format for consumption by CI systems like Jenkins.log.json
is written while running. It contains a line for every run test, connecting it with the PID ofdeqp-vk
, and timestamp and a failure directory containing more information. Tests that are run multiple times because they are flaky are listed multiple times.fails
contains one directory for everydeqp-vk
invocation that had failures. Every failure subdir contains the list of tests that were run for easy reproducibility and stderr ofdeqp-vk
.log.json
references failure subdirs for failed tests.
Install Rust (preferred installation method is rustup)
and build with cargo build --release
.
To install without cloning use cargo install --git https://github.com/Flakebi/deqp-runner
.
Licensed under either of
at your option.