Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing framework should check for performance regression #673

Open
rcabell opened this issue Jan 18, 2023 · 0 comments
Open

Testing framework should check for performance regression #673

rcabell opened this issue Jan 18, 2023 · 0 comments

Comments

@rcabell
Copy link
Collaborator

rcabell commented Jan 18, 2023

The testing framework should compare runtime for each test from candidate to reference and flag significant* differences as a regression.

*"significant" is in the eye of the beholder

Expected Behavior

If a candidate test runs more than xx% slower than the reference test, it should be labeled as a regression

Current Behavior

No differences in test runtimes are noted or flagged

Possible Solution

For each candidate test run by the testing framework, a runtime should be noted and compared against the runtime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant