Clean up run_bench script #197
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
remove upload steps since timing measurement and score upload are happening on pytorch CI now. shorten the runtime of pytest test_bench.py to make this CI faster, and focused on correctness rather than timing.
xuzhao9
approved these changes
Jan 27, 2021
scripts/run_bench_and_upload.sh
Outdated
| # configure benchmark invocation to be as fast as possible since this run isn't used for actual timing results, | ||
| # but it's still useful to see (imprecise) timing output for debug/diagnosis of adding a new model | ||
| # 'real' timing and score computation are handled by pytorch CI github actions runner with performance-tuned machine | ||
| # use --benchmark-disable for now to make things fast; need to modify defaults in test_bench.py before possible to override runtime cmd line: |
There was a problem hiding this comment.
What does "benchmark-disable" do here? Does it run all models for just 1 iteration?
There was a problem hiding this comment.
i didn't end up using this, but what it does is just runs all models once and not time them.
Since we wanted timing, and we also want a json file output to test compute_score.py, I switched to min_rounds=1, max_time= and warmup-disable. It makes the suite run quite a bit faster but still not as fast as the python test.py case, so there is some overhead.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
remove upload steps since timing measurement and score upload are happening on pytorch CI now.
shorten the runtime of pytest test_bench.py to make this CI faster, and focused on correctness rather than timing.