Add machine_util.py for tuning optimal benchmark settings #183
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Applies some configuration settings automatically, but verifies more. Future work - utilize the query part from inside benchmark to embed queried values in json results - additional optimizations like nohz, other nvidia settings, disabling IRQ - add temp/clock query for gpu; try querying nvidia-smi performance mode
xuzhao9
approved these changes
Dec 29, 2020
scripts/machine_util.py
Outdated
| isolcpus.add(int(chunk)) | ||
| return list(isolcpus) | ||
|
|
||
| def get_nvidia_graphics_clock(device_id=0): |
There was a problem hiding this comment.
Does it make sense to get all available graphics clock by default?
There was a problem hiding this comment.
yep, i have already made that change actually. I am adding more stuff still. I'll update the PR in a little bit.
- debugging issue where GOMP_CPU_AFFINITY and taskset are properly specified with a range e.g. 4-47 on the command line for pytest, but the machine util sees process affinity as just 4; this does not happen when using the same command line with python instead of pytest and invoking the machine util - need to verify the extra json file size incurred by logging machine state every benchmark
wconstab
force-pushed
the
wconstab/add_machine_util
branch
from
cedeedb
to
3c2c128
Compare
- be more explicit about checking the machine os config and bail gracefully outside amazon linux - restore limited isolated cpu checks modulo pytorch issue #49971 - implement nvidia clock setting
- add 'ignore_machine_config' option to enable running without checks - bail out before running tests rather than after - log machine info in the json
since this CI runner is not one of the performance tuned machines, just used for correctness.
xuzhao9
reviewed
Jan 26, 2021
| f.write(content) | ||
|
|
||
| def check_intel_turbo_state(turbo_file='/sys/devices/system/cpu/intel_pstate/no_turbo'): | ||
| return int(read_sys_file(turbo_file)) |
There was a problem hiding this comment.
I think it should be 1 - int(read_sys_file(turbo_file)) because if "no_turbo" is 1 means turbo is disabled, this function should return 0.
There was a problem hiding this comment.
oh yea good point.
torchbenchmark isn't installed in a proper way, so it doesn't work to use relative imports from a script with a main function that is nested inside the torchbenchmark package. work around this for now. make machine_config script agnostic to cpu core pinning when run as a script, since pinning is checked by conftest during benchmarking and it's pointless to make the user pin the configuration script.
xuzhao9
added a commit
to xuzhao9/benchmark
that referenced
this pull request
Jan 28, 2021
Applies some configuration settings automatically, but verifies more. Only works on amazon linux so far; tries to bail gracefully on other platforms. Usage: - as a command line script, machine_config.py will check or configure the machine sudo `which python` <path to machine_config.py> --configure - as a library, provides functions for use e.g. in conftest.py - asserts benchmark script is run with configured settings - logs machine settings to benchmark data file Other miscellaneous fixes: * Add score plot, nightly sweep scripts * Add legend to sweep result plotting script. (pytorch#193) Moves compute_score and some other utils around. Needs more work. torchbenchmark isn't installed in a proper way, so it doesn't work to use relative imports from a script with a main function that is nested inside the torchbenchmark package. work around this for now. make machine_config script agnostic to cpu core pinning when run as a script, since pinning is checked by conftest during benchmarking and it's pointless to make the user pin the configuration script. Co-authored-by: xz <[email protected]>
xuzhao9
added a commit
that referenced
this pull request
Jan 29, 2021
* Add machine_util.py for tuning optimal benchmark settings Cherry-picked from master branch PR #183 Applies some configuration settings automatically, but verifies more. Only works on amazon linux so far; tries to bail gracefully on other platforms. Usage: - as a command line script, machine_config.py will check or configure the machine sudo `which python` <path to machine_config.py> --configure - as a library, provides functions for use e.g. in conftest.py - asserts benchmark script is run with configured settings - logs machine settings to benchmark data file Other miscellaneous fixes: * Add score plot, nightly sweep scripts * Add legend to sweep result plotting script. (#193) Moves compute_score and some other utils around. Needs more work. torchbenchmark isn't installed in a proper way, so it doesn't work to use relative imports from a script with a main function that is nested inside the torchbenchmark package. work around this for now. make machine_config script agnostic to cpu core pinning when run as a script, since pinning is checked by conftest during benchmarking and it's pointless to make the user pin the configuration script. Co-authored-by: xz <[email protected]> * Fix CI script paths under the new directory hierarchy. Co-authored-by: Will Constable <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Applies some configuration settings automatically, but verifies more.
Future work