Add --localrun option to get rid of errors reported when running on l…

…ocal machine

README.md

@@ -41,7 +41,7 @@ python install.py
 
 ### Notes
 - Setup steps require connectivity, make sure to enable a proxy if needed.
-- See the [CI scripts](scripts/) and their orchestration in [config.yml](.circleci/config.yml) 
+- See the [CI scripts](scripts/) and their orchestration in [config.yml](.circleci/config.yml)
 for hints about how to replicate the CI environment if you have issues
 - PyTorch versions before 1.6 are not compatible with all the models in torchbenchmark. See branch [wconstab/compare_torch_versions](https://github.com/pytorch/benchmark/tree/wconstab/compare_torch_versions) for a set of models that worked back to torch 1.4.0.
 - torch, torchvision, torchtext must all be installed from the same build process. This means it isn't possible to mix conda torchtext
@@ -66,7 +66,7 @@ There are currently two top-level scripts for running the models.
 
 `test.py` offers the simplest wrapper around the infrastructure for iterating through each model and installing and executing it.
 
-test_bench.py is a pytest-benchmark script that leverages the same infrastructure but collects benchmark statistics and supports filtering ala pytest. 
+test_bench.py is a pytest-benchmark script that leverages the same infrastructure but collects benchmark statistics and supports filtering ala pytest.
 
 In each model repo, the assumption is that the user would already have all of the torch family of packages installed (torch, torchtext, torchvision, ...) but it installs the rest of the dependencies for the model.
 
@@ -85,12 +85,13 @@ The test name follows the following pattern:
 ```
 
 ### Using pytest-benchmark driver
-`pytest test_bench.py` invokes the benchmark driver. See `--help` for a complete list of options. 
+`pytest test_bench.py` invokes the benchmark driver. See `--help` for a complete list of options.
 
 Some useful options include
 - `--benchmark-autosave` (or other save related flags) to get .json output
 - `-k <filter expression>` (standard pytest filtering)
 - `--collect-only` only show what tests would run, useful to see what models there are or debug your filter expression
+- `--localrun` if running on a CPU machine
 
 ## Nightly CI runs
 Currently, models run on nightly pytorch builds and push data to scuba.

conftest.py

@@ -9,11 +9,13 @@ def pytest_addoption(parser):
  parser.addoption("--ignore_machine_config",
  action='store_true',
  help="Disable checks/assertions for machine configuration for stable benchmarks")
- parser.addoption("--disable_nograd", action='store_true', 
+ parser.addoption("--disable_nograd", action='store_true',
  help="Disable no_grad for eval() runs")
  parser.addoption("--check_opt_vs_noopt_jit",
  action='store_true',
  help="The best attempt to check results for inference runs. Not all models support this!")
+ parser.addoption("--localrun", action='store_true',
+ help="Run benchmarks locally on a macbook")
 
 def set_fuser(fuser):
  if fuser == "old":

test_bench.py

@@ -23,11 +23,14 @@
 def pytest_generate_tests(metafunc):
  # This is where the list of models to test can be configured
  # e.g. by using info in metafunc.config
+ devices = ['cpu', 'cuda']
+ if metafunc.config.option.localrun:
+ devices = ['cpu']
  all_models = list_models()
  if metafunc.cls and metafunc.cls.__name__ == "TestBenchNetwork":
  metafunc.parametrize('model_class', all_models,
  ids=[m.name for m in all_models], scope="class")
- metafunc.parametrize('device', ['cpu', 'cuda'], scope='class')
+ metafunc.parametrize('device', devices, scope='class')
  metafunc.parametrize('compiler', ['jit', 'eager'], scope='class')
 
 @pytest.fixture(scope='class')