Set seed per sample for OpInfo tests + support for restricting to a single sample input #128238

jbschlosser · 2024-06-07T19:17:44Z

Stack from ghstack (oldest at bottom):

-> Set seed per sample for OpInfo tests + support for restricting to a single sample input #128238

This PR:

Sets a random seed before generating each sample for an OpInfo test. It does this by intercepting the sample input iterator via TrackedInputIter, optionally setting the seed to a test name specific seed before each iterator call (default is to set the seed).
- Some quick and dirty benchmarking shows (hopefully) negligible overhead from setting the random seed before each sample input generation. For a trivial (single assert) test that uses @ops:
Uncovered a bunch of test issues:
- Test breakdown (>100 total)
  - A lot of tolerance issues (tweaked tolerance values to fix)
  - 1 broken OpInfo (sample_inputs_masked_fill was generating a sample of the wrong dtype)
  - 3 actually broken semantics (for masked tensor; added xfails)
  - 4 Jacobian mismatches (added xfails)
  - 2 nan results (skip for now, need fixing)
  - 3 results too far from reference result (add xfails)
Skips MPS tests for now (there are so many failures!). Those will default to the old behavior.

before (no seed setting):

real	0m21.306s
user	0m19.053s
sys	0m5.192s

after (with seed setting):

real	0m21.905s
user	0m19.578s
sys	0m5.390s

Utilizing the above for reproducible sample input generation, adds support for restricting the iterator to a single sample input. This is done via an env var PYTORCH_OPINFO_SAMPLE_INPUT_INDEX and its usage is included in the repro command.

======================================================================
ERROR: test_bar_add_cuda_uint8 (__main__.TestFooCUDA.test_bar_add_cuda_uint8)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 971, in test_wrapper
    return test(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jbschlosser/branches/testing_updates/test/test_ops.py", line 2671, in test_bar
    self.assertFalse(True)
AssertionError: True is not false

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper
    method(*args, **kwargs)
  File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper
    method(*args, **kwargs)
  File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 419, in instantiated_test
    result = test(self, **param_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 1426, in wrapper
    fn(*args, **kwargs)
  File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 982, in test_wrapper
    raise new_e from e
Exception: Caused by sample input at index 3: SampleInput(input=Tensor[size=(10, 5), device="cuda:0", dtype=torch.uint8], args=TensorList[Tensor[size=(), device="cuda:0", dtype=torch.uint8]], kwargs={}, broadcasts_input=False, name='')

To execute this test, run the following from the base repo dir:
    PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=3 python test/test_ops.py -k TestFooCUDA.test_bar_add_cuda_uint8

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

----------------------------------------------------------------------
Ran 1 test in 0.037s

FAILED (errors=1)

…ingle sample input [ghstack-poisoned]

pytorch-bot · 2024-06-07T19:17:47Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/128238

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit d49a7f0 with merge base e7ab7b8 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

trunk / win-vs2019-cpu-py3 / test (default, 2, 3, windows.4xlarge.nonephemeral) (gh) (similar failure)
test_decomp 11/12 failed!

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…ting to a single sample input" This PR: * Sets a random seed before generating each sample for an OpInfo test. It does this by intercepting the sample input iterator via `TrackedInputIter`, optionally setting the seed to a test name specific seed before each iterator call (default is to set the seed). * Some quick and dirty benchmarking shows (hopefully) negligible overhead from setting the random seed before each sample input generation. For a trivial (single assert) test that uses `ops`: **before (no seed setting):** ``` real 0m21.306s user 0m19.053s sys 0m5.192s ``` **after (with seed setting):** ``` real 0m21.905s user 0m19.578s sys 0m5.390s ``` * Utilizing the above for reproducible sample input generation, adds support for restricting the iterator to a single sample input. Usage looks like this (open to bikeshedding of course): ```python for sample in op.sample_inputs(device, dtype, restrict_to_index=3): ... ``` [ghstack-poisoned]

…ting to a single sample input" This PR: * Sets a random seed before generating each sample for an OpInfo test. It does this by intercepting the sample input iterator via `TrackedInputIter`, optionally setting the seed to a test name specific seed before each iterator call (default is to set the seed). * Some quick and dirty benchmarking shows (hopefully) negligible overhead from setting the random seed before each sample input generation. For a trivial (single assert) test that uses `ops`: **before (no seed setting):** ``` real 0m21.306s user 0m19.053s sys 0m5.192s ``` **after (with seed setting):** ``` real 0m21.905s user 0m19.578s sys 0m5.390s ``` * Utilizing the above for reproducible sample input generation, adds support for restricting the iterator to a single sample input. This is done via an env var `PYTORCH_OPINFO_SAMPLE_INPUT_INDEX` and its usage is included in the repro command. ``` ====================================================================== ERROR: test_bar_add_cuda_uint8 (__main__.TestFooCUDA.test_bar_add_cuda_uint8) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 971, in test_wrapper return test(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/test/test_ops.py", line 2671, in test_bar self.assertFalse(True) AssertionError: True is not false The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 419, in instantiated_test result = test(self, **param_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 1426, in wrapper fn(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 982, in test_wrapper raise new_e from e Exception: Caused by sample input at index 3: SampleInput(input=Tensor[size=(10, 5), device="cuda:0", dtype=torch.uint8], args=TensorList[Tensor[size=(), device="cuda:0", dtype=torch.uint8]], kwargs={}, broadcasts_input=False, name='') To execute this test, run the following from the base repo dir: PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=3 python test/test_ops.py -k TestFooCUDA.test_bar_add_cuda_uint8 This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 ---------------------------------------------------------------------- Ran 1 test in 0.037s FAILED (errors=1) ``` [ghstack-poisoned]

…ingle sample input ghstack-source-id: 74e722f8621e8ec9b9efa209183509314933571f Pull Request resolved: #128238

janeyx99

The code looks good to me. Landing hinges on green CI, as you know.

It would also be good to be able to set the seed, but that can be a followup.

torch/testing/_internal/common_utils.py

janeyx99 · 2024-06-07T21:31:34Z

torch/testing/_internal/common_utils.py

+
+ # allow StopIteration to bubble up
+ input_idx, input_val = next(self.child_iter)
+ if (self.restrict_to_index is None) or (input_idx == self.restrict_to_index):


ahhh this conditional took me a bit. I was confused why you'd have the first if self.restrict_to_index is None and it's because you wanna feed it forward! Cuz if it's specified, you want to ignore every other value and only feed the indexed one forward. Smart.

Is there a way in the nonempty restrict to index case to tell the iterator to short circuit to the end once the index has been found? vs needing to keep going next?

Is there a way in the nonempty restrict to index case to tell the iterator to short circuit to the end once the index has been found? vs needing to keep going next?

let's see, we could manually raise StopIteration once we've passed the restricted index. This approach won't exhaust the child iterator, but maybe that's okay?

This doesn't matter too much to me--I'd only be willing to do that if it's tangibly faster

…ting to a single sample input" This PR: * Sets a random seed before generating each sample for an OpInfo test. It does this by intercepting the sample input iterator via `TrackedInputIter`, optionally setting the seed to a test name specific seed before each iterator call (default is to set the seed). * Some quick and dirty benchmarking shows (hopefully) negligible overhead from setting the random seed before each sample input generation. For a trivial (single assert) test that uses `ops`: **before (no seed setting):** ``` real 0m21.306s user 0m19.053s sys 0m5.192s ``` **after (with seed setting):** ``` real 0m21.905s user 0m19.578s sys 0m5.390s ``` * Utilizing the above for reproducible sample input generation, adds support for restricting the iterator to a single sample input. This is done via an env var `PYTORCH_OPINFO_SAMPLE_INPUT_INDEX` and its usage is included in the repro command. ``` ====================================================================== ERROR: test_bar_add_cuda_uint8 (__main__.TestFooCUDA.test_bar_add_cuda_uint8) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 971, in test_wrapper return test(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/test/test_ops.py", line 2671, in test_bar self.assertFalse(True) AssertionError: True is not false The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 419, in instantiated_test result = test(self, **param_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 1426, in wrapper fn(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 982, in test_wrapper raise new_e from e Exception: Caused by sample input at index 3: SampleInput(input=Tensor[size=(10, 5), device="cuda:0", dtype=torch.uint8], args=TensorList[Tensor[size=(), device="cuda:0", dtype=torch.uint8]], kwargs={}, broadcasts_input=False, name='') To execute this test, run the following from the base repo dir: PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=3 python test/test_ops.py -k TestFooCUDA.test_bar_add_cuda_uint8 This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 ---------------------------------------------------------------------- Ran 1 test in 0.037s FAILED (errors=1) ``` [ghstack-poisoned]

…ingle sample input ghstack-source-id: 0170a5241854fcac0c03c47b7b3a6eadf4d3a1b9 Pull Request resolved: #128238

…ting to a single sample input" This PR: * Sets a random seed before generating each sample for an OpInfo test. It does this by intercepting the sample input iterator via `TrackedInputIter`, optionally setting the seed to a test name specific seed before each iterator call (default is to set the seed). * Some quick and dirty benchmarking shows (hopefully) negligible overhead from setting the random seed before each sample input generation. For a trivial (single assert) test that uses `ops`: **before (no seed setting):** ``` real 0m21.306s user 0m19.053s sys 0m5.192s ``` **after (with seed setting):** ``` real 0m21.905s user 0m19.578s sys 0m5.390s ``` * Utilizing the above for reproducible sample input generation, adds support for restricting the iterator to a single sample input. This is done via an env var `PYTORCH_OPINFO_SAMPLE_INPUT_INDEX` and its usage is included in the repro command. ``` ====================================================================== ERROR: test_bar_add_cuda_uint8 (__main__.TestFooCUDA.test_bar_add_cuda_uint8) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 971, in test_wrapper return test(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/test/test_ops.py", line 2671, in test_bar self.assertFalse(True) AssertionError: True is not false The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 419, in instantiated_test result = test(self, **param_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 1426, in wrapper fn(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 982, in test_wrapper raise new_e from e Exception: Caused by sample input at index 3: SampleInput(input=Tensor[size=(10, 5), device="cuda:0", dtype=torch.uint8], args=TensorList[Tensor[size=(), device="cuda:0", dtype=torch.uint8]], kwargs={}, broadcasts_input=False, name='') To execute this test, run the following from the base repo dir: PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=3 python test/test_ops.py -k TestFooCUDA.test_bar_add_cuda_uint8 This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 ---------------------------------------------------------------------- Ran 1 test in 0.037s FAILED (errors=1) ``` [ghstack-poisoned]

…ting to a single sample input" This PR: * Sets a random seed before generating each sample for an OpInfo test. It does this by intercepting the sample input iterator via `TrackedInputIter`, optionally setting the seed to a test name specific seed before each iterator call (default is to set the seed). * Some quick and dirty benchmarking shows (hopefully) negligible overhead from setting the random seed before each sample input generation. For a trivial (single assert) test that uses `ops`: * Uncovered a bunch of test issues: * Test breakdown (31 total) * 21 tolerance issues (tweaked tolerance values to fix) * 1 broken OpInfo (`sample_inputs_masked_fill` was generating a sample of the wrong dtype) * 2 actually broken semantics (for masked tensor; added xfails) * 3 Jacobian mismatches (added xfails) * 2 nan results (skip for now, need fixing) * 2 results too far from reference result (add xfails) **before (no seed setting):** ``` real 0m21.306s user 0m19.053s sys 0m5.192s ``` **after (with seed setting):** ``` real 0m21.905s user 0m19.578s sys 0m5.390s ``` * Utilizing the above for reproducible sample input generation, adds support for restricting the iterator to a single sample input. This is done via an env var `PYTORCH_OPINFO_SAMPLE_INPUT_INDEX` and its usage is included in the repro command. ``` ====================================================================== ERROR: test_bar_add_cuda_uint8 (__main__.TestFooCUDA.test_bar_add_cuda_uint8) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 971, in test_wrapper return test(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/test/test_ops.py", line 2671, in test_bar self.assertFalse(True) AssertionError: True is not false The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 419, in instantiated_test result = test(self, **param_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 1426, in wrapper fn(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 982, in test_wrapper raise new_e from e Exception: Caused by sample input at index 3: SampleInput(input=Tensor[size=(10, 5), device="cuda:0", dtype=torch.uint8], args=TensorList[Tensor[size=(), device="cuda:0", dtype=torch.uint8]], kwargs={}, broadcasts_input=False, name='') To execute this test, run the following from the base repo dir: PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=3 python test/test_ops.py -k TestFooCUDA.test_bar_add_cuda_uint8 This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 ---------------------------------------------------------------------- Ran 1 test in 0.037s FAILED (errors=1) ``` [ghstack-poisoned]

…ting to a single sample input" This PR: * Sets a random seed before generating each sample for an OpInfo test. It does this by intercepting the sample input iterator via `TrackedInputIter`, optionally setting the seed to a test name specific seed before each iterator call (default is to set the seed). * Some quick and dirty benchmarking shows (hopefully) negligible overhead from setting the random seed before each sample input generation. For a trivial (single assert) test that uses `ops`: * Uncovered a bunch of test issues: * Test breakdown (42 total) * 30 tolerance issues (tweaked tolerance values to fix) * 1 broken OpInfo (`sample_inputs_masked_fill` was generating a sample of the wrong dtype) * 2 actually broken semantics (for masked tensor; added xfails) * 4 Jacobian mismatches (added xfails) * 2 nan results (skip for now, need fixing) * 3 results too far from reference result (add xfails) **before (no seed setting):** ``` real 0m21.306s user 0m19.053s sys 0m5.192s ``` **after (with seed setting):** ``` real 0m21.905s user 0m19.578s sys 0m5.390s ``` * Utilizing the above for reproducible sample input generation, adds support for restricting the iterator to a single sample input. This is done via an env var `PYTORCH_OPINFO_SAMPLE_INPUT_INDEX` and its usage is included in the repro command. ``` ====================================================================== ERROR: test_bar_add_cuda_uint8 (__main__.TestFooCUDA.test_bar_add_cuda_uint8) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 971, in test_wrapper return test(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/test/test_ops.py", line 2671, in test_bar self.assertFalse(True) AssertionError: True is not false The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 419, in instantiated_test result = test(self, **param_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 1426, in wrapper fn(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 982, in test_wrapper raise new_e from e Exception: Caused by sample input at index 3: SampleInput(input=Tensor[size=(10, 5), device="cuda:0", dtype=torch.uint8], args=TensorList[Tensor[size=(), device="cuda:0", dtype=torch.uint8]], kwargs={}, broadcasts_input=False, name='') To execute this test, run the following from the base repo dir: PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=3 python test/test_ops.py -k TestFooCUDA.test_bar_add_cuda_uint8 This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 ---------------------------------------------------------------------- Ran 1 test in 0.037s FAILED (errors=1) ``` [ghstack-poisoned]

pytorch-bot · 2024-07-05T15:47:39Z

PyTorchBot Help

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick,close} ...

In order to invoke the bot on your PR, include a line that starts with
@pytorchbot anywhere in a comment. That line will form the command; no
multi-line commands are allowed. Some commands may be used on issues as specified below.

Example:
    Some extra context, blah blah, wow this PR looks awesome

    @pytorchbot merge

optional arguments:
  -h, --help            Show this help message and exit.

command:
  {merge,revert,rebase,label,drci,cherry-pick,close}
    merge               Merge a PR
    revert              Revert a PR
    rebase              Rebase a PR
    label               Add label to a PR
    drci                Update Dr. CI
    cherry-pick         Cherry pick a PR onto a release branch
    close               Close a PR

Merge

usage: @pytorchbot merge [-f MESSAGE | -i] [-ic] [-r [{viable/strict,main}]]

Merge an accepted PR, subject to the rules in .github/merge_rules.json.
By default, this will wait for all required checks (lint, pull) to succeed before merging.

optional arguments:
  -f MESSAGE, --force MESSAGE
                        Merge without checking anything. This requires a reason for auditting purpose, for example:
                        @pytorchbot merge -f 'Minor update to fix lint. Expecting all PR tests to pass'
                        
                        Please use `-f` as last resort, prefer `--ignore-current` to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.
  -i, --ignore-current  Merge while ignoring the currently failing jobs.  Behaves like -f if there are no pending jobs.
  -ic                   Old flag for --ignore-current. Deprecated in favor of -i.
  -r [{viable/strict,main}], --rebase [{viable/strict,main}]
                        Rebase the PR to re run checks before merging.  Accepts viable/strict or main as branch options and will default to viable/strict if not specified.

Revert

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst}

Revert a merged PR. This requires that you are a Meta employee.

Example:
  @pytorchbot revert -m="This is breaking tests on trunk. hud.pytorch.org/" -c=nosignal

optional arguments:
  -m MESSAGE, --message MESSAGE
                        The reason you are reverting, will be put in the commit message. Must be longer than 3 words.
  -c {nosignal,ignoredsignal,landrace,weird,ghfirst}, --classification {nosignal,ignoredsignal,landrace,weird,ghfirst}
                        A machine-friendly classification of the revert reason.

Rebase

usage: @pytorchbot rebase [-s | -b BRANCH]

Rebase a PR. Rebasing defaults to the stable viable/strict branch of pytorch.
Repeat contributor may use this command to rebase their PR.

optional arguments:
  -s, --stable          [DEPRECATED] Rebase onto viable/strict
  -b BRANCH, --branch BRANCH
                        Branch you would like to rebase to

Label

usage: @pytorchbot label labels [labels ...]

Adds label to a PR or Issue [Can be used on Issues]

positional arguments:
  labels  Labels to add to given Pull Request or Issue [Can be used on Issues]

Dr CI

usage: @pytorchbot drci 

Update Dr. CI. Updates the Dr. CI comment on the PR in case it's gotten out of sync with actual CI results.

cherry-pick

usage: @pytorchbot cherry-pick --onto ONTO [--fixes FIXES] -c
                               {regression,critical,fixnewfeature,docs,release}

Cherry pick a pull request onto a release branch for inclusion in a release

optional arguments:
  --onto ONTO           Branch you would like to cherry pick onto (Example: release/2.1)
  --fixes FIXES         Link to the issue that your PR fixes (Example: https://github.com/pytorch/pytorch/issues/110666)
  -c {regression,critical,fixnewfeature,docs,release}, --classification {regression,critical,fixnewfeature,docs,release}
                        A machine-friendly classification of the cherry-pick reason.

Close

usage: @pytorchbot close

Close a PR [Can be used on issues]

…ting to a single sample input" This PR: * Sets a random seed before generating each sample for an OpInfo test. It does this by intercepting the sample input iterator via `TrackedInputIter`, optionally setting the seed to a test name specific seed before each iterator call (default is to set the seed). * Some quick and dirty benchmarking shows (hopefully) negligible overhead from setting the random seed before each sample input generation. For a trivial (single assert) test that uses `ops`: * Uncovered a bunch of test issues: * Test breakdown (~100 total) * A lot of tolerance issues (tweaked tolerance values to fix) * 1 broken OpInfo (`sample_inputs_masked_fill` was generating a sample of the wrong dtype) * 3 actually broken semantics (for masked tensor; added xfails) * 4 Jacobian mismatches (added xfails) * 2 nan results (skip for now, need fixing) * 3 results too far from reference result (add xfails) * Skips MPS tests for now (there are so many failures!). Those will default to the old behavior. **before (no seed setting):** ``` real 0m21.306s user 0m19.053s sys 0m5.192s ``` **after (with seed setting):** ``` real 0m21.905s user 0m19.578s sys 0m5.390s ``` * Utilizing the above for reproducible sample input generation, adds support for restricting the iterator to a single sample input. This is done via an env var `PYTORCH_OPINFO_SAMPLE_INPUT_INDEX` and its usage is included in the repro command. ``` ====================================================================== ERROR: test_bar_add_cuda_uint8 (__main__.TestFooCUDA.test_bar_add_cuda_uint8) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 971, in test_wrapper return test(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/test/test_ops.py", line 2671, in test_bar self.assertFalse(True) AssertionError: True is not false The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 2816, in wrapper method(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 419, in instantiated_test result = test(self, **param_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_utils.py", line 1426, in wrapper fn(*args, **kwargs) File "/home/jbschlosser/branches/testing_updates/torch/testing/_internal/common_device_type.py", line 982, in test_wrapper raise new_e from e Exception: Caused by sample input at index 3: SampleInput(input=Tensor[size=(10, 5), device="cuda:0", dtype=torch.uint8], args=TensorList[Tensor[size=(), device="cuda:0", dtype=torch.uint8]], kwargs={}, broadcasts_input=False, name='') To execute this test, run the following from the base repo dir: PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=3 python test/test_ops.py -k TestFooCUDA.test_bar_add_cuda_uint8 This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 ---------------------------------------------------------------------- Ran 1 test in 0.037s FAILED (errors=1) ``` [ghstack-poisoned]

…ingle sample input ghstack-source-id: a9d0b0208e90423669f4a2347fc54b23a0bafa68 Pull Request resolved: #128238

jbschlosser · 2024-07-08T15:59:26Z

@pytorchbot merge -i

pytorchmergebot · 2024-07-08T16:01:11Z

Merge started

Your change will be merged while ignoring the following 1 checks: trunk / win-vs2019-cpu-py3 / test (default, 2, 3, windows.4xlarge.nonephemeral)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

malfet · 2024-07-09T18:30:04Z

@pytorchbot revert -m "Broke slow tests, see https://www.torch-ci.com/failure?failureCaptures=%5B%22test_ops.py%3A%3ATestCommonCUDA%3A%3Atest_compare_cpu_mode_cuda_float32%22%5D" -c nosignal

pytorchmergebot · 2024-07-09T18:31:29Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2024-07-09T18:31:37Z

Reverting PR 128238 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit c8ab2e8b637515b6488931f5e59f23848aae9991 returned non-zero exit code 1

Auto-merging torch/testing/_internal/common_methods_invocations.py
CONFLICT (content): Merge conflict in torch/testing/_internal/common_methods_invocations.py
error: could not revert c8ab2e8b63... Set seed per sample for OpInfo tests + support for restricting to a single sample input (#128238)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".
hint: Disable this message with "git config advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

Pull Request resolved: #130360 Approved by: https://github.com/malfet ghstack dependencies: #128238

Pull Request resolved: pytorch#130360 Approved by: https://github.com/malfet ghstack dependencies: pytorch#128238

Fix current failure in periodic trunk https://hud.pytorch.org/failure?name=periodic%20%2F%20linux-focal-cuda11.8-py3.10-gcc9-debug%20%2F%20test%20(default%2C%204%2C%205%2C%20linux.4xlarge.nvidia.gpu)&jobName=undefined&failureCaptures=%5B%22functorch%2Ftest_ops.py%3A%3ATestOperatorsCUDA%3A%3Atest_vjp_linalg_tensorsolve_cuda_float32%22%5D Since it appeared with #128238 that only updates random seed for the test, I expect this is just bad luck of the draw. Thus increasing tolerance like we do for other tests. Pull Request resolved: #130620 Approved by: https://github.com/lezcano, https://github.com/atalman, https://github.com/malfet

… in Windows / debug builds (#130449) Pull Request resolved: #130449 Approved by: https://github.com/zou3519, https://github.com/malfet ghstack dependencies: #128238, #130360

Pull Request resolved: pytorch#130360 Approved by: https://github.com/malfet ghstack dependencies: pytorch#128238

Fix current failure in periodic trunk https://hud.pytorch.org/failure?name=periodic%20%2F%20linux-focal-cuda11.8-py3.10-gcc9-debug%20%2F%20test%20(default%2C%204%2C%205%2C%20linux.4xlarge.nvidia.gpu)&jobName=undefined&failureCaptures=%5B%22functorch%2Ftest_ops.py%3A%3ATestOperatorsCUDA%3A%3Atest_vjp_linalg_tensorsolve_cuda_float32%22%5D Since it appeared with pytorch#128238 that only updates random seed for the test, I expect this is just bad luck of the draw. Thus increasing tolerance like we do for other tests. Pull Request resolved: pytorch#130620 Approved by: https://github.com/lezcano, https://github.com/atalman, https://github.com/malfet

… in Windows / debug builds (pytorch#130449) Pull Request resolved: pytorch#130449 Approved by: https://github.com/zou3519, https://github.com/malfet ghstack dependencies: pytorch#128238, pytorch#130360

Set seed per sample for OpInfo tests + support for restricting to a s…

5bd3602

…ingle sample input [ghstack-poisoned]

jbschlosser requested a review from a team as a code owner June 7, 2024 19:17

jbschlosser requested review from zou3519 and janeyx99 June 7, 2024 19:17

jbschlosser requested a review from mruberry as a code owner June 7, 2024 21:00

jbschlosser added a commit that referenced this pull request Jun 7, 2024

Set seed per sample for OpInfo tests + support for restricting to a s…

0cb5ce9

…ingle sample input ghstack-source-id: 74e722f8621e8ec9b9efa209183509314933571f Pull Request resolved: #128238

janeyx99 approved these changes Jun 7, 2024

View reviewed changes

jbschlosser requested review from Chillee and kshitij12345 as code owners June 12, 2024 20:47

jbschlosser added a commit that referenced this pull request Jun 12, 2024

Set seed per sample for OpInfo tests + support for restricting to a s…

2259924

…ingle sample input ghstack-source-id: 0170a5241854fcac0c03c47b7b3a6eadf4d3a1b9 Pull Request resolved: #128238

jbschlosser added topic: not user facing topic category keep-going Don't stop on first failure, keep running tests until the end labels Jun 20, 2024

jbschlosser requested review from titaiwangms, shubhambhokare1, justinchuby and wschin as code owners June 21, 2024 17:46

justinchuby approved these changes Jun 21, 2024

View reviewed changes

jbschlosser added ciflow/trunk Trigger trunk jobs on your pull request and removed ciflow/trunk Trigger trunk jobs on your pull request labels Jul 5, 2024

jbschlosser added a commit that referenced this pull request Jul 5, 2024

Set seed per sample for OpInfo tests + support for restricting to a s…

dc3e4d3

…ingle sample input ghstack-source-id: a9d0b0208e90423669f4a2347fc54b23a0bafa68 Pull Request resolved: #128238

pytorchmergebot added the merging label Jul 8, 2024

pytorchmergebot added the Merged label Jul 8, 2024

pytorchmergebot closed this in c8ab2e8 Jul 8, 2024

pytorchmergebot removed the merging label Jul 8, 2024

jeffdaily mentioned this pull request Jul 9, 2024

DISABLED test_compare_cpu_mode_cuda_float32 (__main__.TestCommonCUDA) #130353

Closed

jbschlosser mentioned this pull request Jul 9, 2024

Forward fix for test_compare_cpu_cuda_float32 #130360

Closed

pytorchmergebot pushed a commit that referenced this pull request Jul 9, 2024

Forward fix for test_compare_cpu_cuda_float32 (#130360)

fd43a2b

Pull Request resolved: #130360 Approved by: https://github.com/malfet ghstack dependencies: #128238

datagero pushed a commit to datagero/pytorch that referenced this pull request Jul 10, 2024

Forward fix for test_compare_cpu_cuda_float32 (pytorch#130360)

9d17b84

Pull Request resolved: pytorch#130360 Approved by: https://github.com/malfet ghstack dependencies: pytorch#128238

jbschlosser mentioned this pull request Jul 10, 2024

Tweak tolerances for test_vjp_linalg_tensorsolve_cuda_float32 to pass in Windows / debug builds #130449

Closed

albanD mentioned this pull request Jul 12, 2024

Increase tolerance for tensorsolve tests #130620

Closed

xuhancn pushed a commit to xuhancn/pytorch that referenced this pull request Jul 25, 2024

Forward fix for test_compare_cpu_cuda_float32 (pytorch#130360)

aa196ca

Pull Request resolved: pytorch#130360 Approved by: https://github.com/malfet ghstack dependencies: pytorch#128238

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set seed per sample for OpInfo tests + support for restricting to a single sample input #128238

Set seed per sample for OpInfo tests + support for restricting to a single sample input #128238

jbschlosser commented Jun 7, 2024 •

edited

Loading

pytorch-bot bot commented Jun 7, 2024 •

edited

Loading

janeyx99 left a comment

janeyx99 Jun 7, 2024

janeyx99 Jun 7, 2024

jbschlosser Jun 11, 2024

janeyx99 Jun 13, 2024

pytorch-bot bot commented Jul 5, 2024

jbschlosser commented Jul 8, 2024

pytorchmergebot commented Jul 8, 2024

malfet commented Jul 9, 2024

pytorchmergebot commented Jul 9, 2024

pytorchmergebot commented Jul 9, 2024

Set seed per sample for OpInfo tests + support for restricting to a single sample input #128238

Set seed per sample for OpInfo tests + support for restricting to a single sample input #128238

Conversation

jbschlosser commented Jun 7, 2024 • edited Loading

pytorch-bot bot commented Jun 7, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/128238

✅ You can merge normally! (1 Unrelated Failure)

janeyx99 left a comment

Choose a reason for hiding this comment

janeyx99 Jun 7, 2024

Choose a reason for hiding this comment

janeyx99 Jun 7, 2024

Choose a reason for hiding this comment

jbschlosser Jun 11, 2024

Choose a reason for hiding this comment

janeyx99 Jun 13, 2024

Choose a reason for hiding this comment

pytorch-bot bot commented Jul 5, 2024

PyTorchBot Help

Merge

Revert

Rebase

Label

Dr CI

cherry-pick

Close

jbschlosser commented Jul 8, 2024

pytorchmergebot commented Jul 8, 2024

Merge started

malfet commented Jul 9, 2024

pytorchmergebot commented Jul 9, 2024

pytorchmergebot commented Jul 9, 2024

Reverting PR 128238 failed

jbschlosser commented Jun 7, 2024 •

edited

Loading

pytorch-bot bot commented Jun 7, 2024 •

edited

Loading