run_test: Unset cpp stacktraces after reruns

[clee2000]

Rerun the failing test singly with the env var set. If it succeeds, start a new process without the cpp stack traces env var

We don't want to waste time generating these if we don't have to

They can also show up in assertion errors, which may cause unexpected failures if a test wants to check these

Adds new --rs (run single) to be used the same way --scs and --sc are. It will only run the single test in the step current file

https://hud.pytorch.org/pytorch/pytorch/pull/129004?sha=2c349d3557d399020bf1f6a8b7045e2e4957ba46 has some examples of logs

In the above:

• test_checkpoint_valid failed, then passed in another subprocess. The testing continued in a different new subprocess from the test right after it (test_checkpointing_without_reentrant_early_free)
• test_format_traceback_short failed consistently, but it continued to run because keep-going was set

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129004

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 12 Unrelated Failures

As of commit a3312cf with merge base a0e1e20 ():

NEW FAILURE - The following job has failed:

• trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-14) (gh)
  test_mps.py::TestMPS::test_mps_allocator_module

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-13) (gh) (matched macos rule in flaky-rules.json)
  Failure: There is only 1627928KB free space left in /, which is less than the minimum requirement of

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

• pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 1, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)
  Process completed with exit code 9.
• pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)
  Process completed with exit code 9.
• pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)
  Process completed with exit code 9.
• pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)
  Process completed with exit code 9.
• pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (default, 5, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#129080)
  Process completed with exit code 9.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 1, 3, linux.2xlarge, unstable) (gh) (#129248)
  Process completed with exit code 9.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 2, 3, linux.2xlarge, unstable) (gh) (#129248)
  Process completed with exit code 9.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 3, 3, linux.2xlarge, unstable) (gh) (#129248)
  Process completed with exit code 9.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 1, 3, linux.2xlarge, unstable) (gh) (#129256)
  Process completed with exit code 9.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 2, 3, linux.2xlarge, unstable) (gh) (#129256)
  Process completed with exit code 9.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 3, 3, linux.2xlarge, unstable) (gh) (#129256)
  Process completed with exit code 9.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[clee2000]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[pytorchmergebot]

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-14)

Details for Dev Infra team

Raised by workflow job

[clee2000]

@pytorchbot merge -f "failure is present on main"

[pytorchmergebot]

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here