[inductor] Kill mark_node_as_mutating

[peterbell10]

Stack from ghstack (oldest at bottom):

• [inductor] Refactor fusion of inplace operations #128979
• -> [inductor] Kill mark_node_as_mutating #129346
• [inductor] Use multiple outputs for flex-attention #129344
• [inductor] Make UserDefinedTritonKernel a multi-output operation #129325
• [inductor] Separate Buffer and Operation into two concepts #128893

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129346

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 77 Cancelled Jobs, 3 Unrelated Failures

As of commit 40d6db1 with merge base c549629 ():

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• inductor / cuda12.1-py3.10-gcc9-sm86 / build (gh)
  ##[error]The operation was canceled.
• inductor / cuda12.1-py3.12-gcc9-sm86 / build (gh)
  ##[error]The operation was canceled.
• inductor / cuda12.4-py3.10-gcc9-sm86 (gh)
• inductor / cuda12.4-py3.10-gcc9-sm86 / build (gh)
  ##[error]The operation was canceled.
• inductor / linux-jammy-cpu-py3.12-gcc11-inductor-halide / build (gh)
  ##[error]The operation was canceled.
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor (gh)
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / build (gh)
  ##[error]The operation was canceled.
• inductor / rocm6.1-py3.8-inductor / build (gh)
  ##[error]The operation was canceled.
• inductor-periodic / cuda12.1-py3.10-gcc9-sm80 / build (gh)
  ##[error]The operation was canceled.
• inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / build (gh)
  ##[error]The operation was canceled.
• Lint / lintrunner-clang / linux-job (gh)
  ##[error]The operation was canceled.
• Lint / lintrunner-noclang / linux-job (gh)
  ##[error]The operation was canceled.
• linux-binary-libtorch-cxx11-abi / libtorch-cpu-shared-with-deps-cxx11-abi-build / build (gh)
  ##[error]The operation was canceled.
• linux-binary-libtorch-cxx11-abi / libtorch-cpu-shared-with-deps-cxx11-abi-test (gh)
• linux-binary-libtorch-pre-cxx11 / libtorch-cpu-shared-with-deps-pre-cxx11-build / build (gh)
  ##[error]The operation was canceled.
• linux-binary-libtorch-pre-cxx11 / libtorch-cpu-shared-with-deps-pre-cxx11-test (gh)
• linux-binary-manywheel / manywheel-py3_8-cuda11_8-build / build (gh)
  ##[error]The operation was canceled.
• linux-binary-manywheel / manywheel-py3_8-cuda11_8-split-build / build (gh)
  ##[error]The operation was canceled.
• linux-binary-manywheel / manywheel-py3_8-cuda11_8-split-test (gh)
• linux-binary-manywheel / manywheel-py3_8-cuda11_8-test (gh)
• linux-binary-manywheel / manywheel-py3_8-cuda12_1-build / build (gh)
  ##[error]The operation was canceled.
• linux-binary-manywheel / manywheel-py3_8-cuda12_1-split-build / build (gh)
  ##[error]The operation was canceled.
• linux-binary-manywheel / manywheel-py3_8-cuda12_1-split-test (gh)
• linux-binary-manywheel / manywheel-py3_8-cuda12_1-test (gh)
• linux-binary-manywheel / manywheel-py3_8-cuda12_4-build / build (gh)
  ##[error]The operation was canceled.
• linux-binary-manywheel / manywheel-py3_8-cuda12_4-split-build / build (gh)
  ##[error]The operation was canceled.
• linux-binary-manywheel / manywheel-py3_8-cuda12_4-split-test (gh)
• linux-binary-manywheel / manywheel-py3_8-cuda12_4-test (gh)
• pull / linux-docs (gh)
• pull / linux-focal-cpu-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-cuda11.8-py3.10-gcc9 (gh)
• pull / linux-focal-cuda11.8-py3.10-gcc9 / build (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-cuda12.1-py3.10-gcc9 / build (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge.nvidia.gpu) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build (gh)
• pull / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / build (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / build (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-cuda12.4-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge.nvidia.gpu) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3_8-clang9-xla / build (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3-clang9-android-ndk-r21e-gradle-custom-build-single / build-and-test (default, 1, 1, linux.2xlarge) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3-clang9-android-ndk-r21e-gradle-custom-build-single-full-jit / build-and-test (default, 1, 1, linux.2xlarge) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.11-clang10 (gh)
• pull / linux-focal-py3.11-clang10 / build (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.12-clang10 / build (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 1, 3, linux.2xlarge) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 2, 3, linux.2xlarge) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 3, 3, linux.2xlarge) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 1, 3, linux.2xlarge) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 2, 3, linux.2xlarge) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 3, 3, linux.2xlarge) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.8-clang10 / build (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-py3.8-clang10-onnx / build (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-rocm6.1-py3.8 / build (gh)
  ##[error]The operation was canceled.
• pull / linux-jammy-cuda11.8-cudnn9-py3.8-clang12 / build (gh)
  ##[error]The operation was canceled.
• pull / linux-jammy-py3-clang12-executorch / build (gh)
  ##[error]The operation was canceled.
• pull / linux-jammy-py3.10-clang15-asan (gh)
• pull / linux-jammy-py3.10-clang15-asan / build (gh)
  ##[error]The operation was canceled.
• pull / linux-jammy-py3.8-gcc11 / build (gh)
  ##[error]The operation was canceled.
• pull / linux-jammy-py3.8-gcc11-mobile-lightweight-dispatch-build / build (gh)
  ##[error]The operation was canceled.
• pull / linux-jammy-py3.8-gcc11-no-ops / build (gh)
  ##[error]The operation was canceled.
• pull / linux-jammy-py3.8-gcc11-pch / build (gh)
  ##[error]The operation was canceled.
• trunk / libtorch-linux-focal-cuda12.1-py3.7-gcc9-debug / build (gh)
  ##[error]The operation was canceled.
• trunk / libtorch-linux-focal-cuda12.4-py3.7-gcc9-debug / build (gh)
  ##[error]The operation was canceled.
• trunk / linux-focal-cuda11.8-py3.10-gcc9-experimental-split-build / build (gh)
  ##[error]The operation was canceled.
• trunk / linux-focal-cuda11.8-py3.10-gcc9-experimental-split-build-test (gh)
• trunk / linux-focal-cuda12.1-py3.10-gcc9-no-ops / build (gh)
  ##[error]The operation was canceled.
• trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build / build (gh)
  ##[error]The operation was canceled.
• trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test (gh)
• trunk / linux-focal-cuda12.4-py3.10-gcc9-no-ops / build (gh)
  ##[error]The operation was canceled.
• trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / build (gh)
  ##[error]The operation was canceled.
• trunk / linux-focal-rocm6.1-py3.8 / build (gh)
  ##[error]The operation was canceled.
• trunk / macos-py3-arm64 / test (default, 1, 3, macos-m1-stable) (gh)
  ##[error]The operation was canceled.
• trunk / macos-py3-arm64 / test (default, 2, 3, macos-m1-stable) (gh)
  ##[error]The operation was canceled.
• trunk / macos-py3-arm64 / test (default, 3, 3, macos-m1-stable) (gh)
  ##[error]The operation was canceled.
• trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-13) (gh)
  ##[error]The operation was canceled.
• trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-14) (gh)
  ##[error]The operation was canceled.
• trunk / pytorch-linux-focal-py3-clang9-android-ndk-r21e-build / build (default, 1, 1, linux.2xlarge) (gh)
  ##[error]The operation was canceled.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / win-vs2019-cpu-py3 / build (gh) (matched win rule in flaky-rules.json)
  ##[error]The operation was canceled.
• trunk / win-vs2019-cpu-py3 / build (gh) (matched win rule in flaky-rules.json)
  ##[error]The operation was canceled.
• trunk / win-vs2019-cuda12.1-py3 / build (gh) (matched win rule in flaky-rules.json)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[peterbell10]

@ColinPeppler thank you for the reproducer and fix. Should be fixed now in #128893

@yf225 I think this is worth doing as we need to support operations with multiple outputs (including mutations) and up till this point this was done by a variety of hacky workarounds. e.g. MutationOutput as a separate operation doesn't represent the dependencies correctly and implicitly assumes that the scheduler won't reorder nodes.

[peterbell10]

@ColinPeppler would you mind importing this PR to see if the errors are fixed?

[linux-foundation-easycla]

• ✅ login: peterbell10 (44f983b, 04cf759, d672167, 2fadcb9, 50a33de, 85c1c4f, 7210a5a, 6cf1a0e, 538609d, 31abfac, 9781bfd, 5034f03, 4fab046, 5aeecdb, bcee38b, 4fbda96, c91ab26, 77040e5, 36f8328, 1546348, 3cd283a, 40d6db1, f0bbd9f, 2c25710, 7edd4ce, 86278b7)
• ❌ The email address for the commit (ae7f340) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please submit a support request ticket.