[v.2.4.0] Release Tracker

[atalman]

We cut a release branch for the 2.4.0 release.

Our plan from this point from this point is roughly:

• Phase 1 (until 7/1/24): work on finalizing the release branch
• Phase 2 (after 7/1/24): perform extended integration/stability/performance testing based on Release Candidate builds.

This issue is for tracking cherry-picks to the release branch.

Cherry-Pick Criteria

Phase 1 (until 7/1/24):

Only low-risk changes may be cherry-picked from main:

1. Fixes to regressions against the most recent minor release (e.g. 2.3.x for this release; see module: regression issue list)
2. Critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks
3. Critical fixes to new features introduced in the most recent minor release (e.g. 2.3.x for this release)
4. Test/CI fixes
5. Documentation improvements
6. Compilation fixes or ifdefs required for different versions of the compilers or third-party libraries
7. Release branch specific changes (e.g. change version identifiers)

Any other change requires special dispensation from the release managers (currently @atalman, @PaliC , @huydhn, @malfet). If this applies to your change please write "Special Dispensation" in the "Criteria Category:" template below and explain.

Phase 2 (after 7/1/24):

Note that changes here require us to rebuild a Release Candidate and restart extended testing (likely delaying the release). Therefore, the only accepted changes are Release-blocking critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

Changes will likely require a discussion with the larger release team over VC or Slack.

Cherry-Pick Process

1. Ensure your PR has landed in master. This does not apply for release-branch specific changes (see Phase 1 criteria).

2. Create (but do not land) a PR against the release branch.

   # Find the hash of the commit you want to cherry pick
   # (for example, abcdef12345)
   git log
   
   git fetch origin release/2.4
   git checkout release/2.4
   git cherry-pick -x abcdef12345
   
   # Submit a PR based against 'release/2.4' either:
   # via the GitHub UI
   git push my-fork
   
   # via the GitHub CLI
   gh pr create --base release/2.4

3. Make a request below with the following format:

Link to landed trunk PR (if applicable):
* 

Link to release branch PR:
* 

Criteria Category:
* 

1. Someone from the release team will reply with approved / denied or ask for more information.
2. If approved, someone from the release team will merge your PR once the tests pass. Do not land the release branch PR yourself.

NOTE: Our normal tools (ghstack / ghimport, etc.) do not work on the release branch.

Please note HUD Link with branch CI status and link to the HUD to be provided here.
HUD

Versions

2.4.0

[atalman]

Link to landed trunk PR (if applicable):

• NA

Link to release branch PR:

• [Release 2.4] Release only changes #128347
• [Release 2.4] Release only changes - use pinned triton. #128388
• [Release 2.4] Temp change to build triton from pin builder#1863
• [Release only] use triton 3.0.x from pypi #130336

Criteria Category:

• Release only changes, temp changes to build triton from pin rather then branch

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Add AOTriton install step for ROCm manylinux images builder#1862

Link to release branch PR:

• Add AOTriton install step for ROCm manylinux images (#1862) builder#1867

Criteria Category:

• Required to fix rocm nightly build error AOTriton Cmake error breaking PyTorch nightly binary builds for ROCm #128420

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• [dynamo] Fix for #127696 #128358

Link to release branch PR:

• [dynamo] Fix for #127696 #128530

Criteria Category:

• Critical fix, forward fix for Fix 'get_real_value' on placeholder nodes #127698 on release branch

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Revert of [cuDNN][SDPA] Remove TORCH_CUDNN_SDPA_ENABLED=1, enable cuDNN SDPA by default on H100 and 2nd on other archs >= sm80 #125343

Link to release branch PR:

• Revert "[cuDNN][SDPA] Remove TORCH_CUDNN_SDPA_ENABLED=1, enable cuDNN SDPA by default on H100 and 2nd on other archs >= sm80 (#125343)" #128539

Criteria Category:

• Reverted on main

---

@malfet merged

[atalman]

Link to landed trunk PR (if applicable):

• Revert of Set simdlen based on ATEN_CPU_CAPABILITY #123514

Link to release branch PR:

• Revert "Set simdlen based on ATEN_CPU_CAPABILITY (#123514)" #128541

Criteria Category:

• Reverted on main

---

@malfet merged

[atalman]

Link to landed trunk PR (if applicable):

• Revert of Deprecate torch._utils.is_compiling() and torch._dynamo.external_utils.is_compiling() #127690

Link to release branch PR:

• Revert "Deprecate torch._utils.is_compiling() and torch._dynamo.external_utils.is_compiling() (#127690)" #128542

Criteria Category:

• Reverted on main

---

@malfet merged

[zhuhaozhe]

Link to landed trunk PR (if applicable):

• [inductor] fix linear add bias pattern #128473

Link to release branch PR:

• [inductor] fix linear add bias pattern (#128473) #128577

Criteria Category:

• Critical fix, forward fix for Assertion error when running timm_models "levit_128" #128287 on release branch. It's a regression on inductor, will crash on many benchmark models without this fix.

---

@atalman merged

[zou3519]

Link to landed trunk PR (if applicable):

• Change Dynamo's custom ops warning message to be less spammy #128456

Link to release branch PR:

• Change Dynamo's custom ops warning message to be less spammy (#128456) #128581

Criteria Category:

• (1) Fixes to regressions. In 2.4, we started spamming warnings if someone used pybind'ed functions with torch.compile. There were no such warnings in 2.3. This PR adjusts the warnings to be less spammy.

---

@atalman merged

[etaf]

Link to landed trunk PR (if applicable):

• Clean up xpu ut to make CI happy #128383

Link to release branch PR:

• Clean up xpu ut to make CI happy (#128383) #128614

Criteria Category:

• (4) Test/CI fixes. Fix UT and unblock the xpu CI: before [CI] disable td for xpu ci test by default #127611 merged, the xpu-specific UT test/test_xpu.py was skipped temporarily because the test cases were disabled by TD. This PR aims to fix the UT bug introduced by Switch to torch.float16 on XPU AMP mode #127741 .

---

@atalman merged

[zou3519]

Link to landed trunk PR (if applicable):

• [custom_op] stop using nonlocals to store information #128547

Link to release branch PR:

• [custom_op] stop using nonlocals to store information (#128547) #128616

Criteria Category:

• (3) Critical fixes to new features. In 2.4 we are releasing a new torch.library.custom_op API. This PR fixes a critical bug that the API did not compose with FSDP and other distributed APIs.

---

@atalman merged

[chunyuan-w]

Link to landed trunk PR (if applicable):

• Revert of Make torch_geometric models compatible with export #123403 (the revert is done in Revert "Make torch_geometric models compatible with export (#123403)" #128377)

Link to release branch PR:

• Revert "Make torch_geometric models compatible with export (#123403)"… #128511

Criteria Category:

• Reverted on main to fix a critical regression issue ([AOTI] Conv-BN folding on CPU not working anymore after benchmark script change in https://github.com/pytorch/pytorch/pull/123403 #127513) that will impact the AOTI CPU performance on multiple CNN models in the dynamo benchmark.

---

@atalman merged

[etaf]

Link to landed trunk PR (if applicable):

• [Inductor] Update Intel GPU Triton commit pin. #124842

Link to release branch PR:

• [Inductor] Update Intel GPU Triton commit pin. (#124842) #128615

Criteria Category:

• Critical fix, forward fix for Test case test_torchinductor.py:test_nll_loss_backward_xpu failed on Intel GPU #126173. Also update Intel triton for release 2.4.

---

@atalman merged

[wanchaol]

Link to landed trunk PR (if applicable):

• [inductor] fix compile time regression by caching get_gpu_type #128363

Link to release branch PR:

• [inductor] fix compile time regression by caching get_gpu_type (#128363) #128717

Criteria Category:

• Critical fixes for compile time regression

---

@atalman merged

[wanchaol]

Link to landed trunk PR (if applicable):

• [tp] refactor and fix PrepareModuleInput for DTensor inputs #128431

Link to release branch PR:

• [tp] refactor and fix PrepareModuleInput for DTensor inputs (#128431) #128719

Criteria Category:

• Critical fixes for silent correctness

---

@atalman merged

[Xia-Weiwen]

Link to landed trunk PR (if applicable):

• [Quant][Inductor] Bug fix: mutation nodes not handled correctly for QLinearPointwiseBinaryPT2E #127592

Link to release branch PR:

• [Port][Quant][Inductor] Bug fix: mutation nodes not handled correctly for QLinearPointwiseBinaryPT2E #128591

Criteria Category:

• Critical fix for [inductor][cpu]abnormal performance improvement and accuracy drop for huggingface suit static/dynamic quantization #127402. Without the fix, transformer-based models will fail to run with PT2E quantization on X86Inductor backend.

---

@atalman merged

[xuhancn]

Link to landed trunk PR (if applicable):

• [cpp_extension][inductor] Fix sleef windows depends. #128770

Link to release branch PR:

• [cpp_extension][inductor] Fix sleef windows depends. (#128770) #128811

Criteria Category:

• Critical fixes for PyTorch Windows cpp_extension. If user use cpp_extension to build extension and its code contains VEC, it will occur sleef dependency issue.

---

Hi @xuhancn this looks like feature work to enable inductor on PyTorch Windows, however for release 2.4 we don't support this yet.
Do you have any tests with it ?

@atalman merged. ToDo run manual validation on RC with test provided in the PR

[zou3519]

Link to landed trunk PR (if applicable):

• [Inductor] Fix the High Order Op layout issue #128275

Link to release branch PR:

• [Inductor] Fix the High Order Op layout issue (#128275) #128834

Criteria Category:

• (1) Regression (from 2.3.x) and (2) Critical fixes for: silent incorrectness

---

@atalman merged

[lw]

Link to landed trunk PR (if applicable):

• Remove leftover warning causing log spew #128688

Link to release branch PR:

• Remove leftover warning causing log spew #128837

Criteria Category:

• Fixes to regressions against the most recent minor release (changes only logging, low risk)

---

@atalman merged

[clee2000]

Link to landed trunk PR (if applicable):

• Switch to ${{ runner.temp }} test-infra#5327

Link to release branch PR:

• Switch to ${{ runner.temp }} (#5327) test-infra#5348

Criteria Category:

• Release - needed to fix executorch release CI

---

@clee2000 merged

[pytorchbot]

Link to landed trunk PR (if applicable):

• Inductor to fail gracefully on Voltas for bf16 tensors #129288

Link to release branch PR:

• Inductor to fail gracefully on Voltas for bf16 tensors #129699

Criteria Category:
Critical fix inductor on V100

---

@atalman merged

[pytorchbot]

Link to landed trunk PR (if applicable):

• Add threadfence to 2-stage reduction for correct writes visibility #128455

Link to release branch PR:

• Add threadfence to 2-stage reduction for correct writes visibility #129701

Criteria Category:
Critical fix

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Update MKL to 2023.1 on windows conda. update numpy for py3.8 builder#1891

Link to release branch PR:

• Update MKL to 2023.1 on windows conda. update numpy for py3.8 builder#1894

Criteria Category:

• Critical fix MKL dependency

---

@atalman merged

[pytorchbot]

Link to landed trunk PR (if applicable):

• [distributed] NCCL result code update #128777

Link to release branch PR:

• [distributed] NCCL result code update #129704

Criteria Category:
Critical fix, distributed

---

@atalman merged

[oraluben]

Link to landed trunk PR (if applicable):

• (not landed yet) [CMAKE] Look for Development.Module instead of Development #129669

Link to release branch PR:

• [CMAKE] Look for Development.Module instead of Development #129729

Criteria Category:

• Breaks building wheels in specific cases

---

@oraluben please note: the PR must be landed in main in order to be considered for cherry-picking

@malfet merged

[pytorchbot]

Link to landed trunk PR (if applicable):

• [Windows] remove mkl shared library dependency. #129493

Link to release branch PR:

• [Windows] remove mkl shared library dependency. #129740

Criteria Category:
Critical - packaging

---

@atalman merged

[Skylion007]

Wondering if we can get this landined in main so we can backport it. We've encoutered some hanging on certain hardware with certain firmware versions that only occur on this specific NCCL version (fixed in NCCL versions before and after): #124014

[pytorchbot]

Link to landed trunk PR (if applicable):

• Improve error message for weights_only load #129705

Link to release branch PR:

• Improve error message for weights_only load #129783

Criteria Category:
Fixnewfeature

---

@atalman merged

[tinglvv]

Link to landed trunk PR (if applicable):

• [aarch64] Fix for libgfortran.so.5 missing error when running CUDA ARM wheel  builder#1895

Link to release branch PR:

• [aarch64] Fix for libgfortran.so.5 missing error when running CUDA AR… builder#1902

Criteria Category:

• Fix for "libgfortran.so.5" missing error for CUDA ARM wheel at runtime

---

@atalman merged

[wz337]

Link to landed trunk PR (if applicable):

• [FSDP2+TP] Disable 2D state_dict #129519

Link to release branch PR:

• [CherryPick][FSDP2+TP] Disable 2D state_dict (#129519) #129923

Criteria Category:

• Critical fix. Need this to avoid silent correctness for a feature not ready.

---

@atalman merged

[pytorchbot]

Link to landed trunk PR (if applicable):

• Update torchbench model expected accuracy values after pinning numpy #129213

Link to release branch PR:

• Update torchbench model expected accuracy values after pinning numpy #129986

Criteria Category:
Release - Update inductor expected results after pinning numpy on torchbench

[atalman]

Please note, we are in:
Phase 2 (after 7/1/24): perform extended integration/stability/performance testing based on Release Candidate builds.
We do not accept new cherry-pick requests

[ppwwyyxx]

Can we address #130658 and #130659 ? Otherwise 2.4 users will be guaranteed to see a lot of warnings.

[atalman]

hi @ppwwyyxx both of the PRs #130658 and #130659 are targeted for release 2.4.1

[XuehaiPan]

Link to landed trunk PR (if applicable):

• Pin sympy >= 1.13.0 #130895

Link to release branch PR:

• [reland, 2.4] Pin sympy >= 1.13.0 for pip and sympy >= 1.12.0 for conda #131281

Criteria Category:

• Critical fixes for dependency update

---

@XuehaiPan This pr broke linux nightly binaries, please see here: https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/1?per_page=50&name_filter=linux- . Conda builds will require forward fix. Here is the failure: https://github.com/pytorch/pytorch/actions/runs/10036618056/job/27749407507#step:17:663 . Looks like sympy 1.13 is not available on conda: https://anaconda.org/anaconda/sympy