-
Notifications
You must be signed in to change notification settings - Fork 21.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TunableOp hotfix #129281
TunableOp hotfix #129281
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129281
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit 315ba1b with merge base 920ebcc (): NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeffdaily thanks for the fix! I wonder if we can add unit tests to tunable op?
@pytorchbot rebase |
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
Successfully rebased |
4cc0180
to
82381b3
Compare
@mxz297 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Still ran into memory access fault at model level. It looks like my previous repo did not fully capture the tensor layout. A new repo has been provided to @jeffdaily |
@mxz297 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Resolved memory access fault observed for tunable op in internal workload. |
@pytorchbot merge |
Merge failedReason: This PR needs a If not, please add the To add a label, you can comment to pytorchbot, for example For more information, see Details for Dev Infra teamRaised by workflow job |
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 jobs have failed, first few of them are: trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-13) Details for Dev Infra teamRaised by workflow job |
@pytorchbot merge -f "unrelated mac failure" |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
@pytorchmergebot cherry-pick --onto release/2.4 -c critical |
Fixes. - PYTORCH_TUNABLEOP_NUMERICAL_CHECK=1 had a memory leak. - The strided batched gemm size calculation for buffer rotation was incorrect resulting in a mem fault. Pull Request resolved: #129281 Approved by: https://github.com/xw285cornell, https://github.com/eqy, https://github.com/mxz297 (cherry picked from commit e68ee2c)
Cherry picking #129281The cherry pick PR is at #129499 and it is recommended to link a critical cherry pick PR with an issue. The following tracker issues are updated: Details for Dev Infra teamRaised by workflow job |
PR pytorch#129281 was landed to fix critical issues but did not contain unit tests to exercise those issues. This is a follow-up set of unit tests that would exercise the problems seen previously.
PR pytorch#129281 was landed to fix critical issues but did not contain unit tests to exercise those issues. This is a follow-up set of unit tests that would exercise the problems seen previously.
PR #129281 was landed to fix critical issues but did not contain unit tests to exercise those issues. This is a follow-up set of unit tests that would exercise the problems seen previously. Pull Request resolved: #129606 Approved by: https://github.com/atalman
TunableOp hotfix (#129281) Fixes. - PYTORCH_TUNABLEOP_NUMERICAL_CHECK=1 had a memory leak. - The strided batched gemm size calculation for buffer rotation was incorrect resulting in a mem fault. Pull Request resolved: #129281 Approved by: https://github.com/xw285cornell, https://github.com/eqy, https://github.com/mxz297 (cherry picked from commit e68ee2c) Co-authored-by: Jeff Daily <[email protected]>
TunableOp hotfix, unit test follow-up PR #129281 was landed to fix critical issues but did not contain unit tests to exercise those issues. This is a follow-up set of unit tests that would exercise the problems seen previously.
Fixes. - PYTORCH_TUNABLEOP_NUMERICAL_CHECK=1 had a memory leak. - The strided batched gemm size calculation for buffer rotation was incorrect resulting in a mem fault. Pull Request resolved: pytorch#129281 Approved by: https://github.com/xw285cornell, https://github.com/eqy, https://github.com/mxz297
Fixes.