Remove ProcessGroupCudaP2P and change async-TP to use SymmetricMemory #128762

yifuwang · 2024-06-15T01:34:13Z

Stack from ghstack (oldest at bottom):

This PR removes ProcessGroupCudaP2P and changes async-TP to use SymmetricMemory. The async-TP implementation is still workspace-based, but it now doesn't require a buffer size to be specified upfront.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire

[ghstack-poisoned]

pytorch-bot · 2024-06-15T01:34:16Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/128762

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (4 Unrelated Failures)

As of commit 61ca931 with merge base dd00f5e ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

trunk / linux-focal-rocm6.1-py3.8 / test (default, 1, 2, linux.rocm.gpu) (gh) (trunk failure)
test_nn.py::TestNNDeviceTypeCUDA::test_module_to_empty_cuda_float64
trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-13) (gh) (trunk failure)
test_mps.py::TestMPS::test_mps_allocator_module
trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-14) (gh) (trunk failure)
test_mps.py::TestMPS::test_mps_allocator_module

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

inductor / rocm6.1-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2, unstable) (gh) (#128871)
'test/inductor/test_max_autotune.py::TestTuningProcess::test_tuning_pool_multiple_devices'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…emory" cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k voznesenskym EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire [ghstack-poisoned]

ghstack-source-id: a3eb6a484c6db3c9e7e077d448e4415792de872e Pull Request resolved: #128762

…emory" cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k voznesenskym EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire [ghstack-poisoned]

ghstack-source-id: c41f11c4bfdf337d7a9ad67b89767dac54ebfc61 Pull Request resolved: #128762

wanchaol

sounds great! Let's go!

Chillee · 2024-06-17T23:30:25Z

How do we control whether SymmetricMemory is used now?

yifuwang · 2024-06-18T17:40:26Z

@Chillee it is enabled via enable_symm_mem_for_group(group_name), which makes _SymmetricMemory.empty_strided work with the said group. The initialization cost is very small and the workspace is only initialized if used. Once the API matures we could enable it for all eligible process groups by default.

…etricMemory" This PR removes `ProcessGroupCudaP2P` and changes async-TP to use `SymmetricMemory`. The async-TP implementation is still workspace-based, but it now doesn't require a buffer size to be specified upfront. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k voznesenskym EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire [ghstack-poisoned]

ghstack-source-id: 3ca8f70453393c523add323bccfe1d8554001036 Pull Request resolved: #128762

jianc99 · 2024-06-20T09:03:41Z

Hi, I am using gpt-fast to perform multi-GPU inference. When I set ENABLE_INTRA_NODE_COMM=1, I got the RuntimeError: : CUDASymmetricMemory requires pidfd_open and pidfd_getfd support. I am using ubuntu 20.04. Can I solve this problem without updating the kernels? Thanks.

…etricMemory" This PR removes `ProcessGroupCudaP2P` and changes async-TP to use `SymmetricMemory`. The async-TP implementation is still workspace-based, but it now doesn't require a buffer size to be specified upfront. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k voznesenskym EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire [ghstack-poisoned]

yifuwang · 2024-06-24T20:24:46Z

Can I solve this problem without updating the kernels?

@jianc99 I can try to address this. Can you open an issue for it?

…etricMemory" This PR removes `ProcessGroupCudaP2P` and changes async-TP to use `SymmetricMemory`. The async-TP implementation is still workspace-based, but it now doesn't require a buffer size to be specified upfront. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k voznesenskym EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire [ghstack-poisoned]

yifuwang · 2024-06-25T16:42:22Z

@pytorchbot merge

pytorchmergebot · 2024-06-25T16:43:56Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-06-25T16:59:38Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-py3-arm64 / test (default, 2, 3, macos-m1-stable)

Details for Dev Infra team

Raised by workflow job

…etricMemory" This PR removes `ProcessGroupCudaP2P` and changes async-TP to use `SymmetricMemory`. The async-TP implementation is still workspace-based, but it now doesn't require a buffer size to be specified upfront. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k voznesenskym EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire [ghstack-poisoned]

yifuwang · 2024-06-25T21:02:52Z

@pytorchbot merge

pytorchmergebot · 2024-06-25T21:04:25Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

ghstack-source-id: a3eb6a484c6db3c9e7e077d448e4415792de872e Pull Request resolved: pytorch#128762

Remove ProcessGroupCudaP2P and port async-TP to SymmetricMemory

249385d

[ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: inductor oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (pipeline) release notes category labels Jun 15, 2024

yifuwang added a commit that referenced this pull request Jun 15, 2024

Remove ProcessGroupCudaP2P and port async-TP to SymmetricMemory

59e6aed

ghstack-source-id: a3eb6a484c6db3c9e7e077d448e4415792de872e Pull Request resolved: #128762

yifuwang added a commit that referenced this pull request Jun 17, 2024

Remove ProcessGroupCudaP2P and port async-TP to SymmetricMemory

f36556d

ghstack-source-id: c41f11c4bfdf337d7a9ad67b89767dac54ebfc61 Pull Request resolved: #128762

yifuwang changed the title ~~Remove ProcessGroupCudaP2P and port async-TP to SymmetricMemory~~ Remove ProcessGroupCudaP2P and change async-TP to use SymmetricMemory Jun 17, 2024

yifuwang requested review from Chillee and wanchaol June 17, 2024 20:58

yifuwang marked this pull request as ready for review June 17, 2024 20:58

yifuwang requested a review from a team as a code owner June 17, 2024 20:58

wanchaol approved these changes Jun 17, 2024

View reviewed changes

yifuwang added a commit that referenced this pull request Jun 18, 2024

Remove ProcessGroupCudaP2P and port async-TP to SymmetricMemory

07319a2

ghstack-source-id: 3ca8f70453393c523add323bccfe1d8554001036 Pull Request resolved: #128762

This was referenced Jun 20, 2024

Proof-of-concept: manage registered communication buffers with Inductor #129186

Open

Implement operator for micro-pipelined all-gather -> _scaled_mm #129289

Closed

yifuwang added topic: not user facing topic category and removed release notes: distributed (pipeline) release notes category labels Jun 25, 2024

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 25, 2024

pytorchmergebot added the merging label Jun 25, 2024

pytorchmergebot removed the merging label Jun 25, 2024

pytorchmergebot added the merging label Jun 25, 2024

pytorchmergebot added the Merged label Jun 25, 2024

pytorchmergebot closed this in bbd47f7 Jun 25, 2024

pytorchmergebot removed the merging label Jun 25, 2024

yifuwang added a commit to yifuwang/pytorch that referenced this pull request Jul 24, 2024

Remove ProcessGroupCudaP2P and port async-TP to SymmetricMemory

30029c6

ghstack-source-id: a3eb6a484c6db3c9e7e077d448e4415792de872e Pull Request resolved: pytorch#128762

github-actions bot deleted the gh/yifuwang/93/head branch July 26, 2024 01:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove ProcessGroupCudaP2P and change async-TP to use SymmetricMemory #128762

Remove ProcessGroupCudaP2P and change async-TP to use SymmetricMemory #128762

yifuwang commented Jun 15, 2024 •

edited

Loading

pytorch-bot bot commented Jun 15, 2024 •

edited

Loading

wanchaol left a comment

Chillee commented Jun 17, 2024

yifuwang commented Jun 18, 2024

jianc99 commented Jun 20, 2024 •

edited

Loading

yifuwang commented Jun 24, 2024

yifuwang commented Jun 25, 2024

pytorchmergebot commented Jun 25, 2024

pytorchmergebot commented Jun 25, 2024

yifuwang commented Jun 25, 2024

pytorchmergebot commented Jun 25, 2024

Remove ProcessGroupCudaP2P and change async-TP to use SymmetricMemory #128762

Remove ProcessGroupCudaP2P and change async-TP to use SymmetricMemory #128762

Conversation

yifuwang commented Jun 15, 2024 • edited Loading

pytorch-bot bot commented Jun 15, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/128762

✅ You can merge normally! (4 Unrelated Failures)

wanchaol left a comment

Choose a reason for hiding this comment

Chillee commented Jun 17, 2024

yifuwang commented Jun 18, 2024

jianc99 commented Jun 20, 2024 • edited Loading

yifuwang commented Jun 24, 2024

yifuwang commented Jun 25, 2024

pytorchmergebot commented Jun 25, 2024

Merge started

pytorchmergebot commented Jun 25, 2024

Merge failed

yifuwang commented Jun 25, 2024

pytorchmergebot commented Jun 25, 2024

Merge started

yifuwang commented Jun 15, 2024 •

edited

Loading

pytorch-bot bot commented Jun 15, 2024 •

edited

Loading

jianc99 commented Jun 20, 2024 •

edited

Loading