Added hpu backend support in fsdp utils #127757
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/127757
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 1eeaad9 with merge base 406f510 ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@pytorchbot label "topic: not user facing" |
|
@wconstab could you please review. |
|
@wconstab could you please review the change. |
| @@ -817,6 +817,16 @@ def _get_device_from_device_id( | |||
| "index as the `device_id` argument." | |||
| ) | |||
| device = torch.device("cuda", torch.cuda.current_device()) | |||
| if device == torch.device("hpu"): | |||
There was a problem hiding this comment.
nit: it would be nicer to avoid the copy-paste here. Could the code be written in a way to have a list of device types and their cooresponding current_device functions, then just iterate this list and issue the warning/return inside the loop?
There was a problem hiding this comment.
lgtm.
@awgu wonder if you think it is possible as a follow up to replace the list of backend_devices with something that iterates over all the devices types in pytorch, or do you think its necessary to update this code for every device-type we care to support?
There was a problem hiding this comment.
I think we should try to use the device handle abstraction included from previous backend work.
pytorch/torch/distributed/fsdp/_init_utils.py
Line 362 in 2a28299
Maybe we can pass in the _FSDPState into this call:
pytorch/torch/distributed/fsdp/_init_utils.py
Line 555 in 2a28299
and then use
_FSDPState._device_handle in place of torch.cuda or torch.hpu.
|
Lint pops an attribute error. Does it require some registration eve in case to use a string name in list. Error (MYPY) [attr-defined] |
|
Any pointers to resolve Lint error about 'Module has no attribute "hpu"' ? Please share suggestion. |
|
Pushed another patch to resolve Lint error. |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
|
Test failure looks real |
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
|
Successfully rebased |
673973f
to
97f6286
Compare
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
|
Successfully rebased |
4784f01
to
03f537a
Compare
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
Replace `“important”` with `"important"` and `Taylor’s` with `Taylor's` Fixes the obvious symptoms of pytorch#124897 Test plan: Download [wheel](https://github.com/pytorch/pytorch/actions/runs/8833051644/artifacts/1447995459) and check that generated VF.pyi does not have any unicode characters by running following command: ``` % python3 -c "x=open('_VF.pyi', encoding='utf-8').read();uc=[(i, x[i]) for i in range(len(x)) if ord(x[i])>127];print(uc);assert(len(uc)==0)" ```
|
Successfully rebased |
03f537a
to
e7c5ec9
Compare
Replace `“important”` with `"important"` and `Taylor’s` with `Taylor's` Fixes the obvious symptoms of pytorch#124897 Test plan: Download [wheel](https://github.com/pytorch/pytorch/actions/runs/8833051644/artifacts/1447995459) and check that generated VF.pyi does not have any unicode characters by running following command: ``` % python3 -c "x=open('_VF.pyi', encoding='utf-8').read();uc=[(i, x[i]) for i in range(len(x)) if ord(x[i])>127];print(uc);assert(len(uc)==0)" ```
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 3 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
In fsdp init_utils, adding support for hpu backend device on _get_device API. Co-authored-by: Nikita Shulga <[email protected]> Pull Request resolved: #127757 Approved by: https://github.com/wconstab, https://github.com/jgong5, https://github.com/awgu (cherry picked from commit bcdba9f)
In fsdp init_utils, adding support for hpu backend device on _get_device API.
cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @penguinwu @tianyu-l @yf225 @chauhang @rohan-varma