-
Notifications
You must be signed in to change notification settings - Fork 22k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added hpu backend support in fsdp utils #127757
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/127757
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 1eeaad9 with merge base 406f510 ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
@pytorchbot label "topic: not user facing" |
@wconstab could you please review. |
@wconstab could you please review the change. |
@@ -817,6 +817,16 @@ def _get_device_from_device_id( | |||
"index as the `device_id` argument." | |||
) | |||
device = torch.device("cuda", torch.cuda.current_device()) | |||
if device == torch.device("hpu"): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: it would be nicer to avoid the copy-paste here. Could the code be written in a way to have a list of device types and their cooresponding current_device functions, then just iterate this list and issue the warning/return inside the loop?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm.
@awgu wonder if you think it is possible as a follow up to replace the list of backend_devices
with something that iterates over all the devices types in pytorch, or do you think its necessary to update this code for every device-type we care to support?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should try to use the device handle abstraction included from previous backend work.
pytorch/torch/distributed/fsdp/_init_utils.py
Line 362 in 2a28299
def _init_device_handle( |
Maybe we can pass in the _FSDPState
into this call:
pytorch/torch/distributed/fsdp/_init_utils.py
Line 555 in 2a28299
device_from_device_id = _get_device_from_device_id(device_id, state.rank) |
and then use
_FSDPState._device_handle
in place of torch.cuda
or torch.hpu
.
Lint pops an attribute error. Does it require some registration eve in case to use a string name in list. Error (MYPY) [attr-defined]
|
Any pointers to resolve Lint error about 'Module has no attribute "hpu"' ? Please share suggestion. |
Pushed another patch to resolve Lint error. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good.
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
Test failure looks real |
@pytorchbot rebase |
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
Successfully rebased |
673973f
to
97f6286
Compare
@pytorchbot rebase |
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
Successfully rebased |
4784f01
to
03f537a
Compare
@pytorchbot rebase |
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
Replace `“important”` with `"important"` and `Taylor’s` with `Taylor's` Fixes the obvious symptoms of pytorch#124897 Test plan: Download [wheel](https://github.com/pytorch/pytorch/actions/runs/8833051644/artifacts/1447995459) and check that generated VF.pyi does not have any unicode characters by running following command: ``` % python3 -c "x=open('_VF.pyi', encoding='utf-8').read();uc=[(i, x[i]) for i in range(len(x)) if ord(x[i])>127];print(uc);assert(len(uc)==0)" ```
Successfully rebased |
03f537a
to
e7c5ec9
Compare
Replace `“important”` with `"important"` and `Taylor’s` with `Taylor's` Fixes the obvious symptoms of pytorch#124897 Test plan: Download [wheel](https://github.com/pytorch/pytorch/actions/runs/8833051644/artifacts/1447995459) and check that generated VF.pyi does not have any unicode characters by running following command: ``` % python3 -c "x=open('_VF.pyi', encoding='utf-8').read();uc=[(i, x[i]) for i in range(len(x)) if ord(x[i])>127];print(uc);assert(len(uc)==0)" ```
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 3 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
In fsdp init_utils, adding support for hpu backend device on _get_device API. Co-authored-by: Nikita Shulga <[email protected]> Pull Request resolved: #127757 Approved by: https://github.com/wconstab, https://github.com/jgong5, https://github.com/awgu (cherry picked from commit bcdba9f)
In fsdp init_utils, adding support for hpu backend device on _get_device API.
cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @penguinwu @tianyu-l @yf225 @chauhang @rohan-varma