[Train] Support full ray.get_gpu_ids() API in train.torch.get_device()
#28659
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Amog Kamsetty <[email protected]>
ray.get_gpu_ids() APIray.get_gpu_ids() API in train.torch.get_device()
matthewdeng
reviewed
Sep 21, 2022
Signed-off-by: Amog Kamsetty <[email protected]>
matthewdeng
reviewed
Sep 21, 2022
python/ray/train/tests/test_gpu.py
Outdated
Comment on lines
100
to
101
| if cuda_visible_devices: | ||
| assert os.environ["CUDA_VISIBLE_DEVICES"] == cuda_visible_devices |
There was a problem hiding this comment.
This will always return true right? Or is this just to validate that the environment variable is correctly being set?
There was a problem hiding this comment.
Yeah this is to validate that it’s being set correctly
the actual test is just to make sure it can run without raising an error. The value of get_device should be the same regardless of what the env car is set to as get_device returns the index.
matthewdeng
approved these changes
Sep 21, 2022
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
Signed-off-by: Amog Kamsetty <[email protected]>
|
Blocked on #29104 |
Signed-off-by: Amog Kamsetty <[email protected]>
|
Lint failure is unrelated, going to merge |
maxpumperla
pushed a commit
that referenced
this pull request
Oct 7, 2022
…ce()` (#28659) Signed-off-by: Amog Kamsetty [email protected] ray.get_gpu_ids() sometimes returns a list of ints or sometimes returns a list of strings depending on if the user has set the CUDA_VISIBLE_DEVICES environment variable. This has led to the following issue: #28467. It seems like solidifying the Ray Core API will require more discussion (follow the thread here: #28632), so we temporarily account for this in Ray Train for now. Closes #28467
WeichenXu123
pushed a commit
to WeichenXu123/ray
that referenced
this pull request
Dec 19, 2022
…ce()` (ray-project#28659) Signed-off-by: Amog Kamsetty [email protected] ray.get_gpu_ids() sometimes returns a list of ints or sometimes returns a list of strings depending on if the user has set the CUDA_VISIBLE_DEVICES environment variable. This has led to the following issue: ray-project#28467. It seems like solidifying the Ray Core API will require more discussion (follow the thread here: ray-project#28632), so we temporarily account for this in Ray Train for now. Closes ray-project#28467 Signed-off-by: Weichen Xu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Amog Kamsetty [email protected]
ray.get_gpu_ids()sometimes returns a list of ints or sometimes returns a list of strings depending on if the user has set theCUDA_VISIBLE_DEVICESenvironment variable. This has led to the following issue: #28467.It seems like solidifying the Ray Core API will require more discussion (follow the thread here: #28632), so we temporarily account for this in Ray Train for now.
Closes #28467
Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.