-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Datasets] Autodetect dataset parallelism based on available resources and data size #25883
Merged
Merged
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit
Hold shift + click to select a range
94fa31e
wip
ericl d52cbba
wip
ericl b813603
wip
ericl 34c5eb3
wip
ericl 7d46ef0
lint
ericl f7fc883
wip
ericl 622d211
lint
ericl 7163e0b
update
ericl c831447
rename
ericl 5145ab4
update
ericl d2c5a18
update
ericl 7b8794c
update
ericl 55a48d4
update
ericl ed8e8f5
update
ericl 510b8c6
Merge remote-tracking branch 'upstream/master' into detect-parallelism
ericl d88fcd0
update
ericl dad1569
fix from_items
ericl a7a6138
Merge remote-tracking branch 'upstream/master' into detect-parallelism
ericl 8dc156d
fix test
ericl af722b0
wip
ericl 04e7771
update
ericl 745105a
fix
ericl 277c89f
fix
ericl 0b0e967
fx
ericl 42c03b9
update
ericl 748f7e9
update
ericl a7123b3
fix
ericl 3b4277b
Merge remote-tracking branch 'upstream/master' into detect-parallelism
ericl 0f7630e
update
ericl 40a3b13
fix
ericl ee9073d
update
ericl 2ee5e04
fix
ericl 653fb01
Merge remote-tracking branch 'upstream/master' into detect-parallelism
ericl 902e404
fix
ericl e899426
fix
ericl e80b08a
improve doc
ericl 4bca8a3
update
ericl f651581
lint
ericl 558fb4b
update
ericl 2e39005
fix
ericl 1700f73
try out
ericl 59f3bc7
Merge remote-tracking branch 'upstream/master' into detect-parallelism
ericl 2d149cf
add warning
ericl 7a6a07b
fix
ericl 3a00261
Merge remote-tracking branch 'upstream/master' into detect-parallelism
ericl 20e09a8
fix test
ericl f57f5a8
fix
ericl 36f92cb
Merge remote-tracking branch 'upstream/master' into detect-parallelism
ericl c1986c8
try fixing
ericl 5048fbb
fix closure capture
ericl a5c6d85
stale comment
2544e84
lint
a488e9c
Merge branch 'master' of https://github.com/ray-project/ray into eric…
4297fcd
feedback
73a5802
Merge branch 'master' of https://github.com/ray-project/ray into eric…
e74b78d
Merge remote-tracking branch 'upstream/master' into detect-parallelism
ericl df476e2
Merge branch 'detect-parallelism' of github.com:ericl/ray into detect…
ericl b7eee3f
handle subclass
013a449
Merge branch 'master' of https://github.com/ray-project/ray into eric…
4556a15
Merge branch 'detect-parallelism' of github.com:ericl/ray into ericl-…
d6d4ce2
fix file read
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
lint
- Loading branch information
commit 7d46ef0317554ca46ce36a140a429b0eb749dc51
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We shouldn't capture
self
in this closure, since that's going to create serialization issues, e.g. theself.filesystem
deserialization might break since it's not using our wrapper. We should changeFileBasedDatasource._open_input_source()
to a static method (it's not currently using the instance anyway) and close over that function pointer instead ofself
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, done!