[data] Refactor all to all op implementations into a separate file #26585
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
ericl
commented
Jul 14, 2022
| @@ -194,6 +194,8 @@ def schema( | |||
| Returns: | |||
| The schema of the output dataset. | |||
| """ | |||
| from ray.data._internal.stage_impl import RandomizeBlocksStage | |||
There was a problem hiding this comment.
Circular import problems...
ericl
commented
Jul 14, 2022
| from ray.data import Dataset | ||
|
|
||
|
|
||
| class RepartitionStage(AllToAllStage): |
There was a problem hiding this comment.
All code moved verbatim except when commented.
ericl
commented
Jul 14, 2022
| context = DatasetContext.get_current() | ||
| if context.use_push_based_shuffle: | ||
| if output_num_blocks is not None: | ||
| raise NotImplementedError( |
There was a problem hiding this comment.
@stephanie-wang there was a bug where we ignored the num_blocks parameter for random_shuffle, FYI. It appears to crash push-based shuffle, so I disabled it for now.
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
jianoaix
approved these changes
Jul 15, 2022
|
|
||
| if TYPE_CHECKING: | ||
| from ray.data import Dataset | ||
|
|
There was a problem hiding this comment.
Nice refactoring. Shall we also pull out the one-to-one stages?
There was a problem hiding this comment.
I was going to do that later (since OneToOne stages are basically all the same, having them as explicit different classes is not that useful for optimization purposes).
truelegion47
pushed a commit
to truelegion47/ray
that referenced
this pull request
Jul 16, 2022
* master: (35 commits) [data] Refactor all to all op implementations into a separate file (ray-project#26585) [Datasets] Explicitly define Dataset-like APIs in DatasetPipeline class (ray-project#26394) [Serve][Part2] Migrate the tests to use deployment graph api (ray-project#26507) [Serve] Default to EveryNode when starting Serve from REST API (ray-project#26588) Revert "[KubeRay][Autoscaler][Core] Add a flag to disable ray status version check (ray-project#26584)" (ray-project#26597) [air] Add initial benchmark section (ray-project#26608) [Workflow] Remove workflow execution module (ray-project#26504) [air] Add xgboost release test for silver tier(10-node case). (ray-project#26460) Revert "Revert "[serve] Use soft constraint for pinning controller on head node (ray-project#25091)" (ray-project#25857)" (ray-project#25858) [RLlib] Fixes MARWIL release tests (ray-project#26586) [Datasets] Improve read_xxx experience of HTTP file (ray-project#26454) Cleanup ActorContext due to multi actor instances got removed. (ray-project#26497) Print newest_ckpt_path when resuming trial. (ray-project#26561) Fix test_serialization_error_message for pytest 6.x (ray-project#26591) [RLlib] Make DQN update_target use only trainable variables. (ray-project#25226) [RLlib] In env check, step only expected agents. (ray-project#26425) [RLlib] `restart_failed_sub_environments` now works for MA cases and crashes during `reset()`; +more tests and logging; add eval worker sub-env fault tolerance test. (ray-project#26276) [runtime env] plugin refactor[4/n]: remove runtime env protobuf (ray-project#26522) Improve streaming read performance for default configuration. (ray-project#26587) [Dashboard] Fix test dashboard flaky by catch an expected exception (ray-project#26555) ...
truelegion47
pushed a commit
to truelegion47/ray
that referenced
this pull request
Jul 16, 2022
…ay-project#26585) Signed-off-by: Your Name <[email protected]>
xwjiang2010
pushed a commit
to xwjiang2010/ray
that referenced
this pull request
Jul 19, 2022
…ay-project#26585) Signed-off-by: Xiaowei Jiang <[email protected]>
avnishn
pushed a commit
to smorad/ray
that referenced
this pull request
Jul 20, 2022
…ay-project#26585) Signed-off-by: Avnish <[email protected]>
klwuibm
pushed a commit
to yuanchi2807/ray
that referenced
this pull request
Jul 27, 2022
…ay-project#26585) Signed-off-by: klwuibm <[email protected]>
franklsf95
pushed a commit
to franklsf95/ray
that referenced
this pull request
Aug 2, 2022
…ay-project#26585) Signed-off-by: Frank Luan <[email protected]>
gramhagen
pushed a commit
to gramhagen/ray
that referenced
this pull request
Aug 15, 2022
…ay-project#26585) Signed-off-by: Scott Graham <[email protected]>
gramhagen
pushed a commit
to gramhagen/ray
that referenced
this pull request
Aug 15, 2022
Stefan-1313
pushed a commit
to Stefan-1313/ray_mod
that referenced
this pull request
Aug 18, 2022
…ay-project#26585) Signed-off-by: Stefan van der Kleij <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Pull these op implementations into separate classes for readability. This change also simplifies the implementation of #25708