Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Data] Remove old stages optimizer path #41747

Merged
merged 10 commits into from
Dec 28, 2023
Merged

[Data] Remove old stages optimizer path #41747

merged 10 commits into from
Dec 28, 2023

Conversation

scottjlee
Copy link
Contributor

@scottjlee scottjlee commented Dec 8, 2023 •
edited
Loading

Why are these changes needed?

Remove code for the old stages optimizer path (ExecutionPlan._optimize()), as well as related methods and tests. This should all be dead code, now that the streaming executor is enabled by default.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@scottjlee scottjlee changed the title [Data] WIP - Deprecate Stages codepath [Data] Remove old stages optimizer path Dec 14, 2023
@scottjlee scottjlee marked this pull request as ready for review December 14, 2023 19:12
@Zandew Zandew mentioned this pull request Dec 21, 2023
Merged
8 tasks
c21
c21 approved these changes Dec 27, 2023
View reviewed changes
Copy link
Contributor

@c21 c21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally!

python/ray/data/_internal/execution/legacy_compat.py Outdated
@@ -194,7 +178,7 @@ def _get_execution_dag(
dag = get_execution_plan(plan._logical_plan).dag
stats = _get_initial_stats_from_plan(plan)
else:
dag, stats = _to_operator_dag(plan, allow_clear_input_blocks)
raise DeprecationWarning("Execution optimizer should be enabled.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can just remove this else code path.

python/ray/data/_internal/plan.py Outdated
@@ -647,30 +640,11 @@ def execute(
blocks._owned_by_consumer = False

else:
blocks, stats, stages = self._optimize()
raise DeprecationWarning(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

@scottjlee scottjlee requested a review from c21 December 28, 2023 02:08
@c21 c21 merged commit 7c818f2 into ray-project:master Dec 28, 2023
9 checks passed
raulchen pushed a commit that referenced this pull request Jan 12, 2024
@Zandew @scottjlee
[data] remove stages before/after snapshot (#42004)
1af39a5 
Removes `_stages_before/after_snapshot` from `ExecutionPlan`.

This should be merged after #41747 and #41544
---------

Signed-off-by: Andrew Xue <[email protected]>
Signed-off-by: Scott Lee <[email protected]>
Co-authored-by: Scott Lee <[email protected]>
vickytsang pushed a commit to ROCm/ray that referenced this pull request Jan 12, 2024
@scottjlee @vickytsang
[Data] Remove old stages optimizer path (ray-project#41747)
d132fc4 
Remove code for the old stages optimizer path (`ExecutionPlan._optimize()`), as well as related methods and tests. This should all be dead code, now that the streaming executor is enabled by default.

Signed-off-by: Scott Lee <[email protected]>
vickytsang pushed a commit to ROCm/ray that referenced this pull request Jan 12, 2024
@Zandew @scottjlee @vickytsang
[data] remove stages before/after snapshot (ray-project#42004)
dd5c8f5 
Removes `_stages_before/after_snapshot` from `ExecutionPlan`.

This should be merged after ray-project#41747 and ray-project#41544
---------

Signed-off-by: Andrew Xue <[email protected]>
Signed-off-by: Scott Lee <[email protected]>
Co-authored-by: Scott Lee <[email protected]>
yash97 pushed a commit to yash97/ray that referenced this pull request Jan 27, 2024
@scottjlee @yash97
[Data] Remove old stages optimizer path (ray-project#41747)
3b40b19 
Remove code for the old stages optimizer path (`ExecutionPlan._optimize()`), as well as related methods and tests. This should all be dead code, now that the streaming executor is enabled by default.

Signed-off-by: Scott Lee <[email protected]>
Signed-off-by: yash97 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericl ericl Awaiting requested review from ericl

@scv119 scv119 Awaiting requested review from scv119

@amogkam amogkam Awaiting requested review from amogkam

@bveeramani bveeramani Awaiting requested review from bveeramani bveeramani is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen is a code owner

@stephanie-wang stephanie-wang Awaiting requested review from stephanie-wang stephanie-wang is a code owner

@Zandew Zandew Awaiting requested review from Zandew

@c21 c21 Awaiting requested review from c21

Assignees

@raulchen raulchen

@c21 c21

@Zandew Zandew

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@scottjlee @c21 @raulchen @Zandew