[Data] Remove `in_blocks` parameter of `ExecutionPlan` #45860

bveeramani · 2024-06-11T05:43:49Z

bveeramani/ray@change-execute-return...bveeramani:ray:remove-blocks-parameter

Stacked on

[Data] Make ExecutionPlan.execute return RefBundle instead of BlockList #45852
[Data] Refactor count() and schema() to not rely on LazyBlockList #45489

Why are these changes needed?

These changes are part of a larger effort to remove LazyBlockList.

Currently, when you create an ExecutionPlan, you need to pass in a BlockList. A BlockList can either be a regular BlockList which represents in-memory data or a LazyBlockList which represents ReadTasks. The issue is that there's no way to represent lazy input data that doesn't use ReadTasks.

To address this, this PR removes the in_blocks parameter of ExecutionPlan. It follows #45852.

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Balaji Veeramani <[email protected]>

Co-authored-by: Scott Lee <[email protected]> Signed-off-by: Balaji Veeramani <[email protected]>

Signed-off-by: Balaji Veeramani <[email protected]>

raulchen · 2024-06-12T18:01:30Z

python/ray/data/tests/test_consumption.py

+ #  "+- MapBatches(<lambda>)\n"
+ #  "+- Union\n"
+ #  " +- Dataset(num_rows=9, schema={id: int64})"
+ # )


will this be fixed?

Yeah, I'll address in a follow-up PR next week.

The current logic aggregates the metadata (i.e., num_rows and schema) from all the union operators; I think I'll also be able to improve this so that it lists each input dataset separately.

raulchen · 2024-06-12T18:11:58Z

python/ray/data/dataset.py

 datasets = [self] + list(other)
- has_nonlazy = any(not ds._plan.is_read_only() for ds in datasets)


nice getting rid of this logic.

Signed-off-by: Balaji Veeramani <[email protected]>

python/ray/data/_internal/logical/interfaces/operator.py

Signed-off-by: Balaji Veeramani <[email protected]>

After #45860, LazyBlockList is dead code. Signed-off-by: Balaji Veeramani <[email protected]>

After ray-project#45860, LazyBlockList is dead code. Signed-off-by: Balaji Veeramani <[email protected]>

…5860) These changes are part of a larger effort to remove LazyBlockList. Currently, when you create an ExecutionPlan, you need to pass in a BlockList. A BlockList can either be a regular BlockList which represents in-memory data or a LazyBlockList which represents ReadTasks. The issue is that there's no way to represent lazy input data that doesn't use ReadTasks. To address this, this PR removes the in_blocks parameter of ExecutionPlan. It follows ray-project#45852. Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Balaji Veeramani <[email protected]> Co-authored-by: Scott Lee <[email protected]> Signed-off-by: Saihajpreet Singh <[email protected]>

…5860) These changes are part of a larger effort to remove LazyBlockList. Currently, when you create an ExecutionPlan, you need to pass in a BlockList. A BlockList can either be a regular BlockList which represents in-memory data or a LazyBlockList which represents ReadTasks. The issue is that there's no way to represent lazy input data that doesn't use ReadTasks. To address this, this PR removes the in_blocks parameter of ExecutionPlan. It follows ray-project#45852. Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Balaji Veeramani <[email protected]> Co-authored-by: Scott Lee <[email protected]> Signed-off-by: 982945902 <[email protected]>

bveeramani added 30 commits May 21, 2024 21:12

Initial commit

1b9e20d

Signed-off-by: Balaji Veeramani <[email protected]>

Appease lint

3f8b692

Signed-off-by: Balaji Veeramani <[email protected]>

Re-add stuff

70f995e

Signed-off-by: Balaji Veeramani <[email protected]>

Fix typo

4ebc5b3

Signed-off-by: Balaji Veeramani <[email protected]>

Fix tests

daedc9e

Signed-off-by: Balaji Veeramani <[email protected]>

Fix bug

541970f

Signed-off-by: Balaji Veeramani <[email protected]>

Fix stuff

89f822c

Signed-off-by: Balaji Veeramani <[email protected]>

Merge branch 'master' into refactor-count

e479c9b

Signed-off-by: Balaji Veeramani <[email protected]>

Fix tests

7a707dd

Signed-off-by: Balaji Veeramani <[email protected]>

Appease lint

6ee1346

Signed-off-by: Balaji Veeramani <[email protected]>

Fix tests

57ffc6a

Signed-off-by: Balaji Veeramani <[email protected]>

Appease lint

4d010f2

Signed-off-by: Balaji Veeramani <[email protected]>

Appeas elint

fb44202

Signed-off-by: Balaji Veeramani <[email protected]>

Fix test

0b59b02

Signed-off-by: Balaji Veeramani <[email protected]>

Merge branch 'master' into refactor-count

7745c5e

Signed-off-by: Balaji Veeramani <[email protected]>

Fix tests

701a630

Signed-off-by: Balaji Veeramani <[email protected]>

Fix typo

69694a2

Signed-off-by: Balaji Veeramani <[email protected]>

Appease lint

6df829d

Signed-off-by: Balaji Veeramani <[email protected]>

Update stuff

f410d8b

Signed-off-by: Balaji Veeramani <[email protected]>

Fix tests

c6748ed

Signed-off-by: Balaji Veeramani <[email protected]>

Fix test

12a38fa

Signed-off-by: Balaji Veeramani <[email protected]>

Initial commit

3d76aa4

Signed-off-by: Balaji Veeramani <[email protected]>

Appease lint

5421955

Signed-off-by: Balaji Veeramani <[email protected]>

Fix logic

31cc9c0

Signed-off-by: Balaji Veeramani <[email protected]>

Fix some stuff

e67b450

Signed-off-by: Balaji Veeramani <[email protected]>

Update stuff

0f73d18

Signed-off-by: Balaji Veeramani <[email protected]>

Updates tuff

98b9716

Signed-off-by: Balaji Veeramani <[email protected]>

Merge stuff

15e11a0

Signed-off-by: Balaji Veeramani <[email protected]>

Update stuff

4f15924

Signed-off-by: Balaji Veeramani <[email protected]>

Fix stuff

f685adc

Signed-off-by: Balaji Veeramani <[email protected]>

bveeramani and others added 11 commits June 11, 2024 15:04

Merge branch 'refactor-count' into change-execute-return

3a386b7

Signed-off-by: Balaji Veeramani <[email protected]>

Fix test

f81812b

Signed-off-by: Balaji Veeramani <[email protected]>

Appease lint

b7b31e0

Signed-off-by: Balaji Veeramani <[email protected]>

Merge branch 'master' into change-execute-return

94c83a4

Signed-off-by: Balaji Veeramani <[email protected]>

Fix stuff

0608809

Signed-off-by: Balaji Veeramani <[email protected]>

Appease lint

cd215a8

Signed-off-by: Balaji Veeramani <[email protected]>

Update python/ray/data/_internal/plan.py

332cbaa

Co-authored-by: Scott Lee <[email protected]> Signed-off-by: Balaji Veeramani <[email protected]>

Address review comments

cea6feb

Signed-off-by: Balaji Veeramani <[email protected]>

Address review comments

d363e26

Signed-off-by: Balaji Veeramani <[email protected]>

Merge branch 'change-execute-return' into remove-blocks-parameter

68bcc9b

Signed-off-by: Balaji Veeramani <[email protected]>

Merge branch 'master' into remove-blocks-parameter

3dd1681

Signed-off-by: Balaji Veeramani <[email protected]>

raulchen reviewed Jun 12, 2024

View reviewed changes

raulchen approved these changes Jun 12, 2024

View reviewed changes

bveeramani added 4 commits June 12, 2024 15:39

Clean stuff up

6fe5ef6

Signed-off-by: Balaji Veeramani <[email protected]>

Clean more stuff

3e94a01

Signed-off-by: Balaji Veeramani <[email protected]>

Update stuff

c560832

Signed-off-by: Balaji Veeramani <[email protected]>

Clean stuff

da20476

Signed-off-by: Balaji Veeramani <[email protected]>

bveeramani assigned raulchen and scottjlee Jun 12, 2024

scottjlee approved these changes Jun 13, 2024

View reviewed changes

python/ray/data/_internal/logical/interfaces/operator.py Outdated Show resolved Hide resolved

bveeramani added 2 commits June 12, 2024 19:03

Address review comments

6abcef5

Signed-off-by: Balaji Veeramani <[email protected]>

Appease lint

09a0e6e

Signed-off-by: Balaji Veeramani <[email protected]>

bveeramani enabled auto-merge (squash) June 13, 2024 02:24

bveeramani merged commit 384f46c into ray-project:master Jun 13, 2024
7 checks passed

bveeramani mentioned this pull request Jun 17, 2024

[Data] Remove LazyBlockList #46054

Merged

8 tasks

bveeramani added a commit that referenced this pull request Jun 17, 2024

[Data] Remove LazyBlockList (#46054)

ab85dd2

After #45860, LazyBlockList is dead code. Signed-off-by: Balaji Veeramani <[email protected]>

chungen04 pushed a commit to chungen04/ray that referenced this pull request Jun 18, 2024

[Data] Remove LazyBlockList (ray-project#46054)

53be6ef

After ray-project#45860, LazyBlockList is dead code. Signed-off-by: Balaji Veeramani <[email protected]>

liuxsh9 mentioned this pull request Jun 25, 2024

[Data] Fix duplicate execution in dataset.union #45238

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data] Remove `in_blocks` parameter of `ExecutionPlan` #45860

[Data] Remove `in_blocks` parameter of `ExecutionPlan` #45860

bveeramani commented Jun 11, 2024 •

edited

Loading

raulchen Jun 12, 2024

bveeramani Jun 12, 2024

raulchen Jun 12, 2024

		datasets = [self] + list(other)
		has_nonlazy = any(not ds._plan.is_read_only() for ds in datasets)

[Data] Remove in_blocks parameter of ExecutionPlan #45860

[Data] Remove in_blocks parameter of ExecutionPlan #45860

Conversation

bveeramani commented Jun 11, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

raulchen Jun 12, 2024

Choose a reason for hiding this comment

bveeramani Jun 12, 2024

Choose a reason for hiding this comment

raulchen Jun 12, 2024

Choose a reason for hiding this comment

[Data] Remove `in_blocks` parameter of `ExecutionPlan` #45860

[Data] Remove `in_blocks` parameter of `ExecutionPlan` #45860

bveeramani commented Jun 11, 2024 •

edited

Loading