Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Datasets] Streaming executor fixes #3 #32836

Merged
merged 56 commits into from
Feb 27, 2023
Merged

[Datasets] Streaming executor fixes #3 #32836

merged 56 commits into from
Feb 27, 2023

Conversation

jianoaix
Copy link
Contributor

@jianoaix jianoaix commented Feb 25, 2023 •
edited
Loading

Why are these changes needed?

  • Read tasks: The remote args for read tasks are not propagated to streaming executor
  • Execution plan: Run the execution plan is disruptive to the plan in that it will clear snapshot blocks; this is fine, but we should also reset the plan, so that it's re-executable when it has to run again
  • Stats: Add in streaming executor's stats
  • The dataset_format: this is causing different formats of results due to the difference in arrow and pandas

This should be the last batch of fixes and the streaming executor CI should be green from here.

#32132

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@jianoaix
Fix read_tfrecords_benchmark nightly test
edc51bd 
Signed-off-by: jianoaix <[email protected]>
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
61f4d6d
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
a33a943
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
36ebe52
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
ce6763e
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
0e2c29e
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
f2b6ed0
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
bb6c5c4
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
540fe79
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
edad7d0
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
60cc079
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
a3d3980
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
001579c
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
8aeed6c
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
7a9a49b
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
ef97167
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
6f0563c
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
bcec4d6
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
ddef4e5
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
fc9a175
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
f0e90b7
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
999d1de
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
d8159e3
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
d81cd02
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
bc831bb
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
c444395
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
642da6f
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
f713f2f
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
d416a73
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
da5acee
@jianoaix jianoaix changed the title Streaming executor fixes #3 [Datasets] Streaming executor fixes #3 Feb 25, 2023
@ericl ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Feb 25, 2023
@jianoaix jianoaix removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Feb 25, 2023
@ericl
Copy link
Contributor

ericl commented Feb 26, 2023

Lint and other test issues

@ericl ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Feb 26, 2023
@jianoaix
Copy link
Contributor Author

The lint fixed. The failures are new and just introduced by #32795.

@ericl ericl merged commit 4cc3a53 into ray-project:master Feb 27, 2023
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request Mar 21, 2023
@jianoaix @ProjectsByJackHe
[Datasets] Streaming executor fixes ray-project#3 (ray-project#32836)
916e318 
Signed-off-by: Jack He <[email protected]>
cadedaniel pushed a commit to cadedaniel/ray that referenced this pull request Mar 22, 2023
@jianoaix @cadedaniel
[Datasets] Streaming executor fixes #3 (ray-project#32836)
9d8f50e
edoakes pushed a commit to edoakes/ray that referenced this pull request Mar 22, 2023
@jianoaix @edoakes
[Datasets] Streaming executor fixes #3 (ray-project#32836)
0145b01 
Signed-off-by: Edward Oakes <[email protected]>
peytondmurray pushed a commit to peytondmurray/ray that referenced this pull request Mar 22, 2023
@jianoaix @peytondmurray
[Datasets] Streaming executor fixes #3 (ray-project#32836)
c9d4a41
scottsun94 pushed a commit to scottsun94/ray that referenced this pull request Mar 28, 2023
@jianoaix @scottsun94
[Datasets] Streaming executor fixes ray-project#3 (ray-project#32836)
6466e23
cassidylaidlaw pushed a commit to cassidylaidlaw/ray that referenced this pull request Mar 28, 2023
@jianoaix @cassidylaidlaw
[Datasets] Streaming executor fixes ray-project#3 (ray-project#32836)
1a3b95b
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
[Datasets] Streaming executor fixes ray-project#3 (ray-project#32836)
903211e 
Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
[Datasets] Streaming executor fixes ray-project#3 (ray-project#32836)
61d2cfd 
Signed-off-by: Jack He <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericl ericl ericl approved these changes

@scv119 scv119 Awaiting requested review from scv119 scv119 is a code owner

@clarkzinzow clarkzinzow Awaiting requested review from clarkzinzow

@jjyao jjyao Awaiting requested review from jjyao

@c21 c21 Awaiting requested review from c21 c21 is a code owner

Assignees

@ericl ericl

@c21 c21

@clarkzinzow clarkzinzow

Labels
@author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@jianoaix @ericl @c21 @clarkzinzow