Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Datasets] Streaming executor fixes #3 #32836

Merged
merged 56 commits into from
Feb 27, 2023
Merged
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
edc51bd
Fix read_tfrecords_benchmark nightly test
jianoaix Dec 8, 2022
61f4d6d
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 14, 2022
a33a943
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 16, 2022
36ebe52
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 16, 2022
ce6763e
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 19, 2022
0e2c29e
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 21, 2022
f2b6ed0
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 22, 2022
bb6c5c4
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 4, 2023
540fe79
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 10, 2023
edad7d0
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 10, 2023
60cc079
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 11, 2023
a3d3980
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 12, 2023
001579c
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 17, 2023
8aeed6c
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 18, 2023
7a9a49b
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 19, 2023
ef97167
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 20, 2023
6f0563c
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 21, 2023
bcec4d6
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 24, 2023
ddef4e5
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 25, 2023
fc9a175
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 25, 2023
f0e90b7
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 26, 2023
999d1de
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 26, 2023
d8159e3
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 30, 2023
d81cd02
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 31, 2023
bc831bb
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 31, 2023
c444395
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Feb 2, 2023
642da6f
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Feb 3, 2023
f713f2f
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Feb 6, 2023
d416a73
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Feb 9, 2023
da5acee
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Feb 10, 2023
ab64cb6
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Feb 10, 2023
7a5d5e3
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Feb 11, 2023
47010ca
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Feb 13, 2023
b9fcb5f
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Feb 13, 2023
bfd724d
Enable streaming executor by default
jianoaix Feb 13, 2023
9baa73a
Merge branch 'master' of https://github.com/ray-project/ray into enab…
jianoaix Feb 16, 2023
e1d0df5
current resource usage
jianoaix Feb 16, 2023
94a88ee
parquet test: num computed
jianoaix Feb 16, 2023
c670a3b
resource limits
jianoaix Feb 16, 2023
2c608f4
Merge branch 'master' of https://github.com/ray-project/ray into enab…
jianoaix Feb 22, 2023
c5a6aa9
merge
jianoaix Feb 22, 2023
6c82ad2
fix treating numpy as block format
jianoaix Feb 22, 2023
4cbcbae
fix issues related to default dataset_format and partial execution in…
jianoaix Feb 22, 2023
82bc7a7
fix incorrect ownership resolution from bundle iter to blocklist
jianoaix Feb 22, 2023
0c3ac20
Merge branch 'master' of https://github.com/ray-project/ray into enab…
jianoaix Feb 23, 2023
548f5cf
rewind the plan when clear snapshot
jianoaix Feb 24, 2023
aae0e54
propagate the remote args
jianoaix Feb 24, 2023
2f51cc5
pandasrow v.s. pyarrowrow
jianoaix Feb 24, 2023
69328cb
fix stats
jianoaix Feb 24, 2023
59ec39b
fix arrowrow v.s. pandasrow regarding column ordering
jianoaix Feb 25, 2023
322b353
lint
jianoaix Feb 25, 2023
e169caf
unset streaming
jianoaix Feb 25, 2023
85aee0b
fix
jianoaix Feb 25, 2023
e68d1ff
mark streaming CI as stable
jianoaix Feb 25, 2023
0ce558d
Merge branch 'master' of https://github.com/ray-project/ray into stre…
jianoaix Feb 25, 2023
476aac7
fix lint
jianoaix Feb 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Enable streaming executor by default
  • Loading branch information
jianoaix committed Feb 13, 2023
commit bfd724d8ed3afa5c40a8dd17d85f6aee783ab0b1
2 changes: 1 addition & 1 deletion python/ray/data/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@
# Whether to use the streaming executor. This only has an effect if the new execution
# backend is enabled.
DEFAULT_USE_STREAMING_EXECUTOR = bool(
int(os.environ.get("RAY_DATASET_USE_STREAMING_EXECUTOR", "0"))
int(os.environ.get("RAY_DATASET_USE_STREAMING_EXECUTOR", "1"))
)

# Whether to eagerly free memory (new backend only).
Expand Down