Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add objects GC in dataset iterator #34030

Merged
merged 16 commits into from
Apr 6, 2023
Merged

Add objects GC in dataset iterator #34030

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
925a247
Revert "[Datasets] Revert "Enable streaming executor by default (#324…
jianoaix Mar 22, 2023
b33ae23
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Mar 23, 2023
4ef5d35
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Mar 24, 2023
e6dcd6e
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Mar 24, 2023
482e9dc
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Mar 28, 2023
3e2d393
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Mar 29, 2023
cb0840c
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Apr 3, 2023
1696afb
Add objects GC in dataset iterator
jianoaix Apr 3, 2023
cec613c
test it
jianoaix Apr 4, 2023
bfc5f39
more tests
jianoaix Apr 4, 2023
ae6a6b8
fix comment
jianoaix Apr 4, 2023
abb9551
Merge branch 'master' of https://github.com/ray-project/ray into iter…
jianoaix Apr 5, 2023
017f14f
add a little more memory as it's close to the limit and may make test…
jianoaix Apr 5, 2023
b6cb319
Merge branch 'master' of https://github.com/ray-project/ray into iter…
jianoaix Apr 5, 2023
30eb6f5
feedback
jianoaix Apr 5, 2023
5445241
Merge branch 'master' of https://github.com/ray-project/ray into iter…
jianoaix Apr 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add a little more memory as it's close to the limit and may make test… 
… flaky
  • Loading branch information
@jianoaix
jianoaix committed Apr 5, 2023
commit 017f14ffe44d21fd61c58f66c370804ff572b4bf
4 changes: 2 additions & 2 deletions python/ray/data/tests/test_object_gc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ def test_iter_batches_no_spilling_upon_no_transformation(shutdown_only):


def test_torch_iteration(shutdown_only):
# The object store is about 300MB.
ctx = ray.init(num_cpus=1, object_store_memory=300e6)
# The object store is about 400MB.
ctx = ray.init(num_cpus=1, object_store_memory=400e6)
# The size of dataset is 500*(80*80*4)*8B, about 100MB.
ds = ray.data.range_tensor(500, shape=(80, 80, 4), parallelism=100)

Expand Down