Add some sanity checks for memory use in dataset #22642
Conversation
|
|
||
|
|
||
| def test_memory_release_lazy_shuffle(shutdown_only): | ||
| # TODO(ekl) why is this flaky? Due to eviction delay? |
There was a problem hiding this comment.
FYI @stephanie-wang , not sure if you also observed this in your shuffle experiments (memory sometimes not getting released in a timely manner during shuffle)
There was a problem hiding this comment.
Hmm I think @franklsf95 actually did see something like this.
There was a problem hiding this comment.
It's quite odd. I don't think it's related to Python GC or anything. I figured it might be a unit-test artifact since we have some batching/delay before releasing memory, which would only show up in a fast test, but it could also be an issue at larger scale.
| error = e | ||
| print("Failed", e) | ||
| finally: | ||
| ray.shutdown() |
There was a problem hiding this comment.
will this cause double shutdown since you also have shutdown_only?
There was a problem hiding this comment.
Yea; though it is harmless.
Why are these changes needed?
Add some tests on upper bounding memory usage. One is disabled for now until we are able to clear input blocks for normal Datasets.