[Datasets] Autodetect dataset parallelism based on available resources and data size #25883
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ericl
requested review from
scv119,
clarkzinzow,
jjyao and
jianoaix
as code owners
ericl
changed the title
added 3 commits
July 13, 2022 00:07
…l-detect-parallelism
…detect-parallelism
|
test_tensors_shuffle failing |
6 tasks
c21
reviewed
Jul 13, 2022
| self._columns = columns | ||
| self._schema = schema | ||
|
|
||
| def estimate_inmemory_data_size(self) -> Optional[int]: |
There was a problem hiding this comment.
sorry for the late comment. but I think it's probably a bug here to rely on serialized_size, which seems to be size of file footer, not size of actual data. Crafted a quick fix in #26516 , please let me know if it makes sense or not, thanks.
6 tasks
ericl
added a commit
that referenced
this pull request
Jul 14, 2022
… for parallelism detection (#26543) In the previous PR #25883, a subtle regression was introduced in the case where data sizes blow up significantly. For example, suppose you're reading jpeg-image files from a Dataset, which increase in size substantially on decompression. On a small-core cluster (e.g., 4 cores), you end up with 4-8 blocks of ~200MiB each when reading a 1GiB dataset. This can blow up to OOM the node when decompressed (e.g., 25x size increase). Previously the heuristic to use parallelism=200 avoids this small-node problem. This PR avoids this issue by (1) raising the min parallelism back to 200. As an optimization, we also introduce the min block size threshold, which allows using fewer blocks if the data size is really small (<100KiB per block).
6 tasks
xwjiang2010
pushed a commit
to xwjiang2010/ray
that referenced
this pull request
Jul 19, 2022
… for parallelism detection (ray-project#26543) In the previous PR ray-project#25883, a subtle regression was introduced in the case where data sizes blow up significantly. For example, suppose you're reading jpeg-image files from a Dataset, which increase in size substantially on decompression. On a small-core cluster (e.g., 4 cores), you end up with 4-8 blocks of ~200MiB each when reading a 1GiB dataset. This can blow up to OOM the node when decompressed (e.g., 25x size increase). Previously the heuristic to use parallelism=200 avoids this small-node problem. This PR avoids this issue by (1) raising the min parallelism back to 200. As an optimization, we also introduce the min block size threshold, which allows using fewer blocks if the data size is really small (<100KiB per block). Signed-off-by: Xiaowei Jiang <[email protected]>
Stefan-1313
pushed a commit
to Stefan-1313/ray_mod
that referenced
this pull request
Aug 18, 2022
…s and data size (ray-project#25883) This PR defaults the parallelism of Dataset reads to `-1`. The parallelism is determined according to the following rule in this case: - The number of available CPUs is estimated. If in a placement group, the number of CPUs in the cluster is scaled by the size of the placement group compared to the cluster size. If not in a placement group, this is the number of CPUs in the cluster. If the estimated CPUs is less than 8, it is set to 8. - The parallelism is set to the estimated number of CPUs multiplied by 2. - The in-memory data size is estimated. If the parallelism would create in-memory blocks larger than the target block size (512MiB), the parallelism is increased until the blocks are < 512MiB in size. These rules fix two common user problems: 1. Insufficient parallelism in a large cluster, or too much parallelism on a small cluster. 2. Overly large block sizes leading to OOMs when processing a single block. TODO: - [x] Unit tests - [x] Docs update Supercedes part of: ray-project#25708 Co-authored-by: Ubuntu <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>
Stefan-1313
pushed a commit
to Stefan-1313/ray_mod
that referenced
this pull request
Aug 18, 2022
… for parallelism detection (ray-project#26543) In the previous PR ray-project#25883, a subtle regression was introduced in the case where data sizes blow up significantly. For example, suppose you're reading jpeg-image files from a Dataset, which increase in size substantially on decompression. On a small-core cluster (e.g., 4 cores), you end up with 4-8 blocks of ~200MiB each when reading a 1GiB dataset. This can blow up to OOM the node when decompressed (e.g., 25x size increase). Previously the heuristic to use parallelism=200 avoids this small-node problem. This PR avoids this issue by (1) raising the min parallelism back to 200. As an optimization, we also introduce the min block size threshold, which allows using fewer blocks if the data size is really small (<100KiB per block). Signed-off-by: Stefan van der Kleij <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR defaults the parallelism of Dataset reads to
-1. The parallelism is determined according to the following rule in this case:These rules fix two common user problems:
TODO:
Supercedes part of: #25708