[data] Add iterator batch_format=None support, which will yield batches in the current batch format with zero copies #33562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
amogkam
approved these changes
Mar 21, 2023
| @@ -379,7 +379,7 @@ def map_batches( | |||
| *, | |||
| batch_size: Optional[Union[int, Literal["default"]]] = "default", | |||
| compute: Optional[Union[str, ComputeStrategy]] = None, | |||
| batch_format: Literal["default", "pandas", "pyarrow", "numpy"] = "default", | |||
| batch_format: Optional[str] = "default", | |||
There was a problem hiding this comment.
keep it as Optional[Literal] for full explicitness of supported batch formats?
There was a problem hiding this comment.
I feel like that is hard to maintain (per the inconsistencies in the code already), so opted to go unify on the shorter signature.
|
lets also keep the documentation changes from https://github.com/ray-project/ray/pull/33536/files#diff-988f3832ac94d085daf61260175e2580920ebd1521dc760f58b426b94379d5b7L235? |
8 tasks
Signed-off-by: Eric Liang <[email protected]>
Done |
clarkzinzow
approved these changes
Mar 22, 2023
| @@ -540,7 +540,9 @@ def map_batches( | |||
| (promotes tables to Pandas and tensors to NumPy), ``"pandas"`` to select | |||
| ``pandas.DataFrame``, "pyarrow" to select ``pyarrow.Table``, or | |||
| ``"numpy"`` to select ``numpy.ndarray`` for tensor datasets and | |||
| ``Dict[str, numpy.ndarray]`` for tabular datasets. Default is "default". | |||
| ``Dict[str, numpy.ndarray]`` for tabular datasets, or None to return | |||
| the underlying block exactly as is with no additional formatting. | |||
There was a problem hiding this comment.
Nice, I like batch_size=None a good bit more than adding another literal string!
8 tasks
ericl
pushed a commit
that referenced
this pull request
Mar 23, 2023
scottsun94
pushed a commit
to scottsun94/ray
that referenced
this pull request
Mar 28, 2023
…es in the current batch format with zero copies (ray-project#33562) This PR is a cleanup of ray-project#33536 It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format.
scottsun94
pushed a commit
to scottsun94/ray
that referenced
this pull request
Mar 28, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`.
cassidylaidlaw
pushed a commit
to cassidylaidlaw/ray
that referenced
this pull request
Mar 28, 2023
…es in the current batch format with zero copies (ray-project#33562) This PR is a cleanup of ray-project#33536 It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format.
cassidylaidlaw
pushed a commit
to cassidylaidlaw/ray
that referenced
this pull request
Mar 28, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`.
brycehuang30
pushed a commit
to brycehuang30/ray
that referenced
this pull request
Mar 29, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`. Signed-off-by: bhuang <[email protected]>
joncarter1
pushed a commit
to joncarter1/ray
that referenced
this pull request
Apr 2, 2023
…es in the current batch format with zero copies (ray-project#33562) This PR is a cleanup of ray-project#33536 It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format. Signed-off-by: Jonathan Carter <[email protected]>
joncarter1
pushed a commit
to joncarter1/ray
that referenced
this pull request
Apr 2, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`. Signed-off-by: Jonathan Carter <[email protected]>
elliottower
pushed a commit
to elliottower/ray
that referenced
this pull request
Apr 22, 2023
…es in the current batch format with zero copies (ray-project#33562) This PR is a cleanup of ray-project#33536 It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format. Signed-off-by: elliottower <[email protected]>
elliottower
pushed a commit
to elliottower/ray
that referenced
this pull request
Apr 22, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`. Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe
pushed a commit
to ProjectsByJackHe/ray
that referenced
this pull request
May 4, 2023
…es in the current batch format with zero copies (ray-project#33562) This PR is a cleanup of ray-project#33536 It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format. Signed-off-by: Jack He <[email protected]>
ProjectsByJackHe
pushed a commit
to ProjectsByJackHe/ray
that referenced
this pull request
May 4, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`. Signed-off-by: Jack He <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR is a cleanup of #33536
It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format.