Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add multi-file support to dask_cudf.read_json #16057

Merged
merged 14 commits into from
Jul 15, 2024
Merged

Add multi-file support to dask_cudf.read_json #16057

merged 14 commits into from
Jul 15, 2024

Conversation

rjzamora
Copy link
Member

@rjzamora rjzamora commented Jun 18, 2024 •
edited
Loading

Description

Dask cuDF often benefits from a larger partition sizes than pandas-backed Dask DataFrame. This motivates the ability to easily "aggregate" multiple json files into each partition using dask_cudf.read_json. This PR introduces the aggregate_files argument (defaults to True) to make it easier to accomplish multi-file DataFrame partitions.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@rjzamora rjzamora added 2 - In Progress Currently a work in progress dask Dask issue improvement Improvement / enhancement to an existing function labels Jun 18, 2024
@rjzamora rjzamora self-assigned this Jun 18, 2024
@github-actions github-actions bot added the Python Affects Python cuDF API. label Jun 18, 2024
@rjzamora rjzamora added the non-breaking Non-breaking change label Jun 18, 2024
@rjzamora
Copy link
Member Author

cc @randerzander @ayushdg

@rjzamora rjzamora marked this pull request as ready for review July 5, 2024 19:34
@rjzamora rjzamora requested a review from a team as a code owner July 5, 2024 19:34
@rjzamora rjzamora changed the title [WIP] Add multi-file support to dask_cudf.read_json Add multi-file support to dask_cudf.read_json Jul 5, 2024
pentschev
pentschev reviewed Jul 9, 2024
View reviewed changes
Copy link
Member

@pentschev pentschev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments, overall it looks fine to me, but I'm not very knowledgeable of the dask-cudf codebase to offer the most insightful feedback.

python/dask_cudf/dask_cudf/backends.py Outdated Show resolved Hide resolved
python/dask_cudf/dask_cudf/io/tests/test_json.py Outdated Show resolved Hide resolved
python/dask_cudf/dask_cudf/io/json.py Outdated Show resolved Hide resolved
pentschev
pentschev approved these changes Jul 9, 2024
View reviewed changes
Copy link
Member

@pentschev pentschev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, once again I'm not very knowledgeable of dask-cudf so I'll leave it up to you @rjzamora to merge it or wait for other reviews. 🙂

@rjzamora rjzamora added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 2 - In Progress Currently a work in progress labels Jul 15, 2024
@rjzamora
Copy link
Member Author

/merge

@rapids-bot rapids-bot bot merged commit c4ee4a7 into rapidsai:branch-24.08 Jul 15, 2024
79 checks passed
@rjzamora rjzamora deleted the multi-file-json branch July 15, 2024 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pentschev pentschev pentschev approved these changes

Assignees

@rjzamora rjzamora

Labels
5 - Ready to Merge Testing and reviews complete, ready to merge dask Dask issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
cuDF/Dask/Numba/UCX
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@rjzamora @pentschev