Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strongly-typed reading of Parquet data #34

Closed
alamb opened this issue Apr 26, 2021 · 0 comments
Closed

Strongly-typed reading of Parquet data #34

alamb opened this issue Apr 26, 2021 · 0 comments
Labels
parquet Changes to the parquet crate

Comments

@alamb
Copy link
Contributor

alamb commented Apr 26, 2021

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-4314

See the proposal I made on [~csun]'s repository [here|https://github.com/sunchao/parquet-rs/issues/205] for more details.

This aims to let the user opt in to strong typing and substantial performance improvements (2x-7x, see [here|https://github.com/sunchao/parquet-rs/issues/205#issuecomment-446016254]) by optionally specifying the type of the records that they are iterating over.

It is currently a work in progress. All pre-existing tests succeed, bar those in src/record/api.rs which are commented out as they require reworking. Where relevant, pre-existing tests and benchmarks have been duplicated to make new strongly-typed tests and benchmarks, which all also succeed. I've tried to maintain pre-existing APIs where possible. Some changes have been made to better align with prior art in the Rust ecosystem.

Any feedback while I continue working on it very welcome! Looking forward to hopefully seeing this merged when it's ready.

@alamb alamb added arrow Changes to the arrow crate parquet Changes to the parquet crate and removed arrow Changes to the arrow crate labels Apr 26, 2021
@tustvold tustvold closed this as completed Nov 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
parquet Changes to the parquet crate
Projects
None yet
Development

No branches or pull requests

2 participants