-
Notifications
You must be signed in to change notification settings - Fork 689
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Async Arrow Reader #5021
Labels
development-process
Related to development process of arrow-rs
enhancement
Any new improvement worthy of a entry in the changelog
Comments
tustvold
added
the
enhancement
Any new improvement worthy of a entry in the changelog
label
Nov 1, 2023
There is related commentary on https://github.com/apache/arrow-datafusion/pull/7962/files#r1379219105 about how to look at the footer of arrow files to get metadata, etc |
FYI a duplicate/related issue exists: #1207 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
development-process
Related to development process of arrow-rs
enhancement
Any new improvement worthy of a entry in the changelog
This is largely a placeholder and would need further fleshing out if someone wishes to work on this
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Current we only support reading IPC files from a synchronous buffer. This limits the ability to read parts of files.
Describe the solution you'd like
I high-level implementation inspired by parquet might look like
This would then naturally lend itself to being wrapped up in an async interface similar to ParquetRecordBatchStream
Describe alternatives you've considered
Additional context
The text was updated successfully, but these errors were encountered: