Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write parquet statistics for IntervalDayTimeArray , IntervalMonthDayNanoArray and IntervalYearMonthArray #5847

Closed
alamb opened this issue Jun 5, 2024 · 4 comments
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@alamb
Copy link
Contributor

alamb commented Jun 5, 2024

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

As we worked on filling out support for converting parquet statistics into ArrayRefs in DataFusion (see apache/datafusion#10453) @marvinlanhenke discovered that

Describe the solution you'd like
Support writing the following types of data with statistics in the parquet ArrowWriter

This might be the location:

_ => return Err(ParquetError::NYI(

Describe alternatives you've considered

Additional context

apache/datafusion#10801 (comment)

@alamb alamb added enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate labels Jun 5, 2024
@marvinlanhenke
Copy link

for additional context and possibly code pointers where Interval support is missing.

@alamb
Copy link
Contributor Author

alamb commented Jun 6, 2024

@marvinlanhenke also points out that to completely support this ticket probably requires completing #5849 first

@tustvold
Copy link
Contributor

It is incorrect to write statistics for interval types, as per the format specification - https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#interval

@alamb
Copy link
Contributor Author

alamb commented Jun 11, 2024

That makes sense. Thank you @tustvold

@alamb alamb closed this as completed Jun 11, 2024
@alamb alamb reopened this Jun 11, 2024
@alamb alamb closed this as not planned Won't fix, can't repro, duplicate, stale Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Development

No branches or pull requests

3 participants