[EPIC] A collection of items to improve speed of parquet metadata encoding

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

There have been several recent assertions that Parquet is not suitable for handling wide tables with 1000s of columns

The rationale often goes something like wide tables have “large” metadata, which takes a “long time” to decode, often longer than reading the data itself.

This has led to several proposals for new file formats such as in BtrBlocks, Lance V2. Nimble, and recent discussions on the parquet mailing list.

However, there are several ways we can improve the performance of the existing thrift decoding in parquet-rs and this ticket captures several ideas of how to do so

• Report / blog on parquet metadata sizes for "large" (1000+) numbers of columns #5770
• Reduce Allocations When Reading Parquet Metadata #5775
• Use custom thrift decoder to improve speed of parsing parquet metadata #5854
• Selective decoding of a subset (e.g. columns or row groups) of parquet metadata #5855