Limit Parquet Page Row Count By Default to reduce writer memory requirements with highly compressable columns

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

There is a discussion on the mailing list about default settings for page sizing, and one of the suggestions is that the page row count limit should be enabled by default, where currently it is not. Row groups are limited to 1M rows by default, there is some suggestion pages should be limited to 20,000.

Creating this issue to track

Describe the solution you'd like

Describe alternatives you've considered

Additional context

[alamb]

Maybe we can use some data from @XiangpengHao 's analysis in #5770 as part of this decision

[alamb]

Specifically I think @tustvold 's proposal is to change the default value of this

https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterProperties.html#method.data_page_row_count_limit

Currently it seems there is no row count limit (limit is usize::max):

arrow-rs/parquet/src/file/properties.rs

Line 356 in 5a24119

data_page_row_count_limit: usize::MAX,

Note that @XiangpengHao hit this when trying to write large numbers of rows: #5828

[alamb]

I think setting a default page size to 20k is a good idea -- the default "no limit" can result in substantial memory usage on the writer, that people seem to think is a limit of the parquet format itself.

Another potential default value per page would be 64K (the size used in BtrBlocks)

[alamb]

We just found another example in InfluxDB where setting the data page limit to 64k dramatically lowered the amount of memory written to the parquet writer

[alamb]

label_issue.py automatically added labels {'parquet'} from #5957