[REVIEW] Add parquet chunked writing ability for list columns

[devavret]

Closes #6530

Changes:

• Added a method of specifying the nullability of list columns. The API change is as follows: table_metadata_with_nullability.column_nullable[i] used to be the nullability of column[i]. Now it contains the flattened nullability of the table e.g. for a table of three columns, int, list<double>, float, the nullability vector contains the values:

Index	Nullability of
0	int column
1	Level 0 of list column (list itself)
2	Level 1 of list column (double values)
3	float column

• Modified the method of checking schema across write_chunk() calls. Now the entire schema vector is compared rather than just types.
• Fixed a bug introduced in list writing PR where a non-nested column following a list column would have the wrong value of definition bits. Now all such cases where the information was being queried from schema have been fixed to use parquet_column_view
• Fixed a regression introduced in a later commit in list writing PR while adding column_view with offset support to list columns. Changed pinned memory to normal pageable memory.
• Added missing tests for chunked writer where the nullability is mismatched across calls, or nullability is specified in first call.

[GPUtester]

Please update the changelog in order to start CI tests.

View the gpuCI docs here.

[codecov]

Codecov Report

Merging #6831 (0cc1fff) into branch-0.17 (a2d2726) will increase coverage by 0.04%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           branch-0.17    #6831      +/-   ##
===============================================
+ Coverage        81.94%   81.98%   +0.04%     
===============================================
  Files               96       96              
  Lines            16166    16181      +15     
===============================================
+ Hits             13247    13266      +19     
+ Misses            2919     2915       -4     

Impacted Files	Coverage Δ
python/cudf/cudf/core/column/string.py	86.30% <0.00%> (-0.35%)	⬇️
python/cudf/cudf/core/column/numerical.py	94.53% <0.00%> (+0.03%)	⬆️
python/cudf/cudf/core/column/categorical.py	93.37% <0.00%> (+0.03%)	⬆️
python/cudf/cudf/core/column/timedelta.py	89.53% <0.00%> (+0.08%)	⬆️
python/cudf/cudf/core/index.py	93.25% <0.00%> (+0.11%)	⬆️
python/cudf/cudf/core/column/datetime.py	88.55% <0.00%> (+0.11%)	⬆️
python/cudf/cudf/core/dataframe.py	91.14% <0.00%> (+0.15%)	⬆️
python/cudf/cudf/utils/dtypes.py	89.10% <0.00%> (+0.38%)	⬆️
python/cudf/cudf/utils/utils.py	85.35% <0.00%> (+1.10%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a2d2726...0cc1fff. Read the comment docs.

[vuule]

Looks great for the most part, got a bunch of minor suggestions.

[devavret]

• Added a method of specifying the nullability of list columns. The API change is as follows: table_metadata_with_nullability.column_nullable[i] used to be the nullability of column[i]. Now it contains the flattened nullability of the table

@jlowe @revans2 @nvdbaranec , Any objections to this change?

When there are no nested columns in the table, the behaviour stays the same.

[jlowe]

Any objections to this change?

This seems OK to me for now. It's backwards-compatible for the cases before nested types and accounts for expressing nullability if nested types are involved.

Eventually I would expect this to be replaced with an explicitly specified write schema as discussed in #6862

[devavret]

Eventually I would expect this to be replaced with an explicitly specified write schema as discussed in #6862

Great! I was about to ask there whether it would be ok for the input schema to take on the responsibility of prescribing nullability as well.

[harrism]

I don't understand all the parquet stuff, but the code looks clean, except for a potential race condition with async memcopies.

[kkraus14]

pytest LGTM