Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support partitioning on nested fields in Iceberg #15109

Closed
alexjo2144 opened this issue Nov 18, 2022 · 4 comments · Fixed by #15712
Closed

Support partitioning on nested fields in Iceberg #15109

alexjo2144 opened this issue Nov 18, 2022 · 4 comments · Fixed by #15712
Assignees
@krvikash
Labels
enhancement New feature or request

Comments

@alexjo2144
Copy link
Member

Follow up for #15091
Initial work was started here: #9337
@alexjo2144 alexjo2144 added the enhancement New feature or request label Nov 18, 2022
@ebyhr ebyhr self-assigned this Nov 22, 2022
@ebyhr
Copy link
Member

ebyhr commented Nov 25, 2022 •
edited
Loading

Spark can't read an ORC table that is partitioned by nested fields even when the table is created and rows are inserted by Spark. Also, Parquet table made by Trino requires special handling - the nested field on the partitioned column somehow becomes null in Spark.

Let me cancel to implement this enhancement. I will keep this ticket opened though.

@ebyhr ebyhr removed their assignment Nov 25, 2022
@findepi
Copy link
Member

findepi commented Nov 25, 2022

Spark can't read an ORC table that is partitioned by nested fields even when the table is created and rows are inserted by Spark.

@danielcweeks @rdblue is partitioning on nested fields supported by Iceberg?

@alexjo2144
Copy link
Member Author

Found a related Issue apache/iceberg#3139

@danielcweeks
Copy link
Contributor

Spark can't read an ORC table that is partitioned by nested fields even when the table is created and rows are inserted by Spark.

@danielcweeks @rdblue is partitioning on nested fields supported by Iceberg?

@findepi Yes, for structs (not maps or lists/arrays) per the spec

@krvikash krvikash self-assigned this Dec 5, 2022
@krvikash krvikash mentioned this issue Jan 13, 2023
Merged
@findepi findepi mentioned this issue Feb 6, 2023
Merged
@ebyhr ebyhr mentioned this issue Oct 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@krvikash krvikash

Labels
enhancement New feature or request
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support partitioning on nested ROW fields in Iceberg krvikash/trino
5 participants
@findepi @danielcweeks @ebyhr @alexjo2144 @krvikash