parquet Row construct recordbatch

[smallzhongfeng]

Which part is this question about

Describe your question

Additional context

[smallzhongfeng]

For now, can we let parquet's row generate a reoctdBatch?

[tustvold]

I'm not sure I follow what you're requesting, there are native readers for reading parquet files as arrow RecordBatch?

https://docs.rs/parquet/latest/parquet/arrow/index.html#example-of-reading-parquet-file-into-arrow-record-batch

[smallzhongfeng]

Thanks for your reply ! What I mean is that the data of the parquet file is read out through rowIter, and then after certain processing, it is converted into recordBatch or written into the parquet file. Is this currently allowed? @tustvold

[tustvold]

This is not currently supported, and I would be hesitant to add support for this.

What is the motivation for using RowIter and not just doing the processing on arrow data? If you can describe the processing I can suggest how you might achieve this? Keeping the data in a columnar representation will yield orders of magnitude better performance

[smallzhongfeng]

For our business needs, we have made corresponding changes to datafusion. Now in order to meet some needs, we need to merge multiple parquet files. At the same time, we need to maintain order (through a certain field) after merging, so that order can be achieved. Read to increase data processing speed for our business needs.

[tustvold]

At the same time, we need to maintain order (through a certain field) after merging

This sounds like something SortPreservingMergeExec could handle for you?

[smallzhongfeng]

Another question:

Could SerializedFileReader read hdfs path? Doesn't seem to work. @tustvold

[smallzhongfeng]

At the same time, we need to maintain order (through a certain field) after merging

This sounds like something SortPreservingMergeExec could handle for you?

This really help me. thanks!

[tustvold]

Could SerializedFileReader read hdfs path? Doesn't seem to work.

You will likely want to use https://github.com/datafusion-contrib/datafusion-objectstore-hdfs

[smallzhongfeng]

I think not, I want to use like this https://docs.rs/parquet/latest/parquet/file/index.html#example-of-reading-an-existing-file, It seems not to support hdfs path, right?

[tustvold]

It won't support anything that isn't a local filesystem path, correct. If you wish to use a networked API such as HDFS you will need to provide an implementation of IO for this, or use one already implemented for you

[tustvold]

I'm going to close this as the question I believe has been answered, feel free to open a new ticket/discussion/get in touch on Discord if you have any further questions