-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Read from parquet does not work #587
Comments
Hi! |
I changed the Python script with additional compression algo specification like compression = "GZIP" | "SNAPPY" | "BROTLI"
pq.write_table(table, destpth, use_dictionary=False, compression=compression) with readMatrix and all 3 compression algos I get the same error:
|
The Snappy compressed sample parquet files from the link above can be read without error by DAPHNE if I compile Arrow with Snappy support. I'll incorporate all Arrow supported compression formats in the next Docker image updates. |
Added all compression options to the Arrow compilation. This solves parts of the problems described in #587. Reading certain parquet files now does not fail right away due to lack of compression formats support. Parsing correctly is still an issue there.
Hi @corepointer With the new Docker image I am able to read parquet files. Tested for snappy, brotli and gzip. Thank you! Should we close this issue and open a new one for requesting readFrame() on parquet or remain this open? KR |
As I mentioned in the commit message of c1100d8, this is just a partial fix as the parquet reader seems to be quite limited at the moment. |
I just tried a larger file now, and unfortunately it does not properly, at the end of the matrix there are a lot of nan's then instead of the values |
It is very strange: sometimes it works properly, sometimes not |
Parquet file is existing and can be read with https://parquet-viewer-online.com
Execution of DSL script results in printing a frame with all (10) values nan
OUTPUT:
This is how I created the parquet file:
from a csv file which looks like this:
Sample parquet files can be acquired here (data is much more complex here): https://github.com/kaysush/sample-parquet-files/tree/main
The text was updated successfully, but these errors were encountered: