upgrade pyarrow from 12 to 14

[severo]

We temporarily stop supporting array features until datasets fixes the issue on their side.

[albertvillanova]

Thank you. I think you forgot to update it in some of the components: api, rows,...

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (bc8bdcb) 90.49% compared to head (e26ac1c) 87.05%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2089      +/-   ##
==========================================
- Coverage   90.49%   87.05%   -3.45%     
==========================================
  Files         248      132     -116     
  Lines       15202     5245    -9957     
==========================================
- Hits        13757     4566    -9191     
+ Misses       1445      679     -766     

Flag	Coverage Δ
jobs_cache_maintenance	95.30% <ø> (ø)
jobs_mongodb_migration	86.69% <ø> (ø)
libs_libapi	83.37% <ø> (?)
libs_libcommon	?
services_admin	87.63% <ø> (ø)
services_api	87.09% <ø> (ø)
services_rows	85.49% <ø> (ø)
services_search	80.48% <ø> (ø)
services_sse-api	94.21% <ø> (ø)
services_worker	?

Flags with carried forward coverage won't be shown. Click here to find out more.

see 142 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[severo]

I think you forgot to update it in some of the components: api, rows,...

in fact, they were requiring pyarrow 11, which is why it was stuck to a lower version.

But... it seems like the upgrade will not be that simple: https://github.com/huggingface/datasets-server/actions/runs/6823050840/job/18556219132

Looking at it

[albertvillanova]

But... it seems like the upgrade will not be that simple

I see... I'm having a look at datasets side...

EDIT:
I wrongly understood that it may be caused by datasets, but that is not the case: in datasets we already support pyarrow-14 (I did the PR 😅 ):

• Support pyarrow 14.0.0 datasets#6378

[albertvillanova]

I see, the issue is in our viewer_utils...

[severo]

yes, to reproduce:

import numpy as np
from datasets import (Array2D, Dataset, Features)
feature_type = Array2D(shape=(2, 2), dtype="float32")
content = np.zeros((2, 2), dtype="float32")
features = Features({"col": feature_type})
dataset = Dataset.from_dict({"col": [content]}, features=features)

same if we create the content with

content = [[0.0, 0.0], [0.0, 0.0]]

The trace is:

/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py:648: FutureWarning: pyarrow.PyExtensionType is deprecated and will refuse deserialization by default. Instead, please derive from pyarrow.ExtensionType and implement your own serialization mechanism.
  pa.PyExtensionType.__init__(self, self.storage_dtype)
/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py:1661: RuntimeWarning: pickle-based deserialization of pyarrow.PyExtensionType subclasses is disabled by default; if you only ingest trusted data files, you may re-enable this using `pyarrow.PyExtensionType.set_auto_load(True)`.
In the future, Python-defined extension subclasses should derive from pyarrow.ExtensionType (not pyarrow.PyExtensionType) and implement their own serialization mechanism.

  obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py:1661: FutureWarning: pyarrow.PyExtensionType is deprecated and will refuse deserialization by default. Instead, please derive from pyarrow.ExtensionType and implement your own serialization mechanism.
  obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 924, in from_dict
    return cls(pa_table, info=info, split=split)
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 693, in __init__
    inferred_features = Features.from_arrow_schema(arrow_table.schema)
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1661, in from_arrow_schema
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1661, in <dictcomp>
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1381, in generate_from_arrow_type
    return Value(dtype=_arrow_to_datasets_dtype(pa_type))
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 111, in _arrow_to_datasets_dtype
    raise ValueError(f"Arrow type {arrow_type} does not have a datasets dtype equivalent.")
ValueError: Arrow type extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>> does not have a datasets dtype equivalent.

[severo]

Should I open an issue in datasets? Or am I misusing it? It's a bit too cryptic for me

[albertvillanova]

Should I open an issue in datasets?

Yes, I understand now: they have disabled their PyExtensionType and we use it in datasets for arrays...

[severo]

OK, I opened huggingface/datasets#6396

[severo]

Once merged, we'll have to revert 0f58abb if #2090 is merged before.

[lhoestq]

Thanks for the reactivity !
I'm fine with disabling ArrayND types for now using 14.0.1

[severo]

I'm fine with disabling ArrayND types for now using 14.0.1

what does it mean? letting the error propagate, and remove it from the tests?

[lhoestq]

Let's comment the failing tests for now. I guess pyarrow will ship a fix that introduces safe extension arrays soon

[lhoestq]

(or maybe they won't, but in this case datasets should stop using PyExtensionType, let's discuss in the datasets issue)

[albertvillanova]

So we are no longer supporting datasets with arrays features...

[lhoestq]

Yes until datasets fixes it

[severo]

Exactly. I think that it's better to remove support for array features than to let the vulnerability.

[severo]

Thanks for the comments and reviews!