[KED-1642] index option in pandas.ParquetDataSet

[juan-carlos-calvo]

Description

ParquetLocalDataSet accepts the index save option which specifies whether to save the index in parquet or not. pandas.ParquetDataSet doesn't have an equivalent option at the moment.

Context

As the datasets in kedro.io are going to be deprecated and moved to kedro.extras.datasets kedro should have backward compatibility with the load_args save_args options. Also, index is a useful option to have.

Possible Implementation

at the _save method of pandas.ParquetDataSet substitute:

table = pa.Table.from_pandas(data)

with

preserve_index = self._save_args.pop('index', False)
table = pa.Table.from_pandas(data, preserve_index=preserve_index)

[921kiyo]

@juan-carlos-calvo Thank you for opening the issue! This makes sense to me. I've logged this in our backlog and will fix it :)

[andrii-ivaniuk]

Thanks @juan-carlos-calvo for reporting this.
It was fixed in 5330b45 commit.

The arguments for from_pandas() should be passed through a nested key: from_pandas. E.g.: save_args = {"from_pandas": {"preserve_index": False}}