Kedro with minio

[noklam]

What are you trying to do?

Kedro support major cloud storage API. I would like to use self-host storage instead, I am thinking to use Minio (S3 Compatible API)
Does kedro support this?

https://min.io/

[921kiyo]

Hi! Kedro uses fsspec and s3fs to handle IO to cloud storage. Looks like s3fs supports minio, so you could pass it as an argument of fs_args in a dataset class. See how to specify it in catalog Hope this helps :)

[noklam]

@921kiyo
The configuration is a bit trickier, maybe this could be added to kedro documentations. To use s3fs, we have to use a filepath filepath :s3:https://path and an extra endpoint_url client kwargs.

test:
  type: pandas.CSVDataSet
  filepath: s3:https://kedro-test/test.csv
  credentials: dev_s3

for credentials.yml, a typical s3 credentials look like this

dev_s3:
  aws_access_key_id: token
  aws_secret_access_key: key

If we need to use Minio
We will configure like this instead.

dev_s3:
  key: key
  secret: secret
  client_kwargs:
    endpoint_url : 'https://localhost:9000' # default for minio

[921kiyo]

Thank you for the feeback, glad it worked :) We are more than welcome for your contribution :) I'm closing this issue, but, feel free to reopen it if you have any fullow-up question.

[noklam]

Sure, I can make a PR for adding this into documentation if this is useful.

[cupdike]

In the released docs: https://kedro.readthedocs.io/en/stable/data/data_catalog.html#codecell17