fsspec url chaining? #4032
harm-matthias-harms
started this conversation in
Idea
Replies: 1 comment 4 replies
-
So whilst I haven't tested this, I think this should work, we just need to provide
![]() https://docs.kedro.org/en/stable/data/data_catalog.html#load-and-save-arguments |
Beta Was this translation helpful? Give feedback.
4 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I want to utilize ffspec's URL-chaining to cache downloaded files locally and avoid redundant downloads of the same file.
More details about URL-chaining: https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining
As of my current understanding, this approach is not feasible because the protocol checks are quite limited:
kedro/kedro/io/core.py
Line 729 in 5e15044
Is there an alternative method to leverage
simplecache
and similar options?Here's the expected catalog entry:
Possible datasets could include:
Beta Was this translation helpful? Give feedback.
All reactions