fsspec url chaining?

[harm-matthias-harms]

I want to utilize ffspec's URL-chaining to cache downloaded files locally and avoid redundant downloads of the same file.

More details about URL-chaining: https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining

As of my current understanding, this approach is not feasible because the protocol checks are quite limited:

kedro/kedro/io/core.py

Line 729 in 5e15044

if protocol in HTTP_PROTOCOLS:

Is there an alternative method to leverage simplecache and similar options?

Here's the expected catalog entry:

infrastructure_osm_{state}_{layer}:
  type: geopandas.GeoJSONDataset
  load_args:
    layer: "{layer}"
  filepath: "simplecache::https://download.geofabrik.de/europe/germany/{state}-latest-free.shp.zip"

Possible datasets could include:

catalog.load("infrastructure_osm_thueringen_gis_osm_transport_free_1")
catalog.load("infrastructure_osm_thueringen_gis_osm_transport_a_free_1")

[datajoely]

So whilst I haven't tested this, I think this should work, we just need to provide

simplecache={'cache_storage': '/stored/csv/files'} (rewritten as YAML) directly to under the fs_args key:

https://docs.kedro.org/en/stable/data/data_catalog.html#load-and-save-arguments

[harm-matthias-harms]

Thank you for the advice. I tried it and it seems the full fs_args are passed to aiohttp, throwing an error. The problem also exists when using open_args_load.

infrastructure_osm_{state}_{layer}:
  type: geopandas.GeoJSONDataset
  load_args:
    layer: "{layer}"
  fs_args:
    simplecache:
      cache_storage: "./.cache/"
  filepath: "https://download.geofabrik.de/europe/germany/{state}-latest-free.shp.zip"

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /.../.venv/lib/python3.12/site-packages/fsspec/implementatio │
│ ns/http.py:422 in _info                                                                          │
│                                                                                                  │
│   419 │   │   for policy in ["head", "get"]:                                                     │
│   420 │   │   │   try:                                                                           │
│   421 │   │   │   │   info.update(                                                               │
│ ❱ 422 │   │   │   │   │   await _file_info(                                                      │
│   423 │   │   │   │   │   │   self.encode_url(url),                                              │
│   424 │   │   │   │   │   │   size_policy=policy,                                                │
│   425 │   │   │   │   │   │   session=session,                                                   │
│                                                                                                  │
│ /..../.venv/lib/python3.12/site-packages/fsspec/implementatio │
│ ns/http.py:832 in _file_info                                                                     │
│                                                                                                  │
│   829 │   if size_policy == "head":                                                              │
│   830 │   │   r = await session.head(url, allow_redirects=ar, **kwargs)                          │
│   831 │   elif size_policy == "get":                                                             │
│ ❱ 832 │   │   r = await session.get(url, allow_redirects=ar, **kwargs)                           │
│   833 │   else:                                                                                  │
│   834 │   │   raise TypeError(f'size_policy must be "head" or "get", got {size_policy}')         │
│   835 │   async with r:                                                                          │
│                                                                                                  │
│ /.../.venv/lib/python3.12/site-packages/aiohttp/client.py:97 │
│ 8 in get                                                                                         │
│                                                                                                  │
│    975 │   ) -> "_RequestContextManager":                                                        │
│    976 │   │   """Perform HTTP GET request."""                                                   │
│    977 │   │   return _RequestContextManager(                                                    │
│ ❱  978 │   │   │   self._request(hdrs.METH_GET, url, allow_redirects=allow_redirects, **kwargs)  │
│    979 │   │   )                                                                                 │
│    980 │                                                                                         │
│    981 │   def options(                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: ClientSession._request() got an unexpected keyword argument 'simplecache'

[datajoely]

Okay, we're getting somewhere. Can you work out which version of fsspec is on your system? It looks like this functionality was only added in Feb.

[harm-matthias-harms]

It should be the latest.

❯ pip list | grep fsspec
fsspec                    2024.6.1

Looks like fs_args.cache_type is also not helping. Also, because most of the caches are under implementations, the protocols, which are used in URL-chaining. https://github.com/fsspec/filesystem_spec/blob/3ff5fcaff6a4845cdd1985594120105fb35ceb51/fsspec/implementations/cached.py#L711

[datajoely]

Okay it looks like there is a wider issue we've not yet addressed - I'll try and bump up the priority
#1877

We should look to offload as much to fsspec as we can and avoid being opinionated.