Feature request : add delta cache

[djouallah]

I have being playing with delta table and python just for fun, but recently with the performance of duckdb, thing started to make more sense, and maybe even be useful in production environment, but I notice in order to get any good performance using cloud storage we need some form of local cache.

it would be really useful if Delta table cache is supported, that will be a killer feature.

[houqp]

Might be better to implement the caching at the query engine layer because delta-rs doesn't read any of the data files. There is definitely value in caching table metadata in delta-rs.

[djouallah]

"delta-rs doesn't read any of the data files" but it does know which file is needed by the Query ? correct ?

[wjones127]

I wonder if this could be implemented at the object-store level, where it would keep an LRU cache of files recently loaded. Though might not be useful if the query engine is always requesting specific ranges rather than whole files.

[djouallah]

@wjones127 all I can say, anything you do will have a massive impact, Object store throughput is problematic and any cache will make the experience way better !!! and it will be a competitive advantage to delta table !!!

[djouallah]

closing as it is really engine specific and not really delta table responsability