-
Notifications
You must be signed in to change notification settings - Fork 368
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request : add delta cache #769
Comments
Might be better to implement the caching at the query engine layer because delta-rs doesn't read any of the data files. There is definitely value in caching table metadata in delta-rs. |
"delta-rs doesn't read any of the data files" but it does know which file is needed by the Query ? correct ? |
I wonder if this could be implemented at the object-store level, where it would keep an LRU cache of files recently loaded. Though might not be useful if the query engine is always requesting specific ranges rather than whole files. |
@wjones127 all I can say, anything you do will have a massive impact, Object store throughput is problematic and any cache will make the experience way better !!! and it will be a competitive advantage to delta table !!! |
closing as it is really engine specific and not really delta table responsability |
I have being playing with delta table and python just for fun, but recently with the performance of duckdb, thing started to make more sense, and maybe even be useful in production environment, but I notice in order to get any good performance using cloud storage we need some form of local cache.
it would be really useful if Delta table cache is supported, that will be a killer feature.
The text was updated successfully, but these errors were encountered: