-
Notifications
You must be signed in to change notification settings - Fork 282
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cube merge loads NetCDF deferred auxiliary coordinates. #608
Comments
NB. Considering effort/payback, there could be a relatively easy half-way fix when each merge operation is only dealing with a single source cube. This is often the case when loading from netCDF, and is true for the user query which led to this issue. |
This would certainly be a nice to have. |
Pretty certain we have this capability now. Closing this out. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Iris implements deferred loading of auxiliary coordinates payload via the NetCDF loader via a
LazyArray
.This is in response to large multi-dimensional auxiliary coordinates causing a large memory footprint. Loading many such cubes can easily exhaust the available RAM for a process.
At the moment cube merge has no concept of such deferred auxiliary coordinates. As such, cube merge loads the payload of the deferred auxiliary coordinates, thus negating all benefit of such deferred loading.
The only workaround at the moment is to use
iris.load_raw
as this circumvents the merge process. This is a valid step to take in the majority of times, as NetCDF sourced cubes tend to be complete multi-dimensional entities i.e. non-mergeable.We require to extend cube merge to recognise such deferred auxiliary coordinates and not realise the payload.
The text was updated successfully, but these errors were encountered: