Passing NetCDF attributes into cube for CMIP data

[pdearnshaw]

CMIP6 has a specific template of attributes that are required in variables from NetCDF files. Iris currently pulls a few of these out, e.g. axis in coords and _FillValue or missing_value for the variable. These should not be removed and allowed to pass through to the cube so that any resulting cubes conform to CMIP6 data specifications.

[marqh]

hello @pdearnshaw

perhaps we could try to chat about how to handle this?

I believe that a coordinate can be selected from a cube by axis but I can't find the axis stored anywhere
Are you looking to be able to inspect or alter the axis attribute of a coordinate?

a _Fill_Value or missing_value is available from an entity from its masked array
is this sufficient for your needs for missing value metadata?

[pp-mo]

This may take some care to handle.
The missing_value and _Fill_Value attributes are handled by NetCDF4-python, in ways that are not always either useful, controllable or transparent (!).
From a quick look, I think that Iris will "consume" an 'axis' attribute as a part of its interpretation of the CF structure, and therefore recreates rather than preserves it, only as it sees fit. Which could probably be improved.

These should not be removed and allowed to pass through to the cube

I assume that here you mean that we need better control over what is saved by Iris
-- rather than the properties of an Iris cube in Python code ?

In the basic CF rules, virtually everything is optional, so we could maybe view this as adopting an 'extra convention' (like the MoNET conventions).
If no critical information is actually lost in loading, then it might actually be simpler and safer to design a transformation process that ensures CMIP6 compliance, e.g. by adding specific required properties.
Certainly, it could well be better overall to consider the requirements as a totality rather than attempting to patch odd pieces of undesirable behaviour.

Can you supply concrete examples of where the results are currently undesirable ?

[bouweandela]

A concrete example could be the following: the CMOR tables for both CMIP5 and CMIP6 specify an axis attribute for coordinates, see e.g. CMIP6_coordinate.json. Yet it is forbidden to add an axis attribute with iris, which makes it impossible to use iris to write data that complies with those CMOR tables.

Related issue #1404.

[trexfeathers]

Bump: we should either close this or act on it. If there's no more movement by 2022-08-31 I'll go ahead and close the issue.

[bouweandela]

Is there a reason why it is not allowed to set certain attributes on cubes/coordinates? Maybe because they are not necessary?

It might be useful to give this as input for the design of the data request for CMIP7.

[pp-mo]

Is there a reason why it is not allowed to set certain attributes on cubes/coordinates? Maybe because they are not necessary?

It might be useful to give this as input for the design of the data request for CMIP7.

It's simply because certain specific attributes have meanings defined by CF, and so are treated as "managed" by Iris.
Which means ... Iris takes charge of those values, and sets them as it sees fit on saving to netcdf.
Since Iris has its own view of what those attributes "ought" to be, we don't allow them to be set as user attributes since that might conflict.

The defining list is here
In this way we also limit the attributes of all Iris objects, which may in some cases be a bit excessive -- e.g. a cube cannot have a 'calendar' or 'climatology' attribute, even though that would really cause no trouble at all.

[HGWright]

@SciTools/peloton This is at minimum blocked by the Xarray bridge #4994. User intervention would also be needed.