Loading ERA5 data with CHUNK_CONTROL fails with TypeError

[schlunma]

🐛 Bug Report

Loading ERA5 data in netCDF format within a CHUNK_CONTROL.set(time=-1) context leads to a TypeError: 'NoneType' object does not support item assignment. Any coordinate that is present in the file (time, level, latitude, longitude) will give this error regardless of the value specified. No error is raised if coordinates are given that are not present in the file.

The file can be downloaded here (select "Monthly averaged reanalysis", "Temperature", all levels, "2022", all months", "00:00", "Whole available region").

How To Reproduce

import iris
from iris.fileformats.netcdf.loader import CHUNK_CONTROL
print("iris version", iris.__version__)
print()

path = 'adaptor.mars.internal-1710689764.140916-27680-14-21cccc98-2ce5-4fb3-b77d-e5a55df40a17.nc'

# Works
cube = iris.load_cube(path)
print(cube)
print(cube.lazy_data().chunksize)

# Fails
with CHUNK_CONTROL.set(time=-1):
    cube = iris.load_cube(path)

gives

iris version 3.8.1

air_temperature / (K)               (time: 12; pressure_level: 37; latitude: 721; longitude: 1440)
    Dimension coordinates:
        time                             x                   -             -               -
        pressure_level                   -                   x             -               -
        latitude                         -                   -             x               -
        longitude                        -                   -             -               x
    Attributes:
        Conventions                 'CF-1.6'
        history                     '2024-03-17 15:36:09 GMT by grib_to_netcdf-2.25.1: /opt/ecmwf/mars-client/bin/grib_to_netcdf.bin ...'
(1, 12, 721, 1440)

Traceback (most recent call last):
  File "/home/b/b309141/scripts/iris/chunk_control_era5.py", line 17, in <module>
    cube = iris.load_cube(path)
           ^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/iris/__init__.py", line 354, in load_cube
    cubes = _load_collection(uris, constraints, callback).cubes()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/iris/__init__.py", line 294, in _load_collection
    result = _CubeFilterCollection.from_cubes(cubes, constraints)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/iris/cube.py", line 97, in from_cubes
    for cube in cubes:
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/iris/__init__.py", line 275, in _generate_cubes
    for cube in iris.io.load_files(part_names, callback, constraints):
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/iris/io/__init__.py", line 219, in load_files
    for cube in handling_format_spec.handler(fnames, callback, constraints):
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/iris/fileformats/netcdf/loader.py", line 636, in load_cubes
    cube = _load_cube(engine, cf, cf_var, cf.filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/iris/fileformats/netcdf/loader.py", line 322, in _load_cube
    return _load_cube_inner(engine, cf, cf_var, filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/iris/fileformats/netcdf/loader.py", line 329, in _load_cube_inner
    data = _get_cf_var_data(cf_var, filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/iris/fileformats/netcdf/loader.py", line 277, in _get_cf_var_data
    chunks[i_dim] = cf_var.shape[i_dim]
    ~~~~~~^^^^^^^
TypeError: 'NoneType' object does not support item assignment

Expected behaviour

No error. The file should be loaded with a chunksize of (12, ...).

Environment

• OS & Version: Red Hat Enterprise Linux 8.7 (Ootpa)
• Iris Version: 3.8.1

[valeriupredoi]

@schlunma et iris folks - have a look at me comment here ESMValGroup/ESMValCore#2375 (comment) - that data is not storage (HDF5)-chunked, in fact it is not even compatible with HDF5/h5py since it's in NETCDF3 format, so it comes as a contiguous data array of shape (12, 37, 721, 1440), if iris is looking to determine storage chunks, and there are none, then it should fall back to a default synthetic chunk, and rechunk it that way, albeit that's a slow process