Iris 3.3.1 fails unhelpfully when it finds a bad cell_methods attribute

[alistairsellar]

🐛 Bug Report

Loading a file produced by the ESA CCI project, it fails with IndexError: list index out of range (full traceback in additional context section). To be fair on Iris, the file contains a dodgy cell_methods attribute (see also in additional context) and fails cfchecker as a result. (Deleting the cell_methods attribute with ncatted prevents the error.)

The alleged bug is the unhelpful error message (though the traceback gives a clue as to where the issue is).

How To Reproduce

Steps to reproduce the behaviour:

1. iris.load('/data/users/esmval/ESMValTool/temporary/obs/Tier2/ESACCI-SOILMOISTURE/OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201612.nc')

(Not saying which system that path is on, but AVD team may be able to guess ;o). The same file can be found on JASMIN at /gws/nopw/j04/esmeval/obsdata-v2/Tier2/ESACCI-SOILMOISTURE/OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201612.nc)

Expected behaviour

Either return a cube (potentially with a warning about the CF non-compliance), or provide an informative error message.

Iris 3.2.1 returned a cube with no warning, and no cell_methods attribute, which was acceptable behaviour for me at least. Note that both Iris 3.2.1 and 3.3.1 happily ignore bad cell_measures and standard_name attributes in the same file.

Screenshots

Environment

• OS & Version: RHEL7
• Iris Version: 3.3.1 (note that Iris 3.2.1.post0 loaded the file without complaint)

Additional context

Traceback and partial netCDF header in details...

Click to expand this section...

netCDF header for offending variable:

        float sm(time, lat, lon) ;
                sm:standard_name = "" ;
                sm:units = "m3 m-3" ;
                sm:cell_methods = "time" ;
                sm:cell_measures = "area" ;
                sm:long_name = "Volumetric Moisture in Upper Portion of Soil Column" ;
                sm:comment = "the volume of water in all phases in a thin surface soil layer." ;
                sm:_FillValue = 1.e+20f ;

Traceback

>>> import iris
>>> cube = iris.load('OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201612.nc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/__init__.py", line 314, in load
    return _load_collection(uris, constraints, callback).merged().cubes()
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/__init__.py", line 280, in _load_collection
    result = _CubeFilterCollection.from_cubes(cubes, constraints)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/cube.py", line 104, in from_cubes
    for cube in cubes:
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/__init__.py", line 265, in _generate_cubes
    for cube in iris.io.load_files(part_names, callback, constraints):
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/io/__init__.py", line 208, in load_files
    for cube in handling_format_spec.handler(
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/netcdf.py", line 976, in load_cubes
    cube = _load_cube(engine, cf, cf_var, filename)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/netcdf.py", line 689, in _load_cube
    engine.activate()
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/engine.py", line 97, in activate
    run_actions(self)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 521, in run_actions
    action_default(engine)  # This should run the default rules.
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 74, in inner
    rule_name = func(engine, *args, **kwargs)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 90, in action_default
    hh.build_cube_metadata(engine)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/helpers.py", line 215, in build_cube_metadata
    cube.cell_methods = parse_cell_methods(nc_att_cell_methods)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/netcdf.py", line 294, in parse_cell_methods
    for m in _split_cell_methods(nc_cell_methods):
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/netcdf.py", line 253, in _split_cell_methods
    method_indices.append((name_start_inds[-1], len(nc_cell_methods)))
IndexError: list index out of range

[acchamber]

This also was the case with #5237, although with some AVD help we did manage to find the issue

[rcomer]

I can't reproduce the failure with v3.3.1, but the file does not seem to have been modified since January 😕

In [1]: import iris

In [2]: iris.load('/data/users/esmval/ESMValTool/temporary/obs/Tier2/ESACCI-SOILMOISTURE/OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201
   ...: 612.nc')
Out[2]: [<iris 'Cube' of Volumetric Moisture in Upper Portion of Soil Column / (m3 m-3) (time: 456; latitude: 720; longitude: 1440)>]

In [3]: iris.__version__
Out[3]: '3.3.1'

[alistairsellar]

Thanks for investigating @rcomer. I think that the file has been updated, but as a mirror of larger tree, so the file timestamp probably reflects the update time in the source of the mirror.

I've taken a copy and hacked the problem back into the header. New bad file is: /data/users/hadtq/esmvaltool/OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201612.nc. This file reproduces the error, for me at least.

Note that the fix to the source file means that this is a less pressing issue for me at least. Though I'd still argue that Iris should fail more gracefully in such cases, as a nice-to-have.

[bjlittle]

May be related to issue #5119 and addressed by pull-request #5126.

[bjlittle]

Discussed at @SciTools/peloton, when/if #5126 is banked we can see whether it also addresses this issue.

[rcomer]

Have confirmed locally that #5126 would fix this.