Difference lat-lon maps only show half for several obs datasets

[chengzhuzhang]

In v2.9.0rc2, several lat-lon maps (bottom figure) only show half, including SST_HadISST, AOD_550 and OMI-MLS (@golaz's new run also shows this issue)

I saw the same issue when developing the GPCP v3.2 processing script. Rotating longitude from [-180,180) to [0, 360) resolved the issue. I suspect that a dependency library for visualization no longer auto consolidating longitude with [-180,180) or [0, 360).

[chengzhuzhang]

After some troubleshooting, it ruled out change in cartopy as the cause. Looking at the intermediate files being saved, data already missing half after regridding. It suggests regriding with changes in cdms/esmf is the cause. I tried to roll back to older version/builds of cdms and esmf, but ran into numpy, alias deprecation problems. Pinning for old version is not ideal for coordinating with e3sm-unified. For a immediate solution, it might make sense to convert longitude from [-180, 180) to [0, 360) online or offline for input dateset being impacted to work around this problem.

[chengzhuzhang]

Found an old cdms issue pointing out the same behavior.
CDAT/cdms#371
I'm following the approach suggested in this post to wrap the longitude with [0,360), since cdms somehow failed to do it upon open datasets.

[chengzhuzhang]

The regriding step also produce warning as follows:

/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290_numpy/lib/python3.10/site-packages/cdms2/avariable.py:1289: Warning:
avariable.regrid: regridTool = 'esmf' but your version does not
seems to be built with esmf, will switch to regridTool = 'libcf'

  warnings.warn(message, Warning)

It seems cdms is not properly built with esmf. Not sure if this is relevant to the regriding issue here.

[xylar]

@chengzhuzhang, I'm going to see if conda-forge/cdms2-feedstock#87 fixes this. I patched the imports in cdms2 to hopefully bring in esmpy, since the package has been renamed since v8.4.0 from ESMF to esmpy.

[chengzhuzhang]

@xylar I'm glad to report: your fix conda-forge/cdms2-feedstock#87 works, and thanks for adding a check for testing conda-forge/cdms2-feedstock#89.
With esmfy correctly imported by cdms, the regriding behavior gets back to normal.

[xylar]

@chengzhuzhang, could you add a test somewhere in E3SM diags where you test for ESMFRegrid? It could just be as simple as:

from regrid2 import ESMFRegrid

but you might want to have a try/except ImportError where you give an explanation about this meaning that an incompatible version of cdsm2 is installed that is not able to import ESMPy.

Such an error would tell us right away that there's a problem, rather than noticing it only when plots are half missing.

[chengzhuzhang]

Great suggestion. I think it makes sense to add it as one of the CI/CD tests.

[chengzhuzhang]

Fixed.