Any ideas on why there are large differences when regridding E3SM land sea mask with xcdat xESMF compared to cdms2 ESMF? #521

tomvothecoder · 2023-07-26T18:58:30Z

tomvothecoder
Jul 26, 2023
Maintainer

Question criteria

I added a descriptive title here.
I searched the xCDAT GitHub Discussions to find a similar question and didn't find it.
I searched the xCDAT documentation and/or Xarray documentation.

Describe your question

Overview

In e3sm_diags I am replacing the cdms2 horizontal regridder API (ESMF, conservative) with the xcdat horizontal regridder API (xESMF, conservative). The horizontal regridder is used to regrid a land sea mask to a variable's grid before it is applied on the variable.

The Problem

The regridded land sea mask produced by xcdat compared to cdms2 appear to be way off. Since the regridded land sea masks are off, the masked variables are also off. Are there any ideas on why this might be happening?

Note, I regridded other variables and the results are relatively close between both regridders. I mainly run into this problem when regridding the land sea mask.

Implementation Logic

Input variable file: cl_Amon_E3SM-2-0_historical_r1i1p1f1_gr_200001-201412.nc
Input land sea mask: acme_ne30_ocean_land_mask.nc

Regrid an E3SM land sea mask to the grid of an input variable (e.g., "cl")
- Mask grid: 128 lat x 256 lon
- Variable "cl" grid: 180 lat x 360 lon
Update the regridded land sea mask with a lower limit
- All values below the lower limit are masked
- The lower limit is the land and ocean region value (e.g., 0.65)
Apply the land sea mask to the input variable

Test Cases (from MCVE section below)

# 1. COMPARE GRIDS -- identical (PASSING)
np.testing.assert_equal(output_grid_xc.lat.data, output_grid_cd.getAxis(0))
np.testing.assert_equal(output_grid_xc.lon.data, output_grid_cd.getAxis(1))

# 2. COMPARE INPUT MASKS -- the same (PASSING)
np.testing.assert_equal(ds_mask_xr["LANDFRAC"].data, mask_tvar.data)

# 3. COMPARE REGRIDDED MASKS -- not equal, regridding results in differences (FAILING)
# FIXME: This test case is failing
np.testing.assert_equal(dv_mas_var, mask_tvar_regrid.data)

# AssertionError:
# Arrays are not equal

# Mismatched elements: 18426 / 64800 (28.4%)
# Max absolute difference: 0.15945476
# Max relative difference: 1.
#  x: array([[[1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],...
#  y: array([[[1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],...

# 4. COMPARE OUTPUT MASK AVERAGES -- identical (PASSING)
mask_tvar_regrid.mean() == dv_mas_var.mean()

# 5. COMPARE REGRIDDED MASKS WITH LOWER LIMIT -- not equal (FAILING)
# FIXME: This test case is failing
lower_limit = 0.65  # This is based on the region value for land/ocean
mask_var_limit = dv_mas_var < lower_limit
mask_tvar_limit = mask_tvar_regrid < lower_limit

np.testing.assert_equal(mask_var_limit.data, mask_tvar_limit.data)

# AssertionError:
# Arrays are not equal

# Mismatched elements: 261 / 64800 (0.403%)
#  x: array([[[False, False, False, ..., False, False, False],
#         [False, False, False, ..., False, False, False],
#         [False, False, False, ..., False, False, False],...
#  y: array([[[False, False, False, ..., False, False, False],
#         [False, False, False, ..., False, False, False],
#         [False, False, False, ..., False, False, False],...

Are there are any possible answers you came across?

xesmf logic or settings that differ from the cdms2 esmf implementation?
- xesmf settings include: periodic, extrap_method, extra_dist_exponent, extra_num_src_pts (source)

Minimal Complete Verifiable Example (MVCE)

# %%
# flake8 noqa: F401
import cdms2
from cdms2.tvariable import TransientVariable
import numpy as np
import xcdat as xc  # noqa
import xarray as xr


# Source: https://esgf-data2.llnl.gov/thredds/fileServer/user_pub_work/CMIP6/CMIP/E3SM-Project/E3SM-2-0/historical/r1i1p1f1/Amon/cl/gr/v20220830/cl_Amon_E3SM-2-0_historical_r1i1p1f1_gr_200001-201412.nc
FILEPATH = "qa/658-lat-lon-set/cl_Amon_E3SM-2-0_historical_r1i1p1f1_gr_200001-201412.nc"
# Source: https://github.com/E3SM-Project/e3sm_diags/blob/main/e3sm_diags/driver/acme_ne30_ocean_land_mask.nc
LAND_OCEAN_MASK_PATH = "e3sm_diags/driver/acme_ne30_ocean_land_mask.nc"

# %%
# ----------------------------------------------------------------------------
# xCDAT
# ----------------------------------------------------------------------------
# 1. Prepare the input variable and mask datasets.
ds_var = xr.open_dataset(FILEPATH)
ds_mask_xr = xr.open_dataset(LAND_OCEAN_MASK_PATH)

# 2. Regrid the mask variable to the input variable dataset grid.
# NOTE: xesmf "conservative_normed" is equivalent to esmf "conservative"
output_grid_xc = ds_var.regridder.grid
ds_mask_xr_regrid = ds_mask_xr.regridder.horizontal(
    "LANDFRAC", output_grid_xc, tool="xesmf", method="conservative_normed"
)
mask_var_regrid = ds_mask_xr_regrid["LANDFRAC"]

# %%
# ------------------------------------------------------------------------------
# cdms2
# ------------------------------------------------------------------------------
# 1. Prepare the input datasets and variables.
ds_tvar = cdms2.open(FILEPATH)
tvar = TransientVariable(ds_tvar["cl"])

ds_mask_cd = cdms2.open(LAND_OCEAN_MASK_PATH)
mask_tvar = TransientVariable(ds_mask_cd["LANDFRAC"])

# 2. Regrid the mask variable to the input variable dataset grid.
output_grid_cd = tvar.getGrid()
mask_tvar_regrid = mask_tvar.regrid(
    output_grid_cd,
    regridTool="esmf",
    regridMethod="conservative",
)
# ------------------------------------------------------------------------------
# TEST CASES
# ------------------------------------------------------------------------------
# 1. COMPARE GRIDS -- identical (PASSING)
np.testing.assert_equal(output_grid_xc.lat.data, output_grid_cd.getAxis(0))
np.testing.assert_equal(output_grid_xc.lon.data, output_grid_cd.getAxis(1))

# 2. COMPARE INPUT MASKS -- the same (PASSING)
np.testing.assert_equal(ds_mask_xr["LANDFRAC"].data, mask_tvar.data)

# 3. COMPARE REGRIDDED MASKS -- not equal, regridding results in differences (FAILING)
# FIXME: This test case is failing
np.testing.assert_equal(mask_var_regrid, mask_tvar_regrid.data)

# AssertionError:
# Arrays are not equal

# Mismatched elements: 18426 / 64800 (28.4%)
# Max absolute difference: 0.15945476
# Max relative difference: 1.
#  x: array([[[1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],...
#  y: array([[[1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],...

# 4. COMPARE OUTPUT MASK AVERAGES -- identical (PASSING)
mask_tvar_regrid.mean() == mask_var_regrid.mean()

# 5. COMPARE REGRIDDED MASKS WITH LOWER LIMIT -- not equal (FAILING)
# FIXME: This test case is failing
lower_limit = 0.65  # This is based on the region value for land/ocean
mask_var_limit = mask_var_regrid < lower_limit
mask_tvar_limit = mask_tvar_regrid < lower_limit

np.testing.assert_equal(mask_var_limit.data, mask_tvar_limit.data)

# AssertionError:
# Arrays are not equal

# Mismatched elements: 261 / 64800 (0.403%)
#  x: array([[[False, False, False, ..., False, False, False],
#         [False, False, False, ..., False, False, False],
#         [False, False, False, ..., False, False, False],...
#  y: array([[[False, False, False, ..., False, False, False],
#         [False, False, False, ..., False, False, False],
#         [False, False, False, ..., False, False, False],...

#%%
# PLOT THE RESULTS AND DIFFERENCES
mask_xesmf = mask_var_regrid.copy()
mask_xesmf.plot()

# %%
# Using `from_cdms2`  to turn the TransientVariable into an xr.DataArray
# because it is easier to plot.
mask_esmf = xr.DataArray.from_cdms2(mask_tvar_regrid)
mask_esmf.plot()

# Plot the difference
(mask_xesmf - mask_esmf).plot()

Relevant log output

No response

Environment

xcdat=0.6.0rc1
cdms2=3.1.5
xesmf=0.7.1

INSTALLED VERSIONS

commit: None
python: 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.76.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.1
libnetcdf: 4.9.2

xarray: 2023.6.0
pandas: 2.0.3
numpy: 1.24.4
scipy: 1.11.1
netCDF4: 1.6.4
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
iris: None
bottleneck: 1.3.7
dask: 2023.7.0
distributed: 2023.7.0
matplotlib: 3.7.2
cartopy: 0.21.1
seaborn: None
numbagg: None
fsspec: 2023.6.0
cupy: None
pint: 0.19.2
sparse: 0.14.0
flox: None
numpy_groupies: None
setuptools: 68.0.0
pip: 23.1.2
conda: None
pytest: 7.2.0
mypy: 0.991
IPython: 8.14.0
sphinx: 6.2.1

Anything else we need to know?

API Documentation:

Answered by tomvothecoder

Aug 3, 2023

I was wondering with the same thing. Maybe the issue from e3sm_diags is related. Xylar patched cdms2, so that esmf can be properly called by cdms2. When I tested, i installed -c conda-forge/label/cdms2_dev cdms2=3.1.5rc4 that fixed the issue.

The latest e3sm_diags issue and the fix for cdms2. see here: E3SM-Project/e3sm_diags#711

Thank you @chengzhuzhang! I followed your guidance and now the xESMF and ESMF regridding results are close with around 1e-7 absolute and relative max difference. I will mark my comment here as the answer.

To sum up this discussion:

Root cause of the large diffs - cdms2 was not properly importing esmf and using libcf as the fallback regridding tool. You will …

View full answer

tomvothecoder · 2023-07-26T19:00:21Z

tomvothecoder
Jul 26, 2023
Maintainer Author

Any thoughts or ideas @xCDAT/core-developers?

19 replies

chengzhuzhang Aug 2, 2023
Collaborator

I was wondering with the same thing. Maybe the issue from e3sm_diags is related. Xylar patched cdms2, so that esmf can be properly called by cdms2. When I tested, i installed -c conda-forge/label/cdms2_dev cdms2=3.1.5rc4 that fixed the issue.

chengzhuzhang Aug 2, 2023
Collaborator

The latest e3sm_diags issue and the fix for cdms2. see here: E3SM-Project/e3sm_diags#711

chengzhuzhang Aug 2, 2023
Collaborator

Environment related, I'm trying to reproduce, but realized my conda environment for validation (with both xcdat and cdms2 included) is out-dated. It was created early using https://github.com/xCDAT/xcdat-validation/blob/main/conda-env/test_dev.yml.

tomvothecoder Aug 3, 2023
Maintainer Author

I was wondering with the same thing. Maybe the issue from e3sm_diags is related. Xylar patched cdms2, so that esmf can be properly called by cdms2. When I tested, i installed -c conda-forge/label/cdms2_dev cdms2=3.1.5rc4 that fixed the issue.

The latest e3sm_diags issue and the fix for cdms2. see here: E3SM-Project/e3sm_diags#711

Thank you @chengzhuzhang! I followed your guidance and now the xESMF and ESMF regridding results are close with around 1e-7 absolute and relative max difference. I will mark my comment here as the answer.

To sum up this discussion:

Root cause of the large diffs - cdms2 was not properly importing esmf and using libcf as the fallback regridding tool. You will receive this warning below:
- Warning: avariable.regrid: regridTool = 'esmf' but your version does not seems to be built with esmf, will switch to regridTool = 'libcf'
Solution - Use the latest patched cdms2=3.1.5, which properly imports esmf.
- <conda|mamba> update -c conda-forge cdms
- Refer to this PR for more info on this issue and patch

Code with test cases:

# %%
# flake8 noqa: F401
import cdms2
from cdms2.tvariable import TransientVariable
import numpy as np
import xcdat as xc  # noqa
import xarray as xr


# Source: https://esgf-data2.llnl.gov/thredds/fileServer/user_pub_work/CMIP6/CMIP/E3SM-Project/E3SM-2-0/historical/r1i1p1f1/Amon/cl/gr/v20220830/cl_Amon_E3SM-2-0_historical_r1i1p1f1_gr_200001-201412.nc
FILEPATH = "qa/658-lat-lon-set/cl_Amon_E3SM-2-0_historical_r1i1p1f1_gr_200001-201412.nc"
# Source: https://github.com/E3SM-Project/e3sm_diags/blob/main/e3sm_diags/driver/acme_ne30_ocean_land_mask.nc
LAND_OCEAN_MASK_PATH = "e3sm_diags/driver/acme_ne30_ocean_land_mask.nc"

# %%
# ----------------------------------------------------------------------------
# xCDAT
# ----------------------------------------------------------------------------
# 1. Prepare the input variable and mask datasets.
ds_var = xr.open_dataset(FILEPATH)
ds_mask_xr = xr.open_dataset(LAND_OCEAN_MASK_PATH)

# 2. Regrid the mask variable to the input variable dataset grid.
output_grid_xc = ds_var.regridder.grid
ds_mask_xr_regrid = ds_mask_xr.regridder.horizontal(
    "LANDFRAC", output_grid_xc, tool="xesmf", method="conservative_normed", periodic=True,
)
mask_var_regrid = ds_mask_xr_regrid["LANDFRAC"]

# %%
# ------------------------------------------------------------------------------
# cdms2
# ------------------------------------------------------------------------------
# 1. Prepare the input datasets and variables.
ds_tvar = cdms2.open(FILEPATH)
tvar = TransientVariable(ds_tvar["cl"])

ds_mask_cd = cdms2.open(LAND_OCEAN_MASK_PATH)
mask_tvar = TransientVariable(ds_mask_cd["LANDFRAC"])

# 2. Regrid the mask variable to the input variable dataset grid.
output_grid_cd = tvar.getGrid()
mask_tvar_regrid = mask_tvar.regrid(
    output_grid_cd,
    regridTool="esmf",
    regridMethod="conservative",
)

#%%
# ------------------------------------------------------------------------------
# TEST CASES
# ------------------------------------------------------------------------------
# 3. COMPARE REGRIDDED MASKS -- PASSING
np.testing.assert_equal(mask_var_regrid, mask_tvar_regrid.data)

# AssertionError:
# Arrays are not equal

# Mismatched elements: 3182 / 64800 (4.91%)
# Max absolute difference: 1.1920929e-07
# Max relative difference: 1.4111636e-07
#  x: array([[[1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],...
#  y: array([[[1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],
#         [1., 1., 1., ..., 1., 1., 1.],...

# 5. COMPARE REGRIDDED MASKS WITH LOWER LIMIT -- PASSING
lower_limit = 0.65  # This is based on the region value for land/ocean
mask_var_limit = mask_var_regrid < lower_limit
mask_tvar_limit = mask_tvar_regrid < lower_limit

np.testing.assert_equal(mask_var_limit.data, mask_tvar_limit.data)  # True

Difference plot:

Answer selected by tomvothecoder

chengzhuzhang · 2023-08-02T22:15:51Z

chengzhuzhang
Aug 2, 2023
Collaborator

I'm curious where the difference are. If you plot the difference between the two maps, are the differences along the coasts? How big are the differences?

Here are the plots. From what I can tell, the differences seem to be along the coast with an approximate scale of 0.15 to -0.15.

Checking on some old results comparing cdms (conservative) vs xesmf (conservative_normed). The difference is very small (megabit diff: In [13] in validation notebook.)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Any ideas on why there are large differences when regridding E3SM land sea mask with xcdat xESMF compared to cdms2 ESMF? #521

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 19 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Any ideas on why there are large differences when regridding E3SM land sea mask with xcdat xESMF compared to cdms2 ESMF? #521

tomvothecoder Jul 26, 2023 Maintainer

Question criteria

Describe your question

Overview

The Problem

Implementation Logic

Test Cases (from MCVE section below)

Are there are any possible answers you came across?

Minimal Complete Verifiable Example (MVCE)

Relevant log output

Environment

INSTALLED VERSIONS

Anything else we need to know?

Replies: 2 comments · 19 replies

tomvothecoder Jul 26, 2023 Maintainer Author

chengzhuzhang Aug 2, 2023 Collaborator

chengzhuzhang Aug 2, 2023 Collaborator

chengzhuzhang Aug 2, 2023 Collaborator

tomvothecoder Aug 3, 2023 Maintainer Author

chengzhuzhang Aug 2, 2023 Collaborator

tomvothecoder
Jul 26, 2023
Maintainer

Replies: 2 comments 19 replies

tomvothecoder
Jul 26, 2023
Maintainer Author

chengzhuzhang Aug 2, 2023
Collaborator

chengzhuzhang Aug 2, 2023
Collaborator

chengzhuzhang Aug 2, 2023
Collaborator

tomvothecoder Aug 3, 2023
Maintainer Author

chengzhuzhang
Aug 2, 2023
Collaborator