Explore replacement of CDAT modules with xarray/XCDAT

[tomvothecoder]

The CDAT modules used in the codebase include cdp, cdms2, cdtime, cdutil, genutil, and MV2. We should tackle the replacement of these CDAT modules incrementally and methodically since it is a large dependency in e3sm_diags. I will take a deeper look into how we can port over to xarray/XCDAT in a way to minimize regressions in the behavior of the code.

Using VSCode's search functionality, the list below shows where the CDAT modules are referenced. Directories excluded from the search include analysis_data_preprocess, model_data_preprocess, .mypy_cache, and build.

cdp

76 results - 23 files

conda/e3sm_diags_env_dev.yml:
  14    - cartopy_offlinedata
  15:   - cdp=1.7.0
  16    - cdms2=3.1.5

conda/e3sm_diags_env.yml:
  14    - cartopy_offlinedata
  15:   - cdp=1.7.0
  16    - cdms2=3.1.5

conda/meta.yaml:
  24      - python
  25:     - cdp >=1.7.0
  26      - cdms2

docs/source/index.rst:
  55     diagnostics package through a unifying framework: `Community
  56:    Diagnostics Package (CDP) <https://github.com/CDAT/cdp>`_.
  57  -  is flexible for user-specified diagnostics and configuration for

docs/source/dev_guide/adding-new-diags-sets.rst:
  620          from .default_viewer import create_metadata
  621:         from cdp.cdp_viewer import OutputViewer
  622  

  684  
  685: We use the CDP Viewer to create the webpages.
  686  **This is not needed! Use whatever you want to create the webpages.**

docs/source/dev_guide/index.rst:
  10     adding-new-diags-sets
  11:    using-cdp-output-viewer
  12     releasing-e3sm-diags

docs/source/dev_guide/using-cdp-output-viewer.rst:
    1: How to use the CDP Output Viewer
    2  ================================

    6  - The basic framework of the `ESGF Output Viewer <https://github.com/ESGF/output_viewer>`__, consisting of indices, pages, groups, rows, and columns.
    7: - The process of using the `CDP Output Viewer <https://github.com/CDAT/cdp/blob/master/cdp/cdp_viewer.py>`__ (``cdp_viewer``), including explanations and examples related to the relevant functions.
    8  
    9:   - CDP Output Viewer is based on the ESGF Output Viewer
   10  

   20  
   21: .. figure:: _static/cdp_example.png
   22     :alt: Example of a webpage generated by the Output Viewer.

   36  
   37: .. figure:: _static/cdp_index_page.png
   38     :alt: Index page of the Output Viewer

   46  
   47: .. figure:: _static/cdp_page.png
   48     :alt: A page with two groups: ‘Results of addition’ and ‘Results of subtraction’

   86  
   87: Using CDP Output Viewer (``cdp_viewer``)
   88  ----------------------------------------
   89  
   90: ``cdp_viewer`` is a wrapper of the Output Viewer to simplify the API for creating a working viewer.
   91  
   92: If `cdp_viewer` is too abstract/limited, you can instead use the Output Viewer to create your own viewer.
   93  Please be aware that there is no official documentation for the Output Viewer, so you will need refer to the `codebase <https://github.com/ESGF/output_viewer>`__ directly.

  101  
  102:     from cdp.cdp_viewer import OutputViewer
  103  

  124  
  125:     Viewer HTML generated at /Users/shaheen2/github/cdp/jupyter/index.html. Would you like to open in a browser? y/[n]: y
  126  
  127  
  128: Functions of ``cdp_viewer``
  129  ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  130  
  131: The ``cdp_viewer`` five functions:
  132  

e3sm_diags/e3sm_diags_driver.py:
   10  
   11: import cdp.cdp_run
   12  

   45      """
   46:     When using cdp_run, parameters is a list of lists: [[Parameters], ...].
   47      Make this just a list: [Parameters, ...].

  324      if parameters[0].multiprocessing:
  325:         parameters = cdp.cdp_run.multiprocess(run_diag, parameters, context="fork")
  326      elif parameters[0].distributed:
  327:         parameters = cdp.cdp_run.distribute(run_diag, parameters)
  328      else:
  329:         parameters = cdp.cdp_run.serial(run_diag, parameters)
  330  

e3sm_diags/run.py:
  83              # The select() call in get_parameters() was made for the original
  84:             # command-line way of using CDP.
  85              # We just call it manually with the parameter object param.

e3sm_diags/parameter/core_parameter.py:
  2  
  3: import cdp.cdp_parameter
  4  
  5  
  6: class CoreParameter(cdp.cdp_parameter.CDPParameter):
  7      def __init__(self):

e3sm_diags/parser/core_parser.py:
  1: import cdp.cdp_parser
  2  

  5  
  6: class CoreParser(cdp.cdp_parser.CDPParser):
  7      def __init__(self, *args, **kwargs):

e3sm_diags/viewer/annual_cycle_zonal_mean_viewer.py:
  2  
  3: from cdp.cdp_viewer import OutputViewer
  4  

e3sm_diags/viewer/area_mean_time_series_viewer.py:
  2  
  3: from cdp.cdp_viewer import OutputViewer
  4  

e3sm_diags/viewer/arm_diags_viewer.py:
  2  
  3: from cdp.cdp_viewer import OutputViewer
  4  

e3sm_diags/viewer/default_viewer.py:
  12  import numpy
  13: from cdp.cdp_viewer import OutputViewer
  14  

e3sm_diags/viewer/enso_diags_viewer.py:
  3  
  4: from cdp.cdp_viewer import OutputViewer
  5  

e3sm_diags/viewer/mean_2d_viewer.py:
  2  
  3: from cdp.cdp_viewer import OutputViewer
  4  

e3sm_diags/viewer/qbo_viewer.py:
  2  
  3: from cdp.cdp_viewer import OutputViewer
  4  

e3sm_diags/viewer/streamflow_viewer.py:
  2  
  3: from cdp.cdp_viewer import OutputViewer
  4  

e3sm_diags/viewer/tc_analysis_viewer.py:
  2  
  3: from cdp.cdp_viewer import OutputViewer
  4  

tests/test.sh:
  19  printf "==============================================\n"
  20: # Integration tests must be called individually. `e3sm_diags` uses `cdp_parser.get_parameters()`, which incorrectly picks up `python -m unittest` cmd args.
  21: # This results in `cdp_parser.get_parameters()` raising `error: unrecognized arguments <UNITTEST ARGS>`.
  22  # Running tests using these commands do not work:

  24  #  - `python -m unittest discover -s tests/integration --pattern "test_*.py"`
  25: # Related lines of code : https://github.com/CDAT/cdp/blob/2917603097112f7db52be0bf7a2e766e14cc2e16/cdp/cdp_parser.py#L498
  26  command='python -m unittest tests/integration/'

tests/integration/e3sm_diags_run.log:
  10  2021-12-10 10:30:32,869 [INFO]: run.py(_add_parent_attrs_to_children:147) >> ['test_name']
  11: 2021-12-10 10:30:32,870 [WARNING]: warnings.py(_showwarnmsg:109) >> /home/ac.tvo/miniconda3/envs/e3sm_diags_env_dev/lib/python3.9/site-packages/cdp/cdp_parser.py:192: DeprecationWarning: This method will be removed in future versions.  Use 'parser.read_file()' instead.
  12    config.readfp(cfg_file_obj)
  13  
  14: 2021-12-10 10:30:33,071 [WARNING]: warnings.py(_showwarnmsg:109) >> /home/ac.tvo/miniconda3/envs/e3sm_diags_env_dev/lib/python3.9/site-packages/cdp/cdp_parser.py:410: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
  15    if not isinstance(param_v, collections.Iterable):

tests/integration/all_sets_results_test/prov/environment.yml:
  27    - cdms2=3.1.5=py39h6e0a748_9
  28:   - cdp=1.7.0=pyhd8ed1ab_0
  29    - cdtime=3.1.4=py39hd7c74d3_3

cdms2

118 results - 29 files

conda/e3sm_diags_env_dev.yml:
  15    - cdp=1.7.0
  16:   - cdms2=3.1.5
  17    - cdtime=3.1.4

conda/e3sm_diags_env.yml:
  15    - cdp=1.7.0
  16:   - cdms2=3.1.5
  17    - cdtime=3.1.4

conda/meta.yaml:
  25      - cdp >=1.7.0
  26:     - cdms2
  27      - cdutil

e3sm_diags/__init__.py:
  6  
  7: # Disable MPI in cdms2, which is not currently supported by E3SM-unified
  8  os.environ["CDMS_NO_MPI"] = "True"

e3sm_diags/e3sm_diags_vars.py:
  11  
  12: import cdms2
  13  

  48          raise RuntimeError(msg)
  49:     with cdms2.open(path) as f:
  50          return f.variables.keys()

e3sm_diags/derivations/acme.py:
  11  if TYPE_CHECKING:
  12:     from cdms2.axis import FileAxis
  13:     from cdms2.fvariable import FileVariable
  14  

e3sm_diags/driver/annual_cycle_zonal_mean_driver.py:
   6  import MV2
   7: from cdms2 import createAxis
   8  

  18  if TYPE_CHECKING:
  19:     from cdms2.axis import TransientAxis
  20:     from cdms2.tvariable import TransientVariable
  21  

e3sm_diags/driver/area_mean_time_series_driver.py:
   4  
   5: import cdms2
   6  import cdutil

  60          )
  61:         with cdms2.open(mask_path) as f:
  62              land_frac = f("LANDFRAC")

e3sm_diags/driver/arm_diags_driver.py:
    4  
    5: import cdms2
    6  import numpy as np

   96                          ref_file = os.path.join(ref_path, ref_file_name)
   97:                         ref_data = cdms2.open(ref_file)
   98  

  209                      ref_file = os.path.join(ref_path, ref_file_name)
  210:                     ref_data = cdms2.open(ref_file)
  211                      if var == "CLOUD":

  319                          )
  320:                     ref_data = cdms2.open(ref_file)
  321                      vars_funcs = get_vars_funcs_for_derived_var(ref_data, var)

  398              ref_file = os.path.join(ref_path, ref_file_name)
  399:             ref_data = cdms2.open(ref_file)
  400              ref_pr = ref_data("pr")  # mm/hr

e3sm_diags/driver/cosp_histogram_driver.py:
   4  
   5: import cdms2
   6  

  68              )
  69:             with cdms2.open(mask_path) as f:
  70                  land_frac = f("LANDFRAC")

e3sm_diags/driver/diurnal_cycle_driver.py:
   4  
   5: import cdms2
   6  

  40              )
  41:             with cdms2.open(mask_path) as f:
  42                  land_frac = f("LANDFRAC")

e3sm_diags/driver/enso_diags_driver.py:
    6  
    7: import cdms2
    8  import cdutil

  232              )
  233:             with cdms2.open(mask_path) as f:
  234                  land_frac = f("LANDFRAC")

e3sm_diags/driver/lat_lon_driver.py:
    5  
    6: import cdms2
    7  import MV2

   80              )
   81:             with cdms2.open(mask_path) as f:
   82                  land_frac = f("LANDFRAC")

  100              if ref_name == "WARREN":
  101:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  102                  mv2 = MV2.masked_where(mv2 == -0.9, mv2)

  104              if ref_name == "AIRS":
  105:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  106                  mv2 = MV2.masked_where(mv2 > 1e20, mv2)
  107              if ref_name == "WILLMOTT" or ref_name == "CLOUDSAT":
  108:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  109                  mv2 = MV2.masked_where(mv2 == -999.0, mv2)

e3sm_diags/driver/meridional_mean_2d_driver.py:
    2  
    3: import cdms2
    4  import cdutil

   20      """
   21:     orig_bounds = cdms2.getAutoBounds()
   22:     cdms2.setAutoBounds(1)
   23      lev = ref.getLevel()

   41          lev.setBounds(None)
   42:     cdms2.setAutoBounds(orig_bounds)
   43  

  130                      # in order to use regrid tool we need to have at least two latitude bands, so generate new grid first
  131:                     lat = cdms2.createAxis([0])
  132                      lat.setBounds(numpy.array([-1, 1]))
  133                      lat.designateLatitude()
  134:                     grid = cdms2.createRectGrid(lat, lon_out)
  135  

  153                      # latitude bands, so generate new grid first.
  154:                     lat = cdms2.createAxis([0])
  155                      lat.setBounds(numpy.array([-1, 1]))
  156                      lat.designateLatitude()
  157:                     grid = cdms2.createRectGrid(lat, lon_out)
  158  

e3sm_diags/driver/polar_driver.py:
   4  
   5: import cdms2
   6  import MV2

  69              )
  70:             with cdms2.open(mask_path) as f:
  71                  land_frac = f("LANDFRAC")

  89              if ref_name == "WARREN":
  90:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  91                  mv2 = MV2.masked_where(mv2 == -0.9, mv2)

  93              if ref_name == "AIRS":
  94:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  95                  mv2 = MV2.masked_where(mv2 > 1e20, mv2)
  96              if ref_name == "WILLMOTT" or ref_name == "CLOUDSAT":
  97:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  98                  mv2 = MV2.masked_where(mv2 == -999.0, mv2)

e3sm_diags/driver/streamflow_driver.py:
    5  
    6: import cdms2
    7  import cdutil

  283          logger.info("test_array.shape={}".format(test_array.shape))
  284:     if type(area_upstream) == cdms2.tvariable.TransientVariable:
  285          area_upstream = area_upstream.getValue()

  487  
  488:                 if type(monthly) == cdms2.tvariable.TransientVariable:
  489                      monthly = monthly.getValue()

  513  
  514:             if type(monthly) == cdms2.tvariable.TransientVariable:
  515                  monthly = monthly.getValue()

e3sm_diags/driver/tc_analysis_driver.py:
    7  
    8: import cdms2
    9  import numpy as np

   77      )
   78:     test_cyclones_hist = cdms2.open(test_cyclones_file)(
   79          "density", lat=(-60, 60, "ccb"), squeeze=1

   84      )
   85:     test_aew_hist = cdms2.open(test_aew_file)(
   86          "density", lat=(0, 35, "ccb"), lon=(-180, 0, "ccb"), squeeze=1

  113          )
  114:         ref_cyclones_hist = cdms2.open(ref_cyclones_file)("density", squeeze=1)
  115          ref_aew_file = os.path.join(

  118          )
  119:         ref_aew_hist = cdms2.open(ref_aew_file)("density", squeeze=1)
  120          ref_data["metrics"] = generate_tc_metrics_from_te_stitch_file(ref_te_file)

  133          )
  134:         ref_cyclones_hist = cdms2.open(ref_cyclones_file)(
  135              "density", lat=(-60, 60, "ccb"), squeeze=1

  137          ref_aew_file = os.path.join(reference_data_path, "aew_hist_ERA5_2010_2014.nc")
  138:         ref_aew_hist = cdms2.open(ref_aew_file)(
  139              "density", lat=(0, 35, "ccb"), lon=(180, 360, "ccb"), squeeze=1

  176      mask_path = os.path.join(e3sm_diags.INSTALL_PATH, "acme_ne30_ocean_land_mask.nc")
  177:     ocnfrac = cdms2.open(mask_path)("OCNFRAC", squeeze=1)
  178  

  281      vars: Dict[str, Any],
  282:     ocnfrac: cdms2.dataset.DatasetVariable,
  283      basin_info: BasinInfo,

  290      :type vars: Dict[str, Any]
  291:     :param ocnfrac: Ocnfrac CDMS2 dataset variable
  292:     :type ocnfrac: cdms2.dataset.DatasetVariable
  293      :param basin_info: Basin information

e3sm_diags/driver/zonal_mean_2d_driver.py:
    4  
    5: import cdms2
    6  import cdutil

   21      """Creates the mean, max, min, rmse, corr in a dictionary"""
   22:     orig_bounds = cdms2.getAutoBounds()
   23:     cdms2.setAutoBounds(1)
   24      lev = ref.getLevel()

   42          lev.setBounds(None)
   43:     cdms2.setAutoBounds(orig_bounds)
   44  

   95              )
   96:             with cdms2.open(mask_path) as f:
   97                  land_frac = f("LANDFRAC")

  115              if ref_name == "WARREN":
  116:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  117                  mv2 = MV2.masked_where(mv2 == -0.9, mv2)

  119              if ref_name == "AIRS":
  120:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  121                  mv2 = MV2.masked_where(mv2 > 1e20, mv2)
  122              if ref_name == "WILLMOTT" or ref_name == "CLOUDSAT":
  123:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  124                  mv2 = MV2.masked_where(mv2 == -999.0, mv2)

e3sm_diags/driver/zonal_mean_xy_driver.py:
    2  
    3: import cdms2
    4  import cdutil

   38          )
   39:         mv2_reg = cdms2.createVariable(
   40              b0,

   52          )
   53:         mv1_reg = cdms2.createVariable(
   54              a0,

  126              if ref_name == "WARREN":
  127:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  128                  mv2 = MV2.masked_where(mv2 == -0.9, mv2)

  130              if ref_name == "AIRS":
  131:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  132                  mv2 = MV2.masked_where(mv2 > 1e20, mv2)
  133              if ref_name == "WILLMOTT" or ref_name == "CLOUDSAT":
  134:                 # This is cdms2 fix for bad mask, Denis' fix should fix this.
  135                  mv2 = MV2.masked_where(mv2 == -999.0, mv2)

e3sm_diags/driver/utils/climo.py:
   1: import cdms2
   2  import numpy as np

  79  
  80:     trans_var = cdms2.createVariable(climo)(squeeze=1)
  81      # Losing the grid after a squeeze is normal, we need to set it again.

e3sm_diags/driver/utils/dataset.py:
   10  
   11: import cdms2
   12  

  181          file_path = self._get_timeseries_file_path(primary_var, data_path)
  182:         fin = cdms2.open(file_path)
  183          result = fin(static_var)

  231  
  232:         with cdms2.open(filename) as f:
  233              return f.getglobal(attr)

  338  
  339:         with cdms2.open(filename) as data_file:
  340              for var in vars_to_get:

  352  
  353:                     # Get the variables as cdms2.TransientVariables.
  354:                     # Ex: variables is [PRECC, PRECL], where both are cdms2.TransientVariables.
  355                      variables = self._get_original_vars_climo(

  416          Given a dictionary in the form {(vars): func}, get the vars
  417:         from the data_file as cdms2.TransientVariables.
  418  

  463              if not extra_vars_only:
  464:                 # Open the files of the variables and get the cdms2.TransientVariables.
  465                  # Ex: [PRECC, PRECL], where both are TransientVariables.

  623          Given a dictionary in the form {(vars): func}, get the vars
  624:         from files in data_path as cdms2.TransientVariables.
  625  

  683          else:
  684:             # with cdms2.open(fnm) as f:
  685              #    var_time = f(var, time=(start_time, end_time, 'ccb'))(squeeze=1)

  687              # For xml files using above with statement won't work because the Dataset object returned doesn't have attribute __enter__ for content management.
  688:             fin = cdms2.open(fnm)
  689              var_time = fin(var, time=(start_time, end_time, slice_flag))(squeeze=1)

e3sm_diags/driver/utils/general.py:
    7  
    8: import cdms2
    9  import cdutil

   47      var_time[:] = 0.5 * (tbounds[:, 0] + tbounds[:, 1])
   48:     time2 = cdms2.createAxis(var_time)
   49      time2.designateTime()

  298              # Set cdms preferences - no compression, no shuffling, no complaining
  299:             cdms2.setNetcdfDeflateFlag(1)
  300              # 1-9, min to max - Comes at heavy IO (read/write time cost)
  301:             cdms2.setNetcdfDeflateLevelFlag(0)
  302:             cdms2.setNetcdfShuffleFlag(0)
  303:             cdms2.setCompressionWarnings(0)  # Turn off warning messages
  304  

  314              test_pth = os.path.join(path, file_name)
  315:             with cdms2.open(test_pth, "w+") as file_test:
  316                  try:

  329          # Set cdms preferences - no compression, no shuffling, no complaining
  330:         cdms2.setNetcdfDeflateFlag(1)
  331          # 1-9, min to max - Comes at heavy IO (read/write time cost)
  332:         cdms2.setNetcdfDeflateLevelFlag(0)
  333:         cdms2.setNetcdfShuffleFlag(0)
  334:         cdms2.setCompressionWarnings(0)  # Turn off warning messages
  335  

  347  
  348:         with cdms2.open(test_pth, cdms_arg) as file_test:
  349              file_test.write(test)

  354          ref_pth = os.path.join(pth, parameter.output_file + "_ref.nc")
  355:         with cdms2.open(ref_pth, cdms_arg) as file_ref:
  356              file_ref.write(ref)

  362              diff_pth = os.path.join(pth, parameter.output_file + "_diff.nc")
  363:             with cdms2.open(diff_pth, cdms_arg) as file_diff:
  364                  file_diff.write(diff)

tests/e3sm_diags/derivations/test_acme.py:
  13  if TYPE_CHECKING:
  14:     from cdms2.axis import FileAxis
  15  

tests/e3sm_diags/drivers/test_annual_cycle_zonal_mean_driver.py:
  4  import numpy as np
  5: from cdms2.axis import TransientAxis
  6: from cdms2.tvariable import TransientVariable
  7  

tests/e3sm_diags/drivers/test_tc_analysis_driver.py:
  21  if TYPE_CHECKING:
  22:     from cdms2.dataset import DatasetVariable
  23  

tests/integration/e3sm_diags_run.log:
  1: 2021-12-10 10:30:32,854 [WARNING]: warnings.py(_showwarnmsg:109) >> /home/ac.tvo/miniconda3/envs/e3sm_diags_env_dev/lib/python3.9/site-packages/cdms2/variable.py:74: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  2  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

  4  
  5: 2021-12-10 10:30:32,863 [WARNING]: warnings.py(_showwarnmsg:109) >> /home/ac.tvo/miniconda3/envs/e3sm_diags_env_dev/lib/python3.9/site-packages/cdms2/tvariable.py:331: DeprecationWarning: Non-scalar arrays for the fill value are deprecated. Use arrays with scalar values instead. The filled function still supports any array as `fill_value`.
  6    self._missing = numpy.array(value).astype(self.dtype)

tests/integration/test_dataset.py:
    2  
    3: import cdms2
    4  

   15      def test_convert_units(self):
   16:         with cdms2.open(f"{TEST_DATA_PATH}/precc.nc") as precc_file:
   17              var = precc_file("PRECC")

   91  
   92:         precc_file = cdms2.open(get_abs_file_path('integration_test_data/precc.nc'))
   93          acme_derivations.process_derived_var('PRECT', derived_var,

  105  
  106:         precc_file = cdms2.open(get_abs_file_path('integration_test_data/precc.nc'))
  107          with self.assertRaises(RuntimeError):

  125          # add derived_var_dict to default_derived_vars
  126:         precc_file = cdms2.open(get_abs_file_path('integration_test_data/precc.nc'))
  127          self.parameter.derived_variables = derived_var_dict

  155          # add derived_var_dict to default_derived_vars
  156:         precc_file = cdms2.open(get_abs_file_path('integration_test_data/precc.nc'))
  157          self.parameter.derived_variables = derived_var_dict

  177  
  178:         precc_file = cdms2.open(get_abs_file_path('integration_test_data/precc.nc'))
  179          self.parameter.derived_variables = derived_var_dict

  187      def test_mask_by(self):
  188:         with cdms2.open(get_abs_file_path('integration_test_data/precc.nc')) as precc_file:
  189              prcc = precc_file('PRECC')
  190:         with cdms2.open(get_abs_file_path('integration_test_data/precl.nc')) as precl_file:
  191              prcl = precl_file('PRECL')

tests/integration/all_sets_results_test/prov/environment.yml:
  26    - cdat_info=8.2.1=pyh9f0ad1d_1
  27:   - cdms2=3.1.5=py39h6e0a748_9
  28    - cdp=1.7.0=pyhd8ed1ab_0

tests/testing_data/create_test_data.py:
    7  
    8: import cdms2
    9  

  128      orig_path = os.path.join(TEMP_DIR, file_name)
  129:     with cdms2.open(orig_path) as orig_file:
  130          if os.path.exists(f_and_v.local_uri):

  132  
  133:         with cdms2.open(f_and_v.local_uri, "w") as new_file:
  134              # Copy over the attributes.

  136  
  137:             # Lat, lon, and time are considered dimensions in cdms2.
  138              # Those are automatically saved.

cdtime

5 results - 5 files

conda/e3sm_diags_env_dev.yml:
  16    - cdms2=3.1.5
  17:   - cdtime=3.1.4
  18    - cdutil=8.2.1

conda/e3sm_diags_env.yml:
  16    - cdms2=3.1.5
  17:   - cdtime=3.1.4
  18    - cdutil=8.2.1

conda/meta.yaml:
  28      - genutil
  29:     - cdtime
  30      - numpy

e3sm_diags/driver/utils/general.py:
  53      time2.setBounds(tbounds)
  54:     # time2.calendar = cdtime.NoLeapCalendar
  55      time2.id = "time"

tests/integration/all_sets_results_test/prov/environment.yml:
  28    - cdp=1.7.0=pyhd8ed1ab_0
  29:   - cdtime=3.1.4=py39hd7c74d3_3
  30    - cdutil=8.2.1=pyhd8ed1ab_1

cdutil

114 results - 21 files

conda/e3sm_diags_env_dev.yml:
  17    - cdtime=3.1.4
  18:   - cdutil=8.2.1
  19    - genutil=8.2.1

conda/e3sm_diags_env.yml:
  17    - cdtime=3.1.4
  18:   - cdutil=8.2.1
  19    - genutil=8.2.1

conda/meta.yaml:
  26      - cdms2
  27:     - cdutil
  28      - genutil

e3sm_diags/derivations/default_regions.py:
    1: import cdutil
    2  
    3  regions_specs = {
    4:     "NHEX": {"domain": cdutil.region.domain(latitude=(30.0, 90, "ccb"))},
    5:     "SHEX": {"domain": cdutil.region.domain(latitude=(-90.0, -30, "ccb"))},
    6:     "TROPICS": {"domain": cdutil.region.domain(latitude=(-30.0, 30, "ccb"))},
    7      "global": {},
    8:     "TRMM_region": {"domain": cdutil.region.domain(latitude=(-38.0, 38, "ccb"))},
    9:     "90S50S": {"domain": cdutil.region.domain(latitude=(-90.0, -50, "ccb"))},
   10:     "50S20S": {"domain": cdutil.region.domain(latitude=(-50.0, -20, "ccb"))},
   11:     "20S20N": {"domain": cdutil.region.domain(latitude=(-20.0, 20, "ccb"))},
   12:     "50S50N": {"domain": cdutil.region.domain(latitude=(-50.0, 50, "ccb"))},
   13:     "5S5N": {"domain": cdutil.region.domain(latitude=(-5.0, 5, "ccb"))},
   14:     "20N50N": {"domain": cdutil.region.domain(latitude=(20.0, 50, "ccb"))},
   15:     "50N90N": {"domain": cdutil.region.domain(latitude=(50.0, 90, "ccb"))},
   16:     "60S90N": {"domain": cdutil.region.domain(latitude=(-60.0, 90, "ccb"))},
   17:     "60S60N": {"domain": cdutil.region.domain(latitude=(-60.0, 60, "ccb"))},
   18      "ocean": {

   28          "value": 0.65,
   29:         "domain": cdutil.region.domain(latitude=(-60.0, 90, "ccb")),
   30      },

   32          "value": 0.65,
   33:         "domain": cdutil.region.domain(latitude=(-30.0, 30, "ccb")),
   34      },

   36          "value": 0.65,
   37:         "domain": cdutil.region.domain(latitude=(30.0, 90, "ccb")),
   38      },

   40          "value": 0.65,
   41:         "domain": cdutil.region.domain(latitude=(-90.0, -30, "ccb")),
   42      },

   44          "value": 0.65,
   45:         "domain": cdutil.region.domain(latitude=(-30.0, 30, "ccb")),
   46      },

   48          "value": 0.65,
   49:         "domain": cdutil.region.domain(latitude=(30.0, 90, "ccb")),
   50      },

   52          "value": 0.65,
   53:         "domain": cdutil.region.domain(latitude=(-90.0, -30, "ccb")),
   54      },
   55      # follow AMWG polar range,more precise selector
   56:     "polar_N": {"domain": cdutil.region.domain(latitude=(50.0, 90.0, "ccb"))},
   57:     "polar_S": {"domain": cdutil.region.domain(latitude=(-90.0, -55.0, "ccb"))},
   58      # To match AMWG results, the bounds is not as precise in this case

   62      "NAM": {
   63:         "domain": cdutil.region.domain(
   64              latitude=(20.0, 90, "ccb"), longitude=(-180, 180, "ccb")

   67      "NAO": {
   68:         "domain": cdutil.region.domain(
   69              latitude=(20.0, 80, "ccb"), longitude=(-90, 40, "ccb")

   72      "SAM": {
   73:         "domain": cdutil.region.domain(
   74              latitude=(-20.0, -90, "ccb"), longitude=(0, 360, "ccb")

   77      "PNA": {
   78:         "domain": cdutil.region.domain(
   79              latitude=(20.0, 85, "ccb"), longitude=(120, 240, "ccb")

   82      "PDO": {
   83:         "domain": cdutil.region.domain(
   84              latitude=(20.0, 70, "ccb"), longitude=(110, 260, "ccb")

   89      "AllM": {
   90:         "domain": cdutil.region.domain(
   91              latitude=(-45.0, 45.0, "ccb"), longitude=(0.0, 360.0, "ccb")

   95      "NAMM": {
   96:         "domain": cdutil.region.domain(
   97              latitude=(0.0, 45.0, "ccb"), longitude=(210.0, 310.0, "ccb")

  101      "SAMM": {
  102:         "domain": cdutil.region.domain(
  103              latitude=(-45.0, 0.0, "ccb"), longitude=(240.0, 330.0, "ccb")

  107      "NAFM": {
  108:         "domain": cdutil.region.domain(
  109              latitude=(0.0, 45.0, "ccb"), longitude=(310.0, 60.0, "ccb")

  113      "SAFM": {
  114:         "domain": cdutil.region.domain(
  115              latitude=(-45.0, 0.0, "ccb"), longitude=(0.0, 90.0, "ccb")

  119      "ASM": {
  120:         "domain": cdutil.region.domain(
  121              latitude=(0.0, 45.0, "ccb"), longitude=(60.0, 180.0, "ccb")

  125      "AUSM": {
  126:         "domain": cdutil.region.domain(
  127              latitude=(-45.0, 0.0, "ccb"), longitude=(90.0, 160.0, "ccb")

  131      "NINO3": {
  132:         "domain": cdutil.region.domain(
  133              latitude=(-5.0, 5.0, "ccb"), longitude=(210.0, 270.0, "ccb")

  136      "NINO34": {
  137:         "domain": cdutil.region.domain(
  138              latitude=(-5.0, 5.0, "ccb"), longitude=(190.0, 240.0, "ccb")

  141      "NINO4": {
  142:         "domain": cdutil.region.domain(
  143              latitude=(-5.0, 5.0, "ccb"), longitude=(160.0, 210.0, "ccb")

  147      "W_Pacific": {
  148:         "domain": cdutil.region.domain(
  149              latitude=(-20.0, 20.0, "ccb"), longitude=(90.0, 180.0, "ccb")

  152      "CONUS": {
  153:         "domain": cdutil.region.domain(
  154              latitude=(25.0, 50.0, "ccb"), longitude=(-125.0, -65.0, "ccb")

  157      "Amazon": {
  158:         "domain": cdutil.region.domain(
  159              latitude=(-20.0, 5.0, "ccb"), longitude=(-80.0, -45.0, "ccb")

  162      # Below is for RRM(regionally refined model) domains.
  163:     # 'CONUS_RRM': {'domain': cdutil.region.domain(latitude=(20., 50., 'ccb'), longitude=(-125., -65., 'ccb'))},For RRM dataset, negative value won't work
  164      "CONUS_RRM": {
  165:         "domain": cdutil.region.domain(
  166              latitude=(20.0, 50.0, "ccb"), longitude=(235.0, 295.0, "ccb")

  169      # Below is for debugging. A smaller latitude range reduces processing time.
  170:     "DEBUG": {"domain": cdutil.region.domain(latitude=(-2.0, 2, "ccb"))},
  171  }

e3sm_diags/driver/annual_cycle_zonal_mean_driver.py:
   4  
   5: import cdutil
   6  import MV2

  52  
  53:         test_ac_zonal_mean = cdutil.averager(test_ac, axis="x", weights="generate")
  54:         test_ac_reg_zonal_mean = cdutil.averager(
  55              test_ac_reg, axis="x", weights="generate"

  70              else:
  71:                 ref_ac_zonal_mean = cdutil.averager(
  72                      ref_ac, axis="x", weights="generate"
  73                  )
  74:                 ref_ac_reg_zonal_mean = cdutil.averager(
  75                      ref_ac_reg, axis="x", weights="generate"

  78          else:
  79:             ref_ac_zonal_mean = cdutil.averager(ref_ac, axis="x", weights="generate")
  80:             ref_ac_reg_zonal_mean = cdutil.averager(
  81                  ref_ac_reg, axis="x", weights="generate"

  90          # else:
  91:         #    ref_ac_zonal_mean = cdutil.averager(ref_ac, axis="x", weights="generate")
  92:         #    ref_ac_reg_zonal_mean = cdutil.averager(
  93          #        ref_ac_reg, axis="x", weights="generate"

e3sm_diags/driver/area_mean_time_series_driver.py:
    5  import cdms2
    6: import cdutil
    7  

   87              # over months to get the yearly mean.
   88:             test_domain = cdutil.averager(test_domain, axis="xy")
   89:             cdutil.setTimeBoundsMonthly(test_domain)
   90:             test_domain_year = cdutil.YEAR(test_domain)
   91:             # add back attributes since they got lost after applying cdutil.YEAR
   92              test_domain_year.long_name = test.long_name

  113  
  114:                     ref_domain = cdutil.averager(ref_domain, axis="xy")
  115:                     cdutil.setTimeBoundsMonthly(ref_domain)
  116                      logger.info(

  123  
  124:                     ref_domain_year = cdutil.YEAR(ref_domain)
  125                      ref_domain_year.ref_name = ref_name

e3sm_diags/driver/enso_diags_driver.py:
    7  import cdms2
    8: import cdutil
    9  import numpy

   90          # Domain average
   91:         sst_avg = cdutil.averager(sst_nino, axis="xy")
   92          # Get anomaly from annual cycle climatology
   93:         sst_avg_anomaly = cdutil.ANNUALCYCLE.departures(sst_avg)
   94          nino_index = sst_avg_anomaly

  121      # over months to get the yearly mean.
  122:     cdutil.setTimeBoundsMonthly(domain)
  123      # Get anomaly from annual cycle climatology

  125          logger.info("domain.shape: {}".format(domain.shape))
  126:     anomaly = cdutil.ANNUALCYCLE.departures(domain)
  127      nlat = len(anomaly.getLatitude())

  129      reg_coe = anomaly[0, :, :](squeeze=1)
  130:     confidence_levels = cdutil.ANNUALCYCLE.departures(domain)[0, :, :](squeeze=1)
  131      # Neither of the following methods work, so we just set values in confidence_levels

  417              # Domain average
  418:             test_avg = cdutil.averager(test_data_ts_regional, axis="xy")
  419:             ref_avg = cdutil.averager(ref_data_ts_regional, axis="xy")
  420              # Get anomaly from annual cycle climatology
  421:             y["test"] = cdutil.ANNUALCYCLE.departures(test_avg)
  422:             y["ref"] = cdutil.ANNUALCYCLE.departures(ref_avg)
  423              y["units"] = test_avg.units

e3sm_diags/driver/meridional_mean_2d_driver.py:
    3  import cdms2
    4: import cdutil
    5  import MV2

  115  
  116:                 mv1_p = cdutil.averager(mv1_p, axis="y")
  117:                 mv2_p = cdutil.averager(mv2_p, axis="y")
  118  

e3sm_diags/driver/qbo_driver.py:
   5  
   6: import cdutil
   7  import numpy as np

  35      # Average over longitude (i.e., each latitude's average in data_region)
  36:     data_lon_average = cdutil.averager(data_region, axis="x")
  37      # Average over latitude (i.e., average for entire data_region)
  38:     data_lon_lat_average = cdutil.averager(data_lon_average, axis="y")
  39      # Get data by vertical level

  95      # Average over lat and lon
  96:     data_lat_lon_average = cdutil.averager(data_region, axis="xy")
  97      # Average over vertical

e3sm_diags/driver/streamflow_driver.py:
    6  import cdms2
    7: import cdutil
    8  import numpy

  222          ref_data_ts = ref_data.get_timeseries_variable(var)
  223:         var_array = ref_data_ts(cdutil.region.domain(latitude=(-90.0, 90, "ccb")))
  224          if parameter.print_statements:

  265          test_data_ts = test_data.get_timeseries_variable(var)
  266:         var_array = test_data_ts(cdutil.region.domain(latitude=(-90.0, 90, "ccb")))
  267          if parameter.print_statements:

e3sm_diags/driver/zonal_mean_2d_driver.py:
    5  import cdms2
    6: import cdutil
    7  import MV2

  169                  diff_p = mv1_p_reg - mv2_p_reg
  170:                 diff = cdutil.averager(diff_p, axis="x")
  171  
  172:                 mv1_p = cdutil.averager(mv1_p, axis="x")
  173:                 mv2_p = cdutil.averager(mv2_p, axis="x")
  174  

  178  
  179:                 mv1_reg = cdutil.averager(mv1_p_reg, axis="x")
  180:                 mv2_reg = cdutil.averager(mv2_p_reg, axis="x")
  181  

e3sm_diags/driver/zonal_mean_xy_driver.py:
    3  import cdms2
    4: import cdutil
    5  import MV2

  177                          logger.info(f"Selected region: {region}")
  178:                         mv1_zonal = cdutil.averager(mv1, axis="x")
  179:                         mv2_zonal = cdutil.averager(mv2, axis="x")
  180  

  227                      logger.info(f"Selected region: {region}")
  228:                     mv1_zonal = cdutil.averager(mv1, axis="x")
  229:                     mv2_zonal = cdutil.averager(mv2, axis="x")
  230  

e3sm_diags/driver/utils/general.py:
    8  import cdms2
    9: import cdutil
   10  import genutil

   56      var.setAxis(0, time2)
   57:     #    cdutil.setTimeBoundsMonthly(var)
   58  

  136      ps = ps / 100.0  # convert unit from 'Pa' to mb
  137:     levels_orig = cdutil.vertical.reconstructPressureFromHybrid(ps, hyam, hybm, p0)
  138      levels_orig.units = "mb"

  142          levels_orig = levels_orig(lev=slice(-1, None, -1))
  143:     var_p = cdutil.vertical.logLinearInterpolation(
  144          var(squeeze=1), levels_orig(squeeze=1), plev

  167          levels_orig = levels_orig(lev=slice(-1, None, -1))
  168:     var_p = cdutil.vertical.logLinearInterpolation(
  169          var(squeeze=1), levels_orig(squeeze=1), plev

e3sm_diags/metrics/__init__.py:
   1: import cdutil
   2  import genutil

  26  def mean(variable, axis="xy"):
  27:     return cdutil.averager(variable, axis=axis, weights="generate")
  28  

e3sm_diags/plot/cartopy/diurnal_cycle_plot.py:
    6  import cartopy.feature as cfeature
    7: import cdutil
    8  import matplotlib

  104          # Assume global domain
  105:         domain = cdutil.region.domain(latitude=(-90.0, 90, "ccb"))
  106      kargs = domain.components()[0].kargs

e3sm_diags/plot/cartopy/enso_diags_plot.py:
   5  import cartopy.crs as ccrs
   6: import cdutil
   7  import matplotlib

  87          # Assume global domain
  88:         domain = cdutil.region.domain(latitude=(-90.0, 90, "ccb"))
  89      kargs = domain.components()[0].kargs

e3sm_diags/plot/cartopy/lat_lon_plot.py:
   6  import cartopy.feature as cfeature
   7: import cdutil
   8  import matplotlib

  89          # Assume global domain
  90:         domain = cdutil.region.domain(latitude=(-90.0, 90, "ccb"))
  91      kargs = domain.components()[0].kargs

e3sm_diags/plot/cartopy/meridional_mean_2d_plot.py:
  50      #    var_max = float(var.max())
  51:     #    var_mean = cdutil.averager(var, axis='xy', weights='generate')
  52      var = add_cyclic(var)

e3sm_diags/plot/cartopy/streamflow_plot.py:
    6  import cartopy.feature as cfeature
    7: import cdutil
    8  import matplotlib

   80          # Assume global domain
   81:         domain = cdutil.region.domain(latitude=(-90.0, 90, "ccb"))
   82      kargs = domain.components()[0].kargs

  383          # Assume global domain
  384:         domain = cdutil.region.domain(latitude=(-90.0, 90, "ccb"))
  385      kargs = domain.components()[0].kargs

e3sm_diags/plot/cartopy/zonal_mean_2d_plot.py:
  51      #    var_max = float(var.max())
  52:     #    var_mean = cdutil.averager(var, axis='xy', weights='generate')
  53      #    var = add_cyclic(var)

tests/integration/all_sets_results_test/prov/environment.yml:
  29    - cdtime=3.1.4=py39hd7c74d3_3
  30:   - cdutil=8.2.1=pyhd8ed1ab_1
  31    - certifi=2021.10.8=py39hf3d152e_1

genutil

14 results - 8 files

conda/e3sm_diags_env_dev.yml:
  18    - cdutil=8.2.1
  19:   - genutil=8.2.1
  20    - lxml=4.6.3

conda/e3sm_diags_env.yml:
  18    - cdutil=8.2.1
  19:   - genutil=8.2.1
  20    - lxml=4.6.3

conda/meta.yaml:
  27      - cdutil
  28:     - genutil
  29      - cdtime

e3sm_diags/derivations/acme.py:
  8  import numpy as np
  9: from genutil import udunits
  10  

e3sm_diags/driver/enso_diags_driver.py:
  140              independent_var = nino_index
  141:             # Uncomment the following line to use CDAT/genutil instead
  142              # (You'll also need to set pvalue)
  143:             # slope, intercept = genutil.statistics.linearregression(dependent_var, x=independent_var)
  144              slope, _, _, pvalue, _ = scipy.stats.linregress(

e3sm_diags/driver/utils/general.py:
    9  import cdutil
   10: import genutil
   11  import MV2

  158      # grow 1d levels_orig to mv dimention
  159:     var, levels_orig = genutil.grower(var, levels_orig)
  160      # levels_orig.info()

e3sm_diags/metrics/__init__.py:
   1  import cdutil
   2: import genutil
   3  import numpy

  13          corr = float(
  14:             genutil.statistics.correlation(model, obs, axis=axis, weights="generate")
  15          )

  36      try:
  37:         rmse = float(genutil.statistics.rms(model, obs, axis=axis, weights="generate"))
  38      except Exception as err:

  45      try:
  46:         std = float(genutil.statistics.std(variable, axis=axis, weights="generate"))
  47      except Exception as err:

tests/integration/all_sets_results_test/prov/environment.yml:
  67    - g2clib=1.6.3=hdfa6d74_0
  68:   - genutil=8.2.1=py39hce5d2b2_2
  69    - geos=3.9.1=h9c3ff4c_2

  84    - ipython=7.30.1=py39hf3d152e_0
  85:   - ipython_genutils=0.2.0=py_1
  86    - isort=5.9.3=pyhd8ed1ab_0

MV2

279 results - 11 files

docs/source/dev_guide/_static/pre-commit-flow.svg:
  1: ⟪ 1459 characters skipped ⟫bXP/o6f3sQFdwju0J/ankzgi08Xh7RlFvVI06QRTQQbDxsbsyCHBRjDhwL44YuG1/7IHa7r+rDXTxtgf3THtUMcru3sk+ISKHreXElUkWjMqYxUqRl938ajx+pS3t/Y5lTZ2FPTOY7b1379M7WvcCM3F8Cgq7KHKUQTmW5Zzjs92LCuQVj/JVUPWDZdtd35eaRITSJaOMV2PdEOF5FMjNCM4ecavFD+Z4FckRLBOaFqwu5q/ObOyZGZzO/aEzG/3y0sJo2gAyBSuNplsEKtzXPJxYmOo11ZixDO/pXIkO98M+PE3E2wD1PMq+HiDPBMjrSz1nPQA5pwLItToA3XMSx6A5x1qVhIZDBLO3ByV9I/rvqLQYV1iad3BYcoyqu9NtSesnw39LXAx/3/f3U+DB708ay7by/oSZllCyHhh/jCiD6G7dMybn+uOaiM/lCkofA0Eg3H/41cR/AhSG53q3m1mtle6n29+G4v3zX0fxUG3+Y6yzquafWvfqJw==&lt;/diagram&gt;&lt;/mxfile&gt;"><defs/><g><path d="M 120 30 L 250 30 L 250 133.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 250 138.88 L 246.5 131.88 L 250 133.63 L 253.5 131.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http:https://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http:https://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 31

e3sm_diags/derivations/acme.py:
    6  
    7: import MV2
    8  import numpy as np

  407  
  408:     cld_bin_sum = MV2.sum(MV2.sum(cld_bin, axis=1), axis=0)
  409  

e3sm_diags/driver/annual_cycle_zonal_mean_driver.py:
    5  import cdutil
    6: import MV2
    7  from cdms2 import createAxis

  146          if month == "01":
  147:             var_ann_cycle: "TransientVariable" = MV2.zeros([12] + list(var.shape))
  148              var_ann_cycle.id = var.id

e3sm_diags/driver/cosp_histogram_driver.py:
   77              mv1 = test_data.get_climo_variable(var, season)
   78:             mv2 = ref_data.get_climo_variable(var, season)
   79  

   91                  )
   92:                 mv2_domain = utils.general.select_region(
   93:                     region, mv2, land_frac, ocean_frac, parameter
   94                  )

   99                  mv1_domain_mean = mean(mv1_domain)
  100:                 mv2_domain_mean = mean(mv2_domain)
  101:                 diff = mv1_domain_mean - mv2_domain_mean
  102  
  103                  mv1_domain_mean.id = var
  104:                 mv2_domain_mean.id = var
  105                  diff.id = var

  111                      parameter.current_set,
  112:                     mv2_domain_mean,
  113                      mv1_domain_mean,

  120                      mv1_domain,
  121:                     mv2_domain,
  122                      diff,

e3sm_diags/driver/lat_lon_driver.py:
    6  import cdms2
    7: import MV2
    8  

   89              mv1 = test_data.get_climo_variable(var, season)
   90:             mv2 = ref_data.get_climo_variable(var, season)
   91  

  101                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
  102:                 mv2 = MV2.masked_where(mv2 == -0.9, mv2)
  103              # The following should be moved to a derived variable.

  105                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
  106:                 mv2 = MV2.masked_where(mv2 > 1e20, mv2)
  107              if ref_name == "WILLMOTT" or ref_name == "CLOUDSAT":
  108                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
  109:                 mv2 = MV2.masked_where(mv2 == -999.0, mv2)
  110  

  123                      # unit convert from mm/day to cm.
  124:                     mv2 = (
  125:                         mv2 / days_season[season] / 0.1
  126                      )  # Convert cm to mm/day instead.
  127:                     mv2.units = "mm/day"
  128  
  129              # For variables with a z-axis.
  130:             if mv1.getLevel() and mv2.getLevel():
  131                  plev = parameter.plevs

  136                  )
  137:                 mv2_p = utils.general.convert_to_pressure_levels(
  138:                     mv2, plev, ref_data, var, season
  139                  )

  145                      ]
  146:                     mv2 = mv2_p[
  147                          ilev,

  154                          )
  155:                         mv2_domain = utils.general.select_region(
  156:                             region, mv2, land_frac, ocean_frac, parameter
  157                          )

  181                          # variables for calculating the difference.
  182:                         mv1_reg, mv2_reg = utils.general.regrid_to_lower_res(
  183                              mv1_domain,
  184:                             mv2_domain,
  185                              parameter.regrid_tool,

  188  
  189:                         diff = mv1_reg - mv2_reg
  190                          metrics_dict = create_metrics(
  191:                             mv2_domain, mv1_domain, mv2_reg, mv1_reg, diff
  192                          )

  218                              parameter.current_set,
  219:                             mv2_domain,
  220                              mv1_domain,

  227                              mv1_domain,
  228:                             mv2_domain,
  229                              diff,

  233              # For variables without a z-axis.
  234:             elif mv1.getLevel() is None and mv2.getLevel() is None:
  235                  for region in regions:

  239                      )
  240:                     mv2_domain = utils.general.select_region(
  241:                         region, mv2, land_frac, ocean_frac, parameter
  242                      )

  248                      # variables for calculating the difference.
  249:                     mv1_reg, mv2_reg = utils.general.regrid_to_lower_res(
  250                          mv1_domain,
  251:                         mv2_domain,
  252                          parameter.regrid_tool,

  258                          if ref_name == "WILLMOTT":
  259:                             mv2_reg = MV2.masked_where(
  260:                                 mv2_reg == mv2_reg.fill_value, mv2_reg
  261                              )
  262  
  263:                         land_mask = MV2.logical_or(mv1_reg.mask, mv2_reg.mask)
  264:                         mv1_reg = MV2.masked_where(land_mask, mv1_reg)
  265:                         mv2_reg = MV2.masked_where(land_mask, mv2_reg)
  266  
  267:                     diff = mv1_reg - mv2_reg
  268                      metrics_dict = create_metrics(
  269:                         mv2_domain, mv1_domain, mv2_reg, mv1_reg, diff
  270                      )

  294                          parameter.current_set,
  295:                         mv2_domain,
  296                          mv1_domain,

  303                          mv1_domain,
  304:                         mv2_domain,
  305                          diff,

e3sm_diags/driver/meridional_mean_2d_driver.py:
    4  import cdutil
    5: import MV2
    6  import numpy

   90              mv1 = test_data.get_climo_variable(var, season)
   91:             mv2 = ref_data.get_climo_variable(var, season)
   92  

   99              # For variables with a z-axis.
  100:             if mv1.getLevel() and mv2.getLevel():
  101                  # Since the default is now stored in MeridionalMean2dParameter,

  111                  )
  112:                 mv2_p = utils.general.convert_to_pressure_levels(
  113:                     mv2, plevs, ref_data, var, season
  114                  )

  116                  mv1_p = cdutil.averager(mv1_p, axis="y")
  117:                 mv2_p = cdutil.averager(mv2_p, axis="y")
  118  

  125                  # variables for calculating the difference.
  126:                 if len(mv1_p.getLongitude()) < len(mv2_p.getLongitude()):
  127                      mv1_reg = mv1_p

  135  
  136:                     data_shape = list(mv2_p.shape)
  137                      data_shape.append(1)
  138:                     mv2_reg = MV2.resize(mv2_p, data_shape)
  139:                     mv2_reg.setAxis(-1, lat)
  140:                     for i, ax in enumerate(mv2_p.getAxisList()):
  141:                         mv2_reg.setAxis(i, ax)
  142  
  143:                     mv2_reg = mv2_reg.regrid(grid, regridTool="regrid2")[..., 0]
  144                      # Apply the mask back, since crossSectionRegrid
  145                      # doesn't preserve the mask.
  146:                     mv2_reg = MV2.masked_where(mv2_reg == mv2_reg.fill_value, mv2_reg)
  147:                 elif len(mv1_p.getLongitude()) > len(mv2_p.getLongitude()):
  148:                     mv2_reg = mv2_p
  149:                     lev_out = mv2_p.getLevel()
  150:                     lon_out = mv2_p.getLongitude()
  151                      mv1_reg = mv1_p.crossSectionRegrid(lev_out, lon_out)

  160                      data_shape.append(1)
  161:                     mv1_reg = MV2.resize(mv1_p, data_shape)
  162                      mv1_reg.setAxis(-1, lat)

  168                      # doesn't preserve the mask.
  169:                     mv1_reg = MV2.masked_where(mv1_reg == mv1_reg.fill_value, mv1_reg)
  170                  else:
  171                      mv1_reg = mv1_p
  172:                     mv2_reg = mv2_p
  173  
  174:                 diff = mv1_reg - mv2_reg
  175:                 metrics_dict = create_metrics(mv2_p, mv1_p, mv2_reg, mv1_reg, diff)
  176  

  180                      parameter.current_set,
  181:                     mv2_p,
  182                      mv1_p,

  187                  utils.general.save_ncfiles(
  188:                     parameter.current_set, mv1_p, mv2_p, diff, parameter
  189                  )

  191              # For variables without a z-axis.
  192:             elif mv1.getLevel() is None and mv2.getLevel() is None:
  193                  raise RuntimeError(

e3sm_diags/driver/polar_driver.py:
    5  import cdms2
    6: import MV2
    7  

   78              mv1 = test_data.get_climo_variable(var, season)
   79:             mv2 = ref_data.get_climo_variable(var, season)
   80  

   90                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
   91:                 mv2 = MV2.masked_where(mv2 == -0.9, mv2)
   92              # The following should be moved to a derived variable.

   94                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
   95:                 mv2 = MV2.masked_where(mv2 > 1e20, mv2)
   96              if ref_name == "WILLMOTT" or ref_name == "CLOUDSAT":
   97                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
   98:                 mv2 = MV2.masked_where(mv2 == -999.0, mv2)
   99  

  112                      # unit convert from mm/day to cm.
  113:                     mv2 = (
  114:                         mv2 / days_season[season] / 0.1
  115                      )  # Convert cm to mm/day instead.
  116:                     mv2.units = "mm/day"
  117  
  118              # For variables with a z-axis.
  119:             if mv1.getLevel() and mv2.getLevel():
  120                  plev = parameter.plevs

  125                  )
  126:                 mv2_p = utils.general.convert_to_pressure_levels(
  127:                     mv2, plev, ref_data, var, season
  128                  )

  134                      ]
  135:                     mv2 = mv2_p[
  136                          ilev,

  144                          )
  145:                         mv2_domain = utils.general.select_region(
  146:                             region, mv2, land_frac, ocean_frac, parameter
  147                          )

  171                          # variables for calculating the difference.
  172:                         mv1_reg, mv2_reg = utils.general.regrid_to_lower_res(
  173                              mv1_domain,
  174:                             mv2_domain,
  175                              parameter.regrid_tool,

  179                          # Plotting
  180:                         diff = mv1_reg - mv2_reg
  181                          metrics_dict = create_metrics(
  182:                             mv2_domain, mv1_domain, mv2_reg, mv1_reg, diff
  183                          )

  187                              parameter.current_set,
  188:                             mv2_domain,
  189                              mv1_domain,

  196                              mv1_domain,
  197:                             mv2_domain,
  198                              diff,

  202              # For variables without a z-axis.
  203:             elif mv1.getLevel() is None and mv2.getLevel() is None:
  204                  for region in regions:

  209                      )
  210:                     mv2_domain = utils.general.select_region(
  211:                         region, mv2, land_frac, ocean_frac, parameter
  212                      )

  218                      # variables for calculating the difference.
  219:                     mv1_reg, mv2_reg = utils.general.regrid_to_lower_res(
  220                          mv1_domain,
  221:                         mv2_domain,
  222                          parameter.regrid_tool,

  228                          if ref_name == "WILLMOTT":
  229:                             mv2_reg = MV2.masked_where(
  230:                                 mv2_reg == mv2_reg.fill_value, mv2_reg
  231                              )
  232  
  233:                         land_mask = MV2.logical_or(mv1_reg.mask, mv2_reg.mask)
  234:                         mv1_reg = MV2.masked_where(land_mask, mv1_reg)
  235:                         mv2_reg = MV2.masked_where(land_mask, mv2_reg)
  236  
  237:                     diff = mv1_reg - mv2_reg
  238                      metrics_dict = create_metrics(
  239:                         mv2_domain, mv1_domain, mv2_reg, mv1_reg, diff
  240                      )

  244                          parameter.current_set,
  245:                         mv2_domain,
  246                          mv1_domain,

  253                          mv1_domain,
  254:                         mv2_domain,
  255                          diff,

e3sm_diags/driver/zonal_mean_2d_driver.py:
    6  import cdutil
    7: import MV2
    8  import numpy

  104              mv1 = test_data.get_climo_variable(var, season)
  105:             mv2 = ref_data.get_climo_variable(var, season)
  106  

  116                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
  117:                 mv2 = MV2.masked_where(mv2 == -0.9, mv2)
  118              # The following should be moved to a derived variable.

  120                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
  121:                 mv2 = MV2.masked_where(mv2 > 1e20, mv2)
  122              if ref_name == "WILLMOTT" or ref_name == "CLOUDSAT":
  123                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
  124:                 mv2 = MV2.masked_where(mv2 == -999.0, mv2)
  125  

  138                      # unit convert from mm/day to cm.
  139:                     mv2 = (
  140:                         mv2 / days_season[season] / 0.1
  141                      )  # Convert cm to mm/day instead.
  142:                     mv2.units = "mm/day"
  143  
  144              # For variables with a z-axis.
  145:             if mv1.getLevel() and mv2.getLevel():
  146                  # Since the default is now stored in `default_plevs`,

  157                  )
  158:                 mv2_p = utils.general.convert_to_pressure_levels(
  159:                     mv2, plevs, ref_data, var, season
  160                  )

  162                  # variables for calculating the difference.
  163:                 mv1_p_reg, mv2_p_reg = utils.general.regrid_to_lower_res(
  164                      mv1_p,
  165:                     mv2_p,
  166                      parameter.regrid_tool,

  168                  )
  169:                 diff_p = mv1_p_reg - mv2_p_reg
  170                  diff = cdutil.averager(diff_p, axis="x")

  172                  mv1_p = cdutil.averager(mv1_p, axis="x")
  173:                 mv2_p = cdutil.averager(mv2_p, axis="x")
  174  
  175:                 # Make sure mv1_p_reg and mv2_p_reg have same mask
  176:                 mv1_p_reg = mv2_p_reg + diff_p
  177:                 mv2_p_reg = mv1_p_reg - diff_p
  178  
  179                  mv1_reg = cdutil.averager(mv1_p_reg, axis="x")
  180:                 mv2_reg = cdutil.averager(mv2_p_reg, axis="x")
  181  

  186  
  187:                 # Use mv2_p and mv1_p on the original horizonal grids for visualization and their own metrics
  188:                 # Use mv2_reg and mv1_reg for rmse and correlation coefficient calculation
  189:                 metrics_dict = create_metrics(mv2_p, mv1_p, mv2_reg, mv1_reg, diff)
  190  

  194                      parameter.current_set,
  195:                     mv2_p,
  196                      mv1_p,

  201                  utils.general.save_ncfiles(
  202:                     parameter.current_set, mv1_p, mv2_p, diff, parameter
  203                  )

  205              # For variables without a z-axis.
  206:             elif mv1.getLevel() is None and mv2.getLevel() is None:
  207                  for region in regions:

  212                      )
  213:                     mv2_domain = utils.general.select_region(
  214:                         region, mv2, land_frac, ocean_frac, parameter
  215                      )

  221                      # variables for calculating the difference.
  222:                     mv1_reg, mv2_reg = utils.general.regrid_to_lower_res(
  223                          mv1_domain,
  224:                         mv2_domain,
  225                          parameter.regrid_tool,

  231                          if ref_name == "WILLMOTT":
  232:                             mv2_reg = MV2.masked_where(
  233:                                 mv2_reg == mv2_reg.fill_value, mv2_reg
  234                              )
  235:                         land_mask = MV2.logical_or(mv1_reg.mask, mv2_reg.mask)
  236:                         mv1_reg = MV2.masked_where(land_mask, mv1_reg)
  237:                         mv2_reg = MV2.masked_where(land_mask, mv2_reg)
  238  
  239:                     diff = mv1_reg - mv2_reg
  240                      metrics_dict = create_metrics(
  241:                         mv2_domain, mv1_domain, mv2_reg, mv1_reg, diff
  242                      )

  246                          parameter.current_set,
  247:                         mv2_domain,
  248                          mv1_domain,

  255                          mv1_domain,
  256:                         mv2_domain,
  257                          diff,

e3sm_diags/driver/zonal_mean_xy_driver.py:
    4  import cdutil
    5: import MV2
    6  import numpy

   15  
   16: def regrid_to_lower_res_1d(mv1, mv2):
   17      """Regrid 1-D transient variable toward lower resolution of two variables."""
   18  
   19:     if mv1 is None or mv2 is None:
   20          return None

   23      axis1 = mv1.getAxisList()[0]
   24:     axis2 = mv2.getAxisList()[0]
   25      if axis1[-1] < axis1[0]:

   27      if axis2[-1] < axis2[0]:
   28:         mv2 = mv2[::-1]
   29      axis1 = mv1.getAxisList()[0]
   30:     axis2 = mv2.getAxisList()[0]
   31  
   32      if len(axis1) <= len(axis2):
   33:         missing = mv2.get_fill_value()
   34          mv1_reg = mv1
   35:         b0 = numpy.interp(axis1[:], axis2[:], mv2[:], left=missing, right=missing)
   36          b0_mask = numpy.interp(
   37:             axis1[:], axis2[:], mv2.mask[:], left=missing, right=missing
   38          )
   39:         mv2_reg = cdms2.createVariable(
   40              b0,

   60          )
   61:         mv2_reg = mv2
   62  
   63:     return mv1_reg, mv2_reg
   64  

  115              mv1 = test_data.get_climo_variable(var, season)
  116:             mv2 = ref_data.get_climo_variable(var, season)
  117  

  127                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
  128:                 mv2 = MV2.masked_where(mv2 == -0.9, mv2)
  129              # The following should be moved to a derived variable.

  131                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
  132:                 mv2 = MV2.masked_where(mv2 > 1e20, mv2)
  133              if ref_name == "WILLMOTT" or ref_name == "CLOUDSAT":
  134                  # This is cdms2 fix for bad mask, Denis' fix should fix this.
  135:                 mv2 = MV2.masked_where(mv2 == -999.0, mv2)
  136  

  149                      # unit convert from mm/day to cm.
  150:                     mv2 = (
  151:                         mv2 / days_season[season] / 0.1
  152                      )  # Convert cm to mm/day instead.
  153:                     mv2.units = "mm/day"
  154  
  155              # For variables with a z-axis.
  156:             if mv1.getLevel() and mv2.getLevel():
  157                  plev = parameter.plevs

  162                  )
  163:                 mv2_p = utils.general.convert_to_pressure_levels(
  164:                     mv2, plev, test_data, var, season
  165                  )

  171                      ]
  172:                     mv2 = mv2_p[
  173                          ilev,

  178                          mv1_zonal = cdutil.averager(mv1, axis="x")
  179:                         mv2_zonal = cdutil.averager(mv2, axis="x")
  180  

  182                          # variables for calculating the difference.
  183:                         mv1_reg, mv2_reg = regrid_to_lower_res_1d(mv1_zonal, mv2_zonal)
  184  
  185:                         diff = mv1_reg - mv2_reg
  186                          parameter.output_file = "-".join(

  209                              parameter.current_set,
  210:                             mv2_zonal,
  211                              mv1_zonal,

  218                              mv1_zonal,
  219:                             mv2_zonal,
  220                              diff,

  224              # For variables without a z-axis.
  225:             elif mv1.getLevel() is None and mv2.getLevel() is None:
  226                  for region in regions:

  228                      mv1_zonal = cdutil.averager(mv1, axis="x")
  229:                     mv2_zonal = cdutil.averager(mv2, axis="x")
  230  
  231:                     mv1_reg, mv2_reg = regrid_to_lower_res_1d(mv1_zonal, mv2_zonal)
  232  
  233:                     diff = mv1_reg - mv2_reg
  234  

  241                          parameter.current_set,
  242:                         mv2_zonal,
  243                          mv1_zonal,

  250                          mv1_zonal,
  251:                         mv2_zonal,
  252                          diff,

e3sm_diags/driver/utils/diurnal_cycle.py:
    1: import MV2
    2  import numpy

  134          # Save phase, amplitude, and mean for the first homonic,
  135:         amplitude = MV2.zeros((nlat, nlon))
  136          amplitude[:, :] = maxvalue[0]

  142  
  143:         maxtime = MV2.zeros((nlat, nlon))
  144          maxtime[:, :] = tmax[0]

  150  
  151:         cmean = MV2.zeros((nlat, nlon))
  152          cmean[:, :] = cycmean

e3sm_diags/driver/utils/general.py:
   10  import genutil
   11: import MV2
   12  

  155          var_plv[:] = var_plv[:] / 100.0  # convert Pa to mb
  156:     levels_orig = MV2.array(var_plv[:])
  157      levels_orig.setAxis(0, var_plv)

  240  
  241: def regrid_to_lower_res(mv1, mv2, regrid_tool, regrid_method):
  242      """Regrid transient variable toward lower resolution of two variables."""

  244      axes1 = mv1.getAxisList()
  245:     axes2 = mv2.getAxisList()
  246  

  251          mv1_reg = mv1
  252:         mv2_reg = mv2.regrid(
  253              mv_grid, regridTool=regrid_tool, regridMethod=regrid_method
  254          )
  255:         mv2_reg.units = mv2.units
  256  
  257      else:
  258:         mv_grid = mv2.getGrid()
  259:         mv2_reg = mv2
  260          mv1_reg = mv1.regrid(

  264  
  265:     return mv1_reg, mv2_reg
  266  

[chengzhuzhang]

Thank you for getting this process started! Starting this exploration early will also inform xcdat development.