Open multiple files using xcdat #926

lee1043 · 2023-04-26T16:38:50Z

Update xcdat version
Use xcdat open_mfdataset instead of xcdat_open that was defined locally in PMP
- This update allows use of wildcard ('*' or '?') as a part of string giving path of input files.
- E.g., modpath="/MY/FILES/*.nc"

…_open` defined locally in PMP

acordonez · 2023-04-26T16:47:56Z

I'll give this a test run today

lee1043 · 2023-04-26T18:10:30Z

@tomvothecoder As discussed at the xcdat meeting, CI/CD build test is failing and with the log that complains it cannot find xcdat 0.5.0. Could you please take a look?

Set up Conda Environment

Run conda-incubator/setup-miniconda@v2
Gathering Inputs...
Creating bootstrap condarc file in /home/runner/.condarc...
Ensuring installer...
Setup environment variables...
Parsing environment...
Configuring conda package cache...
Applying initial configuration...
Initializing conda shell integration...
Adding tools to 'base' env...
Ensuring environment...
Updating 'pcmdi_metrics_dev' env from conda env update...
  /usr/share/miniconda/condabin/conda env update --name pcmdi_metrics_dev --file conda-env/dev.yml
  Collecting package metadata (repodata.json): ...working... done
  Solving environment: ...working... failed
  Warning: 
  ResolvePackageNotFound: 
    - xcdat=0.5.0
  
  
  
  ResolvePackageNotFound: 
    - xcdat=0.5.0
  

Error: The process '/usr/share/miniconda/condabin/conda' failed with exit code 1

- Add step to install local build of pcmdi_metrics in GH Actions conda env

tomvothecoder · 2023-04-26T22:08:43Z

@tomvothecoder As discussed at the xcdat meeting, CI/CD build test is failing and with the log that complains it cannot find xcdat 0.5.0. Could you please take a look?

Set up Conda Environment

Run conda-incubator/setup-miniconda@v2
Gathering Inputs...
Creating bootstrap condarc file in /home/runner/.condarc...
Ensuring installer...
Setup environment variables...
Parsing environment...
Configuring conda package cache...
Applying initial configuration...
Initializing conda shell integration...
Adding tools to 'base' env...
Ensuring environment...
Updating 'pcmdi_metrics_dev' env from conda env update...
  /usr/share/miniconda/condabin/conda env update --name pcmdi_metrics_dev --file conda-env/dev.yml
  Collecting package metadata (repodata.json): ...working... done
  Solving environment: ...working... failed
  Warning: 
  ResolvePackageNotFound: 
    - xcdat=0.5.0
  
  
  
  ResolvePackageNotFound: 
    - xcdat=0.5.0
  

Error: The process '/usr/share/miniconda/condabin/conda' failed with exit code 1

Hi @lee1043 I'm looking at this issue right now. Ignore my many commits for trying to fix this issue.

I think the root cause might be related to the GH Actions conda environment using a cached version of the packages.

- Update build workflow dependencies - Update build workflow conda caching mechanism based on date and environment file

acordonez · 2023-04-26T22:27:15Z

@lee1043 This fails for me using the demo dataset, where there is a single input file per model. Here's the error text, I can send the full log if that's useful:

--- prepare mean climate metrics calculation ---
--- start mean climate metrics calculation ---
varname: rlut
level: None
reference_data_set (all):  ['alternate1', 'default']
ref: alternate1
ref_data_full_path: demo_data/pmpobs_v1.0/rlut/CERES-EBAF-4-0/v20210804/rlut_mon_CERES-EBAF-4-0_PCMDI_gn.200301-201812.AC.v20210804.nc
Traceback (most recent call last):
  File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_dev/bin/mean_climate_driver.py", line 183, in <module>
    ds_ref = load_and_regrid(data_path=ref_data_full_path, varname=varname, level=level, t_grid=t_grid, decode_times=False, regrid_tool=regrid_tool, debug=debug)
  File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_dev/lib/python3.9/site-packages/pcmdi_metrics/mean_climate/lib/load_and_regrid.py", line 25, in load_and_regrid
    ds = xc.open_mfdataset(data_path, data_var=varname_in_file, decode_times=decode_times)  # NOTE: decode_times=False will be removed once obs4MIP written using xcdat
  File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_dev/lib/python3.9/site-packages/xcdat/dataset.py", line 205, in open_mfdataset
    ds = xr.open_mfdataset(
  File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_dev/lib/python3.9/site-packages/xarray/backends/api.py", line 947, in open_mfdataset
    raise OSError("no files to open")
OSError: no files to open

tomvothecoder

The GH Actions build is now passing. I updated the CI/CD conda environment to correctly update the cache instead of referencing old package versions.

Summary of Changes:

build_workflow.yml

Bumped dependency versions
Updated conda caching to cache entire environment and refresh cache every 24 hours
Update conda environment build step to use mamba for speed
Add "Install pcmdi_metrics" step to install local branch version of the package to run tests against

dev.yml

Bump dependencies versions -- these should be updated periodically (recommended before a new software version release)

lee1043 · 2023-04-27T01:16:00Z

@lee1043 This fails for me using the demo dataset, where there is a single input file per model. Here's the error text, I can send the full log if that's useful:

--- prepare mean climate metrics calculation ---
--- start mean climate metrics calculation ---
varname: rlut
level: None
reference_data_set (all):  ['alternate1', 'default']
ref: alternate1
ref_data_full_path: demo_data/pmpobs_v1.0/rlut/CERES-EBAF-4-0/v20210804/rlut_mon_CERES-EBAF-4-0_PCMDI_gn.200301-201812.AC.v20210804.nc
Traceback (most recent call last):
  File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_dev/bin/mean_climate_driver.py", line 183, in <module>
    ds_ref = load_and_regrid(data_path=ref_data_full_path, varname=varname, level=level, t_grid=t_grid, decode_times=False, regrid_tool=regrid_tool, debug=debug)
  File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_dev/lib/python3.9/site-packages/pcmdi_metrics/mean_climate/lib/load_and_regrid.py", line 25, in load_and_regrid
    ds = xc.open_mfdataset(data_path, data_var=varname_in_file, decode_times=decode_times)  # NOTE: decode_times=False will be removed once obs4MIP written using xcdat
  File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_dev/lib/python3.9/site-packages/xcdat/dataset.py", line 205, in open_mfdataset
    ds = xr.open_mfdataset(
  File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_dev/lib/python3.9/site-packages/xarray/backends/api.py", line 947, in open_mfdataset
    raise OSError("no files to open")
OSError: no files to open

@acordonez thank you for checking the demo. Can you please confirm you have the input file, demo_data/pmpobs_v1.0/rlut/CERES-EBAF-4-0/v20210804/rlut_mon_CERES-EBAF-4-0_PCMDI_gn.200301-201812.AC.v20210804.nc, otherwise you will need to run demo 0 to secure the data. May I have the full log as well?

lee1043 · 2023-04-27T01:17:04Z

@tomvothecoder Thank you very much for making the build test working! That is a huge help!

acordonez · 2023-04-27T22:04:39Z

@lee1043 you're right, once I updated the obs demo data this is working now. I don't have any xml clims to test, but I can also run that if there is some data you can point me to.

lee1043 · 2023-04-27T22:32:31Z

@acordonez thank you for checking. I think it is okay to merge. Merging this now.

update xcdat version and use xcdat open_mfdataset instead of `xcdat…

5fd86d3

…_open` defined locally in PMP

lee1043 added the enhancement label Apr 26, 2023

lee1043 added this to the 3.1 milestone Apr 26, 2023

lee1043 requested review from gleckler1 and acordonez April 26, 2023 16:38

lee1043 self-assigned this Apr 26, 2023

lee1043 linked an issue Apr 26, 2023 that may be closed by this pull request

computing climatolologies spanning multiple files #924

Closed

gleckler1 approved these changes Apr 26, 2023

View reviewed changes

clean up

2eb2d49

Replace conda with mamba in GH Actions build

b41089c

- Add step to install local build of pcmdi_metrics in GH Actions conda env

Add CI conda env and bump deps in dev env

b86fc64

- Update build workflow dependencies - Update build workflow conda caching mechanism based on date and environment file

tomvothecoder force-pushed the 924_lee1043_xcdat_open_multiple_files branch from e5ad015 to b86fc64 Compare April 26, 2023 22:15

tomvothecoder added 2 commits April 26, 2023 15:19

Update order of conda caching steps in CI/CD

691f4d2

Remove ci.yml

e360928

tomvothecoder reviewed Apr 26, 2023

View reviewed changes

lee1043 merged commit cb6f773 into main Apr 27, 2023

lee1043 deleted the 924_lee1043_xcdat_open_multiple_files branch April 27, 2023 22:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Open multiple files using xcdat #926

Open multiple files using xcdat #926

lee1043 commented Apr 26, 2023

acordonez commented Apr 26, 2023

lee1043 commented Apr 26, 2023

tomvothecoder commented Apr 26, 2023

acordonez commented Apr 26, 2023

tomvothecoder left a comment

lee1043 commented Apr 27, 2023

lee1043 commented Apr 27, 2023

acordonez commented Apr 27, 2023

lee1043 commented Apr 27, 2023

Open multiple files using xcdat #926

Open multiple files using xcdat #926

Conversation

lee1043 commented Apr 26, 2023

acordonez commented Apr 26, 2023

lee1043 commented Apr 26, 2023

tomvothecoder commented Apr 26, 2023

acordonez commented Apr 26, 2023

tomvothecoder left a comment

Choose a reason for hiding this comment

lee1043 commented Apr 27, 2023

lee1043 commented Apr 27, 2023

acordonez commented Apr 27, 2023

lee1043 commented Apr 27, 2023