Update mov notebook

[lee1043]

• Update Modes of Variability notebook and code
• Added feature:
  • Add PDO in the notebook
  • In case landmask of OBS not given but need to do landmask, use estimated landmask for OBS

[lee1043]

@msahn could you please take this notebook look and see if that makes sense to you?

@acordonez could you please see if the new notebook is reproducible on your end? This will need to run the updated demo0 from this branch in advance.

[msahn]

@msahn could you please take this notebook look and see if that makes sense to you?

I have tested the notebook in Gates. In my test, an error appears in 3.2.1 Run Metircs. Below is the copy of the error message. It seem like the observation data (ts_mon_20CR_PCMDI_gn_187101-201212.nc) is not downloaded by the current Demo_0. After it is fixed, I will test the rest part of the notebook.

Traceback (most recent call last):
File "/home/ahn6/anaconda3/envs/pmp_v20220110/bin/variability_modes_driver.py", line 4, in
import('pkg_resources').run_script('pcmdi-metrics===v1.2.1-1165-g43e256e', 'variability_modes_driver.py')
File "/home/ahn6/anaconda3/envs/pmp_v20220110/lib/python3.9/site-packages/pkg_resources/init.py", line 656, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/ahn6/anaconda3/envs/pmp_v20220110/lib/python3.9/site-packages/pkg_resources/init.py", line 1460, in run_script
exec(script_code, namespace, namespace)
File "/home/ahn6/anaconda3/envs/pmp_v20220110/lib/python3.9/site-packages/pcmdi_metrics-v1.2.1_1165_g43e256e-py3.9.egg/EGG-INFO/scripts/variability_modes_driver.py", line 321, in
File "/home/ahn6/anaconda3/envs/pmp_v20220110/lib/python3.9/site-packages/pcmdi_metrics-v1.2.1_1165_g43e256e-py3.9.egg/pcmdi_metrics/variability_mode/lib/lib_variability_mode.py", line 57, in read_data_in
File "/home/ahn6/anaconda3/envs/pmp_v20220110/lib/python3.9/site-packages/cdms2/dataset.py", line 514, in openDataset
raise FileNotFoundError(path)
FileNotFoundError: /home/ahn6/PCMDI/branch/lee1043_update_mov_notebook/pcmdi_metrics/doc/jupyter/Demo/demo_data_test/obs4MIPs_PCMDI_monthly/NOAA-ESRL-PSD/20CR/mon/ts/gn/v20210727/ts_mon_20CR_PCMDI_gn_187101-201212.nc

[acordonez]

The notebook ran successfully after I changed the date range in one of the result file names. I left some other suggestions, but changing this file name was the only roadblock issue.

[acordonez]

@msahn Did you run Demo 0 again in this branch? I was able to get the new obs data just now using Demo 0.

[msahn]

@msahn Did you run Demo 0 again in this branch? I was able to get the new obs data just now using Demo 0.

Yes, I ran Demo 0 in this branch. Below is the list of OBS data downloaded. The list does not include "obs4MIPs_PCMDI_monthly/NOAA-ESRL-PSD/20CR/mon/ts/gn/v20210727/ts_mon_20CR_PCMDI_gn_187101-201212.nc" that is needed for PDO.

https://pcmdiweb.llnl.gov/pss/pmpdata/
9d92d486fe3963b29f4d4926e47eab8b CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.pr.198101-200512.AC.v20200426.nc
16fb29fa02cc8c68e170502bca145640 CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc
71aea66241de722a75e26d8fe55a1a22 CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.zg.198101-200512.AC.v20200426.nc
44cce7b8a402a1004ca53565ab378615 CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.pr.198101-200512.AC.v20200426.nc
40b2bfa71a3b7d2febb55652ef551001 CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc
8bb70ae9036280c79d4e305b801d8066 CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.zg.198101-200512.AC.v20200426.nc
1abfd5cbcceac61ac51fed2e2c398eed CMIP5_demo_clims/cmip6.historical.MCM-UA-1-0.r1i1p1f1.mon.zg.198101-200512.AC.v20201119.nc
d5e86e3680cce1a0004bbe9663d20cf6 CMIP5_demo_data/psl_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
dd810f1de8a5db5cddf6a0ab22717cdc CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc
63ea73e990aa7f2f53de8b493eb3e051 CMIP5_demo_data/cmip5.amip.ACCESS1-0.sftlf.nc
aa2384a8957af8ee6e652fb6a27f1f4a CMIP5_demo_data/cmip5.historical.GISS-E2-H.sftlf.nc
2c402715c026eb1d39ec8e450857ef6e CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
ce9aa736ee548e3dc20749de95b6f3fb CMIP5_demo_timeseries/historical/atmos/day/pr/pr_day_GISS-E2-H_historical_r6i1p1_20000101-20051231.nc
2e93205b96244292b10a666fd30ea7ea obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-0/v20210804/rlut_mon_CERES-EBAF-4-0_PCMDI_gn.200301-201812.AC.v20210804.nc
9f3e08d6a4c2d62bce80fcd7d5ed16ee obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-1/v20210804/rlut_mon_CERES-EBAF-4-1_PCMDI_gn.200301-201812.AC.v20210804.nc
3792901034585d3d495722f10a0dfecb obs4MIPs_PCMDI_clims/pr/GPCP-2-3/v20210804/pr_mon_GPCP-2-3_PCMDI_gn.200301-201812.AC.v20210804.nc
46535764dd4c32912944678eb0aa65c3 obs4MIPs_PCMDI_clims/zg/ERA-INT/v20210804/zg_mon_ERA-INT_PCMDI_gn.200301-201812.AC.v20210804.nc
d8fb93f31ff4a6370ccee93db82af86c obs4MIPs_PCMDI_monthly/ECMWF/ERA-INT/mon/zg/gn/v20210727/zg_mon_ERA-INT_PCMDI_gn_198901-201001.nc
61a639f0c8f3d11801352f37702aeaba obs4MIPs_PCMDI_monthly/NASA-LaRC/CERES-EBAF-4-1/mon/rlut/gn/v20210727/rlut_mon_CERES-EBAF-4-1_PCMDI_gn_200301-201812.nc
0877f014868b83547448f96c3e7c83e9 obs4MIPs_PCMDI_monthly/NOAA-NCEI/GPCP-2-3/mon/pr/gn/v20210727/pr_mon_GPCP-2-3_PCMDI_gn_197901-201907.nc
570ce90b3afd1d0b31690ae5dbe32d31 obs4MIPs_PCMDI_monthly/NOAA-ESRL-PSD/20CR/mon/psl/gn/v20210727/psl_mon_20CR_PCMDI_gn_187101-201212.nc
99c8691e0f615dc4d79b4fb5e926cc76 obs4MIPs_PCMDI_monthly/MOHC/HadISST-1-1/mon/ts/gn/v20210727/ts_mon_HadISST-1-1_PCMDI_gn_187001-201907.nc
10496775dae2b5f65fe95b064c1efd42 obs4MIPs_PCMDI_daily/NASA-JPL/GPCP-1-3/day/pr/gn/latest/pr_day_GPCP-1-3_PCMDI_gn_19961002-20170101.nc
eab076619d05c886648f33d507f4a721 misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc
3214c2480d017662d78ae7e50542beaa misc_demo_data/fx/sftlf.GPCP-IP.1x1.nc

[lee1043]

@msahn That is strange that the 20CR ts file was not there. But I see obs4MIPs_PCMDI_monthly/MOHC/HadISST-1-1/mon/ts/gn/v20210727/ts_mon_HadISST-1-1_PCMDI_gn_187001-201907.nc file is included and I think I can revise the notebook to use this data.

@acordonez is this file also downloaded on your end?

[acordonez]

@lee1043 @msahn I had an old copy of it actually! So that is why the notebook worked.

[lee1043]

@acordonez thanks for noticing that. Maybe I was having the old set too. Thanks @msahn for spotting it out. I will update the MoV notebook to use HadSST for PDO's reference dataset -- which is even better because in that way it is consistent to the reference paper.

[lee1043]

@msahn I updated the reference data to HadISST, could you try rerun the notebook to see if that work for you? Cell [27] is needed to be revised accordingly to use the HadISST nc file.

[lee1043]

@msahn I was able to run conda and jupyter notebook on my LOFT laptop so have updated the notebook to use HadISST for PDO. Could you check if the notebook work as expected?

[msahn]

@msahn I updated the reference data to HadISST, could you try rerun the #853 to see if that work for you? Cell [27] is needed to be revised accordingly to use the HadISST nc file.
@msahn I was able to run conda and jupyter notebook on my LOFT laptop so have updated the notebook to use HadISST for PDO. Could you check if the notebook work as expected?

I have tested the notebook again on Gates. Now it is working well! I think the notebook is very well organized, so it would be very useful documentation.

I have a minor suggestion as below.
In "Quick analysis" part, you may add a sub title something like "Comparison between CBF and traditional EOF approaches".

[lee1043]

I have tested the notebook again on Gates. Now it is working well! I think the notebook is very well organized, so it would be very useful documentation.

@msahn that's an excellent news! Thanks for checking it.

I have a minor suggestion as below. In "Quick analysis" part, you may add a sub title something like "Comparison between CBF and traditional EOF approaches".

@msahn great suggestion, added.