Diurnal demo (#616)

* renamed diurnal scripts * Half way thee * cleaned up download/prep notebook, some more diurnal * further down the path * finished follwoing diagram in demo * updated diagram * flake8 * no need to run mac py2, and also no need to run coverage on mac py3, it times out anyway * forcing 3.1.2 because mac py3 has issues * turning off cache * nighlties still tag as 3.1.2 * Removed forced version of cdms2 @muryanto1 seems to have fixed this. Put Mac py2 back in as well * Nightlies are less 3.1.3 * Nightlies still fail this... * Update meta.yaml.in Update recipe so people do not get a bad cdms2 @Muryanto 1 let remove this when cdms2 is fixed
PCMDI · Sep 13, 2019 · db681b7 · db681b7
1 parent f2323c5
commit db681b7
Show file tree

Hide file tree

Showing 24 changed files with 1,077 additions and 180 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -16,7 +16,7 @@ aliases:
  - &create_conda_env
  name: create_conda_env
  environment:
- PKGS: "vcs vcsaddons mesalib matplotlib scipy cia testsrunner 'proj4<5' 'vtk-cdat>8.1' libnetcdf=4.6.2"
+ PKGS: "'cdms2<3.1.2.2019' vcs vcsaddons mesalib matplotlib scipy cia testsrunner 'proj4<5' 'vtk-cdat>8.1' libnetcdf=4.6.2"
  CHANNELS: "-c cdat/label/nightly -c conda-forge -c pcmdi"
  command: |
  if [ -d ${HOME}/miniconda ]; then
@@ -102,9 +102,9 @@ jobs:
  FFMPEG: "'ffmpeg>4' 'libpng>1.6.34'"
  steps:
  - checkout
-  - restore_cache:
-   keys:
-   - macos_py2_2019-06-01
+ #- restore_cache:
+ # keys:
+ # - macos_py2_2019-06-01
  - run: *create_conda_env
  - save_cache:
  key: macos_py2_2019-06-01
@@ -126,20 +126,17 @@ jobs:
  OS: "osx-64"
  PY_VER: "py3"
  FFMPEG: "'ffmpeg>4' 'libpng>1.6.34'"
- COVERAGE: "-c tests/coverage.json --coverage-from-egg"
- COVERAGE_PKGS: "coverage coveralls"
  steps:
  - checkout
-  - restore_cache:
- keys:
- - macos_py3_2019-06-01
+ #- restore_cache:
+ # keys:
+ # - macos_py3_2019-06-01
  - run: *create_conda_env
  - save_cache:
  key: macos_py3_2019-06-01
  paths: /Users/distiller/miniconda
  - run: *setup_pmp
  - run: *run_pmp_tests
- - run: *run_coveralls
  - store_artifacts:
  path: tests_html
  destination: tests_html
@@ -157,9 +154,9 @@ jobs:
  FFMPEG: "ffmpeg"
  steps:
  - checkout
-  - restore_cache:
-   keys:
-   - linux_py2_2019-06-01
+ #- restore_cache:
+ # keys:
+ # - linux_py2_2019-06-01
  - run: *create_conda_env
  - save_cache:
  key: linux_py2_2019-06-01
@@ -185,9 +182,9 @@ jobs:
  COVERAGE_PKGS: "coverage coveralls"
  steps:
  - checkout
-  - restore_cache:
-   keys:
-   - linux_py3_2019-06-01
+ #- restore_cache:
+ # keys:
+ # - linux_py3_2019-06-01
  - run: *create_conda_env
  - save_cache:
  key: linux_py3_2019-06-01
@@ -208,10 +205,8 @@ workflows:
  version: 2
  pcmdi_metrics:
  jobs:
+ - macos_pmp_py3
  - macos_pmp_py2
- - macos_pmp_py3:
- requires:
- - macos_pmp_py2
  - linux_pmp_py2
  - linux_pmp_py3:
  requires:

diff --git a/doc/Diurnal Cycle Diagram.pdf b/doc/Diurnal Cycle Diagram.pdf
diff --git a/doc/Diurnal Cycle Diagram.pptx b/doc/Diurnal Cycle Diagram.pptx
diff --git a/doc/jupyter/Demo/Demo_0_download_data.ipynb b/doc/jupyter/Demo/Demo_0_download_data.ipynb
@@ -4,7 +4,7 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "# Download Data\n",
+ "# Download Data and Prepare Demos\n",
  "\n",
  "This Notebook setup the data for the other demos"
  ]
@@ -26,9 +26,12 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "Now let's retrieve these sample files locally, please edit the foloowing to match a good location on your system.\n",
+ "Now let's retrieve these sample files locally, please edit the followings to match a good location on your system.\n",
  "\n",
- "You will need to edit the subsequent tutorials to match this path"
+ "You will need to edit the subsequent tutorials to match this path.\n",
+ "\n",
+ "`demo_data_directory` is a path where you would like to download our demo data\n",
+ "`demo_output_directory` is a path where you would like the demos output to be dumped."
  ]
  },
  {
@@ -37,7 +40,10 @@
  "metadata": {},
  "outputs": [],
  "source": [
- "demo_data_directory = \"demo_data\""
+ "# This is where you will be downloading the sample_data\n",
+ "demo_data_directory = \"demo_data\"\n",
+ "# this line is where your output will be stored\n",
+ "demo_output_directory = \"demo_output\""
  ]
  },
  {
@@ -50,7 +56,13 @@
  "output_type": "stream",
  "text": [
  "MD5: data_files.txt\n",
- "Downloading: 'PCMDIobs2.0/atmos/mon/pr/GPCP-2-3/gn/v20190301//pr_mon_GPCP-2-3_BE_gn_197901-201803.nc' from 'https://pcmdiweb.llnl.gov/pss/pmpdata/' in: demo_data/PCMDIobs2.0/atmos/mon/pr/GPCP-2-3/gn/v20190301//pr_mon_GPCP-2-3_BE_gn_197901-201803.nc\n"
+ "Downloading: 'PCMDIobs2.0/atmos/mon/pr/GPCP-2-3/gn/v20190301//pr_mon_GPCP-2-3_BE_gn_197901-201803.nc' from 'https://pcmdiweb.llnl.gov/pss/pmpdata/' in: demo_data/PCMDIobs2.0/atmos/mon/pr/GPCP-2-3/gn/v20190301//pr_mon_GPCP-2-3_BE_gn_197901-201803.nc\n",
+ "Downloading: 'pmpobs_v1.0/atm/mo/rlut/CERES/ac/rlut_CERES_000001-000012_ac.nc' from 'https://pcmdiweb.llnl.gov/pss/pmpdata/' in: demo_data/pmpobs_v1.0/atm/mo/rlut/CERES/ac/rlut_CERES_000001-000012_ac.nc\n",
+ "Downloading: 'example_data/atm/mo/rlut/ac/CMIP5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20190225.nc' from 'https://pcmdiweb.llnl.gov/pss/pmpdata/' in: demo_data/example_data/atm/mo/rlut/ac/CMIP5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20190225.nc\n",
+ "Downloading: 'example_data/atm/mo/rlut/ac/CMIP5.historical.CSIRO-Mk3-6-0.r1i1p1.mon.rlut.198101-200512.AC.v20190225.nc' from 'https://pcmdiweb.llnl.gov/pss/pmpdata/' in: demo_data/example_data/atm/mo/rlut/ac/CMIP5.historical.CSIRO-Mk3-6-0.r1i1p1.mon.rlut.198101-200512.AC.v20190225.nc\n",
+ "Downloading: 'example_data/atm/mo/pr/CMIP5.CMIP.historical.CNRM-CERFACS.CNRM-CM5-2.r1i1p1.mon.pr.atmos.glb-2d-gu.v20130401.0000000.0.nc' from 'https://pcmdiweb.llnl.gov/pss/pmpdata/' in: demo_data/example_data/atm/mo/pr/CMIP5.CMIP.historical.CNRM-CERFACS.CNRM-CM5-2.r1i1p1.mon.pr.atmos.glb-2d-gu.v20130401.0000000.0.nc\n",
+ "Downloading: 'example_data/atm/mo/pr/CMIP5.CMIP.historical.NSF-DOE-NCAR.CESM1-WACCM.r1i1p1.mon.pr.atmos.glb-2d-gu.v20130314.0000000.0.nc' from 'https://pcmdiweb.llnl.gov/pss/pmpdata/' in: demo_data/example_data/atm/mo/pr/CMIP5.CMIP.historical.NSF-DOE-NCAR.CESM1-WACCM.r1i1p1.mon.pr.atmos.glb-2d-gu.v20130314.0000000.0.nc\n",
+ "Downloading: 'example_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc' from 'https://pcmdiweb.llnl.gov/pss/pmpdata/' in: demo_data/example_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc\n"
  ]
  }
  ],
@@ -60,13 +72,65 @@
  "cdat_info.download_sample_data_files(\"data_files.txt\", demo_data_directory)"
  ]
  },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Preparing parameter file: basic_param.py\n",
+ "Preparing parameter file: basic_diurnal_std_daily_mean.py\n",
+ "Preparing parameter file: basic_diurnal_fourierAllGrid.py\n",
+ "Preparing parameter file: basic_monsoon_wang_param.py\n",
+ "Preparing parameter file: basic_diurnal_std_hourly_mean.py\n",
+ "Preparing parameter file: basic_diurnal_fourier.py\n",
+ "Preparing parameter file: basic_diurnal_compute_daily_mean.py\n",
+ "Preparing parameter file: basic_diurnal_composite.py\n",
+ "Saving User Choices\n"
+ ]
+ }
+ ],
+ "source": [
+ "# this prepares the various parameter files used in the demo notebooks to reflect where you downloaded the data\n",
+ "import glob\n",
+ "\n",
+ "# Dictionary for template_files substitutions \n",
+ "sub_dict = {\n",
+ " \"INPUT_DIR\": demo_data_directory,\n",
+ " \"OUTPUT_DIR\": demo_output_directory\n",
+ "}\n",
+ "for name in glob.glob(\"*.in\"):\n",
+ " with open(name) as template_file:\n",
+ " print(\"Preparing parameter file: {}\".format(name[:-3]))\n",
+ " template = template_file.read()\n",
+ " for key in sub_dict:\n",
+ " template = template.replace(\"${}$\".format(key), sub_dict[key])\n",
+ " with open(name[:-3], \"w\") as param_file:\n",
+ " param_file.write(template)\n",
+ "\n",
+ "print(\"Saving User Choices\") \n",
+ "with open(\"user_choices.py\", \"w\") as f:\n",
+ " print(\"demo_data_directory = '{}'\".format(demo_data_directory), file=f)\n",
+ " print(\"demo_output_directory = '{}'\".format(demo_output_directory), file=f)"
+ ]
+ },
  {
  "cell_type": "markdown",
  "metadata": {},
  "source": [
  "You're done!\n",
  "Please proceed to the next tutorial"
  ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
  }
  ],
  "metadata": {
@@ -86,8 +150,12 @@
  "nbconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "version": "3.6.7"
- }
+ },
+ "selected_variables": [],
+ "vcdat_file_path": "",
+ "vcdat_loaded_variables": [],
+ "vcdat_variable_info": {}
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }