Add Derecho to supported platforms (ufs-community#1836)

UFS: - Remove Cheyenne (retiring UCAR HPC) support. - Add UCAR's new Derecho HPC system to supported Tier 1 Platforms.
DeniseWorthen · Dec 14, 2023 · 10635ef · 10635ef
1 parent 766ae99
commit 10635ef
Show file tree

Hide file tree

Showing 77 changed files with 1,770 additions and 4,214 deletions.
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -4,7 +4,7 @@ PLEASE DO NOT MODIFY THE TEMPLATE BEYOND FILLING OUT THE PROPER SECTIONS -->
 <!-- Please complete all items in list. -->
 - [ ] I have linked PR's from all sub-components involved in section below. <!-- PLEASE DO NOT LINK SUBCOMPONENT ISSUES -->
 - [ ] I am confirming reviews are completed in ALL sub-component PR's.
-- [ ] I have run the full RT suite on either Hera/Cheyenne AND have attached the log to this PR below this line:
+- [ ] I have run the full RT suite on either Hera/Hercules AND have attached the log to this PR below this line:
  - LOG: 
 - [ ] I have added the list of all failed regression tests to "Anticipated changes" section.
 - [ ] I have filled out all sections of the template.
@@ -79,7 +79,7 @@ PLEASE DO NOT MODIFY THE TEMPLATE BEYOND FILLING OUT THE PROPER SECTIONS -->
  - [ ] Hercules
  - [ ] Jet
  - [ ] Gaea
- - [ ] Cheyenne
+ - [ ] Derecho
 - WCOSS2
  - [ ] Dogwood/Cactus
  - [ ] Acorn

diff --git a/doc/UsersGuide/source/AutomatedTesting.rst b/doc/UsersGuide/source/AutomatedTesting.rst
@@ -91,7 +91,7 @@ On supported HPC systems, a :term:`cron job` runs the ``start_rt_auto.sh`` bash
 This script checks the HPC name and sets certain python paths. Then, it runs ``rt_auto.py``, 
 which uses the Github API (through pyGitHub) to check the labels on pull requests to 
 ``ufs-weather-model``. If a PR label matches the HPC name 
-(e.g., hera-intel-RT or cheyenne-gnu-BL), the label provides the HPC 
+(e.g., hera-intel-RT or derecho-gnu-BL), the label provides the HPC 
 with the compiler and job information to run a test or task on the machine. 
 If no PR label matches HPC name, the script exits.
 

diff --git a/doc/UsersGuide/source/BuildingAndRunning.rst b/doc/UsersGuide/source/BuildingAndRunning.rst
@@ -10,7 +10,7 @@ Supported Platforms & Compilers
 Before running the Weather Model (:term:`WM`), users should determine which of the 
 :ref:`levels of support <SupportedPlatforms>` 
 is applicable to their system. Generally, Level 1 & 2 systems are restricted to those with access 
-through NOAA and its affiliates. These systems are named (e.g., Hera, Orion, Cheyenne). 
+through NOAA and its affiliates. These systems are named (e.g., Hera, Orion, Derecho). 
 Level 3 & 4 systems include certain personal computers or non-NOAA-affiliated HPC systems. 
 The prerequisite software libraries for building the WM already exist in a centralized location on Level 1/preconfigured 
 systems, so users may skip directly to :ref:`getting the data <GetData>` and downloading the code. 
@@ -89,23 +89,23 @@ the data required to run the WM RTs are already available in the following locat
 .. _DataLocations:
 .. table:: Data Locations for Level 1 & 2 Systems
 
- +--------------+-----------------------------------------------------+
- | Machine | File location |
- +==============+=====================================================+
- | Cheyenne | /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT |
- +--------------+-----------------------------------------------------+
- | Gaea | /lustre/f2/pdata/ncep_shared/emc.nemspara/RT |
- +--------------+-----------------------------------------------------+
- | Hera | /scratch1/NCEPDEV/nems/emc.nemspara/RT |
- +--------------+-----------------------------------------------------+
- | Jet | /mnt/lfs4/HFIP/hfv3gfs/role.epic/RT |
- +--------------+-----------------------------------------------------+
- | Orion | /work/noaa/nems/emc.nemspara/RT |
- +--------------+-----------------------------------------------------+
- | S4 | /data/prod/emc.nemspara/RT |
- +--------------+-----------------------------------------------------+ 
- | WCOSS2 | /lfs/h2/emc/nems/noscrub/emc.nems/RT |
- +--------------+-----------------------------------------------------+ 
+ +--------------+--------------------------------------------------------+
+ | Machine | File location  |
+ +==============+========================================================+
+ | Derecho  | /glade/derecho/scratch/epicufsrt/ufs-weather-model/RT |
+ +--------------+--------------------------------------------------------+
+ | Gaea | /lustre/f2/pdata/ncep_shared/emc.nemspara/RT  |
+ +--------------+--------------------------------------------------------+
+ | Hera | /scratch1/NCEPDEV/nems/emc.nemspara/RT  |
+ +--------------+--------------------------------------------------------+
+ | Jet | /mnt/lfs4/HFIP/hfv3gfs/role.epic/RT  |
+ +--------------+--------------------------------------------------------+
+ | Orion | /work/noaa/nems/emc.nemspara/RT  |
+ +--------------+--------------------------------------------------------+
+ | S4 | /data/prod/emc.nemspara/RT  |
+ +--------------+--------------------------------------------------------+ 
+ | WCOSS2 | /lfs/h2/emc/nems/noscrub/emc.nems/RT  |
+ +--------------+--------------------------------------------------------+ 
 
 For Level 3-4 systems, the data must be added to the user's system. 
 Publicly available RT data is available in the `UFS WM Data Bucket <https://registry.opendata.aws/noaa-ufs-regtests/>`__. 

diff --git a/doc/UsersGuide/source/FAQ.rst b/doc/UsersGuide/source/FAQ.rst
@@ -29,14 +29,14 @@ The steps are:
  cd tests
 
  #. Find a configure (``*.conf``) file that contains the machine and compiler you are using. For this
- example, the Intel compiler on Cheyenne is used. To create a custom configure file, two lines are
+ example, the Intel compiler on Derecho is used. To create a custom configure file, two lines are
  needed: a ``COMPILE`` line and a ``RUN`` line. The ``COMPILE`` line should contain the name
- of the machine and compiler ``cheyenne.intel`` and the desired ``SUITES`` for the build. Choose a
+ of the machine and compiler ``derecho.intel`` and the desired ``SUITES`` for the build. Choose a
  ``RUN`` line under this ``COMPILE`` command that uses the desired ``SUITE``. For example:
 
  .. code-block:: console
 
- COMPILE | 32BIT=Y CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta,FV3_GFS_v15p2_no_nsst,FV3_GFS_v16beta_no_nsst | standard | cheyenne.intel | fv3
+ COMPILE | 32BIT=Y CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta,FV3_GFS_v15p2_no_nsst,FV3_GFS_v16beta_no_nsst | standard | derecho.intel | fv3
  RUN | fv3_ccpp_gfs_v16beta | standard | | fv3 |
 
  Put these two lines into a file called ``my_test.conf``. The parameters used in this run can be
@@ -49,16 +49,16 @@ The steps are:
 
  .. code-block:: console
 
- if [[ $MACHINE_ID = cheyenne.* ]]; then stanza:
+ if [[ $MACHINE_ID = derecho.* ]]; then stanza:
  ...
  dprefix=/glade/scratch
 
- This works for Cheyenne, since ``$USER/FV3_RT`` will be appended. Also check that ``RTPWD``
+ This works for Derecho, since ``$USER/FV3_RT`` will be appended. Also check that ``RTPWD``
  points to a diretory that exists:
 
  .. code-block:: console
 
- if [[ $MACHINE_ID = cheyenne.* ]]; then
+ if [[ $MACHINE_ID = derecho.* ]]; then
  RTPWD=${RTPWD:-$DISKNM/ufs-public-release-20200224/${COMPILER^^}}
 
  #. Run the ``rt.sh`` script from the ``tests`` directory:

diff --git a/doc/UsersGuide/source/Introduction.rst b/doc/UsersGuide/source/Introduction.rst
@@ -36,7 +36,7 @@ for those elements, are listed below:
 
 The UFS Weather Model is currently included in two UFS Application releases: The UFS Short-Range Weather (:term:`SRW`) Application v2.0.0 release (June 2022) and the UFS Medium Range Weather Application (:term:`MRW`) v1.1.0 release (October 2020). These UFS Apps also contain pre- and post-processing components, a comprehensive build system, and workflows for configuration and execution of the application. The SRW App v2.0.0 documentation and details can be found `here <https://ufs-srweather-app.readthedocs.io/en/release-public-v2/>`__. The MRW App v1.1.0 documentation and details can be found `here <https://ufs-mrweather-app.readthedocs.io/en/ufs-v1.1.0>`__.
 
-The UFS WM code is portable and can be used with Linux or Mac operating systems and with Intel or GNU compilers. It has been tested on a variety of platforms widely used by atmospheric scientists, such as the NOAA Research Hera system, the National Center for Atmospheric Research (:term:`NCAR`) Cheyenne system, the National Science Foundation Stampede system, and Mac laptops.
+The UFS WM code is portable and can be used with Linux or Mac operating systems and with Intel or GNU compilers. It has been tested on a variety of platforms widely used by atmospheric scientists, such as the NOAA Research Hera system, the National Center for Atmospheric Research (:term:`NCAR`) Derecho system, the National Science Foundation Stampede system, and Mac laptops.
 
 .. note::
 
@@ -66,4 +66,4 @@ This WM User's Guide is organized as follows:
 
 Finally, :numref:`Chapters %s <Acronyms>` and :numref:`%s <Glossary>` contain a list of acronyms and a glossary, respectively.
 
-.. bibliography:: references.bib
+.. bibliography:: references.bib
diff --git a/modulefiles/ufs_cheyenne.gnu.lua b/modulefiles/ufs_cheyenne.gnu.lua
diff --git a/modulefiles/ufs_cheyenne.intel.lua b/modulefiles/ufs_cheyenne.intel.lua
diff --git a/modulefiles/ufs_derecho.gnu.lua b/modulefiles/ufs_derecho.gnu.lua
@@ -0,0 +1,30 @@
+help([[
+loads UFS Model prerequisites for Derecho/GNU
+]])
+
+setenv("LMOD_TMOD_FIND_FIRST","yes")
+prepend_path("MODULEPATH", "/lustre/desc1/scratch/epicufsrt/contrib/modulefiles")
+load("ecflow/5.8.4")
+load("mysql/8.0.33")
+
+setenv("LMOD_TMOD_FIND_FIRST","yes")
+prepend_path("MODULEPATH", "/lustre/desc1/scratch/epicufsrt/contrib/modulefiles_extra")
+prepend_path("MODULEPATH", "/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core")
+
+unload("ncarcompilers")
+stack_gnu_ver=os.getenv("stack_gnu_ver") or "12.2.0"
+load(pathJoin("stack-gcc", stack_gnu_ver))
+
+stack_cray_mpich_ver=os.getenv("stack_cray_mpich_ver") or "8.1.25"
+load(pathJoin("stack-cray-mpich", stack_cray_mpich_ver))
+
+cmake_ver=os.getenv("cmake_ver") or "3.26.3"
+load(pathJoin("cmake", cmake_ver))
+
+stack_python_ver=os.getenv("stack_python_ver") or "3.10.8"
+load(pathJoin("stack-python", stack_python_ver))
+
+setenv("CMAKE_Platform", "derecho.gnu")
+load("ufs-weather-model-env")
+
+whatis("Description: UFS build environment")
diff --git a/modulefiles/ufs_derecho.intel.lua b/modulefiles/ufs_derecho.intel.lua
@@ -0,0 +1,30 @@
+help([[
+loads UFS Model prerequisites for NOAA Parallelworks/Intel
+]])
+
+setenv("LMOD_TMOD_FIND_FIRST","yes")
+prepend_path("MODULEPATH", "/lustre/desc1/scratch/epicufsrt/contrib/modulefiles")
+load("ecflow/5.8.4")
+load("mysql/8.0.33")
+
+setenv("LMOD_TMOD_FIND_FIRST","yes")
+prepend_path("MODULEPATH", "/lustre/desc1/scratch/epicufsrt/contrib/modulefiles_extra")
+prepend_path("MODULEPATH", "/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.0/envs/unified-env/install/modulefiles/Core")
+
+unload("ncarcompilers")
+stack_intel_ver=os.getenv("stack_intel_ver") or "2021.10.0"
+load(pathJoin("stack-intel", stack_intel_ver))
+
+stack_cray_mpich_ver=os.getenv("stack-cray-mpich_ver") or "8.1.25"
+load(pathJoin("stack-cray-mpich", stack_cray_mpich_ver))
+
+cmake_ver=os.getenv("cmake_ver") or "3.26.3"
+load(pathJoin("cmake", cmake_ver))
+
+stack_python_ver=os.getenv("stack_python_ver") or "3.10.8"
+load(pathJoin("stack-python", stack_python_ver))
+
+setenv("CMAKE_Platform", "derecho.intel")
+load("ufs-weather-model-env")
+
+whatis("Description: UFS build environment")
diff --git a/modulefiles/ufs_linux.intel b/modulefiles/ufs_linux.intel
diff --git a/tests/auto-jenkins/jobs/bl.py b/tests/auto-jenkins/jobs/bl.py
@@ -43,10 +43,10 @@ def set_directories(job_obj):
  blstore = '/work/noaa/epic/hercules/UFS-WM_RT'
  rtbldir = '/work/noaa/stmp/role-epic/stmp/role-epic/FV3_RT/'\
  f'REGRESSION_TEST' 
- elif job_obj.machine == 'cheyenne':
- workdir = '/glade/scratch/epicufsrt/autort/jenkins/autort/pr'
- blstore = '/glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/NEMSfv3gfs'
- rtbldir = '/glade/scratch/epicufsrt/FV3_RT/'\
+ elif job_obj.machine == 'derecho':
+ workdir = '/glade/derecho/scratch/epicufsrt/autort/jenkins/autort/pr'
+ blstore = '/glade/derecho/scratch/epicufsrt/ufs-weather-model/RT/NEMSfv3gfs'
+ rtbldir = '/glade/derecho/scratch/epicufsrt/FV3_RT/'\
  f'REGRESSION_TEST'
  else:
  logger.critical(f'Machine {job_obj.machine} is not supported for this job')

diff --git a/tests/auto-jenkins/jobs/rt.py b/tests/auto-jenkins/jobs/rt.py
@@ -33,8 +33,8 @@ def set_directories(job_obj):
  workdir = '/work/noaa/epic-ps/role-epic-ps/autort/tests/auto/pr'
  elif job_obj.machine == 'hercules':
  workdir = '/work/noaa/epic/role-epic/autort/tests/auto/pr'
- elif job_obj.machine == 'cheyenne':
- workdir = '/glade/scratch/epicufsrt/autort/jenkins/autort/pr'
+ elif job_obj.machine == 'derecho':
+ workdir = '/glade/derecho/scratch/epicufsrt/autort/jenkins/autort/pr'
  else:
  print(f'Machine {job_obj.machine} is not supported for this job')
  raise KeyError

diff --git a/tests/auto-jenkins/rt_auto_jenkins.py b/tests/auto-jenkins/rt_auto_jenkins.py
@@ -83,8 +83,8 @@ def delete_pr_dirs(each_pr, machine):
  workdir = '/work/noaa/epic-ps/role-epic-ps/autort/pr'
  elif machine == 'hercules':
  workdir = '/work/noaa/epic/role-epic/autort/pr'
- elif machine == 'cheyenne':
- workdir = '/glade/scratch/epicufsrt/autort/jenkins/autort/pr'
+ elif machine == 'derecho':
+ workdir = '/glade/derecho/scratch/epicufsrt/autort/jenkins/autort/pr'
  else:
  logging.error(f'Machine {machine} is not supported for this job')
  raise KeyError
@@ -120,8 +120,8 @@ def delete_rt_dirs(in_dir, machine, workdir):
  rt_dir = '/work/noaa/stmp/bcurtis/stmp/bcurtis/FV3_RT'
  elif machine == 'hercules':
  rt_dir = '/work/noaa/stmp/bcurtis/stmp/bcurtis/FV3_RT'
- elif machine == 'cheyenne':
- rt_dir = '/glade/scratch/epicufsrt/FV3_RT'
+ elif machine == 'derecho':
+ rt_dir = '/glade/derecho/scratch/epicufsrt/FV3_RT'
  else:
  logging.error(f'Machine {machine} is not supported for this job')
  raise KeyError
@@ -301,12 +301,9 @@ def setup_env():
  elif bool(re.match(re.compile('Hercules-login.+'), hostname)):
  machine = 'hercules'
  os.environ['ACCNR'] = 'epic'
- elif bool(re.match(re.compile('cheyenne.+'), hostname)):
- machine = 'cheyenne'
- os.environ['ACCNR'] = 'SCSG0002'
- elif bool(re.match(re.compile('chadmin.+'), hostname)):
- machine = 'cheyenne'
- os.environ['ACCNR'] = 'SCSG0002'
+ elif bool(re.match(re.compile('derecho.+'), hostname)):
+ machine = 'derecho'
+ os.environ['ACCNR'] = 'NRAL0032'
  else:
  raise KeyError(f'Hostname: {hostname} does not match '\
  'for a supported system. Exiting.')

diff --git a/tests/auto-jenkins/start_rt_auto_jenkins.sh b/tests/auto-jenkins/start_rt_auto_jenkins.sh
@@ -22,9 +22,10 @@ elif [[ $HOSTNAME == tfe* ]]; then
 elif [[ $HOSTNAME == gaea* ]]; then
  export PATH=/lustre/f2/pdata/esrl/gsd/contrib/miniconda3/4.8.3/envs/ufs-weather-model/bin:$PATH
  export PYTHONPATH=/lustre/f2/pdata/esrl/gsd/contrib/miniconda3/4.8.3/lib/python3.8/site-packages
-elif [[ $HOSTNAME == *chadmin* ]] || [[ $HOSTNAME == *cheyenne* ]]; then
- export MACHINE_ID=cheyenne
+elif [[ $HOSTNAME == derecho* ]]; then
+ export MACHINE_ID=derecho
  export PATH=/glade/p/ral/jntp/tools/miniconda3/4.8.3/envs/ufs-weather-model/bin:/glade/p/ral/jntp/tools/miniconda3/4.8.3/bin:$PATH
+ export PATH=/glade/work/epicufsrt/contrib/derecho/rocoto/bin:$PATH
  export PYTHONPATH=/glade/p/ral/jntp/tools/miniconda3/4.8.3/envs/ufs-weather-model/lib/python3.8/site-packages:/glade/p/ral/jntp/tools/miniconda3/4.8.3/lib/python3.8/site-packages
 else
  echo "No Python Path for this machine."