Updates for Gaea C5 & F2/F5 Transition + Fix out of bound errors in block_atmos_copy routines in fv3atm #2124

[ulmononian]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full RT suite (compared to current baselines) on either Hera/Derecho/Hercules AND have committed the log to my PR branch.
• Add list of all failed regression tests in "Regression Tests" section.

PR Information

Description

spack-stack locations have been changed to accommodate the Gaea C5 F2 to F5 transition (see #2101 and #2101 (comment)). This PR updates the WM to accommodate these changes, as well as changes the Gaea C5 name to simply Gaea.

Commit Message

• Gaea C5 modulefile & DISKNM update: closes issues Gaea F2 to F5 file system migration #2101
• Bring in the global-workflow detect_machine.sh to keep consistent between projects. (Closes Bring in detect_machine.sh from global workflow for consistency across the community. #2096 )
• Fix out of bound errors in block_atmos_copy routines in fv3tm

Priority

• Critical Bugfix (This PR contains a critical bug fix and should be prioritized.)
• High (This PR contains a feature or fix needed for a time-sensitive project (eg, retrospectives, implementations))
• Normal

Blocking Dependencies

NOAA-EMC/fv3atm#778

Git Issues Fixed By This PR

Partially addresses #2101
Closes #2124
Closes #2111

Changes

Subcomponent (with links)

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Input data

• No changes are expected to input data.
• Changes are expected to input data:
  • New input data.
  • Updated input data.

Regression Tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

FAILED REGRESSION TESTS

Libraries

• Not Needed
• Needed
  • Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
  • Add issue link from JCSDA/spack-stack following this item

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[ulmononian]

@RatkoVasic-NOAA @natalie-perlin @jkbk2004 @zach1221 FYI

[jkbk2004]

#2124 was combined to this pr. @BrianCurtis-NOAA This PR is ready.

[jkbk2004]

@zach1221 Hercules test log is at /work/noaa/epic/jongkim/UFS-RT/hercules/RegressionTests_hercules.log. I started seeing gnv1_nested_intel comparison failure. Same symptom. Job ran ok but only issue with comparison. We need to trun the case off on hercules. Can you follow up? @FernandoAndrade-NOAA can you check jet log? if we continue to see a case or two for time-out issue, we need to turn them off. Jet resources are not that affordable.

[FernandoAndrade-NOAA]

@zach1221 Hercules test log is at /work/noaa/epic/jongkim/UFS-RT/hercules/RegressionTests_hercules.log. I started seeing gnv1_nested_intel comparison failure. Same symptom. Job ran ok but only issue with comparison. We need to trun the case off on hercules. Can you follow up? @FernandoAndrade-NOAA can you check jet log? if we continue to see a case or two for time-out issue, we need to turn them off. Jet resources are not that affordable.

Just leaving a note that the jet log is missing the cpld_debug_gfsv17 test due to a timeout. Jet seems to have an issue recently generating complete RT logs with that test as well as not posting the summary of results following a failure. It looks like the hash info didn't generate either at the beginning. I'm currently rerunning on jet with an increased time limit, I'll append the log when it finishes.

[FernandoAndrade-NOAA]

@zach1221 Hercules test log is at /work/noaa/epic/jongkim/UFS-RT/hercules/RegressionTests_hercules.log. I started seeing gnv1_nested_intel comparison failure. Same symptom. Job ran ok but only issue with comparison. We need to trun the case off on hercules. Can you follow up? @FernandoAndrade-NOAA can you check jet log? if we continue to see a case or two for time-out issue, we need to turn them off. Jet resources are not that affordable.

I've disabled cpld_debug_gfsv17 in rt.conf for jet. This test has consistently failed on my end due to timeouts when running RTs on jet.

[zach1221]

Testing is complete. We can move on to review and merge process.

[zach1221]

@ulmononian fv3atm sub-pr #778 has been merged. Can you please update the submodule hash and revert .gitmodule url?
NOAA-EMC/fv3atm@28bfc36