Port WW3 to expanse

[JessicaMeixner-NOAA]

Pull Request Summary

Port WW3 to expanse

Description

Ports WW3 to the HPC expanse

Issue(s) addressed

• Is there an issue associated with this development (bug fix, enhancement, new feature)? no

Related to UFS PR: ufs-community/ufs-weather-model#850

Check list

• Is your feature branch up to date with the authoritative repository (NOAA/develop)?

• Make sure you have checked the checklist for a developer submitting to develop and updating version number

• Please list appropriate labels code managers should add for this PR:
  bug, documentation, enhancement, new feature, ..

• Reviewers: @MinsukJi-NOAA

Commit Message

• Ports WW3 code to expanse

Testing

• How were these changes tested? @MinsukJi-NOAA can you add this info?

[JessicaMeixner-NOAA]

@MinsukJi-NOAA - my tests on orion with the S2SW ufs regtests all passed with the develop branch + your changes, so if the PR to your commit here updating to the top of develop works on expanse I think you would be fine to include this in your ufs-weather-model PR. Let me know how you want to proceed - I can run more tests with the updated WW3 or something else depending on what you prefer.

[aliabdolali]

Please merge the upstream noaa/develop and resolve the conflicts

[JessicaMeixner-NOAA]

@MinsukJi-NOAA the PR to WW3 yesterday (#483) means the WW3 baselines change. (It's a superficial reason for a version string change in the binary file, not actual answer changes). However, I know this complicates your updating of WW3. @aliabdolali what's the best path forward for this? We will need a PR to ufs-weather-model with the other update and/or we need that branch for the ufs-weather-model.

[MinsukJi-NOAA]

@JessicaMeixner-NOAA @aliabdolali @junwang-noaa I think bringing the top of develop of WW3 into the ufs-weather-model should be a separate PR, and the Expanse changes come in a later PR.

[JessicaMeixner-NOAA]

@MinsukJi-NOAA okay. Sounds good.

In the meantime, have you had a chance to test the PR I made to your branch bringing this branch up to date with the WW3 develop? Or do I need to create a new branch/PR with those? To get this into WW3 we need to resolve the conflicts. We have a pretty big WW3 commit coming up so we either need to get this PR in now or it might have to wait until after the switch -> CPP PR. I'm okay with either I just want to be prepared with your timeline.

@aliabdolali will you be making a PR to ufs-weather-model with the updated WW3 or should I do that?

[aliabdolali]

@JessicaMeixner-NOAA once the conflicts are solved, I can merge this to WW3 (we can merge this today, the tests are about to finish). Then I will create the PR to ufs-weather-model.

[MinsukJi-NOAA]

@JessicaMeixner-NOAA @aliabdolali I have not had a chance to test it on Expanse.
Here is what I am going to do:

1. Create a baseline using my feature/expanse-7c branch
2. Verify against this baseline using JessicaMeixner-NOAA:feature/expanse

Will try to get this PR ready by the end of today so that it does not delay the WW3 commit coming up.

[aliabdolali]

The tests ran successfully with pre-known different cases.

---

********************* non-identical cases ****************************

---

mww3_test_03/./work_PR2_UQ_MPI_d2 (6 files differ)
mww3_test_03/./work_PR2_UNO_MPI_d2 (9 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2_c (8 files differ)
mww3_test_03/./work_PR1_MPI_d2 (14 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2_c (8 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2 (8 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2 (8 files differ)
mww3_test_07/./work_PR3_UQ (3 files differ)
ww3_tp2.10/./work_MPI_OMPH (7 files differ)
ww3_tp2.16/./work_MPI_OMPH (6 files differ)
ww3_ufs1.1/./work_d (0 files differ)
ww3_ufs1.2/./work_b (0 files differ)
ww3_ufs1.3/./work_a (1 files differ)

matrixCompFull.txt
matrixCompSummary.txt
matrixDiff.txt

[MinsukJi-NOAA]

I am getting MPI aborts when running cpld_bmark_wave_v16_p7b with JessicaMeixner-NOAA:feature/expanse on Expanse:

                    *** WAVEWATCH III Multi-grid shell ***
               =================================================

EXTCDE MPI_ABORT, IEXIT=    11

[JessicaMeixner-NOAA]

@MinsukJi-NOAA I believe this is because we need new mod_def files for WW3. Can you copy from hera the newest WW3 Inputs files (/scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/input-data-20210930/WW3_input_data_20210908) Do you need to run this exact version of ufs-weather-model/WW3 or is putting the updates in the develop of both okay? Because the new mod_defs would be for the develop branch.

[MinsukJi-NOAA]

XSEDE collaborators are running 7c experiments (UFS hash b26a89). Since UFS is changing fast, I am not sure if later (or latest) develop branch still supports the cpld_bmark_wave_v16_p7b(_35d) tests. That being said, we can still include the code changes in the develop of WW3; my WW3/feature/expanse-7c can still be used on Expanse if needed.

[JessicaMeixner-NOAA]

@MinsukJi-NOAA sounds good. To include this into WW3 we need to merge the develop branch into your branch to resolve the conflicts. Another strategy could be that we close this PR and open a PR from the branch that I created that has the develop merged in so that the changes get put in but your branch stays as is. Let me know which you prefer.

[MinsukJi-NOAA]

@JessicaMeixner-NOAA thank you. Can we do the second strategy and close this PR?

For the conflics in:
/model/bin/w3_setup

@@ -417,7 +379,7 @@ then
        [ "$cmplr" == "hera.intel" ] || [ "$cmplr" == "orion.intel" ]                || \
        [ "$cmplr" == "hera.gnu" ]   || [ "$cmplr" == "jet.intel" ]                  || \
        [ "$cmplr" == "stampede.intel" ] || [ "$cmplr" == "gaea.intel" ]             || \
-       [ "$cmplr" == "wcoss2" ]                                                     || \
+       [ "$cmplr" == "wcoss2" ] || [ "$cmplr" == "s4.intel" ]                       || \
        [ "$cmplr" == "expanse.intel" ]                                              || \
        [ "$cmplr" == "cheyenne.intel" ] || [ "$cmplr" == "cheyenne.gnu" ]           || \
        [ "$cmplr" == "wcoss_cray" ] || [ "$cmplr" == "wcoss_dell_p3" ]              || \
@@ -448,7 +410,7 @@ then
        [ "$cmplr" == "hera.intel" ] || [ "$cmplr" == "orion.intel" ]                || \
        [ "$cmplr" == "hera.gnu" ]   || [ "$cmplr" == "jet.intel" ]                  || \
        [ "$cmplr" == "stampede.intel" ] || [ "$cmplr" == "gaea.intel" ]             || \
-       [ "$cmplr" == "wcoss2" ]                                                     || \
+       [ "$cmplr" == "wcoss2" ] || [ "$cmplr" == "s4.intel" ]                       || \
        [ "$cmplr" == "expanse.intel" ]                                              || \
        [ "$cmplr" == "cheyenne.intel" ] || [ "$cmplr" == "cheyenne.gnu" ]           || \
        [ "$cmplr" == "wcoss_cray" ] || [ "$cmplr" == "wcoss_dell_p3" ]              || \

/model/bin/cmplr.env

@@ -111,7 +111,7 @@ if [ "$cmplr" == "intel" ]          || [ "$cmplr" == "intel_debug" ]    || [ "$c
    [ "$cmplr" == "datarmor_intel" ] || [ "$cmplr" == "datarmor_intel_debug" ] || [ "$cmplr" == "datarmor_intel_prof" ] || \
    [ "$cmplr" == "wcoss_cray" ]     || [ "$cmplr" == "wcoss_dell_p3" ]  || [ "$cmplr" == "cheyenne.intel" ]      || \
    [ "$cmplr" == "gaea.intel" ]     || [ "$cmplr" == "jet.intel" ]      || \
-   [ "$cmplr" == "wcoss2" ]         || \
+   [ "$cmplr" == "wcoss2" ]         || [ "$cmplr" == "s4.intel" ]       || \
    [ "$cmplr" == "expanse.intel" ]  || \
    [ "$cmplr" == "hera.intel" ]     || [ "$cmplr" == "orion.intel" ]    || [ "$cmplr" == "stampede.intel" ] || \
    [ "$cmplr" == "ukmo_cray_intel" ] || [ "$cmplr" == "ukmo_cray_intel_debug" ]; then
@@ -147,7 +147,7 @@ if [ "$cmplr" == "intel" ]          || [ "$cmplr" == "intel_debug" ]    || [ "$c
   if [ ! -z "$(echo $cmplr | grep datarmor)" ] || [ "$cmplr" == "hera.intel" ] || [ "$cmplr" == "orion.intel" ] || \
      [ "$cmplr" == "wcoss_cray" ] || [ "$cmplr" == "wcoss_dell_p3" ] || [ "$cmplr" == "cheyenne.intel" ] || \
      [ "$cmplr" == "gaea.intel" ] || [ "$cmplr" == "jet.intel" ] || [ "$cmplr" == "wcoss2" ] || \
-     [ "$cmplr" == "expanse.intel" ]  || [ "$cmplr" == "stampede.intel" ]; then
+     [ "$cmplr" == "expanse.intel" ]  || [ "$cmplr" == "stampede.intel" ] || [ "$cmplr" == "s4.intel" ]; then
      optomp="-qopenmp"
   else
      optomp="-openmp"

[JessicaMeixner-NOAA]

Sounds good -- I'll close this after I open a new PR.

[JessicaMeixner-NOAA]

Closing this PR. Please see it's replacement: #490