Addition of python script to add yaml-based test option

[jkbk2004]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

• Yaml-based test scripts are introduced to follow up with the request to add project-based hierarchical test option development
• Another target feature is to facilitate future integration of https://github.com/ufs-community/uwtools

Commit Message:

* UFSWM - python scripts for yaml and rocoto-xml conversion, experiment setup, and test log output  
 

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Conversion of rt.conf to rt.yaml for the prep of UW tools integration #2182

Sub component Pull Requests:

• None

UFSWM Blocking Dependencies:

• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• No Baseline Changes.

Input data Changes:

• None.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[jkbk2004]

What is needed to add S4 support? I can take care of that.

@InnocentSouopgui-NOAA I added S4 machine config. ufs_test.sh has same functionality as rt.sh but for the use of ufs_test.yaml. So you can run pretty much same way as rt.sh, e.g.: ufs_test.sh -r -a or ufs_test.sh -l -a or ufs_test.sh -n "cpld_control_gfsv17 intel" -a . Other runtime argument options (-d -k, etc) work as well. This pr only support rocoto workflow option. Note that python packages (yaml, re, subprocess, glob) should be available at s4 miniconda as well. Let me know if you want to follow up more.

[jkbk2004]

Conversations are quite general topic not specifics of code changes. Anyway, let me add some comments.

• What is the primary motivation for this PR and which problems does it solve?
  Main motivation is to utilize some flexibility of python and yaml approaches. There are external project requests to add more tests: HSD, JEDI, etc. Bash scripts are not necessarily perfect solution. As we agree rt.sh is internal regression test system, it's better to keep it intact. I don't think adding more complications to rt.sh is good idea in a sense of daily code management. It's not bad option to allow alternative test approach and monitor how we can utilize new approaches. @zach1221 @FernandoAndrade-NOAA What do you think? Note that motivation is not due to the lack of rt.sh functionality. Details of HSD/HTF and some other test tasks will follow on in future. @ulmononian may have some comments. We will need to continue to tune a new test option even if its functionality is ok.
• In terms of actual changes, this PR adds ~2000 lines of code, and deletes 0 lines, which I find strange. Is this meant to replace current scripts or not?
  A goal is to reproduce current test capabilities with Python approaches. Python provides better option to parse with yaml. So, it's good if we can test Python approach.
• I see that only one configuration file (rt.conf) is converted to yaml (ufs_test.yaml). I personally find the ufs_test.yaml less readable than rt.conf. What is the advantage in using yaml.
  I don't agree. Yaml is better than text conf. It has vertical structure and current tests more than 240 cases. We can explore multiple project layers of yaml structure to address lengthy vertical issue in future.
• I find it impossible to review this PR by just looking at the code changes and make sure everything works as expected.
  This PR is absolutely orthogonal to existing test system. So I will ask reviews from @ulmononian @zach1221 @FernandoAndrade-NOAA

[jkbk2004]

For uw tools, it is inevitable to have some level of cleanup in test scripts. I hope this python-based approach may facilitate steps toward adopting uw tools. As uw tools features are getting matured, we can start testing with ufs_test.sh.

[jkbk2004]

@BrianCurtis-NOAA @zach1221 @FernandoAndrade-NOAA FYI: I am turning on py and yaml linting options with this pr.

[BrianCurtis-NOAA]

@BrianCurtis-NOAA @zach1221 @FernandoAndrade-NOAA FYI: I am turning on py and yaml linting options with this pr.

OK, It's good practice to address warnings if possible, I see this:

Warning: orkspace/.github/workflows/aux.yml:21:5: [warning] wrong indentation: expected 6 but found 4 (indentation)
  Warning: orkspace/.github/workflows/aux.yml:34:5: [warning] wrong indentation: expected 6 but found 4 (indentation)
  Warning: orkspace/tests/parm/fd_ufs.yaml:1:2: [warning] wrong indentation: expected 0 but found 1 (indentation)
  Warning: orkspace/tests/parm/fd_ufs.yaml:239:21: [warning] too many spaces before colon (colons)

[ulmononian]

Conversations are quite general topic not specifics of code changes. Anyway, let me add some comments.

* What is the primary motivation for this PR and which problems does it solve?
  Main motivation is to utilize some flexibility of python and yaml approaches. There are external project requests to add more tests: HSD, JEDI, etc. Bash scripts are not necessarily perfect solution. As we agree rt.sh is internal regression test system, it's better to keep it intact. I don't think adding more complications to rt.sh is good idea in a sense of daily code management. It's not bad option to allow alternative test approach and monitor how we can utilize new approaches. @zach1221 @FernandoAndrade-NOAA What do you think? Note that motivation is not due to the lack of rt.sh functionality. Details of HSD/HTF and some other test tasks will follow on in future. @ulmononian may have some comments. We will need to continue to tune a new test option even if its functionality is ok.

* In terms of actual changes, this PR adds ~2000 lines of code, and deletes 0 lines, which I find strange. Is this meant to replace current scripts or not?
  A goal is to reproduce current test capabilities with Python approaches. Python provides better option to parse with yaml. So, it's good if we can test Python approach.

* I see that only one configuration file (rt.conf) is converted to yaml (ufs_test.yaml). I personally find the ufs_test.yaml less readable than rt.conf. What is the advantage in using yaml.
  I don't agree. Yaml is better than text conf. It has vertical structure and current tests more than 240 cases. We can explore multiple project layers of yaml structure to address lengthy vertical issue in future.

* I find it impossible to review this PR by just looking at the code changes and make sure everything works as expected.
  This PR is absolutely orthogonal to existing test system. So I will ask reviews from @ulmononian @zach1221 @FernandoAndrade-NOAA

from my perspective, the capabilities introduced here will help facilitate the integration of cases into a ufs htf, especially through the project options and flexibility enabled through yaml-python interface.

[NickSzapiro-NOAA]

Why not push this to uw tools instead of ufs-weather-model if orthogonal and intended for different tests and hierarchical testing framework?

[jkbk2004]

Why not push this to uw tools instead of ufs-weather-model if orthogonal and intended for different tests and hierarchical testing framework?

@NickSzapiro-NOAA Triggering whole regression test or vertical ufs tests from uw tools? It's a bit oversimplification. uw tools just facilitates a workflow system with parsing and setting up model configuration options. uw tool itself is not a workflow system. In UFS system, all development approaches are based on git flow not trunk-based approach. In this case, allowing another test functionality is actually good option in a sense of strangulation not to tangle with internal test system that code managers actively use on daily basis. So people can adopt or test new test features in gradual approaches. Once all satisfies, new features can be transferred to internal test system.

[christinaholtNOAA]

@NickSzapiro-NOAA I am not sure about the requirements Jong has on the updates for the RT framework itself, but as far as HTF goes, I think it could be a great use of uwtools outside the RT framework to provide some configurable use cases. In that case, a user could build ufs-weather-model with their favorite settings, and then use uwtools to run their experiment.

I ❤️ this idea!

[jkbk2004]

@NickSzapiro-NOAA I am not sure about the requirements Jong has on the updates for the RT framework itself, but as far as HTF goes, I think it could be a great use of uwtools outside the RT framework to provide some configurable use cases. In that case, a user could build ufs-weather-model with their favorite settings, and then use uwtools to run their experiment.

I ❤️ this idea!

@christinaholtNOAA @NickSzapiro-NOAA Feel free to add UFS-WM tests to UW tools side and let us know when the new test options are available. We will use new test feature.