[develop] Transition the var_defns bash file to YAML. #1098

christinaholtNOAA · 2024-06-18T21:30:25Z

DESCRIPTION OF CHANGES:

Use YAML for the configuration language at run time.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

I could use some help on updating documentation. Just pointers to the most important pieces. Let's also talk about whether that could be separated into a follow-on PR.

TESTS CONDUCTED:

DEPENDENCIES:

n/a

DOCUMENTATION:

ISSUE:

CHECKLIST

My code follows the style guidelines in the Contributor's Guide
I have performed a self-review of my own code using the Code Reviewer's Guide
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes do not require updates to the documentation (explain).
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

Use UW-compliant YAML for var_defns.sh file. Update loading tasks script for not having all required variables in environment.

christinaholtNOAA · 2024-06-18T21:32:25Z

jobs/JREGIONAL_CHECK_POST_OUTPUT

+# USHdir
+#
+# workflow:
+# EXPTDIR


This is a pattern that shows up regularly. I did my best to add a doc block to the top of each j-job and ex-script to list the variables that are used in the script and the sections they come from.

The run-time variables may be from Rocoto jobs or from the NCO preamble.

christinaholtNOAA · 2024-06-18T21:34:10Z

jobs/JREGIONAL_CHECK_POST_OUTPUT

-source_config_for_task "" ${GLOBAL_VAR_DEFNS_FP}
+for sect in user nco workflow ; do
+ source_yaml ${GLOBAL_VAR_DEFNS_FP} ${sect}
+done


This is the pattern I've added to explicitly state which sections of the var_defns file should be "sourced". The bash util helps us run a UW tool that translates YAML to bash and then exports it line-by-line.

This sort of thing will go away with a full integration of uwtools drivers (planned work in upcoming EPIC PI 13).

christinaholtNOAA · 2024-06-18T21:35:25Z

jobs/JREGIONAL_CHECK_POST_OUTPUT

@@ -16,7 +31,9 @@
 #-----------------------------------------------------------------------
 #
 . $USHdir/source_util_funcs.sh
-source_config_for_task "" ${GLOBAL_VAR_DEFNS_FP}


This command sources everything in the bash file that does not start with "task_", so the environment is now narrowed to the specific script with the new implementation.

christinaholtNOAA · 2024-06-18T21:37:18Z

jobs/JREGIONAL_MAKE_GRID

@@ -3,97 +3,25 @@
 #
 #-----------------------------------------------------------------------
 #
-# This script generates grid and orography files in NetCDF format that


This text block was not consistent with what this script actually does, nor what its corresponding ex-script does.

christinaholtNOAA · 2024-06-18T21:39:23Z

jobs/JREGIONAL_PLOT_ALLVARS

+ set +u
+ conda activate ${SRW_GRAPHICS_ENV}
+ set -u
+fi


This task uses a different environment to run graphics, so loads it here instead of in the load_modules_run_task.sh file.

christinaholtNOAA · 2024-06-18T21:55:11Z

ush/bash_utils/source_yaml.sh

+ line=$(echo "$line" | sed -E "s/='\[(.*)\]'/=(\1)/")
+ line=${line//,/}
+ line=${line//\"/}
+ line=${line/None/}


These lines handle the transition from YAML lists to bash lists, and "None" values being empty variables in bash.

christinaholtNOAA · 2024-06-18T21:55:45Z

ush/config_defaults.yaml

@@ -468,7 +468,7 @@ workflow:
 #
 #-----------------------------------------------------------------------
 #
- WORKFLOW_ID: !nowtimestamp ''
+ WORKFLOW_ID: ""


This thing isn't supported in uwtools, and isn't actually needed for SRW.

christinaholtNOAA · 2024-06-18T21:56:22Z

ush/generate_FV3LAM_wflow.py

@@ -639,7 +639,7 @@ def generate_FV3LAM_wflow(
 input_format="nml",
 output_file=FV3_NML_STOCH_FP,
 output_format="nml",
- supplemental_configs=[settings],
+ update_config=get_nml_config(settings),


This is a change to the API in the new uwtools version.

christinaholtNOAA · 2024-06-18T21:57:27Z

ush/job_preamble.sh

@@ -1,5 +1,7 @@
 #!/bin/bash

+set +u


Some how this script wasn't subjected to this requirement.

christinaholtNOAA · 2024-06-18T23:10:53Z

ush/load_modules_run_task.sh

-else
- exec "${jjob_fp}"
-fi
+source "${jjob_fp}"


Running the script in the same shell provides access to modify the conda env.

christinaholtNOAA · 2024-06-20T14:43:50Z

I ran the fundamental tests on Hera one more time after my last push and it's still passing.

christinaholtNOAA and others added 21 commits April 11, 2024 21:57

WIP

55b500b

Use UW-compliant YAML for var_defns.sh file. Update loading tasks script for not having all required variables in environment.

Add machine to command

46c4805

Keep same environment behavior for AQM jobs.

87e549f

WIP

59d07d4

Finished up first round of ex-scripts

446f6eb

Order docs.

38f9daa

Order jobs.

8496c68

Ordering for sourced files.

e5b28da

Cleaning up old functionality.

16da1ef

WIP

444f1f0

WIP

a12f6fb

WIP

4bec6b7

Fix all the subshell problems.

0d86ab0

Merge remote-tracking branch 'origin/develop' into to_yaml

a6513cb

Boolified.

f08e3ef

WIP

94d7970

Merge remote-tracking branch 'origin/develop' into to_yaml

4ef89f6

Bump uwtools version.

874af4b

Source from a function.

d2a0bc2

WIP

253590f

Add nco section to be sourced always.

e242490

christinaholtNOAA commented Jun 18, 2024

View reviewed changes

A bit of clean up

241ef5d

MichaelLueken changed the title ~~Transition the var_defns bash file to YAML.~~ [develop] Transition the var_defns bash file to YAML. Jun 20, 2024

christinaholtNOAA marked this pull request as ready for review June 20, 2024 14:43

christinaholtNOAA requested review from mkavulich, gsketefian, JeffBeck-NOAA, RatkoVasic-NOAA and BenjaminBlake-NOAA as code owners June 20, 2024 14:43

christinaholtNOAA requested review from ywangwof, chan-hoo, panll, christopherwharrop-noaa, danielabdi-noaa, mark-a-potts, jkbk2004, willmayfield, dmwright526, gspetro-NOAA, natalie-perlin, EdwardSnyder-NOAA and MichaelLueken as code owners June 20, 2024 14:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[develop] Transition the var_defns bash file to YAML. #1098

[develop] Transition the var_defns bash file to YAML. #1098

christinaholtNOAA commented Jun 18, 2024 •

edited

christinaholtNOAA Jun 18, 2024

christinaholtNOAA Jun 18, 2024

christinaholtNOAA Jun 18, 2024

christinaholtNOAA Jun 18, 2024

christinaholtNOAA Jun 18, 2024

christinaholtNOAA Jun 18, 2024

christinaholtNOAA Jun 18, 2024

christinaholtNOAA Jun 18, 2024

christinaholtNOAA Jun 18, 2024

christinaholtNOAA Jun 18, 2024

christinaholtNOAA commented Jun 20, 2024

[develop] Transition the var_defns bash file to YAML. #1098

Are you sure you want to change the base?

[develop] Transition the var_defns bash file to YAML. #1098

Conversation

christinaholtNOAA commented Jun 18, 2024 • edited

DESCRIPTION OF CHANGES:

Type of change

TESTS CONDUCTED:

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

CHECKLIST

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christinaholtNOAA commented Jun 20, 2024

christinaholtNOAA commented Jun 18, 2024 •

edited