Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Transition the var_defns bash file to YAML. #1098

Open
wants to merge 43 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
55b500b
WIP
christinaholtNOAA Apr 11, 2024
46c4805
Add machine to command
christinaholtNOAA Apr 11, 2024
87e549f
Keep same environment behavior for AQM jobs.
christinaholtNOAA Apr 11, 2024
59d07d4
WIP
christinaholtNOAA Apr 18, 2024
446f6eb
Finished up first round of ex-scripts
christinaholtNOAA Apr 18, 2024
38f9daa
Order docs.
christinaholtNOAA Apr 18, 2024
8496c68
Order jobs.
christinaholtNOAA Apr 18, 2024
e5b28da
Ordering for sourced files.
christinaholtNOAA Apr 18, 2024
16da1ef
Cleaning up old functionality.
christinaholtNOAA Apr 18, 2024
444f1f0
WIP
christinaholtNOAA Apr 18, 2024
a12f6fb
WIP
christinaholtNOAA Apr 19, 2024
4bec6b7
WIP
christinaholtNOAA Apr 19, 2024
0d86ab0
Fix all the subshell problems.
christinaholtNOAA Apr 19, 2024
a6513cb
Merge remote-tracking branch 'origin/develop' into to_yaml
christinaholtNOAA Apr 23, 2024
f08e3ef
Boolified.
christinaholtNOAA Apr 27, 2024
94d7970
WIP
christinaholtNOAA Jun 11, 2024
4ef89f6
Merge remote-tracking branch 'origin/develop' into to_yaml
christinaholtNOAA Jun 11, 2024
874af4b
Bump uwtools version.
christinaholtNOAA Jun 11, 2024
d2a0bc2
Source from a function.
christinaholtNOAA Jun 11, 2024
253590f
WIP
christinaholtNOAA Jun 17, 2024
e242490
Add nco section to be sourced always.
christinaholtNOAA Jun 18, 2024
241ef5d
A bit of clean up
christinaholtNOAA Jun 18, 2024
f360da3
Update jobs/JREGIONAL_MAKE_OROG
christinaholtNOAA Jun 28, 2024
b31f6c7
Applying suggested changes from review.
christinaholtNOAA Jul 12, 2024
2ed368f
Update docs for default values.
christinaholtNOAA Jul 12, 2024
8b9eb07
Merge branch 'to_yaml' of https://github.com/ChristinaholtNOAA/ufs-sr…
christinaholtNOAA Jul 12, 2024
0a8cf97
Merge branch 'develop' of https://github.com/ufs-community/ufs-srweat…
christinaholtNOAA Jul 12, 2024
5502cdc
This should no longer be an executable bash file.
christinaholtNOAA Jul 12, 2024
b368974
Linting.
christinaholtNOAA Jul 13, 2024
b7bacbc
Addressing PR comments.
christinaholtNOAA Jul 15, 2024
d7378dc
Apply suggestions from code review
christinaholtNOAA Jul 15, 2024
e29b3b4
Fix reported stat namespace collision.
christinaholtNOAA Jul 16, 2024
d3f40c6
Merge branch 'to_yaml' of https://github.com/ChristinaholtNOAA/ufs-sr…
christinaholtNOAA Jul 16, 2024
8f7ac15
Apply suggestions from code review
christinaholtNOAA Jul 17, 2024
50be36a
Missed one!
christinaholtNOAA Jul 17, 2024
1acdb48
Making additional PR requested changes.
christinaholtNOAA Jul 17, 2024
ce0a6e6
Merge branch 'to_yaml' of https://github.com/ChristinaholtNOAA/ufs-sr…
christinaholtNOAA Jul 17, 2024
ff4f030
[to_yaml] Update wrapper scripts to allow them to run using var_defns…
MichaelLueken Jul 23, 2024
1afdf51
Merge pull request #4 from MichaelLueken/to_yaml
christinaholtNOAA Jul 23, 2024
060ecc9
Remove tests that are no longer relevant.
christinaholtNOAA Jul 23, 2024
072e8eb
Merge branch 'to_yaml' of https://github.com/ChristinaholtNOAA/ufs-sr…
christinaholtNOAA Jul 23, 2024
2ce08ae
[to_yaml] Update .cicd/scripts/wrapper_srw_ftest.sh to add third argu…
MichaelLueken Jul 25, 2024
85070ad
Merge pull request #5 from MichaelLueken/to_yaml
christinaholtNOAA Jul 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
WIP
Use UW-compliant YAML for var_defns.sh file.
Update loading tasks script for not having all required variables in
environment.
  • Loading branch information
christinaholtNOAA committed Apr 11, 2024
commit 55b500b18782a6a64c6c37363506b8e8fbca2310
19 changes: 18 additions & 1 deletion jobs/JREGIONAL_CHECK_POST_OUTPUT
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,19 @@
#
#-----------------------------------------------------------------------
#
# The J-Job script for checking the post output.
#
# Run-time environment variables:
#
# GLOBAL_VAR_DEFNS_FP
# ENSMEM_INDX
# CDATE
#
# Experiment variables
#
# user.SCRIPTSdir
# user.USHdir
# workflow.EXPTDIR
#
#-----------------------------------------------------------------------
#
Expand All @@ -16,7 +28,12 @@
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
source_config_for_task "" ${GLOBAL_VAR_DEFNS_FP}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This command sources everything in the bash file that does not start with "task_", so the environment is now narrowed to the specific script with the new implementation.

for sect in (user workflow) ; do
for var in $(uw config realize -i ${GLOBAL_VAR_DEFNS_FP} --output-format sh \
--output-block ${sect}) ; do
export $var
done
done
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the pattern I've added to explicitly state which sections of the var_defns file should be "sourced". The bash util helps us run a UW tool that translates YAML to bash and then exports it line-by-line.

This sort of thing will go away with a full integration of uwtools drivers (planned work in upcoming EPIC PI 13).

. $USHdir/job_preamble.sh
#
#-----------------------------------------------------------------------
Expand Down
67 changes: 51 additions & 16 deletions jobs/JREGIONAL_GET_EXTRN_MDL_FILES
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,49 @@
#
#-----------------------------------------------------------------------
#
# This script gets either from the system directory or from mass store
# (HPSS) the files generated by the external model (specified by the
# variable EXTRN_MDL_NAME) for either the initial conditions (ICs) or the
# lateral boundary conditions (LBCs). Which of these we are considering
# depends on the value of the variable ICS_OR_LBCS, which should be defined
# in the environment (when calling this script from a rocoto workflow,
# the workflow should define this variable, e.g. using rocoto's <envar>
# tag).
#
# Note that when we refer to ICs, we are referring to not only the atmospheric
# fields at the initial time but also various surface fields (which are
# for now time-independent) as well as the 0-th forecast hour LBCs. Also,
# when we refer to LBCs, we are referring to the LBCs excluding the one
# at the 0-th hour.
# The J-Job script for getting the model files that will be used for
# either initial conditions or lateral boundary conditions for the
# experiment.
#
# Run-time environment variables:
christinaholtNOAA marked this conversation as resolved.
Show resolved Hide resolved
#
# GLOBAL_VAR_DEFNS_FP
# ICS_OR_LBCS
# PDY
# cyc
# CDATE
# DATA
# COMIN
# EXTRN_MDL_STAGING_DIR (optional; default is DATA)
#
# Experiment variables
#
# user:
# SCRIPTSdir
# USHdir
# workflow:
# EXPTDIR
#
# Depending on ICS_or_LBCS:
#
# task_get_extrn_ics:
# EXTRN_MDL_NAME_ICS
# EXTRN_MDL_ICS_OFFSET_HRS
# FV3GFS_FILE_FMT_ICS
# EXTRN_MDL_SYSBASEDIR_ICS
# USE_USER_STAGED_EXTRN_FILES
# EXTRN_MDL_SOURCE_BASEDIR_ICS
# EXTRN_MDL_FILES_ICS
#
# task_get_extrn_lbcs:
# EXTRN_MDL_NAME_LBCS
# LBC_SPEC_INTVL_HRS
# EXTRN_MDL_LBCS_OFFSET_HRS
# FV3GFS_FILE_FMT_LBCS
# EXTRN_MDL_SYSBASEDIR_LBCS
# USE_USER_STAGED_EXTRN_FILES
# EXTRN_MDL_SOURCE_BASEDIR_LBCS
# EXTRN_MDL_FILES_LBCS
#
#-----------------------------------------------------------------------
#
Expand All @@ -29,8 +58,14 @@
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
source_config_for_task "task_get_extrn_ics|task_get_extrn_lbcs" ${GLOBAL_VAR_DEFNS_FP}
. $USHdir/job_preamble.sh "TRUE"
for sect in (user workflow task_get_extrn_ics task_get_extrn_lbcs) ; do
--output-block ${sect}) ; do
export $var
done
done
. $USHdir/job_preamble.sh


#
#-----------------------------------------------------------------------
#
Expand Down
21 changes: 19 additions & 2 deletions jobs/JREGIONAL_GET_VERIF_OBS
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,19 @@
#
#-----------------------------------------------------------------------
#
# This script checks, pulls, and stages observation data for model verification.
# The J-Job script that checks, pulls, and stages observation data for
# model verification.
#
# Run-time environment variables:
#
# GLOBAL_VAR_DEFNS_FP
# CDATE
#
# Experiment variables
#
# user:
# SCRIPTSdir
# USHdir
#
#-----------------------------------------------------------------------
#
Expand All @@ -16,7 +28,12 @@
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
source_config_for_task " " ${GLOBAL_VAR_DEFNS_FP}
for sect in (user) ; do
for var in $(uw config realize -i ${GLOBAL_VAR_DEFNS_FP} --output-format sh \
--output-block ${sect}) ; do
export $var
done
done
. $USHdir/job_preamble.sh
#
#-----------------------------------------------------------------------
Expand Down
44 changes: 44 additions & 0 deletions jobs/JREGIONAL_INTEGRATION_TEST
Original file line number Diff line number Diff line change
@@ -1,5 +1,49 @@
#!/bin/bash


#
#-----------------------------------------------------------------------
#
# This J-Job script runs a set of tests at the end of WE2E tests.
#
# Run-time environment variables:
#
# GLOBAL_VAR_DEFNS_FP
# CDATE
# FCST_DIR
# SLASH_ENSMEM_SUBDIR
#
# Experiment variables
#
# user:
# SCRIPTSdir
# USHdir
# RUN_ENV
# workflow:
# FCST_LEN_HRS
#
#
#-----------------------------------------------------------------------
#

#
#-----------------------------------------------------------------------
#
# Source the variable definitions file and the bash utility functions.
#
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
for sect in (user workflow) ; do
for var in $(uw config realize -i ${GLOBAL_VAR_DEFNS_FP} --output-format sh \
--output-block ${sect}) ; do
export $var
done
done
. $USHdir/job_preamble.sh



#
#-----------------------------------------------------------------------
#
Expand Down
24 changes: 23 additions & 1 deletion jobs/JREGIONAL_MAKE_GRID
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,23 @@
# orography file generated by orog_fltr_scr down to a width of 4
# cells and store the result in a new filtered orography file in
# WORKDIR_SHVE.

# Run-time environment variables:
#
# GLOBAL_VAR_DEFNS_FP
# DATA
#
# Experiment variables
#
# user:
# SCRIPTSdir
# USHdir
#
# workflow:
# PREEXISTING_DIR_METHOD
#
# task_make_grid:
# GRID_DIR
#
#-----------------------------------------------------------------------
#
Expand All @@ -106,7 +123,12 @@
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
source_config_for_task "task_make_grid" ${GLOBAL_VAR_DEFNS_FP}
for sect in (user workflow task_make_grid) ; do
for var in $(uw config realize -i ${GLOBAL_VAR_DEFNS_FP} --output-format sh \
--output-block ${sect}) ; do
export $var
done
done
. $USHdir/job_preamble.sh
#
#-----------------------------------------------------------------------
Expand Down
30 changes: 29 additions & 1 deletion jobs/JREGIONAL_MAKE_ICS
Original file line number Diff line number Diff line change
@@ -1,5 +1,28 @@
#!/usr/bin/env bash

#
#-----------------------------------------------------------------------
#
# The J-job to run chgres_cube for preparing initial conditions for the
# FV3 forecast
#
# Run-time environment variables:
#
# GLOBAL_VAR_DEFNS_FP
# COMIN
# SLASH_ENSMEM_SUBDIR
# DATA
#
# Experiment variables
#
# user:
# SCRIPTSdir
# USHdir
# RUN_ENVIR
#
#-----------------------------------------------------------------------
#

#
#-----------------------------------------------------------------------
#
Expand All @@ -8,7 +31,12 @@
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
source_config_for_task "task_make_ics" ${GLOBAL_VAR_DEFNS_FP}
for sect in (user) ; do
for var in $(uw config realize -i ${GLOBAL_VAR_DEFNS_FP} --output-format sh \
--output-block ${sect}) ; do
export $var
done
done
. $USHdir/job_preamble.sh
#
#-----------------------------------------------------------------------
Expand Down
31 changes: 30 additions & 1 deletion jobs/JREGIONAL_MAKE_LBCS
Original file line number Diff line number Diff line change
@@ -1,5 +1,29 @@
#!/usr/bin/env bash

#
#-----------------------------------------------------------------------
#
# The J-job to run chgres_cube for preparing lateral boundary conditions
# for the FV3 forecast
#
# Run-time environment variables:
#
# GLOBAL_VAR_DEFNS_FP
# CDATE
# COMIN
# SLASH_ENSMEM_SUBDIR
# DATA
#
# Experiment variables
#
# user:
# SCRIPTSdir
# USHdir
# RUN_ENVIR
#
#-----------------------------------------------------------------------
#

#
#-----------------------------------------------------------------------
#
Expand All @@ -8,7 +32,12 @@
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
source_config_for_task "task_make_lbcs" ${GLOBAL_VAR_DEFNS_FP}
for sect in (user) ; do
for var in $(uw config realize -i ${GLOBAL_VAR_DEFNS_FP} --output-format sh \
--output-block ${sect}) ; do
export $var
done
done
. $USHdir/job_preamble.sh
#
#-----------------------------------------------------------------------
Expand Down
29 changes: 28 additions & 1 deletion jobs/JREGIONAL_MAKE_OROG
Original file line number Diff line number Diff line change
@@ -1,5 +1,27 @@
#!/usr/bin/env bash

#
#-----------------------------------------------------------------------
#
# The J-job to run chgres_cube for preparing lateral boundary conditions
# for the FV3 forecast
christinaholtNOAA marked this conversation as resolved.
Show resolved Hide resolved
#
# Run-time environment variables:
#
# GLOBAL_VAR_DEFNS_FP
#
# Experiment variables
#
# user:
# SCRIPTSdir
# USHdir
#
# task_make_orog:
# OROG_DIR
#
#-----------------------------------------------------------------------
#

#
#-----------------------------------------------------------------------
#
Expand All @@ -8,7 +30,12 @@
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
source_config_for_task "task_make_orog" ${GLOBAL_VAR_DEFNS_FP}
for sect in (user task_make_orog) ; do
for var in $(uw config realize -i ${GLOBAL_VAR_DEFNS_FP} --output-format sh \
--output-block ${sect}) ; do
export $var
done
done
. $USHdir/job_preamble.sh
#
#-----------------------------------------------------------------------
Expand Down
Loading