Enable SHiELD runs in argo workflows #2377
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spencerkclark
force-pushed
the
shield-argo-integration
branch
4 times, most recently
from
ba681f3
to
e743580
Compare
spencerkclark
force-pushed
the
SHiELD-wrapper-regression-tests
branch
2 times, most recently
from
1ba95a9
to
bad6d92
Compare
This references forcing data in the vcm-fv3config bucket, and makes parameters controlling whether we use data from initial conditions or the climatology consistent with our v0.7 FV3GFS base config.
spencerkclark
force-pushed
the
shield-argo-integration
branch
from
e743580
to
fac244f
Compare
spencerkclark
changed the title
This bumps SHiELD-wrapper to include a Q-flux bug fix, and a couple other user experience improvements with the SOM. Note this fix has not been merged to SHiELD yet, so I will update this PR later once we can point to main branches of SHiELD-wrapper and SHiELD_physics.
spencerkclark
commented
Dec 19, 2023
oliverwm1
approved these changes
Dec 21, 2023
|
Thanks @oliverwm1! I went ahead and set |
spencerkclark
added a commit
that referenced
this pull request
Jan 5, 2024
#2377 reorganized the `base_yamls` directory in fv3kube to make room for SHiELD reference configurations, but neglected to update the `package_data` parameter in `setup.py` accordingly. Without this change, installing `fv3kube` via something like: ``` $ pip install git+https://github.com/ai2cm/fv3net.git@b8e2f83b5206539724a4d096d0433ceeb3bc805a#egg=fv3kube&subdirectory=external/fv3kube ``` does not include the `base_yamls` files, which are an important component of the library. I've tested this locally and it fixes the issue, e.g. use: ``` $ pip install git+https://github.com/ai2cm/fv3net.git@8fe01cd6c49ea635a1b07afd4ee4615db7555ce6#egg=fv3kube&subdirectory=external/fv3kube ``` and check for the existence of the `base_yamls` directory (note the different SHA from the original example).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds on #2376 and splits out from #2350 what is necessary to run SHiELD-wrapper-based prognostic simulations through our standard prognostic run argo workflow. No changes to the frontend API are needed; the prognostic run workflow is modified to infer which template (
run-fv3gfsorrun-shield) to run based on the input config.For convenience this also adds a starter base config for SHiELD, which is based on the configuration used in the PIRE simulations (but for simplicity with the mixed layer ocean turned off). I have tested the
prognostic-runandrestart-prognostic-runworkflows using a SHiELD-based config offline. I'm not sure if we want to add an integration test yet or not.Significant internal changes:
prognostic-runworkflow to infer whether to use FV3GFS or SHiELD based on the config.restart-prognostic-runworkflow to infer whether to use FV3GFS or SHiELD based on the config at the provided URL.fv3kubeto better accommodate SHiELD configs. No user-facing changes to the FV3GFS configs are made.Note this PR makes use of YAML anchors and aliases to reduce the amount of duplicate configuration code. Some illustration of how these work can be found here. Use of this concept was already introduced in #2103 within the
training.yamltemplate, though this is the first time using it in the prognostic run.To illustrate the updated workflows I have included some example step outputs from
argo getbelow (we ran theprognostic-runworkflow for two segments and then ran one more segment via therestart-prognostic-runworkflow).prognostic-run
restart-prognostic-run