-
Notifications
You must be signed in to change notification settings - Fork 115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[develop] Streamline SRW App's interface to MET/METplus #1005
[develop] Streamline SRW App's interface to MET/METplus #1005
Conversation
…iable fieldname_in_MET_filedir_names (as it is with APCP).
…UPA, this time in METplus conf files.
… ">=", ">", "<=", and "<" to "ge", "gt", "le", and "lt".
… variables are grouped together, (2) commented-out settings of FCST_VAR<n>_... and OBS_VAR<n>_... are removed, (3) FCST_VAR<n>_OPTIONS and OBS_VAR<n>_OPTIONS come last, and (4) each element in options is on a separate line.
…tool names to template METplus conf files as variables; add new workflow variable that specifies the METplus verbosity level.
…ks for APCP (i.e. not just for accumulation > 1 hr but also for accumulation = 1 hour) use obs and forecast tasks that have been processed by the PcpCombine tool of METplus. These files are all in NetCDF format and are generated from grib2 input files (both for obs and forecast). This involves: * Renaming certain workflow variables for verification so their role/purpose is clearer. * Modifying the names of the arrays for APCP variables in the NetCDF files. * Changing the names of the levels associated with APCP variables in the NetCDF files (e.g. "A1" instead of "A01"). * For consistency, making changes for accumulated snow (ASNOW) analogous to the above changes for APCP. * In ex-scripts for vx tasks, removing if-statement that treats the case of APCP accumulation period > 1 hour separately and performing the same steps for both accumulation = 1 hour and > 1 hour.
…should be replaced with [OBS|FCST]_PCP_COMBINE_INPUT_DATATYPE. Thus, they are only relevant to METplus conf files that run PcpCombine. Thus, remove all use of these variables in conf files that do not run PcpCombine, and in the ones that do, replace them with the appropriate new variables (some already have the new variables).
…lates for APCP01h and APCPgt01h.
…s for combining METplus conf files for APCP01h and APCPgt01h (inadvertantly left out of a previous commit).
…f files no longer differentiate between 1 hour accumulation and > 1 hour accumulation.
…_NATIVE_DATA_TYPE.
… to compare ADP[SFC|UPA] conf files with other branches.
… created using other branches.
…se created using other branches.
… those created using other branches.
… conf files with those generated by another branch.
…ese "BOTH" variables to two, one named "FCST_VAR<n>_..." and another named "OBS_VAR<n>_...".
…the ush directory.
…semble verification.
…onfiguration files.
…iles (instead of creating them, since the creation step is now done by the parse_vx_config_[det|ens] tasks).
…when pretty-printing contents of vx configuration files.
@mkavulich I cleaned up and renamed the script
A final change I made is to call I've tested this latest code on only one WE2E vx test ( |
@MichaelLueken @mkavulich I tested the latest on the following vx WE2E tests on Hera, and they all passed:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gsketefian - While running the verification WE2E tests, I did note two failing tasks:
- parse_vx_config_det
- parse_vx_config_ens
Looking in the logs, the reason for the failure is due to missing JREGIONAL_PARSE_VX_CONFIG
j-job scripts:
/scratch2/NAGAPE/epic/Michael.Lueken/ufs-srweather-app/hera/ush/load_modules_run_task.sh: line 210: /scratch2/NAGAPE/epic/Michael.Lueken/ufs-srweather-app/hera/jobs/JREGIONAL_PARSE_VX_CONFIG: No such file or directory
Was this new j-job script not committed and pushed?
@MichaelLueken Sorry, I forgot to add the j-job and ex-script for the new jobs. Both jobs use the same j-job and ex-script. They should be there now. |
@gsketefian Thanks for addressing my earlier comments. I am a bit concerned about the amount of custom code that the latest commits are adding for writing this custom text format to be read in by metplus tasks rather than just using the original yaml. Is this because of the format that METplus is expecting for config files? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I won't hold this PR up any further, but I do have a few more suggestions. Hopefully we can move from this custom text format to yaml in the future to reduce the amount of custom code we need to maintain.
Alternatively, if delim_str is not be a substring within item_cpld so that | ||
item_cpld has the form | ||
|
||
item_cpld = str1 | ||
|
||
then item_fcst and item_obs are given by | ||
|
||
item_fcst = str1 | ||
item_obs = str1 | ||
|
||
For example, if | ||
|
||
item_cpld = 'ABCD' | ||
|
||
then | ||
|
||
item_fcst = 'ABCD' | ||
item_obs = 'ABCD' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternatively, if delim_str is not be a substring within item_cpld so that | |
item_cpld has the form | |
item_cpld = str1 | |
then item_fcst and item_obs are given by | |
item_fcst = str1 | |
item_obs = str1 | |
For example, if | |
item_cpld = 'ABCD' | |
then | |
item_fcst = 'ABCD' | |
item_obs = 'ABCD' | |
Alternatively, if the input string does not contain the delimiter string delim_str, both return values will be identical to the input. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shortened/modified. Will show up in the next commit(s) I push.
Function to parse the "coupled" value of an item (obtained from the coupled | ||
verification (vx) configuration dictionary) to extract from it the item's | ||
value for forecasts and its value for observations. The coupled item | ||
(item_cpld) is a string that may correspond to a field name, a level, or | ||
a threshold. It can be the name of a key in a key-value pair in the | ||
coupled vx configuration dictionary, or it can be the value (or, if the | ||
value is a list of strings, one of the elements in that list). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggest cutting down less relevant information to make the docstring more readable
Function to parse the "coupled" value of an item (obtained from the coupled | |
verification (vx) configuration dictionary) to extract from it the item's | |
value for forecasts and its value for observations. The coupled item | |
(item_cpld) is a string that may correspond to a field name, a level, or | |
a threshold. It can be the name of a key in a key-value pair in the | |
coupled vx configuration dictionary, or it can be the value (or, if the | |
value is a list of strings, one of the elements in that list). | |
Function to parse the "coupled" value of an item (obtained from the coupled | |
verification (vx) configuration dictionary) to extract from it the item's | |
value for forecasts and its value for observations. The coupled item | |
(item_cpld) is a string that may correspond to a field name, a level, or | |
a threshold. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shortened. Will show up in the next commit(s) I push.
# Set up logging. | ||
log_level = str.upper(log_lvl) | ||
fmt = "[%(levelname)s:%(name)s: %(filename)s, line %(lineno)s: %(funcName)s()] %(message)s" | ||
if log_fp: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't relevant for now, but in my experience logging.basicConfig
does not work as expected if we already have a python logger set up; worth remembering for the future if we move towards python-based exscripts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok.
Thank you very much for adding the
Now preparing to kick off Jenkins testing. |
@mkavulich The files being read in by
Aside: The The parsing by
I know there's the |
The Jenkins tests have all successfully passed. Moving forward with merging this work now. |
DESCRIPTION OF CHANGES:
This PR streamlines the SRW App's interface to the MET/METplus verification tool and implements some bug fixes. Details:
Replace the field-specific METplus configuration jinja2 templates associated with each METplus tool (these templates are hard-coded for each field) with a single template that contains jinja2 code to any valid field to be verified. For example, for the
EnsembleStat
tool, the six templatesare replaced by the single template
EnsembleStat.conf
.Add yaml configuration files for verification that specify the fields to verify (including field levels and thresholds). This is in order to consolidate the field/level/threshold information in one place instead of having it spread out and repeated in several hard-coded configuration files. Two such yaml configuration files are added:
vx_config_det.yaml
-- for deterministic verificationvx_config_ens.yaml
-- for ensemble verificationAdd a python script (
decouple_fcst_obs_vx_config.py
) to parse these two vx configuration files and create a dictionary of the field/level/threshold information that can then be passed to the unified workflow templating tool.Modify the ex-scripts for the verification tasks (
exregional_run_met_....sh
) to allow the use of the new jinja2 METplus config templates. This includes adding code to call the new scriptdecouple_fcst_obs_vx_config.py
and then passing its output to the unified workflow templating tool to generate METplus configuration files from the (new) jinja2 templates.Add new environment variables to the rocoto workflow configuration files (
verify_[pre|det|ens].yaml
) that are needed for using the new jinja2 METplus config templates.Bug fixes:
PointStat
tool, some of the fields to be verified were missing theset_attr_lead
option in the METplus configuration file that accounts for time-lagging when setting the entry in theFCST_LEAD
column of the output stat file. This option causes theFCST_LEAD
entry of time-lagged members to have the same lead time as non-lagged members (i.e. it converts actual lead time to lead time assuming no time-lagging). This option was added where it was missing (e.g. if variable number 14 was missing this, the following line may have been added:FCST_VAR14_OPTIONS = set_attr_lead = "{lead?fmt=%H%M%S}";
). The field/level combinations that were missing this option were:PointStat
verification ofADPSFC
(surface) fields:The
set_attr_lead
option was added to SPFH at Z2, CRAIN at L0, CSNOW at L0, CFRZR at L0, and CICEP at L0.PointStat
verification ofADPUPA
(upper air) fields:The
set_attr_lead
option was added to CAPE at L0-90.vx_config_ens.yaml
, the following bug fixes were automatically implemented:GenEnsProd
:ADPSFC
fields:lt152, lt1520, ge914
tolt152, lt305, lt914
.ADPUPA
fields:le1000, gt1000&<2500, gt2500&<4000, gt2500
tole1000, gt1000&<2500, ge2500&<4000, ge2500
.EnsembleStat
:ADPSFC
fields:lt152, lt1520, ge914
tolt152, lt305, lt914
.lt152, lt305, ge914
tolt152, lt305, lt914
.ADPUPA
fields:ge263, ge286, ge273
toge263, ge268, ge273
.ge263, ge286, ge273
toge263, ge268, ge273
.PointStat_ensmean
(runningPointStat
to calculate ensemble mean fields):ADPUPA
fields:ge263, ge286, ge273
toge263, ge268, ge273
.ge263, ge286, ge273
toge263, ge268, ge273
.ge26, ge31, ge46, ge62
toge26, ge31, ge36, ge46, ge62
.ge26, ge31, ge46, ge62
toge26, ge31, ge36, ge46, ge62
.le1000, gt1000&<2500, gt2500&<4000, gt2500
tole1000, gt1000&<2500, ge2500&<4000, ge2500
.le1000, gt1000&<2500, gt2500&<4000, gt2500
tole1000, gt1000&<2500, ge2500&<4000, ge2500
.PointStat_ensprob
(runningPointStat
to calculate ensemble probabilistic fields):ADPSFC
fields:HGT_L0_ENS_FREQ_lt1520
was changed toHGT_L0_ENS_FREQ_lt305
.lt1520
tolt305
.HGT_L0_ENS_FREQ_ge914
was changed toHGT_L0_ENS_FREQ_lt914
.ge914
tolt914
.ADPUPA
fields:CAPE_L0_ENS_FREQ_gt2500.and.lt4000
was changed toCAPE_L0_ENS_FREQ_ge2500.and.lt4000
.gt2500&<4000
toge2500&<4000
.CAPE_L0_ENS_FREQ_gt2500
was changed toCAPE_L0_ENS_FREQ_ge2500
.gt2500
toge2500
.Note that these bug fixes caused (expected) changes in the outputs of the verification tasks.
Type of change
TESTS CONDUCTED:
The set of fundamental WE2E tests as well as all the verification tests were run on Hera with Intel. All completed successfully. The fundamental tests are:
The verification tests are:
Manual regression tests were also run on the following WE2E tests:
All four had several minor (and expected) differences in results relative to the
develop
branch that did not involve numerical values, only metadata values (e.g. an accumulation level went fromA03
toA3
). There were also some changes to numerical values, but all were expected consequences of the bug fixes in thresholds described above.DOCUMENTATION:
No documentation changes are needed.
CHECKLIST
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):
@michelleharrold @JeffBeck-NOAA