Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Streamline SRW App's interface to MET/METplus #1005

Merged

Conversation

gsketefian
Copy link
Collaborator

@gsketefian gsketefian commented Jan 22, 2024

DESCRIPTION OF CHANGES:

This PR streamlines the SRW App's interface to the MET/METplus verification tool and implements some bug fixes. Details:

  1. Replace the field-specific METplus configuration jinja2 templates associated with each METplus tool (these templates are hard-coded for each field) with a single template that contains jinja2 code to any valid field to be verified. For example, for the EnsembleStat tool, the six templates

    EnsembleStat_APCP.conf
    EnsembleStat_ASNOW.conf
    EnsembleStat_REFC.conf
    EnsembleStat_RETOP.conf
    EnsembleStat_ADPSFC.conf
    EnsembleStat_ADPUPA.conf
    

    are replaced by the single template EnsembleStat.conf.

  2. Add yaml configuration files for verification that specify the fields to verify (including field levels and thresholds). This is in order to consolidate the field/level/threshold information in one place instead of having it spread out and repeated in several hard-coded configuration files. Two such yaml configuration files are added:

    1. vx_config_det.yaml -- for deterministic verification
    2. vx_config_ens.yaml -- for ensemble verification
  3. Add a python script (decouple_fcst_obs_vx_config.py) to parse these two vx configuration files and create a dictionary of the field/level/threshold information that can then be passed to the unified workflow templating tool.

  4. Modify the ex-scripts for the verification tasks (exregional_run_met_....sh) to allow the use of the new jinja2 METplus config templates. This includes adding code to call the new script decouple_fcst_obs_vx_config.py and then passing its output to the unified workflow templating tool to generate METplus configuration files from the (new) jinja2 templates.

  5. Add new environment variables to the rocoto workflow configuration files (verify_[pre|det|ens].yaml) that are needed for using the new jinja2 METplus config templates.

  6. Bug fixes:

    1. For deterministic verification with METplus's PointStat tool, some of the fields to be verified were missing the set_attr_lead option in the METplus configuration file that accounts for time-lagging when setting the entry in the FCST_LEAD column of the output stat file. This option causes the FCST_LEAD entry of time-lagged members to have the same lead time as non-lagged members (i.e. it converts actual lead time to lead time assuming no time-lagging). This option was added where it was missing (e.g. if variable number 14 was missing this, the following line may have been added: FCST_VAR14_OPTIONS = set_attr_lead = "{lead?fmt=%H%M%S}";). The field/level combinations that were missing this option were:
      1. For PointStat verification of ADPSFC (surface) fields:
        The set_attr_lead option was added to SPFH at Z2, CRAIN at L0, CSNOW at L0, CFRZR at L0, and CICEP at L0.
      2. For PointStat verification of ADPUPA (upper air) fields:
        The set_attr_lead option was added to CAPE at L0-90.
    2. For ensemble verification, in unifying the threshold settings into the yaml configuration file vx_config_ens.yaml, the following bug fixes were automatically implemented:
      1. GenEnsProd:
        • ADPSFC fields:
          • (Forecast) Threshold of HGT at L0 was changed from lt152, lt1520, ge914 to lt152, lt305, lt914.
        • ADPUPA fields:
          • (Forecast) Threshold of CAPE at L0 was changed from le1000, gt1000&&lt2500, gt2500&&lt4000, gt2500 to le1000, gt1000&&lt2500, ge2500&&lt4000, ge2500.
      2. EnsembleStat:
        • ADPSFC fields:
          • Forecast threshold of HGT at L0 was changed from lt152, lt1520, ge914 to lt152, lt305, lt914.
          • Observation threshold of CEILING at L0 was changed from lt152, lt305, ge914 to lt152, lt305, lt914.
        • ADPUPA fields:
          • Forecast threshold of DPT at P700 was changed from ge263, ge286, ge273 to ge263, ge268, ge273.
          • Observation threshold of DPT at P700 was changed from ge263, ge286, ge273 to ge263, ge268, ge273.
      3. PointStat_ensmean (running PointStat to calculate ensemble mean fields):
        • ADPUPA fields:
          • Forecast threshold of DPT at P700 was changed from ge263, ge286, ge273 to ge263, ge268, ge273.
          • Observation threshold of DPT at P700 was changed from ge263, ge286, ge273 to ge263, ge268, ge273.
          • Forecast threshold of WIND at P250 was changed from ge26, ge31, ge46, ge62 to ge26, ge31, ge36, ge46, ge62.
          • Observation threshold of WIND at P250 was changed from ge26, ge31, ge46, ge62 to ge26, ge31, ge36, ge46, ge62.
          • Forecast threshold of CAPE_L0_ENS_MEAN at L0 was changed from le1000, gt1000&&lt2500, gt2500&&lt4000, gt2500 to le1000, gt1000&&lt2500, ge2500&&lt4000, ge2500.
          • Observation threshold of CAPE at L0-100000 was changed from le1000, gt1000&&lt2500, gt2500&&lt4000, gt2500 to le1000, gt1000&&lt2500, ge2500&&lt4000, ge2500.
      4. PointStat_ensprob (running PointStat to calculate ensemble probabilistic fields):
        • ADPSFC fields:
          • Name of forecast variable HGT_L0_ENS_FREQ_lt1520 was changed to HGT_L0_ENS_FREQ_lt305.
          • For the observation variable corresponding to this forecast variable, the observation threshold of CEILING at L0 was changed from lt1520 to lt305.
          • Name of forecast variable HGT_L0_ENS_FREQ_ge914 was changed to HGT_L0_ENS_FREQ_lt914.
          • For the observation variable corresponding to this forecast variable, the observation threshold of CEILING at L0 was changed from ge914 to lt914.
        • ADPUPA fields:
          • Name of forecast variable CAPE_L0_ENS_FREQ_gt2500.and.lt4000 was changed to CAPE_L0_ENS_FREQ_ge2500.and.lt4000.
          • For the observation variable corresponding to this forecast variable, the observation threshold of CAPE at L0-100000 was changed from gt2500&&lt4000 to ge2500&&lt4000.
          • Name of forecast variable CAPE_L0_ENS_FREQ_gt2500 was changed to CAPE_L0_ENS_FREQ_ge2500.
          • For the observation variable corresponding to this forecast variable, the observation threshold of CAPE at L0-100000 was changed from gt2500 to ge2500.

    Note that these bug fixes caused (expected) changes in the outputs of the verification tasks.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

TESTS CONDUCTED:

  • hera.intel
  • orion.intel
  • hercules.intel
  • cheyenne.intel
  • cheyenne.gnu
  • derecho.intel
  • gaea.intel
  • gaeac5.intel
  • jet.intel
  • wcoss2.intel
  • NOAA Cloud (indicate which platform)
  • Jenkins
  • fundamental test suite
  • comprehensive tests (specify which if a subset was used)

The set of fundamental WE2E tests as well as all the verification tests were run on Hera with Intel. All completed successfully. The fundamental tests are:

grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_GFS_v16
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16

The verification tests are:

MET_ensemble_verification
MET_ensemble_verification_only_vx
MET_ensemble_verification_only_vx_time_lag
MET_ensemble_verification_winter_wx
MET_verification
MET_verification_only_vx
MET_verification_winter_wx

Manual regression tests were also run on the following WE2E tests:

MET_verification_winter_wx [aka custom_ESGgrid_Great_Lakes_snow_8km]
MET_ensemble_verification_only_vx
MET_ensemble_verification_only_vx_time_lag
MET_ensemble_verification_winter_wx

All four had several minor (and expected) differences in results relative to the develop branch that did not involve numerical values, only metadata values (e.g. an accumulation level went from A03 to A3). There were also some changes to numerical values, but all were expected consequences of the bug fixes in thresholds described above.

DOCUMENTATION:

No documentation changes are needed.

CHECKLIST

  • My code follows the style guidelines in the Contributor's Guide
  • I have performed a self-review of my own code using the Code Reviewer's Guide
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes do not require updates to the documentation (explain).
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

  • Work In Progress
  • bug
  • enhancement
  • documentation
  • release
  • high priority
  • run_ci
  • run_we2e_fundamental_tests
  • run_we2e_comprehensive_tests
  • Needs Cheyenne test
  • Needs Jet test
  • Needs Hera test
  • Needs Orion test
  • help wanted

CONTRIBUTORS (optional):

@michelleharrold @JeffBeck-NOAA

…iable fieldname_in_MET_filedir_names (as it is with APCP).
… ">=", ">", "<=", and "<" to "ge", "gt", "le", and "lt".
… variables are grouped together, (2) commented-out settings of FCST_VAR<n>_... and OBS_VAR<n>_... are removed, (3) FCST_VAR<n>_OPTIONS and OBS_VAR<n>_OPTIONS come last, and (4) each element in options is on a separate line.
…tool names to template METplus conf files as variables; add new workflow variable that specifies the METplus verbosity level.
…ks for APCP (i.e. not just for accumulation > 1 hr but also for accumulation = 1 hour) use obs and forecast tasks that have been processed by the PcpCombine tool of METplus. These files are all in NetCDF format and are generated from grib2 input files (both for obs and forecast). This involves:

* Renaming certain workflow variables for verification so their role/purpose is clearer.
* Modifying the names of the arrays for APCP variables in the NetCDF files.
* Changing the names of the levels associated with APCP variables in the NetCDF files (e.g. "A1" instead of "A01").
* For consistency, making changes for accumulated snow (ASNOW) analogous to the above changes for APCP.
* In ex-scripts for vx tasks, removing if-statement that treats the case of APCP accumulation period > 1 hour separately and performing the same steps for both accumulation = 1 hour and > 1 hour.
…should be replaced with [OBS|FCST]_PCP_COMBINE_INPUT_DATATYPE. Thus, they are only relevant to METplus conf files that run PcpCombine. Thus, remove all use of these variables in conf files that do not run PcpCombine, and in the ones that do, replace them with the appropriate new variables (some already have the new variables).
…s for combining METplus conf files for APCP01h and APCPgt01h (inadvertantly left out of a previous commit).
…f files no longer differentiate between 1 hour accumulation and > 1 hour accumulation.
… to compare ADP[SFC|UPA] conf files with other branches.
… conf files with those generated by another branch.
…ese "BOTH" variables to two, one named "FCST_VAR<n>_..." and another named "OBS_VAR<n>_...".
@gsketefian
Copy link
Collaborator Author

gsketefian commented Apr 30, 2024

@mkavulich I cleaned up and renamed the script separate_fcst_obs_info.py. It is now at ush/metplus/decouple_fcst_obs_vx_config.py. Sorry, it wasn't ready for review before, and I had forgotten to go back and clean it up. It's pretty different from the previous version so feel free to take a look again. To address some of your comments:

  1. I've removed all the commented code.
  2. The print() statements are now replaced with logging.debug().
  3. I now use your code snippet to parse the ..._both values, and that code is now in a separate function to avoid repetition.
  4. There is now an argument to the main function (decouple_fcst_obs_vx_config) that specifies the format in which to write to the output file. It can be txt for pretty-printed text output or yaml for yaml output. We need txt for now because the output file is read in by the (bash) ex-scripts for some of the vx tasks (e.g. exregional_run_met_gridstat_or_pointstat_vx.sh), which then take the contents of that file and place it in the settings variable that gets passed to the uw templater tool. Changing the output of decouple_fcst_obs_vx_config.py to yaml format would require additional changes to those ex-scripts (e.g. to also use yaml for the remaining variables in settings), and I'd rather leave that to another PR.

A final change I made is to call decouple_fcst_obs_vx_config.py only once in a separate workflow task (well once for deterministic vx and another time for ensemble vx). Previously, it was being called for most vx tasks and generating the same output (and I think all those tasks were writing to the same file, which is bad). The output files from these two calls now get placed in the main experiment directory and are called vx_config_det.txt and vx_config_ens.txt.

I've tested this latest code on only one WE2E vx test (MET_ensemble_verification_only_vx). I am now running them on the remaining 6 and will get back here with results.

@gsketefian
Copy link
Collaborator Author

@MichaelLueken @mkavulich I tested the latest on the following vx WE2E tests on Hera, and they all passed:

MET_ensemble_verification
MET_ensemble_verification_only_vx
MET_ensemble_verification_only_vx_time_lag
MET_ensemble_verification_winter_wx
MET_verification
MET_verification_only_vx
MET_verification_winter_wx

Copy link
Collaborator

@MichaelLueken MichaelLueken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gsketefian - While running the verification WE2E tests, I did note two failing tasks:

  • parse_vx_config_det
  • parse_vx_config_ens

Looking in the logs, the reason for the failure is due to missing JREGIONAL_PARSE_VX_CONFIG j-job scripts:

/scratch2/NAGAPE/epic/Michael.Lueken/ufs-srweather-app/hera/ush/load_modules_run_task.sh: line 210: /scratch2/NAGAPE/epic/Michael.Lueken/ufs-srweather-app/hera/jobs/JREGIONAL_PARSE_VX_CONFIG: No such file or directory

Was this new j-job script not committed and pushed?

parm/wflow/verify_det.yaml Show resolved Hide resolved
parm/wflow/verify_ens.yaml Show resolved Hide resolved
@gsketefian
Copy link
Collaborator Author

@MichaelLueken Sorry, I forgot to add the j-job and ex-script for the new jobs. Both jobs use the same j-job and ex-script. They should be there now.

@mkavulich
Copy link
Collaborator

@gsketefian Thanks for addressing my earlier comments. I am a bit concerned about the amount of custom code that the latest commits are adding for writing this custom text format to be read in by metplus tasks rather than just using the original yaml. Is this because of the format that METplus is expecting for config files?

Copy link
Collaborator

@mkavulich mkavulich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't hold this PR up any further, but I do have a few more suggestions. Hopefully we can move from this custom text format to yaml in the future to reduce the amount of custom code we need to maintain.

Comment on lines 130 to 147
Alternatively, if delim_str is not be a substring within item_cpld so that
item_cpld has the form

item_cpld = str1

then item_fcst and item_obs are given by

item_fcst = str1
item_obs = str1

For example, if

item_cpld = 'ABCD'

then

item_fcst = 'ABCD'
item_obs = 'ABCD'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Alternatively, if delim_str is not be a substring within item_cpld so that
item_cpld has the form
item_cpld = str1
then item_fcst and item_obs are given by
item_fcst = str1
item_obs = str1
For example, if
item_cpld = 'ABCD'
then
item_fcst = 'ABCD'
item_obs = 'ABCD'
Alternatively, if the input string does not contain the delimiter string delim_str, both return values will be identical to the input.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shortened/modified. Will show up in the next commit(s) I push.

Comment on lines 103 to 109
Function to parse the "coupled" value of an item (obtained from the coupled
verification (vx) configuration dictionary) to extract from it the item's
value for forecasts and its value for observations. The coupled item
(item_cpld) is a string that may correspond to a field name, a level, or
a threshold. It can be the name of a key in a key-value pair in the
coupled vx configuration dictionary, or it can be the value (or, if the
value is a list of strings, one of the elements in that list).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest cutting down less relevant information to make the docstring more readable

Suggested change
Function to parse the "coupled" value of an item (obtained from the coupled
verification (vx) configuration dictionary) to extract from it the item's
value for forecasts and its value for observations. The coupled item
(item_cpld) is a string that may correspond to a field name, a level, or
a threshold. It can be the name of a key in a key-value pair in the
coupled vx configuration dictionary, or it can be the value (or, if the
value is a list of strings, one of the elements in that list).
Function to parse the "coupled" value of an item (obtained from the coupled
verification (vx) configuration dictionary) to extract from it the item's
value for forecasts and its value for observations. The coupled item
(item_cpld) is a string that may correspond to a field name, a level, or
a threshold.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shortened. Will show up in the next commit(s) I push.

# Set up logging.
log_level = str.upper(log_lvl)
fmt = "[%(levelname)s:%(name)s: %(filename)s, line %(lineno)s: %(funcName)s()] %(message)s"
if log_fp:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't relevant for now, but in my experience logging.basicConfig does not work as expected if we already have a python logger set up; worth remembering for the future if we move towards python-based exscripts.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.

@MichaelLueken
Copy link
Collaborator

@gsketefian -

Thank you very much for adding the JREGIONAL_PARSE_VX_CONFIG j-job and the exregional_parse_vx_config.sh ex-script! The verification WE2E tests are now successfully passing:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
MET_ensemble_verification_only_vx_20240501151627                   COMPLETE               0.99
MET_ensemble_verification_only_vx_time_lag_20240501151629          COMPLETE               3.75
MET_ensemble_verification_winter_wx_20240501151632                 COMPLETE             109.74
MET_verification_20240501151634                                    COMPLETE               9.82
MET_verification_only_vx_20240501151635                            COMPLETE               0.23
MET_verification_winter_wx_20240501151637                          COMPLETE              15.99
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202405  COMPLETE              43.25
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024050115164  COMPLETE              19.67
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             203.44

Now preparing to kick off Jenkins testing.

@MichaelLueken MichaelLueken added run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests and removed Needs Hera test Testing needs to be run on Hera machine labels May 1, 2024
@gsketefian
Copy link
Collaborator Author

@gsketefian Thanks for addressing my earlier comments. I am a bit concerned about the amount of custom code that the latest commits are adding for writing this custom text format to be read in by metplus tasks rather than just using the original yaml. Is this because of the format that METplus is expecting for config files?

@mkavulich The files being read in by decouple_fcst_obs_vx_config.py are yaml (these files are vx_config_det.yaml and vx_config_ens.yaml; I'm not sure what you mean by "the original yaml"). Their contents are in the form

FIELD_GROUP1:
    FIELD1:
        LEVEL1: list_of_thresholds
        LEVEL2: list_of_thresholds
        ...
    FIELD2:
        LEVEL1: list_of_thresholds
        LEVEL2: list_of_thresholds
        ...
    ...
...

Aside: The FIELD_GROUPs are the groups of fields for which we've pooled the verification into SRW workflow tasks, e.g. for the field group ADPSFC, we verify a bunch of fields like TMP, DPT, RH, etc, but for the field group REFC, we verify only the field REFC.

The parsing by decouple_fcst_obs_vx_config.py is needed because each item in this file may contain both the forecast and observation value, e.g. the key FIELD1 may have the form FIELD1_fcst%%FIELD1_obs, where %% is the delimiter string. I do this to avoid needing another level of configuration at the top that distinguishes between forecast and observation values, e.g.

fcst:
    FIELD_GROUP1_fcst:
    ...

obs:
    FIELD_GROUP1_obs:
    ...

I know there's the &label and <<: *label mechanism in yaml, but even with that it seems to me things would get messy. The main difficulty is that there are cases where the names of different forecast fields correspond to the same observation field name. That's a problem because that observation field name can only be used once as a dictionary key, so you'd have to resort to some convoluted solution. I also like this "coupled" approach because the forecast and obs values are always right next to each other; no need to introduce labels, and no need to search in the config file for what obs item further down matches a fcst item up above. This was a much more compact (and to me, intuitive) solution for this situation.

@MichaelLueken
Copy link
Collaborator

The Jenkins tests have all successfully passed. Moving forward with merging this work now.

@MichaelLueken MichaelLueken merged commit 2d94ed4 into ufs-community:develop May 1, 2024
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants