Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UFS-dev PR#131 #120

Merged
merged 6 commits into from
Feb 23, 2024
Merged

UFS-dev PR#131 #120

merged 6 commits into from
Feb 23, 2024

Conversation

grantfirl
Copy link
Collaborator

@grantfirl grantfirl commented Feb 6, 2024

@grantfirl
Copy link
Collaborator Author

Expected RT failures from the UFS develop branch:

Few tests pass. They include hrrr tests, datm_cdeps tests and some hafs tests.

Three tests timed out on the initial test, but ran successfully when retested. The three tests which timed-out were

Test 115 conus13km_debug_intel FAIL
Test 129 hafs_regional_specified_moving_1nest_atm_intel FAIL
Test 133 hafs_regional_storm_following_1nest_atm_ocn_debug_intel FAIL
The following tests failed with non-B4B comparisons.

Test 001 cpld_control_p8_mixedmode_intel FAIL
Test 002 cpld_control_p8_intel FAIL
Test 004 cpld_control_qr_p8_intel FAIL
Test 006 cpld_2threads_p8_intel FAIL
Test 007 cpld_decomp_p8_intel FAIL
Test 008 cpld_mpi_p8_intel FAIL
Test 009 cpld_control_ciceC_p8_intel FAIL
Test 010 cpld_control_c192_p8_intel FAIL
Test 012 cpld_bmark_p8_intel FAIL
Test 014 cpld_control_noaero_p8_intel FAIL
Test 015 cpld_control_nowave_noaero_p8_intel FAIL
Test 016 cpld_debug_p8_intel FAIL
Test 017 cpld_debug_noaero_p8_intel FAIL
Test 019 cpld_control_c48_intel FAIL
Test 020 cpld_control_p8_faster_intel FAIL
Test 021 control_flake_intel FAIL
Test 022 control_CubedSphereGrid_intel FAIL
Test 023 control_CubedSphereGrid_parallel_intel FAIL
Test 024 control_latlon_intel FAIL
Test 025 control_wrtGauss_netcdf_parallel_intel FAIL
Test 026 control_c48_intel FAIL
Test 027 control_c192_intel FAIL
Test 028 control_c384_intel FAIL
Test 029 control_c384gdas_intel FAIL
Test 030 control_stochy_intel FAIL
Test 032 control_lndp_intel FAIL
Test 033 control_iovr4_intel FAIL
Test 034 control_iovr5_intel FAIL
Test 035 control_p8_intel FAIL
Test 036 control_p8_ugwpv1_intel FAIL
Test 038 control_noqr_p8_intel FAIL
Test 040 control_decomp_p8_intel FAIL
Test 041 control_2threads_p8_intel FAIL
Test 042 control_p8_lndp_intel FAIL
Test 043 control_p8_rrtmgp_intel FAIL
Test 044 control_p8_mynn_intel FAIL
Test 045 merra2_thompson_intel FAIL
Test 046 regional_control_intel FAIL
Test 048 regional_decomp_intel FAIL
Test 049 regional_2threads_intel FAIL
Test 050 regional_noquilt_intel FAIL
Test 051 regional_netcdf_parallel_intel FAIL
Test 052 regional_2dwrtdecomp_intel FAIL
Test 053 regional_wofs_intel FAIL
Test 054 rap_control_intel FAIL
Test 056 rap_decomp_intel FAIL
Test 057 rap_2threads_intel FAIL
Test 059 rap_sfcdiff_intel FAIL
Test 060 rap_sfcdiff_decomp_intel FAIL
Test 066 rrfs_v1beta_intel FAIL
Test 067 rrfs_v1nssl_intel FAIL
Test 068 rrfs_v1nssl_nohailnoccn_intel FAIL
Test 069 control_csawmg_intel FAIL
Test 070 control_csawmgt_intel FAIL
Test 071 control_ras_intel FAIL
Test 073 control_p8_faster_intel FAIL
Test 074 regional_control_faster_intel FAIL
Test 075 control_CubedSphereGrid_debug_intel FAIL
Test 076 control_wrtGauss_netcdf_parallel_debug_intel FAIL
Test 077 control_stochy_debug_intel FAIL
Test 078 control_lndp_debug_intel FAIL
Test 079 control_csawmg_debug_intel FAIL
Test 080 control_csawmgt_debug_intel FAIL
Test 081 control_ras_debug_intel FAIL
Test 082 control_diag_debug_intel FAIL
Test 083 control_debug_p8_intel FAIL
Test 084 regional_debug_intel FAIL
Test 085 rap_control_debug_intel FAIL
Test 089 rap_unified_drag_suite_debug_intel FAIL
Test 090 rap_diag_debug_intel FAIL
Test 091 rap_cires_ugwp_debug_intel FAIL
Test 092 rap_unified_ugwp_debug_intel FAIL
Test 093 rap_lndp_debug_intel FAIL
Test 094 rap_progcld_thompson_debug_intel FAIL
Test 095 rap_noah_debug_intel FAIL
Test 096 rap_sfcdiff_debug_intel FAIL
Test 097 rap_noah_sfcdiff_cires_ugwp_debug_intel FAIL
Test 098 rrfs_v1beta_debug_intel FAIL
Test 099 rap_clm_lake_debug_intel FAIL
Test 100 rap_flake_debug_intel FAIL
Test 102 rap_control_dyn32_phy32_intel FAIL
Test 104 rap_2threads_dyn32_phy32_intel FAIL
Test 112 rap_control_dyn64_phy32_intel FAIL
Test 113 rap_control_debug_dyn32_phy32_intel FAIL
Test 115 conus13km_debug_intel FAIL
Test 119 rap_control_dyn64_phy32_debug_intel FAIL
Test 120 hafs_regional_atm_intel FAIL
Test 121 hafs_regional_atm_thompson_gfdlsf_intel FAIL
Test 123 hafs_regional_atm_wav_intel FAIL
Test 125 hafs_regional_1nest_atm_intel FAIL
Test 126 hafs_regional_telescopic_2nests_atm_intel FAIL
Test 127 hafs_global_1nest_atm_intel FAIL
Test 128 hafs_global_multiple_4nests_atm_intel FAIL
Test 129 hafs_regional_specified_moving_1nest_atm_intel FAIL
Test 130 hafs_regional_storm_following_1nest_atm_intel FAIL
Test 132 hafs_global_storm_following_1nest_atm_intel FAIL
Test 133 hafs_regional_storm_following_1nest_atm_ocn_debug_intel FAIL
Test 155 control_p8_atmlnd_sbs_intel FAIL
Test 156 atmwav_control_noaero_p8_intel FAIL
Test 157 control_atmwav_intel FAIL
Test 158 atmaero_control_p8_intel FAIL
Test 159 atmaero_control_p8_rad_intel FAIL
Test 160 atmaero_control_p8_rad_micro_intel FAIL
Test 161 regional_atmaq_intel FAIL
Test 162 regional_atmaq_debug_intel FAIL
Test 163 regional_atmaq_faster_intel FAIL
Test 164 control_c48_gnu FAIL
Test 165 control_stochy_gnu FAIL
Test 166 control_ras_gnu FAIL
Test 167 control_p8_gnu FAIL
Test 168 control_p8_ugwpv1_gnu FAIL
Test 169 control_flake_gnu FAIL
Test 170 rap_control_gnu FAIL
Test 171 rap_decomp_gnu FAIL
Test 172 rap_2threads_gnu FAIL
Test 174 rap_sfcdiff_gnu FAIL
Test 175 rap_sfcdiff_decomp_gnu FAIL
Test 183 rrfs_v1beta_gnu FAIL
Test 184 control_diag_debug_gnu FAIL
Test 185 regional_debug_gnu FAIL
Test 186 rap_control_debug_gnu FAIL
Test 190 rap_diag_debug_gnu FAIL
Test 191 rap_noah_sfcdiff_cires_ugwp_debug_gnu FAIL
Test 192 rap_progcld_thompson_debug_gnu FAIL
Test 193 rrfs_v1beta_debug_gnu FAIL
Test 194 control_ras_debug_gnu FAIL
Test 195 control_stochy_debug_gnu FAIL
Test 196 control_debug_p8_gnu FAIL
Test 197 rap_flake_debug_gnu FAIL
Test 198 rap_clm_lake_debug_gnu FAIL
Test 199 rap_control_dyn32_phy32_gnu FAIL
Test 201 rap_2threads_dyn32_phy32_gnu FAIL
Test 209 rap_control_dyn64_phy32_gnu FAIL
Test 210 rap_control_debug_dyn32_phy32_gnu FAIL
Test 216 rap_control_dyn64_phy32_debug_gnu FAIL
Test 217 cpld_control_p8_gnu FAIL
Test 218 cpld_control_nowave_noaero_p8_gnu FAIL
Test 219 cpld_debug_p8_gnu FAIL
Test 220 cpld_control_pdlib_p8_gnu FAIL
Test 221 cpld_debug_pdlib_p8_gnu FAIL

@grantfirl grantfirl marked this pull request as ready for review February 19, 2024 17:28
@mkavulich mkavulich added hera-RT Run regression test on Hera machine. TESTING ONLY, NOT FOR GENERAL PRS YET and removed hera-RT Run regression test on Hera machine. TESTING ONLY, NOT FOR GENERAL PRS YET labels Feb 20, 2024
@mkavulich
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Job: RT
[RT] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/beta/run//1714125607/20240220195511/ufs-weather-model
[RT] Error: Test 001 cpld_control_p8_mixedmode_intel FAIL Tries: 2
[RT] Error: Test 002 cpld_control_gfsv17_intel FAIL Tries: 2
[RT] Error: Test 005 cpld_mpi_gfsv17_intel FAIL Tries: 2
[RT] Error: Test 006 cpld_debug_gfsv17_intel FAIL Tries: 2
[RT] Error: Test 007 cpld_control_p8_intel FAIL Tries: 2
[RT] Error: Test 009 cpld_control_qr_p8_intel FAIL Tries: 2
[RT] Error: Test 011 cpld_2threads_p8_intel FAIL Tries: 2
[RT] Error: Test 012 cpld_decomp_p8_intel FAIL Tries: 2
[RT] Error: Test 013 cpld_mpi_p8_intel FAIL Tries: 2
[RT] Error: Test 014 cpld_control_ciceC_p8_intel FAIL Tries: 2
[RT] Error: Test 015 cpld_control_c192_p8_intel FAIL Tries: 2
[RT] Error: Test 017 cpld_bmark_p8_intel FAIL Tries: 2
[RT] Error: Test 019 cpld_control_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 020 cpld_control_nowave_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 021 cpld_debug_p8_intel FAIL Tries: 2
[RT] Error: Test 022 cpld_debug_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 024 cpld_control_c48_intel FAIL Tries: 2
[RT] Error: Test 025 cpld_control_p8_faster_intel FAIL Tries: 2
[RT] Error: Test 026 cpld_control_pdlib_p8_intel FAIL Tries: 2
[RT] Error: Test 029 cpld_debug_pdlib_p8_intel FAIL Tries: 2
[RT] Error: Test 030 control_flake_intel FAIL Tries: 2
[RT] Error: Test 031 control_CubedSphereGrid_intel FAIL Tries: 2
[RT] Error: Test 032 control_CubedSphereGrid_parallel_intel FAIL Tries: 2
[RT] Error: Test 033 control_latlon_intel FAIL Tries: 2
[RT] Error: Test 034 control_wrtGauss_netcdf_parallel_intel FAIL Tries: 2
[RT] Error: Test 035 control_c48_intel FAIL Tries: 2
[RT] Error: Test 036 control_c192_intel FAIL Tries: 2
[RT] Error: Test 037 control_c384_intel FAIL Tries: 2
[RT] Error: Test 038 control_c384gdas_intel FAIL Tries: 2
[RT] Error: Test 039 control_stochy_intel FAIL Tries: 2
[RT] Error: Test 041 control_lndp_intel FAIL Tries: 2
[RT] Error: Test 042 control_iovr4_intel FAIL Tries: 2
[RT] Error: Test 043 control_iovr5_intel FAIL Tries: 2
[RT] Error: Test 044 control_p8_intel FAIL Tries: 2
[RT] Error: Test 045 control_p8_ugwpv1_intel FAIL Tries: 2
[RT] Error: Test 047 control_noqr_p8_intel FAIL Tries: 2
[RT] Error: Test 049 control_decomp_p8_intel FAIL Tries: 2
[RT] Error: Test 050 control_2threads_p8_intel FAIL Tries: 2
[RT] Error: Test 051 control_p8_lndp_intel FAIL Tries: 2
[RT] Error: Test 052 control_p8_rrtmgp_intel FAIL Tries: 2
[RT] Error: Test 053 control_p8_mynn_intel FAIL Tries: 2
[RT] Error: Test 054 merra2_thompson_intel FAIL Tries: 2
[RT] Error: Test 055 regional_control_intel FAIL Tries: 2
[RT] Error: Test 057 regional_decomp_intel FAIL Tries: 2
[RT] Error: Test 058 regional_2threads_intel FAIL Tries: 2
[RT] Error: Test 059 regional_noquilt_intel FAIL Tries: 2
[RT] Error: Test 060 regional_netcdf_parallel_intel FAIL Tries: 2
[RT] Error: Test 061 regional_2dwrtdecomp_intel FAIL Tries: 2
[RT] Error: Test 062 regional_wofs_intel FAIL Tries: 2
[RT] Error: Test 063 rap_control_intel FAIL Tries: 2
[RT] Error: Test 065 rap_decomp_intel FAIL Tries: 2
[RT] Error: Test 066 rap_2threads_intel FAIL Tries: 2
[RT] Error: Test 068 rap_sfcdiff_intel FAIL Tries: 2
[RT] Error: Test 069 rap_sfcdiff_decomp_intel FAIL Tries: 2
[RT] Error: Test 075 rrfs_v1beta_intel FAIL Tries: 2
[RT] Error: Test 076 rrfs_v1nssl_intel FAIL Tries: 2
[RT] Error: Test 077 rrfs_v1nssl_nohailnoccn_intel FAIL Tries: 2
[RT] Error: Test 078 control_csawmg_intel FAIL Tries: 2
[RT] Error: Test 079 control_csawmgt_intel FAIL Tries: 2
[RT] Error: Test 080 control_ras_intel FAIL Tries: 2
[RT] Error: Test 082 control_p8_faster_intel FAIL Tries: 2
[RT] Error: Test 083 regional_control_faster_intel FAIL Tries: 2
[RT] Error: Test 084 control_CubedSphereGrid_debug_intel FAIL Tries: 2
[RT] Error: Test 085 control_wrtGauss_netcdf_parallel_debug_intel FAIL Tries: 2
[RT] Error: Test 086 control_stochy_debug_intel FAIL Tries: 2
[RT] Error: Test 087 control_lndp_debug_intel FAIL Tries: 2
[RT] Error: Test 088 control_csawmg_debug_intel FAIL Tries: 2
[RT] Error: Test 089 control_csawmgt_debug_intel FAIL Tries: 2
[RT] Error: Test 090 control_ras_debug_intel FAIL Tries: 2
[RT] Error: Test 091 control_diag_debug_intel FAIL Tries: 2
[RT] Error: Test 092 control_debug_p8_intel FAIL Tries: 2
[RT] Error: Test 093 regional_debug_intel FAIL Tries: 2
[RT] Error: Test 094 rap_control_debug_intel FAIL Tries: 2
[RT] Error: Test 098 rap_unified_drag_suite_debug_intel FAIL Tries: 2
[RT] Error: Test 099 rap_diag_debug_intel FAIL Tries: 2
[RT] Error: Test 100 rap_cires_ugwp_debug_intel FAIL Tries: 2
[RT] Error: Test 101 rap_unified_ugwp_debug_intel FAIL Tries: 2
[RT] Error: Test 102 rap_lndp_debug_intel FAIL Tries: 2
[RT] Error: Test 103 rap_progcld_thompson_debug_intel FAIL Tries: 2
[RT] Error: Test 104 rap_noah_debug_intel FAIL Tries: 2
[RT] Error: Test 105 rap_sfcdiff_debug_intel FAIL Tries: 2
[RT] Error: Test 106 rap_noah_sfcdiff_cires_ugwp_debug_intel FAIL Tries: 2
[RT] Error: Test 107 rrfs_v1beta_debug_intel FAIL Tries: 2
[RT] Error: Test 108 rap_clm_lake_debug_intel FAIL Tries: 2
[RT] Error: Test 109 rap_flake_debug_intel FAIL Tries: 2
[RT] Error: Test 110 gnv1_c96_no_nest_debug_intel FAIL Tries: 2
[RT] Error: Test 113 rap_control_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 115 rap_2threads_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 123 rap_control_dyn64_phy32_intel FAIL Tries: 2
[RT] Error: Test 124 rap_control_debug_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 130 rap_control_dyn64_phy32_debug_intel FAIL Tries: 2
[RT] Error: Test 131 hafs_regional_atm_intel FAIL Tries: 2
[RT] Error: Test 132 hafs_regional_atm_thompson_gfdlsf_intel FAIL Tries: 2
[RT] Error: Test 134 hafs_regional_atm_wav_intel FAIL Tries: 2
[RT] Error: Test 136 hafs_regional_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 137 hafs_regional_telescopic_2nests_atm_intel FAIL Tries: 2
[RT] Error: Test 138 hafs_global_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 139 hafs_global_multiple_4nests_atm_intel FAIL Tries: 2
[RT] Error: Test 140 hafs_regional_specified_moving_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 141 hafs_regional_storm_following_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 143 hafs_global_storm_following_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 144 gnv1_nested_intel FAIL Tries: 2
[RT] Error: Test 167 control_p8_atmlnd_sbs_intel FAIL Tries: 2
[RT] Error: Test 168 atmwav_control_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 169 control_atmwav_intel FAIL Tries: 2
[RT] Error: Test 170 atmaero_control_p8_intel FAIL Tries: 2
[RT] Error: Test 171 atmaero_control_p8_rad_intel FAIL Tries: 2
[RT] Error: Test 172 atmaero_control_p8_rad_micro_intel FAIL Tries: 2
[RT] Error: Test 173 regional_atmaq_intel FAIL Tries: 2
[RT] Error: Test 174 regional_atmaq_debug_intel FAIL Tries: 2
[RT] Error: Test 175 regional_atmaq_faster_intel FAIL Tries: 2
[RT] Error: Test 176 control_c48_gnu FAIL Tries: 2
[RT] Error: Test 177 control_stochy_gnu FAIL Tries: 2
[RT] Error: Test 178 control_ras_gnu FAIL Tries: 2
[RT] Error: Test 179 control_p8_gnu FAIL Tries: 2
[RT] Error: Test 180 control_p8_ugwpv1_gnu FAIL Tries: 2
[RT] Error: Test 181 control_flake_gnu FAIL Tries: 2
[RT] Error: Test 182 rap_control_gnu FAIL Tries: 2
[RT] Error: Test 183 rap_decomp_gnu FAIL Tries: 2
[RT] Error: Test 184 rap_2threads_gnu FAIL Tries: 2
[RT] Error: Test 186 rap_sfcdiff_gnu FAIL Tries: 2
[RT] Error: Test 187 rap_sfcdiff_decomp_gnu FAIL Tries: 2
[RT] Error: Test 195 rrfs_v1beta_gnu FAIL Tries: 2
[RT] Error: Test 196 control_diag_debug_gnu FAIL Tries: 2
[RT] Error: Test 197 regional_debug_gnu FAIL Tries: 2
[RT] Error: Test 198 rap_control_debug_gnu FAIL Tries: 2
[RT] Error: Test 202 rap_diag_debug_gnu FAIL Tries: 2
[RT] Error: Test 203 rap_noah_sfcdiff_cires_ugwp_debug_gnu FAIL Tries: 2
[RT] Error: Test 204 rap_progcld_thompson_debug_gnu FAIL Tries: 2
[RT] Error: Test 205 rrfs_v1beta_debug_gnu FAIL Tries: 2
[RT] Error: Test 206 control_ras_debug_gnu FAIL Tries: 2
[RT] Error: Test 207 control_stochy_debug_gnu FAIL Tries: 2
[RT] Error: Test 208 control_debug_p8_gnu FAIL Tries: 2
[RT] Error: Test 209 rap_flake_debug_gnu FAIL Tries: 2
[RT] Error: Test 210 rap_clm_lake_debug_gnu FAIL Tries: 2
[RT] Error: Test 211 gnv1_c96_no_nest_debug_gnu FAIL Tries: 2
[RT] Error: Test 213 rap_control_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 215 rap_2threads_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 223 rap_control_dyn64_phy32_gnu FAIL Tries: 2
[RT] Error: Test 224 rap_control_debug_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 230 rap_control_dyn64_phy32_debug_gnu FAIL Tries: 2
[RT] Error: Test 231 cpld_control_p8_gnu FAIL Tries: 2
[RT] Error: Test 232 cpld_control_nowave_noaero_p8_gnu FAIL Tries: 2
[RT] Error: Test 233 cpld_debug_p8_gnu FAIL Tries: 2
[RT] Error: Test 234 cpld_control_pdlib_p8_gnu FAIL Tries: 2
[RT] Error: Test 235 cpld_debug_pdlib_p8_gnu FAIL Tries: 2
[RT] Log file shows failures.
[RT] Please obtain logs from /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/beta/run//1714125607/20240220195511/ufs-weather-model

@mkavulich
Copy link
Collaborator

@grantfirl It looks like the numbering of your tests is different than the one that ran....do you know what causes that? I am worried that it has something to do with how I set up the latest tests and it might not have been run correctly.

@grantfirl
Copy link
Collaborator Author

@mkavulich I'm guessing this is a result of the original PR RT failures being reported from the Hercules platform, which is probably running different tests in rt.conf, causing the numbering mismatch. The failures look consistent with the description to me. I think we're OK to run new the new BL tag.

@mkavulich mkavulich added the hera-BL Create new baselines on Hera machine. TESTING ONLY, NOT FOR GENERAL PRS YET label Feb 21, 2024
@mkavulich
Copy link
Collaborator

Ah that makes sense. Starting the baseline generation now.

@mkavulich mkavulich removed the hera-BL Create new baselines on Hera machine. TESTING ONLY, NOT FOR GENERAL PRS YET label Feb 21, 2024
@mkavulich
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Job: BL
[BL] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/beta/run//1714125607/20240221021513/ufs-weather-model
Baseline creation successful on hera
[RT] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/beta/run//1714125607/20240221023821/ufs-weather-model
[RT] Error: Test 001 cpld_control_p8_mixedmode_intel FAIL Tries: 2
[RT] Error: Test 002 cpld_control_gfsv17_intel FAIL Tries: 2
[RT] Error: Test 005 cpld_mpi_gfsv17_intel FAIL Tries: 2
[RT] Error: Test 006 cpld_debug_gfsv17_intel FAIL Tries: 2
[RT] Error: Test 007 cpld_control_p8_intel FAIL Tries: 2
[RT] Error: Test 009 cpld_control_qr_p8_intel FAIL Tries: 2
[RT] Error: Test 011 cpld_2threads_p8_intel FAIL Tries: 2
[RT] Error: Test 012 cpld_decomp_p8_intel FAIL Tries: 2
[RT] Error: Test 013 cpld_mpi_p8_intel FAIL Tries: 2
[RT] Error: Test 014 cpld_control_ciceC_p8_intel FAIL Tries: 2
[RT] Error: Test 015 cpld_control_c192_p8_intel FAIL Tries: 2
[RT] Error: Test 017 cpld_bmark_p8_intel FAIL Tries: 2
[RT] Error: Test 019 cpld_control_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 020 cpld_control_nowave_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 021 cpld_debug_p8_intel FAIL Tries: 2
[RT] Error: Test 022 cpld_debug_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 023 cpld_control_noaero_p8_agrid_intel FAIL Tries: 2
[RT] Error: Test 024 cpld_control_c48_intel FAIL Tries: 2
[RT] Error: Test 025 cpld_control_p8_faster_intel FAIL Tries: 2
[RT] Error: Test 026 cpld_control_pdlib_p8_intel FAIL Tries: 2
[RT] Error: Test 029 cpld_debug_pdlib_p8_intel FAIL Tries: 2
[RT] Error: Test 030 control_flake_intel FAIL Tries: 2
[RT] Error: Test 031 control_CubedSphereGrid_intel FAIL Tries: 2
[RT] Error: Test 032 control_CubedSphereGrid_parallel_intel FAIL Tries: 2
[RT] Error: Test 033 control_latlon_intel FAIL Tries: 2
[RT] Error: Test 034 control_wrtGauss_netcdf_parallel_intel FAIL Tries: 2
[RT] Error: Test 035 control_c48_intel FAIL Tries: 2
[RT] Error: Test 036 control_c192_intel FAIL Tries: 2
[RT] Error: Test 037 control_c384_intel FAIL Tries: 2
[RT] Error: Test 038 control_c384gdas_intel FAIL Tries: 2
[RT] Error: Test 039 control_stochy_intel FAIL Tries: 2
[RT] Error: Test 041 control_lndp_intel FAIL Tries: 2
[RT] Error: Test 042 control_iovr4_intel FAIL Tries: 2
[RT] Error: Test 043 control_iovr5_intel FAIL Tries: 2
[RT] Error: Test 045 control_p8_ugwpv1_intel FAIL Tries: 2
[RT] Error: Test 051 control_p8_lndp_intel FAIL Tries: 2
[RT] Error: Test 052 control_p8_rrtmgp_intel FAIL Tries: 2
[RT] Error: Test 053 control_p8_mynn_intel FAIL Tries: 2
[RT] Error: Test 054 merra2_thompson_intel FAIL Tries: 2
[RT] Error: Test 055 regional_control_intel FAIL Tries: 2
[RT] Error: Test 057 regional_decomp_intel FAIL Tries: 2
[RT] Error: Test 058 regional_2threads_intel FAIL Tries: 2
[RT] Error: Test 059 regional_noquilt_intel FAIL Tries: 2
[RT] Error: Test 060 regional_netcdf_parallel_intel FAIL Tries: 2
[RT] Error: Test 061 regional_2dwrtdecomp_intel FAIL Tries: 2
[RT] Error: Test 062 regional_wofs_intel FAIL Tries: 2
[RT] Error: Test 063 rap_control_intel FAIL Tries: 2
[RT] Error: Test 064 regional_spp_sppt_shum_skeb_intel FAIL Tries: 2
[RT] Error: Test 065 rap_decomp_intel FAIL Tries: 2
[RT] Error: Test 066 rap_2threads_intel FAIL Tries: 2
[RT] Error: Test 068 rap_sfcdiff_intel FAIL Tries: 2
[RT] Error: Test 069 rap_sfcdiff_decomp_intel FAIL Tries: 2
[RT] Error: Test 071 hrrr_control_intel FAIL Tries: 2
[RT] Error: Test 072 hrrr_control_decomp_intel FAIL Tries: 2
[RT] Error: Test 073 hrrr_control_2threads_intel FAIL Tries: 2
[RT] Error: Test 075 rrfs_v1beta_intel FAIL Tries: 2
[RT] Error: Test 076 rrfs_v1nssl_intel FAIL Tries: 2
[RT] Error: Test 077 rrfs_v1nssl_nohailnoccn_intel FAIL Tries: 2
[RT] Error: Test 078 control_csawmg_intel FAIL Tries: 2
[RT] Error: Test 079 control_csawmgt_intel FAIL Tries: 2
[RT] Error: Test 080 control_ras_intel FAIL Tries: 2
[RT] Error: Test 081 control_wam_intel FAIL Tries: 2
[RT] Error: Test 082 control_p8_faster_intel FAIL Tries: 2
[RT] Error: Test 083 regional_control_faster_intel FAIL Tries: 2
[RT] Error: Test 084 control_CubedSphereGrid_debug_intel FAIL Tries: 2
[RT] Error: Test 085 control_wrtGauss_netcdf_parallel_debug_intel FAIL Tries: 2
[RT] Error: Test 086 control_stochy_debug_intel FAIL Tries: 2
[RT] Error: Test 087 control_lndp_debug_intel FAIL Tries: 2
[RT] Error: Test 088 control_csawmg_debug_intel FAIL Tries: 2
[RT] Error: Test 089 control_csawmgt_debug_intel FAIL Tries: 2
[RT] Error: Test 090 control_ras_debug_intel FAIL Tries: 2
[RT] Error: Test 091 control_diag_debug_intel FAIL Tries: 2
[RT] Error: Test 092 control_debug_p8_intel FAIL Tries: 2
[RT] Error: Test 093 regional_debug_intel FAIL Tries: 2
[RT] Error: Test 094 rap_control_debug_intel FAIL Tries: 2
[RT] Error: Test 095 hrrr_control_debug_intel FAIL Tries: 2
[RT] Error: Test 096 hrrr_gf_debug_intel FAIL Tries: 2
[RT] Error: Test 097 hrrr_c3_debug_intel FAIL Tries: 2
[RT] Error: Test 098 rap_unified_drag_suite_debug_intel FAIL Tries: 2
[RT] Error: Test 099 rap_diag_debug_intel FAIL Tries: 2
[RT] Error: Test 100 rap_cires_ugwp_debug_intel FAIL Tries: 2
[RT] Error: Test 101 rap_unified_ugwp_debug_intel FAIL Tries: 2
[RT] Error: Test 102 rap_lndp_debug_intel FAIL Tries: 2
[RT] Error: Test 103 rap_progcld_thompson_debug_intel FAIL Tries: 2
[RT] Error: Test 104 rap_noah_debug_intel FAIL Tries: 2
[RT] Error: Test 105 rap_sfcdiff_debug_intel FAIL Tries: 2
[RT] Error: Test 106 rap_noah_sfcdiff_cires_ugwp_debug_intel FAIL Tries: 2
[RT] Error: Test 107 rrfs_v1beta_debug_intel FAIL Tries: 2
[RT] Error: Test 108 rap_clm_lake_debug_intel FAIL Tries: 2
[RT] Error: Test 109 rap_flake_debug_intel FAIL Tries: 2
[RT] Error: Test 110 gnv1_c96_no_nest_debug_intel FAIL Tries: 2
[RT] Error: Test 111 control_wam_debug_intel FAIL Tries: 2
[RT] Error: Test 112 regional_spp_sppt_shum_skeb_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 113 rap_control_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 114 hrrr_control_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 115 rap_2threads_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 116 hrrr_control_2threads_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 117 hrrr_control_decomp_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 120 conus13km_control_intel FAIL Tries: 2
[RT] Error: Test 123 rap_control_dyn64_phy32_intel FAIL Tries: 2
[RT] Error: Test 124 rap_control_debug_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 125 hrrr_control_debug_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 126 conus13km_debug_intel FAIL Tries: 2
[RT] Error: Test 127 conus13km_debug_qr_intel FAIL Tries: 2
[RT] Error: Test 128 conus13km_debug_2threads_intel FAIL Tries: 2
[RT] Error: Test 129 conus13km_radar_tten_debug_intel FAIL Tries: 2
[RT] Error: Test 130 rap_control_dyn64_phy32_debug_intel FAIL Tries: 2
[RT] Error: Test 131 hafs_regional_atm_intel FAIL Tries: 2
[RT] Error: Test 132 hafs_regional_atm_thompson_gfdlsf_intel FAIL Tries: 2
[RT] Error: Test 133 hafs_regional_atm_ocn_intel FAIL Tries: 2
[RT] Error: Test 134 hafs_regional_atm_wav_intel FAIL Tries: 2
[RT] Error: Test 135 hafs_regional_atm_ocn_wav_intel FAIL Tries: 2
[RT] Error: Test 136 hafs_regional_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 137 hafs_regional_telescopic_2nests_atm_intel FAIL Tries: 2
[RT] Error: Test 138 hafs_global_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 139 hafs_global_multiple_4nests_atm_intel FAIL Tries: 2
[RT] Error: Test 140 hafs_regional_specified_moving_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 141 hafs_regional_storm_following_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 142 hafs_regional_storm_following_1nest_atm_ocn_intel FAIL Tries: 2
[RT] Error: Test 143 hafs_global_storm_following_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 144 gnv1_nested_intel FAIL Tries: 2
[RT] Error: Test 145 hafs_regional_storm_following_1nest_atm_ocn_debug_intel FAIL Tries: 2
[RT] Error: Test 146 hafs_regional_storm_following_1nest_atm_ocn_wav_intel FAIL Tries: 2
[RT] Error: Test 147 hafs_regional_docn_intel FAIL Tries: 2
[RT] Error: Test 148 hafs_regional_docn_oisst_intel FAIL Tries: 2
[RT] Error: Test 149 hafs_regional_datm_cdeps_intel FAIL Tries: 2
[RT] Error: Test 150 datm_cdeps_control_cfsr_intel FAIL Tries: 2
[RT] Error: Test 152 datm_cdeps_control_gefs_intel FAIL Tries: 2
[RT] Error: Test 153 datm_cdeps_iau_gefs_intel FAIL Tries: 2
[RT] Error: Test 154 datm_cdeps_stochy_gefs_intel FAIL Tries: 2
[RT] Error: Test 155 datm_cdeps_ciceC_cfsr_intel FAIL Tries: 2
[RT] Error: Test 156 datm_cdeps_bulk_cfsr_intel FAIL Tries: 2
[RT] Error: Test 157 datm_cdeps_bulk_gefs_intel FAIL Tries: 2
[RT] Error: Test 158 datm_cdeps_mx025_cfsr_intel FAIL Tries: 2
[RT] Error: Test 159 datm_cdeps_mx025_gefs_intel FAIL Tries: 2
[RT] Error: Test 160 datm_cdeps_multiple_files_cfsr_intel FAIL Tries: 2
[RT] Error: Test 161 datm_cdeps_3072x1536_cfsr_intel FAIL Tries: 2
[RT] Error: Test 162 datm_cdeps_gfs_intel FAIL Tries: 2
[RT] Error: Test 163 datm_cdeps_debug_cfsr_intel FAIL Tries: 2
[RT] Error: Test 164 datm_cdeps_control_cfsr_faster_intel FAIL Tries: 2
[RT] Error: Test 165 datm_cdeps_lnd_gswp3_intel FAIL Tries: 2
[RT] Error: Test 167 control_p8_atmlnd_sbs_intel FAIL Tries: 2
[RT] Error: Test 168 atmwav_control_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 169 control_atmwav_intel FAIL Tries: 2
[RT] Error: Test 170 atmaero_control_p8_intel FAIL Tries: 2
[RT] Error: Test 171 atmaero_control_p8_rad_intel FAIL Tries: 2
[RT] Error: Test 172 atmaero_control_p8_rad_micro_intel FAIL Tries: 2
[RT] Error: Test 173 regional_atmaq_intel FAIL Tries: 2
[RT] Error: Test 174 regional_atmaq_debug_intel FAIL Tries: 2
[RT] Error: Test 175 regional_atmaq_faster_intel FAIL Tries: 2
[RT] Error: Test 176 control_c48_gnu FAIL Tries: 2
[RT] Error: Test 177 control_stochy_gnu FAIL Tries: 2
[RT] Error: Test 178 control_ras_gnu FAIL Tries: 2
[RT] Error: Test 179 control_p8_gnu FAIL Tries: 2
[RT] Error: Test 180 control_p8_ugwpv1_gnu FAIL Tries: 2
[RT] Error: Test 181 control_flake_gnu FAIL Tries: 2
[RT] Error: Test 182 rap_control_gnu FAIL Tries: 2
[RT] Error: Test 183 rap_decomp_gnu FAIL Tries: 2
[RT] Error: Test 184 rap_2threads_gnu FAIL Tries: 2
[RT] Error: Test 186 rap_sfcdiff_gnu FAIL Tries: 2
[RT] Error: Test 187 rap_sfcdiff_decomp_gnu FAIL Tries: 2
[RT] Error: Test 189 hrrr_control_gnu FAIL Tries: 2
[RT] Error: Test 190 hrrr_control_noqr_gnu FAIL Tries: 2
[RT] Error: Test 191 hrrr_control_2threads_gnu FAIL Tries: 2
[RT] Error: Test 192 hrrr_control_decomp_gnu FAIL Tries: 2
[RT] Error: Test 195 rrfs_v1beta_gnu FAIL Tries: 2
[RT] Error: Test 196 control_diag_debug_gnu FAIL Tries: 2
[RT] Error: Test 197 regional_debug_gnu FAIL Tries: 2
[RT] Error: Test 198 rap_control_debug_gnu FAIL Tries: 2
[RT] Error: Test 199 hrrr_control_debug_gnu FAIL Tries: 2
[RT] Error: Test 200 hrrr_gf_debug_gnu FAIL Tries: 2
[RT] Error: Test 201 hrrr_c3_debug_gnu FAIL Tries: 2
[RT] Error: Test 202 rap_diag_debug_gnu FAIL Tries: 2
[RT] Error: Test 203 rap_noah_sfcdiff_cires_ugwp_debug_gnu FAIL Tries: 2
[RT] Error: Test 204 rap_progcld_thompson_debug_gnu FAIL Tries: 2
[RT] Error: Test 205 rrfs_v1beta_debug_gnu FAIL Tries: 2
[RT] Error: Test 206 control_ras_debug_gnu FAIL Tries: 2
[RT] Error: Test 207 control_stochy_debug_gnu FAIL Tries: 2
[RT] Error: Test 208 control_debug_p8_gnu FAIL Tries: 2
[RT] Error: Test 209 rap_flake_debug_gnu FAIL Tries: 2
[RT] Error: Test 210 rap_clm_lake_debug_gnu FAIL Tries: 2
[RT] Error: Test 211 gnv1_c96_no_nest_debug_gnu FAIL Tries: 2
[RT] Error: Test 212 control_wam_debug_gnu FAIL Tries: 2
[RT] Error: Test 213 rap_control_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 214 hrrr_control_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 215 rap_2threads_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 216 hrrr_control_2threads_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 217 hrrr_control_decomp_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 220 conus13km_control_gnu FAIL Tries: 2
[RT] Error: Test 223 rap_control_dyn64_phy32_gnu FAIL Tries: 2
[RT] Error: Test 224 rap_control_debug_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 225 hrrr_control_debug_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 226 conus13km_debug_gnu FAIL Tries: 2
[RT] Error: Test 227 conus13km_debug_qr_gnu FAIL Tries: 2
[RT] Error: Test 228 conus13km_debug_2threads_gnu FAIL Tries: 2
[RT] Error: Test 229 conus13km_radar_tten_debug_gnu FAIL Tries: 2
[RT] Error: Test 230 rap_control_dyn64_phy32_debug_gnu FAIL Tries: 2
[RT] Error: Test 231 cpld_control_p8_gnu FAIL Tries: 2
[RT] Error: Test 232 cpld_control_nowave_noaero_p8_gnu FAIL Tries: 2
[RT] Error: Test 233 cpld_debug_p8_gnu FAIL Tries: 2
[RT] Error: Test 234 cpld_control_pdlib_p8_gnu FAIL Tries: 2
[RT] Error: Test 235 cpld_debug_pdlib_p8_gnu FAIL Tries: 2
[RT] Error: Test 236 datm_cdeps_control_cfsr_gnu FAIL Tries: 2
[RT] Log file shows failures.
[RT] Please obtain logs from /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/beta/run//1714125607/20240221023821/ufs-weather-model

@grantfirl
Copy link
Collaborator Author

@mkavulich It looks like a new baseline was only created for control_p8_intel, so the comparison failed due to missing baseline for most of the tests. The control_p8_intel test did correctly compare and passed, so I think you just need to double check that the script is running the entire rt.conf and we should be good to go and run hera-BL again once that's done.

@mkavulich
Copy link
Collaborator

You're right, I had missed one debug setting that hard-coded that single test. I've fixed it now, and will re-run the BL creation.

@mkavulich mkavulich added hera-BL Create new baselines on Hera machine. TESTING ONLY, NOT FOR GENERAL PRS YET and removed hera-BL Create new baselines on Hera machine. TESTING ONLY, NOT FOR GENERAL PRS YET labels Feb 21, 2024
@mkavulich
Copy link
Collaborator

@grantfirl I'm not sure what happened last night... the test seems to have silently failed before creating the new baselines, with some seemingly random failures. Is there a chance we ran out of disk space again? When I run account_params on Hera the gmtb project shows up as DiskInUse=0 GB which is clearly wrong... I'm not sure how else to check our disk usage.

@grantfirl
Copy link
Collaborator Author

@grantfirl I'm not sure what happened last night... the test seems to have silently failed before creating the new baselines, with some seemingly random failures. Is there a chance we ran out of disk space again? When I run account_params on Hera the gmtb project shows up as DiskInUse=0 GB which is clearly wrong... I'm not sure how else to check our disk usage.

It can be found in /scratch2/BMC/public/quotas/gmtb. It is showing something weird there too (0 GB usage), but the individual users have enough to push us over the quota, but not quite at the hard limit. You, Dustin, and I are the largest offenders.

@mkavulich
Copy link
Collaborator

Thanks for checking, I'll clear out some old runs which should do the trick for now, then re-run again.

@mkavulich
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Job: BL
[BL] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/beta/run//1714125607/20240221162516/ufs-weather-model
Please make changes and add the following label back: hera-BL

@mkavulich
Copy link
Collaborator

Ha, it looks like the old job was still running, just hung due to not being able to write to disk. Starting the new test now.

@mkavulich mkavulich added hera-BL Create new baselines on Hera machine. TESTING ONLY, NOT FOR GENERAL PRS YET and removed hera-BL Create new baselines on Hera machine. TESTING ONLY, NOT FOR GENERAL PRS YET labels Feb 22, 2024
@grantfirl
Copy link
Collaborator Author

@mkavulich Hera seems like it has almost ground to a halt. Very slow today. I'm in the middle of another set of RTs for the ufs/dev branch, and the progress is painful.

@mkavulich
Copy link
Collaborator

Machine: hera
Job: BL
[BL] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/beta/run//1714125607/20240222160020/ufs-weather-model
Baseline creation successful on hera
[RT] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/beta/run//1714125607/20240223033009/ufs-weather-model
Regression test successful on hera!

@grantfirl
Copy link
Collaborator Author

@mkavulich This is ready to merge once we have approvals. There is no SCM PR associated with this one since there were no required changes. I could do a submodule update, but it's probably not necessary.

@grantfirl
Copy link
Collaborator Author

@mkavulich I moved the new baselines into beta/baselines already.

Copy link
Collaborator

@mkavulich mkavulich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good once submodules are updated

@grantfirl grantfirl merged commit 640f55a into NCAR:main Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants