Description

In PR #2290, many tests were expected to fail comparison because of the code changes. Multiple tests however failed because they did not actually run. Since developers were expecting test failures, they were not alerted to the fact that the tests failed because of seg-faults. This leads to delays during the code commit process, since fixing this type of test failure requires debugging etc. This type of failure needs to be clearly differentiated from cases where the tests failed because the comparisons failed.

Secondly, looking at the failure in this PR for the cpld_control_gfsv17_intel case, the log reports

* TEST cpld_control_gfsv17_intel: FAILED: UNABLE TO COMPLETE COMPARISON
-- LOG: /scratch1/BMC/gmtb/Grant.Firl/ufs-weather-model-test/tests/logs/log_hera/run_cpld_control_gfsv17_intel.log

In fact, in this case, all the comparisons were made (and failed). The comparison was completed so the log message is misleading.

Solution

A test failure should clearly indicate why it failed.

Alternatives

Related to