Failure in regression test setup scripts not caught

[climbfuji]

If the rt_fv3.sh script exits with a failure, for example if an input namelist template for creating the model namelist is not found, this error is not caught even though the job is never submitted. For regression tests that do not compare files (such as the tests in debug mode), this leads to rt.sh reporting that the test succeeded.

This is because of the following lines in tests/run_test.sh:

...
# Submit the actual test run script
echo "Test ${TEST_NR} ${TEST_NAME} ${TEST_DESCR}"
trap 'echo "run_test.sh: Test ${TEST_NAME} killed"; kill $(jobs -p); wait; trap 0; exit' 1 2 3 4 5 6 7 8 10 12 13 15
./${RUN_SCRIPT} > ${RUNDIR_ROOT}/${TEST_NAME}${RT_SUFFIX}.log 2>&1 &
wait
...

[climbfuji]

One way to fix this is

# Submit the actual test run script
echo "Test ${TEST_NR} ${TEST_NAME} ${TEST_DESCR}"
trap 'echo "run_test.sh: Test ${TEST_NAME} killed"; kill $(jobs -p); wait; trap 0; exit' 1 2 3 4 5 6 7 8 10 12 13 15
./${RUN_SCRIPT} > ${RUNDIR_ROOT}/${TEST_NAME}${RT_SUFFIX}.log 2>&1
...

which, in the event of an error in tests/rt_fv3.sh, leads to

++ echo 'rt.sh error on line 639'
rt.sh error on line 639
++ rt_trap
++ [[ false == true ]]
++ [[ false == true ]]
++ cleanup
++ rm -rf /scratch1/BMC/gmtb/Dom.Heinzeller/NEMSfv3gfs/ufs-weather-model_dtc-develop_20191105/regtestfailure/tests/lock
++ trap 0
++ exit

but I haven't tested this for the full set of regression tests (rt.conf).

[climbfuji]

The issue described above is addressed in #18 for the "develop" branch, and in #25 for the "ufs_public_release" branch.