Testsuite logging

[rs028]

Improving the logs of the various tests (see #423).

[codecov]

Codecov Report

Merging #482 (0257d2d) into master (6650b8a) will decrease coverage by 0.05%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #482      +/-   ##
==========================================
- Coverage   66.81%   66.76%   -0.05%     
==========================================
  Files          17       17              
  Lines        2046     2046              
==========================================
- Hits         1367     1366       -1     
- Misses        679      680       +1     

Flag	Coverage Δ
build	52.81% <ø> (ø)
tests	66.92% <ø> (-0.25%)	⬇️
unittests	54.86% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6650b8a...0257d2d. Read the comment docs.

[rs028]

To avoid overcrowding the terminal, the idea here is to output to screen a message saying that the test has failed or passed at the end of all tests. Initally limited to indent and style tests only for simplicity.
The initial submit works for individual tests (eg make indenttest) but not when they are put together (make alltests) because the temporary file gets deleted at the start of each test.

Desired output:

Running indent script on:
src/argparse.f90
src/atchem2.f90
src/atmosphereFunctions.f90
src/configFunctions.f90
src/constraintFunctions.f90
src/dataStructures.f90
src/inputFunctions.f90
src/interpolationFunctions.f90
src/outputFunctions.f90
src/parameterModules.f90
src/solarFunctions.f90
src/solverFunctions.f90
Indent script finished

Running style script on:
src/argparse.f90
src/atchem2.f90
src/atmosphereFunctions.f90
src/configFunctions.f90
src/constraintFunctions.f90
src/dataStructures.f90
src/inputFunctions.f90
src/interpolationFunctions.f90
src/outputFunctions.f90
src/parameterModules.f90
src/solarFunctions.f90
src/solverFunctions.f90
Style script finished

Indent test PASSED
Style test PASSED

[rs028]

A different approach is to log all the output of the test to a file and just output to screen a message.
This is much simpler and cleaner. @spco, any thoughts?

[rs028]

This works for indent, style, model and old tests. The only annoying thing is that for the test spec_yes_env_no_with_jfac_fail1, which is meant to fail, the executable prints a STOP 2 message (I think) and it looks a bit untidy ;)

[rs028]

I can't find a way to log the unit tests. If I understand it, it is all hard wired in the FRUIT code and I don't really want to modify that.
But the printout to screen is actually reasonably clear, so I think this is okay.

[rs028]

@spco I think I have done the merge correctly, but for some reasons, it is failing the macos tests again!
If you have some time to review this PR as well, it would be great :)

[rs028]

More detail about the tests.

Ubuntu:

• unittests, oldtests and modeltests pass on all versions

MacOS:

• run fails with gfortran 9
• unittests and oldtests pass
• 4 out of 7 modeltests fail

All systems report a warning about cmake becoming obsolete during the installation of sundials, but I don't think that is causing a problem here.

Another warning message is ld: warning: dylib (cvode/lib/libsundials_cvode.dylib) was built for newer macOS version (11.7) than being linked (11.0), I don't know if that is why the MacOS tests fail (?)

In general the whole setup is a bit clunky now. I had to run each test manually and individually (sometimes more than once) to get it to complete the run. Perhaps it is all just a problem on the github actions side. (this is okay now that the yml file has been modified to run all tests, regardless of their success).

[spco]

Can this be correct? A STOP being printed, then further output afterwards? Seems odd.

[rs028]

To be honest I do not know! The test is meant to fail, and it seems that STOP is being printed to screen at least on my Linux machine. It seems it is not necessary for the jobs that run on actions. A bit puzzled.

[spco]

The difference between the two is probably something to do with how the two streams (output and error) are being handled, either piped to the same location or different ones. The main question is does it pass/fail the tests correctly I guess :)

[rs028]

This particular test passes. It is four others that fail but only on mac.

[spco]

Hi @rs028 . All looks fine to me really, so long as you've tested it yourself then I see no problems (see the one comment above for a question). I don't want to hold things up if this is all working!