Sort and separate the tests scripts

[leouieda]

GMT has a lot of test scripts that create and compare plots. What we don't have is a measure of how many modules these tests call and how much of the code is covered. The tests also take a long time to run so they're not good for running on TravisCI (we don't want to wait 40min for a build to finish).

Quoting @PaulWessel s email:

See plot of test durations (log-scale in seconds). So most are around a few seconds (1-10) with a few slow ones (70 s); this is on my laptop. I also attach the sorted list (time, test). Before I do anything else we can talk about what might be useful? Make a plot of “coverage” somehow, meaning how many times any gmt module is called within the 812 tests? I think we wanted a very basic set that at least ran all modules once on a minimal data set.

Distribution of test times.

Here is a file with the timing information for each test script: test_sorted.txt

[leouieda]

To run these things on CI, it would be good to be able to specify different levels of tests that we want to run. Tests that are run on every build should be fast, ideally taking no more than 10min to run in total. We can run the full test suite with the slower tests on a nightly cron job. So it would help if we could "make test-fast" and "make test-full", for example.

[leouieda]

cc @joa-quim @seisman

[seisman]

Inspired by https://github.com/obspy/obspy_github_api and https://github.com/obspy/obspy/blob/master/.travis.yml.

Maybe we can separate it into several tests, and use some magic comment strings like +DOCS, +TESTS:FULL, +TESTS:0 to trigger specific tests.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions.

[joa-quim]

Closing on a too old and stalled criteria.