Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GMT_KNOWN_FAILURE flag to "pass" the known failures #1001

Merged
merged 5 commits into from
Jun 27, 2019

Conversation

seisman
Copy link
Member

@seisman seisman commented Jun 25, 2019

Fixes #852.

@seisman
Copy link
Member Author

seisman commented Jun 25, 2019

mgd77/test_cm4.sh and psxy/antwrap.sh still fail.

@seisman seisman changed the title Add GMT_KNOWN_FAILURE flag to "pass" the known failures WIP: Add GMT_KNOWN_FAILURE flag to "pass" the known failures Jun 25, 2019
@joa-quim
Copy link
Member

I still don't understand well this idea of hiding the failing tests. What then are they good for?

@seisman
Copy link
Member Author

seisman commented Jun 26, 2019

I still don't understand well this idea of hiding the failing tests. What then are they good for?

We run all the tests nightly to make sure recent changes don't break the codes. Usually, we have two ways to know if all the tests pass:

  1. Check the badges in README.md TravisCI Azure

  2. Check the build logs.

It's obvious that the first one is much simpler. However, it doesn't work well for our case. Since we always have some "known to fail" tests, we have to read build logs to check if there're any new "failing" tests. After hiding these "known to fail" tests, we can know if all the tests pass at a glance.

@seisman seisman changed the title WIP: Add GMT_KNOWN_FAILURE flag to "pass" the known failures Add GMT_KNOWN_FAILURE flag to "pass" the known failures Jun 26, 2019
@seisman seisman requested a review from PaulWessel June 26, 2019 01:51
@seisman
Copy link
Member Author

seisman commented Jun 26, 2019

There is still one failing test (mapproject/proj4.sh) on Linux. Probably it's caused by the old versions of gdal (1.11.3) or proj (4.9.2).

@joa-quim
Copy link
Member

I see. But the price of this is to almost throw away the failing tests. We should move them into a separate category (directory?) to not hide them under the carpet.

@joa-quim
Copy link
Member

The mapproject/proj4.sh failures are boring. They mostly happen due to changes in PROJ4.

@PaulWessel
Copy link
Member

We need a way to run the filling test so they fail without having to find and edit individual scripts. but we can discuss this at Scripps.

@seisman
Copy link
Member Author

seisman commented Jun 27, 2019

Can we merge this? Or leave it to the summit?

@PaulWessel
Copy link
Member

Either way, we will discuss this later this summer so perhaps nice to have things pass for now? I won't forget about the failures!

@seisman seisman merged commit c598ae4 into 6.0 Jun 27, 2019
@seisman
Copy link
Member Author

seisman commented Jun 27, 2019

OK. merged.

@seisman seisman deleted the pass-failure-tests branch June 27, 2019 01:55
obaney pushed a commit to obaney/gmt that referenced this pull request Aug 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants