Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

List of failing tests for GMT 6.4 #1885

Closed
69 tasks done
seisman opened this issue Apr 18, 2022 · 5 comments · Fixed by #1883
Closed
69 tasks done

List of failing tests for GMT 6.4 #1885

seisman opened this issue Apr 18, 2022 · 5 comments · Fixed by #1883
Labels
maintenance Boring but important stuff for the core devs
Milestone

Comments

@seisman
Copy link
Member

seisman commented Apr 18, 2022

Following is a long list of 70 failing tests for the upcoming GMT dev version.

@weiji14 weiji14 added the maintenance Boring but important stuff for the core devs label Apr 18, 2022
@seisman seisman added this to the 0.7.0 milestone Apr 19, 2022
@michaelgrund
Copy link
Member

michaelgrund commented Apr 24, 2022

Is there still something to do here or all steps are handled in #1883 now @seisman?

@seisman
Copy link
Member Author

seisman commented Apr 24, 2022

I'm slowly checking the failing tests and pushing the new baseline images to #1883, so all will be done in #1883 unless some failing tests are caused by GMT or PyGMT bugs.

@seisman
Copy link
Member Author

seisman commented Apr 25, 2022

FYI, I've updated 46 baseline images in the gmt-6.4 branch (PR #1883), which should fix ~56 failing tests (some tests share the same baseline image). These tests fail due to tiny changes in gridlines or texts, or upstream fixes that improve the layout (e.g., GenericMappingTools/gmt#6163).

The remaining tests show much bigger changes between the old and new baseline images, so we must carefully check if the changes are correct or not. For these tests, I'll open separate PRs instead of directly pushing the baseline images to the gmt-6.4 branch, so that we can have more discussions.

@seisman
Copy link
Member Author

seisman commented Apr 26, 2022

FYI, I've updated all baseline images.

Some baseline images are pushed directly to the gmt-6.4 branch (#1883) and the related failures are marked as solved in this issue, because the baseline differences are tiny.

For some other tests, I've opened PRs and also added links to the upstream PRs/issues that help explain why the baseline images change.

For all the remaining tests, the new baseline images are generated using GMT bash script, which are always believed to the correct ones. I've opened separate PRs for each grdimage test.

@seisman
Copy link
Member Author

seisman commented May 5, 2022

FYI, all failing baselines images have been updated and merged into the gmt-6.4 branch (PR #1883).

If you look at the scheduled "GMT Dev Test" CI runs (e.g., https://github.com/GenericMappingTools/pygmt/runs/6298988995?check_suite_focus=true), you will see all tests pass except four. The four tests fail because I've updated the test functions in the gmt-6.4 branch to generate more reasonable baseline images, but the scheduled CI jobs run "tests in the main branch" + "baseline images in the gmt-6.4 branch" + "GMT dev source code".

Instead, the CI job triggered by /test-gmt-dev slash command (#1883 (comment)) tests PyGMT in the gmt-6.4 branch, and all tests pass (https://github.com/GenericMappingTools/pygmt/actions/runs/2274271836).

@seisman seisman modified the milestones: 0.7.0, 0.8.0 Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Boring but important stuff for the core devs
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants