Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache GMT remote data as artifacts on Github #530

Merged
merged 14 commits into from
Jul 16, 2020
Merged

Cache GMT remote data as artifacts on Github #530

merged 14 commits into from
Jul 16, 2020

Conversation

weiji14
Copy link
Member

@weiji14 weiji14 commented Jul 16, 2020

Description of proposed changes

Use GMT on macOS to download the GMT remote files from the GMT data server, and then upload them as artifacts on Github.

Separates the 'cache' task from our Tests CI. Meaning we can update our cache independently of our Continuous Integration tests, allowing the cache to be updated every Sunday for example without needing to manually trigger it with a PR.

References:

Fixes #523, Closes #527

Reminders

  • Run make format and make check to make sure the code follows the style guide.
  • Add tests for new features or tests that would have caught the bug that you're fixing.
  • Add new public functions/methods/classes to doc/api/index.rst.
  • Write detailed docstrings for all functions/methods.
  • If adding new functionality, add an example to docstrings or tutorials.

Use GMT on macOS to download the GMT remote files from the GMT data server, and then upload them as artifacts on Github.
@weiji14 weiji14 added the maintenance Boring but important stuff for the core devs label Jul 16, 2020
@weiji14 weiji14 changed the title WIP: Cache GMT remote data as artifacts on Github Cache GMT remote data as artifacts on Github Jul 16, 2020
@weiji14 weiji14 marked this pull request as ready for review July 16, 2020 03:07
github_token: ${{secrets.GITHUB_TOKEN}}
workflow: cache_data.yaml
name: gmt-cache
path: ~/.gmt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the path correct? Downloading artifacts is fast, but it may not be placed in the correct directories. GMT still tries to download all files during testing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, it looks like it's unzipped to the right location (e.g. ~/.gmt/server/earth/earth_relief/earth_relief_01d_g.grd), and that's what wget was doing before?? Maybe we should use $HOME or something?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can add a step listing all files in the ~/.gmt directory after downloading the artifacts.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, $HOME doesn't work either. Then I'm not sure where the action is actually unzipping the files to!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've some tests to do, and will make changes to this branch.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, go for it 👍

Copy link
Member Author

@weiji14 weiji14 Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that a ~ folder is created 🤦. How about trying ./.gmt, since the action uses ./ as the default, see https://github.com/dawidd6/action-download-artifact/blob/2f29da2bb95f5580767170955fca0cf48714d48b/action.yml#L31. Also, we should use ls -alh to see hidden dot files.

Edit: What am I saying, I should just use .gmt!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw the .gmt directory in the root directory. At least we can download the artifact, and then move it to ~/.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's one way of doing it.

@vercel vercel bot temporarily deployed to Preview July 16, 2020 03:40 Inactive
So we don't need to set a Github token
.github/workflows/cache_data.yaml Outdated Show resolved Hide resolved
@seisman seisman added this to the 0.2.x milestone Jul 16, 2020
@seisman seisman added this to In progress in Release v0.2.x via automation Jul 16, 2020
@weiji14 weiji14 merged commit f0d3762 into master Jul 16, 2020
Release v0.2.x automation moved this from In progress to Done Jul 16, 2020
@weiji14 weiji14 deleted the cache-artifact branch July 16, 2020 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Boring but important stuff for the core devs
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

The GitHub Actions caches don't work for PRs
2 participants