Let libcurl downloads be managed via lockfiles #3735
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
When the time is up to refresh the gmt_data_server.txt file, and you run several commands, possibly simultaneously, that all need to access remote files (not necessarily the same file), they all realize that the local copy of gmt_data_server.txt is too old and must be refreshed. So then the mad dash of being the first to do this starts among competing processes. What seems to happen is that they are clobbering each other and sometimes we are left with a zero-size gmt_data_server.txt file. This file now has a fresh date, so it is no longer downloaded, but it has no contents so any remote grids will fail to be aquired. This problems seems particulary bad when I run ctest with 20+ cores and that file is due for a refresh.
This PR tries to use the lockfile mechanism we have used for years for the gmt.history file (to protect them from being clobbered with your run process1 | process2, for instance. Except here, the file is acquired inside libcurl. The solution explored in this PR is to start the download sequence as outlined in #3723. So far so good. I did two experiments:
I hope @joa-quim and @seisman can give this branch a spin and see if they discovered any issues. It already seems to me it works better than the no-lock release we have. I added WIP so that we dont merge until we test a bit more. Comments welcome.