Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If GSHHG cannot we be found we download file from remote server #3645

Merged
merged 8 commits into from
Aug 23, 2020

Conversation

PaulWessel
Copy link
Member

Instead of giving up if GSHHG files are not found, we simply download them from the server and read from the local ./gmt/geography/gshhg directory.

Tested and works. Like DCW, probably need to change the cmake setup so that not having GSHHG_ROOT set is not a fatal problem, @seisman.

Instead of giving up if GSHHG files are not found, we simply download them from the server and read from the local ./gmt/geography/gshhg directory.
@joa-quim
Copy link
Member

I would still prefer that it first tried to put the files under share/coast (its natural location) and only resort to .gmt if permission don't allow it. Storing in .gmt means that a gmt clear will wipe them out.

@seisman
Copy link
Member

seisman commented Jul 12, 2020

I've improved the GSHHG message in bd20180.

If GSHHG is found, you'll see:

*  Found GSHHG database       : /path/to/coast (2.3.7)

otherwise:

*  Found GSHHG database       : GSHHG_PATH-NOTFOUND (GMT will download on demand)

@seisman
Copy link
Member

seisman commented Jul 12, 2020

If GSHHG is not found, CMake still reports a fatal error when tests or examples are enabled. I think we can remove these message since GMT will download the coastline data when requested.

gmt/src/CMakeLists.txt

Lines 306 to 311 in 308386e

# Examples and tests depend on Shorelines
find_package (GSHHG)
if (DO_EXAMPLES OR DO_TESTS AND NOT GSHHG_FOUND)
message (FATAL_ERROR "Cannot proceed without GSHHG Shorelines. "
"Need to either set GSHHG_ROOT or disable tests.")
endif (DO_EXAMPLES OR DO_TESTS AND NOT GSHHG_FOUND)

@PaulWessel
Copy link
Member Author

Thanks @seisman . Hi @joa-quim, the problem with your suggestion is that it will go against the policy of making things simple for the user. Like our remote data service, we want to simply update grids and coastlines on the server and the users automatically get them when needed. Thus, trying to place these files in an install directory that presumably may go away in 6.2 is not good. Yes you can run gmt clear and delete it if you want to, but they get downloaded again when you ask for them. The 'natural location' will become ./gmt/geography in the users directory and the whole business of tarballs etc will go away.

@PaulWessel
Copy link
Member Author

If GSHHG is not found, CMake still reports a fatal error when tests or examples are enabled. I think we can remove these message since GMT will download the coastline data when requested.

yes, I saw that too and thought the same. It is no longer relevant. However, I think for our CI testing we would do the same we do for the remote data and pre-download to avoid any race condition of multiple downloads of the same file, etc.

@joa-quim
Copy link
Member

Well, that's a big novelty. You mean that you intend to get rid of the tarballs altogether and remove the GSHHG-DCW from cmake?

But the files will still be searched in share/coast, right? I have my modified version of the full resolution with high detail Portuguese coastlines and certainly want to continue using it

@PaulWessel
Copy link
Member Author

I am in no hurry to remove the search path system we have since we know ubuntu will have these until 2028. So this will let you continue as before. By 2028 we either have an openstreetmap solution or I will still be tinkering with this in retirement.

@seisman
Copy link
Member

seisman commented Jul 12, 2020

I have my modified version of the full resolution with high detail Portuguese coastlines and certainly want to continue using it.

Wow, does it take a lot of efforts? The official GSHHG data isn't good for Chinese users due to the China-India border dispute. I'm also planning to modify GSHHG for Chinese users.

@PaulWessel
Copy link
Member Author

It is a bit rabbit hole to creep into. Better to wait for OpenStreetMap and our postdoc. News: We have applicants (4 or more I think). We will discuss this at next GMT zoom. We may want to have that zoom this Wednesday or Thursday due to the UNAVCO course as well.

@PaulWessel
Copy link
Member Author

Family zoom will take up next 2 hours.

@joa-quim
Copy link
Member

I wrote tools in Mirone to help with the process but the biggest problem is that to build the nc files one needs tools that link with GMT4 and I suspect I wont be able to build GMT4 again, at least without substantial pain.

@PaulWessel
Copy link
Member Author

Are we happy with this solution, in which we will still look in the old places first (which lets @joa-quim have a custom file somewhere) but going forward we should

  1. Stop requiring gshhg tar install as a prerequisite [there are non-GMT users of GSHHG so I wont remove the files from SOEST]
  2. Deliver coastline in whatever format we end up with down the road via the data server only

If so then I can remove the WIP and we can let it rip.

@PaulWessel PaulWessel changed the title WIP if GSHHG cannot we be found we download from server If GSHHG cannot we be found we download file from remote server Aug 23, 2020
@PaulWessel
Copy link
Member Author

Let's decide to merge this in. It does not affect any installed files and is only used if no files are found. It certainly makes life easier for people who do not install from a repo or installer.

@PaulWessel PaulWessel merged commit 868b19f into master Aug 23, 2020
@PaulWessel PaulWessel deleted the remote-gshhg branch August 23, 2020 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants