Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let gmt download DCW from server if not found locallly #3644

Merged
merged 10 commits into from
Aug 23, 2020

Conversation

PaulWessel
Copy link
Member

@PaulWessel PaulWessel commented Jul 12, 2020

For now we look in all the usual pre-installed places, but if that fails we also look in ./gmt/geography/dcw and if no such dir or no files we download from server and use this path.
I have tested this in debug and command line and it works fine. However, I think our cmake setup expects DCW_ROOT to be set correctly. We would need to let this be an optional setting and if not then we do this all just-in-time. I may need help from @seisman on the cmake side. I dont think we (yet) want to remove the pre-install mechanism but just add the server download as a fallback if we cannot find the files where we expect them. I will do something similar for gshhg.

For now we look in all the usual pre-installed places, but if that fails we also look in ./gmt/geography/dcw and if no such dir or no files we download from server and use this path.
@seisman
Copy link
Member

seisman commented Jul 12, 2020

I think our cmake setup expects DCW_ROOT to be set correctly.

Perhaps not. I can still build and install GMT without setting GSHHG_ROOT and DCW_ROOT.

cmake says:

*  Found GSHHG database       : GSHHG_PATH-NOTFOUND ()
*  Found DCW-GMT database     : DCW_PATH-NOTFOUND ()

Running gmt pscoast gives me errors like:

coast [WARNING]: GSHHG version 2.2.0 or newer is needed to use coastlines with GMT.
	Get and install GSHHG from ftp:https://ftp.soest.hawaii.edu/gshhg/.
coast [ERROR]: Could not find file [GSHHG intermediate resolution shorelines]
coast [ERROR]: No GSHHG databases available - must abort

@PaulWessel
Copy link
Member Author

OK, yes I guess that is true. I got some build errors because I renamed my path to gshhg during testing after cmake had done its config. Would we want the above message to say something different, e.g.

*  Found GSHHG database       : GMT will download on demand
*  Found DCW-GMT database     : GMT will ownload on demand

so a user wont freak out?

@seisman
Copy link
Member

seisman commented Jul 12, 2020

How about this?

*  Found GSHHG database       : GSHHG_PATH-NOTFOUND (GMT will download on demand)
*  Found DCW-GMT database     : DCW_PATH-NOTFOUND (GMT will download on demand)

@PaulWessel
Copy link
Member Author

Sure, if that is simpler that is good enough.

@PaulWessel
Copy link
Member Author

Same here, are you OK with me removing WIP and that means that for GMT 6.2 there will be no discussion of how to install DCW as part of the installation. We can discuss later if the installers should continue to contain DCW and GSHHG or if they should be lighter and data appear during use.

@PaulWessel
Copy link
Member Author

The way I am seeing this PR is that it will greatly simplify life for GMT users as they do not have to add one extra prerequisite that they may need to do manually via our gshhg tarball. However, we would not simplify it if we continue to distribute the dcw tar balls and talk about it on the installation page. The biggest issue is the lag it takes before the ubuntus of the world will reach and distribute 6.2. Only then will people no longer look for the dcw tarball.

@PaulWessel PaulWessel changed the title WIP Let gmt download DCW from server if not found locallly Let gmt download DCW from server if not found locallly Aug 23, 2020
@PaulWessel
Copy link
Member Author

Same as for #3645, I think it is time to merge this in for the future, and for the same reasons.

@seisman
Copy link
Member

seisman commented Aug 23, 2020

Looks good to me. Just one small issue:

When I ran gmt coast -RCN -W1p -pdf map -Vd, I saw the following messages, so I can know the GSHHG file path GMT is reading:

coast [DEBUG]: 1. GSHHG: GSHHGDIR set, trying /Users/seisman/Gits/gmt/gmt/share/coast/binned_GSHHS_i.nc
coast [DEBUG]: 1. GSHHG: OK, could access /Users/seisman/Gits/gmt/gmt/share/coast/binned_GSHHS_i.nc
coast [DEBUG]: NetCDF Library Version: 4.7.3 of Jan 23 2020 20:10:56 $
coast [DEBUG]: GSHHG with two Antarcticas, read in extra ANT flgs.
coast [INFORMATION]: Selected ice front line as Antarctica coastline
coast [INFORMATION]: GSHHG version 2.3.7
coast [INFORMATION]: Derived from World Vector Shoreline, CIA WDB-II, and Atlas of the Cryosphere
coast [INFORMATION]: Processed by Paul Wessel and Walter H. F. Smith, 1994-2017

but for DCW data, I only saw these messages, and it's unclear to me which file GMT is reading (I have multiple DCW installed in my system):

coast [DEBUG]: Got country code for region (CN)
coast [DEBUG]: GMT: 0. Will try to find subdir=dcw stem = dcw-countries suffix=.txt
coast [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
coast [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /Users/seisman/.gmt
coast [DEBUG]: GMT: 3. gmt_getsharepath trying USERDIR subdir /Users/seisman/.gmt/dcw
coast [DEBUG]: GMT: 4. gmt_getsharepath trying SHAREDIR subdir /Users/seisman/Gits/gmt/gmt/share/dcw
coast [DEBUG]: GMT: 5. gmt_getsharepath trying SHAREDIR /Users/seisman/Gits/gmt/gmt/share
coast [DEBUG]: GMT: 6. gmt_getsharepath failed
coast [DEBUG]: GMT: 0. Will try to find subdir=dcw stem = dcw-states suffix=.txt
coast [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
coast [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /Users/seisman/.gmt
coast [DEBUG]: GMT: 3. gmt_getsharepath trying USERDIR subdir /Users/seisman/.gmt/dcw
coast [DEBUG]: GMT: 4. gmt_getsharepath trying SHAREDIR subdir /Users/seisman/Gits/gmt/gmt/share/dcw
coast [DEBUG]: GMT: 5. gmt_getsharepath trying SHAREDIR /Users/seisman/Gits/gmt/gmt/share
coast [DEBUG]: GMT: 6. gmt_getsharepath failed
coast [DEBUG]: DCW: Found 250 countries, 8 countries with states, and 281 states
coast [DEBUG]: Requested 1 DCW items: CN
coast [DEBUG]: GMT: 0. Will try to find subdir=dcw stem = dcw-gmt suffix=.nc
coast [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
coast [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /Users/seisman/.gmt
coast [DEBUG]: GMT: 3. gmt_getsharepath trying USERDIR subdir /Users/seisman/.gmt/dcw
coast [DEBUG]: GMT: 4. gmt_getsharepath trying SHAREDIR subdir /Users/seisman/Gits/gmt/gmt/share/dcw
coast [DEBUG]: GMT: 5. gmt_getsharepath trying SHAREDIR /Users/seisman/Gits/gmt/gmt/share
coast [DEBUG]: GMT: 6. gmt_getsharepath failed
coast [INFORMATION]: Using country and state data from dcw-gmt
coast [INFORMATION]: Title  : DCW-GMT - The Digital Chart of the World for the Generic Mapping Tools
coast [INFORMATION]: Source : Processed by the GMT Team, 2018-JUL-01
coast [INFORMATION]: Version: 1.1.4
coast [INFORMATION]: China

@PaulWessel
Copy link
Member Author

I think that is useful to show, let me try.

@PaulWessel
Copy link
Member Author

Please give this a spin since you have several DCW files.

@seisman
Copy link
Member

seisman commented Aug 23, 2020

Now it works well for me.

One small issue. Why use \r here? Should it be \n?

gmt/src/pscoast.c

Line 1017 in 868b19f

GMT_Report (API, GMT_MSG_DEBUG, "Working on bin # %5d\r", bin);

@PaulWessel
Copy link
Member Author

Well, we have lots of those throughout GMT, try

grep '\\r' *.c

The idea was that these were more progress indicators so we did not want them to scroll off.

@seisman
Copy link
Member

seisman commented Aug 23, 2020

I don't see any these messages when running commands in terminal (perhaps because it's running too fast), but when I redirect all stderr to a file, I can see something like.

coast [DEBUG]: Working on bin #   518^Mcoast [DEBUG]: Working on bin #   519^M

Copy link
Member

@seisman seisman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK to me.

@PaulWessel PaulWessel merged commit f4650b5 into master Aug 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants