Let 6.1 have more flexibility to discover new remote data sets #3370

PaulWessel · 2020-05-22T21:22:23Z

For some background, see GenericMappingTools/gmtserver-admin#37.

This branch will implement the changes needed for 6.1. First change is to replace the hardwired "gmt_hash_server.txt" name with a constant:

#define GMT_HASH_SERVER_FILE "gmt_hash_server.txt"

Using a arameter for data hash filename makes it easier to test features by temporarily changing the hash name.

I expect the hash name to remain the same upon release though. We want GMT <= 6.0.0 to function exactly as before, except get the updated grids for the data types known to <= 6.0.0. However, 6.1 will have access to more files.

Having this as a variable makes it easier to test features by changing the hash name.

seisman · 2020-05-23T06:11:18Z

This branch fails on Windows:

..\src\gmt_io.c(8604): error C2065: 'ext': undeclared identifier

PaulWessel · 2020-05-23T06:11:41Z

Perhaps @seisman can think of ways to update gmt_getremote.sh once we are not just serving earth_relief files? How do we find out that an entry in gmt_hash_server.txt is a cache file vs data file when we cannot use earth_relief?

PaulWessel · 2020-05-23T06:19:47Z

I added this too
if ((FindFileData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) == 0) continue;

Per the web that should work for Windows to identify directories.

seisman · 2020-05-23T06:24:02Z

Perhaps @seisman can think of ways to update gmt_getremote.sh once we are not just serving earth_relief files? How do we find out that an entry in gmt_hash_server.txt is a cache file vs data file when we cannot use earth_relief?

OK. I'll try and open a PR for it.

seisman · 2020-05-23T23:35:19Z

gmt which -Gu @earth_relief_01d_g downloads the data to .gmt/server/earth_relief_01d_g.grd. Below are the debug messages:

gmt [DEBUG]: Revised options: -Gu @earth_relief_01d_g -Vd
gmtwhich [DEBUG]: gmtapi_init_export: Passed family = Data Table and geometry = Text
gmtwhich [DEBUG]: Object ID 0 : Registered Data Table Stream 7fff996d1e68 as an Output resource with geometry Text [n_objects = 1]
gmtwhich [DEBUG]: gmtapi_init_export: Added stdout to registered destinations
gmtwhich [DEBUG]: GMT_Init_IO: Returned first Output object ID = 0
gmtwhich [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Output
gmtwhich [DEBUG]: gmtapi_next_io_source: Selected object 0
gmtwhich [INFORMATION]: Writing Data Table to Standard Output stream
gmtwhich [DEBUG]: GMT_Begin_IO: Output resource access is now enabled [record-by-record]
gmtwhich [DEBUG]: File @earth_relief_01d_g.grd: Type is Data File
gmtwhich [DEBUG]: Look for file earth_relief_01d_g.grd in /Users/seisman/.gmt
gmtwhich [DEBUG]: Look for file earth_relief_01d_g.grd in /Users/seisman/.gmt/cache
gmtwhich [DEBUG]: Look for file earth_relief_01d_g.grd in /Users/seisman/.gmt/server
gmtwhich [NOTICE]: earth_relief_01d.grd: Download file from the GMT data server [data set size is 128K].
gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].

gmtwhich [INFORMATION]: Downloading file https://oceania.generic-mapping-tools.org/earth_relief_01d.grd ...
gmtwhich [INFORMATION]: Download complete [Got 127.373 kb].
gmtwhich [DEBUG]: Look for file earth_relief_01d_g.grd in /Users/seisman/.gmt
gmtwhich [DEBUG]: Look for file earth_relief_01d_g.grd in /Users/seisman/.gmt/cache
gmtwhich [DEBUG]: Look for file earth_relief_01d_g.grd in /Users/seisman/.gmt/server
gmtwhich [DEBUG]: Found file /Users/seisman/.gmt/server/earth_relief_01d_g.grd
/Users/seisman/.gmt/server/earth_relief_01d_g.grd
gmtwhich [DEBUG]: GMT_End_IO: Output resource access is now disabled
gmtwhich [DEBUG]: gmtlib_unregister_io: Unregistering object no 0 [n_objects = 0]
==> 0 API Objects at end of GMTAPI_Garbage_Collection
gmt [DEBUG]: Entering GMT_Destroy_Session

PaulWessel · 2020-05-24T02:20:56Z

Working on this as part of the update. Is this what you expect from gmtwhich -G:

-Gl: Place the file right in the current directory and nowhere else.
-Gc: Place the file in the cache dir and nowhere else.
-Gu: Place the file in the user dir (typically ~/.gmt) and nowhere else.

Of course, if the file already exists in cache or server/* we just copy to the requested place, but if the files do not exist locally, we do not place them in their natural directories as we do in all other situations. gmt which is special and only allows specific placements.

Is this how you interpret gmtwhich?

…ools/gmt into new-remote-files

PaulWessel · 2020-06-05T03:07:10Z

Is the testing stuck?

seisman · 2020-06-05T03:12:46Z

The CI is OK, but github didn't get the reports.

PaulWessel · 2020-06-05T03:29:04Z

Do we wait?

seisman · 2020-06-05T03:29:59Z

Do we wait?

The CI reports are here https://dev.azure.com/GenericMappingTools/GMT/_build/results?buildId=8258&view=results. No need to wait.

PaulWessel · 2020-06-05T03:32:51Z

I dont understand. I says several of the test failed?

seisman · 2020-06-05T03:41:15Z

There is a segment fault for command

gmt pscoast -R0/10/0/10 -JM6i -Ba -Ggray -ENG+p1p,blue -P -Vd > map.ps

after commit 4e7c9b1. I think it's because GMT is using the old gmt_data_server.txt file.

The error message is:

gmt [DEBUG]: Local file /home/vsts/.gmt/server/gmt_data_server.txt found
gmt [DEBUG]: File /home/vsts/.gmt/server/gmt_data_server.txt less than 24 hours old, refresh is premature.
gmt [DEBUG]: Load contents from /home/vsts/.gmt/server/gmt_data_server.txt
gmt [WARNING]: File /home/vsts/.gmt/server/gmt_data_server.txt should have 12 fields but only 5 read for record 0 - download error???
gmt [INFORMATION]: Unable to read server information file
gmt [DEBUG]: Local file /home/vsts/.gmt/server/gmt_hash_server.txt found
gmt [DEBUG]: File /home/vsts/.gmt/server/gmt_hash_server.txt less than 24 hours old, refresh is premature.
gmt [DEBUG]: GMT_Create_Session initialized GMT structure
gmt [DEBUG]: Loading core GMT shared library: libgmt.so
gmt [DEBUG]: Shared Library # 0 (core). Path = libgmt.so
gmt [DEBUG]: Loading GMT plugins from: /home/vsts/work/1/s/gmt-install-dir/lib/gmt/plugins
gmt [DEBUG]: Shared Library # 1 (supplements). Path = /home/vsts/work/1/s/gmt-install-dir/lib/gmt/plugins/supplements.so
gmt [DEBUG]: Revised options: -R0/10/0/10 -JM6i -Ba -Ggray -ENG+p1p,blue -P -Vd
pscoast [DEBUG]: History: Process -R0/10/0/10
pscoast [DEBUG]: History: Process -JM6i
pscoast [DEBUG]: History: Process -Ba
pscoast [DEBUG]: Map distance calculation will be using great circle approximation with authalic auxiliary latitudes and authalic (R_2) radius = 6371007.1809 m, in meter.
pscoast [DEBUG]: gmt_get_filename: In: 0/10/0/10 Out: 0/10/0/10
ERROR: Caught signal number 11 (Segmentation fault) at
/lib/x86_64-linux-gnu/libc.so.6(+0x185540)[0x7f7b8649b540]
[0x2da5]
Stack backtrace:
/home/vsts/work/1/s/gmt-install-dir/lib/libgmt.so.6(sig_handler+0x2b1)[0x7f7b869605d1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f7b86719890]
/lib/x86_64-linux-gnu/libc.so.6(+0x185540)[0x7f7b8649b540]
/home/vsts/work/1/s/gmt-install-dir/lib/libgmt.so.6(gmt_file_is_remotedata+0x89)[0x7f7b86964579]
/home/vsts/work/1/s/gmt-install-dir/lib/libgmt.so.6(gmt_remote_no_extension+0xf)[0x7f7b8696474f]
/home/vsts/work/1/s/gmt-install-dir/lib/libgmt.so.6(gmt_access+0x123)[0x7f7b869dde93]
/home/vsts/work/1/s/gmt-install-dir/lib/libgmt.so.6(gmt_parse_R_option+0x288)[0x7f7b86a8d5f8]
/home/vsts/work/1/s/gmt-install-dir/lib/libgmt.so.6(gmt_parse_common_options+0x182b)[0x7f7b86a9b2db]
/home/vsts/work/1/s/gmt-install-dir/lib/libgmt.so.6(GMT_Parse_Common+0x522)[0x7f7b86a61182]
/home/vsts/work/1/s/gmt-install-dir/lib/libgmt.so.6(GMT_pscoast+0x1ec)[0x7f7b86c50cbc]
/home/vsts/work/1/s/gmt-install-dir/lib/libgmt.so.6(GMT_Call_Module+0x206)[0x7f7b8697d486]
gmt(main+0x6b0)[0x55bf51d9d6b0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f7b86337b97]
gmt(_start+0x2a)[0x55bf51d9e15a]

PaulWessel · 2020-06-05T03:42:06Z

Yes, since I removed the special case that let us read the old format gmt_data_server.c. Perhaps we should move on a merge, then check again for issues

seisman · 2020-06-05T03:45:32Z

Agreed.

PaulWessel · 2020-06-05T03:49:39Z

I will now change the data pointer on the server.

PaulWessel · 2020-06-05T03:52:04Z

gmtserver data now pointing to the new data_6.1 directory. Will build master and see if OK, then run 6.0.

PaulWessel · 2020-06-05T03:57:32Z

OK, local test with master still work as before (minus ex52.sh and gdal_nn.sh which crashes when run in ctest).

PaulWessel · 2020-06-05T03:59:32Z

How might I switch to 6.0.0 and run all tests locally? THere is no 6.0.0 branch (I think) and the tarball does not have the tests. There is probably a 6.0.0 tag but my GUI Git Desktop does not seem to know about tags?

seisman · 2020-06-05T04:00:49Z

For command line, run git checkout 6.0.0 will checkout the 6.0.0 release.

PaulWessel · 2020-06-05T04:18:21Z

That gave me an empty test dir so no tests could be run...

PaulWessel · 2020-06-05T04:42:23Z

Fails. This fails too:
curl -k -O -L https://oceania.generic-mapping-tools.org/earth_relief_05.grd
But

-bash-4.2$ cd data
-bash-4.2$ ll earth_relief_05m.grd
lrwxrwxrwx 1 pwessel gmt 22 Jun  1 19:19 earth_relief_05m.grd -> earth_relief_05m_p.grd
-bash-4.2$ ll earth_relief_05m_p.grd
-rw-r--r-- 1 pwessel gmt 13022374 Jun  1 19:03 earth_relief_05m_p.grd
-bash-4.2$ pwd
/export/gmtserver/gmt/data

seisman · 2020-06-05T04:44:29Z

curl -k -O -L https://oceania.generic-mapping-tools.org/earth_relief_05.grd this works for me.

PaulWessel · 2020-06-05T04:55:00Z

That is pretty interesting. That is not even a GMT command so not affected by versions. I get a tiny file that contains

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.17.3</center>
</body>
</html>

seisman · 2020-06-05T04:57:20Z

curl -k -O -L https://oceania.generic-mapping-tools.org/earth_relief_05.grd this works for me.

Sorry, I also got the "404 not found" HTML page, not the grid.

seisman · 2020-06-05T04:58:23Z

curl -k -O -L https://oceania.generic-mapping-tools.org/earth_relief_05m.grd

Did you use earth_relief_05.grd or earth_relief_05m.grd?

PaulWessel · 2020-06-05T05:00:09Z

He-he-he.....
THanks.

PaulWessel · 2020-06-05T05:31:01Z

With GMT 6.0.0 and blank .gmt

time gmt grdimage @earth_relief_03s  -R-156/-155/18/20 -JM4i  -B -pdf ma2 -V
real	4m38.105s

due to downloading the 15s 2.6 Gb grid. With 6.1,

time gmt grdimage @earth_relief_03s  -R-156/-155/18/20 -JM4i  -B -pdf map -V
real	0m6.537s

That is why are doing the tiling.

seisman · 2020-06-05T05:58:13Z

gmt grdimage @earth_relief_60m -JH10c -Baf -pdf map -Vd doesn't work because earth_relief_60m is not listed in gmt_data_server.txt

PaulWessel · 2020-06-05T07:46:43Z

I guess the only way it could work for 6.1 is to add another entry for 60m. I link won't do since it gives up by not finding it in the table. But it is not even listed on our dataset page - do we really need to go there?

Use parameter for data hash filename

31de70e

Having this as a variable makes it easier to test features by changing the hash name.

seisman added this to the 6.1.0 milestone May 22, 2020

PaulWessel added 6 commits May 22, 2020 17:26

Compiles after major changes, testing next

55f0bd3

Update gmt_remote.c

89f68cd

Add dir listing

edce592

Eliminate specific loop over srtm dirs since now under server

d1e74b4

Merge branch 'master' into new-remote-files

0509264

Update earth-relief.rst

033f246

PaulWessel added 2 commits May 22, 2020 20:15

Windows fix

b6840bf

Update gmt_io.c

bea4b7d

seisman and others added 3 commits May 23, 2020 15:53

Update gmt_getremote.sh (#3373)

b351a73

Move remote hash/index update to gmt_begin

abd5920

Rely on binary search to find matching entry

76ad3a1

PaulWessel added 2 commits May 23, 2020 14:43

New order of things

3a5cd1e

Update gmt_remote.c

d4b9e5e

PaulWessel added 4 commits May 23, 2020 22:05

Add new API function GMT_Get_FilePath

9e50204

Simlify

54707fa

Update gmtwhich.c

52b5394

fix pos

9375e36

joa-quim approved these changes May 24, 2020

View reviewed changes

PaulWessel added 2 commits May 24, 2020 08:06

typos

97da6aa

Add family and direction back in

56a071f

PaulWessel mentioned this pull request May 24, 2020

Robust handling of filename modifiers #3383

Open

add macro

bf0d129

Merge branch 'new-remote-files' of https://github.com/GenericMappingT…

c56c4c2

…ools/gmt into new-remote-files

seisman approved these changes Jun 5, 2020

View reviewed changes

seisman closed this Jun 5, 2020

seisman reopened this Jun 5, 2020

PaulWessel merged commit 4bffd86 into master Jun 5, 2020

PaulWessel deleted the new-remote-files branch June 5, 2020 03:49

seisman mentioned this pull request Jun 5, 2020

Simpler options to get or remove remote data and cache files #3412

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Let 6.1 have more flexibility to discover new remote data sets #3370

Let 6.1 have more flexibility to discover new remote data sets #3370

PaulWessel commented May 22, 2020

seisman commented May 23, 2020

PaulWessel commented May 23, 2020

PaulWessel commented May 23, 2020

seisman commented May 23, 2020

seisman commented May 23, 2020

PaulWessel commented May 24, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

Let 6.1 have more flexibility to discover new remote data sets #3370

Let 6.1 have more flexibility to discover new remote data sets #3370

Conversation

PaulWessel commented May 22, 2020

seisman commented May 23, 2020

PaulWessel commented May 23, 2020

PaulWessel commented May 23, 2020

seisman commented May 23, 2020

seisman commented May 23, 2020

PaulWessel commented May 24, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

PaulWessel commented Jun 5, 2020

seisman commented Jun 5, 2020

PaulWessel commented Jun 5, 2020