Generalize default CPT selection for datasets #3513

PaulWessel · 2020-06-21T05:26:21Z

Description of proposed changes

With several global remote datasets coming online, the initial implementation of switching to geo or srtm if earth_relief_ or srtm_relief_ files were selected is no longer correct. The scheme partly breaks because we now have tiles and the filename grdimage sees may no longer contain "earth_relief". Furthermore, other datasets may have their own default CPTs and we should not hardwire those in GMT. Now, such CPTs are listed on the server in gmt_data_server.txt. I had to change several functions to implement this feature.

This PR implements the changes but expects an updated gmt_data_server.txt file with the extra CPT column. I have added WIP for this branch so it is not merged until read. Testing can be done by placing this next-gen gmt_data_server.txt in your .gmt/server directory.

gmt_data_server.txt

WIth many remote data sets the simple scheme before is broken. I will implement a new one by checking is a given remote dataset has a specified master cpt.

/home/dir/place/earth_relief_30m_p is still a remote data set.

seisman · 2020-06-21T06:28:58Z

src/gmt_remote.c

- /* New format - soon the only format once testing of 6.1 is over */
- GMT_Report (GMT->parent, GMT_MSG_WARNING, "File %s should have 12 fields but only %d read for record %d - download error???\n", file, nr, k);
+ if ((nr = sscanf (line, "%s %s %s %c %lg %lg %s %lg %s %s %s %s %[^\n]", I[k].dir, I[k].file, I[k].inc, &I[k].reg, &I[k].scale, &I[k].offset, I[k].size, &I[k].tile_size, I[k].date, I[k].coverage, I[k].filler, I[k].CPT, I[k].remark)) != 13) {
+ GMT_Report (GMT->parent, GMT_MSG_WARNING, "File %s should have 13 fields but only %d read for record %d - download error???\n", file, nr, k);


Like what we're doing here, we may have to update the format of gmt_data_server.txt someday, which will break old GMT versions. Perhaps we should say download error or incompatible GMT versions..

I'm thinking if we should/can add a line in gmt_data_server.txt, which gives the minimin GMT version for the GMT server.

OK, reading your comment now. Hm. Since 6.1 is unreleased and 6.0.0 does not know about gmt_data_server.txt, the worst that can happen is that somebody complains and we tell them to update.

What I mean is, after GMT 6.1.0 is released, we may find another serious bug and have to add one more column in gmt_data_server.txt to fix it. The new format will break GMT 6.1.0 users, while the old format will break GMT master-branch users. We have other choices and have to release GMT 6.2.0 immediately.

Even we release GMT 6.2.0 as soon as we can, there are still many GMT 6.1.0 users. They will see the "download error????" warning but the warning is not exactly correct.

If we add a line 6.2.0 in gmt_data_server.txt, then GMT can compare if the current GMT version is smaller than the minimum required version for the data server. If yes, then GMT can report a warning to tell users that "they have to update GMT to version x.x.x" to use remote dataset.

OK, fine, so I will add 6.1.0 to that file and parse it. If your version is less then update GMT.

Great, but be careful that GMT version can be 6.1.0, 6.1.0rc1 and 6.1.0_2b42bb7_2020.06.21.

This has to work for GMT 6.1.0 regardless of what version we are on because that is what ubuntu will offer until 2026...

Is there any way to push them update to the latest GMT version

You mean ubuntu? No control, they have their own schedules. So all the linux people taking the UNAVCO course in some weeks will have 6.0.0 at best, so we will need to have them build from git and make that super easy.

src/gmt_remote.c

seisman · 2020-06-21T06:41:39Z

It seems the gmt_data_server.txt you attached is still in old format and doesn't include the number of records at the top.

I downloaded the file from https://github.com/GenericMappingTools/gmtserver-admin/pull/62/files and manually added 77 at the beginning of the file. However, all tiled earth_relief data are using turbo now.

PaulWessel · 2020-06-21T07:59:41Z

Yes, sorry, must have attached the old file. With the right file in .gmt/server all tests pass for me. But perhaps I need to wipe my server files and start over.. TOmorow...

PaulWessel · 2020-06-21T23:39:12Z

Reran all test and all except grdimage_img_tif.sh passed. Are you really getting failed tests with turbo colors?

seisman · 2020-06-21T23:55:00Z

Please upload the new gmt_data_server.txt so that I can test it again.

PaulWessel · 2020-06-21T23:56:30Z

This is what I just used so it has the 77 etc.
gmt_data_server.txt

seisman · 2020-06-22T00:00:43Z

Still turbo for gmt grdimage @earth_relief_01m -RMG+r2 -Vd -png noshading.

See the attached log: log.txt

PaulWessel · 2020-06-22T00:09:13Z

OK, good. Same for me. But tests pass though, right? Will debug this guy (again).

seisman · 2020-06-22T00:11:56Z

I didn't run the full tests, but it's weird that all tests in the master branch pass in yesterday CI cron jobs.

Messed up case with user tile name.

PaulWessel · 2020-06-22T00:29:56Z

Now works for me.

seisman · 2020-06-22T00:54:27Z

Works for me too.

PaulWessel · 2020-06-22T01:07:34Z

How shall we do this merging? We need the gmtserver file merge - which updates on the hour (or manually). Then the gmt master merge. It is Sunday so perhaps we can commit the gmtserver now (and let it git update on the hour), then do the master gmt merge about that time as well, i.e., in ~50 minutes?

PaulWessel · 2020-06-22T02:33:28Z

I added the check for version that should be foolproof. The data file is attached.
gmt_data_server.txt

seisman · 2020-06-22T02:58:20Z

src/gmt_remote.c

+ GMT_Report (GMT->parent, GMT_MSG_ERROR, "Your GMT version too old to use the remote data mechanism - please upgrade to %s or later\n", line);
+ else
+ GMT_Report (GMT->parent, GMT_MSG_ERROR, "Unable to parse %s to extract GMT version\n", line);
+ return NULL;


Even though users' GMT is too old, they can still use it as long as they don't use remote dataset. Perhaps make it a notice or a warning?

Since the gmt_data_server.txt on the GMT data server is still in old format, our CI jobs report warnings like:

gmt [ERROR]: Unable to parse # Master table with information about all the remote data sets available on the GMT Data serve

Please add quotes after parse before to.

The command gmt pscoast -R0/10/0/10 -JM6i -Ba -Ggray -ENG+p1p,blue -P -Vd > map.ps crashes:

gmt [DEBUG]: Initialize FFTW with 4 threads. gmt [DEBUG]: Local file /Users/runner/.gmt/server/gmt_data_server.txt found gmt [DEBUG]: File /Users/runner/.gmt/server/gmt_data_server.txt less than 24 hours old, refresh is premature. gmt [DEBUG]: Load contents from /Users/runner/.gmt/server/gmt_data_server.txt gmt [ERROR]: Unable to parse # Master table with information about all the remote data sets available on the GMT Data server to extract GMT version gmt [INFORMATION]: Unable to read server information file gmt [DEBUG]: Local file /Users/runner/.gmt/server/gmt_hash_server.txt found gmt [DEBUG]: File /Users/runner/.gmt/server/gmt_hash_server.txt less than 24 hours old, refresh is premature. gmt [DEBUG]: GMT_Create_Session initialized GMT structure gmt [DEBUG]: Loading core GMT shared library: libgmt.dylib gmt [DEBUG]: Shared Library # 0 (core). Path = libgmt.dylib gmt [DEBUG]: Loading GMT plugins from: /Users/runner/runners/2.170.1/work/1/s/gmt-install-dir/lib/gmt/plugins gmt [DEBUG]: Shared Library # 1 (supplements). Path = /Users/runner/runners/2.170.1/work/1/s/gmt-install-dir/lib/gmt/plugins/supplements.so gmt [DEBUG]: Revised options: -R0/10/0/10 -JM6i -Ba -Ggray -ENG+p1p,blue -P -Vd pscoast [DEBUG]: History: Process -R0/10/0/10 pscoast [DEBUG]: History: Process -JM6i pscoast [DEBUG]: History: Process -Ba pscoast [DEBUG]: Map distance calculation will be using great circle approximation with authalic auxiliary latitudes and authalic (R_2) radius = 6371007.1809 m, in meter. pscoast [DEBUG]: gmt_get_filename: In: 0/10/0/10 Out: 0/10/0/10 ERROR: Caught signal number 11 (Segmentation fault) at 0 libsystem_platform.dylib 0x00007fff7121ed45 _platform_strncmp + 325 1 ??? 0x0000000000005825 0x0 + 22565 Stack backtrace: 0 libgmt.6.dylib 0x00000001057da235 sig_handler + 581 1 libsystem_platform.dylib 0x00007fff712215fd _sigtramp + 29 2 ??? 0x0000000000000001 0x0 + 1 3 libsystem_c.dylib 0x00007fff710ce313 bsearch + 57 4 libgmt.6.dylib 0x00000001057dd196 gmt_remote_no_extension + 134 5 libgmt.6.dylib 0x000000010584b415 gmt_access + 421 6 libgmt.6.dylib 0x00000001058ef889 gmt_parse_R_option + 1481 7 libgmt.6.dylib 0x0000000105917311 gmt_parse_common_options + 6897 8 libgmt.6.dylib 0x00000001058df13b GMT_Parse_Common + 5067

Did 1 and 2. 3 died not crash for me. I suppose it will if not updating the remote file soon

If you remove your local gmt_data_server.txt, you may see the crash.

Sure, but once we upload the new file...

Co-authored-by: Dongdong Tian <[email protected]>

I screwed up earlier I think since this orig is cleary turbo, but should be geo, which now works.

PaulWessel · 2020-06-22T19:35:47Z

Let me know how you see this, @seisman. I thought we would update gmt_data_server.txt on oceania and more or less simultaneously merge this branch into master. There is no way to let an un-updated master version used by someone to not have trouble with this new file. It is also not possible to add a message to their outdated versions. Today I will ahve time to do this between 10:30-1 pm HST and after 2 pm HSG.

src/grdimage.c

Co-authored-by: Dongdong Tian <[email protected]>

PaulWessel · 2020-06-22T19:48:37Z

OK, as soon as my zoom is over I will do this.

PaulWessel added 5 commits June 20, 2020 16:40

Update how default master CPT is picked for remote datasets

fd666b9

WIth many remote data sets the simple scheme before is broken. I will implement a new one by checking is a given remote dataset has a specified master cpt.

Eliminate short list of default CPTs, use remote info instead

2f91c7c

Update gmt_support.c

ce9049b

Update gmt_remote.c

1dc6214

Skip leading directory when getting local local copy of remote file

a13cc51

/home/dir/place/earth_relief_30m_p is still a remote data set.

PaulWessel requested review from joa-quim and seisman June 21, 2020 05:26

PaulWessel changed the title ~~Generalize default CPT selection for datasets~~ WIP Generalize default CPT selection for datasets Jun 21, 2020

seisman reviewed Jun 21, 2020

View reviewed changes

Merge branch 'master' into relief-cpt-check

165eadf

Fix gmt_cpt_default

1b1bcd2

Messed up case with user tile name.

Check min GMT version needed

9a7f304

Merge branch 'master' into relief-cpt-check

d673a6c

seisman reviewed Jun 22, 2020

View reviewed changes

PaulWessel and others added 3 commits June 21, 2020 19:57

Update src/gmt_remote.c

52b6090

Co-authored-by: Dongdong Tian <[email protected]>

Update gmt_remote.c

9af2198

Update autointense.ps

c6670cc

I screwed up earlier I think since this orig is cleary turbo, but should be geo, which now works.

seisman reviewed Jun 22, 2020

View reviewed changes

src/grdimage.c Outdated Show resolved Hide resolved

Update src/grdimage.c

b1fa44d

Co-authored-by: Dongdong Tian <[email protected]>

seisman approved these changes Jun 22, 2020

View reviewed changes

PaulWessel changed the title ~~WIP Generalize default CPT selection for datasets~~ Generalize default CPT selection for datasets Jun 22, 2020

PaulWessel merged commit 58483ca into master Jun 22, 2020

PaulWessel deleted the relief-cpt-check branch June 22, 2020 20:15

seisman mentioned this pull request Jun 22, 2020

Wrong master CPT for shading? #3511

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generalize default CPT selection for datasets #3513

Generalize default CPT selection for datasets #3513

PaulWessel commented Jun 21, 2020

seisman Jun 21, 2020

PaulWessel Jun 22, 2020

seisman Jun 22, 2020

PaulWessel Jun 22, 2020

seisman Jun 22, 2020

PaulWessel Jun 22, 2020

seisman Jun 22, 2020

PaulWessel Jun 22, 2020

seisman commented Jun 21, 2020 •

edited

Loading

PaulWessel commented Jun 21, 2020

PaulWessel commented Jun 21, 2020

seisman commented Jun 21, 2020

PaulWessel commented Jun 21, 2020

seisman commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

seisman commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

seisman commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

seisman Jun 22, 2020 •

edited

Loading

PaulWessel Jun 22, 2020 •

edited

Loading

seisman Jun 22, 2020

PaulWessel Jun 22, 2020

PaulWessel commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

Generalize default CPT selection for datasets #3513

Generalize default CPT selection for datasets #3513

Conversation

PaulWessel commented Jun 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seisman commented Jun 21, 2020 • edited Loading

PaulWessel commented Jun 21, 2020

PaulWessel commented Jun 21, 2020

seisman commented Jun 21, 2020

PaulWessel commented Jun 21, 2020

seisman commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

seisman commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

seisman commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

seisman Jun 22, 2020 • edited Loading

Choose a reason for hiding this comment

PaulWessel Jun 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PaulWessel commented Jun 22, 2020

PaulWessel commented Jun 22, 2020

seisman commented Jun 21, 2020 •

edited

Loading

seisman Jun 22, 2020 •

edited

Loading

PaulWessel Jun 22, 2020 •

edited

Loading