Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

To read run length compressed data #72

Merged
merged 2 commits into from
Jan 19, 2023
Merged

To read run length compressed data #72

merged 2 commits into from
Jan 19, 2023

Conversation

jfsimoneau
Copy link

This is a rework of Tatsuya Noyori's PR #70. With his authorization, I am submitting it here without the public domain requirements after testing on real JMA data.

@FussyDuck
Copy link

FussyDuck commented Jan 18, 2023

CLA assistant check
All committers have signed the CLA.

@shahramn
Copy link
Collaborator

Thank you
Can you please sign our Contributor License Agreement before we can accept your contribution

@jfsimoneau
Copy link
Author

jfsimoneau commented Jan 18, 2023

Thank you Can you please sign our Contributor License Agreement before we can accept your contribution

@shahramn I did. I'm not sure why it doesn't show.

@jfsimoneau
Copy link
Author

Figured it out. Noyori-san was sill an author, so I edited the history to list only myself.

@shahramn shahramn changed the base branch from develop to feature/ECC-1510-RunLengthPacking January 19, 2023 20:10
@shahramn shahramn merged commit b4234ae into ecmwf:feature/ECC-1510-RunLengthPacking Jan 19, 2023
@shahramn
Copy link
Collaborator

shahramn commented Jan 19, 2023

Dear @jfsimoneau
I have merged this into a branch for further testing.
I came across several issues:

First of all I took one of the files in the data set from PR#70
i.e., http:https://www.jmbsc-west.jp/jp/online/online-sample/NOWC_GPV_Ggis1km/NOWC_GPV_Ggis1km_Prr05lv.zip

I extracted the first message in
NOWC_GPV_Ggis1km_Prr05lv/Z__C_RJTD_20170807032000_NOWC_GPV_Ggis1km_Prr05lv_FH0005-0100_grib2.bin

via

% grib_copy -wcount=1 NOWC_GPV_Ggis1km_Prr05lv/Z__C_RJTD_20170807032000_NOWC_GPV_Ggis1km_Prr05lv_FH0005-0100_grib2.bin rle.msg01.grib2

Running wgrib2 -stats prints:

% wgrib2 -stats rle.msg01.grib2 
check_pdt_size: pdt=50008 needs to be added to pdt_len(..)
1:0:ndata=8601600:undef=6259785:mean=0.701772:min=0:max=201:cos_wt_mean=0.722284

whereas ecCodes does:

% grib_ls -n statistics rle.msg01.grib2
max         min         avg           sd          skew        kurt        const
201         -1e+100     -7.27747e+99  4.4512e+99  1.02331     nan         0

So the minimum is incorrect. The maximum however is correct.

Also wgrib2 says there are 6259785 undefined (missing) values but as far as I can see there is no bitmap in this file so I cannot work out why wgrib2 prints that

@jfsimoneau
Copy link
Author

Thank you for the comments. I will look into that.

I noticed the -1e+100 values too. I'll take a look at the source and the standard to make sure it fits.

@jfsimoneau
Copy link
Author

Ok, the code in this PR converts the invalid values to GRIB_MISSING_DOUBLE instead of 0 as wgrib2 does. I will see if we can mark those values as missing directly.

@shahramn
Copy link
Collaborator

shahramn commented Jan 19, 2023

One other thing:
I cannot include the file

template.4.50008.def

in our definitions as that is a local product definition number. We can use it just for testing that GRIB data from Japan.

Q. Do you know if there is some other data which uses Run Length Packing but not a local section 4?

@jfsimoneau
Copy link
Author

I don't know of any, unfortunately. I can always try to generate one, but I'm not an expert on the matter.

Q. right back at you: If I were to need the template.4.50008.def file locally, how would I install it so ecCodes would be able to use it?

@shahramn
Copy link
Collaborator

Dear @jfsimoneau
The way ecCodes finds various definition files/tables is similar to how UNIX finds executable in its PATH environment variable.
So the steps are as follows:

mkdir -p mydefs/grib2
cp /path/to/src/template.4.50008.def mydefs/grib2
export ECCODES_DEFINITION_PATH=$PWD/mydefs/grib2

Now the tools like grib_ls, grib_dump should work

@jfsimoneau
Copy link
Author

Thank you for your answers @shahramn!

All in all, I think we can simply drop the template.4.50008.def file from this PR since we will pick it up on our side. For the minimum value of -1e100, I don't think it's wrong, since wgrib2 was not right by saying it was a value of 0. It is an invalid (missing) value encoded inside the RLE stream. I have not found a way to tell ecCodes the value is missing without a bitmap.

Is there anything else I can do to help?

@shahramn
Copy link
Collaborator

I would like to get the same answers as wgrib2. Even if you think they're doing the 'wrong' thing.

@jfsimoneau
Copy link
Author

Very well, will do!

@jfsimoneau
Copy link
Author

Made new PR: #73

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants