To read run length compressed data

[jfsimoneau]

This is a rework of Tatsuya Noyori's PR #70. With his authorization, I am submitting it here without the public domain requirements after testing on real JMA data.

[FussyDuck]

All committers have signed the CLA.

[shahramn]

Thank you
Can you please sign our Contributor License Agreement before we can accept your contribution

[jfsimoneau]

Thank you Can you please sign our Contributor License Agreement before we can accept your contribution

@shahramn I did. I'm not sure why it doesn't show.

[jfsimoneau]

Figured it out. Noyori-san was sill an author, so I edited the history to list only myself.

[shahramn]

Dear @jfsimoneau
I have merged this into a branch for further testing.
I came across several issues:

First of all I took one of the files in the data set from PR#70
i.e., http:https://www.jmbsc-west.jp/jp/online/online-sample/NOWC_GPV_Ggis1km/NOWC_GPV_Ggis1km_Prr05lv.zip

I extracted the first message in
NOWC_GPV_Ggis1km_Prr05lv/Z__C_RJTD_20170807032000_NOWC_GPV_Ggis1km_Prr05lv_FH0005-0100_grib2.bin

via

% grib_copy -wcount=1 NOWC_GPV_Ggis1km_Prr05lv/Z__C_RJTD_20170807032000_NOWC_GPV_Ggis1km_Prr05lv_FH0005-0100_grib2.bin rle.msg01.grib2

Running wgrib2 -stats prints:

% wgrib2 -stats rle.msg01.grib2 
check_pdt_size: pdt=50008 needs to be added to pdt_len(..)
1:0:ndata=8601600:undef=6259785:mean=0.701772:min=0:max=201:cos_wt_mean=0.722284

whereas ecCodes does:

% grib_ls -n statistics rle.msg01.grib2
max         min         avg           sd          skew        kurt        const
201         -1e+100     -7.27747e+99  4.4512e+99  1.02331     nan         0

So the minimum is incorrect. The maximum however is correct.

Also wgrib2 says there are 6259785 undefined (missing) values but as far as I can see there is no bitmap in this file so I cannot work out why wgrib2 prints that

[jfsimoneau]

Thank you for the comments. I will look into that.

I noticed the -1e+100 values too. I'll take a look at the source and the standard to make sure it fits.

[jfsimoneau]

Ok, the code in this PR converts the invalid values to GRIB_MISSING_DOUBLE instead of 0 as wgrib2 does. I will see if we can mark those values as missing directly.

[shahramn]

One other thing:
I cannot include the file

template.4.50008.def

in our definitions as that is a local product definition number. We can use it just for testing that GRIB data from Japan.

Q. Do you know if there is some other data which uses Run Length Packing but not a local section 4?

[jfsimoneau]

I don't know of any, unfortunately. I can always try to generate one, but I'm not an expert on the matter.

Q. right back at you: If I were to need the template.4.50008.def file locally, how would I install it so ecCodes would be able to use it?

[shahramn]

Dear @jfsimoneau
The way ecCodes finds various definition files/tables is similar to how UNIX finds executable in its PATH environment variable.
So the steps are as follows:

mkdir -p mydefs/grib2
cp /path/to/src/template.4.50008.def mydefs/grib2
export ECCODES_DEFINITION_PATH=$PWD/mydefs/grib2

Now the tools like grib_ls, grib_dump should work

[jfsimoneau]

Thank you for your answers @shahramn!

All in all, I think we can simply drop the template.4.50008.def file from this PR since we will pick it up on our side. For the minimum value of -1e100, I don't think it's wrong, since wgrib2 was not right by saying it was a value of 0. It is an invalid (missing) value encoded inside the RLE stream. I have not found a way to tell ecCodes the value is missing without a bitmap.

Is there anything else I can do to help?

[shahramn]

I would like to get the same answers as wgrib2. Even if you think they're doing the 'wrong' thing.

[jfsimoneau]

Very well, will do!

[jfsimoneau]

Made new PR: #73