NetCDF-Java and HDF dynamic filters

[joaquinrgu]

The latest versión of HDF-Java is able to load HDF dynamic filters (BZIP2 in my case), while NetCDF-Java fails:

!$ java -Xmx1g -classpath ~/Programs/toolsUI-4.6.jar ucar.nc2.NCdumpW h5ex_d_bzip2.h5 -vall
java.lang.RuntimeException: Unknown filter type=307
at ucar.nc2.iosp.hdf5.H5tiledLayoutBB$DataChunk.getByteBuffer(H5tiledLayoutBB.java:198)
at ucar.nc2.iosp.LayoutBBTiled.hasNext(LayoutBBTiled.java:128)
at ucar.nc2.iosp.hdf5.H5tiledLayoutBB.hasNext(H5tiledLayoutBB.java:126)
at ucar.nc2.iosp.IospHelper.readData(IospHelper.java:336)
at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:292)
at ucar.nc2.iosp.hdf5.H5iosp.readData(H5iosp.java:162)
at ucar.nc2.iosp.hdf5.H5iosp.readData(H5iosp.java:140)
at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1986)
at ucar.nc2.Variable.reallyRead(Variable.java:868)
at ucar.nc2.Variable._read(Variable.java:839)
at ucar.nc2.Variable.read(Variable.java:717)
at ucar.nc2.NCdumpW.print(NCdumpW.java:328)
at ucar.nc2.NCdumpW.print(NCdumpW.java:228)
at ucar.nc2.NCdumpW.print(NCdumpW.java:171)
at ucar.nc2.NCdumpW.main(NCdumpW.java:1009)
netcdf h5ex_d_bzip2.h5 {
variables:
int DS1(32, 64);
:_ChunkSizes = 4, 8; // int

data:
Unknown filter type=307

NetCDF-Java is not built on top of HDF-Java (HDF-Java also had some problems in the past to load dynamic filters), is it? are there plans to support HDF dynamic filters?

Thanks!
Joaquín

[lesserwhirls]

Greetings!

This seems to be a very popular question here recently!

netCDF-java is not built on top of HDF-Java. The HDF implementation in netCDF-Java totally seperate from HDF tools, and is pure java. It was written before my time here at Unidata. However, from what I gather, if you want to use a custom filter in netCDF-Java, you will need to implement that filter in Java as well.

You will need to modify java code in the HDF5 IOSP used by netCDF-Java. More specifically by adding the filter to the H5header class:

https://github.com/Unidata/thredds/blob/master/cdm/src/main/java/ucar/nc2/iosp/hdf5/H5header.java

and implementing the filter in the H5tiledLayoutBB class:

https://github.com/Unidata/thredds/blob/master/cdm/src/main/java/ucar/nc2/iosp/hdf5/H5tiledLayoutBB.java

Again, this code predates my time at Unidata, so there may be more to do that what I've outlined above, in terms of reading your file.

Also note, netCDF-Java will not read HDF5 files written using superblock 3, which is the default of the new 1.10 HDF5 lib. You will need to compile it to use the 1.8 compatibility option.

[joaquinrgu]

Thanks for the quick reply and the helpful information Sean!
It is then not very likely that netCDF-java based viewers like IDV or Panoply support this in the future, right? we would have to overwrite the NetCDF jar file that comes with these programs with our modified one, correct?

[DennisHeimbigner]

It should be noted that there is an experimental branch of the netcdf-c library that
supports adding filters. Currently, the filters need to be added at library compile time
but we are working on the dynamic loading option.
So, if you are willing to use JNI/JNA, you can read netcdf files using the netcdf c library.

[joaquinrgu]

Thanks Dennis, in fact NetCDF-C 4.3.2 built on top of HDF5 (compiled with --enable-hl) has no problems loading the HDF dynamic filters located in $HDF5_PLUGIN_PATH. Ncdump from NetCDF-C successfully displays compressed BZIP2 variables when the filter is installed.

[DennisHeimbigner]

Thanks. It will work for decoding, but not encoding. I will add that note to my
documentation.

[lesserwhirls]

Hi @joaquinrgu - you are correct, you would need to overwrite the netcdf jars shipped with those applications. The current HDF5 code in netCDF Java is pretty complex, and we're considering calling into the C library to read hdf5 (or at least make that an option), much like we do to write netCDF-4 with the netCDF-C library. However, that would be a pretty involved, so I do not have a time estimate for that work, if we do go down that path.

[joaquinrgu]

Thanks Sean! that would be great! even better as an option I guess so there is no dependency with netCFD-C for reading. Have you consider invoking HDF5-java in netCDF-java as netCDF-C invokes HDF5-C?

[joaquinrgu]

It should be noted that there is an experimental branch of the netcdf-c library that
supports adding filters. Currently, the filters need to be added at library compile time
but we are working on the dynamic loading option.

Hi, I was wondering if there was any progress on the netcdf-c and filters for compressing. Thanks!

[DennisHeimbigner]

We just merged the new filter code into our master branch.
So it is official.

[joaquinrgu]

That´s fantastic news, thanks Dennis! any chance it has already been documented?