NetCDF-C for reading (nj22Config.xml) in non-Unidata netCDF-Java based tools

[joaquinrgu]

NetCDF-Java can be configured to use libnetcdf (if installed) to read datasets, instead of its own classes.
As an example, the following config at ~/.unidata/nj22Config.xml allows to use libnetcdf for reading in ToolsUI:

<nj22Config>
  <Netcdf4Clibrary>
     <libraryPath>/opt/netcdf-4.4.1/lib</libraryPath>
     <libraryName>netcdf</libraryName>
     <useForReading>true</useForReading>
  </Netcdf4Clibrary>
</nj22Config>

However other NetCDF-Java based non-Unidata tools, such as Panoply, don't seem to pick up the configuration and fail to use netCDF-C for reading. Do they need to do anything special to take advantage of this netCDF-java feature? This support would be beneficial as currently only netCDF-C allows to decompress datasets previously compressed with HDF filters.

[cofinoa]

Dear @joaquinrgu,

I'm not sure if you are having a particular issue with a HFD5 filter, but be sure that netcdf-c's library path is in the LD_LIBRARY_PATH enviroment variable for the netcdf-java tool you want to use.

$ export LD_LIBRARY_PATH=/opt/netcdf-4.4.1/lib:$LD_LIBRARY_PATH
$ ./panoply.sh

[DennisHeimbigner]

I believe that the Java HDF5 reader in netcdf-java will support libz compression.
If you are using some other (de-)compressor, then you will need to use the
native netcdf as Sean describes above. Also, you may need to set the environment
variable HDF5_PLUGIN_PATH

[joaquinrgu]

Dear both,

Thanks for your advise. The HDF5 filter seems to be working fine, as I am able to open the compressed dataset with both ToolsUI and IDV and show the data, which is uncompressed on the fly. I can see traces of this in the HDF5 filter logs and also the log "Nc4Iosp:NetCDF-4 C library loaded (jna_path=..." in the console.

However, when I try to do the same with Panoply, in the same environment, I end up with the error message below, and can't find any logs regarding netCDF-C or the HDF filter.

"There was an error while preparing the plot. Failed to create the data handler because: Unknown filter type=-8754. "

I wonder if netCDF-java is prepared to load the netCDF-C library by default when finding the nj22Config.xml configuration or if there is some explicit code in IDV and ToolsUI to do so. It would be good to know if anyone made any non-Unidata netCDF-java based tool, such as Panoply, run with netCDF-C.

[ethanrd]

Yes, there is explicit code in ToolsUI.main(...), around line 5753, to read the nj22Config file and hand it to RuntimeConfigParser.read(...) which configures the netCDF-C library here.

[joaquinrgu]

Thanks Ethan for checking! would it make sense to add that piece of code to netCDF-java so non-Unidata tools can benefit from this feature out of the box? or are there any plans to support HDF5 filters in netCDF-java (natively or adding a dependency on hdf-java?)

[lesserwhirls]

Greetings @joaquinrgu - we discussed this today at our meeting and came up with a few ways this could be done at the library level. For now, however, applications built on-top of netCDF-java will need to add the hook to look for a config file (as described here in our docs).

In terms of filters, we support the compression algorithms that netCDF-C natively supports, but not any any filters that are added in to hdf at this point. Are there some standard ones that are used often? We could look at adding native java support for these, where possible.

[joaquinrgu]

Thanks @lesserwhirls - it would be great to see this feature in the library :)
NetCDF-C 4.6 has introduced full support for HDF5 dynamic filters and it works fine with a custom filter we use (https://support.hdfgroup.org/services/filters.html#fcidecomp). Thanks to this nj22Config.xml option, ToolsUI and IDV are able to decompress datasets compressed with this codec. It would be great if others, such as Panoply, could do the same.
PD: HDFView, based on hdf-java, supports HDF dynamic filters. Has it ever been considered to rely on hdf-java for these kinds of tasks?

[lesserwhirls]

Hi @joaquinrgu - we have only spoken about adding hdf-java as an option at this point. It would need to be added as an IOSP. Unfortunately, it is till based on the HDF C library. Our HDF5 reading is pure java, which gives us much better server side performance (but that's only one consideration) - HDF5 and netCDF-C are not "thread safe" libraries, as they have global state, so we must lock read/write calls that use those c libraries via jna. There used to be a thredds-safe option for HDF5, but it didn't not fully work in our server environments (and last I checked, it was not a fully supported option anymore).

From a data standards point of view, dynamic filters makes me a bit nervous, unless there is a list of fully open source filters available (the list on the HDF page is a good start). I'm not saying a variety of dynamic filters are a bad idea, it just complicates the landscape quite a bit, and allows people to produce netCDF files that can only be read, potentially, if you pay big bucks for the use of a particular filter library. As far as I know, there is no consideration of filters in any data standards yet.

[joaquinrgu]

Thanks for the explanation @lesserwhirls , sometimes we the users are not aware of all these complications. It would be great if .nj22config could be used to set how the netCDF-java library behaves and weather it loads netCDF-C for reading or not.

[rschmunk]

Looking at adding an automatic check in Panoply for a $HOME/.unidata/nj22config.xml file, as referenced in ToolsUI by ethanrd above. However, it seems it will have to wait until I commit to upgrading to 5.0 — i.e., maybe next month.

[joaquinrgu]

That is excellent news @msdsoftware , thanks!

[rschmunk]

@joaquinrgu, Could you download the just released Panoply 4.9 and see if it reads the nj22Config file?

[joaquinrgu]

Great work @msdsoftware ! it worked fine for me as I was able to export to .csv some compressed variables. Thanks!