Support to read HDF4 files

[Yujie-W]

The libnetcdf.so shipped with NCDatasets does not support reading HDF4 files, here is the chunk in the settting file

# Features
--------
NetCDF-2 API:		yes
HDF4 Support:		no
HDF5 Support:		yes
NetCDF-4 API:		yes
CDF5 Support:		yes
NC-4 Parallel Support:	yes
PnetCDF Support:	no

Thus, I got the error like this when reading a HDF4 file (from MODIS),

ERROR: NetCDF error: Opening path MOD09A1.A2019361.h35v10.006.2020005030852.hdf: NetCDF: Attempt to use feature that was not turned on when netCDF was built. (NetCDF error code: -128)

So the problem is within Netcdf_jll shipped with NCDatasets.jl. I guess the primary reason is to enable NC-4 Parallel Support?

I looked around, and found that the libnetcdf.so shipped with Conda.jl has the HDF4 support enabled, and this is in its setting file (v4.8.1) in my case

NetCDF-2 API:           yes
HDF4 Support:           yes
HDF5 Support:           yes
NetCDF-4 API:           yes
NC-4 Parallel Support:  no
PnetCDF Support:        no

After reading through your code in file netcdf_c.jl, e.g.,

code = ccall((:nc_open,libnetcdf),Cint,(Cstring,Cint,Ptr{Cint}),path,mode,ncidp)

it occurred to me the reference to the string libnetcdf can be swapped to an external source. So I tried this, and it works for HDF4 files

import NetCDF_jll

const LIBNETCDF  = deepcopy(NetCDF_jll.libnetcdf);

function switch_netcdf_lib!(; use_default::Bool = true, user_defined::String = "$(homedir())/.julia/conda/3/lib/libnetcdf.so")
    if use_default
        NetCDF_jll.libnetcdf = LIBNETCDF;
    else
        if isfile(user_defined)
            NetCDF_jll.libnetcdf = user_defined;
        else
            @warn "File '$(user_defined)' not found!";
            @info "Hint: You may libnetcdf shipped with Conda.jl using Conda.add(\"libnetcdf\"). A version above 4.8.1 is recommended.";
            @warn "The file '$(user_defined)' does not exist, please make sure you have provided the correct path!";
        end;
    end;

    return nothing
end

I am pasting a solution here in case you want to add HDF4 support to NCDatasets. If so, I am thinking maybe you can add an alternative Netcdf_jll (say Netcdf_HDF4_jll) to the dependency and switch among the library when required.

[Yujie-W]

Seems like this in you issues.md already

NCDatasets.jl/docs/src/issues.md

Line 75 in 66d1e67

The NetCDF library `libnetcdf.so` is installed as an artifact via the package `NetCDF_jll`.

So, there is not a hard fix?

[Yujie-W]

Then I found my old solution and yours will not allow for dynamically loading and unloading the libnetcdf library. Inspired by this: JuliaLang/julia#23459 (comment)

I made some changes like

function switch_netcdf_lib!(; use_default::Bool = true, user_defined::String = "$(homedir())/.julia/conda/3/lib/libnetcdf.so")
    if use_default
        NetCDF_jll.libnetcdf = LIBNETCDF;
        NetCDF_jll.libnetcdf_path = LIBNETCDF;
        Base.Libc.Libdl.dlclose(NetCDF_jll.libnetcdf_handle);
        NetCDF_jll.libnetcdf_handle = Base.Libc.Libdl.dlopen(LIBNETCDF);
    else
        if isfile(user_defined)
            NetCDF_jll.libnetcdf = user_defined;
            NetCDF_jll.libnetcdf_path = user_defined;
            Base.Libc.Libdl.dlclose(NetCDF_jll.libnetcdf_handle);
            NetCDF_jll.libnetcdf_handle = Base.Libc.Libdl.dlopen(user_defined);
        else
            @warn "File '$(user_defined)' not found!";
            @info "Hint: You may libnetcdf shipped with Conda.jl using Conda.add(\"libnetcdf\"). A version above 4.8.1 is recommended.";
            @warn "The file '$(user_defined)' does not exist, please make sure you have provided the correct path!";
        end;
    end;

    return nothing
end

The functions with ccall need to be chaned accordingly to use the handle; otherwise the already loaded library won't be unloaded

function nc_open(path,mode::Integer)
    @debug "nc_open $path with mode $mode"
    ncidp = Ref(Cint(0))

    @show ncidp;
    _sym = Base.Libc.Libdl.dlsym(NetCDF_jll.libnetcdf_handle, :nc_open)
    code = ccall(_sym,Cint,(Cstring,Cint,Ptr{Cint}),path,mode,ncidp)

    if code == NC_NOERR
        return ncidp[]
    else
        # otherwise throw an error message
        # with a more helpful error message (i.e. with the path)
        throw(NetCDFError(code, "Opening path $(path): $(nc_strerror(code))"))
    end
end

Now when I run this sort of code, I will be able to load/unload the library dynamically

using NCDatasets; fn="test.hdf"; vn="sur_refl_b01";
switch_netcdf_lib!(use_default=true); data=read_nc(fn, vn)     # error here
switch_netcdf_lib!(use_default=false); data=read_nc(fn, vn)    # succeed
switch_netcdf_lib!(use_default=true); data=read_nc(fn, vn)     # error again

[Alexander-Barth]

I am wondering if you did see the section "Using a custom NetCDF library" in the documentation?

https://alexander-barth.github.io/NCDatasets.jl/stable/issues/#Using-a-custom-NetCDF-library

(you need to restart julia if NCDatasets was already loaded).

The cleanest solution would indeed to be build NetCDF_jll with HDF4 support. If somebody has the time to contribute a build script for HDF4 to https://github.com/JuliaPackaging/Yggdrasil/ using BinaryBuilder that would be really great! Note that for HDF5, cross-compilation was a long-standing issue.

I think it is better to keep track of improvement of NetCDF_jll at https://github.com/JuliaPackaging/Yggdrasil/.

[meggart]

I started making a HDF4 build here: JuliaPackaging/Yggdrasil#7465 and if it works we could try to support this by default in NetCDF_jll