Segfault on NetCDF-Java

[LupusUmbrae]

Hi,

I've encountered an interesting issue with NetCDF-Java where creating a netcdf file from an NCML file with a large dimension causes a segfault.

In short when i have a large dimension (in this case 40,000) the netcdf file seems to write a netcdf file but there is a segfault which crashes the toolsUi and the Java application we're using the netcdf-java library under.

This is a test ncml file, note that we're creating a new NetCDF file directly from the ncml file:

I can't find anything that states that this dimension length would be too big and if it was i would of expected an error from the netcdf library not a segfault in libc

Any help with this would be apprecaited, thanks in advance

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http:https://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <dimension name="rows" length="40000" />
  <dimension name="columns" length="1500" />
  <attribute name="title" value="Test Dataset" />
  <variable name="test_var" shape="rows columns" type="short">
    <attribute name="standard_name" value="test_veriable" />
    <attribute name="_FillValue" type="short" value="-32768" />
  </variable>
</netcdf>

ncml from opening the outputted nc file

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http:https://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="file:/home/rmoss/tmp/test.nc">
  <dimension name="rows" length="40000" />
  <dimension name="columns" length="1500" />
  <attribute name="title" value="Test Dataset" />
  <variable name="test_var" shape="rows columns" type="short">
    <attribute name="standard_name" value="test_veriable" />
    <attribute name="_FillValue" type="short" value="-32768" />
    <attribute name="_ChunkSizes" type="int" value="87 1500" />
  </variable>
</netcdf>

Running on Ubuntu (uname -a : Linux UKWGPVL32 3.13.0-39-generic #66-Ubuntu SMP Tue Oct 28 13:30:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux)

nc-config --version : netCDF 4.1.3

toolsUi.txt

hs_err_pid29553.txt

[LupusUmbrae]

Have built a work around locally but not using the FileWriter2(NetcdfFile, String, Version, Nc4Chunking). I've ended up having to use NetcdfFileWriter.createNew(Version, String) and then looping over the contents of the NetcdfDataset.

Not an ideal solution as its very basic and covers only the simplest of use cases

[lesserwhirls]

Greetings Ross,

In order to write netcdf4 files, you need to have at least netCDF-C 4.3.1
or above. We usually recomend the latest release, which in this case would
be 4.4.0. Any chance you could upgrade your version of netCDF-C and give
FileWriter2 a try again?

Thanks!

Sean

On Mon, Apr 4, 2016 at 7:05 AM, Robin Moss [email protected] wrote:

Have built a work around locally but not using the FileWriter2(NetcdfFile,
String, Version, Nc4Chunking). I've ended up having to use NetcdfFileWriter.createNew(Version,
String) and then looping over the contents of the NetcdfDataset.

Not an ideal solution as its very basic and covers only the simplest of
use cases

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#510 (comment)

[LupusUmbrae]

I did wonder if the version would be an issue so i built NetCDF 4.4.0 and still get the same issue, however i didn't do much in terms on configuring the NetCDF build, so here's the nc-config --all output just in case one of the options is wrong

This netCDF 4.4.0 has been built with the following features:

  --cc        -> gcc
  --cflags    ->  -I/usr/local/include -I/home/rmoss/workspace/hdf5-1.8.16/hdf5/include
  --libs      ->

  --has-c++   -> no
  --cxx       ->
  --has-c++4  -> no
  --cxx4      ->

  --fc        ->
  --fflags    ->
  --flibs     ->
  --has-f90   -> no

  --has-dap   -> no
  --has-nc2   -> yes
  --has-nc4   -> yes
  --has-hdf5  -> yes
  --has-hdf4  -> no
  --has-pnetcdf-> no

  --prefix    -> /usr/local
  --includedir-> /usr/local/include
  --version   -> netCDF 4.4.0

[lesserwhirls]

Thanks for testing out 4.4.0! In terms of library configureation, all it needs to have enabled is netCDF-4 output, so your config looks ok.

I can confirm this behavior on Mac OSX using toolsUI (current master). It may take us a bit to track down exactly what is going on here. It's interesting that using FileWriter works, but not FileWriter2. @DennisHeimbigner - would you have any ideas regarding the Java - C communications path for netCDF4 writing? I have not been down at that layer myself.

[LupusUmbrae]

I think its worth noting that we are creating a NetCDF file from scratch using NcML. Using FileWriter2 looks like it tries to copy the file, which also lead to assertion errors in Nc4Iosp.

The way i ended up doing it is something like this:

NetcdfDataset ds = NcMLReader.readNcML(ncmlFilename, null);
NetcdfFileWriter writer = NetcdfFileWriter.createNew(Version.netcdf4, outputFilename);
for(Dimension dim : ds.getDimensions) {
   writer.addDimension(null, dim.getShortName(), dim.getLength());
}
 //ect

[LupusUmbrae]

One last note :)

If i use ncml to write an existing file (i.e. have a location="") it works fine, even if the location file has no data.

[lesserwhirls]

@DennisHeimbigner - just checking in to see if you have any ideas about this one.

[DennisHeimbigner]

Ok, so the size of test_var is 40000_1500_2 = 120,000,000 bytes.
The seg fault may be in the netcdf-c library, but it may also be in the JNA
support library. Did you try increasing the heap space (-Xm...) to see if the problem
goes away?

[LupusUmbrae]

Just written a tiny test, and changed the heap space (-Xms1024m -Xmx2048m), and different sizes but still get the segfault.

It's worth noting that i am trying to create an empty NetCDF 4 file, we are using NcML as a file template.

    public static void main(String[] args) throws IOException {
        File ncmlExample = new File("src/main/resources/example.ncml");
        if (!ncmlExample.exists()) {
            throw new IllegalStateException("Failed to find ncml example file");
        }
        InputStream ncmlStream = new FileInputStream(ncmlExample);
        NetcdfDataset ds = NcMLReader.readNcML(ncmlStream, null);
        ncmlStream.close();
        FileWriter2 writer = new FileWriter2(ds, "target/test.nc4", Version.netcdf4, null);
        writer.write();
    }

With NcML:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http:https://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <dimension name="rows" length="40000" />
  <dimension name="columns" length="1500" />
  <attribute name="title" value="Test Dataset" />
  <variable name="test_var" shape="rows columns" type="short">
    <attribute name="standard_name" value="test_veriable" />
    <attribute name="_FillValue" type="short" value="-32768" />
  </variable>
</netcdf>

[LupusUmbrae]

The issues seems to be in the dataset that the NcMLReader generates, internally i've had to take that dataset and re-create it. Admittedly i'm only touching the surface of the Dataset (dimensions, variables, attributes).

public static NetcdfFile create(InputStream ncmlStream, String outputFilename, Version netcdfVersion)
            throws IOException {
        NetcdfDataset ds = NcMLReader.readNcML(ncmlStream, null);

        return create(ds, outputFilename, netcdfVersion);
    }

    private static NetcdfFile create(NetcdfDataset ds, String outputFilename, Version netcdfVersion)
            throws IOException {
        NetcdfFileWriter writer = NetcdfFileWriter.createNew(netcdfVersion, outputFilename);

        List<Dimension> dims = createDimensions(writer, null, ds.getDimensions());
        createGroupAttributes(writer, null, ds.getGlobalAttributes());
        createVariables(writer, null, ds.getVariables(), dims);
        writer.create();
        return writer.getNetcdfFile();
    }

    private static List<Dimension> createDimensions(NetcdfFileWriter writer, Group group, List<Dimension> dimensions) {
        List<Dimension> dims = new ArrayList<>();
        for (Dimension dim : dimensions) {
            int length = dim.getLength();
            if (dim.isUnlimited()) {
                length = 0;
            }
            Dimension newDim = writer.addDimension(group, dim.getShortName(), length, dim.isShared(), dim.isUnlimited(),
                    dim.isVariableLength());
            dims.add(newDim);
        }
        return dims;
    }

    private static void createGroupAttributes(NetcdfFileWriter writer, Group group, List<Attribute> attributes) {
        for (Attribute attr : attributes) {
            writer.addGroupAttribute(group, attr);
        }
    }

    private static void createVariableAttributes(NetcdfFileWriter writer, Variable var, List<Attribute> attributes) {
        for (Attribute attr : attributes) {
            writer.addVariableAttribute(var, attr);
        }
    }

    private static void createVariables(NetcdfFileWriter writer, Group group, List<Variable> variables,
            List<Dimension> dims) {
        for (Variable var : variables) {
            List<Dimension> dimsToUse = new ArrayList<>();
            for (Dimension curDim : var.getDimensions()) {
                for (Dimension myDim : dims) {
                    if (myDim.getFullName().equals(curDim.getFullName())) {
                        dimsToUse.add(myDim);
                        break;
                    }
                }
            }

            Variable newVar = writer.addVariable(group, var.getShortName(), var.getDataType(), dimsToUse);
            createVariableAttributes(writer, newVar, var.getAttributes());
        }
    }

This works for my use case, but won't for any more details uses of NcML, although it may give some insight into whats happening.