NetcdfFile.openInMemory() fails in 4.3

[kenahoo]

In NetCDF 4.2, I do the following, which works:

FileInputStream is = new FileInputStream(testPath);
DataInputStream dis = new DataInputStream(is);
byte[] buf = new byte[(int) is.getChannel().size()];
dis.readFully(buf);

ncf = NetcdfFile.openInMemory("in-mem file", buf);

However, in NetCDF 4.3.19, this now fails because opening the "file" seems to force the creation of an index on disk, and something goes haywire in this process. See stack trace below.

Any way to disable index creation, or a better strategy to use?

My use case is one where I'm handed the bytes of a Grib file, which doesn't necessarily exist anywhere on disk.

Thanks.

java.lang.StringIndexOutOfBoundsException: String index out of range: -37
    at java.lang.String.substring(String.java:1937)
    at java.lang.String.substring(String.java:1904)
    at ucar.nc2.grib.GribCollectionBuilder.makeFiles(GribCollectionBuilder.java:39)
    at ucar.nc2.grib.grib2.Grib2CollectionBuilder.createIndex(Grib2CollectionBuilder.java:610)
    at ucar.nc2.grib.grib2.Grib2CollectionBuilder.createIndex(Grib2CollectionBuilder.java:435)
    at ucar.nc2.grib.grib2.Grib2CollectionBuilder.readOrCreateIndex(Grib2CollectionBuilder.java:148)
    at ucar.nc2.grib.grib2.Grib2CollectionBuilder.readOrCreateIndexFromSingleFile(Grib2CollectionBuilder.java:76)
    at ucar.nc2.grib.GribIndex.makeGribCollectionFromSingleFile(GribIndex.java:122)
    at ucar.nc2.grib.grib2.Grib2Iosp.open(Grib2Iosp.java:311)
    at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1521)
    at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
    at ucar.nc2.NetcdfFile.openInMemory(NetcdfFile.java:719)
    at com.windlogics.dmf.weather.HadoopGribCrackerTest.testFileOpening(HadoopGribCrackerTest.java:37)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at junit.framework.TestCase.runTest(TestCase.java:154)
    at junit.framework.TestCase.runBare(TestCase.java:127)
    at junit.framework.TestResult$1.protect(TestResult.java:106)
    at junit.framework.TestResult.runProtected(TestResult.java:124)
    at junit.framework.TestResult.run(TestResult.java:109)
    at junit.framework.TestCase.run(TestCase.java:118)
    at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

[kenahoo]

ping

[JohnLCaron]

Hi Ken:

Thanks for the ping, I didnt see this, not sure why.

Ill have to think about this use case, I hadnt considered it. How big is the GRIB file?

John

[kenahoo]

They can get pretty big. I'm doing this in a Hadoop context, so I want to keep my files big (see https://blog.cloudera.com/blog/2009/02/the-small-files-problem/). Right now they're typically about 1 or 2 GB. Currently I read the entire thing at once, but on my to-do list is to read smaller chunks at a time.

[JohnLCaron]

Im not sure why 4.2 worked, as we have always written indexes for GRIB.

The problem is that theres a lot of processing that one needs to do to convert a GRIB file to a netCDF file. We write both a .gbx9 index and an .ncx index.

You might be able to use a lower level interface. What exactly are you trying to do, eg what do you do once you have a NetcdfFile ?

[kenahoo]

I'm actually just using the NetCDF libraries to read data from a GRIB file, I don't actually need to convert from one file format to another. Once I've extracted the relevant data I'm done using the file.

[JohnLCaron]

Sorry, i didnt mean write to a netcdf file, i meant create a NetcdfFile object. Do you read all of the data, or are you using netcdf index subsetting to extract parts of it? Do you reuse the GRIB files or is it one-time use?

[JohnLCaron]

You can just scan the grib file and create grib records in memory, without creating an index

See

https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/reference/formats/GribFiles.html

at the end "lower level interface to GRIB".