Write variables with VLEN dimensions to NetCDF-4 #588
Comments
|
Also, I should point out that the helpful error message is only printed for 1-D VLEN variables. Something like: Is going to print a much less useful message: |
|
I believe I am running into this. When I go to write the file, from Java with an NetcdfFileWriter.write(variable, value) call, I get the ENOTAVAR (-49) error. Making the I have about 11,000 groups of ~30x60 matrixes, roughly 20 to 100 per group, about 500,000 total. It's currently not possible to write these to a NetCDF file from Java? Is there a work around? |
|
Hi @chuston - could you tell us which version of netCDF-java you are using, as well as netCDF-C? @DennisHeimbigner - can you take a look at this one? |
|
Thanks for the reply. Versions |
|
The artifacts on maven central are quite old (not managed by us). If you are using maven/gradle to build, try these instructions to pull v4.6.11 from our nexus server: https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/reference/BuildDependencies.html Quite a lot changed between v4.5 and v4.6 in terms of netCDF-4 support, so that might fix the issue. you might also give v5.0.0-beta6 a try as well, as there have been some tweaks there specific to VLEN support. |
|
Thanks. I followed the instructions at the referenced link. I am seeing the same results with version If I replace the variable length Dimension with a fixed length Dimension the problem goes away. If I move to 5.0.0-beta6, I see: I guess I don't know how to create an unshared Dimension. Just using the Dimension constructor leads to a null pointer except during |
|
It's probably nothing you are doing wrong @chuston - @DennisHeimbigner, any thoughts? |
|
Okay. I've successfully made my variable length dimension non-shared. (By using Nc4Iosp:2492 (createVariable) agrees the dimension is not shared. Now, with 5.0.0-beta6, I am back to err -49 EVARNOTFOUND: |
|
First, I need to understand exactly what you meta-data
So we have three dimensions:
I assume that "netcdf" is of type NetcdfFileWriter. So, somewhere, you create a variable that uses these three dimensions Something like this.
The order of the dimensions is important because I believe that Sometimes we see people using a variable length dimension when what |
|
Thanks for the reply. Unlimited vs Variable: My understanding is that UNLIMITED is to allow appending data in the future. VARIABLE is to accommodate irregularly shaped variables. I'm after the latter. I have about 11,000 groups of ~30x60 matrixes, roughly 20 to 100 per group, I'm attempting to write those as 3d matrixes. I'm doing this mainly because it appears I have to define all of the Variables before I write to them. I can easily see the total number of groups, but to see the number of matrixes in each group I have to read gigs of data and fetch extra info from a DB - very time consuming. Ideally, I'd be able to just add 1 new variable at a time and write to it. To work around both the inability to append new variables and the inability to write variable dimensions - I'm writing new code to generate all 500,000 matrixes, store them as serialized Java objects, then create a NetCDF from the serialized matrixes. (And wondering if a directory full of CSVs wouldn't be the better way to go.) Dimensions. I've tried both VARLEN as first dimension and as last dimension: Both ways produce EVARNOTFOUND (-49): |
|
ok, so unlimited won't work for you.
In any case, we still have to deal with this error:
|
|
Another thought: try creating one variable and then close the dataset without ALso, I am afraid we will need a minimal example showing this problem. |
|
The following unit test replicates both that pattern I'm using to write and the problem: I did try to append variables with alternating calls to |
|
Thanks, the test case should help a lot. |
|
The following quick and dirty example executes without err (no variable length dimensions) but indicated that the file was rewritten during There is a message sent to stdout saying |
|
Interesting. The underlying netcdf-4 code in the netcdf-c library does not require
|
|
So with respect to rewriting the file. The method Nc4Iosp.rewriteheader |
|
Ok, there are two problems.
We need a command decision from Sean about adding VLEN support. |
|
I took a closer look at Nc4Iosp.java and the situation is that has code |
I didn't even realize that we don't support this in
Nc4Iosp. Maybe we have this documented somewhere, but I only found the clue buried in the source. According to the code, we can't writeNC_OPAQUEeither, but we don't really have an analogue for that in the CDM.@DennisHeimbigner Do you plan to take this on?
The text was updated successfully, but these errors were encountered: