Add support for multiple, grouped coordinate systems in the extended model

[ecs-abdulgkhan]

Below is the text of an email conversation that led to this issue. You can contact me for the sample NetCDF4/HDF5 file.

Thanks!

Greetings Abdul,

Unlike netCDF3 or netCDF4-classic, there are not any standards like the
Climate and Forecast (CF) standards for the netCDF4-extended model
(which uses groups). Because of that, we do not have a way to build
coordinate systems in a standard way. It's probably possible to write a
coordinate system builder for your specific files though. See the following
link for more info:

https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/tutorial/CoordSysBuilder.html

Note that there is nothing in netCDF Java that necessarily excludes
multiple coordinate systems in a dataset (for example, we routinely
find these in GRIB datasets).

We have an active proposal (just recently awarded) to work on CF 2.0,
in which we hope to address the the netCDF4 extended model in a
standard way, so this issue will likely be addressed there.

Would you be willing to submit an issue to our github issue tracker?

https://github.com/unidata/thredds/issues

Thanks!

Sean

Hello,

I was wondering if you were aware of the issue with the NetCDF-Java library not working with multiple coordinates systems in a given dataset. If so, do you have an estimated timeframe in which this issue will be addressed.

Thanks,

Abdul

On Feb 19, 2016, at 6:05 PM, Schmunk, Robert B. (GISS-611.0)[TRINNOVIM, LLC] [email protected] wrote:

Abdul,

This has been reported before, most recently I think by someone else at JPL.

Generally speaking, the underlying problem probably is that Unidata's netCDF-Java library, which Panoply uses to parse the dataset metadata, does not work well (and often not at all) with multiple coordinate systems in a single dataset.

There have been cases where I have been able to add code to Panoply to sidestep or to augment what the library reports, although that additional code can sometimes run into trouble with HDF datasets using groups. I will have to take a look at your sample dataset to see if this applies there and what if anything I can do to better interpret the metadata.

rbs

On Feb 19, 2016, at 16:49, Abdul Khan [email protected] wrote:

Dr. Schmunk,

After following your advice to upgrade our version of Panoply to fix the “negative seek offset” issue, we were able to perform quite a bit of testing against the SMAP data we are processing for NSIDC using Panoply.

The only other issue we have encountered with our reformatted outputs being rendered properly in Panoply is with the geolocation of datasets in files with multiple groups. Although there are exceptions, we seem to see a pattern in which our NetCDF-4/HDF5 output files with multiple groups will only geolocate the datasets in the first group.

I have attached a screen shot and a NetCDF-4/HDF5 file that can be used to reproduce the issue. If you have any insight into what may be causing this it would be very much appreciated. It could very well be an issue with the way we are generating the NetCDF files, but from my understanding Panoply should just see the “coordinates” attribute and follow the paths in that string to geolocate the dataset.

Thanks,

Abdul

[JohnLCaron]

can we get the sample file?

[ecs-abdulgkhan]

John,

Attached is the file that we have been working with. Let me know if you need another example file. I think this may also be an issue with HDF5, but would have to find an example for you.

Note that the attached file does not contain the CF 2.0 convention metadata, so if this is the direction you want to go let me know and we will add a ticket to our bucket to get that done.

Thanks,

Abdul

On Feb 25, 2016, at 7:21 PM, John Caron [email protected] wrote:

can we get the sample file?

—
Reply to this email directly or view it on GitHub #441 (comment).

[JohnLCaron]

Im not seeing the attachment

[ecs-abdulgkhan]

I think it is able to get through my firewall, but maybe github is filtering it.

Do you have another email I can forward it to?

-abdul

On Feb 26, 2016, at 10:37 AM, John Caron [email protected] wrote:

Im not seeing the attachment

—
Reply to this email directly or view it on GitHub #441 (comment).

[JohnLCaron]

can you just drag and drop to the github page?
how big is file?

[ecs-abdulgkhan]

Looks like github does not support .nc files, but they do support .zips. Thanks!
processed_SMAP_L1C_TB_03969_A_20151029T221445_R12170_001.h5.nc.zip

[cwardgar]

Ok, I just took a look at this. For those interested, the CDL is:

netcdf /Users/cwardgar/Downloads/processed_SMAP_L1C_TB_03969_A_20151029T221445_R12170_001.h5.nc {

  group: South_Polar_Projection {
    dimensions:
      lat = 500;
      lon = 500;
    variables:
      short cell_column(lon=500, lat=500);
        :_FillValue = -2US; // short
        :long_name = "EASE grid column index of cell on world grid in longitude direction.";
        :coordinates = "/South_Polar_Projection/cell_lat /South_Polar_Projection/cell_lon";
        :valid_min = 0US; // short
        :valid_max = 963US; // short
        :_Unsigned = "true";
        :_ChunkSizes = 500, 22; // int

      float cell_lat(lon=500, lat=500);
        :_FillValue = -9999.0f; // float
        :long_name = "Latitude of the center of the Earth based grid cell";
        :units = "degrees_north";
        :_ChunkSizes = 500, 22; // int

      float cell_lon(lon=500, lat=500);
        :_FillValue = -9999.0f; // float
        :long_name = "Longitude of the center of the Earth based grid cell";
        :units = "degrees_east";
        :_ChunkSizes = 500, 22; // int

      short cell_row(lon=500, lat=500);
        :_FillValue = -2US; // short
        :long_name = "EASE grid row index of cell on world grid in latitude direction.";
        :coordinates = "/South_Polar_Projection/cell_lat /South_Polar_Projection/cell_lon";
        :valid_min = 0US; // short
        :valid_max = 405US; // short
        :_Unsigned = "true";
        :_ChunkSizes = 500, 22; // int

  }

  group: Global_Projection {
    dimensions:
      lat = 406;
      lon = 964;
    variables:
      float cell_lon(lon=964, lat=406);
        :_FillValue = -9999.0f; // float
        :long_name = "Longitude of the center of the Earth based grid cell";
        :units = "degrees_east";
        :_ChunkSizes = 406, 31; // int

      short cell_row(lon=964, lat=406);
        :_FillValue = -2US; // short
        :long_name = "EASE grid row index of cell on world grid in latitude direction.";
        :coordinates = "/Global_Projection/cell_lat /Global_Projection/cell_lon";
        :valid_min = 0US; // short
        :valid_max = 405US; // short
        :_Unsigned = "true";
        :_ChunkSizes = 406, 31; // int

      short cell_column(lon=964, lat=406);
        :_FillValue = -2US; // short
        :long_name = "EASE grid column index of cell on world grid in longitude direction.";
        :coordinates = "/Global_Projection/cell_lat /Global_Projection/cell_lon";
        :valid_min = 0US; // short
        :valid_max = 963US; // short
        :_Unsigned = "true";
        :_ChunkSizes = 406, 31; // int

      float cell_lat(lon=964, lat=406);
        :_FillValue = -9999.0f; // float
        :long_name = "Latitude of the center of the Earth based grid cell";
        :units = "degrees_north";
        :_ChunkSizes = 406, 31; // int

  }

  group: North_Polar_Projection {
    dimensions:
      lat = 500;
      lon = 500;
    variables:
      short cell_row(lon=500, lat=500);
        :_FillValue = -2US; // short
        :long_name = "EASE grid row index of cell on world grid in latitude direction.";
        :coordinates = "/North_Polar_Projection/cell_lat /North_Polar_Projection/cell_lon";
        :valid_min = 0US; // short
        :valid_max = 405US; // short
        :_Unsigned = "true";
        :_ChunkSizes = 500, 22; // int

      float cell_lon(lon=500, lat=500);
        :_FillValue = -9999.0f; // float
        :long_name = "Longitude of the center of the Earth based grid cell";
        :units = "degrees_east";
        :_ChunkSizes = 500, 22; // int

      short cell_column(lon=500, lat=500);
        :_FillValue = -2US; // short
        :long_name = "EASE grid column index of cell on world grid in longitude direction.";
        :coordinates = "/North_Polar_Projection/cell_lat /North_Polar_Projection/cell_lon";
        :valid_min = 0US; // short
        :valid_max = 963US; // short
        :_Unsigned = "true";
        :_ChunkSizes = 500, 22; // int

      float cell_lat(lon=500, lat=500);
        :_FillValue = -9999.0f; // float
        :long_name = "Latitude of the center of the Earth based grid cell";
        :units = "degrees_north";
        :_ChunkSizes = 500, 22; // int

  }
  // global attributes:
  :iso_19139_dataset_xml = <lots_of_xml>
}

As Sean stated, the issue is that we don't have any coordinate system convention for the netCDF4-extended model (which uses groups). As it stands, this dataset is being parsed by ucar.nc2.dataset.conv.DefaultConvention (because no other convention was specified). It processes the South_Polar_Projection group first and finds 2 coordinate axes: cell_lat and cell_lon. It is programmed to find at most one axis of each type (e.g. Lat, Lon, Time, etc), so the cell_lat and cell_lon variables belonging to the other groups are rejected as axis candidates (see DefaultConvention#L157).

We could relax the constraint that a dataset have at most one axis of each type, but that doesn't seem like the right thing to do for DefaultConvention. This dataset should really specify a non-default convention that property handles the grouped coordinate systems. We don't have such a thing right now, but we hope that CF 2.0 will address it. "Proper" resolution of this issue is on hold until then.

In the interim, there's nothing stopping you from implementing your own CoordSysBuilder, as described here.