Lon, lat axes construction error in HdfEosOmiConvention

[rschmunk]

A Panoply user has noted that an OMI-Aura_L3 HDF-EOS dataset is plotted with all cells a half degree out of position both in lon and lat.

Doing some checking I find that Panoply reports the dataset is opened using the HdfEosOmiConvention class. Checking the source code for that convention, I see the makeLonCoordAxis and makeLatCoordAxis methods are constructing CoordinateAxis1D axes such that the first grid cell is centered on -180°E -90°S, whereas that should presumably be the lower left corner of the first cell.

So it looks to me like makeLatCoordAxis should instead say

v.setValues(n, -90.0 + 0.5 * incr, incr);

and makeLonCoordAxis should say

v.setValues(n, -180.0 + 0.5 * incr, incr);

Does my interpretation of this sound right? Unfortunately, the URL at top of HdfEosOmiConvention for more info is a dead link. Using Google I did find a PDF document that seems to pertain, and I note that it also suggests the possibility that an OMI-Aura dataset might not be full global, as HdfEosOmiConvention currently seems to expect.

[rschmunk]

Submitted pull request #1195

[lesserwhirls]

Thanks for the PR @msdsoftware! I don't know much about HdfEos, but it looks like you are saying that when geolocating a cell, the lat/lon associated with that cell should be defined as the center of the cell and not the lower left corner? That sounds reasonable to me. @ethanrd - thoughts?

[rschmunk]

@lesserwhirls, I don't know much about HDF-Eos either, but when plotting the data from the sample dataset, Panoply needs to be given the cell centers. I haven't poked about too much for documentation or other help, but I did find, e.g. , the example code in "Figure 5" of https://hdfeos.org/examples/matlab5.php effectively does the same as what my PR implements.

[lesserwhirls]

Having read more about HdfEos, I worry that the 0.5 offset isn't always the right one to use.

It looks like there is a a lot of information tucked away in the HDFEOS_INFORMATION group. For example, inside the StructMetadata.0 of the HDFEOS_INFORMATION group there is a PixelRegistration, which can have two values: HDFE_CENTER or HDFE_CORNER. Currently, I think the code assumes HDFE_CORNER, but in your case it's HDFE_CENTER, thus the 0.5 offset. This variable also has UpperLeftPointMtrs and LowerRightMtrs, which would give us the actual domain size.

Unfortunately, the variable StructMetadata.0 is one big string. In the case of a sample file I found:

char StructMetadata.0(32000);

 data:
  "GROUP=SwathStructure
END_GROUP=SwathStructure
GROUP=GridStructure
	GROUP=GRID_1
		GridName="OMI Column Amount O3"
		XDim=1440
		YDim=720
		UpperLeftPointMtrs=(-180000000.000000,90000000.000000)
		LowerRightMtrs=(180000000.000000,-90000000.000000)
		PixelRegistration=HE5_HDFE_CENTER
		GridOrigin=HE5_HDFE_GD_LL
		Projection=HE5_GCTP_GEO
		GROUP=Dimension
			OBJECT=Dimension_1
				DimensionName="XDim"
				Size=1440
			END_OBJECT=Dimension_1
			OBJECT=Dimension_2
				DimensionName="YDim"
				Size=720
			END_OBJECT=Dimension_2
		END_GROUP=Dimension
		GROUP=DataField
			OBJECT=DataField_1
				DataFieldName="SolarZenithAngle"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=5
			END_OBJECT=DataField_1
			OBJECT=DataField_2
				DataFieldName="ViewingZenithAngle"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=5
			END_OBJECT=DataField_2
			OBJECT=DataField_3
				DataFieldName="ColumnAmountO3"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=5
			END_OBJECT=DataField_3
			OBJECT=DataField_4
				DataFieldName="RadiativeCloudFraction"
				DataType=H5T_NATIVE_FLOAT
				DimList=("YDim","XDim")
				MaxdimList=("YDim","XDim")
				CompressionType=HE5_HDFE_COMP_DEFLATE
				DeflateLevel=5
			END_OBJECT=DataField_4
		END_GROUP=DataField
		GROUP=MergedFields
		END_GROUP=MergedFields
	END_GROUP=GRID_1
END_GROUP=GridStructure
GROUP=PointStructure
END_GROUP=PointStructure
GROUP=ZaStructure
END_GROUP=ZaStructure
END
"

Perhaps we should keep HDFE_CORNER as the default, but look for the string HDFE_CENTER in the value of the StructMetata.0 variable and override as needed? Trying to back out the actual domain size will be a bit more work, but at least this would get things going. But string matching does not quite feel right, though. Some of that metadata is available in the GRIDS group, for example:

<netcdf xmlns="http:https://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="C:/Users/lesserwhirls/Desktop/OMI-Aura_L3-OMTO3e_2017m0105_v003-2017m0203t091906.he5">
  <group name="HDFEOS">
    <!--SNIP-->
    <group name="GRIDS">
      <!--SNIP-->
      <group name="OMI_Column_Amount_O3">
        <!--SNIP-->
        <attribute name="GCTPProjectionCode" type="int" value="0" />
        <attribute name="Projection" value="Geographic" />
        <attribute name="GridOrigin" value="Center" />
        <attribute name="GridSpacing" value="(0.25,0.25)" />
        <attribute name="GridSpacingUnit" value="deg" />
        <attribute name="GridSpan" value="(-180,180,-90,90)" />
        <attribute name="GridSpanUnit" value="deg" />
        <attribute name="NumberOfLongitudesInGrid" type="int" value="1440" />
        <attribute name="NumberOfLatitudesInGrid" type="int" value="720" />
      </group>
    </group>
  </group>

In this case, the attribute GridOrigin appears to be the same as PixelRegistration (as opposed to the GridOrigin parameter in the HDFEOS_INFORMATION group). This also has GridSpan, which would be easier to use.

[rschmunk]

@lesserwhirls, The convention class in question, HdfEosOmiConvention, seems to be somewhat more constrained in its specification and I believe that the GRID group's GridOrigin attribute will only have value "Center". Also it appears that for OMI, the GridSpan is always (-180,180,-90,90).. I'm basing those statements on what I see in http:https://views.cira.colostate.edu/data/Documents/Guidelines/HDFEOS_Aura_File_Format_Guidelines.pdf

[lesserwhirls]

That's how I read that document as well. Thanks for digging around and making the PR! I'll go ahead and merge it in. I might make another PR with some notes, updating the url at the top of the file and referencing the pdf above, but nothing you need to bother with unless you want to. I appreciate your efforts!

[rschmunk]

BTW, @lesserwhirls, what was your procedure for extracting the content of StructMetadata.0? ToolsUI doesn't seem to show it, nor does HDFView. Only tool I had on hand that allowed for extracting that info was HexFiend, which doesn't present helpfully formatted info.

[pjtleonard]

Most of the original OMI L2G and L3 gridded products included errors in the HDF-EOS grid-level metadata.

All of these products are now either fixed, or are in the process of being fixed.

Here are the corrected values of the grid-level metadata:
UpperLeftPointMtrs = (-180000000.0,-90000000.0)
LowerRightPointMtrs = (180000000.0,90000000.0)

Also, the Grid_Origin attribute has been removed.

Finally,
PixelRegistration = HE5_HDFE_CENTER
and
Projection = HE5_GCTP_GEO
have always been correct, and so these have not been changed.

[lesserwhirls]

Thanks for the info @pjtleonard! It looks like @msdsoftware fix is all we need.

@msdsoftware - I am able to view the content of StructMetadata.0 by going through the Viewer tab of ToolsUI, right-clicking the variable, and selecting NCdump Data.

[rschmunk]

Ach, @lesserwhirls, control-click! (I'm a Mac user) I was trying double-click.