IFPSConvention

[rschmunk]

Anyone around familiar with the IFPS convention for NWS data?

An NWS user was trying to figure out why Panoply was displaying "virtual" time coordinates for one dataset, but not another. The difference turned out to be that the dataset with the missing variables did not include a DIM_0 dimension, apparently because it didn't need one.

The problem is that when NJ's IFPSConvention is queried as to whether it claims a dataset, one of the 4 conditions it tests is the presence of DIM_0.

So I'm wondering if removing that condition from IFPSConvention.isItMine() would be a good idea, or if it would cause other trouble, perhaps with datasets created using some other condition.

The class appears to have been contributed by someone at NWS, so perhaps asking here won't receive any especially knowledgeable answers.

[lesserwhirls]

I have one sample file in our test datasets directory that is 40 MB. I've zipped the file and attached it to this issue. Having looked at the metadata in the file, there's not really much to go off of, which is why the isMine() feels like such a long shot, so to say, at least to me. Even taken together, the four checks:

public static boolean isMine( NetcdfFile ncfile) {
    Variable v = ncfile.findVariable("latitude");

    return (null != ncfile.findDimension("DIM_0")) && (null != ncfile.findVariable("longitude"))
            && (null != v) && (null != ncfile.findAttValueIgnoreCase(v, "projectionType", null));
  }

do not inspire a lot of confidence in being able to identify the convention being used in the file. Of the four, the DIM_0 and projectionType checks seem to be critical. I wonder if DIM_0 check could be replace by a check for the attribute descriptiveName being present on the latitude variable? Maybe also gridPointLL? It's not great no matter what, and it's hard to tell since I only have one file to examine. The global metadata in my sample file are sadly lacking:

  :creationTime = 1110842162; // int
  :creationTimeString = "Mon Mar 14 23:16:02 2005";
  :fileFormatVersion = "20010816";
  :startProcTime = "19700101_0000";
  :endProcTime = "20371231_2359";

Maybe the fileFormatVersion with both startProcTime and endProcTime in combination with some of the above? I'm certain we can do better, but there isn't much to work with. How does your file look? Are the global metadata any better?

HUNGrids.zip

[rschmunk]

@lesserwhirls, Global metadata in the sample "good" and "bad" files I was provided are the same as yours. Including the same value for the fileFormatVersion even thought the files were just created yesterday.

Use of the projectionType attribute on the latitude variable does look key. It also appears on the longitude var in the files I was provided. But from what I was provided, there is a use case when DIM_0 might not be present.

But yeah, I have my concerns that there doesn't seem to be anything else in the file that waves its hand and says, "I could be an IFPS file."

Sample files I was provided are both 170 MB. Large, although not as bad as the 3.4 GB 😮 file someone else sent my way today.

[dopplershift]

I think calling it a "convention" is a bit of a stretch. Our own API docs say: "IFPS Convention Allows Local NWS forecast office generated forecast datasets to be brought into IDV." I can't find this data anywhere, so I'm guess they're some kind of an ad hoc dump out of another data store.

For future google searching, IPFS stands for: "interactive forecast preparation system".

[lesserwhirls]

The AWIPS Graphical Forecast Editor (GFE) produces these grids. Looks like ifpnetCDF.py will tell us quite a bit, especially this

    if argDict['krunch']:
        setattr(file, "fileFormatVersion", "20030117")
    else:
        setattr(file, "fileFormatVersion", "20010816")

I bet looking for a match on one of those two specific values of fileFormatVersion, plus the presence of a latitude variable with an attribute of descriptiveName, is "ok enough" for now, or will at least keep out false positives.

It looks like this comment before the getDims function provides a clue as to why DIM_0 is missing from your file:

### Special case, if dimension of zero, use a different name.

It'd be nice to know if Raython would add a global attribute to their file, something like:

Conventions = NWS-IFPS`

(the term "conventions" being used loosely), to allow for better identification of the file.

What do you think the chances are of that, @mjames-upc? Is that something we could push for with a reasonable chance of success? Would making a PR help (it's only one line)? Something tells me going for full CF compliance would be a non-starter for AWIPS, but adding a global attribute seems more than reasonable.

[lesserwhirls]

I think this is addressed for now (PR #1247) while options are being weighed for a better solution upstream when the products are generated.

[rschmunk]

Yes, @lesserwhirls, I think this will have to do. Even if NWS does sometime start adding more useful metadata, there are going to be plenty of old datasets floating around without it.