NcML Aggregations don't issue error when dimension size changes

[dopplershift]

Taken from esupport (EWU-865844)

'''
If TDS is asked to "joinExisting" aggregate 2 .nc files that are identical
(e.g., same dimension and variable names),
except that one has twice as many lat and lon values (2x the resolution)
and thus 4x as many data values, TDS will aggregate the files (as far as I
know, without throwing an error message) and present the dataset to users.

At least right now, you can see this behavior with this dataset
http:https://oceanwatch.pfeg.noaa.gov/thredds/dodsC/satellite/QS/ux10/14day.html
The .dds reflects the data files with the larger lat and lon dimension
sizes:

Dataset {
Float64 altitude[altitude = 1];
Float64 lat[lat = 1201];
Float64 lon[lon = 2881];
Float64 time[time = 2454];
Grid {
ARRAY:
Float32 QSux10[time = 2454][altitude = 1][lat = 1201][lon = 2881];
MAPS:
Float64 time[time = 2454];
Float64 altitude[altitude = 1];
Float64 lat[lat = 1201];
Float64 lon[lon = 2881];
} QSux10;
} satellite/QS/ux10/14day;

If you request data from the last time point
http:https://oceanwatch.pfeg.noaa.gov/thredds/dodsC/satellite/QS/ux10/14day.ascii?QSux10[2453][0:1:0][0:1200:1200][0:2880:2880]
it succeeds because the file for that time point has 1201 lat values and
2881 lon values, which is also what the .dds indicates.

If you request data from the first time point
http:https://oceanwatch.pfeg.noaa.gov/thredds/dodsC/satellite/QS/ux10/14day.ascii?QSux10[0][0:1:0][0:1200:1200][0:2880:2880]
it fails with the message
Error {
code = 500;
message = "NcSDArray InvalidRangeException=Illegal Range for dimension
2: last requested 1200 > max 600";
};

If you want 2 different-sized sample files from this dataset, let me know
and I can email them to you (or make them available to you some other way
if email fails).

*** Caveats ***

• I didn't see a note/warning about this behavior in

https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/ncml/Aggregation.html
If I missed it, I'm sorry.

• Is there a switch in the TDS setup that allows me to turn on/ turn off
  strict checking of files to be aggregated?
• Maybe TDS does output an error message to a log file when it does the
  aggregation. I didn't see one. (I confess I didn't look super hard.)
  In any case, TDS successfully creates the aggregation and displays it to
  users.
• Maybe you think this is appropriate behavior ("It's not a bug; it's a
  feature!").
  Perhaps there are use cases where it is useful for TDS to be able
  aggregated different-sized files.
  Fine. If so, please just tell me that this is a feature (not a bug), and
  I will accept that.

Thank you for looking into this.
'''

[dopplershift]

There's also the open question of whether FMRC can be used to perform the task of time aggregation of netcdf files.

[JohnLCaron]

NcML error checking, in general, sucks.

FMRC can be used if the file is recognized as a Grid.

On Thu, Sep 3, 2015 at 10:52 AM, Ryan May [email protected] wrote:

There's also the open question of whether FMRC can be used to perform the
task of time aggregation of netcdf files.

—
Reply to this email directly or view it on GitHub
#168 (comment).