RDA diurnal averages

[JohnLCaron]

diurnal averages not handled well.

[JohnLCaron]

ds628.5 JRA diurnal
time_unit = 132 (non-standard, coded in JmaTables)

<featureCollection 
name="JRA-55AMIP Monthly Diurnal Variance of 1.25 Degree 2-Dimensional Instantaneous Diagnostic Fields"
                                featureType="GRIB1"
                                harvest="true"
                                path="aggregations/JRA55/ds628.5/112">

 <collection name="ds628.5.fcst_surf125_var_diurnal"                             spec="/glade/p/rda/data/ds628.5/fcst_surf125_var_diurnal/**/fcst_surf125_var_diurnal\..*" />

file naming captures which interval is intended, eg

fcst_surf125_var_diurnal/1958/fcst_surf125_var_diurnal.AMIP.001_pres.00.195801_195812

has 12 records for each month of 1958. there are files for 00, 03, 06, ...21. I think this is the monthly stat (in this case variance) for all days with 00 = hour of day.

the time encoding is unexpected, eg for january:

hour     reftime                                 intv        result date
00        1957-12-31T18:00:00Z       6-726 == (1958-01-01T00:00:00Z,1958-01-31T00:00:00Z)
03        1958-01-01T00:00:00Z       3-723 == (1958-01-01T03:00:00Z,1958-01-31T03:00:00Z)
06        1958-01-01T00:00:00Z       6-726 == (1958-01-01T06:00:00Z,1958-01-31T06:00:00Z)
09        1958-01-01T06:00:00Z       3-723 == (1958-01-01T09:00:00Z,1958-01-31T09:00:00Z)
12        1958-01-01T06:00:00Z       6-726 == (1958-01-01T12:00:00Z,1958-01-31T12:00:00Z)
15        1958-01-01T12:00:00Z       3-723 == (1958-01-01T15:00:00Z,1958-01-31T15:00:00Z)
18        1958-01-01T12:00:00Z       6-726 == (1958-01-01T18:00:00Z,1958-01-31T18:00:00Z)
21        1958-01-01T18:00:00Z       3-723 == (1958-01-01T21:00:00Z,1958-01-31T21:00:00Z)

this is not particulatly bad, if you already know what it means.
however, because CDM groups records by reference time, we see 2 times for each reference, and these time intervals vary in strange ways across reference time, creating a "non-orthogonal" time2D which we dont currently handle.

as a first pass, better to not group by reference time, but just use the time interval. could get reference time as aux variable if it was important.

this may mean we need an option to "use time coordinate" instead of the current "group by reftime coordinate" option. TBD

[JohnLCaron]

From Dave S:

Issue 357:

I produced single parameter time series and set up the file naming conventions such that
fcst_surf125_var_diurnal/1958/fcst_surf125_var_diurnal.AMIP.001_pres.00.195801_195812
represents the monthly variance of the 00Z valid time of the 3 hourly diurnal forecast times.
Siimilarly,
fcst_surf125_var_diurnal/1958/fcst_surf125_var_diurnal.AMIP.001_pres.03.195801_195812
represents the monthly variance of the 03Z valid time of the 3 hourly diurnal forecast times.

For January, as encoded by JMA, one has

1957123118 + 6 -> 1958010100 in the fcst_surf125_var_diurnal.AMIP.001_pres.00.195801_195812 file
1958010100 + 3 -> 1958010103 in the fcst_surf125_var_diurnal.AMIP.001_pres.03.195801_195812 file
1958010100 + 6 -> 1958010106 in the fcst_surf125_var_diurnal.AMIP.001_pres.06.195801_195812 file
1958010106 + 3 -> 1958010109 in the fcst_surf125_var_diurnal.AMIP.001_pres.09.195801_195812 file
1958010106 + 6 -> 1958010112 in the fcst_surf125_var_diurnal.AMIP.001_pres.12.195801_195812 file
1958010112 + 3 -> 1958010115 in the fcst_surf125_var_diurnal.AMIP.001_pres.15.195801_195812 file
1958010112 + 6 -> 1958010118 in the fcst_surf125_var_diurnal.AMIP.001_pres.18.195801_195812 file
1958010118 + 3 -> 1958010121 in the fcst_surf125_var_diurnal.AMIP.001_pres.21.195801_195812 file

For February, as encoded by JMA, one has

1958013118 + 6 -> 1958020100 in the fcst_surf125_var_diurnal.AMIP.001_pres.00.195801_195812 file
1958020100 + 3 -> 1958020103 in the fcst_surf125_var_diurnal.AMIP.001_pres.03.195801_195812 file
1958020100 + 6 -> 1958020106 in the fcst_surf125_var_diurnal.AMIP.001_pres.06.195801_195812 file
1958020106 + 3 -> 1958020109 in the fcst_surf125_var_diurnal.AMIP.001_pres.09.195801_195812 file
1958020106 + 6 -> 1958020112 in the fcst_surf125_var_diurnal.AMIP.001_pres.12.195801_195812 file
1958020112 + 3 -> 1958020115 in the fcst_surf125_var_diurnal.AMIP.001_pres.15.195801_195812 file
1958020112 + 6 -> 1958020118 in the fcst_surf125_var_diurnal.AMIP.001_pres.18.195801_195812 file
1958020118 + 3 -> 1958020121 in the fcst_surf125_var_diurnal.AMIP.001_pres.21.195801_195812 file

And so on.

For monthly means and variances of the diurnal cycle, in this case 3,6,9,12,15,18,21 hours, people are usually interest in only on or two of these hours depending on their particular research application.

[JohnLCaron]

1957123118 + 6 -> 1958010100 in the fcst_surf125_var_diurnal.AMIP.001_pres.00.195801_195812 file
1958010100 + 3 -> 1958010103 in the fcst_surf125_var_diurnal.AMIP.001_pres.03.195801_195812 file
1958010100 + 6 -> 1958010106 in the fcst_surf125_var_diurnal.AMIP.001_pres.06.195801_195812 file
1958010106 + 3 -> 1958010109 in the fcst_surf125_var_diurnal.AMIP.001_pres.09.195801_195812 file

1. probably obvious to you, but why not use

1958010100 + 0 -> 1958010100 in the fcst_surf125_var_diurnal.AMIP.001_pres.00.195801_195812 file
1958010100 + 3 -> 1958010103 in the fcst_surf125_var_diurnal.AMIP.001_pres.03.195801_195812 file
1958010106 + 0 -> 1958010106 in the fcst_surf125_var_diurnal.AMIP.001_pres.06.195801_195812 file
1958010106 + 3 -> 1958010109 in the fcst_surf125_var_diurnal.AMIP.001_pres.09.195801_195812 file

answer:

JMA initializes a forecast cycle every 6 hours.
For each forecast cycle the forecast times are 3, 6, 9, 12, 15, 18, 21, etc.
There is no "0".

2. my real question is, do people want to know which reftime was used?

[JohnLCaron]

The "Collectiion status" shows missing records, but the MRUTX deals correctly with this. Should fix that diagnostic, since missing vertical records may get ignored. eg:

ds628.5.fcst_column125_var_diurnal (JRA-55AMIP Monthly Diurnal Variance of 1.25 Degree Total Column Forecast Fields)
 Group LatLon_145X288 (Center 0.6250S 180.0000E) total nrecords=42240, ndups=768 (0.018182), nmiss=10560 (0.250000)

ds628.5.fcst_land125_diurnal (JRA-55AMIP Monthly Diurnal Mean 1.25 Degree Land Surface Forecast Fields)
 Group LatLon_145X288 (Center 0.6250S 180.0000E) total nrecords=68640, ndups=0 (0.000000), nmiss=17160 (0.250000)

ds628.5.fcst_land125_var_diurnal (JRA-55AMIP Monthly Diurnal Variance of 1.25 Degree Land Surface Forecast Fields)
 Group LatLon_145X288 (Center 0.6250S 180.0000E) total nrecords=68640, ndups=0 (0.000000), nmiss=17160 (0.250000)

ds628.5.fcst_surf125_var_diurnal (JRA-55AMIP Monthly Diurnal Variance of 1.25 Degree 2-Dimensional Instantaneous Diagnostic Fields)
 Group LatLon_145X288 (Center 0.6250S 180.0000E) total nrecords=73920, ndups=0 (0.000000), nmiss=18480 (0.250000)

note that percent (.25) is constant.