Transition from 4.5 to 4.6: GRIB collection not updating

[dopplershift]

From e-support NLB-782132

Package Version: 4.6.3
Operating System: windows server 2008
Hardware:
Description of problem: Hi there,

I am upgrading TDS from 4.5 to 4.6
In 4.5, everything works fine and the updating works fine but

for example, I am dealing with ECMWF C1D data, which
after being transformed to simple-coding GRIB1 from complex-coding GRIB1+GRIB2,
I use following xmls try to make collection:

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="NMC THREDDS Server Catalog" xmlns="http:https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http:https://www.w3.org/1999/xlink"  version="1.0.2" >

<service name="Default" serviceType="Compound" base="">
    <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/"/>
    <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/" />
</service>

<featureCollection name="ecmwf_model_c1d_surface" featureType="GRIB1" harvest="true" path="nwp/ECMWF/C1DS" >
    <metadata inherited="true">
        <serviceName>Default</serviceName>
        <documentation type="summary"> ECMWF Main High Resolution Model Products C1D Surface Variables </documentation>
        <documentation type="rights"> Users of these data files are expected to follow the CMA rules </documentation>
        <contributor role="data manager">Tang Jian and Dai Kan</contributor>
        <keyword>Numerical weather forecasting</keyword>
    </metadata>

    <collection spec="D:/THREDDS_NEW/nwp/ECMWF/C1D_surface/**/.*grib1$"
                timePartition="directory"
                name="ecmwf_deterministic_surface"
                dateFormatMark="#C1D_surface/#yyyyMMddHH" />

    <update startup="never" trigger="allow"/>
    <tdm rewrite="test" rescan="0 0/5 * * * ? *" trigger="allow"/>

    <gribConfig datasetTypes="TwoD Best Latest Files" >
      <filesSort>
        <lexigraphicByName increasing="false" />
      </filesSort>
    </gribConfig>
</featureCollection>
</catalog>

tomcat is upgraded to 7.0.63 as told to upgrade greater than 7.0.50
tdm is 4.6 also
use this script to start
"C:/Program Files/Java/jre7/bin/java" -Xmx4g -Dtds.content.root.path="C:/Program Files/Apache Software Foundation/Tomcat 7.0/content" -jar tdm-4.6.jar -tds "http:https://thredds.cma.gov.cn/" -cred -log DEBUG

everything in log looks fine

  spec='D:/THREDDS_NEW/nwp/ECMWF/C1D_surface/**/.*grib1$'
  dateFormatMark ='#C1D_surface/#yyyyMMddHH'
  timePartition =directory
  updateConfig =UpdateConfig{userDefined=true, recheckAfter='null', rescan='null', triggerOk=true, updateType=never}
  tdmConfig =UpdateConfig{userDefined=true, recheckAfter='null', rescan='0 0/5 * * * ? *', triggerOk=true, updateType=test}
  ProtoConfig{choice=Penultimate, change='null', param='null', outerNcml='null', cacheAll=true}
  hasInnerNcml =false
  GribConfig{datasets=[TwoD, Best, Files, Latest], useGenType=false, useTableVersion=false, intvMerge=true, useCenter=false, filesSortIncreasing=false}

2015-10-16T18:07:47.509 +0800 INFO  - scheduleJob startup scan force=test for 'ecmwf_deterministic_surface' at 2015-10-16T18:07:50.508+0800
2015-10-16T18:07:47.510 +0800 INFO  - scheduleJob recurring scan for 'ecmwf_deterministic_surface' cronExpr=0 0/5 * * * ? *
2015-10-16T18:08:15.781 +0800 INFO  - updateGribCollection ecmwf_deterministic_surface changed true took 1323 msecs
2015-10-16T18:08:15.814 +0800 INFO  - send trigger to http:https://thredds.cma.gov.cn/thredds/admin/collection/trigger?trigger=never&collection=ecmwf_deterministic_surface status = 200
2015-10-16T18:15:15.093 +0800 ERROR - gc2tp cant delete D:\THREDDS_NEW\nwp\ECMWF\C1D_surface\ecmwf_deterministic_surface.ncx3
2015-10-16T18:15:15.099 +0800 INFO  - updateGribCollection ecmwf_deterministic_surface changed true took 1033 msecs
2015-10-16T18:15:15.124 +0800 INFO  - send trigger to http:https://thredds.cma.gov.cn/thredds/admin/collection/trigger?trigger=never&collection=ecmwf_deterministic_surface status = 200

THE error LINE existed before, and it did not get into the way of updating collection.
But this time the collection only can be updated for the first time, and later when more file added into the collection, the time dimension of collection just won't change.

In tdm.log
there is

2015-10-16T18:15:04.131 +0800 WARN  - ucar.coord.CoordinateTime2DUnionizer - CoordinateTime2DUnionizer duplicate runtimes from different partitions java.lang.Thread.getStackTrace(Unknown Source)
ucar.coord.CoordinateTime2DUnionizer.addAll(CoordinateTime2DUnionizer.java:47)
ucar.coord.CoordinateUnionizer.addCoords(CoordinateUnionizer.java:64)
ucar.nc2.grib.collection.GribPartitionBuilder.makeDataset2D(GribPartitionBuilder.java:327)
ucar.nc2.grib.collection.GribPartitionBuilder.createPartitionedIndex(GribPartitionBuilder.java:166)
ucar.nc2.grib.collection.GribCdmIndex.updatePartition(GribCdmIndex.java:370)
ucar.nc2.grib.collection.GribCdmIndex.updateFilePartition(GribCdmIndex.java:583)
ucar.nc2.grib.collection.GribCdmIndex.updateLeafCollection(GribCdmIndex.java:509)
ucar.nc2.grib.collection.GribCdmIndex.updateDirectoryCollectionRecurse(GribCdmIndex.java:463)
ucar.nc2.grib.collection.GribCdmIndex.updateGribCollection(GribCdmIndex.java:333)
thredds.tdm.Tdm$IndexTask.run(Tdm.java:339)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)

the website looks like this
Dataset {
Float32 lat[lat = 1441];
Float32 lon[lon = 2880];
Float64 reftime;
Float64 time[time = 52];
Float64 time1[time1 = 39];
TimeCoverage:

Start: 2015-10-16T00:00:00Z
End: 2015-10-25T18:00:00Z

it should end with 2015-10-26T00:00:00Z

also I try to send trigger manually, by typing the following into browser
https://thredds.cma.gov.cn:8443/thredds/admin/collection/trigger?collection=ecmwf_deterministic_surface&trigger=nocheck
it shows :

Collection ecmwf_deterministic_surface

TRIGGER SENT

seems SSL is working

and after I restart the tomcat, everything is fine
It seems to me that the TDM can send trigger, but tomcat does not respond to it.

[JohnLCaron]

is this fixed?

[sprite2200]

HI John
haven't got chance to test the later version, I will have it tested by this week on 4.6.5.
Thanks for your attention!
Jian

[sprite2200]

@JohnLCaron
Dear John
I have been testing this whole afternoon, and here are some results.

With TDS 4.6.5, it still not work in my case.
I tested it in Win7/Windows Server 2008, with tomcat 7 and Java 7.

You see, I try to use time Partition "directory", it works fine in 4.5.4.
the data structure is like
C1D_surface\2016040800\surface_YYYYMMDDHH.grib1
so TDS will aggregate every dir as one partition.

The problem is when the YYYYMMDDHH directory first generated, the TDM can send trigger to TDS, and the TDS will update.
But with more and more files comes in, gbx9 will be generated and ncx3 will be updated, but the TDS will not update anymore.

I thought it maybe I should use time Partition "File", So I try it out.
I use "cat" generate three file all named "2016040800.grib1", only difference is three files have different records. First one consist of only +000 forecast, second one have +000 and +003 forecast, the third one have +000 and +003 and +006.
And I manually put the first one in, then replace it, the rename the third one and put it in.
TDS will update for the first and third time, but no respond to the second time, which is only replace the file( which means the file name wont change).
So it seems that it has the same problem as the "Directory" way.

Please help me out here if you have any clue about whats going on.
Many thanks!

Best,
Jian