ncWMS is not working for some dataset elements in thredds config catalogs #618
Comments
|
@lesserwhirls , just a note that the latest artifact |
|
So this does work on our test datasets that use the dataset element, so there is something more to this. I'll take a deeper look at your configs. |
lesserwhirls
changed the title
|
I just tested the latest TDS 5.0 artifact from Sep 21, and still godiva 3 is not working for our datasets. We are getting these errors: erviceExceptionReport xmlns="http:https://www.opengis.net/ogc" xmlns:xsi="http:https://www.w3.org/2001/XMLSchema-instance" version="1.3.0" xsi:schemaLocation="http:https://www.opengis.net/ogc http:https://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
<ServiceException>
The location file:/home/rsignell/data/models/delft3d/vs_trim2nc/00_dir.ncml doesn't refer to any existing files.
</ServiceException>
</ServiceExceptionReport> |
|
@lesserwhirls is there anything I could do here to help? Oh, wait, is there a workaround such that I can just eliminate the |
|
I only ask because most of the catalogs I've tested are written this way and are not currently working with ncWMS2 in TDS5.0. Yet many of these datasets would benefit strongly from the new features in ncWMS2, so we would really like to get this going! |
|
@rsignell-usgs - sorry this has taken so long to get to. Ill be digging into and working on this for the rest of the week until (hopefully) I can find a solution. I have some ideas, so maybe it won't take too crazy long. |
|
@lesserwhirls, any update here? Guessing either (a) ran out of time to test, or (b) hit roadblocks. FWIW, huge improvements for unstructured grid and all projected data in today's release of ncWMS2: |
|
Hey @rsignell-usgs - I thought I had replied to this already, but I'm not seeing it. My apologies. Working with the
About number 2 above, the data file was incorrectly named as |
|
@lesserwhirls , I'm trying to get this To be frank, this always confused me, and since my catalogs worked fine without it (at least until TDS 5.0), I never used it, except in So for the bathy catalog and etopo2 dataset you mention, my catalog entry looks like: <dataset name="ETOPO2 Version 2c (2 arc minute - Worldwide)" ID="bathy/etopo2_v2c.nc"
urlPath="bathy/etopo2_v2c.nc">
...
<netcdf xmlns="http:https://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
location="/usgs/data0/bathy/ETOPO2v2c_i2.nc">
<variable name="topo">
<attribute name="units" value="meters"/>
<attribute name="long_name" value="Topography"/>
</variable>
<attribute name="title" value="NGDC ETOPO2v2c Global DEM (2 min))"/>
</netcdf>
</dataset>
My understanding was that because I point to the full path name to the datafile explicitly inside the netcdf tags, the So if these catalogs no longer work, it's a backwards compatibility issue. I guess that's okay since you get to break the API in a major version, but it at all possible, it would be nice if these catalogs would still work. It's not just the 30 catalogs I have, but all the folks I've helped make catalogs also... BTW, I tried adding <datasetRoot path="bathy" location="/usgs/data0/bathy/" />to the top of the file but that didn't work. Perhaps you could point me toward the version you used? |
|
According to the docs for 5.0 (not official), you should not need to define a datasetRoot when you are using NcML to point to a particular data file, like you are doing: So, this is a bug that should be fixed before a release. Of course, bug or not, I'd say we would want this to work no matter what, as you have put a lot of effort into help others get things up and running (greatly appreciated, btw!). In terms of adding the datasetRoot, I also had to change the name of the file used in the URL path to use the name of the file on disk - from |
|
Ok, I finally found the underlying issue. Ultimately, it has to do with the handoff of datasets between netCDF-Java/TDS and edal-java when NcML is involved. @guygriffiths - there does not appear to be a way to cleanly use |
|
@lesserwhirls - Currently we use the location to create a NetcdfDataset object, from the NetcdfDatasetAggregator class. This works by either just opening the file from disk, or creating an in-memory NcML dataset to aggregate glob expressions. It also keeps a cache of the 20 most recently used datasets so as to avoid having to re-create them each time they are needed. The various grid dataset classes store the location and then use this class to obtain a NetcdfDataset when they need it (which may or may not retrieve it from the cache). I propose that I create a protected method in CdmDatasetFactory called something like getDatasetFromLocation, which I then delegate to NetcdfDatasetAggregator. You'd need to create a subclass of CdmGridDatasetFactory which overrides this method for cases where the dataset has been modified by NcML (or for all cases if you prefer). From what I recall of the TDS code it's probably the case that you'll create a new ThreddsGridDatasetFactory for each dataset, and getDatasetFromLocation will just ignore the location parameter and just return the appropriate NetcdfDataset object (which you've used in the constructor to ThreddsGridDatasetFactory). Does that sound like it would work? I was also going to suggest the following: I create a NetcdfDatasetAggregator.addDataset(String location, NetcdfDataset dataset) method, which will add a specific location -> NetcdfDataset object mapping to the cache. I would also make it mark the dataset as "active", which means that it will not be expired until it has been finished with. In ThreddsWmsCatalogue you'd have to insert the object into the cache each time you wanted to create it. But whilst writing it down, I came to think that has a lot more potential for weird bugs, and it feels a bit hacky. |
|
Either would work from my point of view, but I'm always in favor of reducing the number of places weird bugs can crop up (we have too many of those already laughs). How hard would it be for you to implement the first option? I don't want to burden you. |
|
Not hard at all. I did it earlier 😁 |
|
Is this in the develop branch? |
|
Oh awesome @guygriffiths! |
|
@lesserwhirls - I haven't had a chance to look into this specifically because I've been working on another project which uses EDAL. However, that project threw up the same bug! I've just release a version 1.3.1 containing a fix. |
|
Excellent - thank you for the update @guygriffiths! I'll update to 1.3.1 and let you know how it goes. |
|
Perfect - works well locally. I'll upgrade our dev server once I get to work and will report back, but looks like 1.3.1 did the trick! Thank you once again @guygriffiths! |
|
@lesserwhirls and @cwardgar , godiva3 still isn't working for me for netcdf files under a datasetScan. Here is godiva2 on thredds 4.6.10 accessing one such file: Here is godiva3 on thredds 5.0 accessing the same file: If I turn on developer tools, I can see that godiva3 is calling This is using the thredds 5.0 snapshot docker container from 5 days ago |
|
Hmm, okay, according to Guy the So some other problem. @lesserwhirls, can you try accessing a netcdf file in a |
|
Interesting. The WMS server is working: but godiva3 is bombing out on a null value: I have not been able to successfully debug https://gamone.whoi.edu/thredds/dodsC/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc |
|
Actually, now that I have actually updated to the |
|
Hi @rsignell-usgs - can you clear your cache directory and retry? |
|
Blew away the cache, but same problem: |
|
I should also note that for this dataset:
|
|
Ok, I see an error being generated in the ncWMS code: I'm digging into it to see what is happening, but it does not happen on other |
|
I'm so glad you can reproduce! The only things I saw in the logs were some H5 warnings... |
|
So it looks like things are going wrong in |
|
@guygriffiths - I am also seeing this kind of message in the logs: But I am getting this for other datasets as well (non SGRID), not just this one. Maybe the file is being prematurely released? |
|
Not sure what's happening, I haven't seen this one before. Any chance I could get a copy of the data? Doing a download from the dataset @rsignell-usgs linked to just give be a 403 error. |
|
@guygriffiths , I just tried the link and it worked fine for me: https://gamone.whoi.edu/thredds/fileServer/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc But if you don't want to download a 1.7GB file, I made a one-time-step version (170mb) you can download here: |
|
I've had a quick look and I think it's to do with the fact that some of the variables ( I'll have a look at fixing it at some point tomorrow, but for the moment you can get around it by removing the |
|
Excellent - thank you for your help @guygriffiths! |
|
The issue with SGRID has been fixed in the 1.4.0 release, along with a couple of other SGRID issues. Not sure about the dataset release one, I'm not getting that with a basic ncWMS2 installation. It's also not so important - it won't cause any actual problems in itself. |
|
Thank you @guygriffiths! |
|
Hi Sean, I've tried seeing whether Godiva would display different timesteps with the latest WAR snapshot here: using this NetCDF file: And it still shows data from the same timestep (the last one in the file), regardless of which date is selected in the "Time" menu. The catalina.out file does not show any errors related to this page, and the tomcat access logs don't show any non-200 responses. I also don't see any obvious javascript problems in the Chrome developer tools while browsing the page and changing dates. Any idea whether progress has been made yet on this issue? |
|
I was hoping it was something similar to this issue, but it looks like I was wrong. I'll dig into it and see how things look with the latest version. |
|
Greetings @bonnland! I just gave the dataset you mentioned above (tas.hist.CanESM2.CanRCM4.day.NAM-22.raw.nc) in the latest version of 5.0 and it appears to be working. When you have a chance, give this war file a spin: |
ncWMS is not working when a thredds config catalog is using the dataset element. The issue is that the the dataset urlPath is being directly passed into the call to open a dataset, and thus there is an error as the file does not exists.
This is happening using all versions of 5.0, including the latest published snapshot at this time:
http:https://artifacts.unidata.ucar.edu/content/repositories/unidata-snapshots/edu/ucar/tds/5.0.0-SNAPSHOT/tds-5.0.0-20160808.141712-14.war
This is a continuation of the disccusion here
Here is the stacktrace showing the error, as found in catalina.out:
It should be noted in the dataset element, the attribute
urlPathwas set tourlPath, andIDwas set toid.Also running into this same thing for the threddsIso services, as well as WCS.
The text was updated successfully, but these errors were encountered: