ESGF non-data file serving with datasetScan

[JohnLCaron]

ESGF wants to serve files with a datasetScan. The files should display instead of download.

Claim this used to work in 4.3? using httpService with ususal catalog.xml?dataset=id extra indirection, AFAIU. Does not work in 4.6/5.0.

Perhaps add another filter to datasetScan describing files that should be served as files rather than datasets. Then allow

/thredds/catalog/work/p86denv/IPSLCM6/DEVT/piControl/CM6PI.12/MONITORING/index.html

instead of

/thredds/catalog/work/p86denv/IPSLCM6/DEVT/piControl/CM6PI12/MONITORING/catalog.html?
dataset=DatasetScanWORK/p86denv/IPSLCM6/DEVT/piControl/CM6PI.12/MONITORING/index.html

[JohnLCaron]

Nicolas Carenton ([email protected]

The context is that users will browse manually their home dirs which
contain bulk data (.nc files but also any other possible extension) via
a datasetScan. So the ideal behaviour would be that a compound of
ds_services is applied to this datsetScan including whatever exists
(HTTPServer, opendap, WCS, WMS, ...) meaning it is not a problem if we
end up with innapropriate services for a certain file extension. But, we
want the HTTPServer service to display html, png, txt, ... files instead
of download them.

[JohnLCaron]

GET /thredds/fileServer/IPSLFS/ncarenton/test.html HTTP/1.1
Host: vesg.ipsl.upmc.fr
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: JSESSIONID=B66B105B044FA83FD6247B4B779F4969
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36

HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: close
Content-Disposition: attachment; filename="test.html"
Content-Length: 5
Content-Type: text/html;charset=UTF-8
Date: Tue, 26 Jan 2016 19:02:21 GMT

[DennisHeimbigner]

I find this confusing. Are they asking for a different format for catalogs so that the web browser
sees directories and files rather than the current catalog display format?

[JohnLCaron]

They want to use TDS as traditional web server that displays eg html instead of downloading it.

[DennisHeimbigner]

Ok, I see. Well some access protocols (e.g. opendap) provide html access. I suppose the most straightforward approach is to create a parallel to the current subset service that serves up results in some selected html formats (or xml plus css plus xslt). The problem is agreeing on the result format(s).

[JohnLCaron]

The problem is distinguishing between data files and non-data files in DatasetScan. Data files are correctly dealt with now. Non-data files could be served as a web server would, deciding on their mime-type and then letting the browser decide what to do with it. They should not be served though the various services, as that doesnt work.

[JohnLCaron]

use case: use the TDS of an ESGF node as a web server.

assume can distinguish dataset from non-dataset by suffix

[DennisHeimbigner]

Would it make more sense for tds to pretend to be an FTP server?

[JohnLCaron]

No, they want a web server.

On Thu, Feb 4, 2016 at 1:36 PM, DennisHeimbigner [email protected]
wrote:

Would it make more sense for tds to pretend to be an FTP server?

—
Reply to this email directly or view it on GitHub
#381 (comment).

[JohnLCaron]

Ok heres what I have in tdsGlobalConfig:

    <!-- for datasetScan blacklist -->
    <property name="isNotAThreddsDataset">
      <list>
        <value>.html</value>
        <value>.csv</value>
        <value>.txt</value>
        <value>.doc</value>
      </list>
    </property>

    <!--  for datasetScan whitelist -->
    <property name="isAThreddsDataset">
      <list>
        <value>.nc</value>
        <value>.nc4</value>
      </list>
    </property>

datasetScan checks suffixes against these by calling AllowedServices.isAThreddsDataset(). (if no matches, than true). if false, then put in direct link to file in HTML.

can evolve into fancier, but this should give basic functionality.

[cwardgar]

So is this issue finished or does it need more polish?