| layout | title |
|---|---|
default |
Projects |
The ESGF P2P Node is the central project. It is an amalgam of several sub-projects and external software components and libraries to create a single Data Node artifact capable of searching, sharing and posting large amounts of metadata.
- Build, install, configure and maintain the ESGF Node.
- Central component for managing constituent components and overall inter and intra node activity.
- The node manager provides the framework for the coordination and management of the various components that comprise the ESGF Node.
- coordination, monitoring, metrics, notification, routing, discovery, etc
- The shell provides an interactive commandline (REPL) environment from which the user can manipulate the node and take advantage of node features.
- Use the esgf shell (esgf-sh) as a command line alternative interface into all the features of the ESGF Node. Write your own custom commands in Java, Clojure, Scala, Groovy and/or BeanShell. The shell provides familiar shell features and full set of commands to let you take advantage of the simple power of the command line.
- The esgf shell provides a simple shell and command line interface to the node's services. Users can use the shell to perform any of the tasks that may be performed through the web user interface. To help end users leverage the shell to it's maximum the shell provides the ability for end-users to develop custom shell commands. Commands can be written in Java, Closure, Scala, Jython, Groovy, or BeanShell. Through custom commands end-users have a direct mechanism for extending the node features and creating new ones!
- Power, simplicity, flexibility, extensibility, etc.
- Federation level monitoring tool. The dashboard and the node manager enjoy a symbiotic relationship as they both participate in federation level activities. Different from the node manager's role the dashboard monitors nodes once they are discovered and join the federation. The dashboard maintains historical information, whereas the node manager is primarily interested in keeping federation data up to date and in sync. The dashboard has a 'back-end' written and a quite attractive :-) front-end displaying not only node participation but node location as well!
- Federation level desktop
- The ESGF Desktop extends the functionalities provided by the data node adding an intuitive interface to back-end esgf node "monitoring" statistics and information - using the desktop metaphor. It extends the data node by providing a back-end monitoring system and a front-end user interface for monitoring purposes.
- Harvests, publishes, replicates and maintains meta data.
- Primary feature component for performing meta data harvesting, publishing and maintenance. Features pluggable handler design architecture for multi data format support.
- The identity provider in the ESGF SSO federation as well as account management and access control system. Also including a web-based interface for site administrators.
- Identity Provider and User Manager
- Where users are managed and identities kept.
- identity, security, naming, access, people
- Identity Provider and User Manager
- Where users are managed and identities kept.
- identity, security, naming, access, people
- The OpenID Relying Party - relays end-users to their home organization's IDP (identity provider) for secure login and connection to local resources
- OpenID relying party
- Provides the federation of nodes the ability to have single sign-on, with OpenID support.
- Single sign-on
- Security infrastructure for nodes and federation.
- Security infrastructure protecting data integrity and access on each node and across the federation.
- Search mechanism that supports search across entire corpus of ever-changing published metadata.
- The ESGF search module contains the next generation search functionality for
the Earth System Grid Federation, built upon the Apache Solr search engine. The
package contains functionality for:
- Publishing and unpublishing search metadata records into and from a Solr server. Metadata records are generated by harvesting a remote metadata source (a hierarchy of THREDDS catalogs, a OAI repository, a CAS metadata catalog).
- Searching the Solr engine content via a free text, faceted search or a geo-spatial and temporal search.
- Set of libraries for handling the DRS taxonomy that model data adheres to.
- A system for connecting to and retrieving data from back-end deep storage archives.
- The Stager provides the means to transparently access files located at a remote system. Remote files are cached locally preserving the directory structure from the remote structure.
- Implementation
- The current implementation uses a Java web Filter for intercepting the file
requests targeted at servlets running locally. These calls are handled
differently depending on the user-agent attribute from the http request
header.
- If the request was started from within a browser, the user is redirected to a "waiting page" from where it will be automatically redirected to the original servlet when the file is ready.
- In case the request was started by a tool, the request will just block until the file is available.
- The file retrieval will be triggered only once and in cache the file is already in the cache the filter will return immediately allowing the call to proceed normally without any delay. This call is logged to optimize the cache-cleaning mechanism.
- The current implementation uses a Java web Filter for intercepting the file
requests targeted at servlets running locally. These calls are handled
differently depending on the user-agent attribute from the http request
header.
- The web front-end to the federation. Interface to fasceted search and other services.
- Download published datasets and stay up-to-date.
- Fetching and keeping updated with ESGF Published data.
- Fast, Secure and Easy
- Stay Tuned... - Work in progress
- The ESGF Fetcher leverages the ESGF security mechanisms (esgf-security) and provides fast and secure HTTP file downloads. It also maintains and updates downloaded files with its built-in "broadcatching" capabilities. The fetcher has a secured RESTful interface as its control channel and provides a web and esgf shell (esgf-sh) command line interface. The Fetcher is a stand alone service that may be run on any host without any other dependencies.
- Algorithms and tools to help you wrangle your data.
- The Premiere Toolbox of computation algorithms
- Big Data -> Big Computation -> Big Analytics... No Problem
- Stay Tuned... - Work in progress
- The ESGF Toolbox is a collection of algorithms written in Java, Scala, Clojure and Python that leverages ESGF Compute Tools' "Big Data" infrastructure to perform computation, analytics and querying over ESGF datasets.