PCMDI · lee1043 · Nov 4, 2020 · Nov 4, 2020 · Nov 4, 2020 · Nov 4, 2020
diff --git a/docs/.nojekyll b/docs/.nojekyll
diff --git a/doc/Diurnal Cycle Diagram.pdf → docs/Diurnal Cycle Diagram.pdf b/doc/Diurnal Cycle Diagram.pdf → docs/Diurnal Cycle Diagram.pdf
diff --git a/doc/Diurnal Cycle Diagram.pptx → docs/Diurnal Cycle Diagram.pptx b/doc/Diurnal Cycle Diagram.pptx → docs/Diurnal Cycle Diagram.pptx
diff --git a/doc/HOWTO_devel_package.md → docs/HOWTO_devel_package.md b/doc/HOWTO_devel_package.md → docs/HOWTO_devel_package.md
diff --git a/docs/Makefile b/docs/Makefile
@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/docs/_sources/index.rst.txt b/docs/_sources/index.rst.txt
@@ -0,0 +1,50 @@
+.. pcmdi_metrics documentation master file, created by
+   sphinx-quickstart on Wed Nov  4 13:15:37 2020.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+***************************
+PCMDI metrics package (PMP)
+***************************
+
+The PCMDI metrics package is used to provide "quick-look" objective comparisons of Earth System Models (ESMs) with one another and available observations.  Results are produced in the context of all model simulations contributed to CMIP6 and earlier CMIP phases.  Among other purposes, this enables modeling groups to evaluate changes during the development cycle in the context of the structural error distribution of the multi-model ensemble. Currently, the comparisons emphasize metrics of large- to global-scale annual cycle and both tropcial and extra-tropical modes of variability. Ongoing work in v1.x development branches include established statistics for ENSO, MJO, regional monsoons, and high frequency characteristics of simulated precipitation. 
+
+PCMDI uses the PMP to produce `quick-look simulation summaries across generations of CMIP <https://cmec.llnl.gov/results/physical.html>`_
+
+The metrics package consists of four parts: 1) Analysis software, 2) an observationally-based database of global (or near global, land or ocean) annual cycle climatologies, 3) a database of performance metrics computed for CMIP models and 4) package documentation (in preparation).
+
+The package expects model data to be `CF-compliant <https://cfconventions.org/>`_. To successfully use the package some input data "conditioning" may be required. We provide several demo scripts within the package.
+
+Users of the current release (v1.2) will need to contact the PMP developers ([email protected]) to obtain supporting datasets and get started using the package.
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Contents:
+
+   self
+   install
+   install-using-anaconda
+   using-the-package
+   ipsl-corner
+   pmp-using-cdp-guide
+   pmpparser
+
+
+GETTING STARTED
+===============
+Installation requirements and instructions are available on the :ref:`install` page
+
+An overview for using the package and template scripts are detailed on the :ref:`using-the-package` page
+
+Some installation support for CMIP participating modeling groups is available: [email protected]
+
+PMP versions
+============
+
+v1.2 - Tied to CDAT 8.0.  Now includes extensive regression testing.  New metrics: Diurnal cycle and intermittency of precipitation, Sperber and Wang Monsoon metrics
+
+v1.1.2 - Now managed through Anaconda, and tied to UV-CDAT 2.10.  Weights on bias statistic added.   Extensive provenance information incorporated into json files.
+
+v1.1 - First public release, emphasizing climatological statistics, with development branches for ENSO and regional monsoon precipitation indices
+
+v1.0 - Prototype version of the PMP
diff --git a/docs/_sources/install-using-anaconda.rst.txt b/docs/_sources/install-using-anaconda.rst.txt
@@ -0,0 +1,39 @@
+.. _install-anaconda:
+
+**********************
+Install using Anaconda
+**********************
+
+We offer an installation for anaconda users under linux-64 or osx-64.
+Support for Windows is not available yet.
+
+https://anaconda.org/PCMDI/pcmdi_metrics/files
+
+All Platforms System Requirements
+=================================
+  * Install the `Anaconda for Python 3.7 <https://www.anaconda.com/products/individual#Downloads>`_ package (we recommend installing this for each user)
+  * Make sure anaconda is in your PATH (assuming anaconda is installed in ${HOME}/anaconda
+      * ``export PATH=${HOME}/anaconda/bin:${PATH}`` # for [ba]sh
+      * ``setenv PATH ${HOME}/anaconda/bin:${PATH}`` # for [t]csh
+
+Make sure you have no environment variables set from an old UV-CDAT installation in your PATH/PYTHONPATH,LD_LIBRARY_PATH etc
+
+Bypassing firewalls
+===================
+If your institution has tight ssl certificate/security issues try:
+  * ``conda config --set ssl_verify False``
+  * ``binstar config --set ssl_verify False``
+
+Installing the PCMDI Metrics Package (PMP)
+==========================================
+Using the conda package manager, you can install the PCMDI Metrics package from the PCMDI conda channel, and from an environment containing [CDAT](https://cdat.llnl.gov/).
+  * ``source activate [YOUR_CDAT_ENABLED_CONDA_ENVIRONMENT]`` (See `CDAT Install <https://github.com/CDAT/cdat/wiki/install>`_)
+  * ``conda install pcmdi_metrics -c cdat-forge``
+
+Getting the latest nightly of PMP and CDAT
+==========================================
+  * ``conda create -n [YOUR_ENV_NAME_HERE] -c cdat/label/nightly -c pcmdi/label/nightly -c conda-forge -c cdat pcmdi_metrics``
+  * ``source activate [YOUR_ENV_NAME_HERE]``
+
+
+To learn more about conda environments see: https://conda.pydata.org/docs/using/envs.html
diff --git a/docs/_sources/install.rst.txt b/docs/_sources/install.rst.txt
@@ -0,0 +1,14 @@
+.. _install:
+
+*******
+Install
+*******
+
+We offer two paths to install the PCMDI Metrics package. The recommended method is to install the Anaconda package and then install PCMDI Metrics using our conda channel. Alternatively, the older, more complex build method (from the source install method) is also supported, and used by developers contributing to the Github repository.   
+
+Anaconda installation (recommended)
+===================================
+
+* :ref:`install-anaconda` - Installation instructions for Anaconda users
+
+A primary advantage of the Anaconda install path is that it is user-centric. This allows users on super computing systems with limited permissions to fully configure their local PMP install
diff --git a/docs/_sources/ipsl-corner.rst.txt b/docs/_sources/ipsl-corner.rst.txt
@@ -0,0 +1,134 @@
+IPSL Corner
+***********
+
+Summary
+=======
+We want to develop what we have called "the Metrics Garden": this is an end-to-end facility to produce metrics (this part will be largely be done thanks to the PCMDI-MP, controlled by libIGCM, i.e. the library that controls the production of our simulations), store them on a database (through csv or json files, or directly from the package), and finally search for the metrics results on request and display them on interactive portrait plots via a web interface. Sebastien Denvil and I are dealing with the production of the metrics ; then we work with Mark Morgan for the storage of the results on a dedicated database ; Patrick Brockmann has designed the UI that will search for the results, and produce the interactive portrait plots ; he is also actively working with Mark Morgan for the exploitation of the database.
+
+I detailed here point by point the different developments we are looking for. The aim is to discuss with you guys which ones are of interest for you and can be handled at PCMDI, and which ones will be done here at IPSL.
+
+Indexation of the metric results
+--------------------------------
+
+The philosophy of the Metrics Garden is to allow crossing the results of various simulations and metrics on the portrait plots by:
+* storing all the results on the same database
+* and fetching the results we want (on request from the UI) for display
+
+Following this, an important issue is to store the metric results with the appropriate indexation (set of keywords) on the database, so that we can find them afterwards. The indexation has to be univocal. In attached file [(see wiki page)](https://github.com/PCMDI/wgne-wgcm_metrics/wiki/Reference-Syntax-indexation) , you will find the set of keywords we identified as necessary to index the results. As you will see, we require much more information than what is currently available in the output json files. This is largely inspired from the CMIP5 DRS, and we already know that the CMIP6 DRS will be potentially different... anyway, the present document aims at opening discussions on this issue and we will be very happy to discuss it with you.
+ The question here is to see with you guys if extending the information provided by the PCMDI-MP is something that could be interesting for you as well (notably for the information related to the reference and the metric). We could discuss this issue together, at least to define the good practices to develop this aspect within the PCMDI-MP.
+
+Adding new metrics
+------------------
+
+ Another issue is to add a collection of metrics to document the mean state of the model. We are looking for the following metrics:
+
+* latitude of the subtropical jets (in progress)
+* ENSO metrics (with Eric)
+* metrics for the sea ice following Massonnet et al. (2012) (attached file ; in progress)
+* metrics for the turbulent fluxes (to be discussed with you Pete)
+* ocean metrics ; the collection of metrics is not defined yet but it should be done in the coming months
+* metrics for the land surfaces (work in progress with the ORCHIDEE people)
+
+Among all those, what are the plans at PCMDI to include additionnal metrics to the PCMDI-MP (like the ENSO Metrics of Eric)? We can discuss about a little bit of coordination, both for the metrics and the references that will be added in the obs directory.
+ We will create an "ipsl" directory within the "metrics" module, where we will add all the diagnostics developed at IPSL (in parallel of the "wgne" module, so that we don't interfere with what you guys do).
+
+Miscellaneous
+-------------
+
+* a Python dictionnary correspondance table for the variable names: would you have specific recommendation for this? Do you plan to include a template of such correspondance table in the PCMDI-MP?
+* when I was at the PCMDI, we talked about a way to recognize the variable units ; do you confirm that it is still of interest (I've seen that there is a branch on github that should deal with this issue)
+* in some cases, the regridding is not necessary for the calculation of the metrics (as for the position of the subtropical jets). Would you consider adding an option in the parameter file so that the regridding can be switched off?
+
+Future developments and challenges
+----------------------------------
+
+ In the future, the question is also whether we could plan to calculate metrics derived from temporal or spatio-temporal analysis (spectrums, EOFs) ; the aim would be to have metrics for the main variability modes (AO-NAO, PNA...), weather regimes and QBO (to give some examples). Is it in the plans of the PCMDI to develop the package to handle time series (longer than the 12 months of the annual cycle)?
+
+If we want to describe a given metric and that this description is univocal,
+we have to give enough information on the numerical experiment (or
+simulation), the reference, the regridding method and the metric itself. We
+have identified the following suite of keywords that could be able to fully identify a metric.
+
+In the json files, at IPSL we want to have the keyword/value pairs. Example: "MetricName":"rms_xyt"
+We would need to do this for all the keywords.
+
+Concerning the different tracking dates, that would be a good point to agree on a given format for all the date provided in the json file.
+
+Some elements (quoted with "!!!") necessitate more thoughts. As well, we could add more details on the respective grids of both the numerical experiment and the reference. Feedbacks and further discussions are welcome!
+
+
+
+  * Description of the simulation
+
+    * ModelActivity = project name (CMIP3, CMIP5…), or the type of simulation that we run at IPSL (PROD, DEVT or TEST) ; this is surely a keyword that has to be discussed
+
+    * ModellingGroup = name of the modelling group (ex: IPSL, KNMI…); same names as in the CMIP5 tree
+
+    * Model = model name (ex: HadGEM, CNRM-CM5...)
+
+    * !!! MIPtable = Amon, Omon… we are not sure that the MIPtable has to be part of the indexation; maybe there is something more clever to be done
+
+    * Experiment = the type of experiment (ex: historical, amip…)
+
+    * SimName = the name of the numerical experiment of simulation (ex: r1i1p1, mysimname1…)
+
+    * ModelFreeSpace = a custom comment to add some more information that can be helpful to document some details on the simulation
+
+    * Login = for us at IPSL, login on which the simulation has been ran
+
+    * Center = for us at IPSL, name of the computing center
+
+    * SimTrackingDate = date of the creation of the file (end of the simulation or update)
+
+  * Description of the reference
+
+    * !!! RefActivity = if there is one, the project associated with the reference (obs4mips…)
+
+    * RefName = the reference name (ex: ERAInterim, CERES-EBAF, AIRS…)
+
+    * !!! RefType = what type of reference is it ? (ex: reanalysis, in-situ, satellite…) (should we merge RefActivity and RefType ?)
+
+    * RefTrackingDate = date of creation of the file (a date that gives a possibility to track down an update)
+
+    * RefFreeSpace = a custom comment to add some more information ; maybe not needed but say it's rather cheap to add it.
+
+  * Description of the regridding
+
+    * RegridMethod = the regridding method that has been used to do the model/reference comparison (bilinear, ESMF, None…)
+
+    * GridName = the name of the grid (ex: regular, gaussian, ORCA, None…)
+
+    * !!! GridResolution = the effective average horizontal grid resolution (ex: 2x2) or the number of grid points and vertical levels (ex: 96x96x39) ; at the moment we provide a tuple, ex: ( 96, 96 ). Should we rather go for a character chain?
+
+  * Description of the metric:
+
+    * Period = period on which the metric is calculated (ex: 1980-01-01_2005-12-31)
+
+    * MetricName = name of the metric (ex: RMSE, corr, ENSO_Metrics_1…)
+
+    * MetricReferenceArticle = explicit ; example: "Gleckler et al. 2009"
+
+    * DataType = the type of data we are using (ex: “SE” for a seasonal cycle, “TS_MO” for a monthly time series, “Ann” for the annual mean…)
+
+    * DomainName = the name of the domain we are working on (ex: Globe, NHEX, MyCustomDomain…)
+
+    * Region = we have to decide whether we use region or domain to provide the information if we are working on land surface, ocean or global
+
+    * GeographicalBounds = the geographical limits of the domain (if it is a lon x lat domain) (ex: 20.00E_120.00E_20.00S_20.00N)
+
+    * Variable (if more than one variable => separate them with a _: t2m_slp; if incorrect/insufficient, leave the field empty and find an explicit and unique MetricName)
+
+    * MetricFreeSpace = a custom comment to add some more information
+
+    * Result = explicit
+
+    * P-value = the p-value of the associated statistical test (ex: 2.e-16, 0.002, None…)
+
+    * 95signifThreshold = the threshold value of the statistic corresponding to the 95% significance level (ex: the value of the correlation coefficient that is significant at 95% for the related number of dof)
+
+    * MetricContactExpert = the email of the contact, in general the developer of the metric (Ex: [email protected])
+
+    * MetricTrackingDate = date of calculation of the metric
+
+
+
diff --git a/docs/_sources/pmp-using-cdp-guide.rst.txt b/docs/_sources/pmp-using-cdp-guide.rst.txt
@@ -0,0 +1,39 @@
+PMP using CDP Guide
+*******************
+
+Introduction
+============
+
+This guide is intended to bring developers (and maybe users) up to speed with the changes done when refactoring pmp to use cdp. If you don't know what cdp is, look `here <https://github.com/UV-CDAT/CDP>`_.
+
+What changed
+------------
+Vocabulary for the parameter has changed to account for the new paradigm of reference data set vs test data set, instead of just observation vs model. `See here <https://github.com/PCMDI/pcmdi_metrics/wiki/PMPParser#default-arguments>`_
+
+All other cdp related stuff is in the ``src/python/pcmdi/scripts/driver/`` folder. This include the ``pmp_parser``, which is no longer in ``src/python/pcmdi/``. 
+
+The majority of the work was done to the ``pcmdi_metrics_driver.py``, which is now named ``pcmdi_metrics_driver_legacy.py``. The new driver is now named ``pcmdi_metrics_driver.py``. Both are executable via the command line. The next section details the changes done to the driver.
+
+Changes to the driver
+---------------------
+
+Though not a requirement of cdp, the driver is now programmed in an object-oriented fashion. There are many good reasons to this, which you can see by googling it. Below is an explanation of the classes, which are located in ``src/python/pcmdi/scripts/driver/``.
+
+* **PMPParameter**: Inherits from ``CDPParameter``. Contains the stuff that's usually in a Python parameter script. Eventually, we want to add error checking to the ``heck_values()`` function.
+
+* **PMPParser**: Inherits from ``CDPParser``, which it based on ``ArgumentParser``. You can add/remove/change the arguments in the ``load_default_args()`` function if needed.
+
+* **DataSet**: One of the largest forthcoming changes to pmp is that observations and models can be used interchangeably. To do so, both must be of the same class, which is ``DataSet``. ``DataSet`` is an abstract class that acts as an `interface <https://en.wikipedia.org/wiki/Interface_(computing)#Programming_to_the_interface>`_, with some functionality through static methods. Each ``DataSet`` object also has an attribute of type ``pmp_io``. 
+
+* **Model**: A concrete version of ``DataSet``. Looking at this from the legacy code, this is all of the stuff in the ``model_versions`` loop. It just does stuff related to ``_model_file``, which was called ``MODEL`` in the legacy version.
+
+* **Observation**: Another concrete version of ``DataSet``. Looking at this from the legacy code, this is all of the stuff in the ``refs`` loop. It just does stuff related to ``_obs_file``, which was called ``OBS`` in the legacy version.
+
+* **PMPDriver**: Inherits from ``CDPDriver``. Has a ``PMPParser`` to get command line arguments. Composed of three functions, ``check_parameter()``, ``run_diags()``, ``export()``. ``check_parameter()`` checks that the ``self.parameter`` has all of the stuff needed for this driver. ``run_diags()`` runs the diags. ``export()`` should export the results, but doesn't do that yet because that's already done in ``run_diags`` (but eventually will do it).
+
+* **RunDiags**: The actual work for ``PMPDriver.run_diags()`` is done by this class. **This is where the main functionality is**. This loops through all of the ``vars``, ``regions``, ``reference_data_set`` and ``test_data_set`` in that order. This also determines if the comparison is obs vs obs, obs vs model, or model vs model.
+
+* **OutputMetrics** When ``RunDiags`` gets the data from ``Model`` or ``Observation`` (via ``DataSet.get()``), these get sent to ``OutputMetrics`` which creates the ``metrics_dictionary``, computes the metrics needed, and outputs the results. Also has an ``out_file`` and ``clim_file``, which were respectively ``OUT`` and ``CLIM`` previously. 
+
+
+