This repository contains all that is required to install and run various Jupyter notebooks which demonstrate the simplified access to model data via FDB, the Polytope server and the Polytope library (ECMWF), and then the subsequent post-processing via a data processing framework meteodata-lab (in development). The following use cases are covered by the notebooks:

• Retrieving a specified selection of GRIB fields into an xarray object and processing with meteorological operators.
• Remotely retrieving a very reduced subset of GRIB fields, eg just a few grid points, via the Polytope algorithm, to create a timeseries over an extended period.
• Writing GRIB fields data back to FDB having processed the model data at CSCS.

The notebooks for the demo can be found in the directory notebooks.

Notebooks to run from CSCS (Balfrin/Eiger):

• 1_Data_Retrieval_from_FDB_Preprocessing
• 2_Precompute_and_Store_Echotop_to_FDB

Notebooks to run from the LabVM:

• 3_Retrieve_Echotop_and_Regrid
• 4_Location_and_TimeSeries_Access

Forecasts available in FDB

To use the Jupyter notebooks you have the following two options regarding the runtime dependencies and the jupyter server:

1. Jupyter server on the lab-vm or at CSCS (Balfrin/Eiger) and runtime dependencies in a container
2. Both the jupyter server and the runtime dependencies in a container (LabVM only)

With this approach you define a kernel definiton in your jupyter server with a reference to the container.

sudo apt install pipx
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
pipx install jupyterlab

./host/install_kernel.sh

Connect to the jupyter server

• from VSCode:
  Open the notebook and select the polytope-demo kernel in "Select Kernel" -> "Select another Kernel..." -> "Jupyter Kernel..."
  
• from your browser:

  pipx run jupyter lab --port 8080

  Open the URL given in the code and select the polytope-demo kernel.

Setup spack for the machine and build FDB.

spack env activate -p spack-env
spack install --no-checksum

Setup the python environment.

conda env create -n demo -f environment.yaml
conda activate demo
git clone -b mars-levtype-echotop-2 https://github.com/cfkanesan/eccodes-cosmo-resources.git $CONDA_PREFIX/share/eccodes-cosmo-resources
pip install jupyterlab
python -m jupyter lab

Use VSCode to forward the port that is binded to the jupyter lab server to your local machine and open the link in the jupyter lab server logs. Ctrl-C once to show the link again.

With this approach you have both the Jupyter server and the runtime dependencies in a container.

podman run \
  -e https_proxy=$https_proxy \
  -e REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt \
  -e SSL_CERT_DIR=/etc/ssl/certs/ \
  --network=host \
  --rm \
  dockerhub.apps.cp.meteoswiss.ch/numericalweatherpredictions/polytope/demo/notebook:<TAG>

<TAG>:The current container tag can be retrieved from: https://nexus.meteoswiss.ch/nexus/service/rest/repository/browse/docker-all/v2/numericalweatherpredictions/polytope/demo/notebook/tags/

Afterwards connect to the external Jupyter server from the notebook with the url from container log. Click "Select Kernel" -> "Existing Jupyter Server..." and then paste the url form the container log.