notebooks/neural_tangents_cookbook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/google/neural-tangents/blob/main/notebooks/neural_tangents_cookbook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cIkq2O7yVWYJ"
   },
   "source": [
    "Copyright 2019 Google LLC\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "     https://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AcGrb0snO_S6"
   },
   "source": [
    "### Imports and Utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Iwc6IU5EwORY"
   },
   "outputs": [],
   "source": [
    "!pip install -q --upgrade pip\n",
    "!pip install -q --upgrade jax[cuda11_cudnn82] -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html\n",
    "!pip install -q git+https://www.github.com/google/neural-tangents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Lt74vgCVNN2b"
   },
   "outputs": [],
   "source": [
    "import jax.numpy as jnp\n",
    "\n",
    "from jax import random\n",
    "from jax.example_libraries import optimizers\n",
    "from jax import jit, grad, vmap\n",
    "\n",
    "import functools\n",
    "\n",
    "import neural_tangents as nt\n",
    "from neural_tangents import stax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "U8lzkWnROEQs"
   },
   "outputs": [],
   "source": [
    "from IPython.display import set_matplotlib_formats\n",
    "set_matplotlib_formats('pdf', 'svg')\n",
    "import matplotlib\n",
    "import seaborn as sns\n",
    "\n",
    "sns.set(font_scale=1.3)\n",
    "sns.set_style(\"darkgrid\", {\"axes.facecolor\": \".95\"})\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def format_plot(x=None, y=None):\n",
    "  # plt.grid(False)\n",
    "  ax = plt.gca()\n",
    "  if x is not None:\n",
    "    plt.xlabel(x, fontsize=20)\n",
    "  if y is not None:\n",
    "    plt.ylabel(y, fontsize=20)\n",
    "\n",
    "def finalize_plot(shape=(1, 1)):\n",
    "  plt.gcf().set_size_inches(\n",
    "    shape[0] * 1.5 * plt.gcf().get_size_inches()[1],\n",
    "    shape[1] * 1.5 * plt.gcf().get_size_inches()[1])\n",
    "  plt.tight_layout()\n",
    "\n",
    "legend = functools.partial(plt.legend, fontsize=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YLa63IKQRnE0"
   },
   "outputs": [],
   "source": [
    "def plot_fn(train, test, *fs):\n",
    "  train_xs, train_ys = train\n",
    "\n",
    "  plt.plot(train_xs, train_ys, 'ro', markersize=10, label='train')\n",
    "\n",
    "  if test != None:\n",
    "    test_xs, test_ys = test\n",
    "    plt.plot(test_xs, test_ys, 'k--', linewidth=3, label='$f(x)$')\n",
    "\n",
    "    for f in fs:\n",
    "      plt.plot(test_xs, f(test_xs), '-', linewidth=3)\n",
    "\n",
    "  plt.xlim([-jnp.pi, jnp.pi])\n",
    "  plt.ylim([-1.5, 1.5])\n",
    "\n",
    "  format_plot('$x$', '$f$')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5zNUrSHJQOan"
   },
   "outputs": [],
   "source": [
    "def loss_fn(predict_fn, ys, t, xs=None):\n",
    "  mean, cov = predict_fn(t=t, get='ntk', x_test=xs, compute_cov=True)\n",
    "  mean = jnp.reshape(mean, mean.shape[:1] + (-1,))\n",
    "  var = jnp.diagonal(cov, axis1=1, axis2=2)\n",
    "  ys = jnp.reshape(ys, (1, -1))\n",
    "\n",
    "  mean_predictions = 0.5 * jnp.mean(ys ** 2 - 2 * mean * ys + var + mean ** 2,\n",
    "                                   axis=1)\n",
    "\n",
    "  return mean_predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xOVCymQgVn9R"
   },
   "source": [
    "# Neural Tangents Cookbook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LKnYe66NVzFF"
   },
   "source": [
    "In this notebook we explore the training of infinitely-wide, Bayesian, neural networks using a library called [Neural Tangents](https://github.com/google/neural-tangents). Recent work has shown that such networks are Gaussian Processes with a particular compositional kernel called the NNGP kernel. More recently, it was shown that predictions resulting from these networks following Gradient Descent are Gaussian with a distribution that can be computed in closed form using the [Neural Tangent Kernel](). Neural Tangents provides a high level library to compute NNGP and NT kernels for a wide range of neural networks. See [the paper]() for a more detailed description of the library itself.\n",
    "\n",
    "Our goal will be to train an ensemble of neural networks on a simple synthetic task. We'll then compare the results of this ensemble with the prediction of the NTK theory. Finally, we'll play around with different neural network architectures to see how this affects the resulting kernel. However, Neural Tangents is built on JAX which may be new to you. To get warmed up with JAX, we'll start out by generating some data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MM9rp-EqOuWT"
   },
   "source": [
    "## Warm Up: Creating a Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jEjKnmJsa6AC"
   },
   "source": [
    "We're going to build a widely used synthetic dataset that's used extensively in [Pattern Recognition and Machine Learning](). Incidentally, Pattern Recognition and Machine Learning is an outstanding book by Christopher Bishop that was recently released for free.\n",
    "\n",
    "Our training data is going to be drawn from a process,\n",
    "    $$y = f(x) + \\epsilon$$\n",
    "where $f(x)$ is a deterministic function and $\\epsilon\\sim\\mathcal N(0, \\sigma)$ is Gaussian noise with some scale. We're going to choose $f(x) = \\sin(x)$ with $x\\sim\\text{Uniform}(-\\pi, \\pi)$. Our testing data will be $y = f(x)$ for $x$ linearly spaced in $[-\\pi, \\pi]$. Feel free to try out different functions and domains!\n",
    "\n",
    "Since we want to generate our data randomly, we'll need to generate random numbers. Unlike most random number generators that store a global random state, JAX makes the random state explicit (see the [JAX documentation](https://github.com/google/jax#random-numbers-are-different) for more information). Let's therefore start by making some random state using a seed of 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c8KMAdhnO9i6"
   },
   "outputs": [],
   "source": [
    "key = random.PRNGKey(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jvlnKFRindkt"
   },
   "source": [
    "Now let's set up some constants that will define our dataset. In particular, we will use a small training set of 5 points and 50 tests points. Finally, we'll define a noise scale and target function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Fan7RcUcngvS"
   },
   "outputs": [],
   "source": [
    "train_points = 5\n",
    "test_points = 50\n",
    "noise_scale = 1e-1\n",
    "\n",
    "target_fn = lambda x: jnp.sin(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YduzMTM-nsEl"
   },
   "source": [
    "Next we generate our training data. We know that we will want to have randomly chosen $x$'s and noise. To generate random numbers in JAX, we have to explicitly evolve the random number state using `random.split` each time we draw a random number.\n",
    "\n",
    "Then we'll want to generate the random inputs, apply the target function, and add the random noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JQSL0tCjOxVP"
   },
   "outputs": [],
   "source": [
    "key, x_key, y_key = random.split(key, 3)\n",
    "\n",
    "train_xs = random.uniform(x_key, (train_points, 1), minval=-jnp.pi, maxval=jnp.pi)\n",
    "\n",
    "train_ys = target_fn(train_xs)\n",
    "train_ys += noise_scale * random.normal(y_key, (train_points, 1))\n",
    "train = (train_xs, train_ys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MgaBJLDCo2h9"
   },
   "source": [
    "Finally, we want to generate our test data. The $x$'s will be linearly spaced with no noise. Note, we want the inputs to have shape `(N, 1)` instead of `(N,)` since we treat this as a model with one feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5QRjxH8ZpKuX"
   },
   "outputs": [],
   "source": [
    "test_xs = jnp.linspace(-jnp.pi, jnp.pi, test_points)\n",
    "test_xs = jnp.reshape(test_xs, (test_points, 1))\n",
    "\n",
    "test_ys = target_fn(test_xs)\n",
    "test = (test_xs, test_ys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Mi5fGc11puDz"
   },
   "source": [
    "Having generated our data, let's plot it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 341
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 589,
     "status": "ok",
     "timestamp": 1588653667417,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "gUwuP34vPexj",
    "outputId": "0c8bbd6e-5120-4b59-9612-0f09065c0d41"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"239.859969pt\" version=\"1.1\" viewBox=\"0 0 348.854672 239.859969\" width=\"348.854672pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 239.859969 \nL 348.854672 239.859969 \nL 348.854672 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \nL 335.654672 13.2 \nL 74.780672 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 122.178899 185.964 \nL 122.178899 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- −2 -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(111.638235 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 205.217672 185.964 \nL 205.217672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(200.668484 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 288.256445 185.964 \nL 288.256445 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(283.707258 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_4\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(199.217672 228.500594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_5\">\n      <!-- −1.5 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 191.396883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 74.780672 157.17 \nL 335.654672 157.17 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_6\">\n      <!-- −1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 162.602883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 74.780672 128.376 \nL 335.654672 128.376 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- −0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 133.808883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 74.780672 99.582 \nL 335.654672 99.582 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 105.014883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 74.780672 70.788 \nL 335.654672 70.788 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 76.220883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 74.780672 41.994 \nL 335.654672 41.994 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 47.426883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_10\">\n      <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 18.632883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 103.182)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_11\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"m10f1cd75a7\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#pd2713cb1b4)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"332.990361\" xlink:href=\"#m10f1cd75a7\" y=\"89.211043\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"312.709007\" xlink:href=\"#m10f1cd75a7\" y=\"61.934552\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"113.996599\" xlink:href=\"#m10f1cd75a7\" y=\"143.295353\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"197.320621\" xlink:href=\"#m10f1cd75a7\" y=\"113.447504\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"255.905476\" xlink:href=\"#m10f1cd75a7\" y=\"35.617422\"/>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#pd2713cb1b4)\" d=\"M 74.780668 99.581995 \nL 80.104624 106.946181 \nL 85.42859 114.189459 \nL 90.752545 121.192871 \nL 96.076501 127.841434 \nL 101.400467 134.025987 \nL 106.724422 139.644962 \nL 112.048388 144.606113 \nL 117.372344 148.827959 \nL 122.696304 152.241194 \nL 128.020265 154.789766 \nL 133.34422 156.431825 \nL 138.668181 157.140412 \nL 143.992142 156.903891 \nL 149.316097 155.726148 \nL 154.640058 153.626521 \nL 159.964019 150.639481 \nL 165.287974 146.814083 \nL 170.61194 142.213126 \nL 175.935896 136.912177 \nL 181.259861 130.998257 \nL 186.583817 124.568494 \nL 191.907773 117.728453 \nL 197.231738 110.590436 \nL 202.555694 103.271674 \nL 207.87965 95.892326 \nL 213.203615 88.57355 \nL 218.527571 81.435547 \nL 223.851527 74.595506 \nL 229.175492 68.165732 \nL 234.499448 62.251823 \nL 239.823414 56.950863 \nL 245.147369 52.349917 \nL 250.471325 48.524519 \nL 255.795281 45.537482 \nL 261.119256 43.437849 \nL 266.443212 42.260106 \nL 271.767168 42.023588 \nL 277.091123 42.732175 \nL 282.415079 44.374234 \nL 287.739054 46.922813 \nL 293.06301 50.336048 \nL 298.386966 54.557898 \nL 303.710922 59.519038 \nL 309.034877 65.138013 \nL 314.358833 71.322554 \nL 319.682808 77.971141 \nL 325.006764 84.974554 \nL 330.33072 92.217819 \nL 335.654676 99.582005 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:11.1,4.8;stroke-dashoffset:0;stroke-width:3;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 74.780672 185.964 \nL 74.780672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 335.654672 185.964 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 81.780672 50.55625 \nL 137.055672 50.55625 \nQ 139.055672 50.55625 139.055672 48.55625 \nL 139.055672 20.2 \nQ 139.055672 18.2 137.055672 18.2 \nL 81.780672 18.2 \nQ 79.780672 18.2 79.780672 20.2 \nL 79.780672 48.55625 \nQ 79.780672 50.55625 81.780672 50.55625 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_13\"/>\n    <g id=\"line2d_14\">\n     <g>\n      <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"93.780672\" xlink:href=\"#m10f1cd75a7\" y=\"26.298437\"/>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- train -->\n     <defs>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 29.798437)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_15\">\n     <path d=\"M 83.780672 40.976562 \nL 103.780672 40.976562 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:11.1,4.8;stroke-dashoffset:0;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_16\"/>\n    <g id=\"text_14\">\n     <!-- $f(x)$ -->\n     <defs>\n      <path d=\"M 31 75.875 \nQ 24.46875 64.65625 21.28125 53.65625 \nQ 18.109375 42.671875 18.109375 31.390625 \nQ 18.109375 20.125 21.3125 9.0625 \nQ 24.515625 -2 31 -13.1875 \nL 23.1875 -13.1875 \nQ 15.875 -1.703125 12.234375 9.375 \nQ 8.59375 20.453125 8.59375 31.390625 \nQ 8.59375 42.28125 12.203125 53.3125 \nQ 15.828125 64.359375 23.1875 75.875 \nz\n\" id=\"DejaVuSans-40\"/>\n      <path d=\"M 8.015625 75.875 \nL 15.828125 75.875 \nQ 23.140625 64.359375 26.78125 53.3125 \nQ 30.421875 42.28125 30.421875 31.390625 \nQ 30.421875 20.453125 26.78125 9.375 \nQ 23.140625 -1.703125 15.828125 -13.1875 \nL 8.015625 -13.1875 \nQ 14.5 -2 17.703125 9.0625 \nQ 20.90625 20.125 20.90625 31.390625 \nQ 20.90625 42.671875 17.703125 53.65625 \nQ 14.5 64.65625 8.015625 75.875 \nz\n\" id=\"DejaVuSans-41\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 44.476562)scale(0.1 -0.1)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n      <use transform=\"translate(35.205078 0.015625)\" xlink:href=\"#DejaVuSans-40\"/>\n      <use transform=\"translate(74.21875 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n      <use transform=\"translate(133.398438 0.015625)\" xlink:href=\"#DejaVuSans-41\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pd2713cb1b4\">\n   <rect height=\"172.764\" width=\"260.874\" x=\"74.780672\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 510x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_fn(train, test)\n",
    "legend(loc='upper left')\n",
    "finalize_plot((0.85, 0.6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gu1yKzZcqY2b"
   },
   "source": [
    "What a good looking dataset! Let's train a neural network on it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Af6ufsPafoFO"
   },
   "source": [
    "## Defining a Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "luzPB8YvfxhC"
   },
   "source": [
    "The first thing we need to do is define a neural network. We'll start out with a simple fully-connected network using `Erf` nonlinearities. We describe our network using our neural network library that shares syntax and code with JAX's own library called [stax](https://github.com/google/jax#neural-net-building-with-stax). Layers in `jax.example_libraries.stax` are pairs of functions `(init_fn, apply_fn)` where `init_fn(key, input_shape)` draws parameters randomly and `apply_fn(params, xs)` computes outputs of the function for specific inputs. These layers can be composed using `serial` and `parallel` combinators to produce new `(init_fn, apply_fn)` pairs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Uc8XZxuUgTTd"
   },
   "source": [
    "Similarly, layers in `neural_tangents.stax` are triplets of functions `(init_fn, apply_fn, kernel_fn)` where the first two functions are the same as their `stax` equivalent but the third function, `kernel_fn`, computes infinite-width GP kernels corresponding to the layer. Again these layers can be composed using `serial` and `parallel` to build kernels for complicated architectures. Fully-connected layers in `neural_tangents.stax` are created using the `Dense` layer which is defined by,\n",
    "$$z^{l+1}_i = \\frac{\\sigma_w}{\\sqrt{N_{in}}} \\sum_j W_{ij} z^{l}_i + \\sigma_b b_i$$\n",
    "where $W_{ij}, b_i\\sim\\mathcal N(0,1)$ at initialization and $\\sigma_w, \\sigma_b$ sets the scales of the weights and biases respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4u4oqkyEkiAE"
   },
   "outputs": [],
   "source": [
    "init_fn, apply_fn, kernel_fn = stax.serial(\n",
    "    stax.Dense(512, W_std=1.5, b_std=0.05), stax.Erf(),\n",
    "    stax.Dense(512, W_std=1.5, b_std=0.05), stax.Erf(),\n",
    "    stax.Dense(1, W_std=1.5, b_std=0.05)\n",
    ")\n",
    "\n",
    "apply_fn = jit(apply_fn)\n",
    "kernel_fn = jit(kernel_fn, static_argnames='get')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UQ87i8-7kpg6"
   },
   "source": [
    "Here the lines `apply_fn = jit(apply_fn)` and `kernel_fn = jit(kernel_fn, static_argnames='get')` use a [JAX feature](https://github.com/google/jax#compilation-with-jit) that compiles functions so that they are executed as single calls to the GPU.\n",
    "\n",
    "Next, let's take several random draws of the parameters of the network and plot what the functions look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zOz_S-IPlAqZ"
   },
   "outputs": [],
   "source": [
    "prior_draws = []\n",
    "for _ in range(10):\n",
    "  key, net_key = random.split(key)\n",
    "  _, params = init_fn(net_key, (-1, 1))\n",
    "  prior_draws += [apply_fn(params, test_xs)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 341
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 860,
     "status": "ok",
     "timestamp": 1588653669127,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "FCC6_ER2ng5W",
    "outputId": "ad46f165-e33f-4855-c26b-d1cfbc63ad73"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"239.859969pt\" version=\"1.1\" viewBox=\"0 0 348.854672 239.859969\" width=\"348.854672pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 239.859969 \nL 348.854672 239.859969 \nL 348.854672 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \nL 335.654672 13.2 \nL 74.780672 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 122.178899 185.964 \nL 122.178899 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- −2 -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(111.638235 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 205.217672 185.964 \nL 205.217672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(200.668484 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 288.256445 185.964 \nL 288.256445 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(283.707258 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_4\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(199.217672 228.500594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_5\">\n      <!-- −1.5 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 191.396883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780672 157.17 \nL 335.654672 157.17 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_6\">\n      <!-- −1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 162.602883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780672 128.376 \nL 335.654672 128.376 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- −0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 133.808883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780672 99.582 \nL 335.654672 99.582 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 105.014883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780672 70.788 \nL 335.654672 70.788 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 76.220883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780672 41.994 \nL 335.654672 41.994 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 47.426883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_10\">\n      <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 18.632883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 103.182)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_11\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"m4af95c9f67\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#p0fa4f723df)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"332.990361\" xlink:href=\"#m4af95c9f67\" y=\"89.211043\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"312.709007\" xlink:href=\"#m4af95c9f67\" y=\"61.934552\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"113.996599\" xlink:href=\"#m4af95c9f67\" y=\"143.295353\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"197.320621\" xlink:href=\"#m4af95c9f67\" y=\"113.447504\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"255.905476\" xlink:href=\"#m4af95c9f67\" y=\"35.617422\"/>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780668 99.581995 \nL 80.104624 106.946181 \nL 85.42859 114.189459 \nL 90.752545 121.192871 \nL 96.076501 127.841434 \nL 101.400467 134.025987 \nL 106.724422 139.644962 \nL 112.048388 144.606113 \nL 117.372344 148.827959 \nL 122.696304 152.241194 \nL 128.020265 154.789766 \nL 133.34422 156.431825 \nL 138.668181 157.140412 \nL 143.992142 156.903891 \nL 149.316097 155.726148 \nL 154.640058 153.626521 \nL 159.964019 150.639481 \nL 165.287974 146.814083 \nL 170.61194 142.213126 \nL 175.935896 136.912177 \nL 181.259861 130.998257 \nL 186.583817 124.568494 \nL 191.907773 117.728453 \nL 197.231738 110.590436 \nL 202.555694 103.271674 \nL 207.87965 95.892326 \nL 213.203615 88.57355 \nL 218.527571 81.435547 \nL 223.851527 74.595506 \nL 229.175492 68.165732 \nL 234.499448 62.251823 \nL 239.823414 56.950863 \nL 245.147369 52.349917 \nL 250.471325 48.524519 \nL 255.795281 45.537482 \nL 261.119256 43.437849 \nL 266.443212 42.260106 \nL 271.767168 42.023588 \nL 277.091123 42.732175 \nL 282.415079 44.374234 \nL 287.739054 46.922813 \nL 293.06301 50.336048 \nL 298.386966 54.557898 \nL 303.710922 59.519038 \nL 309.034877 65.138013 \nL 314.358833 71.322554 \nL 319.682808 77.971141 \nL 325.006764 84.974554 \nL 330.33072 92.217819 \nL 335.654676 99.582005 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:11.1,4.8;stroke-dashoffset:0;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 82.218899 -1 \nL 85.42859 -0.500605 \nL 90.752545 0.368521 \nL 96.076501 1.273386 \nL 101.400467 2.206267 \nL 106.724422 3.156517 \nL 112.048388 4.110598 \nL 117.372344 5.05274 \nL 122.696304 5.967058 \nL 128.020265 6.841065 \nL 133.34422 7.671232 \nL 138.668181 8.470362 \nL 143.992142 9.275362 \nL 149.316097 10.154352 \nL 154.640058 11.211388 \nL 159.964019 12.590942 \nL 165.287974 14.487899 \nL 170.61194 17.173208 \nL 175.935896 21.053011 \nL 181.259861 26.791952 \nL 186.583817 35.45814 \nL 191.907773 48.299247 \nL 197.231738 66.258189 \nL 202.555694 90.527189 \nL 207.87965 117.010954 \nL 213.203615 137.913785 \nL 218.527571 153.34463 \nL 223.851527 164.027679 \nL 229.175492 170.755492 \nL 234.499448 175.004057 \nL 239.823414 177.858726 \nL 245.147369 179.889147 \nL 250.471325 181.396802 \nL 255.795281 182.564687 \nL 261.119256 183.521507 \nL 266.443212 184.366543 \nL 271.767168 185.175243 \nL 277.091123 185.999938 \nL 282.415079 186.869799 \nL 287.739054 187.794566 \nL 293.06301 188.769955 \nL 298.386966 189.783473 \nL 303.710922 190.81911 \nL 309.034877 191.860603 \nL 314.358833 192.893494 \nL 319.682808 193.90601 \nL 325.006764 194.889095 \nL 330.33072 195.836489 \nL 335.654676 196.744025 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_14\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780668 126.684161 \nL 80.104624 127.201456 \nL 85.42859 127.759893 \nL 90.752545 128.359694 \nL 96.076501 129.001023 \nL 101.400467 129.683849 \nL 106.724422 130.407985 \nL 112.048388 131.172859 \nL 117.372344 131.976407 \nL 122.696304 132.814053 \nL 128.020265 133.676341 \nL 133.34422 134.545265 \nL 138.668181 135.390057 \nL 143.992142 136.161123 \nL 149.316097 136.785607 \nL 154.640058 137.166455 \nL 159.964019 137.196115 \nL 165.287974 136.798678 \nL 170.61194 136.013724 \nL 175.935896 135.083407 \nL 181.259861 134.396503 \nL 186.583817 134.110932 \nL 191.907773 133.571711 \nL 197.231738 130.555614 \nL 202.555694 119.705694 \nL 207.87965 100.570667 \nL 213.203615 87.612371 \nL 218.527571 82.483249 \nL 223.851527 79.655568 \nL 229.175492 76.990155 \nL 234.499448 74.252045 \nL 239.823414 71.764793 \nL 245.147369 69.857343 \nL 250.471325 68.671084 \nL 255.795281 68.170648 \nL 261.119256 68.227418 \nL 266.443212 68.689314 \nL 271.767168 69.417721 \nL 277.091123 70.301273 \nL 282.415079 71.257549 \nL 287.739054 72.229303 \nL 293.06301 73.179347 \nL 298.386966 74.085246 \nL 303.710922 74.934724 \nL 309.034877 75.722226 \nL 314.358833 76.446344 \nL 319.682808 77.10812 \nL 325.006764 77.709827 \nL 330.33072 78.254323 \nL 335.654676 78.744742 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_15\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 192.79315 -1 \nL 197.231738 19.483543 \nL 202.555694 65.062264 \nL 207.87965 126.664074 \nL 213.203615 174.913535 \nL 218.527571 200.798584 \nL 223.851527 214.530699 \nL 229.175492 224.103083 \nL 234.499448 232.190947 \nL 239.823414 239.121629 \nL 241.442736 240.859969 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_16\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780668 65.883252 \nL 80.104624 64.867774 \nL 85.42859 63.77685 \nL 90.752545 62.60728 \nL 96.076501 61.356362 \nL 101.400467 60.02232 \nL 106.724422 58.604957 \nL 112.048388 57.106037 \nL 117.372344 55.5305 \nL 122.696304 53.88735 \nL 128.020265 52.191434 \nL 133.34422 50.465258 \nL 138.668181 48.74153 \nL 143.992142 47.067318 \nL 149.316097 45.510269 \nL 154.640058 44.169495 \nL 159.964019 43.194113 \nL 165.287974 42.811724 \nL 170.61194 43.369198 \nL 175.935896 45.385487 \nL 181.259861 49.588358 \nL 186.583817 56.796802 \nL 191.907773 67.42583 \nL 197.231738 81.620223 \nL 202.555694 101.267863 \nL 207.87965 121.462723 \nL 213.203615 133.418728 \nL 218.527571 141.985503 \nL 223.851527 149.728945 \nL 229.175492 155.220367 \nL 234.499448 158.003476 \nL 239.823414 158.670221 \nL 245.147369 157.949221 \nL 250.471325 156.409218 \nL 255.795281 154.439361 \nL 261.119256 152.28525 \nL 266.443212 150.091432 \nL 271.767168 147.938691 \nL 277.091123 145.869885 \nL 282.415079 143.906738 \nL 287.739054 142.058862 \nL 293.06301 140.329113 \nL 298.386966 138.716272 \nL 303.710922 137.217084 \nL 309.034877 135.826728 \nL 314.358833 134.540003 \nL 319.682808 133.351327 \nL 325.006764 132.25524 \nL 330.33072 131.246359 \nL 335.654676 130.319643 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_17\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780668 2.876074 \nL 80.104624 1.697088 \nL 85.42859 0.489695 \nL 90.752545 -0.747561 \nL 91.810449 -1 \nM 193.126302 -1 \nL 197.231738 22.599077 \nL 202.555694 69.659778 \nL 207.87965 126.530999 \nL 213.203615 172.217653 \nL 218.527571 200.627253 \nL 223.851527 216.512179 \nL 229.175492 223.861997 \nL 234.499448 225.65959 \nL 239.823414 224.178832 \nL 245.147369 221.046472 \nL 250.471325 217.318229 \nL 255.795281 213.602479 \nL 261.119256 210.197036 \nL 266.443212 207.207173 \nL 271.767168 204.632164 \nL 277.091123 202.421565 \nL 282.415079 200.508461 \nL 287.739054 198.826832 \nL 293.06301 197.319451 \nL 298.386966 195.940467 \nL 303.710922 194.655101 \nL 309.034877 193.43811 \nL 314.358833 192.271969 \nL 319.682808 191.145055 \nL 325.006764 190.049987 \nL 330.33072 188.982455 \nL 335.654676 187.940283 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_18\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780668 64.033235 \nL 80.104624 64.116579 \nL 85.42859 64.166378 \nL 90.752545 64.174095 \nL 96.076501 64.129036 \nL 101.400467 64.017785 \nL 106.724422 63.823879 \nL 112.048388 63.527523 \nL 117.372344 63.105344 \nL 122.696304 62.530683 \nL 128.020265 61.774278 \nL 133.34422 60.805485 \nL 138.668181 59.594073 \nL 143.992142 58.111787 \nL 149.316097 56.333794 \nL 154.640058 54.239302 \nL 159.964019 51.815897 \nL 165.287974 49.078973 \nL 170.61194 46.134406 \nL 175.935896 43.318318 \nL 181.259861 41.393791 \nL 186.583817 41.611742 \nL 191.907773 45.562158 \nL 197.231738 56.184219 \nL 202.555694 80.674506 \nL 207.87965 117.783581 \nL 213.203615 143.136918 \nL 218.527571 154.822065 \nL 223.851527 160.343358 \nL 229.175492 162.192418 \nL 234.499448 161.608479 \nL 239.823414 159.80814 \nL 245.147369 157.670817 \nL 250.471325 155.629585 \nL 255.795281 153.82422 \nL 261.119256 152.26542 \nL 266.443212 150.928474 \nL 271.767168 149.788094 \nL 277.091123 148.824516 \nL 282.415079 148.020831 \nL 287.739054 147.360594 \nL 293.06301 146.827054 \nL 298.386966 146.403723 \nL 303.710922 146.07483 \nL 309.034877 145.826127 \nL 314.358833 145.645076 \nL 319.682808 145.520956 \nL 325.006764 145.444717 \nL 330.33072 145.408871 \nL 335.654676 145.407189 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_19\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780668 36.491978 \nL 80.104624 35.855398 \nL 85.42859 35.214376 \nL 90.752545 34.570622 \nL 96.076501 33.925516 \nL 101.400467 33.279936 \nL 106.724422 32.634095 \nL 112.048388 31.987389 \nL 117.372344 31.33828 \nL 122.696304 30.684572 \nL 128.020265 30.023628 \nL 133.34422 29.353533 \nL 138.668181 28.674974 \nL 143.992142 27.99535 \nL 149.316097 27.336541 \nL 154.640058 26.74816 \nL 159.964019 26.327224 \nL 165.287974 26.239318 \nL 170.61194 26.730249 \nL 175.935896 28.111129 \nL 181.259861 30.688465 \nL 186.583817 34.596455 \nL 191.907773 39.950124 \nL 197.231738 49.697707 \nL 202.555694 74.526584 \nL 207.87965 113.891882 \nL 213.203615 140.614362 \nL 218.527571 152.967664 \nL 223.851527 160.75828 \nL 229.175492 166.482969 \nL 234.499448 170.045765 \nL 239.823414 171.761212 \nL 245.147369 172.243974 \nL 250.471325 172.037282 \nL 255.795281 171.509121 \nL 261.119256 170.869912 \nL 266.443212 170.220501 \nL 271.767168 169.597912 \nL 277.091123 169.008048 \nL 282.415079 168.444217 \nL 287.739054 167.896752 \nL 293.06301 167.356516 \nL 298.386966 166.816616 \nL 303.710922 166.272391 \nL 309.034877 165.721184 \nL 314.358833 165.162014 \nL 319.682808 164.595135 \nL 325.006764 164.021638 \nL 330.33072 163.443314 \nL 335.654676 162.862272 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_20\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780668 115.269391 \nL 80.104624 115.416521 \nL 85.42859 115.622295 \nL 90.752545 115.893731 \nL 96.076501 116.237679 \nL 101.400467 116.660367 \nL 106.724422 117.166599 \nL 112.048388 117.757999 \nL 117.372344 118.432091 \nL 122.696304 119.17974 \nL 128.020265 119.983518 \nL 133.34422 120.815686 \nL 138.668181 121.63809 \nL 143.992142 122.404436 \nL 149.316097 123.068394 \nL 154.640058 123.59946 \nL 159.964019 124.009081 \nL 165.287974 124.382651 \nL 170.61194 124.90411 \nL 175.935896 125.846863 \nL 181.259861 127.493721 \nL 186.583817 129.924163 \nL 191.907773 132.266914 \nL 197.231738 129.643126 \nL 202.555694 109.770623 \nL 207.87965 76.366854 \nL 213.203615 59.321037 \nL 218.527571 57.57648 \nL 223.851527 60.4517 \nL 229.175492 63.718926 \nL 234.499448 66.194447 \nL 239.823414 67.746904 \nL 245.147369 68.663766 \nL 250.471325 69.294579 \nL 255.795281 69.887254 \nL 261.119256 70.564279 \nL 266.443212 71.355835 \nL 271.767168 72.241137 \nL 277.091123 73.179275 \nL 282.415079 74.127318 \nL 287.739054 75.049037 \nL 293.06301 75.917594 \nL 298.386966 76.715496 \nL 303.710922 77.433114 \nL 309.034877 78.066968 \nL 314.358833 78.617883 \nL 319.682808 79.089461 \nL 325.006764 79.486966 \nL 330.33072 79.816319 \nL 335.654676 80.08365 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_21\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780668 168.741356 \nL 80.104624 168.412782 \nL 85.42859 168.06695 \nL 90.752545 167.70219 \nL 96.076501 167.315985 \nL 101.400467 166.904262 \nL 106.724422 166.460898 \nL 112.048388 165.976687 \nL 117.372344 165.438174 \nL 122.696304 164.825814 \nL 128.020265 164.112002 \nL 133.34422 163.258405 \nL 138.668181 162.21383 \nL 143.992142 160.912303 \nL 149.316097 159.273327 \nL 154.640058 157.206254 \nL 159.964019 154.618576 \nL 165.287974 151.428025 \nL 170.61194 147.565879 \nL 175.935896 142.949102 \nL 181.259861 137.411495 \nL 186.583817 130.699056 \nL 191.907773 122.700114 \nL 197.231738 114.262771 \nL 202.555694 109.435413 \nL 207.87965 107.001822 \nL 213.203615 101.160947 \nL 218.527571 91.328859 \nL 223.851527 81.151537 \nL 229.175492 71.970734 \nL 234.499448 64.211245 \nL 239.823414 57.890438 \nL 245.147369 52.842146 \nL 250.471325 48.852884 \nL 255.795281 45.712108 \nL 261.119256 43.232835 \nL 266.443212 41.26028 \nL 271.767168 39.673121 \nL 277.091123 38.379427 \nL 282.415079 37.311181 \nL 287.739054 36.41844 \nL 293.06301 35.664482 \nL 298.386966 35.022073 \nL 303.710922 34.470846 \nL 309.034877 33.995148 \nL 314.358833 33.582965 \nL 319.682808 33.224714 \nL 325.006764 32.912678 \nL 330.33072 32.640528 \nL 335.654676 32.403025 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_22\">\n    <path clip-path=\"url(#p0fa4f723df)\" d=\"M 74.780668 34.654094 \nL 80.104624 34.243065 \nL 85.42859 33.764648 \nL 90.752545 33.217499 \nL 96.076501 32.601932 \nL 101.400467 31.920792 \nL 106.724422 31.179898 \nL 112.048388 30.389163 \nL 117.372344 29.563706 \nL 122.696304 28.725294 \nL 128.020265 27.904395 \nL 133.34422 27.143146 \nL 138.668181 26.499763 \nL 143.992142 26.055926 \nL 149.316097 25.928133 \nL 154.640058 26.283432 \nL 159.964019 27.35463 \nL 165.287974 29.437218 \nL 170.61194 32.829007 \nL 175.935896 37.664909 \nL 181.259861 43.677232 \nL 186.583817 50.135288 \nL 191.907773 56.513764 \nL 197.231738 64.951394 \nL 202.555694 83.891197 \nL 207.87965 115.307147 \nL 213.203615 139.2733 \nL 218.527571 151.414109 \nL 223.851527 158.147943 \nL 229.175492 162.668795 \nL 234.499448 166.028298 \nL 239.823414 168.374901 \nL 245.147369 169.734354 \nL 250.471325 170.260462 \nL 255.795281 170.181734 \nL 261.119256 169.712187 \nL 266.443212 169.012161 \nL 271.767168 168.187987 \nL 277.091123 167.305838 \nL 282.415079 166.406856 \nL 287.739054 165.517348 \nL 293.06301 164.655513 \nL 298.386966 163.834353 \nL 303.710922 163.063788 \nL 309.034877 162.350746 \nL 314.358833 161.700079 \nL 319.682808 161.114314 \nL 325.006764 160.594425 \nL 330.33072 160.1397 \nL 335.654676 159.748345 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 74.780672 185.964 \nL 74.780672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 335.654672 185.964 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 81.780672 65.234375 \nL 180.508797 65.234375 \nQ 182.508797 65.234375 182.508797 63.234375 \nL 182.508797 20.2 \nQ 182.508797 18.2 180.508797 18.2 \nL 81.780672 18.2 \nQ 79.780672 18.2 79.780672 20.2 \nL 79.780672 63.234375 \nQ 79.780672 65.234375 81.780672 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_23\"/>\n    <g id=\"line2d_24\">\n     <g>\n      <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"93.780672\" xlink:href=\"#m4af95c9f67\" y=\"26.298437\"/>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- train -->\n     <defs>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 29.798437)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_25\">\n     <path d=\"M 83.780672 40.976562 \nL 103.780672 40.976562 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:11.1,4.8;stroke-dashoffset:0;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_26\"/>\n    <g id=\"text_14\">\n     <!-- $f(x)$ -->\n     <defs>\n      <path d=\"M 31 75.875 \nQ 24.46875 64.65625 21.28125 53.65625 \nQ 18.109375 42.671875 18.109375 31.390625 \nQ 18.109375 20.125 21.3125 9.0625 \nQ 24.515625 -2 31 -13.1875 \nL 23.1875 -13.1875 \nQ 15.875 -1.703125 12.234375 9.375 \nQ 8.59375 20.453125 8.59375 31.390625 \nQ 8.59375 42.28125 12.203125 53.3125 \nQ 15.828125 64.359375 23.1875 75.875 \nz\n\" id=\"DejaVuSans-40\"/>\n      <path d=\"M 8.015625 75.875 \nL 15.828125 75.875 \nQ 23.140625 64.359375 26.78125 53.3125 \nQ 30.421875 42.28125 30.421875 31.390625 \nQ 30.421875 20.453125 26.78125 9.375 \nQ 23.140625 -1.703125 15.828125 -13.1875 \nL 8.015625 -13.1875 \nQ 14.5 -2 17.703125 9.0625 \nQ 20.90625 20.125 20.90625 31.390625 \nQ 20.90625 42.671875 17.703125 53.65625 \nQ 14.5 64.65625 8.015625 75.875 \nz\n\" id=\"DejaVuSans-41\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 44.476562)scale(0.1 -0.1)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n      <use transform=\"translate(35.205078 0.015625)\" xlink:href=\"#DejaVuSans-40\"/>\n      <use transform=\"translate(74.21875 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n      <use transform=\"translate(133.398438 0.015625)\" xlink:href=\"#DejaVuSans-41\"/>\n     </g>\n    </g>\n    <g id=\"line2d_27\">\n     <path d=\"M 83.780672 55.654687 \nL 103.780672 55.654687 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_28\"/>\n    <g id=\"text_15\">\n     <!-- random draw -->\n     <defs>\n      <path d=\"M 45.40625 46.390625 \nL 45.40625 75.984375 \nL 54.390625 75.984375 \nL 54.390625 0 \nL 45.40625 0 \nL 45.40625 8.203125 \nQ 42.578125 3.328125 38.25 0.953125 \nQ 33.9375 -1.421875 27.875 -1.421875 \nQ 17.96875 -1.421875 11.734375 6.484375 \nQ 5.515625 14.40625 5.515625 27.296875 \nQ 5.515625 40.1875 11.734375 48.09375 \nQ 17.96875 56 27.875 56 \nQ 33.9375 56 38.25 53.625 \nQ 42.578125 51.265625 45.40625 46.390625 \nz\nM 14.796875 27.296875 \nQ 14.796875 17.390625 18.875 11.75 \nQ 22.953125 6.109375 30.078125 6.109375 \nQ 37.203125 6.109375 41.296875 11.75 \nQ 45.40625 17.390625 45.40625 27.296875 \nQ 45.40625 37.203125 41.296875 42.84375 \nQ 37.203125 48.484375 30.078125 48.484375 \nQ 22.953125 48.484375 18.875 42.84375 \nQ 14.796875 37.203125 14.796875 27.296875 \nz\n\" id=\"DejaVuSans-100\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 52 44.1875 \nQ 55.375 50.25 60.0625 53.125 \nQ 64.75 56 71.09375 56 \nQ 79.640625 56 84.28125 50.015625 \nQ 88.921875 44.046875 88.921875 33.015625 \nL 88.921875 0 \nL 79.890625 0 \nL 79.890625 32.71875 \nQ 79.890625 40.578125 77.09375 44.375 \nQ 74.3125 48.1875 68.609375 48.1875 \nQ 61.625 48.1875 57.5625 43.546875 \nQ 53.515625 38.921875 53.515625 30.90625 \nL 53.515625 0 \nL 44.484375 0 \nL 44.484375 32.71875 \nQ 44.484375 40.625 41.703125 44.40625 \nQ 38.921875 48.1875 33.109375 48.1875 \nQ 26.21875 48.1875 22.15625 43.53125 \nQ 18.109375 38.875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.1875 51.21875 25.484375 53.609375 \nQ 29.78125 56 35.6875 56 \nQ 41.65625 56 45.828125 52.96875 \nQ 50 49.953125 52 44.1875 \nz\n\" id=\"DejaVuSans-109\"/>\n      <path id=\"DejaVuSans-32\"/>\n      <path d=\"M 4.203125 54.6875 \nL 13.1875 54.6875 \nL 24.421875 12.015625 \nL 35.59375 54.6875 \nL 46.1875 54.6875 \nL 57.421875 12.015625 \nL 68.609375 54.6875 \nL 77.59375 54.6875 \nL 63.28125 0 \nL 52.6875 0 \nL 40.921875 44.828125 \nL 29.109375 0 \nL 18.5 0 \nz\n\" id=\"DejaVuSans-119\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"41.113281\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"102.392578\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"165.771484\" xlink:href=\"#DejaVuSans-100\"/>\n      <use x=\"229.248047\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"290.429688\" xlink:href=\"#DejaVuSans-109\"/>\n      <use x=\"387.841797\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"419.628906\" xlink:href=\"#DejaVuSans-100\"/>\n      <use x=\"483.105469\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"524.21875\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"585.498047\" xlink:href=\"#DejaVuSans-119\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p0fa4f723df\">\n   <rect height=\"172.764\" width=\"260.874\" x=\"74.780672\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 510x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_fn(train, test)\n",
    "\n",
    "for p in prior_draws:\n",
    "  plt.plot(test_xs, p, linewidth=3, color=[1, 0.65, 0.65])\n",
    "\n",
    "legend(['train', '$f(x)$', 'random draw'], loc='upper left')\n",
    "\n",
    "finalize_plot((0.85, 0.6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HUh41cmZl4If"
   },
   "source": [
    "Next we can look at the exact prior over functions in the infinite-width limit using the `kernel_fn`. The kernel function has the signature `kernel = kernel_fn(x_1, x_2)` which computes the kernel between two sets of inputs `x_1` and `x_2`. The `kernel_fn` can compute two different kernels: the NNGP kernel which describes the Bayesian infinite network and the NT kernel which describes how this network evolves under gradient descent. We would like to visualize the standard deviation of the infinite-width function distribution at each $x$. This is given by the diagonal of the NNGP kernel. We compute this now and then plot it compared with the draws above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YLo8K96Pl2xw"
   },
   "outputs": [],
   "source": [
    "kernel = kernel_fn(test_xs, test_xs, 'nngp')\n",
    "std_dev = jnp.sqrt(jnp.diag(kernel))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 341
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 643,
     "status": "ok",
     "timestamp": 1588653670279,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "60k0Lq3loAz9",
    "outputId": "127e49c1-6df8-4c7c-e156-9358548575c9"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"239.859969pt\" version=\"1.1\" viewBox=\"0 0 348.854672 239.859969\" width=\"348.854672pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 239.859969 \nL 348.854672 239.859969 \nL 348.854672 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \nL 335.654672 13.2 \nL 74.780672 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 122.178899 185.964 \nL 122.178899 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- −2 -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(111.638235 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 205.217672 185.964 \nL 205.217672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(200.668484 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 288.256445 185.964 \nL 288.256445 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(283.707258 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_4\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(199.217672 228.500594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_5\">\n      <!-- −1.5 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 191.396883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780672 157.17 \nL 335.654672 157.17 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_6\">\n      <!-- −1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 162.602883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780672 128.376 \nL 335.654672 128.376 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- −0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 133.808883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780672 99.582 \nL 335.654672 99.582 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 105.014883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780672 70.788 \nL 335.654672 70.788 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 76.220883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780672 41.994 \nL 335.654672 41.994 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 47.426883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_10\">\n      <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 18.632883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 103.182)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"PolyCollection_1\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 231.975317 \nL 74.780668 -32.811317 \nL 80.104624 -32.686854 \nL 85.42859 -32.550377 \nL 90.752545 -32.400116 \nL 96.076501 -32.233886 \nL 101.400467 -32.049025 \nL 106.724422 -31.84225 \nL 112.048388 -31.609471 \nL 117.372344 -31.345552 \nL 122.696304 -31.043861 \nL 128.020265 -30.695887 \nL 133.34422 -30.290233 \nL 138.668181 -29.811727 \nL 143.992142 -29.239321 \nL 149.316097 -28.543345 \nL 154.640058 -27.68059 \nL 159.964019 -26.585975 \nL 165.287974 -25.157309 \nL 170.61194 -23.226219 \nL 175.935896 -20.498744 \nL 181.259861 -16.424833 \nL 186.583817 -9.88946 \nL 191.907773 1.559005 \nL 197.231738 23.449056 \nL 202.555694 63.664833 \nL 207.87965 63.664833 \nL 213.203615 23.449001 \nL 218.527571 1.559005 \nL 223.851527 -9.88946 \nL 229.175492 -16.424846 \nL 234.499448 -20.498744 \nL 239.823414 -23.226219 \nL 245.147369 -25.157309 \nL 250.471325 -26.585975 \nL 255.795281 -27.68059 \nL 261.119256 -28.543345 \nL 266.443212 -29.239321 \nL 271.767168 -29.811727 \nL 277.091123 -30.290233 \nL 282.415079 -30.695887 \nL 287.739054 -31.043861 \nL 293.06301 -31.345552 \nL 298.386966 -31.609471 \nL 303.710922 -31.84225 \nL 309.034877 -32.049025 \nL 314.358833 -32.233886 \nL 319.682808 -32.400116 \nL 325.006764 -32.550377 \nL 330.33072 -32.686854 \nL 335.654676 -32.811317 \nL 335.654676 231.975317 \nL 335.654676 231.975317 \nL 330.33072 231.850854 \nL 325.006764 231.714377 \nL 319.682808 231.564116 \nL 314.358833 231.397886 \nL 309.034877 231.213025 \nL 303.710922 231.00625 \nL 298.386966 230.773471 \nL 293.06301 230.509552 \nL 287.739054 230.207861 \nL 282.415079 229.859887 \nL 277.091123 229.454233 \nL 271.767168 228.975727 \nL 266.443212 228.403321 \nL 261.119256 227.707345 \nL 255.795281 226.84459 \nL 250.471325 225.749975 \nL 245.147369 224.321309 \nL 239.823414 222.390219 \nL 234.499448 219.662744 \nL 229.175492 215.588846 \nL 223.851527 209.05346 \nL 218.527571 197.604995 \nL 213.203615 175.714999 \nL 207.87965 135.499167 \nL 202.555694 135.499167 \nL 197.231738 175.714944 \nL 191.907773 197.604995 \nL 186.583817 209.05346 \nL 181.259861 215.588833 \nL 175.935896 219.662744 \nL 170.61194 222.390219 \nL 165.287974 224.321309 \nL 159.964019 225.749975 \nL 154.640058 226.84459 \nL 149.316097 227.707345 \nL 143.992142 228.403321 \nL 138.668181 228.975727 \nL 133.34422 229.454233 \nL 128.020265 229.859887 \nL 122.696304 230.207861 \nL 117.372344 230.509552 \nL 112.048388 230.773471 \nL 106.724422 231.00625 \nL 101.400467 231.213025 \nL 96.076501 231.397886 \nL 90.752545 231.564116 \nL 85.42859 231.714377 \nL 80.104624 231.850854 \nL 74.780668 231.975317 \nz\n\" style=\"fill:#4c72b0;fill-opacity:0.4;stroke:#ffffff;stroke-opacity:0.4;\"/>\n   </g>\n   <g id=\"line2d_11\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"md0e1ee1990\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#pa67f7adc25)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"332.990361\" xlink:href=\"#md0e1ee1990\" y=\"89.211043\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"312.709007\" xlink:href=\"#md0e1ee1990\" y=\"61.934552\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"113.996599\" xlink:href=\"#md0e1ee1990\" y=\"143.295353\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"197.320621\" xlink:href=\"#md0e1ee1990\" y=\"113.447504\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"255.905476\" xlink:href=\"#md0e1ee1990\" y=\"35.617422\"/>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 99.581995 \nL 80.104624 106.946181 \nL 85.42859 114.189459 \nL 90.752545 121.192871 \nL 96.076501 127.841434 \nL 101.400467 134.025987 \nL 106.724422 139.644962 \nL 112.048388 144.606113 \nL 117.372344 148.827959 \nL 122.696304 152.241194 \nL 128.020265 154.789766 \nL 133.34422 156.431825 \nL 138.668181 157.140412 \nL 143.992142 156.903891 \nL 149.316097 155.726148 \nL 154.640058 153.626521 \nL 159.964019 150.639481 \nL 165.287974 146.814083 \nL 170.61194 142.213126 \nL 175.935896 136.912177 \nL 181.259861 130.998257 \nL 186.583817 124.568494 \nL 191.907773 117.728453 \nL 197.231738 110.590436 \nL 202.555694 103.271674 \nL 207.87965 95.892326 \nL 213.203615 88.57355 \nL 218.527571 81.435547 \nL 223.851527 74.595506 \nL 229.175492 68.165732 \nL 234.499448 62.251823 \nL 239.823414 56.950863 \nL 245.147369 52.349917 \nL 250.471325 48.524519 \nL 255.795281 45.537482 \nL 261.119256 43.437849 \nL 266.443212 42.260106 \nL 271.767168 42.023588 \nL 277.091123 42.732175 \nL 282.415079 44.374234 \nL 287.739054 46.922813 \nL 293.06301 50.336048 \nL 298.386966 54.557898 \nL 303.710922 59.519038 \nL 309.034877 65.138013 \nL 314.358833 71.322554 \nL 319.682808 77.971141 \nL 325.006764 84.974554 \nL 330.33072 92.217819 \nL 335.654676 99.582005 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:11.1,4.8;stroke-dashoffset:0;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 82.218899 -1 \nL 85.42859 -0.500605 \nL 90.752545 0.368521 \nL 96.076501 1.273386 \nL 101.400467 2.206267 \nL 106.724422 3.156517 \nL 112.048388 4.110598 \nL 117.372344 5.05274 \nL 122.696304 5.967058 \nL 128.020265 6.841065 \nL 133.34422 7.671232 \nL 138.668181 8.470362 \nL 143.992142 9.275362 \nL 149.316097 10.154352 \nL 154.640058 11.211388 \nL 159.964019 12.590942 \nL 165.287974 14.487899 \nL 170.61194 17.173208 \nL 175.935896 21.053011 \nL 181.259861 26.791952 \nL 186.583817 35.45814 \nL 191.907773 48.299247 \nL 197.231738 66.258189 \nL 202.555694 90.527189 \nL 207.87965 117.010954 \nL 213.203615 137.913785 \nL 218.527571 153.34463 \nL 223.851527 164.027679 \nL 229.175492 170.755492 \nL 234.499448 175.004057 \nL 239.823414 177.858726 \nL 245.147369 179.889147 \nL 250.471325 181.396802 \nL 255.795281 182.564687 \nL 261.119256 183.521507 \nL 266.443212 184.366543 \nL 271.767168 185.175243 \nL 277.091123 185.999938 \nL 282.415079 186.869799 \nL 287.739054 187.794566 \nL 293.06301 188.769955 \nL 298.386966 189.783473 \nL 303.710922 190.81911 \nL 309.034877 191.860603 \nL 314.358833 192.893494 \nL 319.682808 193.90601 \nL 325.006764 194.889095 \nL 330.33072 195.836489 \nL 335.654676 196.744025 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_14\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 126.684161 \nL 80.104624 127.201456 \nL 85.42859 127.759893 \nL 90.752545 128.359694 \nL 96.076501 129.001023 \nL 101.400467 129.683849 \nL 106.724422 130.407985 \nL 112.048388 131.172859 \nL 117.372344 131.976407 \nL 122.696304 132.814053 \nL 128.020265 133.676341 \nL 133.34422 134.545265 \nL 138.668181 135.390057 \nL 143.992142 136.161123 \nL 149.316097 136.785607 \nL 154.640058 137.166455 \nL 159.964019 137.196115 \nL 165.287974 136.798678 \nL 170.61194 136.013724 \nL 175.935896 135.083407 \nL 181.259861 134.396503 \nL 186.583817 134.110932 \nL 191.907773 133.571711 \nL 197.231738 130.555614 \nL 202.555694 119.705694 \nL 207.87965 100.570667 \nL 213.203615 87.612371 \nL 218.527571 82.483249 \nL 223.851527 79.655568 \nL 229.175492 76.990155 \nL 234.499448 74.252045 \nL 239.823414 71.764793 \nL 245.147369 69.857343 \nL 250.471325 68.671084 \nL 255.795281 68.170648 \nL 261.119256 68.227418 \nL 266.443212 68.689314 \nL 271.767168 69.417721 \nL 277.091123 70.301273 \nL 282.415079 71.257549 \nL 287.739054 72.229303 \nL 293.06301 73.179347 \nL 298.386966 74.085246 \nL 303.710922 74.934724 \nL 309.034877 75.722226 \nL 314.358833 76.446344 \nL 319.682808 77.10812 \nL 325.006764 77.709827 \nL 330.33072 78.254323 \nL 335.654676 78.744742 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_15\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 192.79315 -1 \nL 197.231738 19.483543 \nL 202.555694 65.062264 \nL 207.87965 126.664074 \nL 213.203615 174.913535 \nL 218.527571 200.798584 \nL 223.851527 214.530699 \nL 229.175492 224.103083 \nL 234.499448 232.190947 \nL 239.823414 239.121629 \nL 241.442736 240.859969 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_16\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 65.883252 \nL 80.104624 64.867774 \nL 85.42859 63.77685 \nL 90.752545 62.60728 \nL 96.076501 61.356362 \nL 101.400467 60.02232 \nL 106.724422 58.604957 \nL 112.048388 57.106037 \nL 117.372344 55.5305 \nL 122.696304 53.88735 \nL 128.020265 52.191434 \nL 133.34422 50.465258 \nL 138.668181 48.74153 \nL 143.992142 47.067318 \nL 149.316097 45.510269 \nL 154.640058 44.169495 \nL 159.964019 43.194113 \nL 165.287974 42.811724 \nL 170.61194 43.369198 \nL 175.935896 45.385487 \nL 181.259861 49.588358 \nL 186.583817 56.796802 \nL 191.907773 67.42583 \nL 197.231738 81.620223 \nL 202.555694 101.267863 \nL 207.87965 121.462723 \nL 213.203615 133.418728 \nL 218.527571 141.985503 \nL 223.851527 149.728945 \nL 229.175492 155.220367 \nL 234.499448 158.003476 \nL 239.823414 158.670221 \nL 245.147369 157.949221 \nL 250.471325 156.409218 \nL 255.795281 154.439361 \nL 261.119256 152.28525 \nL 266.443212 150.091432 \nL 271.767168 147.938691 \nL 277.091123 145.869885 \nL 282.415079 143.906738 \nL 287.739054 142.058862 \nL 293.06301 140.329113 \nL 298.386966 138.716272 \nL 303.710922 137.217084 \nL 309.034877 135.826728 \nL 314.358833 134.540003 \nL 319.682808 133.351327 \nL 325.006764 132.25524 \nL 330.33072 131.246359 \nL 335.654676 130.319643 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_17\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 2.876074 \nL 80.104624 1.697088 \nL 85.42859 0.489695 \nL 90.752545 -0.747561 \nL 91.810449 -1 \nM 193.126302 -1 \nL 197.231738 22.599077 \nL 202.555694 69.659778 \nL 207.87965 126.530999 \nL 213.203615 172.217653 \nL 218.527571 200.627253 \nL 223.851527 216.512179 \nL 229.175492 223.861997 \nL 234.499448 225.65959 \nL 239.823414 224.178832 \nL 245.147369 221.046472 \nL 250.471325 217.318229 \nL 255.795281 213.602479 \nL 261.119256 210.197036 \nL 266.443212 207.207173 \nL 271.767168 204.632164 \nL 277.091123 202.421565 \nL 282.415079 200.508461 \nL 287.739054 198.826832 \nL 293.06301 197.319451 \nL 298.386966 195.940467 \nL 303.710922 194.655101 \nL 309.034877 193.43811 \nL 314.358833 192.271969 \nL 319.682808 191.145055 \nL 325.006764 190.049987 \nL 330.33072 188.982455 \nL 335.654676 187.940283 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_18\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 64.033235 \nL 80.104624 64.116579 \nL 85.42859 64.166378 \nL 90.752545 64.174095 \nL 96.076501 64.129036 \nL 101.400467 64.017785 \nL 106.724422 63.823879 \nL 112.048388 63.527523 \nL 117.372344 63.105344 \nL 122.696304 62.530683 \nL 128.020265 61.774278 \nL 133.34422 60.805485 \nL 138.668181 59.594073 \nL 143.992142 58.111787 \nL 149.316097 56.333794 \nL 154.640058 54.239302 \nL 159.964019 51.815897 \nL 165.287974 49.078973 \nL 170.61194 46.134406 \nL 175.935896 43.318318 \nL 181.259861 41.393791 \nL 186.583817 41.611742 \nL 191.907773 45.562158 \nL 197.231738 56.184219 \nL 202.555694 80.674506 \nL 207.87965 117.783581 \nL 213.203615 143.136918 \nL 218.527571 154.822065 \nL 223.851527 160.343358 \nL 229.175492 162.192418 \nL 234.499448 161.608479 \nL 239.823414 159.80814 \nL 245.147369 157.670817 \nL 250.471325 155.629585 \nL 255.795281 153.82422 \nL 261.119256 152.26542 \nL 266.443212 150.928474 \nL 271.767168 149.788094 \nL 277.091123 148.824516 \nL 282.415079 148.020831 \nL 287.739054 147.360594 \nL 293.06301 146.827054 \nL 298.386966 146.403723 \nL 303.710922 146.07483 \nL 309.034877 145.826127 \nL 314.358833 145.645076 \nL 319.682808 145.520956 \nL 325.006764 145.444717 \nL 330.33072 145.408871 \nL 335.654676 145.407189 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_19\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 36.491978 \nL 80.104624 35.855398 \nL 85.42859 35.214376 \nL 90.752545 34.570622 \nL 96.076501 33.925516 \nL 101.400467 33.279936 \nL 106.724422 32.634095 \nL 112.048388 31.987389 \nL 117.372344 31.33828 \nL 122.696304 30.684572 \nL 128.020265 30.023628 \nL 133.34422 29.353533 \nL 138.668181 28.674974 \nL 143.992142 27.99535 \nL 149.316097 27.336541 \nL 154.640058 26.74816 \nL 159.964019 26.327224 \nL 165.287974 26.239318 \nL 170.61194 26.730249 \nL 175.935896 28.111129 \nL 181.259861 30.688465 \nL 186.583817 34.596455 \nL 191.907773 39.950124 \nL 197.231738 49.697707 \nL 202.555694 74.526584 \nL 207.87965 113.891882 \nL 213.203615 140.614362 \nL 218.527571 152.967664 \nL 223.851527 160.75828 \nL 229.175492 166.482969 \nL 234.499448 170.045765 \nL 239.823414 171.761212 \nL 245.147369 172.243974 \nL 250.471325 172.037282 \nL 255.795281 171.509121 \nL 261.119256 170.869912 \nL 266.443212 170.220501 \nL 271.767168 169.597912 \nL 277.091123 169.008048 \nL 282.415079 168.444217 \nL 287.739054 167.896752 \nL 293.06301 167.356516 \nL 298.386966 166.816616 \nL 303.710922 166.272391 \nL 309.034877 165.721184 \nL 314.358833 165.162014 \nL 319.682808 164.595135 \nL 325.006764 164.021638 \nL 330.33072 163.443314 \nL 335.654676 162.862272 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_20\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 115.269391 \nL 80.104624 115.416521 \nL 85.42859 115.622295 \nL 90.752545 115.893731 \nL 96.076501 116.237679 \nL 101.400467 116.660367 \nL 106.724422 117.166599 \nL 112.048388 117.757999 \nL 117.372344 118.432091 \nL 122.696304 119.17974 \nL 128.020265 119.983518 \nL 133.34422 120.815686 \nL 138.668181 121.63809 \nL 143.992142 122.404436 \nL 149.316097 123.068394 \nL 154.640058 123.59946 \nL 159.964019 124.009081 \nL 165.287974 124.382651 \nL 170.61194 124.90411 \nL 175.935896 125.846863 \nL 181.259861 127.493721 \nL 186.583817 129.924163 \nL 191.907773 132.266914 \nL 197.231738 129.643126 \nL 202.555694 109.770623 \nL 207.87965 76.366854 \nL 213.203615 59.321037 \nL 218.527571 57.57648 \nL 223.851527 60.4517 \nL 229.175492 63.718926 \nL 234.499448 66.194447 \nL 239.823414 67.746904 \nL 245.147369 68.663766 \nL 250.471325 69.294579 \nL 255.795281 69.887254 \nL 261.119256 70.564279 \nL 266.443212 71.355835 \nL 271.767168 72.241137 \nL 277.091123 73.179275 \nL 282.415079 74.127318 \nL 287.739054 75.049037 \nL 293.06301 75.917594 \nL 298.386966 76.715496 \nL 303.710922 77.433114 \nL 309.034877 78.066968 \nL 314.358833 78.617883 \nL 319.682808 79.089461 \nL 325.006764 79.486966 \nL 330.33072 79.816319 \nL 335.654676 80.08365 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_21\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 168.741356 \nL 80.104624 168.412782 \nL 85.42859 168.06695 \nL 90.752545 167.70219 \nL 96.076501 167.315985 \nL 101.400467 166.904262 \nL 106.724422 166.460898 \nL 112.048388 165.976687 \nL 117.372344 165.438174 \nL 122.696304 164.825814 \nL 128.020265 164.112002 \nL 133.34422 163.258405 \nL 138.668181 162.21383 \nL 143.992142 160.912303 \nL 149.316097 159.273327 \nL 154.640058 157.206254 \nL 159.964019 154.618576 \nL 165.287974 151.428025 \nL 170.61194 147.565879 \nL 175.935896 142.949102 \nL 181.259861 137.411495 \nL 186.583817 130.699056 \nL 191.907773 122.700114 \nL 197.231738 114.262771 \nL 202.555694 109.435413 \nL 207.87965 107.001822 \nL 213.203615 101.160947 \nL 218.527571 91.328859 \nL 223.851527 81.151537 \nL 229.175492 71.970734 \nL 234.499448 64.211245 \nL 239.823414 57.890438 \nL 245.147369 52.842146 \nL 250.471325 48.852884 \nL 255.795281 45.712108 \nL 261.119256 43.232835 \nL 266.443212 41.26028 \nL 271.767168 39.673121 \nL 277.091123 38.379427 \nL 282.415079 37.311181 \nL 287.739054 36.41844 \nL 293.06301 35.664482 \nL 298.386966 35.022073 \nL 303.710922 34.470846 \nL 309.034877 33.995148 \nL 314.358833 33.582965 \nL 319.682808 33.224714 \nL 325.006764 32.912678 \nL 330.33072 32.640528 \nL 335.654676 32.403025 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_22\">\n    <path clip-path=\"url(#pa67f7adc25)\" d=\"M 74.780668 34.654094 \nL 80.104624 34.243065 \nL 85.42859 33.764648 \nL 90.752545 33.217499 \nL 96.076501 32.601932 \nL 101.400467 31.920792 \nL 106.724422 31.179898 \nL 112.048388 30.389163 \nL 117.372344 29.563706 \nL 122.696304 28.725294 \nL 128.020265 27.904395 \nL 133.34422 27.143146 \nL 138.668181 26.499763 \nL 143.992142 26.055926 \nL 149.316097 25.928133 \nL 154.640058 26.283432 \nL 159.964019 27.35463 \nL 165.287974 29.437218 \nL 170.61194 32.829007 \nL 175.935896 37.664909 \nL 181.259861 43.677232 \nL 186.583817 50.135288 \nL 191.907773 56.513764 \nL 197.231738 64.951394 \nL 202.555694 83.891197 \nL 207.87965 115.307147 \nL 213.203615 139.2733 \nL 218.527571 151.414109 \nL 223.851527 158.147943 \nL 229.175492 162.668795 \nL 234.499448 166.028298 \nL 239.823414 168.374901 \nL 245.147369 169.734354 \nL 250.471325 170.260462 \nL 255.795281 170.181734 \nL 261.119256 169.712187 \nL 266.443212 169.012161 \nL 271.767168 168.187987 \nL 277.091123 167.305838 \nL 282.415079 166.406856 \nL 287.739054 165.517348 \nL 293.06301 164.655513 \nL 298.386966 163.834353 \nL 303.710922 163.063788 \nL 309.034877 162.350746 \nL 314.358833 161.700079 \nL 319.682808 161.114314 \nL 325.006764 160.594425 \nL 330.33072 160.1397 \nL 335.654676 159.748345 \n\" style=\"fill:none;stroke:#ffa6a6;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 74.780672 185.964 \nL 74.780672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 335.654672 185.964 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pa67f7adc25\">\n   <rect height=\"172.764\" width=\"260.874\" x=\"74.780672\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 510x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_fn(train, test)\n",
    "\n",
    "plt.fill_between(\n",
    "    jnp.reshape(test_xs, (-1,)), 2 * std_dev, -2 * std_dev, alpha=0.4)\n",
    "\n",
    "for p in prior_draws:\n",
    "  plt.plot(test_xs, p, linewidth=3, color=[1, 0.65, 0.65])\n",
    "\n",
    "finalize_plot((0.85, 0.6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5nG_q22a7aJw"
   },
   "source": [
    "## Infinite Width Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c3lXqB1t3U9g"
   },
   "source": [
    "We can use the infinite-width GP defined above to perform exact Bayesian inference using the infinite width network. To do this, we will use the function `neural_tangents.predict.gradient_descent_mse_ensemble` that performs this inference exactly. `predict_fn = nt.predict.gradient_descent_mse_ensemble(kernel_fn, train_xs, train_ys); mean, cov = predict_fn(x_test=test_xs, get='ntk', compute_cov=True)` computes the mean and covariance of the network evaluated on the test points after training. This `predict_fn` function includes two different modes: in \"NNGP\" mode we compute the Bayesian posterior (which is equivalent to gradient descent with all but the last-layer weights frozen), in \"NTK\" mode we compute the distribution of networks after gradient descent training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rdTH1Ern4Bt3"
   },
   "source": [
    "We want to do exact Bayesian inference so we'll start off using the \"NNGP\" setting. We will compute and plot these predictions now; we will be concerned with the standard deviation of the predictions on test points which will be given by the diagonal of the covariance matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JeKzXr6Y4Erl"
   },
   "outputs": [],
   "source": [
    "predict_fn = nt.predict.gradient_descent_mse_ensemble(kernel_fn, train_xs,\n",
    "                                                      train_ys, diag_reg=1e-4)\n",
    "\n",
    "nngp_mean, nngp_covariance = predict_fn(x_test=test_xs, get='nngp',\n",
    "                                        compute_cov=True)\n",
    "\n",
    "nngp_mean = jnp.reshape(nngp_mean, (-1,))\n",
    "nngp_std = jnp.sqrt(jnp.diag(nngp_covariance))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 341
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 586,
     "status": "ok",
     "timestamp": 1588653671515,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "elaecFoO4Erp",
    "outputId": "9122f676-c8b9-4fac-e08a-f83de9ebb87d"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"239.859969pt\" version=\"1.1\" viewBox=\"0 0 348.854672 239.859969\" width=\"348.854672pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 239.859969 \nL 348.854672 239.859969 \nL 348.854672 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \nL 335.654672 13.2 \nL 74.780672 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 122.178899 185.964 \nL 122.178899 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- −2 -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(111.638235 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 205.217672 185.964 \nL 205.217672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(200.668484 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 288.256445 185.964 \nL 288.256445 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(283.707258 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_4\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(199.217672 228.500594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_5\">\n      <!-- −1.5 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 191.396883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 74.780672 157.17 \nL 335.654672 157.17 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_6\">\n      <!-- −1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 162.602883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 74.780672 128.376 \nL 335.654672 128.376 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- −0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 133.808883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 74.780672 99.582 \nL 335.654672 99.582 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 105.014883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 74.780672 70.788 \nL 335.654672 70.788 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 76.220883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 74.780672 41.994 \nL 335.654672 41.994 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 47.426883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_10\">\n      <path clip-path=\"url(#p6966447886)\" d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 18.632883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 103.182)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"PolyCollection_1\">\n    <defs>\n     <path d=\"M 74.780668 -139.193632 \nL 74.780668 -132.073324 \nL 80.104624 -127.303292 \nL 85.42859 -122.424214 \nL 90.752545 -117.424722 \nL 96.076501 -112.3211 \nL 101.400467 -107.229397 \nL 106.724422 -102.154016 \nL 112.048388 -97.140238 \nL 117.372344 -92.177553 \nL 122.696304 -87.41615 \nL 128.020265 -83.103305 \nL 133.34422 -79.459721 \nL 138.668181 -76.720397 \nL 143.992142 -75.139732 \nL 149.316097 -74.884621 \nL 154.640058 -76.088465 \nL 159.964019 -78.885997 \nL 165.287974 -83.120849 \nL 170.61194 -88.608334 \nL 175.935896 -94.955842 \nL 181.259861 -101.703035 \nL 186.583817 -108.65665 \nL 191.907773 -116.226046 \nL 197.231738 -125.17209 \nL 202.555694 -122.704802 \nL 207.87965 -121.695344 \nL 213.203615 -126.305528 \nL 218.527571 -134.132426 \nL 223.851527 -143.872137 \nL 229.175492 -155.525916 \nL 234.499448 -168.042366 \nL 239.823414 -179.96643 \nL 245.147369 -190.232923 \nL 250.471325 -198.210895 \nL 255.795281 -203.430861 \nL 261.119256 -205.807113 \nL 266.443212 -206.208141 \nL 271.767168 -205.228557 \nL 277.091123 -203.116456 \nL 282.415079 -200.106081 \nL 287.739054 -196.333239 \nL 293.06301 -192.008891 \nL 298.386966 -187.271163 \nL 303.710922 -182.255689 \nL 309.034877 -177.028659 \nL 314.358833 -171.699996 \nL 319.682808 -166.299695 \nL 325.006764 -160.801567 \nL 330.33072 -155.258654 \nL 335.654676 -149.757693 \nL 335.654676 -152.28654 \nL 335.654676 -152.28654 \nL 330.33072 -157.306532 \nL 325.006764 -162.569277 \nL 319.682808 -168.072763 \nL 314.358833 -173.754039 \nL 309.034877 -179.492894 \nL 303.710922 -185.182242 \nL 298.386966 -190.677176 \nL 293.06301 -195.834749 \nL 287.739054 -200.491827 \nL 282.415079 -204.474598 \nL 277.091123 -207.508081 \nL 271.767168 -209.387141 \nL 266.443212 -209.819665 \nL 261.119256 -208.617902 \nL 255.795281 -205.858896 \nL 250.471325 -202.281086 \nL 245.147369 -197.795249 \nL 239.823414 -192.396744 \nL 234.499448 -186.50206 \nL 229.175492 -180.816639 \nL 223.851527 -176.214038 \nL 218.527571 -173.119999 \nL 213.203615 -170.198794 \nL 207.87965 -160.233159 \nL 202.555694 -140.820163 \nL 197.231738 -127.624105 \nL 191.907773 -130.342638 \nL 186.583817 -129.066807 \nL 181.259861 -123.075951 \nL 175.935896 -114.56148 \nL 170.61194 -105.526061 \nL 165.287974 -97.363461 \nL 159.964019 -90.817262 \nL 154.640058 -86.121918 \nL 149.316097 -83.334601 \nL 143.992142 -82.229765 \nL 138.668181 -82.597229 \nL 133.34422 -84.243613 \nL 128.020265 -86.924199 \nL 122.696304 -90.447807 \nL 117.372344 -94.712649 \nL 112.048388 -99.589026 \nL 106.724422 -104.935394 \nL 101.400467 -110.58395 \nL 96.076501 -116.372597 \nL 90.752545 -122.223138 \nL 85.42859 -127.991519 \nL 80.104624 -133.654484 \nL 74.780668 -139.193632 \nz\n\" id=\"mcefd0b0820\" style=\"stroke:#ff0000;stroke-opacity:0.2;\"/>\n    </defs>\n    <g clip-path=\"url(#p6966447886)\">\n     <use style=\"fill:#ff0000;fill-opacity:0.2;stroke:#ff0000;stroke-opacity:0.2;\" x=\"0\" xlink:href=\"#mcefd0b0820\" y=\"239.859969\"/>\n    </g>\n   </g>\n   <g id=\"line2d_11\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"m3ef57eb98a\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#p6966447886)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"332.990361\" xlink:href=\"#m3ef57eb98a\" y=\"89.211043\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"312.709007\" xlink:href=\"#m3ef57eb98a\" y=\"61.934552\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"113.996599\" xlink:href=\"#m3ef57eb98a\" y=\"143.295353\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"197.320621\" xlink:href=\"#m3ef57eb98a\" y=\"113.447504\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"255.905476\" xlink:href=\"#m3ef57eb98a\" y=\"35.617422\"/>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#p6966447886)\" d=\"M 74.780668 99.581995 \nL 80.104624 106.946181 \nL 85.42859 114.189459 \nL 90.752545 121.192871 \nL 96.076501 127.841434 \nL 101.400467 134.025987 \nL 106.724422 139.644962 \nL 112.048388 144.606113 \nL 117.372344 148.827959 \nL 122.696304 152.241194 \nL 128.020265 154.789766 \nL 133.34422 156.431825 \nL 138.668181 157.140412 \nL 143.992142 156.903891 \nL 149.316097 155.726148 \nL 154.640058 153.626521 \nL 159.964019 150.639481 \nL 165.287974 146.814083 \nL 170.61194 142.213126 \nL 175.935896 136.912177 \nL 181.259861 130.998257 \nL 186.583817 124.568494 \nL 191.907773 117.728453 \nL 197.231738 110.590436 \nL 202.555694 103.271674 \nL 207.87965 95.892326 \nL 213.203615 88.57355 \nL 218.527571 81.435547 \nL 223.851527 74.595506 \nL 229.175492 68.165732 \nL 234.499448 62.251823 \nL 239.823414 56.950863 \nL 245.147369 52.349917 \nL 250.471325 48.524519 \nL 255.795281 45.537482 \nL 261.119256 43.437849 \nL 266.443212 42.260106 \nL 271.767168 42.023588 \nL 277.091123 42.732175 \nL 282.415079 44.374234 \nL 287.739054 46.922813 \nL 293.06301 50.336048 \nL 298.386966 54.557898 \nL 303.710922 59.519038 \nL 309.034877 65.138013 \nL 314.358833 71.322554 \nL 319.682808 77.971141 \nL 325.006764 84.974554 \nL 330.33072 92.217819 \nL 335.654676 99.582005 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:11.1,4.8;stroke-dashoffset:0;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#p6966447886)\" d=\"M 74.780668 104.226491 \nL 80.104624 109.381081 \nL 85.42859 114.652102 \nL 90.752545 120.036039 \nL 96.076501 125.51312 \nL 101.400467 130.953295 \nL 106.724422 136.315264 \nL 112.048388 141.495337 \nL 117.372344 146.414868 \nL 122.696304 150.92799 \nL 128.020265 154.846217 \nL 133.34422 158.008302 \nL 138.668181 160.201155 \nL 143.992142 161.17522 \nL 149.316097 160.750357 \nL 154.640058 158.754777 \nL 159.964019 155.008341 \nL 165.287974 149.617814 \nL 170.61194 142.792771 \nL 175.935896 135.101308 \nL 181.259861 127.470476 \nL 186.583817 120.99824 \nL 191.907773 116.575627 \nL 197.231738 113.461871 \nL 202.555694 108.097486 \nL 207.87965 98.895717 \nL 213.203615 91.607807 \nL 218.527571 86.233756 \nL 223.851527 79.81688 \nL 229.175492 71.688691 \nL 234.499448 62.587756 \nL 239.823414 53.678382 \nL 245.147369 45.845883 \nL 250.471325 39.613978 \nL 255.795281 35.21509 \nL 261.119256 32.647461 \nL 266.443212 31.846066 \nL 271.767168 32.55212 \nL 277.091123 34.5477 \nL 282.415079 37.569629 \nL 287.739054 41.447434 \nL 293.06301 45.938149 \nL 298.386966 50.885799 \nL 303.710922 56.141003 \nL 309.034877 61.599192 \nL 314.358833 67.132951 \nL 319.682808 72.67374 \nL 325.006764 78.174547 \nL 330.33072 83.577376 \nL 335.654676 88.837852 \n\" style=\"fill:none;stroke:#c44e52;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 74.780672 185.964 \nL 74.780672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 335.654672 185.964 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 81.780672 65.234375 \nL 209.683797 65.234375 \nQ 211.683797 65.234375 211.683797 63.234375 \nL 211.683797 20.2 \nQ 211.683797 18.2 209.683797 18.2 \nL 81.780672 18.2 \nQ 79.780672 18.2 79.780672 20.2 \nL 79.780672 63.234375 \nQ 79.780672 65.234375 81.780672 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_14\"/>\n    <g id=\"line2d_15\">\n     <g>\n      <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"93.780672\" xlink:href=\"#m3ef57eb98a\" y=\"26.298437\"/>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- Train -->\n     <defs>\n      <path d=\"M -0.296875 72.90625 \nL 61.375 72.90625 \nL 61.375 64.59375 \nL 35.5 64.59375 \nL 35.5 0 \nL 25.59375 0 \nL 25.59375 64.59375 \nL -0.296875 64.59375 \nz\n\" id=\"DejaVuSans-84\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 29.798437)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"60.865234\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"101.978516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"163.257812\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"191.041016\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_16\">\n     <path d=\"M 83.780672 40.976562 \nL 103.780672 40.976562 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:11.1,4.8;stroke-dashoffset:0;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_17\"/>\n    <g id=\"text_14\">\n     <!-- f(x) -->\n     <defs>\n      <path d=\"M 37.109375 75.984375 \nL 37.109375 68.5 \nL 28.515625 68.5 \nQ 23.6875 68.5 21.796875 66.546875 \nQ 19.921875 64.59375 19.921875 59.515625 \nL 19.921875 54.6875 \nL 34.71875 54.6875 \nL 34.71875 47.703125 \nL 19.921875 47.703125 \nL 19.921875 0 \nL 10.890625 0 \nL 10.890625 47.703125 \nL 2.296875 47.703125 \nL 2.296875 54.6875 \nL 10.890625 54.6875 \nL 10.890625 58.5 \nQ 10.890625 67.625 15.140625 71.796875 \nQ 19.390625 75.984375 28.609375 75.984375 \nz\n\" id=\"DejaVuSans-102\"/>\n      <path d=\"M 31 75.875 \nQ 24.46875 64.65625 21.28125 53.65625 \nQ 18.109375 42.671875 18.109375 31.390625 \nQ 18.109375 20.125 21.3125 9.0625 \nQ 24.515625 -2 31 -13.1875 \nL 23.1875 -13.1875 \nQ 15.875 -1.703125 12.234375 9.375 \nQ 8.59375 20.453125 8.59375 31.390625 \nQ 8.59375 42.28125 12.203125 53.3125 \nQ 15.828125 64.359375 23.1875 75.875 \nz\n\" id=\"DejaVuSans-40\"/>\n      <path d=\"M 54.890625 54.6875 \nL 35.109375 28.078125 \nL 55.90625 0 \nL 45.3125 0 \nL 29.390625 21.484375 \nL 13.484375 0 \nL 2.875 0 \nL 24.125 28.609375 \nL 4.6875 54.6875 \nL 15.28125 54.6875 \nL 29.78125 35.203125 \nL 44.28125 54.6875 \nz\n\" id=\"DejaVuSans-120\"/>\n      <path d=\"M 8.015625 75.875 \nL 15.828125 75.875 \nQ 23.140625 64.359375 26.78125 53.3125 \nQ 30.421875 42.28125 30.421875 31.390625 \nQ 30.421875 20.453125 26.78125 9.375 \nQ 23.140625 -1.703125 15.828125 -13.1875 \nL 8.015625 -13.1875 \nQ 14.5 -2 17.703125 9.0625 \nQ 20.90625 20.125 20.90625 31.390625 \nQ 20.90625 42.671875 17.703125 53.65625 \nQ 14.5 64.65625 8.015625 75.875 \nz\n\" id=\"DejaVuSans-41\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-40\"/>\n      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-120\"/>\n      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-41\"/>\n     </g>\n    </g>\n    <g id=\"line2d_18\">\n     <path d=\"M 83.780672 55.654687 \nL 103.780672 55.654687 \n\" style=\"fill:none;stroke:#c44e52;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_19\"/>\n    <g id=\"text_15\">\n     <!-- Bayesian Inference -->\n     <defs>\n      <path d=\"M 19.671875 34.8125 \nL 19.671875 8.109375 \nL 35.5 8.109375 \nQ 43.453125 8.109375 47.28125 11.40625 \nQ 51.125 14.703125 51.125 21.484375 \nQ 51.125 28.328125 47.28125 31.5625 \nQ 43.453125 34.8125 35.5 34.8125 \nz\nM 19.671875 64.796875 \nL 19.671875 42.828125 \nL 34.28125 42.828125 \nQ 41.5 42.828125 45.03125 45.53125 \nQ 48.578125 48.25 48.578125 53.8125 \nQ 48.578125 59.328125 45.03125 62.0625 \nQ 41.5 64.796875 34.28125 64.796875 \nz\nM 9.8125 72.90625 \nL 35.015625 72.90625 \nQ 46.296875 72.90625 52.390625 68.21875 \nQ 58.5 63.53125 58.5 54.890625 \nQ 58.5 48.1875 55.375 44.234375 \nQ 52.25 40.28125 46.1875 39.3125 \nQ 53.46875 37.75 57.5 32.78125 \nQ 61.53125 27.828125 61.53125 20.40625 \nQ 61.53125 10.640625 54.890625 5.3125 \nQ 48.25 0 35.984375 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-66\"/>\n      <path d=\"M 32.171875 -5.078125 \nQ 28.375 -14.84375 24.75 -17.8125 \nQ 21.140625 -20.796875 15.09375 -20.796875 \nL 7.90625 -20.796875 \nL 7.90625 -13.28125 \nL 13.1875 -13.28125 \nQ 16.890625 -13.28125 18.9375 -11.515625 \nQ 21 -9.765625 23.484375 -3.21875 \nL 25.09375 0.875 \nL 2.984375 54.6875 \nL 12.5 54.6875 \nL 29.59375 11.921875 \nL 46.6875 54.6875 \nL 56.203125 54.6875 \nz\n\" id=\"DejaVuSans-121\"/>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n      <path id=\"DejaVuSans-32\"/>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-73\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-66\"/>\n      <use x=\"68.603516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"129.882812\" xlink:href=\"#DejaVuSans-121\"/>\n      <use x=\"189.0625\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"250.585938\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"302.685547\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"330.46875\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"391.748047\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"455.126953\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"486.914062\" xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"516.40625\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"579.785156\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"614.990234\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"676.513672\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"717.595703\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"779.119141\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"842.498047\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"897.478516\" xlink:href=\"#DejaVuSans-101\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p6966447886\">\n   <rect height=\"172.764\" width=\"260.874\" x=\"74.780672\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 510x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_fn(train, test)\n",
    "\n",
    "plt.plot(test_xs, nngp_mean, 'r-', linewidth=3)\n",
    "plt.fill_between(\n",
    "    jnp.reshape(test_xs, (-1)),\n",
    "    nngp_mean - 2 * nngp_std,\n",
    "    nngp_mean +  2 * nngp_std,\n",
    "    color='red', alpha=0.2)\n",
    "\n",
    "plt.xlim([-jnp.pi, jnp.pi])\n",
    "plt.ylim([-1.5, 1.5])\n",
    "\n",
    "legend(['Train', 'f(x)', 'Bayesian Inference'], loc='upper left')\n",
    "\n",
    "finalize_plot((0.85, 0.6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B0bPmNfX42tI"
   },
   "source": [
    "We see that our posterior exactly fits all of the training points as expected. We also see that the there is significant uncertainty in the predictions between the points in the middle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WoCJKtoo8b3N"
   },
   "source": [
    "Next, we would like to compute the result of doing gradient descent on our infinite network for an *infinite* amount of time. To do this, we will use the \"NTK\" inference mode. Note that otherwise the call to `predict_fn` looks identical. We will compare the result of true Bayesian inference with gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nQ3xu5lr8MPN"
   },
   "outputs": [],
   "source": [
    "ntk_mean, ntk_covariance = predict_fn(x_test=test_xs, get='ntk',\n",
    "                                      compute_cov=True)\n",
    "\n",
    "ntk_mean = jnp.reshape(ntk_mean, (-1,))\n",
    "ntk_std = jnp.sqrt(jnp.diag(ntk_covariance))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 341
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 612,
     "status": "ok",
     "timestamp": 1588653672749,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "25U1oRJHq3M9",
    "outputId": "3a3d67ca-dccc-44e5-ae74-ea6a34573f33"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"239.859969pt\" version=\"1.1\" viewBox=\"0 0 348.854672 239.859969\" width=\"348.854672pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 239.859969 \nL 348.854672 239.859969 \nL 348.854672 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \nL 335.654672 13.2 \nL 74.780672 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 122.178899 185.964 \nL 122.178899 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- −2 -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(111.638235 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 205.217672 185.964 \nL 205.217672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(200.668484 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 288.256445 185.964 \nL 288.256445 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(283.707258 206.329766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_4\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(199.217672 228.500594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_5\">\n      <!-- −1.5 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 191.396883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780672 157.17 \nL 335.654672 157.17 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_6\">\n      <!-- −1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 162.602883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780672 128.376 \nL 335.654672 128.376 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- −0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 133.808883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780672 99.582 \nL 335.654672 99.582 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 105.014883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780672 70.788 \nL 335.654672 70.788 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 76.220883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780672 41.994 \nL 335.654672 41.994 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 47.426883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_10\">\n      <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1.5 -->\n      <g style=\"fill:#262626;\" transform=\"translate(42.539203 18.632883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 103.182)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"PolyCollection_1\">\n    <defs>\n     <path d=\"M 74.780668 -139.193632 \nL 74.780668 -132.073324 \nL 80.104624 -127.303292 \nL 85.42859 -122.424214 \nL 90.752545 -117.424722 \nL 96.076501 -112.3211 \nL 101.400467 -107.229397 \nL 106.724422 -102.154016 \nL 112.048388 -97.140238 \nL 117.372344 -92.177553 \nL 122.696304 -87.41615 \nL 128.020265 -83.103305 \nL 133.34422 -79.459721 \nL 138.668181 -76.720397 \nL 143.992142 -75.139732 \nL 149.316097 -74.884621 \nL 154.640058 -76.088465 \nL 159.964019 -78.885997 \nL 165.287974 -83.120849 \nL 170.61194 -88.608334 \nL 175.935896 -94.955842 \nL 181.259861 -101.703035 \nL 186.583817 -108.65665 \nL 191.907773 -116.226046 \nL 197.231738 -125.17209 \nL 202.555694 -122.704802 \nL 207.87965 -121.695344 \nL 213.203615 -126.305528 \nL 218.527571 -134.132426 \nL 223.851527 -143.872137 \nL 229.175492 -155.525916 \nL 234.499448 -168.042366 \nL 239.823414 -179.96643 \nL 245.147369 -190.232923 \nL 250.471325 -198.210895 \nL 255.795281 -203.430861 \nL 261.119256 -205.807113 \nL 266.443212 -206.208141 \nL 271.767168 -205.228557 \nL 277.091123 -203.116456 \nL 282.415079 -200.106081 \nL 287.739054 -196.333239 \nL 293.06301 -192.008891 \nL 298.386966 -187.271163 \nL 303.710922 -182.255689 \nL 309.034877 -177.028659 \nL 314.358833 -171.699996 \nL 319.682808 -166.299695 \nL 325.006764 -160.801567 \nL 330.33072 -155.258654 \nL 335.654676 -149.757693 \nL 335.654676 -152.28654 \nL 335.654676 -152.28654 \nL 330.33072 -157.306532 \nL 325.006764 -162.569277 \nL 319.682808 -168.072763 \nL 314.358833 -173.754039 \nL 309.034877 -179.492894 \nL 303.710922 -185.182242 \nL 298.386966 -190.677176 \nL 293.06301 -195.834749 \nL 287.739054 -200.491827 \nL 282.415079 -204.474598 \nL 277.091123 -207.508081 \nL 271.767168 -209.387141 \nL 266.443212 -209.819665 \nL 261.119256 -208.617902 \nL 255.795281 -205.858896 \nL 250.471325 -202.281086 \nL 245.147369 -197.795249 \nL 239.823414 -192.396744 \nL 234.499448 -186.50206 \nL 229.175492 -180.816639 \nL 223.851527 -176.214038 \nL 218.527571 -173.119999 \nL 213.203615 -170.198794 \nL 207.87965 -160.233159 \nL 202.555694 -140.820163 \nL 197.231738 -127.624105 \nL 191.907773 -130.342638 \nL 186.583817 -129.066807 \nL 181.259861 -123.075951 \nL 175.935896 -114.56148 \nL 170.61194 -105.526061 \nL 165.287974 -97.363461 \nL 159.964019 -90.817262 \nL 154.640058 -86.121918 \nL 149.316097 -83.334601 \nL 143.992142 -82.229765 \nL 138.668181 -82.597229 \nL 133.34422 -84.243613 \nL 128.020265 -86.924199 \nL 122.696304 -90.447807 \nL 117.372344 -94.712649 \nL 112.048388 -99.589026 \nL 106.724422 -104.935394 \nL 101.400467 -110.58395 \nL 96.076501 -116.372597 \nL 90.752545 -122.223138 \nL 85.42859 -127.991519 \nL 80.104624 -133.654484 \nL 74.780668 -139.193632 \nz\n\" id=\"m01166b935e\" style=\"stroke:#ff0000;stroke-opacity:0.2;\"/>\n    </defs>\n    <g clip-path=\"url(#p063d696f5f)\">\n     <use style=\"fill:#ff0000;fill-opacity:0.2;stroke:#ff0000;stroke-opacity:0.2;\" x=\"0\" xlink:href=\"#m01166b935e\" y=\"239.859969\"/>\n    </g>\n   </g>\n   <g id=\"PolyCollection_2\">\n    <defs>\n     <path d=\"M 74.780668 -146.530753 \nL 74.780668 -141.394871 \nL 80.104624 -134.55743 \nL 85.42859 -127.802123 \nL 90.752545 -121.190269 \nL 96.076501 -114.812522 \nL 101.400467 -108.793912 \nL 106.724422 -103.21689 \nL 112.048388 -98.224654 \nL 117.372344 -93.029794 \nL 122.696304 -88.141275 \nL 128.020265 -84.03966 \nL 133.34422 -80.866261 \nL 138.668181 -78.740575 \nL 143.992142 -77.75911 \nL 149.316097 -77.964257 \nL 154.640058 -79.308388 \nL 159.964019 -81.666133 \nL 165.287974 -84.769807 \nL 170.61194 -88.348476 \nL 175.935896 -92.250888 \nL 181.259861 -96.716807 \nL 186.583817 -102.656326 \nL 191.907773 -111.794781 \nL 197.231738 -126.137875 \nL 202.555694 -125.470279 \nL 207.87965 -125.411355 \nL 213.203615 -131.472998 \nL 218.527571 -140.929072 \nL 223.851527 -150.971593 \nL 229.175492 -161.498237 \nL 234.499448 -172.112557 \nL 239.823414 -182.164889 \nL 245.147369 -191.083273 \nL 250.471325 -198.479826 \nL 255.795281 -204.170443 \nL 261.119256 -206.055291 \nL 266.443212 -207.079072 \nL 271.767168 -207.17418 \nL 277.091123 -206.265525 \nL 282.415079 -204.348103 \nL 287.739054 -201.450562 \nL 293.06301 -197.643768 \nL 298.386966 -193.031982 \nL 303.710922 -187.741448 \nL 309.034877 -181.833651 \nL 314.358833 -175.402139 \nL 319.682808 -168.669208 \nL 325.006764 -161.589663 \nL 330.33072 -154.398707 \nL 335.654676 -147.180202 \nL 335.654676 -147.443983 \nL 335.654676 -147.443983 \nL 330.33072 -154.650213 \nL 325.006764 -161.767505 \nL 319.682808 -168.748739 \nL 314.358833 -175.64074 \nL 309.034877 -182.223283 \nL 303.710922 -188.495962 \nL 298.386966 -194.329129 \nL 293.06301 -199.514062 \nL 287.739054 -203.912236 \nL 282.415079 -207.365117 \nL 277.091123 -209.678033 \nL 271.767168 -210.673626 \nL 266.443212 -210.177793 \nL 261.119256 -208.046793 \nL 255.795281 -204.249982 \nL 250.471325 -201.946581 \nL 245.147369 -199.402361 \nL 239.823414 -196.951602 \nL 234.499448 -194.840758 \nL 229.175492 -193.101549 \nL 223.851527 -191.407076 \nL 218.527571 -188.928864 \nL 213.203615 -183.709931 \nL 207.87965 -169.860987 \nL 202.555694 -146.213459 \nL 197.231738 -126.430099 \nL 191.907773 -127.236875 \nL 186.583817 -125.174685 \nL 181.259861 -120.351878 \nL 175.935896 -114.02433 \nL 170.61194 -107.357516 \nL 165.287974 -101.133698 \nL 159.964019 -95.786088 \nL 154.640058 -91.512222 \nL 149.316097 -88.404903 \nL 143.992142 -86.487824 \nL 138.668181 -85.770137 \nL 133.34422 -86.227516 \nL 128.020265 -87.797959 \nL 122.696304 -90.392214 \nL 117.372344 -93.915437 \nL 112.048388 -98.5977 \nL 106.724422 -104.685559 \nL 101.400467 -111.170604 \nL 96.076501 -117.957847 \nL 90.752545 -124.96871 \nL 85.42859 -132.110137 \nL 80.104624 -139.318125 \nL 74.780668 -146.530753 \nz\n\" id=\"mf0e6340043\" style=\"stroke:#0000ff;stroke-opacity:0.2;\"/>\n    </defs>\n    <g clip-path=\"url(#p063d696f5f)\">\n     <use style=\"fill:#0000ff;fill-opacity:0.2;stroke:#0000ff;stroke-opacity:0.2;\" x=\"0\" xlink:href=\"#mf0e6340043\" y=\"239.859969\"/>\n    </g>\n   </g>\n   <g id=\"line2d_11\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"mc9c0c5814a\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#p063d696f5f)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"332.990361\" xlink:href=\"#mc9c0c5814a\" y=\"89.211043\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"312.709007\" xlink:href=\"#mc9c0c5814a\" y=\"61.934552\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"113.996599\" xlink:href=\"#mc9c0c5814a\" y=\"143.295353\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"197.320621\" xlink:href=\"#mc9c0c5814a\" y=\"113.447504\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"255.905476\" xlink:href=\"#mc9c0c5814a\" y=\"35.617422\"/>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780668 99.581995 \nL 80.104624 106.946181 \nL 85.42859 114.189459 \nL 90.752545 121.192871 \nL 96.076501 127.841434 \nL 101.400467 134.025987 \nL 106.724422 139.644962 \nL 112.048388 144.606113 \nL 117.372344 148.827959 \nL 122.696304 152.241194 \nL 128.020265 154.789766 \nL 133.34422 156.431825 \nL 138.668181 157.140412 \nL 143.992142 156.903891 \nL 149.316097 155.726148 \nL 154.640058 153.626521 \nL 159.964019 150.639481 \nL 165.287974 146.814083 \nL 170.61194 142.213126 \nL 175.935896 136.912177 \nL 181.259861 130.998257 \nL 186.583817 124.568494 \nL 191.907773 117.728453 \nL 197.231738 110.590436 \nL 202.555694 103.271674 \nL 207.87965 95.892326 \nL 213.203615 88.57355 \nL 218.527571 81.435547 \nL 223.851527 74.595506 \nL 229.175492 68.165732 \nL 234.499448 62.251823 \nL 239.823414 56.950863 \nL 245.147369 52.349917 \nL 250.471325 48.524519 \nL 255.795281 45.537482 \nL 261.119256 43.437849 \nL 266.443212 42.260106 \nL 271.767168 42.023588 \nL 277.091123 42.732175 \nL 282.415079 44.374234 \nL 287.739054 46.922813 \nL 293.06301 50.336048 \nL 298.386966 54.557898 \nL 303.710922 59.519038 \nL 309.034877 65.138013 \nL 314.358833 71.322554 \nL 319.682808 77.971141 \nL 325.006764 84.974554 \nL 330.33072 92.217819 \nL 335.654676 99.582005 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:11.1,4.8;stroke-dashoffset:0;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780668 104.226491 \nL 80.104624 109.381081 \nL 85.42859 114.652102 \nL 90.752545 120.036039 \nL 96.076501 125.51312 \nL 101.400467 130.953295 \nL 106.724422 136.315264 \nL 112.048388 141.495337 \nL 117.372344 146.414868 \nL 122.696304 150.92799 \nL 128.020265 154.846217 \nL 133.34422 158.008302 \nL 138.668181 160.201155 \nL 143.992142 161.17522 \nL 149.316097 160.750357 \nL 154.640058 158.754777 \nL 159.964019 155.008341 \nL 165.287974 149.617814 \nL 170.61194 142.792771 \nL 175.935896 135.101308 \nL 181.259861 127.470476 \nL 186.583817 120.99824 \nL 191.907773 116.575627 \nL 197.231738 113.461871 \nL 202.555694 108.097486 \nL 207.87965 98.895717 \nL 213.203615 91.607807 \nL 218.527571 86.233756 \nL 223.851527 79.81688 \nL 229.175492 71.688691 \nL 234.499448 62.587756 \nL 239.823414 53.678382 \nL 245.147369 45.845883 \nL 250.471325 39.613978 \nL 255.795281 35.21509 \nL 261.119256 32.647461 \nL 266.443212 31.846066 \nL 271.767168 32.55212 \nL 277.091123 34.5477 \nL 282.415079 37.569629 \nL 287.739054 41.447434 \nL 293.06301 45.938149 \nL 298.386966 50.885799 \nL 303.710922 56.141003 \nL 309.034877 61.599192 \nL 314.358833 67.132951 \nL 319.682808 72.67374 \nL 325.006764 78.174547 \nL 330.33072 83.577376 \nL 335.654676 88.837852 \n\" style=\"fill:none;stroke:#c44e52;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_14\">\n    <path clip-path=\"url(#p063d696f5f)\" d=\"M 74.780668 95.897157 \nL 80.104624 102.922191 \nL 85.42859 109.903839 \nL 90.752545 116.780479 \nL 96.076501 123.474784 \nL 101.400467 129.87771 \nL 106.724422 135.908744 \nL 112.048388 141.448792 \nL 117.372344 146.387353 \nL 122.696304 150.593224 \nL 128.020265 153.941159 \nL 133.34422 156.31308 \nL 138.668181 157.604611 \nL 143.992142 157.736502 \nL 149.316097 156.675389 \nL 154.640058 154.449665 \nL 159.964019 151.133858 \nL 165.287974 146.908216 \nL 170.61194 142.006973 \nL 175.935896 136.72236 \nL 181.259861 131.325626 \nL 186.583817 125.944463 \nL 191.907773 120.344141 \nL 197.231738 113.575982 \nL 202.555694 104.0181 \nL 207.87965 92.223799 \nL 213.203615 82.268504 \nL 218.527571 74.931001 \nL 223.851527 68.670634 \nL 229.175492 62.560076 \nL 234.499448 56.383311 \nL 239.823414 50.301723 \nL 245.147369 44.617153 \nL 250.471325 39.646766 \nL 255.795281 35.649756 \nL 261.119256 32.808927 \nL 266.443212 31.231536 \nL 271.767168 30.936066 \nL 277.091123 31.88819 \nL 282.415079 34.003359 \nL 287.739054 37.17857 \nL 293.06301 41.281053 \nL 298.386966 46.179413 \nL 303.710922 51.741264 \nL 309.034877 57.831502 \nL 314.358833 64.338529 \nL 319.682808 71.150995 \nL 325.006764 78.181384 \nL 330.33072 85.335509 \nL 335.654676 92.547876 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 74.780672 185.964 \nL 74.780672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 335.654672 185.964 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 74.780672 185.964 \nL 335.654672 185.964 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 74.780672 13.2 \nL 335.654672 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 81.780672 79.9125 \nL 209.683797 79.9125 \nQ 211.683797 79.9125 211.683797 77.9125 \nL 211.683797 20.2 \nQ 211.683797 18.2 209.683797 18.2 \nL 81.780672 18.2 \nQ 79.780672 18.2 79.780672 20.2 \nL 79.780672 77.9125 \nQ 79.780672 79.9125 81.780672 79.9125 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_15\"/>\n    <g id=\"line2d_16\">\n     <g>\n      <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"93.780672\" xlink:href=\"#mc9c0c5814a\" y=\"26.298437\"/>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- Train -->\n     <defs>\n      <path d=\"M -0.296875 72.90625 \nL 61.375 72.90625 \nL 61.375 64.59375 \nL 35.5 64.59375 \nL 35.5 0 \nL 25.59375 0 \nL 25.59375 64.59375 \nL -0.296875 64.59375 \nz\n\" id=\"DejaVuSans-84\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 29.798437)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"60.865234\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"101.978516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"163.257812\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"191.041016\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_17\">\n     <path d=\"M 83.780672 40.976562 \nL 103.780672 40.976562 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:11.1,4.8;stroke-dashoffset:0;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_18\"/>\n    <g id=\"text_14\">\n     <!-- f(x) -->\n     <defs>\n      <path d=\"M 37.109375 75.984375 \nL 37.109375 68.5 \nL 28.515625 68.5 \nQ 23.6875 68.5 21.796875 66.546875 \nQ 19.921875 64.59375 19.921875 59.515625 \nL 19.921875 54.6875 \nL 34.71875 54.6875 \nL 34.71875 47.703125 \nL 19.921875 47.703125 \nL 19.921875 0 \nL 10.890625 0 \nL 10.890625 47.703125 \nL 2.296875 47.703125 \nL 2.296875 54.6875 \nL 10.890625 54.6875 \nL 10.890625 58.5 \nQ 10.890625 67.625 15.140625 71.796875 \nQ 19.390625 75.984375 28.609375 75.984375 \nz\n\" id=\"DejaVuSans-102\"/>\n      <path d=\"M 31 75.875 \nQ 24.46875 64.65625 21.28125 53.65625 \nQ 18.109375 42.671875 18.109375 31.390625 \nQ 18.109375 20.125 21.3125 9.0625 \nQ 24.515625 -2 31 -13.1875 \nL 23.1875 -13.1875 \nQ 15.875 -1.703125 12.234375 9.375 \nQ 8.59375 20.453125 8.59375 31.390625 \nQ 8.59375 42.28125 12.203125 53.3125 \nQ 15.828125 64.359375 23.1875 75.875 \nz\n\" id=\"DejaVuSans-40\"/>\n      <path d=\"M 54.890625 54.6875 \nL 35.109375 28.078125 \nL 55.90625 0 \nL 45.3125 0 \nL 29.390625 21.484375 \nL 13.484375 0 \nL 2.875 0 \nL 24.125 28.609375 \nL 4.6875 54.6875 \nL 15.28125 54.6875 \nL 29.78125 35.203125 \nL 44.28125 54.6875 \nz\n\" id=\"DejaVuSans-120\"/>\n      <path d=\"M 8.015625 75.875 \nL 15.828125 75.875 \nQ 23.140625 64.359375 26.78125 53.3125 \nQ 30.421875 42.28125 30.421875 31.390625 \nQ 30.421875 20.453125 26.78125 9.375 \nQ 23.140625 -1.703125 15.828125 -13.1875 \nL 8.015625 -13.1875 \nQ 14.5 -2 17.703125 9.0625 \nQ 20.90625 20.125 20.90625 31.390625 \nQ 20.90625 42.671875 17.703125 53.65625 \nQ 14.5 64.65625 8.015625 75.875 \nz\n\" id=\"DejaVuSans-41\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-40\"/>\n      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-120\"/>\n      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-41\"/>\n     </g>\n    </g>\n    <g id=\"line2d_19\">\n     <path d=\"M 83.780672 55.654687 \nL 103.780672 55.654687 \n\" style=\"fill:none;stroke:#c44e52;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_20\"/>\n    <g id=\"text_15\">\n     <!-- Bayesian Inference -->\n     <defs>\n      <path d=\"M 19.671875 34.8125 \nL 19.671875 8.109375 \nL 35.5 8.109375 \nQ 43.453125 8.109375 47.28125 11.40625 \nQ 51.125 14.703125 51.125 21.484375 \nQ 51.125 28.328125 47.28125 31.5625 \nQ 43.453125 34.8125 35.5 34.8125 \nz\nM 19.671875 64.796875 \nL 19.671875 42.828125 \nL 34.28125 42.828125 \nQ 41.5 42.828125 45.03125 45.53125 \nQ 48.578125 48.25 48.578125 53.8125 \nQ 48.578125 59.328125 45.03125 62.0625 \nQ 41.5 64.796875 34.28125 64.796875 \nz\nM 9.8125 72.90625 \nL 35.015625 72.90625 \nQ 46.296875 72.90625 52.390625 68.21875 \nQ 58.5 63.53125 58.5 54.890625 \nQ 58.5 48.1875 55.375 44.234375 \nQ 52.25 40.28125 46.1875 39.3125 \nQ 53.46875 37.75 57.5 32.78125 \nQ 61.53125 27.828125 61.53125 20.40625 \nQ 61.53125 10.640625 54.890625 5.3125 \nQ 48.25 0 35.984375 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-66\"/>\n      <path d=\"M 32.171875 -5.078125 \nQ 28.375 -14.84375 24.75 -17.8125 \nQ 21.140625 -20.796875 15.09375 -20.796875 \nL 7.90625 -20.796875 \nL 7.90625 -13.28125 \nL 13.1875 -13.28125 \nQ 16.890625 -13.28125 18.9375 -11.515625 \nQ 21 -9.765625 23.484375 -3.21875 \nL 25.09375 0.875 \nL 2.984375 54.6875 \nL 12.5 54.6875 \nL 29.59375 11.921875 \nL 46.6875 54.6875 \nL 56.203125 54.6875 \nz\n\" id=\"DejaVuSans-121\"/>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n      <path id=\"DejaVuSans-32\"/>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-73\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-66\"/>\n      <use x=\"68.603516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"129.882812\" xlink:href=\"#DejaVuSans-121\"/>\n      <use x=\"189.0625\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"250.585938\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"302.685547\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"330.46875\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"391.748047\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"455.126953\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"486.914062\" xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"516.40625\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"579.785156\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"614.990234\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"676.513672\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"717.595703\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"779.119141\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"842.498047\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"897.478516\" xlink:href=\"#DejaVuSans-101\"/>\n     </g>\n    </g>\n    <g id=\"line2d_21\">\n     <path d=\"M 83.780672 70.332812 \nL 103.780672 70.332812 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_22\"/>\n    <g id=\"text_16\">\n     <!-- Gradient Descent -->\n     <defs>\n      <path d=\"M 59.515625 10.40625 \nL 59.515625 29.984375 \nL 43.40625 29.984375 \nL 43.40625 38.09375 \nL 69.28125 38.09375 \nL 69.28125 6.78125 \nQ 63.578125 2.734375 56.6875 0.65625 \nQ 49.8125 -1.421875 42 -1.421875 \nQ 24.90625 -1.421875 15.25 8.5625 \nQ 5.609375 18.5625 5.609375 36.375 \nQ 5.609375 54.25 15.25 64.234375 \nQ 24.90625 74.21875 42 74.21875 \nQ 49.125 74.21875 55.546875 72.453125 \nQ 61.96875 70.703125 67.390625 67.28125 \nL 67.390625 56.78125 \nQ 61.921875 61.421875 55.765625 63.765625 \nQ 49.609375 66.109375 42.828125 66.109375 \nQ 29.4375 66.109375 22.71875 58.640625 \nQ 16.015625 51.171875 16.015625 36.375 \nQ 16.015625 21.625 22.71875 14.15625 \nQ 29.4375 6.6875 42.828125 6.6875 \nQ 48.046875 6.6875 52.140625 7.59375 \nQ 56.25 8.5 59.515625 10.40625 \nz\n\" id=\"DejaVuSans-71\"/>\n      <path d=\"M 45.40625 46.390625 \nL 45.40625 75.984375 \nL 54.390625 75.984375 \nL 54.390625 0 \nL 45.40625 0 \nL 45.40625 8.203125 \nQ 42.578125 3.328125 38.25 0.953125 \nQ 33.9375 -1.421875 27.875 -1.421875 \nQ 17.96875 -1.421875 11.734375 6.484375 \nQ 5.515625 14.40625 5.515625 27.296875 \nQ 5.515625 40.1875 11.734375 48.09375 \nQ 17.96875 56 27.875 56 \nQ 33.9375 56 38.25 53.625 \nQ 42.578125 51.265625 45.40625 46.390625 \nz\nM 14.796875 27.296875 \nQ 14.796875 17.390625 18.875 11.75 \nQ 22.953125 6.109375 30.078125 6.109375 \nQ 37.203125 6.109375 41.296875 11.75 \nQ 45.40625 17.390625 45.40625 27.296875 \nQ 45.40625 37.203125 41.296875 42.84375 \nQ 37.203125 48.484375 30.078125 48.484375 \nQ 22.953125 48.484375 18.875 42.84375 \nQ 14.796875 37.203125 14.796875 27.296875 \nz\n\" id=\"DejaVuSans-100\"/>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 19.671875 64.796875 \nL 19.671875 8.109375 \nL 31.59375 8.109375 \nQ 46.6875 8.109375 53.6875 14.9375 \nQ 60.6875 21.78125 60.6875 36.53125 \nQ 60.6875 51.171875 53.6875 57.984375 \nQ 46.6875 64.796875 31.59375 64.796875 \nz\nM 9.8125 72.90625 \nL 30.078125 72.90625 \nQ 51.265625 72.90625 61.171875 64.09375 \nQ 71.09375 55.28125 71.09375 36.53125 \nQ 71.09375 17.671875 61.125 8.828125 \nQ 51.171875 0 30.078125 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-68\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(111.780672 73.832812)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-71\"/>\n      <use x=\"77.490234\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"118.603516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"179.882812\" xlink:href=\"#DejaVuSans-100\"/>\n      <use x=\"243.359375\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"271.142578\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"332.666016\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"396.044922\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"435.253906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"467.041016\" xlink:href=\"#DejaVuSans-68\"/>\n      <use x=\"544.042969\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"605.566406\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"657.666016\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"712.646484\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"774.169922\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"837.548828\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p063d696f5f\">\n   <rect height=\"172.764\" width=\"260.874\" x=\"74.780672\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 510x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_fn(train, test)\n",
    "\n",
    "plt.plot(test_xs, nngp_mean, 'r-', linewidth=3)\n",
    "plt.fill_between(\n",
    "    jnp.reshape(test_xs, (-1)),\n",
    "    nngp_mean - 2 * nngp_std,\n",
    "    nngp_mean +  2 * nngp_std,\n",
    "    color='red', alpha=0.2)\n",
    "\n",
    "\n",
    "plt.plot(test_xs, ntk_mean, 'b-', linewidth=3)\n",
    "plt.fill_between(\n",
    "    jnp.reshape(test_xs, (-1)),\n",
    "    ntk_mean - 2 * ntk_std,\n",
    "    ntk_mean +  2 * ntk_std,\n",
    "    color='blue', alpha=0.2)\n",
    "\n",
    "plt.xlim([-jnp.pi, jnp.pi])\n",
    "plt.ylim([-1.5, 1.5])\n",
    "\n",
    "legend(['Train', 'f(x)', 'Bayesian Inference', 'Gradient Descent'],\n",
    "       loc='upper left')\n",
    "\n",
    "finalize_plot((0.85, 0.6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SwV7HILPsNrz"
   },
   "source": [
    "We see that while the result of gradient descent and bayesian inference are similar, they are not identical.\n",
    "\n",
    "Not only can we do inference at infinite times, but we can also perform finite-time inference. We will use this to predict the mean of the train and test losses over the course of training. To compute the mean MSE loss, we need to access the mean and variance of our networks predictions as a function of time. To do this, we call our `predict_fn` function with finite times `t` (as opposed to using the default value `t=None` earlier, considered as infinite time). Note that `predict` can act on both scalar and array values, so we simply invoke the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mNrvoLXx7Jbp"
   },
   "outputs": [],
   "source": [
    "ts = jnp.arange(0, 10 ** 3, 10 ** -1)\n",
    "ntk_train_loss_mean = loss_fn(predict_fn, train_ys, ts)\n",
    "ntk_test_loss_mean = loss_fn(predict_fn, test_ys, ts, test_xs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 338
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1628,
     "status": "ok",
     "timestamp": 1588653675621,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "0iwoN0LDR28N",
    "outputId": "5116d0e4-9757-4c84-f1e7-6dffb26d8fdf"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"237.771969pt\" version=\"1.1\" viewBox=\"0 0 629.256869 237.771969\" width=\"629.256869pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 237.771969 \nL 629.256869 237.771969 \nL 629.256869 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 73.66125 183.876 \nL 300.120059 183.876 \nL 300.120059 13.2 \nL 73.66125 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 83.954832 183.876 \nL 83.954832 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(67.152332 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 135.423302 183.876 \nL 135.423302 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(122.839302 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 186.891772 183.876 \nL 186.891772 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(174.307772 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 238.360242 183.876 \nL 238.360242 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_4\">\n      <!-- $\\mathdefault{10^{2}}$ -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(225.776242 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 289.828712 183.876 \nL 289.828712 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_5\">\n      <!-- $\\mathdefault{10^{3}}$ -->\n      <defs>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(277.244712 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\"/>\n    <g id=\"xtick_7\"/>\n    <g id=\"xtick_8\"/>\n    <g id=\"xtick_9\"/>\n    <g id=\"xtick_10\"/>\n    <g id=\"xtick_11\"/>\n    <g id=\"xtick_12\"/>\n    <g id=\"xtick_13\"/>\n    <g id=\"xtick_14\"/>\n    <g id=\"xtick_15\"/>\n    <g id=\"xtick_16\"/>\n    <g id=\"xtick_17\"/>\n    <g id=\"xtick_18\"/>\n    <g id=\"xtick_19\"/>\n    <g id=\"xtick_20\"/>\n    <g id=\"xtick_21\"/>\n    <g id=\"xtick_22\"/>\n    <g id=\"xtick_23\"/>\n    <g id=\"xtick_24\"/>\n    <g id=\"xtick_25\"/>\n    <g id=\"xtick_26\"/>\n    <g id=\"xtick_27\"/>\n    <g id=\"xtick_28\"/>\n    <g id=\"xtick_29\"/>\n    <g id=\"xtick_30\"/>\n    <g id=\"xtick_31\"/>\n    <g id=\"xtick_32\"/>\n    <g id=\"xtick_33\"/>\n    <g id=\"xtick_34\"/>\n    <g id=\"xtick_35\"/>\n    <g id=\"xtick_36\"/>\n    <g id=\"xtick_37\"/>\n    <g id=\"xtick_38\"/>\n    <g id=\"xtick_39\"/>\n    <g id=\"xtick_40\"/>\n    <g id=\"text_6\">\n     <!-- Step -->\n     <defs>\n      <path d=\"M 53.515625 70.515625 \nL 53.515625 60.890625 \nQ 47.90625 63.578125 42.921875 64.890625 \nQ 37.9375 66.21875 33.296875 66.21875 \nQ 25.25 66.21875 20.875 63.09375 \nQ 16.5 59.96875 16.5 54.203125 \nQ 16.5 49.359375 19.40625 46.890625 \nQ 22.3125 44.4375 30.421875 42.921875 \nL 36.375 41.703125 \nQ 47.40625 39.59375 52.65625 34.296875 \nQ 57.90625 29 57.90625 20.125 \nQ 57.90625 9.515625 50.796875 4.046875 \nQ 43.703125 -1.421875 29.984375 -1.421875 \nQ 24.8125 -1.421875 18.96875 -0.25 \nQ 13.140625 0.921875 6.890625 3.21875 \nL 6.890625 13.375 \nQ 12.890625 10.015625 18.65625 8.296875 \nQ 24.421875 6.59375 29.984375 6.59375 \nQ 38.421875 6.59375 43.015625 9.90625 \nQ 47.609375 13.234375 47.609375 19.390625 \nQ 47.609375 24.75 44.3125 27.78125 \nQ 41.015625 30.8125 33.5 32.328125 \nL 27.484375 33.5 \nQ 16.453125 35.6875 11.515625 40.375 \nQ 6.59375 45.0625 6.59375 53.421875 \nQ 6.59375 63.09375 13.40625 68.65625 \nQ 20.21875 74.21875 32.171875 74.21875 \nQ 37.3125 74.21875 42.625 73.28125 \nQ 47.953125 72.359375 53.515625 70.515625 \nz\n\" id=\"DejaVuSans-83\"/>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(164.120342 226.412594)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-83\"/>\n      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"102.685547\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"164.208984\" xlink:href=\"#DejaVuSans-112\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 73.66125 157.023279 \nL 300.120059 157.023279 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- $\\mathdefault{10^{-4}}$ -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 162.456162)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-52\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 73.66125 122.22088 \nL 300.120059 122.22088 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-3}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 127.653763)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 73.66125 87.418481 \nL 300.120059 87.418481 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{-2}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 92.851364)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 73.66125 52.616082 \nL 300.120059 52.616082 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 58.048965)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_10\">\n      <path clip-path=\"url(#p23f3c146c7)\" d=\"M 73.66125 17.813683 \nL 300.120059 17.813683 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_11\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(38.99325 23.246565)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\"/>\n    <g id=\"ytick_7\"/>\n    <g id=\"ytick_8\"/>\n    <g id=\"ytick_9\"/>\n    <g id=\"ytick_10\"/>\n    <g id=\"ytick_11\"/>\n    <g id=\"ytick_12\"/>\n    <g id=\"ytick_13\"/>\n    <g id=\"ytick_14\"/>\n    <g id=\"ytick_15\"/>\n    <g id=\"ytick_16\"/>\n    <g id=\"ytick_17\"/>\n    <g id=\"ytick_18\"/>\n    <g id=\"ytick_19\"/>\n    <g id=\"ytick_20\"/>\n    <g id=\"ytick_21\"/>\n    <g id=\"ytick_22\"/>\n    <g id=\"ytick_23\"/>\n    <g id=\"ytick_24\"/>\n    <g id=\"ytick_25\"/>\n    <g id=\"ytick_26\"/>\n    <g id=\"ytick_27\"/>\n    <g id=\"ytick_28\"/>\n    <g id=\"ytick_29\"/>\n    <g id=\"ytick_30\"/>\n    <g id=\"ytick_31\"/>\n    <g id=\"ytick_32\"/>\n    <g id=\"ytick_33\"/>\n    <g id=\"ytick_34\"/>\n    <g id=\"ytick_35\"/>\n    <g id=\"ytick_36\"/>\n    <g id=\"ytick_37\"/>\n    <g id=\"ytick_38\"/>\n    <g id=\"ytick_39\"/>\n    <g id=\"ytick_40\"/>\n    <g id=\"ytick_41\"/>\n    <g id=\"ytick_42\"/>\n    <g id=\"ytick_43\"/>\n    <g id=\"ytick_44\"/>\n    <g id=\"ytick_45\"/>\n    <g id=\"text_12\">\n     <!-- Loss -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 8.296875 \nL 55.171875 8.296875 \nL 55.171875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-76\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 120.645812)rotate(-90)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use x=\"55.697266\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"116.878906\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"168.978516\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_11\">\n    <path clip-path=\"url(#p23f3c146c7)\" d=\"M -1 36.262195 \nL 83.954832 36.286846 \nL 99.448386 46.793491 \nL 108.511534 52.413614 \nL 114.941939 54.961997 \nL 119.929749 56.223694 \nL 133.068233 58.941167 \nL 137.553721 60.116122 \nL 142.944288 61.819487 \nL 147.284156 63.454506 \nL 152.007435 65.527566 \nL 155.904665 67.478557 \nL 159.980003 69.748584 \nL 164.667776 72.625456 \nL 170.015149 76.183158 \nL 177.603974 81.527122 \nL 187.768451 88.954771 \nL 204.001871 100.973825 \nL 207.283546 102.98928 \nL 210.380234 104.632287 \nL 213.511054 106.04215 \nL 216.673448 107.252725 \nL 221.002903 108.673861 \nL 227.602738 110.630393 \nL 235.654736 113.174816 \nL 240.833465 115.147455 \nL 245.084874 117.088 \nL 248.655442 119.005085 \nL 252.18354 121.209347 \nL 256.276934 124.219398 \nL 258.234386 125.853734 \nL 261.028849 128.43285 \nL 264.959133 132.598864 \nL 270.891 140.256912 \nL 271.396336 140.993638 \nL 273.296598 143.881849 \nL 273.833342 144.738775 \nL 273.992792 144.989662 \nL 274.491081 145.802103 \nL 275.181728 146.94348 \nL 275.608071 147.6643 \nL 275.797303 147.990203 \nL 276.558855 149.305297 \nL 276.856358 149.828778 \nL 277.334439 150.678872 \nL 278.05375 151.982447 \nL 278.603383 152.998818 \nL 279.186651 154.090859 \nL 279.57546 154.831581 \nL 280.507021 156.634708 \nL 280.517194 156.639154 \nL 280.551072 156.722104 \nL 280.561225 156.732209 \nL 280.88372 157.367853 \nL 281.208202 158.007831 \nL 281.296779 158.184033 \nL 281.692696 158.980018 \nL 281.695911 158.978288 \nL 281.734473 159.066077 \nL 281.737682 159.066012 \nL 281.913549 159.42823 \nL 282.059583 159.707673 \nL 282.223521 160.050398 \nL 282.516857 160.655825 \nL 282.584958 160.782159 \nL 282.803332 161.244028 \nL 282.806394 161.239934 \nL 282.861411 161.349061 \nL 284.946824 165.702901 \nL 284.949606 165.693905 \nL 285.013466 165.841924 \nL 285.19571 166.244012 \nL 285.20121 166.231064 \nL 285.250644 166.341848 \nL 285.253388 166.339657 \nL 285.466322 166.811565 \nL 285.469038 166.802207 \nL 285.536851 166.96122 \nL 285.542267 166.958721 \nL 285.585549 167.068859 \nL 285.588252 167.051248 \nL 285.728326 167.355908 \nL 285.822115 167.553739 \nL 286.710721 169.454084 \nL 286.713292 169.444732 \nL 286.779999 169.605552 \nL 286.782561 169.604257 \nL 288.611481 173.527313 \nL 288.613841 173.491447 \nL 288.708046 173.724067 \nL 288.864956 174.055765 \nL 288.712746 173.723217 \nL 288.86729 174.049162 \nL 288.869623 174.039439 \nL 288.937186 174.214195 \nL 288.941836 174.203488 \nL 289.163998 174.696078 \nL 289.1663 174.669452 \nL 289.25821 174.885063 \nL 289.622117 175.676778 \nL 289.63114 175.656288 \nL 289.705434 175.836038 \nL 289.709929 175.834083 \nL 289.826477 176.118 \nL 289.826477 176.118 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#p23f3c146c7)\" d=\"M -1 34.9827 \nL 83.954832 35.005911 \nL 108.511534 51.066914 \nL 114.941939 54.211496 \nL 119.929749 55.961716 \nL 127.450735 57.979576 \nL 135.423302 60.063025 \nL 141.287791 61.864208 \nL 145.929045 63.514364 \nL 152.007435 65.959369 \nL 165.844493 71.751425 \nL 169.534433 72.965989 \nL 173.118489 73.889688 \nL 176.564138 74.540553 \nL 180.75743 75.087531 \nL 199.14364 77.158039 \nL 226.184243 81.295801 \nL 234.195315 82.75612 \nL 241.44536 84.299591 \nL 249.185015 86.179862 \nL 263.03584 89.618809 \nL 268.15371 90.560201 \nL 272.13848 91.045546 \nL 275.964176 91.289526 \nL 279.315814 91.342593 \nL 284.395023 91.221223 \nL 289.826477 90.976459 \nL 289.826477 90.976459 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 73.66125 183.876 \nL 73.66125 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 300.120059 183.876 \nL 300.120059 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 73.66125 183.876 \nL 300.120059 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 73.66125 13.2 \nL 300.120059 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 197.726309 50.55625 \nL 293.120059 50.55625 \nQ 295.120059 50.55625 295.120059 48.55625 \nL 295.120059 20.2 \nQ 295.120059 18.2 293.120059 18.2 \nL 197.726309 18.2 \nQ 195.726309 18.2 195.726309 20.2 \nL 195.726309 48.55625 \nQ 195.726309 50.55625 197.726309 50.55625 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_13\">\n     <path d=\"M 199.726309 26.298438 \nL 219.726309 26.298438 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_14\"/>\n    <g id=\"text_13\">\n     <!-- Infinite Train -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-73\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n      <path d=\"M 37.109375 75.984375 \nL 37.109375 68.5 \nL 28.515625 68.5 \nQ 23.6875 68.5 21.796875 66.546875 \nQ 19.921875 64.59375 19.921875 59.515625 \nL 19.921875 54.6875 \nL 34.71875 54.6875 \nL 34.71875 47.703125 \nL 19.921875 47.703125 \nL 19.921875 0 \nL 10.890625 0 \nL 10.890625 47.703125 \nL 2.296875 47.703125 \nL 2.296875 54.6875 \nL 10.890625 54.6875 \nL 10.890625 58.5 \nQ 10.890625 67.625 15.140625 71.796875 \nQ 19.390625 75.984375 28.609375 75.984375 \nz\n\" id=\"DejaVuSans-102\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path id=\"DejaVuSans-32\"/>\n      <path d=\"M -0.296875 72.90625 \nL 61.375 72.90625 \nL 61.375 64.59375 \nL 35.5 64.59375 \nL 35.5 0 \nL 25.59375 0 \nL 25.59375 64.59375 \nL -0.296875 64.59375 \nz\n\" id=\"DejaVuSans-84\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(227.726309 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"440.40625\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"481.519531\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"542.798828\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"570.582031\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_15\">\n     <path d=\"M 199.726309 40.976562 \nL 219.726309 40.976562 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_16\"/>\n    <g id=\"text_14\">\n     <!-- Infinite Test -->\n     <g style=\"fill:#262626;\" transform=\"translate(227.726309 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"440.359375\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"501.882812\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"553.982422\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n  <g id=\"axes_2\">\n   <g id=\"patch_8\">\n    <path d=\"M 389.598059 183.876 \nL 616.056869 183.876 \nL 616.056869 13.2 \nL 389.598059 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_3\">\n    <g id=\"xtick_41\">\n     <g id=\"line2d_17\">\n      <path clip-path=\"url(#pa7436bb53b)\" d=\"M 430.743386 183.876 \nL 430.743386 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_15\">\n      <!-- −2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(420.202722 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_42\">\n     <g id=\"line2d_18\">\n      <path clip-path=\"url(#pa7436bb53b)\" d=\"M 502.827464 183.876 \nL 502.827464 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_16\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(498.278276 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_43\">\n     <g id=\"line2d_19\">\n      <path clip-path=\"url(#pa7436bb53b)\" d=\"M 574.911542 183.876 \nL 574.911542 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_17\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(570.362354 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_18\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(496.827464 226.412594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_4\">\n    <g id=\"ytick_46\">\n     <g id=\"line2d_20\">\n      <path clip-path=\"url(#pa7436bb53b)\" d=\"M 389.598059 155.43 \nL 616.056869 155.43 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_19\">\n      <!-- −1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(359.016731 160.862883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_47\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#pa7436bb53b)\" d=\"M 389.598059 98.538 \nL 616.056869 98.538 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_20\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(370.999684 103.970883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_48\">\n     <g id=\"line2d_22\">\n      <path clip-path=\"url(#pa7436bb53b)\" d=\"M 389.598059 41.646 \nL 616.056869 41.646 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_21\">\n      <!-- 1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(370.999684 47.078883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_22\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(350.857356 102.138)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"PolyCollection_1\">\n    <defs>\n     <path d=\"M 389.598056 -145.411183 \nL 389.598056 -140.337372 \nL 394.219661 -133.582567 \nL 398.841275 -126.908904 \nL 403.462881 -120.37696 \nL 408.084486 -114.076294 \nL 412.7061 -108.130424 \nL 417.327705 -102.620805 \nL 421.949319 -97.688904 \nL 426.570924 -92.556829 \nL 431.192534 -87.727392 \nL 435.814144 -83.675348 \nL 440.435749 -80.540302 \nL 445.057359 -78.440307 \nL 449.678968 -77.470704 \nL 454.300574 -77.673372 \nL 458.922183 -79.001257 \nL 463.543793 -81.330507 \nL 468.165398 -84.39667 \nL 472.787012 -87.932088 \nL 477.408617 -91.787337 \nL 482.030231 -96.19928 \nL 486.651837 -102.067016 \nL 491.273442 -111.095025 \nL 495.895056 -125.26477 \nL 500.516661 -124.605242 \nL 505.138267 -124.54703 \nL 509.75988 -130.535414 \nL 514.381486 -139.877203 \nL 519.003091 -149.798352 \nL 523.624705 -160.197772 \nL 528.24631 -170.683809 \nL 532.867924 -180.61465 \nL 537.48953 -189.425248 \nL 542.111135 -196.732407 \nL 546.73274 -202.354249 \nL 551.354363 -204.216316 \nL 555.975968 -205.227724 \nL 560.597573 -205.321682 \nL 565.219179 -204.42401 \nL 569.840784 -202.529761 \nL 574.462407 -199.667239 \nL 579.084012 -195.906454 \nL 583.705617 -191.350405 \nL 588.327223 -186.123811 \nL 592.948828 -180.287415 \nL 597.570433 -173.933634 \nL 602.192056 -167.282076 \nL 606.813661 -160.288094 \nL 611.435266 -153.184046 \nL 616.056872 -146.052783 \nL 616.056872 -146.313376 \nL 616.056872 -146.313376 \nL 611.435266 -153.432512 \nL 606.813661 -160.463786 \nL 602.192056 -167.360646 \nL 597.570433 -174.169351 \nL 592.948828 -180.672338 \nL 588.327223 -186.869206 \nL 583.705617 -192.631874 \nL 579.084012 -197.754143 \nL 574.462407 -202.099162 \nL 569.840784 -205.510312 \nL 565.219179 -207.795274 \nL 560.597573 -208.778835 \nL 555.975968 -208.288994 \nL 551.354363 -206.183749 \nL 546.73274 -202.432825 \nL 542.111135 -200.157263 \nL 537.48953 -197.643793 \nL 532.867924 -195.222653 \nL 528.24631 -193.137321 \nL 523.624705 -191.419131 \nL 519.003091 -189.745137 \nL 514.381486 -187.296876 \nL 509.75988 -182.141019 \nL 505.138267 -168.459451 \nL 500.516661 -145.097723 \nL 495.895056 -125.553462 \nL 491.273442 -126.350487 \nL 486.651837 -124.313221 \nL 482.030231 -119.548702 \nL 477.408617 -113.297627 \nL 472.787012 -106.711387 \nL 468.165398 -100.56279 \nL 463.543793 -95.27981 \nL 458.922183 -91.057598 \nL 454.300574 -87.987833 \nL 449.678968 -86.093924 \nL 445.057359 -85.384911 \nL 440.435749 -85.836762 \nL 435.814144 -87.388225 \nL 431.192534 -89.951126 \nL 426.570924 -93.431768 \nL 421.949319 -98.057441 \nL 417.327705 -104.071724 \nL 412.7061 -110.478392 \nL 408.084486 -117.183604 \nL 403.462881 -124.109735 \nL 398.841275 -131.164852 \nL 394.219661 -138.285726 \nL 389.598056 -145.411183 \nz\n\" id=\"m0cf8bb0a6e\" style=\"stroke:#0000ff;stroke-opacity:0.2;\"/>\n    </defs>\n    <g clip-path=\"url(#pa7436bb53b)\">\n     <use style=\"fill:#0000ff;fill-opacity:0.2;stroke:#0000ff;stroke-opacity:0.2;\" x=\"0\" xlink:href=\"#m0cf8bb0a6e\" y=\"237.771969\"/>\n    </g>\n   </g>\n   <g id=\"line2d_23\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"mae383d4d60\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#pa7436bb53b)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"613.744041\" xlink:href=\"#mae383d4d60\" y=\"88.292384\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"596.138257\" xlink:href=\"#mae383d4d60\" y=\"61.345553\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"423.640517\" xlink:href=\"#mae383d4d60\" y=\"141.72304\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"495.972213\" xlink:href=\"#mae383d4d60\" y=\"112.235927\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"546.828399\" xlink:href=\"#mae383d4d60\" y=\"35.346488\"/>\n    </g>\n   </g>\n   <g id=\"line2d_24\">\n    <path clip-path=\"url(#pa7436bb53b)\" d=\"M 389.598056 94.897691 \nL 394.219661 101.837822 \nL 398.841275 108.735091 \nL 403.462881 115.528621 \nL 408.084486 122.14202 \nL 412.7061 128.467561 \nL 417.327705 134.425705 \nL 421.949319 139.898796 \nL 426.570924 144.777671 \nL 431.192534 148.93271 \nL 435.814144 152.240183 \nL 440.435749 154.583437 \nL 445.057359 155.859358 \nL 449.678968 155.989655 \nL 454.300574 154.941367 \nL 458.922183 152.742543 \nL 463.543793 149.46681 \nL 468.165398 145.292239 \nL 472.787012 140.450231 \nL 477.408617 135.229487 \nL 482.030231 129.897978 \nL 486.651837 124.58185 \nL 491.273442 119.049213 \nL 495.895056 112.362853 \nL 500.516661 102.920486 \nL 505.138267 91.268729 \nL 509.75988 81.433752 \nL 514.381486 74.184929 \nL 519.003091 68.000224 \nL 523.624705 61.963517 \nL 528.24631 55.861404 \nL 532.867924 49.853317 \nL 537.48953 44.23745 \nL 542.111135 39.327134 \nL 546.73274 35.378432 \nL 551.354363 32.571936 \nL 555.975968 31.01361 \nL 560.597573 30.72171 \nL 565.219179 31.662327 \nL 569.840784 33.751932 \nL 574.462407 36.888768 \nL 579.084012 40.94167 \nL 583.705617 45.780829 \nL 588.327223 51.27546 \nL 592.948828 57.292092 \nL 597.570433 63.720476 \nL 602.192056 70.450608 \nL 606.813661 77.396029 \nL 611.435266 84.46369 \nL 616.056872 91.58889 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"patch_9\">\n    <path d=\"M 389.598059 183.876 \nL 389.598059 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_10\">\n    <path d=\"M 616.056869 183.876 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_11\">\n    <path d=\"M 389.598059 183.876 \nL 616.056869 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_12\">\n    <path d=\"M 389.598059 13.2 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_2\">\n    <g id=\"patch_13\">\n     <path d=\"M 396.598059 50.55625 \nL 508.304309 50.55625 \nQ 510.304309 50.55625 510.304309 48.55625 \nL 510.304309 20.2 \nQ 510.304309 18.2 508.304309 18.2 \nL 396.598059 18.2 \nQ 394.598059 18.2 394.598059 20.2 \nL 394.598059 48.55625 \nQ 394.598059 50.55625 396.598059 50.55625 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_25\"/>\n    <g id=\"line2d_26\">\n     <g>\n      <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"408.598059\" xlink:href=\"#mae383d4d60\" y=\"26.298438\"/>\n     </g>\n    </g>\n    <g id=\"text_23\">\n     <!-- Train -->\n     <g style=\"fill:#262626;\" transform=\"translate(426.598059 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"60.865234\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"101.978516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"163.257812\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"191.041016\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_27\">\n     <path d=\"M 398.598059 40.976562 \nL 418.598059 40.976562 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_28\"/>\n    <g id=\"text_24\">\n     <!-- Infinite Network -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 23.09375 72.90625 \nL 55.421875 11.921875 \nL 55.421875 72.90625 \nL 64.984375 72.90625 \nL 64.984375 0 \nL 51.703125 0 \nL 19.390625 60.984375 \nL 19.390625 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-78\"/>\n      <path d=\"M 4.203125 54.6875 \nL 13.1875 54.6875 \nL 24.421875 12.015625 \nL 35.59375 54.6875 \nL 46.1875 54.6875 \nL 57.421875 12.015625 \nL 68.609375 54.6875 \nL 77.59375 54.6875 \nL 63.28125 0 \nL 52.6875 0 \nL 40.921875 44.828125 \nL 29.109375 0 \nL 18.5 0 \nz\n\" id=\"DejaVuSans-119\"/>\n      <path d=\"M 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 31.109375 \nL 44.921875 54.6875 \nL 56.390625 54.6875 \nL 27.390625 29.109375 \nL 57.625 0 \nL 45.90625 0 \nL 18.109375 26.703125 \nL 18.109375 0 \nL 9.078125 0 \nz\n\" id=\"DejaVuSans-107\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(426.598059 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-78\"/>\n      <use x=\"454.345703\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"515.869141\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"555.078125\" xlink:href=\"#DejaVuSans-119\"/>\n      <use x=\"636.865234\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"698.046875\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"739.160156\" xlink:href=\"#DejaVuSans-107\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p23f3c146c7\">\n   <rect height=\"170.676\" width=\"226.458809\" x=\"73.66125\" y=\"13.2\"/>\n  </clipPath>\n  <clipPath id=\"pa7436bb53b\">\n   <rect height=\"170.676\" width=\"226.458809\" x=\"389.598059\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 900x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(1, 2, 1)\n",
    "\n",
    "plt.loglog(ts, ntk_train_loss_mean, linewidth=3)\n",
    "plt.loglog(ts, ntk_test_loss_mean, linewidth=3)\n",
    "format_plot('Step', 'Loss')\n",
    "legend(['Infinite Train', 'Infinite Test'])\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "\n",
    "plot_fn(train, None)\n",
    "\n",
    "plt.plot(test_xs, ntk_mean, 'b-', linewidth=3)\n",
    "plt.fill_between(\n",
    "    jnp.reshape(test_xs, (-1)),\n",
    "    ntk_mean - 2 * ntk_std,\n",
    "    ntk_mean +  2 * ntk_std,\n",
    "    color='blue', alpha=0.2)\n",
    "\n",
    "legend(\n",
    "    ['Train', 'Infinite Network'],\n",
    "    loc='upper left')\n",
    "\n",
    "finalize_plot((1.5, 0.6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "98jPUzUEOXgy"
   },
   "source": [
    "## Training a Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L9Zu-EXywF22"
   },
   "source": [
    "We will now compare the results of gradient descent GP-inference computed above to the result of training an ensemble of finite width neural networks. We first train a single network drawn from the prior and then we will show how to generalize this to an ensemble. To do this we use JAX's gradient descent [optimizer](https://github.com/google/jax#first-order-optimization). Optimizers are described by a triple of functions `(init_fn, update_fn, get_params)`. Here, `init_fn(params)` takes an initial set of parameters and returns an optimizer state that can include extra information (like the velocity in the `momentum` optimizer). `opt_update(step, grads, state)` takes a new state and updates it using gradients. Finally, `get_params(state)` returns the parameters for a given state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gG36dJv_VUaR"
   },
   "outputs": [],
   "source": [
    "learning_rate = 0.1\n",
    "training_steps = 10000\n",
    "\n",
    "opt_init, opt_update, get_params = optimizers.sgd(learning_rate)\n",
    "opt_update = jit(opt_update)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4KBX5kpiUKHi"
   },
   "source": [
    "Next we need to define a loss and a gradient of the loss. We'll use an MSE loss. The function `grad` is another [JAX function](https://github.com/google/jax#automatic-differentiation-with-grad) that takes a function and returns a new function that computes its gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wTGamMDaUTu5"
   },
   "outputs": [],
   "source": [
    "loss = jit(lambda params, x, y: 0.5 * jnp.mean((apply_fn(params, x) - y) ** 2))\n",
    "grad_loss = jit(lambda state, x, y: grad(loss)(get_params(state), x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1plIKII_USSM"
   },
   "source": [
    "Now we want to actually train the network. To do this we just initialize the optimizer state and then update it however many times we want. We'll record the train and test loss after each step.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BNRGGcNLUE2l"
   },
   "outputs": [],
   "source": [
    "train_losses = []\n",
    "test_losses = []\n",
    "\n",
    "opt_state = opt_init(params)\n",
    "\n",
    "for i in range(training_steps):\n",
    "  opt_state = opt_update(i, grad_loss(opt_state, *train), opt_state)\n",
    "\n",
    "  train_losses += [loss(get_params(opt_state), *train)]\n",
    "  test_losses += [loss(get_params(opt_state), *test)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R7jQRWzRy4OF"
   },
   "source": [
    "Finally, lets plot the loss over the course of training and the function after training compared with our GP inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 338
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2650,
     "status": "ok",
     "timestamp": 1588653690209,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "KcqvB9M-WZiv",
    "outputId": "579b4b84-868f-4f05-bb2d-2ee4357c5a32"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"237.771969pt\" version=\"1.1\" viewBox=\"0 0 629.256869 237.771969\" width=\"629.256869pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 237.771969 \nL 629.256869 237.771969 \nL 629.256869 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 73.66125 183.876 \nL 300.120059 183.876 \nL 300.120059 13.2 \nL 73.66125 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 83.954832 183.876 \nL 83.954832 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(67.152332 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 135.423302 183.876 \nL 135.423302 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(122.839302 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 186.891772 183.876 \nL 186.891772 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(174.307772 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 238.360242 183.876 \nL 238.360242 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_4\">\n      <!-- $\\mathdefault{10^{2}}$ -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(225.776242 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 289.828712 183.876 \nL 289.828712 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_5\">\n      <!-- $\\mathdefault{10^{3}}$ -->\n      <defs>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(277.244712 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\"/>\n    <g id=\"xtick_7\"/>\n    <g id=\"xtick_8\"/>\n    <g id=\"xtick_9\"/>\n    <g id=\"xtick_10\"/>\n    <g id=\"xtick_11\"/>\n    <g id=\"xtick_12\"/>\n    <g id=\"xtick_13\"/>\n    <g id=\"xtick_14\"/>\n    <g id=\"xtick_15\"/>\n    <g id=\"xtick_16\"/>\n    <g id=\"xtick_17\"/>\n    <g id=\"xtick_18\"/>\n    <g id=\"xtick_19\"/>\n    <g id=\"xtick_20\"/>\n    <g id=\"xtick_21\"/>\n    <g id=\"xtick_22\"/>\n    <g id=\"xtick_23\"/>\n    <g id=\"xtick_24\"/>\n    <g id=\"xtick_25\"/>\n    <g id=\"xtick_26\"/>\n    <g id=\"xtick_27\"/>\n    <g id=\"xtick_28\"/>\n    <g id=\"xtick_29\"/>\n    <g id=\"xtick_30\"/>\n    <g id=\"xtick_31\"/>\n    <g id=\"xtick_32\"/>\n    <g id=\"xtick_33\"/>\n    <g id=\"xtick_34\"/>\n    <g id=\"xtick_35\"/>\n    <g id=\"xtick_36\"/>\n    <g id=\"xtick_37\"/>\n    <g id=\"xtick_38\"/>\n    <g id=\"xtick_39\"/>\n    <g id=\"xtick_40\"/>\n    <g id=\"text_6\">\n     <!-- Step -->\n     <defs>\n      <path d=\"M 53.515625 70.515625 \nL 53.515625 60.890625 \nQ 47.90625 63.578125 42.921875 64.890625 \nQ 37.9375 66.21875 33.296875 66.21875 \nQ 25.25 66.21875 20.875 63.09375 \nQ 16.5 59.96875 16.5 54.203125 \nQ 16.5 49.359375 19.40625 46.890625 \nQ 22.3125 44.4375 30.421875 42.921875 \nL 36.375 41.703125 \nQ 47.40625 39.59375 52.65625 34.296875 \nQ 57.90625 29 57.90625 20.125 \nQ 57.90625 9.515625 50.796875 4.046875 \nQ 43.703125 -1.421875 29.984375 -1.421875 \nQ 24.8125 -1.421875 18.96875 -0.25 \nQ 13.140625 0.921875 6.890625 3.21875 \nL 6.890625 13.375 \nQ 12.890625 10.015625 18.65625 8.296875 \nQ 24.421875 6.59375 29.984375 6.59375 \nQ 38.421875 6.59375 43.015625 9.90625 \nQ 47.609375 13.234375 47.609375 19.390625 \nQ 47.609375 24.75 44.3125 27.78125 \nQ 41.015625 30.8125 33.5 32.328125 \nL 27.484375 33.5 \nQ 16.453125 35.6875 11.515625 40.375 \nQ 6.59375 45.0625 6.59375 53.421875 \nQ 6.59375 63.09375 13.40625 68.65625 \nQ 20.21875 74.21875 32.171875 74.21875 \nQ 37.3125 74.21875 42.625 73.28125 \nQ 47.953125 72.359375 53.515625 70.515625 \nz\n\" id=\"DejaVuSans-83\"/>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(164.120342 226.412594)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-83\"/>\n      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"102.685547\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"164.208984\" xlink:href=\"#DejaVuSans-112\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 73.66125 157.023279 \nL 300.120059 157.023279 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- $\\mathdefault{10^{-4}}$ -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 162.456162)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-52\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 73.66125 122.22088 \nL 300.120059 122.22088 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-3}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 127.653763)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 73.66125 87.418481 \nL 300.120059 87.418481 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{-2}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 92.851364)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 73.66125 52.616082 \nL 300.120059 52.616082 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 58.048965)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_10\">\n      <path clip-path=\"url(#pcc88ee0d92)\" d=\"M 73.66125 17.813683 \nL 300.120059 17.813683 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_11\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(38.99325 23.246565)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\"/>\n    <g id=\"ytick_7\"/>\n    <g id=\"ytick_8\"/>\n    <g id=\"ytick_9\"/>\n    <g id=\"ytick_10\"/>\n    <g id=\"ytick_11\"/>\n    <g id=\"ytick_12\"/>\n    <g id=\"ytick_13\"/>\n    <g id=\"ytick_14\"/>\n    <g id=\"ytick_15\"/>\n    <g id=\"ytick_16\"/>\n    <g id=\"ytick_17\"/>\n    <g id=\"ytick_18\"/>\n    <g id=\"ytick_19\"/>\n    <g id=\"ytick_20\"/>\n    <g id=\"ytick_21\"/>\n    <g id=\"ytick_22\"/>\n    <g id=\"ytick_23\"/>\n    <g id=\"ytick_24\"/>\n    <g id=\"ytick_25\"/>\n    <g id=\"ytick_26\"/>\n    <g id=\"ytick_27\"/>\n    <g id=\"ytick_28\"/>\n    <g id=\"ytick_29\"/>\n    <g id=\"ytick_30\"/>\n    <g id=\"ytick_31\"/>\n    <g id=\"ytick_32\"/>\n    <g id=\"ytick_33\"/>\n    <g id=\"ytick_34\"/>\n    <g id=\"ytick_35\"/>\n    <g id=\"ytick_36\"/>\n    <g id=\"ytick_37\"/>\n    <g id=\"ytick_38\"/>\n    <g id=\"ytick_39\"/>\n    <g id=\"ytick_40\"/>\n    <g id=\"ytick_41\"/>\n    <g id=\"ytick_42\"/>\n    <g id=\"ytick_43\"/>\n    <g id=\"ytick_44\"/>\n    <g id=\"ytick_45\"/>\n    <g id=\"text_12\">\n     <!-- Loss -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 8.296875 \nL 55.171875 8.296875 \nL 55.171875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-76\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 120.645812)rotate(-90)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use x=\"55.697266\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"116.878906\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"168.978516\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_11\">\n    <path clip-path=\"url(#pcc88ee0d92)\" d=\"M -1 36.262195 \nL 83.954832 36.286846 \nL 99.448386 46.793491 \nL 108.511534 52.413614 \nL 114.941939 54.961997 \nL 119.929749 56.223694 \nL 133.068233 58.941167 \nL 137.553721 60.116122 \nL 142.944288 61.819487 \nL 147.284156 63.454506 \nL 152.007435 65.527566 \nL 155.904665 67.478557 \nL 159.980003 69.748584 \nL 164.667776 72.625456 \nL 170.015149 76.183158 \nL 177.603974 81.527122 \nL 187.768451 88.954771 \nL 204.001871 100.973825 \nL 207.283546 102.98928 \nL 210.380234 104.632287 \nL 213.511054 106.04215 \nL 216.673448 107.252725 \nL 221.002903 108.673861 \nL 227.602738 110.630393 \nL 235.654736 113.174816 \nL 240.833465 115.147455 \nL 245.084874 117.088 \nL 248.655442 119.005085 \nL 252.18354 121.209347 \nL 256.276934 124.219398 \nL 258.234386 125.853734 \nL 261.028849 128.43285 \nL 264.959133 132.598864 \nL 270.891 140.256912 \nL 271.396336 140.993638 \nL 273.296598 143.881849 \nL 273.833342 144.738775 \nL 273.992792 144.989662 \nL 274.491081 145.802103 \nL 275.181728 146.94348 \nL 275.608071 147.6643 \nL 275.797303 147.990203 \nL 276.558855 149.305297 \nL 276.856358 149.828778 \nL 277.334439 150.678872 \nL 278.05375 151.982447 \nL 278.603383 152.998818 \nL 279.186651 154.090859 \nL 279.57546 154.831581 \nL 280.507021 156.634708 \nL 280.517194 156.639154 \nL 280.551072 156.722104 \nL 280.561225 156.732209 \nL 280.88372 157.367853 \nL 281.208202 158.007831 \nL 281.296779 158.184033 \nL 281.692696 158.980018 \nL 281.695911 158.978288 \nL 281.734473 159.066077 \nL 281.737682 159.066012 \nL 281.913549 159.42823 \nL 282.059583 159.707673 \nL 282.223521 160.050398 \nL 282.516857 160.655825 \nL 282.584958 160.782159 \nL 282.803332 161.244028 \nL 282.806394 161.239934 \nL 282.861411 161.349061 \nL 284.946824 165.702901 \nL 284.949606 165.693905 \nL 285.013466 165.841924 \nL 285.19571 166.244012 \nL 285.20121 166.231064 \nL 285.250644 166.341848 \nL 285.253388 166.339657 \nL 285.466322 166.811565 \nL 285.469038 166.802207 \nL 285.536851 166.96122 \nL 285.542267 166.958721 \nL 285.585549 167.068859 \nL 285.588252 167.051248 \nL 285.728326 167.355908 \nL 285.822115 167.553739 \nL 286.710721 169.454084 \nL 286.713292 169.444732 \nL 286.779999 169.605552 \nL 286.782561 169.604257 \nL 288.611481 173.527313 \nL 288.613841 173.491447 \nL 288.708046 173.724067 \nL 288.864956 174.055765 \nL 288.712746 173.723217 \nL 288.86729 174.049162 \nL 288.869623 174.039439 \nL 288.937186 174.214195 \nL 288.941836 174.203488 \nL 289.163998 174.696078 \nL 289.1663 174.669452 \nL 289.25821 174.885063 \nL 289.622117 175.676778 \nL 289.63114 175.656288 \nL 289.705434 175.836038 \nL 289.709929 175.834083 \nL 289.826477 176.118 \nL 289.826477 176.118 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#pcc88ee0d92)\" d=\"M -1 34.9827 \nL 83.954832 35.005911 \nL 108.511534 51.066914 \nL 114.941939 54.211496 \nL 119.929749 55.961716 \nL 127.450735 57.979576 \nL 135.423302 60.063025 \nL 141.287791 61.864208 \nL 145.929045 63.514364 \nL 152.007435 65.959369 \nL 165.844493 71.751425 \nL 169.534433 72.965989 \nL 173.118489 73.889688 \nL 176.564138 74.540553 \nL 180.75743 75.087531 \nL 199.14364 77.158039 \nL 226.184243 81.295801 \nL 234.195315 82.75612 \nL 241.44536 84.299591 \nL 249.185015 86.179862 \nL 263.03584 89.618809 \nL 268.15371 90.560201 \nL 272.13848 91.045546 \nL 275.964176 91.289526 \nL 279.315814 91.342593 \nL 284.395023 91.221223 \nL 289.826477 90.976459 \nL 289.826477 90.976459 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#pcc88ee0d92)\" d=\"M -1 45.426612 \nL 83.954832 45.443146 \nL 99.448386 48.102664 \nL 114.941939 50.080625 \nL 119.929749 50.909322 \nL 124.005087 51.723406 \nL 130.435492 53.319016 \nL 135.423302 54.872447 \nL 139.49864 56.38344 \nL 142.944288 57.851794 \nL 147.284156 59.97376 \nL 150.916855 61.99836 \nL 154.992194 64.545564 \nL 159.22222 67.485495 \nL 164.05534 71.156741 \nL 171.398219 77.090039 \nL 179.236266 83.342376 \nL 186.440192 88.766617 \nL 200.52154 99.033848 \nL 205.018067 102.023297 \nL 208.843655 104.302261 \nL 212.680526 106.328952 \nL 217.198039 108.448156 \nL 223.743368 111.240629 \nL 242.323537 119.109785 \nL 247.997126 121.874866 \nL 254.263537 125.33594 \nL 258.63501 128.026613 \nL 265.370145 132.660227 \nL 270.723486 136.86662 \nL 273.933701 139.662872 \nL 276.655778 142.213002 \nL 279.624914 145.198963 \nL 281.625043 147.337844 \nL 283.065018 148.945193 \nL 285.90752 152.298511 \nL 289.29487 156.643023 \nL 289.826477 157.363201 \nL 289.826477 157.363201 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"line2d_14\">\n    <path clip-path=\"url(#pcc88ee0d92)\" d=\"M -1 43.218469 \nL 83.954832 43.236367 \nL 99.448386 47.390043 \nL 124.005087 52.729237 \nL 130.435492 54.659573 \nL 135.423302 56.522192 \nL 139.49864 58.326562 \nL 142.944288 60.069644 \nL 147.284156 62.558712 \nL 152.007435 65.61621 \nL 158.437841 70.14864 \nL 163.425652 73.553875 \nL 166.410408 75.316626 \nL 169.043151 76.592438 \nL 171.398219 77.470316 \nL 173.528637 78.043764 \nL 175.843027 78.448111 \nL 178.597581 78.690478 \nL 182.455902 78.764598 \nL 190.591429 78.851319 \nL 195.042446 79.169418 \nL 200.154094 79.772636 \nL 210.380234 81.270705 \nL 224.462432 83.486323 \nL 232.806516 85.040589 \nL 241.269432 86.841077 \nL 252.612948 89.514236 \nL 262.366307 91.760926 \nL 269.419875 93.190378 \nL 275.772166 94.23707 \nL 281.039927 94.869461 \nL 286.058847 95.248651 \nL 289.826477 95.391808 \nL 289.826477 95.391808 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 73.66125 183.876 \nL 73.66125 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 300.120059 183.876 \nL 300.120059 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 73.66125 183.876 \nL 300.120059 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 73.66125 13.2 \nL 300.120059 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 197.726309 65.234375 \nL 293.120059 65.234375 \nQ 295.120059 65.234375 295.120059 63.234375 \nL 295.120059 20.2 \nQ 295.120059 18.2 293.120059 18.2 \nL 197.726309 18.2 \nQ 195.726309 18.2 195.726309 20.2 \nL 195.726309 63.234375 \nQ 195.726309 65.234375 197.726309 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_15\">\n     <path d=\"M 199.726309 26.298438 \nL 219.726309 26.298438 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_16\"/>\n    <g id=\"text_13\">\n     <!-- Infinite Train -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-73\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n      <path d=\"M 37.109375 75.984375 \nL 37.109375 68.5 \nL 28.515625 68.5 \nQ 23.6875 68.5 21.796875 66.546875 \nQ 19.921875 64.59375 19.921875 59.515625 \nL 19.921875 54.6875 \nL 34.71875 54.6875 \nL 34.71875 47.703125 \nL 19.921875 47.703125 \nL 19.921875 0 \nL 10.890625 0 \nL 10.890625 47.703125 \nL 2.296875 47.703125 \nL 2.296875 54.6875 \nL 10.890625 54.6875 \nL 10.890625 58.5 \nQ 10.890625 67.625 15.140625 71.796875 \nQ 19.390625 75.984375 28.609375 75.984375 \nz\n\" id=\"DejaVuSans-102\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path id=\"DejaVuSans-32\"/>\n      <path d=\"M -0.296875 72.90625 \nL 61.375 72.90625 \nL 61.375 64.59375 \nL 35.5 64.59375 \nL 35.5 0 \nL 25.59375 0 \nL 25.59375 64.59375 \nL -0.296875 64.59375 \nz\n\" id=\"DejaVuSans-84\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(227.726309 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"440.40625\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"481.519531\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"542.798828\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"570.582031\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_17\">\n     <path d=\"M 199.726309 40.976562 \nL 219.726309 40.976562 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_18\"/>\n    <g id=\"text_14\">\n     <!-- Infinite Test -->\n     <g style=\"fill:#262626;\" transform=\"translate(227.726309 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"440.359375\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"501.882812\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"553.982422\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n    <g id=\"line2d_19\">\n     <path d=\"M 199.726309 55.654687 \nL 219.726309 55.654687 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n    </g>\n    <g id=\"line2d_20\"/>\n    <g id=\"text_15\">\n     <!-- Finite -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 51.703125 72.90625 \nL 51.703125 64.59375 \nL 19.671875 64.59375 \nL 19.671875 43.109375 \nL 48.578125 43.109375 \nL 48.578125 34.8125 \nL 19.671875 34.8125 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-70\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(227.726309 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"57.410156\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"85.193359\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"148.572266\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"176.355469\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"215.564453\" xlink:href=\"#DejaVuSans-101\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n  <g id=\"axes_2\">\n   <g id=\"patch_8\">\n    <path d=\"M 389.598059 183.876 \nL 616.056869 183.876 \nL 616.056869 13.2 \nL 389.598059 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_3\">\n    <g id=\"xtick_41\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#pee34f021cb)\" d=\"M 430.743386 183.876 \nL 430.743386 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_16\">\n      <!-- −2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(420.202722 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_42\">\n     <g id=\"line2d_22\">\n      <path clip-path=\"url(#pee34f021cb)\" d=\"M 502.827464 183.876 \nL 502.827464 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_17\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(498.278276 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_43\">\n     <g id=\"line2d_23\">\n      <path clip-path=\"url(#pee34f021cb)\" d=\"M 574.911542 183.876 \nL 574.911542 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_18\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(570.362354 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_19\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(496.827464 226.412594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_4\">\n    <g id=\"ytick_46\">\n     <g id=\"line2d_24\">\n      <path clip-path=\"url(#pee34f021cb)\" d=\"M 389.598059 155.43 \nL 616.056869 155.43 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_20\">\n      <!-- −1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(359.016731 160.862883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_47\">\n     <g id=\"line2d_25\">\n      <path clip-path=\"url(#pee34f021cb)\" d=\"M 389.598059 98.538 \nL 616.056869 98.538 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_21\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(370.999684 103.970883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_48\">\n     <g id=\"line2d_26\">\n      <path clip-path=\"url(#pee34f021cb)\" d=\"M 389.598059 41.646 \nL 616.056869 41.646 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_22\">\n      <!-- 1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(370.999684 47.078883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_23\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(350.857356 102.138)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"PolyCollection_1\">\n    <defs>\n     <path d=\"M 389.598056 -145.411183 \nL 389.598056 -140.337372 \nL 394.219661 -133.582567 \nL 398.841275 -126.908904 \nL 403.462881 -120.37696 \nL 408.084486 -114.076294 \nL 412.7061 -108.130424 \nL 417.327705 -102.620805 \nL 421.949319 -97.688904 \nL 426.570924 -92.556829 \nL 431.192534 -87.727392 \nL 435.814144 -83.675348 \nL 440.435749 -80.540302 \nL 445.057359 -78.440307 \nL 449.678968 -77.470704 \nL 454.300574 -77.673372 \nL 458.922183 -79.001257 \nL 463.543793 -81.330507 \nL 468.165398 -84.39667 \nL 472.787012 -87.932088 \nL 477.408617 -91.787337 \nL 482.030231 -96.19928 \nL 486.651837 -102.067016 \nL 491.273442 -111.095025 \nL 495.895056 -125.26477 \nL 500.516661 -124.605242 \nL 505.138267 -124.54703 \nL 509.75988 -130.535414 \nL 514.381486 -139.877203 \nL 519.003091 -149.798352 \nL 523.624705 -160.197772 \nL 528.24631 -170.683809 \nL 532.867924 -180.61465 \nL 537.48953 -189.425248 \nL 542.111135 -196.732407 \nL 546.73274 -202.354249 \nL 551.354363 -204.216316 \nL 555.975968 -205.227724 \nL 560.597573 -205.321682 \nL 565.219179 -204.42401 \nL 569.840784 -202.529761 \nL 574.462407 -199.667239 \nL 579.084012 -195.906454 \nL 583.705617 -191.350405 \nL 588.327223 -186.123811 \nL 592.948828 -180.287415 \nL 597.570433 -173.933634 \nL 602.192056 -167.282076 \nL 606.813661 -160.288094 \nL 611.435266 -153.184046 \nL 616.056872 -146.052783 \nL 616.056872 -146.313376 \nL 616.056872 -146.313376 \nL 611.435266 -153.432512 \nL 606.813661 -160.463786 \nL 602.192056 -167.360646 \nL 597.570433 -174.169351 \nL 592.948828 -180.672338 \nL 588.327223 -186.869206 \nL 583.705617 -192.631874 \nL 579.084012 -197.754143 \nL 574.462407 -202.099162 \nL 569.840784 -205.510312 \nL 565.219179 -207.795274 \nL 560.597573 -208.778835 \nL 555.975968 -208.288994 \nL 551.354363 -206.183749 \nL 546.73274 -202.432825 \nL 542.111135 -200.157263 \nL 537.48953 -197.643793 \nL 532.867924 -195.222653 \nL 528.24631 -193.137321 \nL 523.624705 -191.419131 \nL 519.003091 -189.745137 \nL 514.381486 -187.296876 \nL 509.75988 -182.141019 \nL 505.138267 -168.459451 \nL 500.516661 -145.097723 \nL 495.895056 -125.553462 \nL 491.273442 -126.350487 \nL 486.651837 -124.313221 \nL 482.030231 -119.548702 \nL 477.408617 -113.297627 \nL 472.787012 -106.711387 \nL 468.165398 -100.56279 \nL 463.543793 -95.27981 \nL 458.922183 -91.057598 \nL 454.300574 -87.987833 \nL 449.678968 -86.093924 \nL 445.057359 -85.384911 \nL 440.435749 -85.836762 \nL 435.814144 -87.388225 \nL 431.192534 -89.951126 \nL 426.570924 -93.431768 \nL 421.949319 -98.057441 \nL 417.327705 -104.071724 \nL 412.7061 -110.478392 \nL 408.084486 -117.183604 \nL 403.462881 -124.109735 \nL 398.841275 -131.164852 \nL 394.219661 -138.285726 \nL 389.598056 -145.411183 \nz\n\" id=\"m881061ced4\" style=\"stroke:#0000ff;stroke-opacity:0.2;\"/>\n    </defs>\n    <g clip-path=\"url(#pee34f021cb)\">\n     <use style=\"fill:#0000ff;fill-opacity:0.2;stroke:#0000ff;stroke-opacity:0.2;\" x=\"0\" xlink:href=\"#m881061ced4\" y=\"237.771969\"/>\n    </g>\n   </g>\n   <g id=\"line2d_27\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"m4514306ed8\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#pee34f021cb)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"613.744041\" xlink:href=\"#m4514306ed8\" y=\"88.292384\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"596.138257\" xlink:href=\"#m4514306ed8\" y=\"61.345553\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"423.640517\" xlink:href=\"#m4514306ed8\" y=\"141.72304\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"495.972213\" xlink:href=\"#m4514306ed8\" y=\"112.235927\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"546.828399\" xlink:href=\"#m4514306ed8\" y=\"35.346488\"/>\n    </g>\n   </g>\n   <g id=\"line2d_28\">\n    <path clip-path=\"url(#pee34f021cb)\" d=\"M 389.598056 94.897691 \nL 394.219661 101.837822 \nL 398.841275 108.735091 \nL 403.462881 115.528621 \nL 408.084486 122.14202 \nL 412.7061 128.467561 \nL 417.327705 134.425705 \nL 421.949319 139.898796 \nL 426.570924 144.777671 \nL 431.192534 148.93271 \nL 435.814144 152.240183 \nL 440.435749 154.583437 \nL 445.057359 155.859358 \nL 449.678968 155.989655 \nL 454.300574 154.941367 \nL 458.922183 152.742543 \nL 463.543793 149.46681 \nL 468.165398 145.292239 \nL 472.787012 140.450231 \nL 477.408617 135.229487 \nL 482.030231 129.897978 \nL 486.651837 124.58185 \nL 491.273442 119.049213 \nL 495.895056 112.362853 \nL 500.516661 102.920486 \nL 505.138267 91.268729 \nL 509.75988 81.433752 \nL 514.381486 74.184929 \nL 519.003091 68.000224 \nL 523.624705 61.963517 \nL 528.24631 55.861404 \nL 532.867924 49.853317 \nL 537.48953 44.23745 \nL 542.111135 39.327134 \nL 546.73274 35.378432 \nL 551.354363 32.571936 \nL 555.975968 31.01361 \nL 560.597573 30.72171 \nL 565.219179 31.662327 \nL 569.840784 33.751932 \nL 574.462407 36.888768 \nL 579.084012 40.94167 \nL 583.705617 45.780829 \nL 588.327223 51.27546 \nL 592.948828 57.292092 \nL 597.570433 63.720476 \nL 602.192056 70.450608 \nL 606.813661 77.396029 \nL 611.435266 84.46369 \nL 616.056872 91.58889 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_29\">\n    <path clip-path=\"url(#pee34f021cb)\" d=\"M 389.598056 97.933126 \nL 394.219661 104.151452 \nL 398.841275 110.422003 \nL 403.462881 116.685904 \nL 408.084486 122.871916 \nL 412.7061 128.895315 \nL 417.327705 134.657397 \nL 421.949319 140.045492 \nL 426.570924 144.934136 \nL 431.192534 149.187885 \nL 435.814144 152.666523 \nL 440.435749 155.234358 \nL 445.057359 156.77343 \nL 449.678968 157.204559 \nL 454.300574 156.514925 \nL 458.922183 154.790466 \nL 463.543793 152.243611 \nL 468.165398 149.208791 \nL 472.787012 146.057529 \nL 477.408617 142.966333 \nL 482.030231 139.521463 \nL 486.651837 134.366955 \nL 491.273442 125.488331 \nL 495.895056 112.407683 \nL 500.516661 100.939802 \nL 505.138267 96.410667 \nL 509.75988 89.720444 \nL 514.381486 79.860294 \nL 519.003091 70.616778 \nL 523.624705 62.737387 \nL 528.24631 55.927285 \nL 532.867924 49.769213 \nL 537.48953 44.138795 \nL 542.111135 39.202738 \nL 546.73274 35.236246 \nL 551.354363 32.473284 \nL 555.975968 31.042054 \nL 560.597573 30.961931 \nL 565.219179 32.167509 \nL 569.840784 34.538549 \nL 574.462407 37.925561 \nL 579.084012 42.168259 \nL 583.705617 47.108057 \nL 588.327223 52.595177 \nL 592.948828 58.492977 \nL 597.570433 64.679954 \nL 602.192056 71.050264 \nL 606.813661 77.513304 \nL 611.435266 83.993254 \nL 616.056872 90.427305 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"patch_9\">\n    <path d=\"M 389.598059 183.876 \nL 389.598059 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_10\">\n    <path d=\"M 616.056869 183.876 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_11\">\n    <path d=\"M 389.598059 183.876 \nL 616.056869 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_12\">\n    <path d=\"M 389.598059 13.2 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_2\">\n    <g id=\"patch_13\">\n     <path d=\"M 396.598059 65.234375 \nL 508.304309 65.234375 \nQ 510.304309 65.234375 510.304309 63.234375 \nL 510.304309 20.2 \nQ 510.304309 18.2 508.304309 18.2 \nL 396.598059 18.2 \nQ 394.598059 18.2 394.598059 20.2 \nL 394.598059 63.234375 \nQ 394.598059 65.234375 396.598059 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_30\"/>\n    <g id=\"line2d_31\">\n     <g>\n      <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"408.598059\" xlink:href=\"#m4514306ed8\" y=\"26.298438\"/>\n     </g>\n    </g>\n    <g id=\"text_24\">\n     <!-- Train -->\n     <g style=\"fill:#262626;\" transform=\"translate(426.598059 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"60.865234\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"101.978516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"163.257812\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"191.041016\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_32\">\n     <path d=\"M 398.598059 40.976562 \nL 418.598059 40.976562 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_33\"/>\n    <g id=\"text_25\">\n     <!-- Infinite Network -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 23.09375 72.90625 \nL 55.421875 11.921875 \nL 55.421875 72.90625 \nL 64.984375 72.90625 \nL 64.984375 0 \nL 51.703125 0 \nL 19.390625 60.984375 \nL 19.390625 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-78\"/>\n      <path d=\"M 4.203125 54.6875 \nL 13.1875 54.6875 \nL 24.421875 12.015625 \nL 35.59375 54.6875 \nL 46.1875 54.6875 \nL 57.421875 12.015625 \nL 68.609375 54.6875 \nL 77.59375 54.6875 \nL 63.28125 0 \nL 52.6875 0 \nL 40.921875 44.828125 \nL 29.109375 0 \nL 18.5 0 \nz\n\" id=\"DejaVuSans-119\"/>\n      <path d=\"M 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 31.109375 \nL 44.921875 54.6875 \nL 56.390625 54.6875 \nL 27.390625 29.109375 \nL 57.625 0 \nL 45.90625 0 \nL 18.109375 26.703125 \nL 18.109375 0 \nL 9.078125 0 \nz\n\" id=\"DejaVuSans-107\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(426.598059 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-78\"/>\n      <use x=\"454.345703\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"515.869141\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"555.078125\" xlink:href=\"#DejaVuSans-119\"/>\n      <use x=\"636.865234\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"698.046875\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"739.160156\" xlink:href=\"#DejaVuSans-107\"/>\n     </g>\n    </g>\n    <g id=\"line2d_34\">\n     <path d=\"M 398.598059 55.654687 \nL 418.598059 55.654687 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n    </g>\n    <g id=\"line2d_35\"/>\n    <g id=\"text_26\">\n     <!-- Finite Network -->\n     <g style=\"fill:#262626;\" transform=\"translate(426.598059 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"57.410156\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"85.193359\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"148.572266\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"176.355469\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"215.564453\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"277.087891\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"308.875\" xlink:href=\"#DejaVuSans-78\"/>\n      <use x=\"383.679688\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"445.203125\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"484.412109\" xlink:href=\"#DejaVuSans-119\"/>\n      <use x=\"566.199219\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"627.380859\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"668.494141\" xlink:href=\"#DejaVuSans-107\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pcc88ee0d92\">\n   <rect height=\"170.676\" width=\"226.458809\" x=\"73.66125\" y=\"13.2\"/>\n  </clipPath>\n  <clipPath id=\"pee34f021cb\">\n   <rect height=\"170.676\" width=\"226.458809\" x=\"389.598059\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 900x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(1, 2, 1)\n",
    "\n",
    "plt.loglog(ts, ntk_train_loss_mean, linewidth=3)\n",
    "plt.loglog(ts, ntk_test_loss_mean, linewidth=3)\n",
    "\n",
    "plt.loglog(ts, train_losses, 'k-', linewidth=2)\n",
    "plt.loglog(ts, test_losses, 'k-', linewidth=2)\n",
    "\n",
    "format_plot('Step', 'Loss')\n",
    "legend(['Infinite Train', 'Infinite Test', 'Finite'])\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "\n",
    "plot_fn(train, None)\n",
    "\n",
    "plt.plot(test_xs, ntk_mean, 'b-', linewidth=3)\n",
    "plt.fill_between(\n",
    "    jnp.reshape(test_xs, (-1)),\n",
    "    ntk_mean - 2 * ntk_std,\n",
    "    ntk_mean +  2 * ntk_std,\n",
    "    color='blue', alpha=0.2)\n",
    "\n",
    "plt.plot(test_xs, apply_fn(get_params(opt_state), test_xs), 'k-', linewidth=2)\n",
    "\n",
    "legend(\n",
    "    ['Train', 'Infinite Network', 'Finite Network'],\n",
    "    loc='upper left')\n",
    "\n",
    "finalize_plot((1.5, 0.6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RcLr3DjtaCzE"
   },
   "source": [
    "## Training an Ensemble of Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GC-gh5Gez7xd"
   },
   "source": [
    "The draw above certainly seems consistent with exact inference. However, as discussed above to make a more quantitative comparison we want to train an ensemble of finite-width networks. We could use a for-loop to loop over all the different instantiations that we wanted to evaluate. However, it is more convenient and efficient to use another [JAX feature](https://github.com/google/jax#auto-vectorization-with-vmap) called `vmap`. `vmap` takes a function and vectorizes it over a batch dimension. In this case, we're going to add a batch dimension to our training loop so that we train a whole batch of neural networks at once. To do that, let's first wrap our whole training loop in a function. The function will take a random state and train a network based on that random state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "P-w2MxZz1yO1"
   },
   "outputs": [],
   "source": [
    "def train_network(key):\n",
    "  train_losses = []\n",
    "  test_losses = []\n",
    "\n",
    "  _, params = init_fn(key, (-1, 1))\n",
    "  opt_state = opt_init(params)\n",
    "\n",
    "  for i in range(training_steps):\n",
    "    train_losses += [jnp.reshape(loss(get_params(opt_state), *train), (1,))]\n",
    "    test_losses += [jnp.reshape(loss(get_params(opt_state), *test), (1,))]\n",
    "    opt_state = opt_update(i, grad_loss(opt_state, *train), opt_state)\n",
    "\n",
    "  train_losses = jnp.concatenate(train_losses)\n",
    "  test_losses = jnp.concatenate(test_losses)\n",
    "  return get_params(opt_state), train_losses, test_losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IVrlZGhh2Ck-"
   },
   "source": [
    "We can test it to make sure that we get a trained network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "83ld3fTm2MiI"
   },
   "outputs": [],
   "source": [
    "# test {\"skip\": true}\n",
    "params, train_loss, test_loss = train_network(key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 338
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1962,
     "status": "ok",
     "timestamp": 1588653706584,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "_8XLiCZrvCjQ",
    "outputId": "a15cb757-bdb8-4c56-c8a9-f503316b5054"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"237.771969pt\" version=\"1.1\" viewBox=\"0 0 629.256869 237.771969\" width=\"629.256869pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 237.771969 \nL 629.256869 237.771969 \nL 629.256869 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 73.66125 183.876 \nL 300.120059 183.876 \nL 300.120059 13.2 \nL 73.66125 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 83.954832 183.876 \nL 83.954832 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(67.152332 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 135.423302 183.876 \nL 135.423302 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(122.839302 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 186.891772 183.876 \nL 186.891772 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(174.307772 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 238.360242 183.876 \nL 238.360242 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_4\">\n      <!-- $\\mathdefault{10^{2}}$ -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(225.776242 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 289.828712 183.876 \nL 289.828712 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_5\">\n      <!-- $\\mathdefault{10^{3}}$ -->\n      <defs>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(277.244712 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\"/>\n    <g id=\"xtick_7\"/>\n    <g id=\"xtick_8\"/>\n    <g id=\"xtick_9\"/>\n    <g id=\"xtick_10\"/>\n    <g id=\"xtick_11\"/>\n    <g id=\"xtick_12\"/>\n    <g id=\"xtick_13\"/>\n    <g id=\"xtick_14\"/>\n    <g id=\"xtick_15\"/>\n    <g id=\"xtick_16\"/>\n    <g id=\"xtick_17\"/>\n    <g id=\"xtick_18\"/>\n    <g id=\"xtick_19\"/>\n    <g id=\"xtick_20\"/>\n    <g id=\"xtick_21\"/>\n    <g id=\"xtick_22\"/>\n    <g id=\"xtick_23\"/>\n    <g id=\"xtick_24\"/>\n    <g id=\"xtick_25\"/>\n    <g id=\"xtick_26\"/>\n    <g id=\"xtick_27\"/>\n    <g id=\"xtick_28\"/>\n    <g id=\"xtick_29\"/>\n    <g id=\"xtick_30\"/>\n    <g id=\"xtick_31\"/>\n    <g id=\"xtick_32\"/>\n    <g id=\"xtick_33\"/>\n    <g id=\"xtick_34\"/>\n    <g id=\"xtick_35\"/>\n    <g id=\"xtick_36\"/>\n    <g id=\"xtick_37\"/>\n    <g id=\"xtick_38\"/>\n    <g id=\"xtick_39\"/>\n    <g id=\"xtick_40\"/>\n    <g id=\"text_6\">\n     <!-- Step -->\n     <defs>\n      <path d=\"M 53.515625 70.515625 \nL 53.515625 60.890625 \nQ 47.90625 63.578125 42.921875 64.890625 \nQ 37.9375 66.21875 33.296875 66.21875 \nQ 25.25 66.21875 20.875 63.09375 \nQ 16.5 59.96875 16.5 54.203125 \nQ 16.5 49.359375 19.40625 46.890625 \nQ 22.3125 44.4375 30.421875 42.921875 \nL 36.375 41.703125 \nQ 47.40625 39.59375 52.65625 34.296875 \nQ 57.90625 29 57.90625 20.125 \nQ 57.90625 9.515625 50.796875 4.046875 \nQ 43.703125 -1.421875 29.984375 -1.421875 \nQ 24.8125 -1.421875 18.96875 -0.25 \nQ 13.140625 0.921875 6.890625 3.21875 \nL 6.890625 13.375 \nQ 12.890625 10.015625 18.65625 8.296875 \nQ 24.421875 6.59375 29.984375 6.59375 \nQ 38.421875 6.59375 43.015625 9.90625 \nQ 47.609375 13.234375 47.609375 19.390625 \nQ 47.609375 24.75 44.3125 27.78125 \nQ 41.015625 30.8125 33.5 32.328125 \nL 27.484375 33.5 \nQ 16.453125 35.6875 11.515625 40.375 \nQ 6.59375 45.0625 6.59375 53.421875 \nQ 6.59375 63.09375 13.40625 68.65625 \nQ 20.21875 74.21875 32.171875 74.21875 \nQ 37.3125 74.21875 42.625 73.28125 \nQ 47.953125 72.359375 53.515625 70.515625 \nz\n\" id=\"DejaVuSans-83\"/>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(164.120342 226.412594)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-83\"/>\n      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"102.685547\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"164.208984\" xlink:href=\"#DejaVuSans-112\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 73.66125 157.245869 \nL 300.120059 157.245869 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- $\\mathdefault{10^{-4}}$ -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 162.678752)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-52\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 73.66125 122.849167 \nL 300.120059 122.849167 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-3}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 128.28205)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 73.66125 88.452465 \nL 300.120059 88.452465 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{-2}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 93.885348)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 73.66125 54.055763 \nL 300.120059 54.055763 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 59.488646)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_10\">\n      <path clip-path=\"url(#p397120f2d2)\" d=\"M 73.66125 19.659061 \nL 300.120059 19.659061 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_11\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(38.99325 25.091944)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\"/>\n    <g id=\"ytick_7\"/>\n    <g id=\"ytick_8\"/>\n    <g id=\"ytick_9\"/>\n    <g id=\"ytick_10\"/>\n    <g id=\"ytick_11\"/>\n    <g id=\"ytick_12\"/>\n    <g id=\"ytick_13\"/>\n    <g id=\"ytick_14\"/>\n    <g id=\"ytick_15\"/>\n    <g id=\"ytick_16\"/>\n    <g id=\"ytick_17\"/>\n    <g id=\"ytick_18\"/>\n    <g id=\"ytick_19\"/>\n    <g id=\"ytick_20\"/>\n    <g id=\"ytick_21\"/>\n    <g id=\"ytick_22\"/>\n    <g id=\"ytick_23\"/>\n    <g id=\"ytick_24\"/>\n    <g id=\"ytick_25\"/>\n    <g id=\"ytick_26\"/>\n    <g id=\"ytick_27\"/>\n    <g id=\"ytick_28\"/>\n    <g id=\"ytick_29\"/>\n    <g id=\"ytick_30\"/>\n    <g id=\"ytick_31\"/>\n    <g id=\"ytick_32\"/>\n    <g id=\"ytick_33\"/>\n    <g id=\"ytick_34\"/>\n    <g id=\"ytick_35\"/>\n    <g id=\"ytick_36\"/>\n    <g id=\"ytick_37\"/>\n    <g id=\"ytick_38\"/>\n    <g id=\"ytick_39\"/>\n    <g id=\"ytick_40\"/>\n    <g id=\"ytick_41\"/>\n    <g id=\"ytick_42\"/>\n    <g id=\"ytick_43\"/>\n    <g id=\"ytick_44\"/>\n    <g id=\"ytick_45\"/>\n    <g id=\"text_12\">\n     <!-- Loss -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 8.296875 \nL 55.171875 8.296875 \nL 55.171875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-76\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 120.645812)rotate(-90)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use x=\"55.697266\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"116.878906\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"168.978516\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_11\">\n    <path clip-path=\"url(#p397120f2d2)\" d=\"M -1 37.892517 \nL 83.954832 37.916879 \nL 99.448386 48.301047 \nL 108.511534 53.855656 \nL 114.941939 56.374331 \nL 119.929749 57.621321 \nL 133.068233 60.307116 \nL 137.553721 61.468375 \nL 142.944288 63.151883 \nL 147.284156 64.767842 \nL 152.007435 66.816736 \nL 155.904665 68.744984 \nL 159.980003 70.988549 \nL 164.667776 73.831885 \nL 170.015149 77.348115 \nL 177.940107 82.868131 \nL 187.982353 90.129052 \nL 203.792967 101.714315 \nL 207.193596 103.79092 \nL 210.380234 105.465607 \nL 213.511054 106.859036 \nL 216.673448 108.055498 \nL 221.051442 109.475136 \nL 237.307628 114.492928 \nL 241.542502 116.155784 \nL 246.401995 118.441765 \nL 249.661922 120.259455 \nL 254.816282 123.703056 \nL 256.276934 124.824388 \nL 258.24357 126.447572 \nL 261.358809 129.309346 \nL 266.311443 134.680412 \nL 267.041897 135.568861 \nL 271.114137 140.996439 \nL 273.230938 144.15335 \nL 273.529535 144.619727 \nL 274.069831 145.475856 \nL 275.112753 147.171108 \nL 275.237614 147.376697 \nL 276.092652 148.812601 \nL 276.322879 149.208926 \nL 276.936088 150.273997 \nL 278.503438 153.084272 \nL 278.695528 153.435725 \nL 279.943714 155.776333 \nL 280.285455 156.431531 \nL 281.910364 159.606221 \nL 282.094362 159.976471 \nL 282.311302 160.420894 \nL 283.08316 161.976939 \nL 283.13147 162.054058 \nL 283.134487 162.071804 \nL 284.269254 164.41422 \nL 284.272121 164.404422 \nL 284.3408 164.557067 \nL 284.343658 164.553799 \nL 284.50591 164.896435 \nL 285.319121 166.610105 \nL 285.321857 166.596675 \nL 285.398294 166.766327 \nL 285.450013 166.863584 \nL 285.590953 167.185492 \nL 285.593656 167.180297 \nL 285.647621 167.295009 \nL 285.65301 167.296427 \nL 285.803388 167.624338 \nL 285.806064 167.616425 \nL 286.135447 168.326627 \nL 286.14072 168.304093 \nL 286.209157 168.475053 \nL 286.599944 169.300759 \nL 288.32643 172.964466 \nL 288.32882 172.955703 \nL 288.459906 173.244562 \nL 288.464659 173.236281 \nL 288.783126 173.944913 \nL 288.785469 173.905989 \nL 288.878955 174.126073 \nL 288.883618 174.098535 \nL 289.004543 174.382534 \nL 289.006863 174.367119 \nL 289.622117 175.681921 \nL 289.628884 175.663198 \nL 289.653679 175.689954 \nL 289.662688 175.766086 \nL 289.667192 175.765895 \nL 289.826477 176.118 \nL 289.826477 176.118 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#p397120f2d2)\" d=\"M -1 36.627937 \nL 83.954832 36.650877 \nL 108.511534 52.524655 \nL 114.941939 55.632579 \nL 119.929749 57.362396 \nL 127.450735 59.356735 \nL 135.423302 61.415896 \nL 141.287791 63.196082 \nL 145.929045 64.827002 \nL 152.007435 67.243506 \nL 165.844493 72.968043 \nL 169.534433 74.168448 \nL 173.118489 75.08138 \nL 176.564138 75.724657 \nL 180.75743 76.265259 \nL 199.14364 78.311631 \nL 226.222748 82.407563 \nL 234.383039 83.881193 \nL 241.56188 85.396255 \nL 249.240035 87.242164 \nL 263.396074 90.702372 \nL 268.382426 91.591313 \nL 272.256538 92.04789 \nL 276.607369 92.299017 \nL 281.125872 92.309764 \nL 286.03767 92.142346 \nL 289.826477 91.968967 \nL 289.826477 91.968967 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#p397120f2d2)\" d=\"M -1 41.843886 \nL 83.954832 41.878126 \nL 99.448386 54.869655 \nL 108.511534 59.825358 \nL 114.941939 61.472708 \nL 127.450735 63.719195 \nL 133.068233 65.036289 \nL 137.553721 66.326379 \nL 141.287791 67.589679 \nL 145.929045 69.432193 \nL 149.770323 71.208681 \nL 154.04088 73.469071 \nL 158.437841 76.110914 \nL 162.777709 78.998506 \nL 169.534433 83.824923 \nL 175.097876 87.703881 \nL 178.597581 89.887608 \nL 181.903962 91.681981 \nL 185.027986 93.125558 \nL 188.612042 94.526154 \nL 193.433661 96.146803 \nL 204.105595 99.677425 \nL 210.767635 102.171938 \nL 247.228823 116.50645 \nL 252.111162 118.846785 \nL 258.003517 122.030036 \nL 262.744914 124.857485 \nL 266.253757 127.115442 \nL 270.496395 130.094871 \nL 275.186033 133.804864 \nL 277.905631 136.1941 \nL 283.14956 141.363579 \nL 289.653679 148.959123 \nL 289.826477 149.180277 \nL 289.826477 149.180277 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"line2d_14\">\n    <path clip-path=\"url(#p397120f2d2)\" d=\"M -1 41.359413 \nL 83.954832 41.393177 \nL 99.448386 55.64643 \nL 108.511534 62.608289 \nL 114.941939 65.493976 \nL 119.929749 66.951567 \nL 133.068233 70.270191 \nL 137.553721 71.636101 \nL 142.944288 73.513297 \nL 150.916855 76.619012 \nL 156.781344 78.810504 \nL 159.980003 79.784331 \nL 162.777709 80.416941 \nL 165.263876 80.765631 \nL 168.026955 80.894769 \nL 170.485747 80.785069 \nL 173.528637 80.396426 \nL 177.262709 79.653577 \nL 187.114187 77.551536 \nL 190.591429 77.101515 \nL 193.92857 76.891661 \nL 197.397515 76.904528 \nL 201.120837 77.152835 \nL 205.31413 77.664217 \nL 210.536 78.536062 \nL 217.822928 79.997973 \nL 227.238089 82.122296 \nL 236.178353 84.358253 \nL 245.462207 86.917464 \nL 263.279083 91.918845 \nL 270.549413 93.617928 \nL 275.472523 94.567243 \nL 279.290757 95.136888 \nL 284.269254 95.61032 \nL 288.523977 95.764075 \nL 289.826477 95.767772 \nL 289.826477 95.767772 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 73.66125 183.876 \nL 73.66125 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 300.120059 183.876 \nL 300.120059 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 73.66125 183.876 \nL 300.120059 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 73.66125 13.2 \nL 300.120059 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 233.412247 65.234375 \nL 293.120059 65.234375 \nQ 295.120059 65.234375 295.120059 63.234375 \nL 295.120059 20.2 \nQ 295.120059 18.2 293.120059 18.2 \nL 233.412247 18.2 \nQ 231.412247 18.2 231.412247 20.2 \nL 231.412247 63.234375 \nQ 231.412247 65.234375 233.412247 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_15\">\n     <path d=\"M 235.412247 26.298438 \nL 255.412247 26.298438 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_16\"/>\n    <g id=\"text_13\">\n     <!-- Train -->\n     <defs>\n      <path d=\"M -0.296875 72.90625 \nL 61.375 72.90625 \nL 61.375 64.59375 \nL 35.5 64.59375 \nL 35.5 0 \nL 25.59375 0 \nL 25.59375 64.59375 \nL -0.296875 64.59375 \nz\n\" id=\"DejaVuSans-84\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(263.412247 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"60.865234\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"101.978516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"163.257812\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"191.041016\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_17\">\n     <path d=\"M 235.412247 40.976562 \nL 255.412247 40.976562 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_18\"/>\n    <g id=\"text_14\">\n     <!-- Test -->\n     <g style=\"fill:#262626;\" transform=\"translate(263.412247 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"60.818359\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"122.341797\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"174.441406\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n    <g id=\"line2d_19\">\n     <path d=\"M 235.412247 55.654687 \nL 255.412247 55.654687 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n    </g>\n    <g id=\"line2d_20\"/>\n    <g id=\"text_15\">\n     <!-- Finite -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 51.703125 72.90625 \nL 51.703125 64.59375 \nL 19.671875 64.59375 \nL 19.671875 43.109375 \nL 48.578125 43.109375 \nL 48.578125 34.8125 \nL 19.671875 34.8125 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-70\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(263.412247 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"57.410156\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"85.193359\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"148.572266\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"176.355469\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"215.564453\" xlink:href=\"#DejaVuSans-101\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n  <g id=\"axes_2\">\n   <g id=\"patch_8\">\n    <path d=\"M 389.598059 183.876 \nL 616.056869 183.876 \nL 616.056869 13.2 \nL 389.598059 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_3\">\n    <g id=\"xtick_41\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#p187a8efeaf)\" d=\"M 430.743386 183.876 \nL 430.743386 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_16\">\n      <!-- −2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(420.202722 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_42\">\n     <g id=\"line2d_22\">\n      <path clip-path=\"url(#p187a8efeaf)\" d=\"M 502.827464 183.876 \nL 502.827464 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_17\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(498.278276 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_43\">\n     <g id=\"line2d_23\">\n      <path clip-path=\"url(#p187a8efeaf)\" d=\"M 574.911542 183.876 \nL 574.911542 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_18\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(570.362354 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_19\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(496.827464 226.412594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_4\">\n    <g id=\"ytick_46\">\n     <g id=\"line2d_24\">\n      <path clip-path=\"url(#p187a8efeaf)\" d=\"M 389.598059 155.43 \nL 616.056869 155.43 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_20\">\n      <!-- −1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(359.016731 160.862883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_47\">\n     <g id=\"line2d_25\">\n      <path clip-path=\"url(#p187a8efeaf)\" d=\"M 389.598059 98.538 \nL 616.056869 98.538 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_21\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(370.999684 103.970883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_48\">\n     <g id=\"line2d_26\">\n      <path clip-path=\"url(#p187a8efeaf)\" d=\"M 389.598059 41.646 \nL 616.056869 41.646 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_22\">\n      <!-- 1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(370.999684 47.078883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_23\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(350.857356 102.138)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"PolyCollection_1\">\n    <defs>\n     <path d=\"M 389.598056 -145.411183 \nL 389.598056 -140.337372 \nL 394.219661 -133.582567 \nL 398.841275 -126.908904 \nL 403.462881 -120.37696 \nL 408.084486 -114.076294 \nL 412.7061 -108.130424 \nL 417.327705 -102.620805 \nL 421.949319 -97.688904 \nL 426.570924 -92.556829 \nL 431.192534 -87.727392 \nL 435.814144 -83.675348 \nL 440.435749 -80.540302 \nL 445.057359 -78.440307 \nL 449.678968 -77.470704 \nL 454.300574 -77.673372 \nL 458.922183 -79.001257 \nL 463.543793 -81.330507 \nL 468.165398 -84.39667 \nL 472.787012 -87.932088 \nL 477.408617 -91.787337 \nL 482.030231 -96.19928 \nL 486.651837 -102.067016 \nL 491.273442 -111.095025 \nL 495.895056 -125.26477 \nL 500.516661 -124.605242 \nL 505.138267 -124.54703 \nL 509.75988 -130.535414 \nL 514.381486 -139.877203 \nL 519.003091 -149.798352 \nL 523.624705 -160.197772 \nL 528.24631 -170.683809 \nL 532.867924 -180.61465 \nL 537.48953 -189.425248 \nL 542.111135 -196.732407 \nL 546.73274 -202.354249 \nL 551.354363 -204.216316 \nL 555.975968 -205.227724 \nL 560.597573 -205.321682 \nL 565.219179 -204.42401 \nL 569.840784 -202.529761 \nL 574.462407 -199.667239 \nL 579.084012 -195.906454 \nL 583.705617 -191.350405 \nL 588.327223 -186.123811 \nL 592.948828 -180.287415 \nL 597.570433 -173.933634 \nL 602.192056 -167.282076 \nL 606.813661 -160.288094 \nL 611.435266 -153.184046 \nL 616.056872 -146.052783 \nL 616.056872 -146.313376 \nL 616.056872 -146.313376 \nL 611.435266 -153.432512 \nL 606.813661 -160.463786 \nL 602.192056 -167.360646 \nL 597.570433 -174.169351 \nL 592.948828 -180.672338 \nL 588.327223 -186.869206 \nL 583.705617 -192.631874 \nL 579.084012 -197.754143 \nL 574.462407 -202.099162 \nL 569.840784 -205.510312 \nL 565.219179 -207.795274 \nL 560.597573 -208.778835 \nL 555.975968 -208.288994 \nL 551.354363 -206.183749 \nL 546.73274 -202.432825 \nL 542.111135 -200.157263 \nL 537.48953 -197.643793 \nL 532.867924 -195.222653 \nL 528.24631 -193.137321 \nL 523.624705 -191.419131 \nL 519.003091 -189.745137 \nL 514.381486 -187.296876 \nL 509.75988 -182.141019 \nL 505.138267 -168.459451 \nL 500.516661 -145.097723 \nL 495.895056 -125.553462 \nL 491.273442 -126.350487 \nL 486.651837 -124.313221 \nL 482.030231 -119.548702 \nL 477.408617 -113.297627 \nL 472.787012 -106.711387 \nL 468.165398 -100.56279 \nL 463.543793 -95.27981 \nL 458.922183 -91.057598 \nL 454.300574 -87.987833 \nL 449.678968 -86.093924 \nL 445.057359 -85.384911 \nL 440.435749 -85.836762 \nL 435.814144 -87.388225 \nL 431.192534 -89.951126 \nL 426.570924 -93.431768 \nL 421.949319 -98.057441 \nL 417.327705 -104.071724 \nL 412.7061 -110.478392 \nL 408.084486 -117.183604 \nL 403.462881 -124.109735 \nL 398.841275 -131.164852 \nL 394.219661 -138.285726 \nL 389.598056 -145.411183 \nz\n\" id=\"mbca7e295a2\" style=\"stroke:#0000ff;stroke-opacity:0.2;\"/>\n    </defs>\n    <g clip-path=\"url(#p187a8efeaf)\">\n     <use style=\"fill:#0000ff;fill-opacity:0.2;stroke:#0000ff;stroke-opacity:0.2;\" x=\"0\" xlink:href=\"#mbca7e295a2\" y=\"237.771969\"/>\n    </g>\n   </g>\n   <g id=\"line2d_27\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"m01a7a5a974\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#p187a8efeaf)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"613.744041\" xlink:href=\"#m01a7a5a974\" y=\"88.292384\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"596.138257\" xlink:href=\"#m01a7a5a974\" y=\"61.345553\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"423.640517\" xlink:href=\"#m01a7a5a974\" y=\"141.72304\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"495.972213\" xlink:href=\"#m01a7a5a974\" y=\"112.235927\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"546.828399\" xlink:href=\"#m01a7a5a974\" y=\"35.346488\"/>\n    </g>\n   </g>\n   <g id=\"line2d_28\">\n    <path clip-path=\"url(#p187a8efeaf)\" d=\"M 389.598056 94.897691 \nL 394.219661 101.837822 \nL 398.841275 108.735091 \nL 403.462881 115.528621 \nL 408.084486 122.14202 \nL 412.7061 128.467561 \nL 417.327705 134.425705 \nL 421.949319 139.898796 \nL 426.570924 144.777671 \nL 431.192534 148.93271 \nL 435.814144 152.240183 \nL 440.435749 154.583437 \nL 445.057359 155.859358 \nL 449.678968 155.989655 \nL 454.300574 154.941367 \nL 458.922183 152.742543 \nL 463.543793 149.46681 \nL 468.165398 145.292239 \nL 472.787012 140.450231 \nL 477.408617 135.229487 \nL 482.030231 129.897978 \nL 486.651837 124.58185 \nL 491.273442 119.049213 \nL 495.895056 112.362853 \nL 500.516661 102.920486 \nL 505.138267 91.268729 \nL 509.75988 81.433752 \nL 514.381486 74.184929 \nL 519.003091 68.000224 \nL 523.624705 61.963517 \nL 528.24631 55.861404 \nL 532.867924 49.853317 \nL 537.48953 44.23745 \nL 542.111135 39.327134 \nL 546.73274 35.378432 \nL 551.354363 32.571936 \nL 555.975968 31.01361 \nL 560.597573 30.72171 \nL 565.219179 31.662327 \nL 569.840784 33.751932 \nL 574.462407 36.888768 \nL 579.084012 40.94167 \nL 583.705617 45.780829 \nL 588.327223 51.27546 \nL 592.948828 57.292092 \nL 597.570433 63.720476 \nL 602.192056 70.450608 \nL 606.813661 77.396029 \nL 611.435266 84.46369 \nL 616.056872 91.58889 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_29\">\n    <path clip-path=\"url(#p187a8efeaf)\" d=\"M 389.598056 100.260148 \nL 394.219661 106.198222 \nL 398.841275 112.175577 \nL 403.462881 118.133978 \nL 408.084486 124.003345 \nL 412.7061 129.700488 \nL 417.327705 135.128163 \nL 421.949319 140.175561 \nL 426.570924 144.718877 \nL 431.192534 148.624038 \nL 435.814144 151.752458 \nL 440.435749 153.968911 \nL 445.057359 155.154513 \nL 449.678968 155.223568 \nL 454.300574 154.143329 \nL 458.922183 151.953135 \nL 463.543793 148.773501 \nL 468.165398 144.793381 \nL 472.787012 140.224874 \nL 477.408617 135.244997 \nL 482.030231 130.005924 \nL 486.651837 124.76482 \nL 491.273442 119.579376 \nL 495.895056 112.329047 \nL 500.516661 98.685409 \nL 505.138267 84.438352 \nL 509.75988 76.276339 \nL 514.381486 70.293461 \nL 519.003091 63.938804 \nL 523.624705 57.544062 \nL 528.24631 51.686615 \nL 532.867924 46.523722 \nL 537.48953 42.010434 \nL 542.111135 38.171153 \nL 546.73274 35.136068 \nL 551.354363 33.066606 \nL 555.975968 32.087155 \nL 560.597573 32.252976 \nL 565.219179 33.548369 \nL 569.840784 35.89989 \nL 574.462407 39.194986 \nL 579.084012 43.299689 \nL 583.705617 48.072424 \nL 588.327223 53.374269 \nL 592.948828 59.075071 \nL 597.570433 65.057514 \nL 602.192056 71.218354 \nL 606.813661 77.468885 \nL 611.435266 83.734412 \nL 616.056872 89.953095 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"patch_9\">\n    <path d=\"M 389.598059 183.876 \nL 389.598059 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_10\">\n    <path d=\"M 616.056869 183.876 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_11\">\n    <path d=\"M 389.598059 183.876 \nL 616.056869 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_12\">\n    <path d=\"M 389.598059 13.2 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_2\">\n    <g id=\"patch_13\">\n     <path d=\"M 396.598059 65.234375 \nL 508.304309 65.234375 \nQ 510.304309 65.234375 510.304309 63.234375 \nL 510.304309 20.2 \nQ 510.304309 18.2 508.304309 18.2 \nL 396.598059 18.2 \nQ 394.598059 18.2 394.598059 20.2 \nL 394.598059 63.234375 \nQ 394.598059 65.234375 396.598059 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_30\"/>\n    <g id=\"line2d_31\">\n     <g>\n      <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"408.598059\" xlink:href=\"#m01a7a5a974\" y=\"26.298438\"/>\n     </g>\n    </g>\n    <g id=\"text_24\">\n     <!-- Train -->\n     <g style=\"fill:#262626;\" transform=\"translate(426.598059 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"60.865234\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"101.978516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"163.257812\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"191.041016\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_32\">\n     <path d=\"M 398.598059 40.976562 \nL 418.598059 40.976562 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_33\"/>\n    <g id=\"text_25\">\n     <!-- Infinite Network -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-73\"/>\n      <path d=\"M 37.109375 75.984375 \nL 37.109375 68.5 \nL 28.515625 68.5 \nQ 23.6875 68.5 21.796875 66.546875 \nQ 19.921875 64.59375 19.921875 59.515625 \nL 19.921875 54.6875 \nL 34.71875 54.6875 \nL 34.71875 47.703125 \nL 19.921875 47.703125 \nL 19.921875 0 \nL 10.890625 0 \nL 10.890625 47.703125 \nL 2.296875 47.703125 \nL 2.296875 54.6875 \nL 10.890625 54.6875 \nL 10.890625 58.5 \nQ 10.890625 67.625 15.140625 71.796875 \nQ 19.390625 75.984375 28.609375 75.984375 \nz\n\" id=\"DejaVuSans-102\"/>\n      <path id=\"DejaVuSans-32\"/>\n      <path d=\"M 9.8125 72.90625 \nL 23.09375 72.90625 \nL 55.421875 11.921875 \nL 55.421875 72.90625 \nL 64.984375 72.90625 \nL 64.984375 0 \nL 51.703125 0 \nL 19.390625 60.984375 \nL 19.390625 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-78\"/>\n      <path d=\"M 4.203125 54.6875 \nL 13.1875 54.6875 \nL 24.421875 12.015625 \nL 35.59375 54.6875 \nL 46.1875 54.6875 \nL 57.421875 12.015625 \nL 68.609375 54.6875 \nL 77.59375 54.6875 \nL 63.28125 0 \nL 52.6875 0 \nL 40.921875 44.828125 \nL 29.109375 0 \nL 18.5 0 \nz\n\" id=\"DejaVuSans-119\"/>\n      <path d=\"M 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 31.109375 \nL 44.921875 54.6875 \nL 56.390625 54.6875 \nL 27.390625 29.109375 \nL 57.625 0 \nL 45.90625 0 \nL 18.109375 26.703125 \nL 18.109375 0 \nL 9.078125 0 \nz\n\" id=\"DejaVuSans-107\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(426.598059 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-78\"/>\n      <use x=\"454.345703\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"515.869141\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"555.078125\" xlink:href=\"#DejaVuSans-119\"/>\n      <use x=\"636.865234\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"698.046875\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"739.160156\" xlink:href=\"#DejaVuSans-107\"/>\n     </g>\n    </g>\n    <g id=\"line2d_34\">\n     <path d=\"M 398.598059 55.654687 \nL 418.598059 55.654687 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n    </g>\n    <g id=\"line2d_35\"/>\n    <g id=\"text_26\">\n     <!-- Finite Network -->\n     <g style=\"fill:#262626;\" transform=\"translate(426.598059 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"57.410156\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"85.193359\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"148.572266\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"176.355469\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"215.564453\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"277.087891\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"308.875\" xlink:href=\"#DejaVuSans-78\"/>\n      <use x=\"383.679688\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"445.203125\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"484.412109\" xlink:href=\"#DejaVuSans-119\"/>\n      <use x=\"566.199219\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"627.380859\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"668.494141\" xlink:href=\"#DejaVuSans-107\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p397120f2d2\">\n   <rect height=\"170.676\" width=\"226.458809\" x=\"73.66125\" y=\"13.2\"/>\n  </clipPath>\n  <clipPath id=\"p187a8efeaf\">\n   <rect height=\"170.676\" width=\"226.458809\" x=\"389.598059\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 900x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# test {\"skip\": true}\n",
    "plt.subplot(1, 2, 1)\n",
    "\n",
    "plt.loglog(ts, ntk_train_loss_mean, linewidth=3)\n",
    "plt.loglog(ts, ntk_test_loss_mean, linewidth=3)\n",
    "\n",
    "plt.loglog(ts, train_loss, 'k-', linewidth=2)\n",
    "plt.loglog(ts, test_loss, 'k-', linewidth=2)\n",
    "\n",
    "format_plot('Step', 'Loss')\n",
    "legend(['Train', 'Test', 'Finite'])\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "\n",
    "plot_fn(train, None)\n",
    "\n",
    "plt.plot(test_xs, ntk_mean, 'b-', linewidth=3)\n",
    "plt.fill_between(\n",
    "    jnp.reshape(test_xs, (-1)),\n",
    "    ntk_mean - 2 * ntk_std,\n",
    "    ntk_mean +  2 * ntk_std,\n",
    "    color='blue', alpha=0.2)\n",
    "\n",
    "plt.plot(test_xs, apply_fn(params, test_xs), 'k-', linewidth=2)\n",
    "\n",
    "legend(['Train', 'Infinite Network', 'Finite Network'], loc='upper left')\n",
    "\n",
    "finalize_plot((1.5, 0.6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ar-Zs2LW2u7P"
   },
   "source": [
    "Now, to train an ensemble we just have to apply `vmap` to `train_network`. The resulting function will take a vector of random states and will train one network for each random state in the vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YJHWdE6V21i1"
   },
   "outputs": [],
   "source": [
    "# test {\"skip\": true}\n",
    "ensemble_size = 100\n",
    "ensemble_key = random.split(key, ensemble_size)\n",
    "params, train_loss, test_loss = vmap(train_network)(ensemble_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lYeAAo2v3gPn"
   },
   "source": [
    "Let's plot the empirical standard deviation in the loss over the course of training as well as well as for the function after gradient descent compared with the exact inference above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 338
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1790,
     "status": "ok",
     "timestamp": 1588653743139,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "CUG5v8jQvhqU",
    "outputId": "4c4969f4-8045-47d8-bcb7-613d17525aaf"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"237.771969pt\" version=\"1.1\" viewBox=\"0 0 629.256869 237.771969\" width=\"629.256869pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 237.771969 \nL 629.256869 237.771969 \nL 629.256869 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 73.66125 183.876 \nL 295.953457 183.876 \nL 295.953457 13.2 \nL 73.66125 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p2c843578b0)\" d=\"M 73.66125 183.876 \nL 73.66125 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(61.07725 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#p2c843578b0)\" d=\"M 147.758652 183.876 \nL 147.758652 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(135.174652 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p2c843578b0)\" d=\"M 221.856054 183.876 \nL 221.856054 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- $\\mathdefault{10^{2}}$ -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(209.272054 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#p2c843578b0)\" d=\"M 295.953457 183.876 \nL 295.953457 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_4\">\n      <!-- $\\mathdefault{10^{3}}$ -->\n      <defs>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(283.369457 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\"/>\n    <g id=\"xtick_6\"/>\n    <g id=\"xtick_7\"/>\n    <g id=\"xtick_8\"/>\n    <g id=\"xtick_9\"/>\n    <g id=\"xtick_10\"/>\n    <g id=\"xtick_11\"/>\n    <g id=\"xtick_12\"/>\n    <g id=\"xtick_13\"/>\n    <g id=\"xtick_14\"/>\n    <g id=\"xtick_15\"/>\n    <g id=\"xtick_16\"/>\n    <g id=\"xtick_17\"/>\n    <g id=\"xtick_18\"/>\n    <g id=\"xtick_19\"/>\n    <g id=\"xtick_20\"/>\n    <g id=\"xtick_21\"/>\n    <g id=\"xtick_22\"/>\n    <g id=\"xtick_23\"/>\n    <g id=\"xtick_24\"/>\n    <g id=\"xtick_25\"/>\n    <g id=\"xtick_26\"/>\n    <g id=\"xtick_27\"/>\n    <g id=\"xtick_28\"/>\n    <g id=\"text_5\">\n     <!-- Step -->\n     <defs>\n      <path d=\"M 53.515625 70.515625 \nL 53.515625 60.890625 \nQ 47.90625 63.578125 42.921875 64.890625 \nQ 37.9375 66.21875 33.296875 66.21875 \nQ 25.25 66.21875 20.875 63.09375 \nQ 16.5 59.96875 16.5 54.203125 \nQ 16.5 49.359375 19.40625 46.890625 \nQ 22.3125 44.4375 30.421875 42.921875 \nL 36.375 41.703125 \nQ 47.40625 39.59375 52.65625 34.296875 \nQ 57.90625 29 57.90625 20.125 \nQ 57.90625 9.515625 50.796875 4.046875 \nQ 43.703125 -1.421875 29.984375 -1.421875 \nQ 24.8125 -1.421875 18.96875 -0.25 \nQ 13.140625 0.921875 6.890625 3.21875 \nL 6.890625 13.375 \nQ 12.890625 10.015625 18.65625 8.296875 \nQ 24.421875 6.59375 29.984375 6.59375 \nQ 38.421875 6.59375 43.015625 9.90625 \nQ 47.609375 13.234375 47.609375 19.390625 \nQ 47.609375 24.75 44.3125 27.78125 \nQ 41.015625 30.8125 33.5 32.328125 \nL 27.484375 33.5 \nQ 16.453125 35.6875 11.515625 40.375 \nQ 6.59375 45.0625 6.59375 53.421875 \nQ 6.59375 63.09375 13.40625 68.65625 \nQ 20.21875 74.21875 32.171875 74.21875 \nQ 37.3125 74.21875 42.625 73.28125 \nQ 47.953125 72.359375 53.515625 70.515625 \nz\n\" id=\"DejaVuSans-83\"/>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(162.037041 226.412594)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-83\"/>\n      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"102.685547\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"164.208984\" xlink:href=\"#DejaVuSans-112\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p2c843578b0)\" d=\"M 73.66125 157.023279 \nL 295.953457 157.023279 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_6\">\n      <!-- $\\mathdefault{10^{-4}}$ -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 162.456162)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-52\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#p2c843578b0)\" d=\"M 73.66125 122.22088 \nL 295.953457 122.22088 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- $\\mathdefault{10^{-3}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 127.653763)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p2c843578b0)\" d=\"M 73.66125 87.418481 \nL 295.953457 87.418481 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-2}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 92.851364)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#p2c843578b0)\" d=\"M 73.66125 52.616082 \nL 295.953457 52.616082 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 58.048965)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p2c843578b0)\" d=\"M 73.66125 17.813683 \nL 295.953457 17.813683 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(38.99325 23.246565)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\"/>\n    <g id=\"ytick_7\"/>\n    <g id=\"ytick_8\"/>\n    <g id=\"ytick_9\"/>\n    <g id=\"ytick_10\"/>\n    <g id=\"ytick_11\"/>\n    <g id=\"ytick_12\"/>\n    <g id=\"ytick_13\"/>\n    <g id=\"ytick_14\"/>\n    <g id=\"ytick_15\"/>\n    <g id=\"ytick_16\"/>\n    <g id=\"ytick_17\"/>\n    <g id=\"ytick_18\"/>\n    <g id=\"ytick_19\"/>\n    <g id=\"ytick_20\"/>\n    <g id=\"ytick_21\"/>\n    <g id=\"ytick_22\"/>\n    <g id=\"ytick_23\"/>\n    <g id=\"ytick_24\"/>\n    <g id=\"ytick_25\"/>\n    <g id=\"ytick_26\"/>\n    <g id=\"ytick_27\"/>\n    <g id=\"ytick_28\"/>\n    <g id=\"ytick_29\"/>\n    <g id=\"ytick_30\"/>\n    <g id=\"ytick_31\"/>\n    <g id=\"ytick_32\"/>\n    <g id=\"ytick_33\"/>\n    <g id=\"ytick_34\"/>\n    <g id=\"ytick_35\"/>\n    <g id=\"ytick_36\"/>\n    <g id=\"ytick_37\"/>\n    <g id=\"ytick_38\"/>\n    <g id=\"ytick_39\"/>\n    <g id=\"ytick_40\"/>\n    <g id=\"ytick_41\"/>\n    <g id=\"ytick_42\"/>\n    <g id=\"ytick_43\"/>\n    <g id=\"ytick_44\"/>\n    <g id=\"ytick_45\"/>\n    <g id=\"text_11\">\n     <!-- Loss -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 8.296875 \nL 55.171875 8.296875 \nL 55.171875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-76\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 120.645812)rotate(-90)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use x=\"55.697266\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"116.878906\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"168.978516\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_10\">\n    <path clip-path=\"url(#p2c843578b0)\" d=\"M -1 36.286732 \nL -0.436152 36.286846 \nL 21.869389 46.793491 \nL 34.917295 52.413614 \nL 44.17493 54.961997 \nL 51.355709 56.223694 \nL 70.270738 58.941167 \nL 76.728341 60.116122 \nL 84.488957 61.819487 \nL 90.736917 63.454506 \nL 97.536861 65.527566 \nL 103.147571 67.478557 \nL 109.014696 69.748584 \nL 115.763522 72.625456 \nL 123.461954 76.183158 \nL 133.896018 81.282654 \nL 147.758652 88.300388 \nL 161.02038 95.288854 \nL 168.750539 99.253109 \nL 173.854462 101.626557 \nL 178.258602 103.435089 \nL 182.571243 104.964882 \nL 186.950005 106.288446 \nL 192.289184 107.651189 \nL 198.570334 109.033472 \nL 221.173076 113.980935 \nL 225.272299 115.105227 \nL 233.478373 117.778188 \nL 237.892944 119.503122 \nL 240.627742 120.690202 \nL 242.981745 121.79268 \nL 245.97592 123.310665 \nL 250.481405 125.861727 \nL 254.712285 128.583126 \nL 257.582761 130.62765 \nL 258.108592 131.019658 \nL 261.538718 133.726077 \nL 263.094499 135.042191 \nL 264.062526 135.889695 \nL 265.470363 137.163322 \nL 265.858031 137.521367 \nL 266.5956 138.214915 \nL 269.731093 141.314087 \nL 275.224116 147.36063 \nL 275.571551 147.774314 \nL 275.656317 147.872065 \nL 279.045013 152.038985 \nL 279.364516 152.447401 \nL 285.247801 160.387632 \nL 285.413437 160.628784 \nL 285.697941 161.037301 \nL 285.77751 161.142973 \nL 286.115671 161.621901 \nL 287.164368 163.13243 \nL 287.333071 163.378193 \nL 287.337277 163.373306 \nL 287.622023 163.796304 \nL 287.804934 164.064585 \nL 288.118448 164.521166 \nL 288.241366 164.70584 \nL 288.732426 165.416171 \nL 288.736453 165.40705 \nL 288.80484 165.524677 \nL 288.861051 165.606023 \nL 289.053028 165.892768 \nL 289.140636 166.003334 \nL 289.362568 166.341848 \nL 289.417814 166.420699 \nL 289.421755 166.430245 \nL 289.696532 166.846575 \nL 289.708255 166.843125 \nL 289.762911 166.930931 \nL 290.579396 168.15173 \nL 290.583198 168.140565 \nL 290.659157 168.252947 \nL 290.662952 168.245966 \nL 290.765199 168.412275 \nL 290.938659 168.684818 \nL 290.949938 168.673252 \nL 291.017535 168.794833 \nL 291.021288 168.78355 \nL 291.234387 169.130395 \nL 291.238115 169.113972 \nL 291.31998 169.24043 \nL 291.32741 169.233986 \nL 291.53481 169.559925 \nL 291.538501 169.556442 \nL 291.590137 169.654042 \nL 291.593822 169.63704 \nL 291.733543 169.861682 \nL 291.737213 169.856154 \nL 291.74455 169.852995 \nL 291.974786 170.220057 \nL 292.112867 170.410401 \nL 292.250358 170.63473 \nL 292.261187 170.629881 \nL 292.318884 170.731063 \nL 292.444735 170.908651 \nL 292.630811 171.20733 \nL 292.634378 171.200855 \nL 293.783788 172.899399 \nL 292.641512 171.191365 \nL 293.787232 172.892157 \nL 293.917766 173.100652 \nL 294.030696 173.252629 \nL 294.163652 173.476054 \nL 294.167053 173.449656 \nL 294.251983 173.620894 \nL 294.255375 173.612961 \nL 294.262159 173.605369 \nL 294.481865 173.93772 \nL 294.488602 173.957732 \nL 294.55589 174.023305 \nL 294.713463 174.282303 \nL 294.716807 174.269961 \nL 294.793628 174.395971 \nL 294.943404 174.625722 \nL 294.946726 174.621572 \nL 295.059416 174.779334 \nL 295.21455 175.015404 \nL 295.221135 174.994865 \nL 295.280337 175.118888 \nL 295.319746 175.147251 \nL 295.286909 175.135302 \nL 295.342712 175.209013 \nL 295.499752 175.43861 \nL 295.503015 175.435754 \nL 295.509541 175.41863 \nL 295.587752 175.567497 \nL 295.950239 176.118 \nL 295.950239 176.118 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_11\">\n    <path clip-path=\"url(#p2c843578b0)\" d=\"M -1 35.005804 \nL -0.436152 35.005911 \nL 34.917295 51.066914 \nL 44.17493 54.211496 \nL 51.355709 55.961716 \nL 62.183417 57.979576 \nL 73.66125 60.063025 \nL 82.104155 61.864208 \nL 88.786011 63.514364 \nL 97.536861 65.959369 \nL 117.457603 71.751425 \nL 122.769883 72.965989 \nL 127.929726 73.889688 \nL 132.890309 74.540553 \nL 138.927249 75.087531 \nL 165.397249 77.158039 \nL 204.271153 81.289316 \nL 215.743426 82.740127 \nL 226.409523 84.317361 \nL 237.618065 86.211427 \nL 257.487193 89.634539 \nL 264.192302 90.500776 \nL 269.284597 90.963612 \nL 275.32813 91.271185 \nL 281.606977 91.339347 \nL 289.731692 91.175175 \nL 295.950239 90.976459 \nL 295.950239 90.976459 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#p2c843578b0)\" d=\"M -1 41.542466 \nL -0.436152 41.542615 \nL 21.869389 50.81412 \nL 34.917295 54.055672 \nL 44.17493 55.348897 \nL 62.183417 57.433284 \nL 70.270738 58.64519 \nL 76.728341 59.825112 \nL 84.488957 61.540143 \nL 90.736917 63.187807 \nL 97.536861 65.277359 \nL 103.147571 67.243247 \nL 109.014696 69.52884 \nL 115.763522 72.421548 \nL 123.461954 75.990805 \nL 135.347999 81.796377 \nL 169.740771 98.682998 \nL 174.978771 100.894351 \nL 179.721587 102.65454 \nL 184.374225 104.14404 \nL 189.423093 105.52153 \nL 195.436824 106.920692 \nL 204.712739 108.816522 \nL 219.067719 111.767878 \nL 226.798243 113.59821 \nL 233.411026 115.387072 \nL 239.64312 117.301558 \nL 245.547254 119.346661 \nL 251.148713 121.516969 \nL 256.5046 123.821983 \nL 261.828082 126.357039 \nL 266.874839 129.006582 \nL 271.526407 131.680965 \nL 276.290957 134.670495 \nL 281.405315 138.179263 \nL 286.106934 141.69595 \nL 290.32742 145.109466 \nL 294.98986 149.189132 \nL 295.950239 150.072663 \nL 295.950239 150.072663 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#p2c843578b0)\" d=\"M -1 40.005114 \nL -0.436152 40.005256 \nL 21.869389 49.582607 \nL 34.917295 53.645571 \nL 44.17493 55.519756 \nL 73.66125 60.229918 \nL 82.104155 62.049235 \nL 88.786011 63.735877 \nL 95.966791 65.774063 \nL 119.066941 72.595349 \nL 124.139457 73.703956 \nL 129.100039 74.531463 \nL 133.896018 75.099955 \nL 139.763143 75.558224 \nL 167.90141 77.39777 \nL 195.363605 79.989578 \nL 208.435935 81.401449 \nL 219.624364 82.836352 \nL 230.863324 84.505307 \nL 244.497726 86.771663 \nL 264.544374 90.112416 \nL 272.773723 91.227471 \nL 279.418356 91.909122 \nL 285.856879 92.343271 \nL 292.264797 92.54572 \nL 295.950239 92.564942 \nL 295.950239 92.564942 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 73.66125 183.876 \nL 73.66125 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 295.953457 183.876 \nL 295.953457 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 73.66125 183.876 \nL 295.953457 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 73.66125 13.2 \nL 295.953457 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 177.028457 65.234375 \nL 288.953457 65.234375 \nQ 290.953457 65.234375 290.953457 63.234375 \nL 290.953457 20.2 \nQ 290.953457 18.2 288.953457 18.2 \nL 177.028457 18.2 \nQ 175.028457 18.2 175.028457 20.2 \nL 175.028457 63.234375 \nQ 175.028457 65.234375 177.028457 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_14\">\n     <path d=\"M 179.028457 26.298438 \nL 199.028457 26.298438 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_15\"/>\n    <g id=\"text_12\">\n     <!-- Infinite Train -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-73\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n      <path d=\"M 37.109375 75.984375 \nL 37.109375 68.5 \nL 28.515625 68.5 \nQ 23.6875 68.5 21.796875 66.546875 \nQ 19.921875 64.59375 19.921875 59.515625 \nL 19.921875 54.6875 \nL 34.71875 54.6875 \nL 34.71875 47.703125 \nL 19.921875 47.703125 \nL 19.921875 0 \nL 10.890625 0 \nL 10.890625 47.703125 \nL 2.296875 47.703125 \nL 2.296875 54.6875 \nL 10.890625 54.6875 \nL 10.890625 58.5 \nQ 10.890625 67.625 15.140625 71.796875 \nQ 19.390625 75.984375 28.609375 75.984375 \nz\n\" id=\"DejaVuSans-102\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path id=\"DejaVuSans-32\"/>\n      <path d=\"M -0.296875 72.90625 \nL 61.375 72.90625 \nL 61.375 64.59375 \nL 35.5 64.59375 \nL 35.5 0 \nL 25.59375 0 \nL 25.59375 64.59375 \nL -0.296875 64.59375 \nz\n\" id=\"DejaVuSans-84\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(207.028457 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"440.40625\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"481.519531\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"542.798828\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"570.582031\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_16\">\n     <path d=\"M 179.028457 40.976562 \nL 199.028457 40.976562 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_17\"/>\n    <g id=\"text_13\">\n     <!-- Infinite Test -->\n     <g style=\"fill:#262626;\" transform=\"translate(207.028457 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"440.359375\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"501.882812\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"553.982422\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n    <g id=\"line2d_18\">\n     <path d=\"M 179.028457 55.654687 \nL 199.028457 55.654687 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n    </g>\n    <g id=\"line2d_19\"/>\n    <g id=\"text_14\">\n     <!-- Finite Ensemble -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 51.703125 72.90625 \nL 51.703125 64.59375 \nL 19.671875 64.59375 \nL 19.671875 43.109375 \nL 48.578125 43.109375 \nL 48.578125 34.8125 \nL 19.671875 34.8125 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-70\"/>\n      <path d=\"M 9.8125 72.90625 \nL 55.90625 72.90625 \nL 55.90625 64.59375 \nL 19.671875 64.59375 \nL 19.671875 43.015625 \nL 54.390625 43.015625 \nL 54.390625 34.71875 \nL 19.671875 34.71875 \nL 19.671875 8.296875 \nL 56.78125 8.296875 \nL 56.78125 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-69\"/>\n      <path d=\"M 52 44.1875 \nQ 55.375 50.25 60.0625 53.125 \nQ 64.75 56 71.09375 56 \nQ 79.640625 56 84.28125 50.015625 \nQ 88.921875 44.046875 88.921875 33.015625 \nL 88.921875 0 \nL 79.890625 0 \nL 79.890625 32.71875 \nQ 79.890625 40.578125 77.09375 44.375 \nQ 74.3125 48.1875 68.609375 48.1875 \nQ 61.625 48.1875 57.5625 43.546875 \nQ 53.515625 38.921875 53.515625 30.90625 \nL 53.515625 0 \nL 44.484375 0 \nL 44.484375 32.71875 \nQ 44.484375 40.625 41.703125 44.40625 \nQ 38.921875 48.1875 33.109375 48.1875 \nQ 26.21875 48.1875 22.15625 43.53125 \nQ 18.109375 38.875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.1875 51.21875 25.484375 53.609375 \nQ 29.78125 56 35.6875 56 \nQ 41.65625 56 45.828125 52.96875 \nQ 50 49.953125 52 44.1875 \nz\n\" id=\"DejaVuSans-109\"/>\n      <path d=\"M 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\nM 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nz\n\" id=\"DejaVuSans-98\"/>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(207.028457 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"57.410156\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"85.193359\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"148.572266\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"176.355469\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"215.564453\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"277.087891\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"308.875\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"372.058594\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"435.4375\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"487.537109\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"549.060547\" xlink:href=\"#DejaVuSans-109\"/>\n      <use x=\"646.472656\" xlink:href=\"#DejaVuSans-98\"/>\n      <use x=\"709.949219\" xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"737.732422\" xlink:href=\"#DejaVuSans-101\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n  <g id=\"axes_2\">\n   <g id=\"patch_8\">\n    <path d=\"M 393.764662 183.876 \nL 616.056869 183.876 \nL 616.056869 13.2 \nL 393.764662 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_3\">\n    <g id=\"xtick_29\">\n     <g id=\"line2d_20\">\n      <path clip-path=\"url(#pb3b09c626c)\" d=\"M 434.152958 183.876 \nL 434.152958 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_15\">\n      <!-- −2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(423.612294 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_30\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#pb3b09c626c)\" d=\"M 504.910765 183.876 \nL 504.910765 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_16\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(500.361578 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_31\">\n     <g id=\"line2d_22\">\n      <path clip-path=\"url(#pb3b09c626c)\" d=\"M 575.668572 183.876 \nL 575.668572 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_17\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(571.119385 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_18\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(498.910765 226.412594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_4\">\n    <g id=\"ytick_46\">\n     <g id=\"line2d_23\">\n      <path clip-path=\"url(#pb3b09c626c)\" d=\"M 393.764662 155.43 \nL 616.056869 155.43 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_19\">\n      <!-- −1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(363.183334 160.862883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_47\">\n     <g id=\"line2d_24\">\n      <path clip-path=\"url(#pb3b09c626c)\" d=\"M 393.764662 98.538 \nL 616.056869 98.538 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_20\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(375.166287 103.970883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_48\">\n     <g id=\"line2d_25\">\n      <path clip-path=\"url(#pb3b09c626c)\" d=\"M 393.764662 41.646 \nL 616.056869 41.646 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_21\">\n      <!-- 1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(375.166287 47.078883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_22\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(355.023959 102.138)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"PolyCollection_1\">\n    <defs>\n     <path d=\"M 393.764659 -145.411183 \nL 393.764659 -140.337372 \nL 398.301231 -133.582567 \nL 402.837812 -126.908904 \nL 407.374385 -120.37696 \nL 411.910958 -114.076294 \nL 416.447539 -108.130424 \nL 420.984112 -102.620805 \nL 425.520693 -97.688904 \nL 430.057265 -92.556829 \nL 434.593842 -87.727392 \nL 439.130419 -83.675348 \nL 443.666992 -80.540302 \nL 448.203569 -78.440307 \nL 452.740146 -77.470704 \nL 457.276718 -77.673372 \nL 461.813295 -79.001257 \nL 466.349872 -81.330507 \nL 470.886445 -84.39667 \nL 475.423026 -87.932088 \nL 479.959599 -91.787337 \nL 484.49618 -96.19928 \nL 489.032752 -102.067016 \nL 493.569325 -111.095025 \nL 498.105906 -125.26477 \nL 502.642479 -124.605242 \nL 507.179051 -124.54703 \nL 511.715633 -130.535414 \nL 516.252205 -139.877203 \nL 520.788778 -149.798352 \nL 525.325359 -160.197772 \nL 529.861932 -170.683809 \nL 534.398513 -180.61465 \nL 538.935086 -189.425248 \nL 543.471658 -196.732407 \nL 548.008231 -202.354249 \nL 552.54482 -204.216316 \nL 557.081393 -205.227724 \nL 561.617966 -205.321682 \nL 566.154538 -204.42401 \nL 570.691111 -202.529761 \nL 575.227701 -199.667239 \nL 579.764273 -195.906454 \nL 584.300846 -191.350405 \nL 588.837419 -186.123811 \nL 593.373991 -180.287415 \nL 597.910564 -173.933634 \nL 602.447154 -167.282076 \nL 606.983726 -160.288094 \nL 611.520299 -153.184046 \nL 616.056872 -146.052783 \nL 616.056872 -146.313376 \nL 616.056872 -146.313376 \nL 611.520299 -153.432512 \nL 606.983726 -160.463786 \nL 602.447154 -167.360646 \nL 597.910564 -174.169351 \nL 593.373991 -180.672338 \nL 588.837419 -186.869206 \nL 584.300846 -192.631874 \nL 579.764273 -197.754143 \nL 575.227701 -202.099162 \nL 570.691111 -205.510312 \nL 566.154538 -207.795274 \nL 561.617966 -208.778835 \nL 557.081393 -208.288994 \nL 552.54482 -206.183749 \nL 548.008231 -202.432825 \nL 543.471658 -200.157263 \nL 538.935086 -197.643793 \nL 534.398513 -195.222653 \nL 529.861932 -193.137321 \nL 525.325359 -191.419131 \nL 520.788778 -189.745137 \nL 516.252205 -187.296876 \nL 511.715633 -182.141019 \nL 507.179051 -168.459451 \nL 502.642479 -145.097723 \nL 498.105906 -125.553462 \nL 493.569325 -126.350487 \nL 489.032752 -124.313221 \nL 484.49618 -119.548702 \nL 479.959599 -113.297627 \nL 475.423026 -106.711387 \nL 470.886445 -100.56279 \nL 466.349872 -95.27981 \nL 461.813295 -91.057598 \nL 457.276718 -87.987833 \nL 452.740146 -86.093924 \nL 448.203569 -85.384911 \nL 443.666992 -85.836762 \nL 439.130419 -87.388225 \nL 434.593842 -89.951126 \nL 430.057265 -93.431768 \nL 425.520693 -98.057441 \nL 420.984112 -104.071724 \nL 416.447539 -110.478392 \nL 411.910958 -117.183604 \nL 407.374385 -124.109735 \nL 402.837812 -131.164852 \nL 398.301231 -138.285726 \nL 393.764659 -145.411183 \nz\n\" id=\"mcad1b7dafb\" style=\"stroke:#0000ff;stroke-opacity:0.2;\"/>\n    </defs>\n    <g clip-path=\"url(#pb3b09c626c)\">\n     <use style=\"fill:#0000ff;fill-opacity:0.2;stroke:#0000ff;stroke-opacity:0.2;\" x=\"0\" xlink:href=\"#mcad1b7dafb\" y=\"237.771969\"/>\n    </g>\n   </g>\n   <g id=\"line2d_26\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"m59bf1ab113\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#pb3b09c626c)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"613.786595\" xlink:href=\"#m59bf1ab113\" y=\"88.292384\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"596.504738\" xlink:href=\"#m59bf1ab113\" y=\"61.345553\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"427.180774\" xlink:href=\"#m59bf1ab113\" y=\"141.72304\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"498.181644\" xlink:href=\"#m59bf1ab113\" y=\"112.235927\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"548.102129\" xlink:href=\"#m59bf1ab113\" y=\"35.346488\"/>\n    </g>\n   </g>\n   <g id=\"line2d_27\">\n    <path clip-path=\"url(#pb3b09c626c)\" d=\"M 393.764659 94.897691 \nL 398.301231 101.837822 \nL 402.837812 108.735091 \nL 407.374385 115.528621 \nL 411.910958 122.14202 \nL 416.447539 128.467561 \nL 420.984112 134.425705 \nL 425.520693 139.898796 \nL 430.057265 144.777671 \nL 434.593842 148.93271 \nL 439.130419 152.240183 \nL 443.666992 154.583437 \nL 448.203569 155.859358 \nL 452.740146 155.989655 \nL 457.276718 154.941367 \nL 461.813295 152.742543 \nL 466.349872 149.46681 \nL 470.886445 145.292239 \nL 475.423026 140.450231 \nL 479.959599 135.229487 \nL 484.49618 129.897978 \nL 489.032752 124.58185 \nL 493.569325 119.049213 \nL 498.105906 112.362853 \nL 502.642479 102.920486 \nL 507.179051 91.268729 \nL 511.715633 81.433752 \nL 516.252205 74.184929 \nL 520.788778 68.000224 \nL 525.325359 61.963517 \nL 529.861932 55.861404 \nL 534.398513 49.853317 \nL 538.935086 44.23745 \nL 543.471658 39.327134 \nL 548.008231 35.378432 \nL 552.54482 32.571936 \nL 557.081393 31.01361 \nL 561.617966 30.72171 \nL 566.154538 31.662327 \nL 570.691111 33.751932 \nL 575.227701 36.888768 \nL 579.764273 40.94167 \nL 584.300846 45.780829 \nL 588.837419 51.27546 \nL 593.373991 57.292092 \nL 597.910564 63.720476 \nL 602.447154 70.450608 \nL 606.983726 77.396029 \nL 611.520299 84.46369 \nL 616.056872 91.58889 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_28\">\n    <path clip-path=\"url(#pb3b09c626c)\" d=\"M 393.764659 102.182969 \nL 398.301231 107.909929 \nL 402.837812 113.676159 \nL 407.374385 119.416407 \nL 411.910958 125.052661 \nL 416.447539 130.494303 \nL 420.984112 135.639943 \nL 425.520693 140.393896 \nL 430.057265 145.283866 \nL 434.593842 150.015024 \nL 439.130419 154.047949 \nL 443.666992 157.243067 \nL 448.203569 159.478148 \nL 452.740146 160.656903 \nL 457.276718 160.728602 \nL 461.813295 159.710948 \nL 466.349872 157.704464 \nL 470.886445 154.875842 \nL 475.423026 151.387624 \nL 479.959599 147.280764 \nL 484.49618 142.33137 \nL 489.032752 135.856338 \nL 493.569325 126.482114 \nL 498.105906 112.441767 \nL 502.642479 112.516457 \nL 507.179051 112.500851 \nL 511.715633 106.576541 \nL 516.252205 96.779235 \nL 520.788778 86.151569 \nL 525.325359 75.38173 \nL 529.861932 64.9767 \nL 534.398513 55.420052 \nL 538.935086 47.11552 \nL 543.471658 40.347813 \nL 548.008231 35.296199 \nL 552.54482 33.861096 \nL 557.081393 33.477368 \nL 561.617966 34.016324 \nL 566.154538 35.497694 \nL 570.691111 37.885439 \nL 575.227701 41.102839 \nL 579.764273 45.046868 \nL 584.300846 49.600587 \nL 588.837419 54.642869 \nL 593.373991 60.055769 \nL 597.910564 65.729904 \nL 602.447154 71.569891 \nL 606.983726 77.583938 \nL 611.520299 84.284098 \nL 616.056872 91.054621 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_29\">\n    <path clip-path=\"url(#pb3b09c626c)\" d=\"M 393.764659 95.320883 \nL 398.301231 101.917347 \nL 402.837812 108.544093 \nL 407.374385 115.148941 \nL 411.910958 121.669622 \nL 416.447539 128.03218 \nL 420.984112 134.148732 \nL 425.520693 139.902075 \nL 430.057265 144.530563 \nL 434.593842 148.079521 \nL 439.130419 150.831594 \nL 443.666992 152.674705 \nL 448.203569 153.50139 \nL 452.740146 153.221763 \nL 457.276718 151.769844 \nL 461.813295 149.108857 \nL 466.349872 145.242482 \nL 470.886445 140.247404 \nL 475.423026 134.336772 \nL 479.959599 127.930931 \nL 484.49618 121.68175 \nL 489.032752 116.43136 \nL 493.569325 113.096644 \nL 498.105906 112.159718 \nL 502.642479 92.991836 \nL 507.179051 70.872964 \nL 511.715633 57.903481 \nL 516.252205 52.279319 \nL 520.788778 49.068552 \nL 525.325359 46.626707 \nL 529.861932 44.258024 \nL 534.398513 41.774706 \nL 538.935086 39.266258 \nL 543.471658 36.946151 \nL 548.008231 35.032513 \nL 552.54482 31.877467 \nL 557.081393 30.149318 \nL 561.617966 29.94861 \nL 566.154538 31.12472 \nL 570.691111 33.51043 \nL 575.227701 36.934683 \nL 579.764273 41.230843 \nL 584.300846 46.242107 \nL 588.837419 51.824477 \nL 593.373991 57.848109 \nL 597.910564 64.197228 \nL 602.447154 70.767577 \nL 606.983726 77.377529 \nL 611.520299 83.367472 \nL 616.056872 89.22995 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_30\">\n    <path clip-path=\"url(#pb3b09c626c)\" d=\"M 393.764659 98.751926 \nL 398.301231 104.913638 \nL 402.837812 111.110126 \nL 407.374385 117.282674 \nL 411.910958 123.361142 \nL 416.447539 129.263242 \nL 420.984112 134.894338 \nL 425.520693 140.147986 \nL 430.057265 144.907215 \nL 434.593842 149.047272 \nL 439.130419 152.439772 \nL 443.666992 154.958888 \nL 448.203569 156.489771 \nL 452.740146 156.939333 \nL 457.276718 156.249225 \nL 461.813295 154.409901 \nL 466.349872 151.473471 \nL 470.886445 147.561623 \nL 475.423026 142.862198 \nL 479.959599 137.605848 \nL 484.49618 132.006559 \nL 489.032752 126.143849 \nL 493.569325 119.789379 \nL 498.105906 112.300743 \nL 502.642479 102.754147 \nL 507.179051 91.686907 \nL 511.715633 82.24001 \nL 516.252205 74.529276 \nL 520.788778 67.61006 \nL 525.325359 61.004219 \nL 529.861932 54.617362 \nL 534.398513 48.597379 \nL 538.935086 43.190888 \nL 543.471658 38.646982 \nL 548.008231 35.164356 \nL 552.54482 32.869282 \nL 557.081393 31.813343 \nL 561.617966 31.982467 \nL 566.154538 33.311207 \nL 570.691111 35.697934 \nL 575.227701 39.018761 \nL 579.764273 43.138856 \nL 584.300846 47.921347 \nL 588.837419 53.233673 \nL 593.373991 58.951939 \nL 597.910564 64.963566 \nL 602.447154 71.168734 \nL 606.983726 77.480733 \nL 611.520299 83.825785 \nL 616.056872 90.142286 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"patch_9\">\n    <path d=\"M 393.764662 183.876 \nL 393.764662 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_10\">\n    <path d=\"M 616.056869 183.876 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_11\">\n    <path d=\"M 393.764662 183.876 \nL 616.056869 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_12\">\n    <path d=\"M 393.764662 13.2 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_2\">\n    <g id=\"patch_13\">\n     <path d=\"M 400.764662 65.234375 \nL 512.689662 65.234375 \nQ 514.689662 65.234375 514.689662 63.234375 \nL 514.689662 20.2 \nQ 514.689662 18.2 512.689662 18.2 \nL 400.764662 18.2 \nQ 398.764662 18.2 398.764662 20.2 \nL 398.764662 63.234375 \nQ 398.764662 65.234375 400.764662 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_31\"/>\n    <g id=\"line2d_32\">\n     <g>\n      <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"412.764662\" xlink:href=\"#m59bf1ab113\" y=\"26.298438\"/>\n     </g>\n    </g>\n    <g id=\"text_23\">\n     <!-- Train -->\n     <g style=\"fill:#262626;\" transform=\"translate(430.764662 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"60.865234\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"101.978516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"163.257812\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"191.041016\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_33\">\n     <path d=\"M 402.764662 40.976562 \nL 422.764662 40.976562 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_34\"/>\n    <g id=\"text_24\">\n     <!-- Infinite Network -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 23.09375 72.90625 \nL 55.421875 11.921875 \nL 55.421875 72.90625 \nL 64.984375 72.90625 \nL 64.984375 0 \nL 51.703125 0 \nL 19.390625 60.984375 \nL 19.390625 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-78\"/>\n      <path d=\"M 4.203125 54.6875 \nL 13.1875 54.6875 \nL 24.421875 12.015625 \nL 35.59375 54.6875 \nL 46.1875 54.6875 \nL 57.421875 12.015625 \nL 68.609375 54.6875 \nL 77.59375 54.6875 \nL 63.28125 0 \nL 52.6875 0 \nL 40.921875 44.828125 \nL 29.109375 0 \nL 18.5 0 \nz\n\" id=\"DejaVuSans-119\"/>\n      <path d=\"M 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 31.109375 \nL 44.921875 54.6875 \nL 56.390625 54.6875 \nL 27.390625 29.109375 \nL 57.625 0 \nL 45.90625 0 \nL 18.109375 26.703125 \nL 18.109375 0 \nL 9.078125 0 \nz\n\" id=\"DejaVuSans-107\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(430.764662 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-78\"/>\n      <use x=\"454.345703\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"515.869141\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"555.078125\" xlink:href=\"#DejaVuSans-119\"/>\n      <use x=\"636.865234\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"698.046875\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"739.160156\" xlink:href=\"#DejaVuSans-107\"/>\n     </g>\n    </g>\n    <g id=\"line2d_35\">\n     <path d=\"M 402.764662 55.654687 \nL 422.764662 55.654687 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:round;stroke-width:2;\"/>\n    </g>\n    <g id=\"line2d_36\"/>\n    <g id=\"text_25\">\n     <!-- Finite Ensemble -->\n     <g style=\"fill:#262626;\" transform=\"translate(430.764662 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"57.410156\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"85.193359\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"148.572266\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"176.355469\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"215.564453\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"277.087891\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"308.875\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"372.058594\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"435.4375\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"487.537109\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"549.060547\" xlink:href=\"#DejaVuSans-109\"/>\n      <use x=\"646.472656\" xlink:href=\"#DejaVuSans-98\"/>\n      <use x=\"709.949219\" xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"737.732422\" xlink:href=\"#DejaVuSans-101\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p2c843578b0\">\n   <rect height=\"170.676\" width=\"222.292207\" x=\"73.66125\" y=\"13.2\"/>\n  </clipPath>\n  <clipPath id=\"pb3b09c626c\">\n   <rect height=\"170.676\" width=\"222.292207\" x=\"393.764662\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 900x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# test {\"skip\": true}\n",
    "plt.subplot(1, 2, 1)\n",
    "\n",
    "mean_train_loss = jnp.mean(train_loss, axis=0)\n",
    "mean_test_loss = jnp.mean(test_loss, axis=0)\n",
    "\n",
    "plt.loglog(ts, ntk_train_loss_mean, linewidth=3)\n",
    "plt.loglog(ts, ntk_test_loss_mean, linewidth=3)\n",
    "\n",
    "plt.loglog(ts, mean_train_loss, 'k-', linewidth=2)\n",
    "plt.loglog(ts, mean_test_loss, 'k-', linewidth=2)\n",
    "\n",
    "plt.xlim([10 ** 0, 10 ** 3])\n",
    "\n",
    "plt.xscale('log')\n",
    "plt.yscale('log')\n",
    "format_plot('Step', 'Loss')\n",
    "\n",
    "legend(['Infinite Train', 'Infinite Test', 'Finite Ensemble'])\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "\n",
    "plot_fn(train, None)\n",
    "\n",
    "plt.plot(test_xs, ntk_mean, 'b-', linewidth=3)\n",
    "plt.fill_between(\n",
    "    jnp.reshape(test_xs, (-1)),\n",
    "    ntk_mean - 2 * ntk_std,\n",
    "    ntk_mean +  2 * ntk_std,\n",
    "    color='blue', alpha=0.2)\n",
    "\n",
    "ensemble_fx = vmap(apply_fn, (0, None))(params, test_xs)\n",
    "\n",
    "mean_fx = jnp.reshape(jnp.mean(ensemble_fx, axis=0), (-1,))\n",
    "std_fx = jnp.reshape(jnp.std(ensemble_fx, axis=0), (-1,))\n",
    "\n",
    "plt.plot(test_xs, mean_fx - 2 * std_fx, 'k--', label='_nolegend_')\n",
    "plt.plot(test_xs, mean_fx + 2 * std_fx, 'k--', label='_nolegend_')\n",
    "plt.plot(test_xs, mean_fx, linewidth=2, color='black')\n",
    "\n",
    "legend(['Train', 'Infinite Network', 'Finite Ensemble'], loc='upper left')\n",
    "\n",
    "plt.xlim([-jnp.pi, jnp.pi])\n",
    "plt.ylim([-1.5, 1.5])\n",
    "\n",
    "format_plot('$x$', '$f$')\n",
    "finalize_plot((1.5, 0.6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IvqFKuxY5pWx"
   },
   "source": [
    "We see pretty nice agreement between exact inference of the infinite-width networks and the result of training an ensemble! Note that we do see some deviations in the training loss at the end of training. This is ameliorated by using a wider network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c39JDz2tD2di"
   },
   "source": [
    "## Playing Around with the Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O1i0fwz3EfR8"
   },
   "source": [
    "To demonstrate the ease of specifying more exotic architecture, can try to reproduce the above results with different choices of architecture. For fn, let's see what happens if we add residual connections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "txOpM97RE_EE"
   },
   "outputs": [],
   "source": [
    "ResBlock = stax.serial(\n",
    "    stax.FanOut(2),\n",
    "    stax.parallel(\n",
    "        stax.serial(\n",
    "            stax.Erf(),\n",
    "            stax.Dense(512, W_std=1.1, b_std=0),\n",
    "        ),\n",
    "        stax.Identity()\n",
    "    ),\n",
    "    stax.FanInSum()\n",
    ")\n",
    "\n",
    "init_fn, apply_fn, kernel_fn = stax.serial(\n",
    "    stax.Dense(512, W_std=1, b_std=0),\n",
    "    ResBlock, ResBlock, stax.Erf(),\n",
    "    stax.Dense(1, W_std=1.5, b_std=0)\n",
    ")\n",
    "\n",
    "apply_fn = jit(apply_fn)\n",
    "kernel_fn = jit(kernel_fn, static_argnames='get')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eli8h43jGc2n"
   },
   "source": [
    "Given this new architecture, let's train a new ensemble of models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7xz1cWyCE39Z"
   },
   "outputs": [],
   "source": [
    "# test {\"skip\": true}\n",
    "ensemble_size = 100\n",
    "learning_rate = 0.1\n",
    "ts = jnp.arange(0, 10 ** 3, learning_rate)\n",
    "\n",
    "opt_init, opt_update, get_params = optimizers.sgd(learning_rate)\n",
    "opt_update = jit(opt_update)\n",
    "\n",
    "key, = random.split(key, 1)\n",
    "ensemble_key = random.split(key, ensemble_size)\n",
    "params, train_loss, test_loss = vmap(train_network)(ensemble_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o9ttNWWSGgBV"
   },
   "source": [
    "Finally, let's repeat our NTK-GP inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JEMTCSGpFcDL"
   },
   "outputs": [],
   "source": [
    "# test {\"skip\": true}\n",
    "predict_fn = nt.predict.gradient_descent_mse_ensemble(kernel_fn, train_xs,\n",
    "                                                      train_ys, diag_reg=1e-4)\n",
    "ntk_mean, ntk_var = predict_fn(x_test=test_xs, get='ntk', compute_cov=True)\n",
    "\n",
    "ntk_mean = jnp.reshape(ntk_mean, (-1,))\n",
    "ntk_std = jnp.sqrt(jnp.diag(ntk_var))\n",
    "\n",
    "ntk_train_loss_mean = loss_fn(predict_fn, train_ys, ts)\n",
    "ntk_test_loss_mean = loss_fn(predict_fn, test_ys, ts, test_xs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4uLi8wdcGjaU"
   },
   "source": [
    "Now let's draw the result!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 338
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1620,
     "status": "ok",
     "timestamp": 1588653810888,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "nlOJ4QuK0-Ss",
    "outputId": "d828770f-c37b-4371-e742-8473f7e6a489"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"237.771969pt\" version=\"1.1\" viewBox=\"0 0 629.256869 237.771969\" width=\"629.256869pt\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 237.771969 \nL 629.256869 237.771969 \nL 629.256869 0 \nL 0 0 \nz\n\" style=\"fill:#ffffff;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 73.66125 183.876 \nL 295.953457 183.876 \nL 295.953457 13.2 \nL 73.66125 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p0ccaef7514)\" d=\"M 73.66125 183.876 \nL 73.66125 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_1\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(61.07725 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <path clip-path=\"url(#p0ccaef7514)\" d=\"M 147.758652 183.876 \nL 147.758652 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_2\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(135.174652 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p0ccaef7514)\" d=\"M 221.856054 183.876 \nL 221.856054 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_3\">\n      <!-- $\\mathdefault{10^{2}}$ -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(209.272054 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <path clip-path=\"url(#p0ccaef7514)\" d=\"M 295.953457 183.876 \nL 295.953457 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_4\">\n      <!-- $\\mathdefault{10^{3}}$ -->\n      <defs>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(283.369457 204.241766)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\"/>\n    <g id=\"xtick_6\"/>\n    <g id=\"xtick_7\"/>\n    <g id=\"xtick_8\"/>\n    <g id=\"xtick_9\"/>\n    <g id=\"xtick_10\"/>\n    <g id=\"xtick_11\"/>\n    <g id=\"xtick_12\"/>\n    <g id=\"xtick_13\"/>\n    <g id=\"xtick_14\"/>\n    <g id=\"xtick_15\"/>\n    <g id=\"xtick_16\"/>\n    <g id=\"xtick_17\"/>\n    <g id=\"xtick_18\"/>\n    <g id=\"xtick_19\"/>\n    <g id=\"xtick_20\"/>\n    <g id=\"xtick_21\"/>\n    <g id=\"xtick_22\"/>\n    <g id=\"xtick_23\"/>\n    <g id=\"xtick_24\"/>\n    <g id=\"xtick_25\"/>\n    <g id=\"xtick_26\"/>\n    <g id=\"xtick_27\"/>\n    <g id=\"xtick_28\"/>\n    <g id=\"text_5\">\n     <!-- Step -->\n     <defs>\n      <path d=\"M 53.515625 70.515625 \nL 53.515625 60.890625 \nQ 47.90625 63.578125 42.921875 64.890625 \nQ 37.9375 66.21875 33.296875 66.21875 \nQ 25.25 66.21875 20.875 63.09375 \nQ 16.5 59.96875 16.5 54.203125 \nQ 16.5 49.359375 19.40625 46.890625 \nQ 22.3125 44.4375 30.421875 42.921875 \nL 36.375 41.703125 \nQ 47.40625 39.59375 52.65625 34.296875 \nQ 57.90625 29 57.90625 20.125 \nQ 57.90625 9.515625 50.796875 4.046875 \nQ 43.703125 -1.421875 29.984375 -1.421875 \nQ 24.8125 -1.421875 18.96875 -0.25 \nQ 13.140625 0.921875 6.890625 3.21875 \nL 6.890625 13.375 \nQ 12.890625 10.015625 18.65625 8.296875 \nQ 24.421875 6.59375 29.984375 6.59375 \nQ 38.421875 6.59375 43.015625 9.90625 \nQ 47.609375 13.234375 47.609375 19.390625 \nQ 47.609375 24.75 44.3125 27.78125 \nQ 41.015625 30.8125 33.5 32.328125 \nL 27.484375 33.5 \nQ 16.453125 35.6875 11.515625 40.375 \nQ 6.59375 45.0625 6.59375 53.421875 \nQ 6.59375 63.09375 13.40625 68.65625 \nQ 20.21875 74.21875 32.171875 74.21875 \nQ 37.3125 74.21875 42.625 73.28125 \nQ 47.953125 72.359375 53.515625 70.515625 \nz\n\" id=\"DejaVuSans-83\"/>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(162.037041 226.412594)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-83\"/>\n      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"102.685547\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"164.208984\" xlink:href=\"#DejaVuSans-112\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p0ccaef7514)\" d=\"M 73.66125 156.325368 \nL 295.953457 156.325368 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_6\">\n      <!-- $\\mathdefault{10^{-4}}$ -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 161.75825)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-52\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_6\">\n      <path clip-path=\"url(#p0ccaef7514)\" d=\"M 73.66125 122.311744 \nL 295.953457 122.311744 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_7\">\n      <!-- $\\mathdefault{10^{-3}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 127.744627)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p0ccaef7514)\" d=\"M 73.66125 88.29812 \nL 295.953457 88.29812 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-2}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 93.731003)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_8\">\n      <path clip-path=\"url(#p0ccaef7514)\" d=\"M 73.66125 54.284496 \nL 295.953457 54.284496 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(30.55625 59.717379)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p0ccaef7514)\" d=\"M 73.66125 20.270872 \nL 295.953457 20.270872 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g style=\"fill:#262626;\" transform=\"translate(38.99325 25.703755)scale(0.143 -0.143)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\"/>\n    <g id=\"ytick_7\"/>\n    <g id=\"ytick_8\"/>\n    <g id=\"ytick_9\"/>\n    <g id=\"ytick_10\"/>\n    <g id=\"ytick_11\"/>\n    <g id=\"ytick_12\"/>\n    <g id=\"ytick_13\"/>\n    <g id=\"ytick_14\"/>\n    <g id=\"ytick_15\"/>\n    <g id=\"ytick_16\"/>\n    <g id=\"ytick_17\"/>\n    <g id=\"ytick_18\"/>\n    <g id=\"ytick_19\"/>\n    <g id=\"ytick_20\"/>\n    <g id=\"ytick_21\"/>\n    <g id=\"ytick_22\"/>\n    <g id=\"ytick_23\"/>\n    <g id=\"ytick_24\"/>\n    <g id=\"ytick_25\"/>\n    <g id=\"ytick_26\"/>\n    <g id=\"ytick_27\"/>\n    <g id=\"ytick_28\"/>\n    <g id=\"ytick_29\"/>\n    <g id=\"ytick_30\"/>\n    <g id=\"ytick_31\"/>\n    <g id=\"ytick_32\"/>\n    <g id=\"ytick_33\"/>\n    <g id=\"ytick_34\"/>\n    <g id=\"ytick_35\"/>\n    <g id=\"ytick_36\"/>\n    <g id=\"ytick_37\"/>\n    <g id=\"ytick_38\"/>\n    <g id=\"ytick_39\"/>\n    <g id=\"ytick_40\"/>\n    <g id=\"ytick_41\"/>\n    <g id=\"ytick_42\"/>\n    <g id=\"ytick_43\"/>\n    <g id=\"ytick_44\"/>\n    <g id=\"ytick_45\"/>\n    <g id=\"text_11\">\n     <!-- Loss -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 8.296875 \nL 55.171875 8.296875 \nL 55.171875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-76\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(22.396875 120.645812)rotate(-90)scale(0.2 -0.2)\">\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use x=\"55.697266\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"116.878906\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"168.978516\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_10\">\n    <path clip-path=\"url(#p0ccaef7514)\" d=\"M -1 34.404739 \nL -0.436152 34.404841 \nL 21.869389 44.777933 \nL 34.917295 51.496778 \nL 44.17493 55.388891 \nL 51.355709 57.755848 \nL 66.48047 62.183843 \nL 73.66125 64.63686 \nL 79.528376 66.955825 \nL 84.488957 69.155141 \nL 90.736917 72.22696 \nL 97.536861 75.894748 \nL 107.923742 81.848459 \nL 117.457603 87.236519 \nL 127.328212 92.524713 \nL 141.76255 100.263981 \nL 151.405579 105.405469 \nL 156.201559 107.710138 \nL 160.156432 109.382201 \nL 164.068091 110.794165 \nL 167.90141 111.938395 \nL 171.78714 112.878213 \nL 176.594848 113.792284 \nL 181.798441 114.565582 \nL 192.127474 115.791966 \nL 203.426483 117.137228 \nL 214.715474 118.880679 \nL 222.587814 120.489711 \nL 233.72411 123.538398 \nL 240.465717 125.950195 \nL 245.7623 128.218559 \nL 254.677488 132.937256 \nL 255.817532 133.635659 \nL 261.180422 137.236785 \nL 261.679059 137.601195 \nL 262.453853 138.178275 \nL 267.049127 141.852419 \nL 268.006546 142.678972 \nL 268.266249 142.904997 \nL 269.159051 143.697996 \nL 271.201666 145.586272 \nL 271.615652 145.982985 \nL 272.414611 146.7579 \nL 272.694265 147.031286 \nL 273.024049 147.358362 \nL 273.252904 147.58355 \nL 273.622162 147.956387 \nL 280.497472 155.482835 \nL 280.704886 155.734306 \nL 280.78748 155.825154 \nL 281.054452 156.150937 \nL 281.12595 156.232137 \nL 281.470993 156.649301 \nL 281.54661 156.729958 \nL 281.737384 156.968455 \nL 282.293166 157.640129 \nL 282.533336 157.941108 \nL 282.606501 158.028787 \nL 283.037222 158.5673 \nL 283.109251 158.651572 \nL 283.391009 159.0043 \nL 283.457507 159.084939 \nL 283.717426 159.417941 \nL 284.46183 160.354293 \nL 284.626973 160.580754 \nL 284.63155 160.555159 \nL 284.718355 160.683875 \nL 284.904882 160.925325 \nL 284.909417 160.916237 \nL 284.972854 161.012814 \nL 284.977379 161.010924 \nL 285.171412 161.266435 \nL 285.328487 161.470362 \nL 285.542664 161.748626 \nL 285.547112 161.731396 \nL 285.622617 161.8501 \nL 285.627051 161.845383 \nL 285.887693 162.19828 \nL 285.892091 162.192221 \nL 285.944839 162.272417 \nL 285.958012 162.276121 \nL 286.115671 162.503526 \nL 286.124407 162.493245 \nL 286.185496 162.583333 \nL 286.381068 162.847694 \nL 286.385399 162.838883 \nL 286.450318 162.944025 \nL 286.458964 162.939202 \nL 286.644291 163.195869 \nL 286.64859 163.190787 \nL 286.70869 163.276432 \nL 286.71298 163.277138 \nL 286.896854 163.535313 \nL 287.054236 163.719258 \nL 287.290978 164.049076 \nL 287.366706 164.141635 \nL 287.713607 164.628004 \nL 287.717766 164.602153 \nL 287.804934 164.736037 \nL 288.097915 165.132231 \nL 288.102022 165.117454 \nL 288.175867 165.236884 \nL 288.179966 165.229643 \nL 288.506091 165.68909 \nL 288.510145 165.675715 \nL 288.849015 166.143377 \nL 288.989163 166.341085 \nL 288.993158 166.335449 \nL 288.997151 166.327324 \nL 289.061002 166.436575 \nL 289.243866 166.694236 \nL 289.247831 166.689574 \nL 289.251793 166.679477 \nL 289.311185 166.784781 \nL 289.330957 166.776293 \nL 289.465092 167.000842 \nL 289.528019 167.055355 \nL 289.524091 167.08189 \nL 289.539804 167.073344 \nL 289.696532 167.299712 \nL 292.416013 171.092045 \nL 292.419604 171.082927 \nL 292.548639 171.292426 \nL 292.552218 171.291517 \nL 292.559371 171.267589 \nL 292.620103 171.380756 \nL 292.623673 171.379077 \nL 292.890228 171.770885 \nL 292.893766 171.766814 \nL 292.897307 171.755706 \nL 292.971531 171.871113 \nL 293.11244 172.090261 \nL 292.98212 171.86733 \nL 293.11947 172.079707 \nL 293.18617 172.179483 \nL 293.193184 172.16613 \nL 293.423768 172.525373 \nL 293.427248 172.505777 \nL 293.51068 172.657282 \nL 293.593894 172.744151 \nL 293.728663 172.946031 \nL 293.893759 173.187194 \nL 293.897189 173.171349 \nL 293.972569 173.304318 \nL 293.979413 173.278634 \nL 294.105764 173.490643 \nL 294.129613 173.485895 \nL 294.177256 173.600281 \nL 294.184057 173.585933 \nL 294.793628 174.471079 \nL 294.796966 174.443474 \nL 294.886906 174.595661 \nL 294.890231 174.585239 \nL 295.418056 175.348894 \nL 295.4246 175.340717 \nL 295.431142 175.336531 \nL 295.551928 175.553974 \nL 295.561703 175.527905 \nL 295.643036 175.667067 \nL 295.646284 175.649142 \nL 295.795386 175.884917 \nL 295.801853 175.869618 \nL 295.863226 175.988704 \nL 295.86968 175.979749 \nL 295.943802 176.118 \nL 295.950239 176.100359 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_11\">\n    <path clip-path=\"url(#p0ccaef7514)\" d=\"M -1 33.520389 \nL -0.436152 33.520485 \nL 21.869389 43.156682 \nL 34.917295 49.596191 \nL 44.17493 53.613718 \nL 51.355709 56.247967 \nL 70.270738 62.641161 \nL 76.728341 65.159203 \nL 84.488957 68.544322 \nL 104.409696 77.659207 \nL 109.014696 79.35018 \nL 113.042458 80.5923 \nL 117.457603 81.716237 \nL 122.769883 82.825663 \nL 159.26865 89.673985 \nL 165.397249 90.459846 \nL 171.939293 91.067429 \nL 179.482327 91.532 \nL 188.347334 91.841647 \nL 198.035102 91.955184 \nL 208.872447 91.843074 \nL 218.927043 91.519104 \nL 228.570072 90.983738 \nL 237.101266 90.288489 \nL 245.43919 89.375697 \nL 254.080136 88.179539 \nL 263.853788 86.576274 \nL 288.518256 82.528738 \nL 295.950239 81.682706 \nL 295.950239 81.682706 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#p0ccaef7514)\" d=\"M -1 39.095045 \nL -0.436152 39.095174 \nL 21.869389 48.483287 \nL 34.917295 53.055235 \nL 44.17493 55.553251 \nL 57.222835 58.818943 \nL 62.183417 60.219256 \nL 66.48047 61.566314 \nL 73.66125 64.15325 \nL 79.528376 66.613232 \nL 84.488957 68.945187 \nL 90.736917 72.195783 \nL 97.536861 76.063161 \nL 119.842402 89.131807 \nL 127.328212 93.059502 \nL 142.147943 100.489156 \nL 151.116968 104.824115 \nL 156.448152 107.158806 \nL 161.02038 108.917991 \nL 165.2107 110.297193 \nL 169.577836 111.495066 \nL 174.139244 112.513331 \nL 179.482327 113.46979 \nL 186.373631 114.461259 \nL 213.860545 118.178243 \nL 221.271535 119.538858 \nL 228.122937 121.019945 \nL 234.602203 122.641856 \nL 240.878174 124.439375 \nL 246.757174 126.342651 \nL 252.592121 128.454816 \nL 258.056393 130.650181 \nL 263.599745 133.112944 \nL 269.416994 135.97285 \nL 275.021242 139.020691 \nL 280.104931 142.069733 \nL 284.337413 144.842179 \nL 288.599242 147.870454 \nL 293.196691 151.426594 \nL 295.950239 153.712196 \nL 295.950239 153.712196 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#p0ccaef7514)\" d=\"M -1 37.302795 \nL -0.436152 37.302915 \nL 21.869389 46.256948 \nL 34.917295 51.080076 \nL 44.17493 53.96997 \nL 57.222835 57.802183 \nL 62.183417 59.383654 \nL 70.270738 62.290869 \nL 76.728341 64.966713 \nL 82.104155 67.434213 \nL 90.736917 71.717927 \nL 99.033882 75.813424 \nL 103.147571 77.641388 \nL 106.794498 79.067715 \nL 111.091552 80.470801 \nL 114.881819 81.45737 \nL 119.066941 82.312939 \nL 124.802987 83.220491 \nL 143.644963 85.985909 \nL 158.815402 88.372106 \nL 166.132828 89.256265 \nL 173.567137 89.922077 \nL 181.686512 90.420615 \nL 190.634316 90.743894 \nL 199.997911 90.857551 \nL 209.539907 90.750609 \nL 219.416747 90.41512 \nL 229.36978 89.854971 \nL 239.4388 89.061963 \nL 248.97935 88.090053 \nL 258.810204 86.86426 \nL 270.669519 85.147594 \nL 286.359396 82.644133 \nL 295.950239 81.123223 \nL 295.950239 81.123223 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 73.66125 183.876 \nL 73.66125 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 295.953457 183.876 \nL 295.953457 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 73.66125 183.876 \nL 295.953457 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 73.66125 13.2 \nL 295.953457 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 177.028457 65.234375 \nL 288.953457 65.234375 \nQ 290.953457 65.234375 290.953457 63.234375 \nL 290.953457 20.2 \nQ 290.953457 18.2 288.953457 18.2 \nL 177.028457 18.2 \nQ 175.028457 18.2 175.028457 20.2 \nL 175.028457 63.234375 \nQ 175.028457 65.234375 177.028457 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_14\">\n     <path d=\"M 179.028457 26.298438 \nL 199.028457 26.298438 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_15\"/>\n    <g id=\"text_12\">\n     <!-- Infinite Train -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 19.671875 72.90625 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-73\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n      <path d=\"M 37.109375 75.984375 \nL 37.109375 68.5 \nL 28.515625 68.5 \nQ 23.6875 68.5 21.796875 66.546875 \nQ 19.921875 64.59375 19.921875 59.515625 \nL 19.921875 54.6875 \nL 34.71875 54.6875 \nL 34.71875 47.703125 \nL 19.921875 47.703125 \nL 19.921875 0 \nL 10.890625 0 \nL 10.890625 47.703125 \nL 2.296875 47.703125 \nL 2.296875 54.6875 \nL 10.890625 54.6875 \nL 10.890625 58.5 \nQ 10.890625 67.625 15.140625 71.796875 \nQ 19.390625 75.984375 28.609375 75.984375 \nz\n\" id=\"DejaVuSans-102\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path id=\"DejaVuSans-32\"/>\n      <path d=\"M -0.296875 72.90625 \nL 61.375 72.90625 \nL 61.375 64.59375 \nL 35.5 64.59375 \nL 35.5 0 \nL 25.59375 0 \nL 25.59375 64.59375 \nL -0.296875 64.59375 \nz\n\" id=\"DejaVuSans-84\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(207.028457 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"440.40625\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"481.519531\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"542.798828\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"570.582031\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_16\">\n     <path d=\"M 179.028457 40.976562 \nL 199.028457 40.976562 \n\" style=\"fill:none;stroke:#dd8452;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_17\"/>\n    <g id=\"text_13\">\n     <!-- Infinite Test -->\n     <g style=\"fill:#262626;\" transform=\"translate(207.028457 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"440.359375\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"501.882812\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"553.982422\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n    <g id=\"line2d_18\">\n     <path d=\"M 179.028457 55.654687 \nL 199.028457 55.654687 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-linecap:round;stroke-width:2;\"/>\n    </g>\n    <g id=\"line2d_19\"/>\n    <g id=\"text_14\">\n     <!-- Finite Ensemble -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 51.703125 72.90625 \nL 51.703125 64.59375 \nL 19.671875 64.59375 \nL 19.671875 43.109375 \nL 48.578125 43.109375 \nL 48.578125 34.8125 \nL 19.671875 34.8125 \nL 19.671875 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-70\"/>\n      <path d=\"M 9.8125 72.90625 \nL 55.90625 72.90625 \nL 55.90625 64.59375 \nL 19.671875 64.59375 \nL 19.671875 43.015625 \nL 54.390625 43.015625 \nL 54.390625 34.71875 \nL 19.671875 34.71875 \nL 19.671875 8.296875 \nL 56.78125 8.296875 \nL 56.78125 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-69\"/>\n      <path d=\"M 52 44.1875 \nQ 55.375 50.25 60.0625 53.125 \nQ 64.75 56 71.09375 56 \nQ 79.640625 56 84.28125 50.015625 \nQ 88.921875 44.046875 88.921875 33.015625 \nL 88.921875 0 \nL 79.890625 0 \nL 79.890625 32.71875 \nQ 79.890625 40.578125 77.09375 44.375 \nQ 74.3125 48.1875 68.609375 48.1875 \nQ 61.625 48.1875 57.5625 43.546875 \nQ 53.515625 38.921875 53.515625 30.90625 \nL 53.515625 0 \nL 44.484375 0 \nL 44.484375 32.71875 \nQ 44.484375 40.625 41.703125 44.40625 \nQ 38.921875 48.1875 33.109375 48.1875 \nQ 26.21875 48.1875 22.15625 43.53125 \nQ 18.109375 38.875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.1875 51.21875 25.484375 53.609375 \nQ 29.78125 56 35.6875 56 \nQ 41.65625 56 45.828125 52.96875 \nQ 50 49.953125 52 44.1875 \nz\n\" id=\"DejaVuSans-109\"/>\n      <path d=\"M 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\nM 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nz\n\" id=\"DejaVuSans-98\"/>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(207.028457 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"57.410156\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"85.193359\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"148.572266\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"176.355469\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"215.564453\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"277.087891\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"308.875\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"372.058594\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"435.4375\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"487.537109\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"549.060547\" xlink:href=\"#DejaVuSans-109\"/>\n      <use x=\"646.472656\" xlink:href=\"#DejaVuSans-98\"/>\n      <use x=\"709.949219\" xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"737.732422\" xlink:href=\"#DejaVuSans-101\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n  <g id=\"axes_2\">\n   <g id=\"patch_8\">\n    <path d=\"M 393.764662 183.876 \nL 616.056869 183.876 \nL 616.056869 13.2 \nL 393.764662 13.2 \nz\n\" style=\"fill:#f2f2f2;\"/>\n   </g>\n   <g id=\"matplotlib.axis_3\">\n    <g id=\"xtick_29\">\n     <g id=\"line2d_20\">\n      <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 434.152958 183.876 \nL 434.152958 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_15\">\n      <!-- −2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(423.612294 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_30\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 504.910765 183.876 \nL 504.910765 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_16\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(500.361578 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_31\">\n     <g id=\"line2d_22\">\n      <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 575.668572 183.876 \nL 575.668572 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_17\">\n      <!-- 2 -->\n      <g style=\"fill:#262626;\" transform=\"translate(571.119385 204.241766)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_18\">\n     <!-- $x$ -->\n     <defs>\n      <path d=\"M 60.015625 54.6875 \nL 34.90625 27.875 \nL 50.296875 0 \nL 39.984375 0 \nL 28.421875 21.6875 \nL 8.296875 0 \nL -2.59375 0 \nL 24.3125 28.8125 \nL 10.015625 54.6875 \nL 20.3125 54.6875 \nL 30.8125 34.90625 \nL 49.125 54.6875 \nz\n\" id=\"DejaVuSans-Oblique-120\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(498.910765 226.412594)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.3125)\" xlink:href=\"#DejaVuSans-Oblique-120\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_4\">\n    <g id=\"ytick_46\">\n     <g id=\"line2d_23\">\n      <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 393.764662 155.43 \nL 616.056869 155.43 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_19\">\n      <!-- −1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(363.183334 160.862883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-8722\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_47\">\n     <g id=\"line2d_24\">\n      <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 393.764662 98.538 \nL 616.056869 98.538 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_20\">\n      <!-- 0 -->\n      <g style=\"fill:#262626;\" transform=\"translate(375.166287 103.970883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_48\">\n     <g id=\"line2d_25\">\n      <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 393.764662 41.646 \nL 616.056869 41.646 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:round;\"/>\n     </g>\n     <g id=\"text_21\">\n      <!-- 1 -->\n      <g style=\"fill:#262626;\" transform=\"translate(375.166287 47.078883)scale(0.143 -0.143)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_22\">\n     <!-- $f$ -->\n     <defs>\n      <path d=\"M 47.796875 75.984375 \nL 46.390625 68.5 \nL 37.796875 68.5 \nQ 32.90625 68.5 30.6875 66.578125 \nQ 28.46875 64.65625 27.390625 59.515625 \nL 26.421875 54.6875 \nL 41.21875 54.6875 \nL 39.890625 47.703125 \nL 25.09375 47.703125 \nL 15.828125 0 \nL 6.78125 0 \nL 16.109375 47.703125 \nL 7.515625 47.703125 \nL 8.796875 54.6875 \nL 17.390625 54.6875 \nL 18.109375 58.5 \nQ 19.96875 68.171875 24.625 72.078125 \nQ 29.296875 75.984375 39.3125 75.984375 \nz\n\" id=\"DejaVuSans-Oblique-102\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(355.023959 102.138)rotate(-90)scale(0.2 -0.2)\">\n      <use transform=\"translate(0 0.015625)\" xlink:href=\"#DejaVuSans-Oblique-102\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"PolyCollection_1\">\n    <defs>\n     <path d=\"M 393.764659 -133.559857 \nL 393.764659 -133.115389 \nL 398.301231 -124.31491 \nL 402.837812 -116.085245 \nL 407.374385 -109.070807 \nL 411.910958 -103.744669 \nL 416.447539 -99.617979 \nL 420.984112 -97.084342 \nL 425.520693 -95.921396 \nL 430.057265 -95.291221 \nL 434.593842 -94.175753 \nL 439.130419 -92.273176 \nL 443.666992 -89.313257 \nL 448.203569 -85.447319 \nL 452.740146 -81.292398 \nL 457.276718 -77.806775 \nL 461.813295 -75.939322 \nL 466.349872 -70.671761 \nL 470.886445 -67.103519 \nL 475.423026 -66.312535 \nL 479.959599 -69.219425 \nL 484.49618 -76.548602 \nL 489.032752 -88.767935 \nL 493.569325 -105.765552 \nL 498.105906 -125.211446 \nL 502.642479 -132.085771 \nL 507.179051 -140.392168 \nL 511.715633 -152.995884 \nL 516.252205 -156.178302 \nL 520.788778 -162.119033 \nL 525.325359 -170.882781 \nL 529.861932 -180.810108 \nL 534.398513 -190.090594 \nL 538.935086 -197.248385 \nL 543.471658 -201.404768 \nL 548.008231 -202.353326 \nL 552.54482 -196.631659 \nL 557.081393 -191.143929 \nL 561.617966 -186.801193 \nL 566.154538 -183.979078 \nL 570.691111 -182.641616 \nL 575.227701 -182.39287 \nL 579.764273 -182.601951 \nL 584.300846 -182.269403 \nL 588.837419 -180.95324 \nL 593.373991 -178.589365 \nL 597.910564 -174.633579 \nL 602.447154 -168.884163 \nL 606.983726 -161.809888 \nL 611.520299 -153.77621 \nL 616.056872 -144.90808 \nL 616.056872 -145.352549 \nL 616.056872 -145.352549 \nL 611.520299 -154.153028 \nL 606.983726 -162.392592 \nL 602.447154 -169.368511 \nL 597.910564 -174.809268 \nL 593.373991 -178.849959 \nL 588.837419 -181.383596 \nL 584.300846 -182.477287 \nL 579.764273 -183.151955 \nL 575.227701 -184.288387 \nL 570.691111 -186.194762 \nL 566.154538 -189.154681 \nL 561.617966 -193.019293 \nL 557.081393 -197.175553 \nL 552.54482 -200.665693 \nL 548.008231 -202.545787 \nL 543.471658 -207.796176 \nL 538.935086 -211.364419 \nL 534.398513 -212.155545 \nL 529.861932 -209.248513 \nL 525.325359 -201.919607 \nL 520.788778 -189.700002 \nL 516.252205 -172.702386 \nL 511.715633 -153.256478 \nL 507.179051 -146.382166 \nL 502.642479 -138.07577 \nL 498.105906 -125.47204 \nL 493.569325 -122.289635 \nL 489.032752 -116.348904 \nL 484.49618 -107.585428 \nL 479.959599 -97.657829 \nL 475.423026 -88.377204 \nL 470.886445 -81.219552 \nL 466.349872 -77.06317 \nL 461.813295 -76.096463 \nL 457.276718 -81.835437 \nL 452.740146 -87.324534 \nL 448.203569 -91.668396 \nL 443.666992 -94.48886 \nL 439.130419 -95.826322 \nL 434.593842 -96.074524 \nL 430.057265 -95.889608 \nL 425.520693 -96.263883 \nL 420.984112 -97.514697 \nL 416.447539 -99.878572 \nL 411.910958 -103.744669 \nL 407.374385 -109.609465 \nL 402.837812 -116.646359 \nL 398.301231 -124.691727 \nL 393.764659 -133.559857 \nz\n\" id=\"m26d6eb9637\" style=\"stroke:#0000ff;stroke-opacity:0.2;\"/>\n    </defs>\n    <g clip-path=\"url(#pf0a37ae83e)\">\n     <use style=\"fill:#0000ff;fill-opacity:0.2;stroke:#0000ff;stroke-opacity:0.2;\" x=\"0\" xlink:href=\"#m26d6eb9637\" y=\"237.771969\"/>\n    </g>\n   </g>\n   <g id=\"line2d_26\">\n    <defs>\n     <path d=\"M 0 5 \nC 1.326016 5 2.597899 4.473168 3.535534 3.535534 \nC 4.473168 2.597899 5 1.326016 5 0 \nC 5 -1.326016 4.473168 -2.597899 3.535534 -3.535534 \nC 2.597899 -4.473168 1.326016 -5 0 -5 \nC -1.326016 -5 -2.597899 -4.473168 -3.535534 -3.535534 \nC -4.473168 -2.597899 -5 -1.326016 -5 0 \nC -5 1.326016 -4.473168 2.597899 -3.535534 3.535534 \nC -2.597899 4.473168 -1.326016 5 0 5 \nz\n\" id=\"m6aa0739c5f\" style=\"stroke:#c44e52;\"/>\n    </defs>\n    <g clip-path=\"url(#pf0a37ae83e)\">\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"613.786595\" xlink:href=\"#m6aa0739c5f\" y=\"88.292384\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"596.504738\" xlink:href=\"#m6aa0739c5f\" y=\"61.345553\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"427.180774\" xlink:href=\"#m6aa0739c5f\" y=\"141.72304\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"498.181644\" xlink:href=\"#m6aa0739c5f\" y=\"112.235927\"/>\n     <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"548.102129\" xlink:href=\"#m6aa0739c5f\" y=\"35.346488\"/>\n    </g>\n   </g>\n   <g id=\"line2d_27\">\n    <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 393.764659 104.434346 \nL 398.301231 113.26865 \nL 402.837812 121.406167 \nL 407.374385 128.431833 \nL 411.910958 134.0273 \nL 416.447539 138.023693 \nL 420.984112 140.472449 \nL 425.520693 141.679329 \nL 430.057265 142.181554 \nL 434.593842 142.64683 \nL 439.130419 143.72222 \nL 443.666992 145.870911 \nL 448.203569 149.214111 \nL 452.740146 153.463502 \nL 457.276718 157.950863 \nL 461.813295 161.754076 \nL 466.349872 163.904503 \nL 470.886445 163.610433 \nL 475.423026 160.427099 \nL 479.959599 154.333342 \nL 484.49618 145.704954 \nL 489.032752 135.213549 \nL 493.569325 123.744375 \nL 498.105906 112.430226 \nL 502.642479 102.691198 \nL 507.179051 94.384802 \nL 511.715633 84.645788 \nL 516.252205 73.331625 \nL 520.788778 61.862451 \nL 525.325359 51.370775 \nL 529.861932 42.742658 \nL 534.398513 36.648901 \nL 538.935086 33.465567 \nL 543.471658 33.171497 \nL 548.008231 35.322412 \nL 552.54482 39.123293 \nL 557.081393 43.612226 \nL 561.617966 47.861726 \nL 566.154538 51.205089 \nL 570.691111 53.35378 \nL 575.227701 54.43134 \nL 579.764273 54.895016 \nL 584.300846 55.398624 \nL 588.837419 56.603551 \nL 593.373991 59.052307 \nL 597.910564 63.050545 \nL 602.447154 68.645632 \nL 606.983726 75.670728 \nL 611.520299 83.80735 \nL 616.056872 92.641654 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n   </g>\n   <g id=\"line2d_28\">\n    <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 393.764659 106.08875 \nL 398.301231 113.502134 \nL 402.837812 121.369752 \nL 407.374385 128.275741 \nL 411.910958 133.815788 \nL 416.447539 137.851483 \nL 420.984112 140.434778 \nL 425.520693 142.483871 \nL 430.057265 144.646546 \nL 434.593842 146.601491 \nL 439.130419 148.691723 \nL 443.666992 151.202399 \nL 448.203569 154.218009 \nL 452.740146 157.46697 \nL 457.276718 160.248372 \nL 461.813295 161.768888 \nL 466.349872 166.5837 \nL 470.886445 170.047777 \nL 475.423026 170.937854 \nL 479.959599 168.184977 \nL 484.49618 161.038298 \nL 489.032752 149.052434 \nL 493.569325 132.190105 \nL 498.105906 112.57911 \nL 502.642479 105.258966 \nL 507.179051 97.461108 \nL 511.715633 84.731194 \nL 516.252205 80.104978 \nL 520.788778 73.200443 \nL 525.325359 64.457052 \nL 529.861932 55.252389 \nL 534.398513 46.853377 \nL 538.935086 40.357742 \nL 543.471658 36.517118 \nL 548.008231 35.52879 \nL 552.54482 40.511335 \nL 557.081393 45.241452 \nL 561.617966 49.079695 \nL 566.154538 51.81794 \nL 570.691111 53.466064 \nL 575.227701 54.27108 \nL 579.764273 54.692517 \nL 584.300846 55.31477 \nL 588.837419 57.350309 \nL 593.373991 60.838778 \nL 597.910564 65.249382 \nL 602.447154 70.595018 \nL 606.983726 76.795042 \nL 611.520299 83.826038 \nL 616.056872 92.553845 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_29\">\n    <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 393.764659 104.52215 \nL 398.301231 113.249964 \nL 402.837812 120.280968 \nL 407.374385 126.480992 \nL 411.910958 131.826611 \nL 416.447539 136.237219 \nL 420.984112 139.725694 \nL 425.520693 141.761223 \nL 430.057265 142.383483 \nL 434.593842 142.804923 \nL 439.130419 143.609936 \nL 443.666992 145.258064 \nL 448.203569 147.996308 \nL 452.740146 151.834555 \nL 457.276718 156.564672 \nL 461.813295 161.547196 \nL 466.349872 160.558888 \nL 470.886445 156.718258 \nL 475.423026 150.222626 \nL 479.959599 141.823611 \nL 484.49618 132.618935 \nL 489.032752 123.875562 \nL 493.569325 116.971022 \nL 498.105906 112.344803 \nL 502.642479 99.614892 \nL 507.179051 91.817034 \nL 511.715633 84.496856 \nL 516.252205 64.885891 \nL 520.788778 48.023563 \nL 525.325359 36.037675 \nL 529.861932 28.89103 \nL 534.398513 26.138133 \nL 538.935086 27.028223 \nL 543.471658 30.492294 \nL 548.008231 35.307112 \nL 552.54482 36.827621 \nL 557.081393 39.609037 \nL 561.617966 42.857994 \nL 566.154538 45.873597 \nL 570.691111 48.384277 \nL 575.227701 50.474506 \nL 579.764273 52.429461 \nL 584.300846 54.592122 \nL 588.837419 56.641212 \nL 593.373991 59.224513 \nL 597.910564 63.260206 \nL 602.447154 68.800269 \nL 606.983726 75.706261 \nL 611.520299 83.573864 \nL 616.056872 90.987257 \n\" style=\"fill:none;stroke:#1a1a1a;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_30\">\n    <path clip-path=\"url(#pf0a37ae83e)\" d=\"M 393.764659 105.30545 \nL 398.301231 113.376049 \nL 402.837812 120.82536 \nL 407.374385 127.378366 \nL 411.910958 132.821199 \nL 416.447539 137.044351 \nL 420.984112 140.080236 \nL 425.520693 142.122547 \nL 430.057265 143.515015 \nL 434.593842 144.703207 \nL 439.130419 146.15083 \nL 443.666992 148.230232 \nL 448.203569 151.107159 \nL 452.740146 154.650762 \nL 457.276718 158.406522 \nL 461.813295 161.658042 \nL 466.349872 163.571294 \nL 470.886445 163.383017 \nL 475.423026 160.580238 \nL 479.959599 155.004294 \nL 484.49618 146.828618 \nL 489.032752 136.463997 \nL 493.569325 124.580564 \nL 498.105906 112.461956 \nL 502.642479 102.436929 \nL 507.179051 94.639071 \nL 511.715633 84.614025 \nL 516.252205 72.495435 \nL 520.788778 60.612003 \nL 525.325359 50.247365 \nL 529.861932 42.07171 \nL 534.398513 36.495755 \nL 538.935086 33.692983 \nL 543.471658 33.504706 \nL 548.008231 35.417951 \nL 552.54482 38.669478 \nL 557.081393 42.425245 \nL 561.617966 45.968844 \nL 566.154538 48.845768 \nL 570.691111 50.92517 \nL 575.227701 52.372793 \nL 579.764273 53.560989 \nL 584.300846 54.953446 \nL 588.837419 56.99576 \nL 593.373991 60.031646 \nL 597.910564 64.254794 \nL 602.447154 69.697644 \nL 606.983726 76.250652 \nL 611.520299 83.699951 \nL 616.056872 91.770551 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:round;stroke-width:2;\"/>\n   </g>\n   <g id=\"patch_9\">\n    <path d=\"M 393.764662 183.876 \nL 393.764662 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_10\">\n    <path d=\"M 616.056869 183.876 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_11\">\n    <path d=\"M 393.764662 183.876 \nL 616.056869 183.876 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"patch_12\">\n    <path d=\"M 393.764662 13.2 \nL 616.056869 13.2 \n\" style=\"fill:none;stroke:#ffffff;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n   </g>\n   <g id=\"legend_2\">\n    <g id=\"patch_13\">\n     <path d=\"M 400.764662 65.234375 \nL 512.689662 65.234375 \nQ 514.689662 65.234375 514.689662 63.234375 \nL 514.689662 20.2 \nQ 514.689662 18.2 512.689662 18.2 \nL 400.764662 18.2 \nQ 398.764662 18.2 398.764662 20.2 \nL 398.764662 63.234375 \nQ 398.764662 65.234375 400.764662 65.234375 \nz\n\" style=\"fill:#f2f2f2;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_31\"/>\n    <g id=\"line2d_32\">\n     <g>\n      <use style=\"fill:#c44e52;stroke:#c44e52;\" x=\"412.764662\" xlink:href=\"#m6aa0739c5f\" y=\"26.298438\"/>\n     </g>\n    </g>\n    <g id=\"text_23\">\n     <!-- Train -->\n     <g style=\"fill:#262626;\" transform=\"translate(430.764662 29.798438)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-84\"/>\n      <use x=\"60.865234\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"101.978516\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"163.257812\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"191.041016\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_33\">\n     <path d=\"M 402.764662 40.976562 \nL 422.764662 40.976562 \n\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:3;\"/>\n    </g>\n    <g id=\"line2d_34\"/>\n    <g id=\"text_24\">\n     <!-- Infinite Network -->\n     <defs>\n      <path d=\"M 9.8125 72.90625 \nL 23.09375 72.90625 \nL 55.421875 11.921875 \nL 55.421875 72.90625 \nL 64.984375 72.90625 \nL 64.984375 0 \nL 51.703125 0 \nL 19.390625 60.984375 \nL 19.390625 0 \nL 9.8125 0 \nz\n\" id=\"DejaVuSans-78\"/>\n      <path d=\"M 4.203125 54.6875 \nL 13.1875 54.6875 \nL 24.421875 12.015625 \nL 35.59375 54.6875 \nL 46.1875 54.6875 \nL 57.421875 12.015625 \nL 68.609375 54.6875 \nL 77.59375 54.6875 \nL 63.28125 0 \nL 52.6875 0 \nL 40.921875 44.828125 \nL 29.109375 0 \nL 18.5 0 \nz\n\" id=\"DejaVuSans-119\"/>\n      <path d=\"M 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 31.109375 \nL 44.921875 54.6875 \nL 56.390625 54.6875 \nL 27.390625 29.109375 \nL 57.625 0 \nL 45.90625 0 \nL 18.109375 26.703125 \nL 18.109375 0 \nL 9.078125 0 \nz\n\" id=\"DejaVuSans-107\"/>\n     </defs>\n     <g style=\"fill:#262626;\" transform=\"translate(430.764662 44.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"29.492188\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"92.871094\" xlink:href=\"#DejaVuSans-102\"/>\n      <use x=\"128.076172\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"155.859375\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"219.238281\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"247.021484\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"286.230469\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"347.753906\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"379.541016\" xlink:href=\"#DejaVuSans-78\"/>\n      <use x=\"454.345703\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"515.869141\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"555.078125\" xlink:href=\"#DejaVuSans-119\"/>\n      <use x=\"636.865234\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"698.046875\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"739.160156\" xlink:href=\"#DejaVuSans-107\"/>\n     </g>\n    </g>\n    <g id=\"line2d_35\">\n     <path d=\"M 402.764662 55.654687 \nL 422.764662 55.654687 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:round;stroke-width:2;\"/>\n    </g>\n    <g id=\"line2d_36\"/>\n    <g id=\"text_25\">\n     <!-- Finite Ensemble -->\n     <g style=\"fill:#262626;\" transform=\"translate(430.764662 59.154687)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"57.410156\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"85.193359\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"148.572266\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"176.355469\" xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"215.564453\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"277.087891\" xlink:href=\"#DejaVuSans-32\"/>\n      <use x=\"308.875\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"372.058594\" xlink:href=\"#DejaVuSans-110\"/>\n      <use x=\"435.4375\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"487.537109\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"549.060547\" xlink:href=\"#DejaVuSans-109\"/>\n      <use x=\"646.472656\" xlink:href=\"#DejaVuSans-98\"/>\n      <use x=\"709.949219\" xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"737.732422\" xlink:href=\"#DejaVuSans-101\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p0ccaef7514\">\n   <rect height=\"170.676\" width=\"222.292207\" x=\"73.66125\" y=\"13.2\"/>\n  </clipPath>\n  <clipPath id=\"pf0a37ae83e\">\n   <rect height=\"170.676\" width=\"222.292207\" x=\"393.764662\" y=\"13.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 900x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# test {\"skip\": true}\n",
    "plt.subplot(1, 2, 1)\n",
    "\n",
    "mean_train_loss = jnp.mean(train_loss, axis=0)\n",
    "mean_test_loss = jnp.mean(test_loss, axis=0)\n",
    "\n",
    "plt.loglog(ts, ntk_train_loss_mean, linewidth=3)\n",
    "plt.loglog(ts, ntk_test_loss_mean, linewidth=3)\n",
    "\n",
    "plt.loglog(ts, mean_train_loss, 'k-', linewidth=2)\n",
    "plt.loglog(ts, mean_test_loss, 'k-', linewidth=2)\n",
    "\n",
    "plt.xlim([10 ** 0, 10 ** 3])\n",
    "\n",
    "plt.xscale('log')\n",
    "plt.yscale('log')\n",
    "format_plot('Step', 'Loss')\n",
    "\n",
    "legend(['Infinite Train', 'Infinite Test', 'Finite Ensemble'])\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "\n",
    "plot_fn(train, None)\n",
    "\n",
    "plt.plot(test_xs, ntk_mean, 'b-', linewidth=3)\n",
    "plt.fill_between(\n",
    "    jnp.reshape(test_xs, (-1)),\n",
    "    ntk_mean - 2 * ntk_std,\n",
    "    ntk_mean +  2 * ntk_std,\n",
    "    color='blue', alpha=0.2)\n",
    "\n",
    "ensemble_fx = vmap(apply_fn, (0, None))(params, test_xs)\n",
    "\n",
    "mean_fx = jnp.reshape(jnp.mean(ensemble_fx, axis=0), (-1,))\n",
    "std_fx = jnp.reshape(jnp.std(ensemble_fx, axis=0), (-1,))\n",
    "\n",
    "plt.plot(test_xs, mean_fx - 2 * std_fx, 'k--', label='_nolegend_')\n",
    "plt.plot(test_xs, mean_fx + 2 * std_fx, 'k--', label='_nolegend_')\n",
    "plt.plot(test_xs, mean_fx, linewidth=2, color='black')\n",
    "\n",
    "legend(['Train', 'Infinite Network', 'Finite Ensemble'], loc='upper left')\n",
    "\n",
    "plt.xlim([-jnp.pi, jnp.pi])\n",
    "plt.ylim([-1.5, 1.5])\n",
    "\n",
    "format_plot('$x$', '$f$')\n",
    "finalize_plot((1.5, 0.6))\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "last_runtime": {
    "build_target": "//learning/deepmind/dm_python:dm_notebook3",
    "kind": "private"
   },
   "name": "Neural Tangents Cookbook.ipynb",
   "provenance": [
    {
     "file_id": "/piper/depot/google3/third_party/py/neural_tangents/notebooks/neural_tangents_cookbook.ipynb",
     "timestamp": 1569345842698
    }
   ],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}