transformer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "transformer.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "s_qNSzzyaCbD"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dhirensk/ai/blob/master/transformer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "s_qNSzzyaCbD"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "jmjh290raIky",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "J0Qjg6vuaHNt"
      },
      "source": [
        "# Transformer model for language understanding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AOpGoE2T-YXS"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/text/transformer\">\n",
        "    <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />\n",
        "    View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/text/transformer.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
        "    Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/text/transformer.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
        "    View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/text/transformer.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "M-f8TnGpE_ex"
      },
      "source": [
        "This tutorial trains a <a href=\"https://arxiv.org/abs/1706.03762\" class=\"external\">Transformer model</a> to translate Portuguese to English. This is an advanced example that assumes knowledge of [text generation](text_generation.ipynb) and [attention](nmt_with_attention.ipynb).\n",
        "\n",
        "The core idea behind the Transformer model is *self-attention*—the ability to attend to different positions of the input sequence to compute a representation of that sequence. Transformer creates stacks of self-attention layers and is explained below in the sections *Scaled dot product attention* and *Multi-head attention*.\n",
        "\n",
        "A transformer model handles variable-sized input using stacks of self-attention layers instead of [RNNs](text_classification_rnn.ipynb) or [CNNs](../images/intro_to_cnns.ipynb). This general architecture has a number of advantages:\n",
        "\n",
        "* It make no assumptions about the temporal/spatial relationships across the data. This is ideal for processing a set of objects (for example, [StarCraft units](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/#block-8)).\n",
        "* Layer outputs can be calculated in parallel, instead of a series like an RNN.\n",
        "* Distant items can affect each other's output without passing through many RNN-steps, or convolution layers (see [Scene Memory Transformer](https://arxiv.org/pdf/1903.03878.pdf) for example).\n",
        "* It can learn long-range dependencies. This is a challenge in many sequence tasks.\n",
        "\n",
        "The downsides of this architecture are:\n",
        "\n",
        "* For a time-series, the output for a time-step is calculated from the *entire history* instead of only the inputs and current hidden-state. This _may_ be less efficient.   \n",
        "* If the input *does* have a  temporal/spatial relationship, like text, some positional encoding must be added or the model will effectively see a bag of words. \n",
        "\n",
        "After training the model in this notebook, you will be able to input a Portuguese sentence and return the English translation.\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/attention_map_portuguese.png\" width=\"800\" alt=\"Attention heatmap\">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JjJJyJTZYebt",
        "colab": {}
      },
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "try:\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow_datasets as tfds\n",
        "import tensorflow as tf\n",
        "\n",
        "import time\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fd1NWMxjfsDd"
      },
      "source": [
        "## Setup input pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "t4_Qt8W1hJE_"
      },
      "source": [
        "Use [TFDS](https://www.tensorflow.org/datasets) to load the [Portugese-English translation dataset](https://github.com/neulab/word-embeddings-for-nmt) from the [TED Talks Open Translation Project](https://www.ted.com/participate/translate).\n",
        "\n",
        "This dataset contains approximately 50000 training examples, 1100 validation examples, and 2000 test examples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8q9t4FmN96eN",
        "colab": {}
      },
      "source": [
        "examples, metadata = tfds.load('ted_hrlr_translate/pt_to_en', with_info=True,\n",
        "                               as_supervised=True)\n",
        "train_examples, val_examples = examples['train'], examples['validation']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s69vQrjJFVZD",
        "colab_type": "text"
      },
      "source": [
        "as_supervised: bool, if True, the returned tf.data.Dataset will have a 2-tuple structure (input, label) according to builder.info.supervised_keys. If False, the default, the returned tf.data.Dataset will have a dictionary with all the features.\n",
        "\n",
        "\n",
        "with_info: bool, if True, tfds.load will return the tuple (tf.data.Dataset, tfds.core.DatasetInfo) containing the info associated with the builder.\n",
        "\n",
        "\n",
        "https://www.tensorflow.org/datasets/api_docs/python/tfds/load\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ofl7C4rF7STs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 73
        },
        "outputId": "3df39e9a-d5ee-482a-88e0-7d43c6ac119c"
      },
      "source": [
        "print(type(examples))\n",
        "print(type(examples['train']))\n",
        "print(train_examples.output_shapes)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'dict'>\n",
            "<class 'tensorflow.python.data.ops.dataset_ops._OptionsDataset'>\n",
            "(TensorShape([]), TensorShape([]))\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RCEKotqosGfq"
      },
      "source": [
        "Create a custom subwords tokenizer from the training dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "KVBg5Q8tBk5z",
        "colab": {}
      },
      "source": [
        "tokenizer_en = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
        "    (en.numpy() for pt, en in train_examples), target_vocab_size=2**13)\n",
        "\n",
        "tokenizer_pt = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
        "    (pt.numpy() for pt, en in train_examples), target_vocab_size=2**13)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LoUzO5IzM6QP",
        "colab_type": "text"
      },
      "source": [
        "target_vocab_size: int, approximate size of the vocabulary to create.\n",
        "\n",
        "Builds a SubwordTextEncoder based on the corpus_generator.\n",
        "Args:\n",
        "\n",
        "    corpus_generator: generator yielding str, from which subwords will be constructed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "4DYWukNFkGQN",
        "colab": {}
      },
      "source": [
        "sample_string = 'Transformer is awesome.'\n",
        "\n",
        "tokenized_string = tokenizer_en.encode(sample_string)\n",
        "print ('Tokenized string is {}'.format(tokenized_string))\n",
        "\n",
        "original_string = tokenizer_en.decode(tokenized_string)\n",
        "print ('The original string: {}'.format(original_string))\n",
        "\n",
        "assert original_string == sample_string"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "o9KJWJjrsZ4Y"
      },
      "source": [
        "The tokenizer encodes the string by breaking it into subwords if the word is not in its dictionary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "bf2ntBxjkqK6",
        "colab": {}
      },
      "source": [
        "for ts in tokenized_string:\n",
        "  print ('{} ----> {}'.format(ts, tokenizer_en.decode([ts])))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "bcRp7VcQ5m6g",
        "colab": {}
      },
      "source": [
        "BUFFER_SIZE = 20000\n",
        "BATCH_SIZE = 64"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fbh_v0-9S2iI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "e896353b-4955-47c6-b3da-8849d2cb781f"
      },
      "source": [
        "print(tokenizer_pt.vocab_size)\n",
        "print(tokenizer_en.vocab_size)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "8214\n",
            "8087\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "soK7v61dTNqN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "outputId": "31270a36-eecd-47e0-acda-131e5ca06d65"
      },
      "source": [
        "print(tokenizer_pt.decode([8213]))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "�\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kGi4PoVakxdc"
      },
      "source": [
        "Add a start and end token to the input and target.  vocab_size & vocab_size+1 for imaginary \\<start> and \\<end> tags"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "UZwnPr4R055s",
        "colab": {}
      },
      "source": [
        "def encode(lang1, lang2):\n",
        "  lang1 = [tokenizer_pt.vocab_size] + tokenizer_pt.encode(\n",
        "      lang1.numpy()) + [tokenizer_pt.vocab_size+1]\n",
        "\n",
        "  lang2 = [tokenizer_en.vocab_size] + tokenizer_en.encode(\n",
        "      lang2.numpy()) + [tokenizer_en.vocab_size+1]\n",
        "  \n",
        "  return lang1, lang2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6JrGp5Gek6Ql"
      },
      "source": [
        "Note: To keep this example small and relatively fast, drop examples with a length of over 40 tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "2QEgbjntk6Yf",
        "colab": {}
      },
      "source": [
        "MAX_LENGTH = 40"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "c081xPGv1CPI",
        "colab": {}
      },
      "source": [
        "def filter_max_length(x, y, max_length=MAX_LENGTH):\n",
        "  return tf.logical_and(tf.size(x) <= max_length,\n",
        "                        tf.size(y) <= max_length)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Tx1sFbR-9fRs"
      },
      "source": [
        "Operations inside `.map()` run in graph mode and receive a graph tensor that do not have a numpy attribute. The `tokenizer` expects a string or Unicode symbol to encode it into integers. Hence, you need to run the encoding inside a `tf.py_function`, which receives an eager tensor having a numpy attribute that contains the string value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Mah1cS-P70Iz",
        "colab": {}
      },
      "source": [
        "def tf_encode(pt, en):\n",
        "  return tf.py_function(encode, [pt, en], [tf.int64, tf.int64])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "9mk9AZdZ5bcS",
        "colab": {}
      },
      "source": [
        "train_dataset = train_examples.map(tf_encode)\n",
        "train_dataset = train_dataset.filter(filter_max_length)\n",
        "# cache the dataset to memory to get a speedup while reading from it.\n",
        "train_dataset = train_dataset.cache()\n",
        "train_dataset = train_dataset.shuffle(BUFFER_SIZE).padded_batch(\n",
        "    BATCH_SIZE, padded_shapes=([-1], [-1]))\n",
        "train_dataset = train_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n",
        "\n",
        "\n",
        "val_dataset = val_examples.map(tf_encode)\n",
        "val_dataset = val_dataset.filter(filter_max_length).padded_batch(\n",
        "    BATCH_SIZE, padded_shapes=([-1], [-1]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "_fXvfYVfQr2n",
        "colab": {}
      },
      "source": [
        "pt_batch, en_batch = next(iter(val_dataset))\n",
        "pt_batch, en_batch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nBQuibYA4n0n"
      },
      "source": [
        "## Positional encoding\n",
        "\n",
        "Since this model doesn't contain any recurrence or convolution, positional encoding is added to give the model some information about the relative position of the words in the sentence. \n",
        "\n",
        "The positional encoding vector is added to the embedding vector. Embeddings represent a token in a d-dimensional space where tokens with similar meaning will be closer to each other. But the embeddings do not encode the relative position of words in a sentence. So after adding the positional encoding, words will be closer to each other based on the *similarity of their meaning and their position in the sentence*, in the d-dimensional space.\n",
        "\n",
        "See the notebook on [positional encoding](https://github.com/tensorflow/examples/blob/master/community/en/position_encoding.ipynb) to learn more about it. The formula for calculating the positional encoding is as follows:\n",
        "\n",
        "$$\\Large{PE_{(pos, 2i)} = sin(pos / 10000^{2i / d_{model}})} $$\n",
        "$$\\Large{PE_{(pos, 2i+1)} = cos(pos / 10000^{2i / d_{model}})} $$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WhIOZjMNKujn",
        "colab": {}
      },
      "source": [
        "def get_angles(pos, i, d_model):\n",
        "  angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))\n",
        "  return pos * angle_rates"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "1Rz82wEs5biZ",
        "colab": {}
      },
      "source": [
        "def positional_encoding(position, d_model):\n",
        "  angle_rads = get_angles(np.arange(position)[:, np.newaxis],\n",
        "                          np.arange(d_model)[np.newaxis, :],\n",
        "                          d_model)\n",
        "  \n",
        "  # apply sin to even indices in the array; 2i\n",
        "  angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])\n",
        "  \n",
        "  # apply cos to odd indices in the array; 2i+1\n",
        "  angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])\n",
        "\n",
        " \n",
        "  pos_encoding = angle_rads[np.newaxis, ...]\n",
        "    \n",
        "  return tf.cast(pos_encoding, dtype=tf.float32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xHt0IJFDUFWP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "OqJr6Z0aUFnO",
        "colab": {}
      },
      "source": [
        "def positional_encoding2(position, d_model):\n",
        "  angle_rads = get_angles(np.arange(position)[:, np.newaxis],\n",
        "                          np.arange(d_model)[np.newaxis, :],\n",
        "                          d_model)\n",
        "  \n",
        "  angle_rads= np.concatenate([np.sin(angle_rads),np.cos(angle_rads)], axis = -1)\n",
        "\n",
        " \n",
        "  pos_encoding = angle_rads[np.newaxis, ...]\n",
        "    \n",
        "  return tf.cast(pos_encoding, dtype=tf.float32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "1kLCla68EloE",
        "colab": {}
      },
      "source": [
        "pos_encoding = positional_encoding(50, 512)\n",
        "print (pos_encoding.shape)\n",
        "plt.figure(figsize=(14,8))\n",
        "plt.pcolormesh(pos_encoding[0], cmap='RdBu')\n",
        "plt.xlabel('Depth')\n",
        "plt.xlim((0, 512))\n",
        "plt.ylabel('Position')\n",
        "plt.colorbar()\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R3JgPFscXi4h",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 520
        },
        "outputId": "41beb169-b1af-4a55-d78a-4949b793c6db"
      },
      "source": [
        "pos_encoding2 = positional_encoding2(50, 256)\n",
        "print (pos_encoding2.shape)\n",
        "plt.figure(figsize=(14,8))\n",
        "plt.pcolormesh(pos_encoding2[0], cmap='RdBu', vmin=-1, vmax=1)\n",
        "plt.xlabel('Depth')\n",
        "plt.xlim((0, 512))\n",
        "plt.ylabel('Position')\n",
        "plt.colorbar()\n",
        "plt.show()"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1, 50, 512)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwMAAAHkCAYAAABi/PMwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXxcVd3H8c+ZydqkTdO9dC8UWmwF\nyiIIIkU2QQEVARVkkQdEEMqDsoiiIiLgAig8CmIVNxBQdmQTCgKyU6CALaX7vqfZl5n7/HHuPeck\nmaYpTdqG+b5fr77m5nfuNmnmJHfO3PM1URQhIiIiIiL5J7WtT0BERERERLYNXQyIiIiIiOQpXQyI\niIiIiOQpXQyIiIiIiOQpXQyIiIiIiOQpXQyIiIiIiOSpbr0YMMbMN8a8ZYyZYYx5Ja71M8Y8box5\nL36s7M5zEBERERHZXhhjphljVhpjZm6k3RhjfmmMmWOMedMYMzloOyX+G/o9Y8wpXXE+W2NkYEoU\nRbtHUbRX/PUlwL+iKBoH/Cv+WkREREQkH/wBOKKD9k8D4+J/ZwK/BvuGOvB94GPAPsD3u+JN9W3x\nMaFjgNvi5duAY7fBOYiIiIiIbHVRFD0DrO1glWOAP0bWC0BfY8xQ4HDg8SiK1kZRtA54nI4vKjql\nuy8GIuAxY8yrxpgz49rgKIqWxcvLgcHdfA4iIiIiIj3FMGBR8PXiuLax+hYp2NIdbMIBURQtMcYM\nAh43xvw3bIyiKDLGRLk2jC8ezgQoxOzZzxSysqQ3AP3rq916a3r1AWBMU42rZceNAyAT2V03/ne2\na6vdYTQAZaX+qZu5cwAYvusYV3ttnr1gGzlyiKul57xn14+/Lpow3rXNmrvU1sp6u9quQ3sBUPX2\ne65W3ZIFYGD/UgAKRo51bXNW1gJQt359+H0AoKRPH1cb0c/ut5xGV2tYvtw+v3UNdh+ZrGtLvsHF\nKeNqvUrs8y+uKHW1wor4GCX+OTRm7TbVTRkAahqa/TEbba2lOeNq2Zam+NGvF2WzyQLtmOSc/LmZ\nVCpu8teqJp3OUSuI17fbpoLnl3zfTFBL2lPG19Jpu5w24Xq0qoX7TbYNSv5YwdNK2k2wX/dMk7bw\nOZvW67RdbnusXDbeEnybN6HD1bpiH9247bba85bufkNjCwAVhX4nUfxznmqqc7X34m6vbNF8AFaW\n93Ntu4+wr9kNcxe72ryS/gCMHuZHkDe8bbvg4buOcrU3Ftkdl1f2tW11y1zbqtX1tjZ+uKvN2mBf\nd3Xr1rha74EDABjXx78+1/93HgBVcZ9XWeL73N5jdrBtqV6utmS1fa6NNRtcLekzCuN+vnfvYtc0\nOF4ujZpcrWW9PaeGdbWuVl9v+6LGjP9Vk/RYydkWBC+QovjFW1Tkn0thaaFdr5c/fqrELqeK/XOI\nCorsY8o+1+agy2uJ+8Gm4Dyak1qLX7E5bm8J+vBMXIuy9jGb9fuIotZtthZvmw1/D0RJY6vtwlrY\nR0dt20JBrV1rrvXbr7X5cu5XeoKofs3qKIoGbuvzyCXVZ3hES0OX7zeqX/M2EO74liiKbunyA3WR\nbr0YiKJoSfy40hhzD/bzTSuMMUOjKFoWD3ms3Mi2twC3AAxJFUenFAzjul0OAuALM5506/1hVzs6\ncs3Cf7ta7Z8eAqC6yf6Sff8TB7u2F875PwD2mTDI1UpOOAaAq+7+rauVn3IHAN/91UWu1vezRwIQ\n//3IyIeecm0HfumHtrb3FFf7z2X2fo+HJh7uak+tsr/wzv7sJAAG/fIO13b0TS8A8Pp997pautD+\nctn1EL+Pn51k97t/1l9kvHfttfaYf38XgNfW+5/B5KJox15FrjZ5F/uHwk5HTXS1QUfY72X0Ef8c\n3q+zPyL/XrAOgKdn+f+u9+bZ2tpl/uKsesUCABrWrXC15np7oZZpqqctk7J/5KcK/LkVFMcXSiVl\nrlZYVmEfS8rb1UrKSgAojn9hAxTFF3vFJb5WHNd6BX+U9O1l28uD9XoXF8S11o8ApYX2fHvFjwDF\nBfaPhsLgCqG4wLaXFAR/UMTthfEPUGEqaEsuSoJ9pHNceCS18I/7pD1nW/wYXkSkcvzhmjTnWr/D\n7dqXcq7nj9PxX80dbbup43a83+69GNiS3T8+1178Hz7cvwaaC+xroHThK6722en2Z2qfC74KwA37\nfsW1vfiLAwF48ou+vzp5V7vedVee6GqPTdwXgF/89UZXG3DhMwB84vijAPjpS9e4tv/7/RsA/Gza\nta72yafsRcOrd/3F1fY/6wwAHjzMP4cH9z8VgIeX29f/F0b3d21T/nSFbSvd09W+M80+17nPPeZq\nyZsLQ3azfdJBB49zbd+eshMA41sWuNr6+/4IwKy7XnS1mW/aPmt2jb9oqIn/+E7+8B9Q5F/PY8ps\nXzB8hH8DZvAk+/tiwEf9mze9x+9s97HTR/359hsBQGO5HfBeVdfi2lbU2ouSJRt837ysxr6hs3C1\nv+hbVmX7yZUb/Js9tdX23Bvq7GNjvd9vc3wxmTwCNDfY/WUafZ+bfC8z7g0b//3INrevJW/iRNlM\nULPL2bCW8ctt1++otjntnV1Htk/NM36/YNNrbSMtDRTscnSX77Z5xu8bgntlP4glwIjg6+FxbQlw\nUJv69C04DtCNHxMyxpQZY3ony8BhwEzgfiC5+/kU4L7uOgcRERERkZyMwaTSXf6vC9wPfDWeVWhf\noCr+iP2jwGHGmMr4xuHD4toW6c6RgcHAPfE7gQXAX6MoesQY8zJwpzHma8AC4PhuPAcRERERke2G\nMeZ27Dv8A4wxi7EzBBUCRFH0G+Bh4EhgDlAHnBa3rTXG/Ah4Od7VFVEUdXQjcqd028VAFEVzgd1y\n1NcAn+qu44qIiIiIdEYXvZO/WaIo+tIm2iPgnI20TQOmdeX5KIFYRERERCRPdfdsQiIiIiIi2yGz\nTUYGtje6GBARERGR/GN0MQD6mJCIiIiISN7qESMDfQrTTBlewQsn2Xmzzxjqp6x9qL+dN/upVx9x\ntb9f9Q8AFn93FwBeO8SHid3y8IN2/alXutpFcaDLiyW7uloyp/FJEwe42qVr7NzN53x9bwB+/qI/\nj+baKgD22GOoq2VfeRiAt6r8fNGD4znshx24OwBvrPLzQK9cZPfR0uAD1HoNsHNZDx/u57weWWHD\nbppefdvV1sdz/q+Kg8CaggCa0ngO+37BXNq9Bth5zMuG+Lm/C/rbgLXaAh9Etj6er3pdPL/1mmCu\n7qZ4rutkTmvw81rnmpu6I+GVebIc1lI58gjSBa1Dx4I8MjevfFhL1isK5v5Px3P9h6FjyVz/bR/D\n/ebSKswsx2o+sKx9+Jmr5ZhBf1Nz2H/QKe67eer9Nsfq6PvWyX100blsb0Z+z860PHHI113t19Ov\nso8X+jyA+z9r+4BV62zmyDX3rnZtT51wCQCfvPViV1t7ru0Tjyze39X+FX8TZw/e19Xq1tisk4sO\nsX3NG9/1IY0T+9i+pmX3o1xt0Y02X6CkwmcIHbOHDcCsf/FPrvZOPE9+efx6GzTJZ7tEI23Oyquv\n+YDFDattHxb2f8W9bbBaeV+bOTJusM8Z6Vdq+4Ro3hJXq11ivyd1q32/mmQKhH1iInmdlqZ9n5Cc\nb3EfHzBW1Mf2iYVlvm9M9bJBb6bI16IiG0CWBIuFx2yMz6MxCBirj8Mcw1oSQNbSEoaOJXP+5wgY\nyxFElkvSD7vHzOb10V1NuQGyPTD4UNN8ppEBEREREZE81SNGBkREREREupQx7pMH+UwXAyIiIiKS\nl3QDsT4mJCIiIiKStzQyICIiIiL5R1OLAhoZEBERERHJWxoZEBEREZG8YwCT0vviuhgQERERkTyk\njwlBD7kYKJkwngn/eoZ/LXwWgIWn3OvaHm2Jw13qbne13y18F4DHPzcNgEOfv9O1RV+4CYDV10x1\ntU+PsGE+F9/1pqsN+ogN6sncd52r1cdBMqNOOQmAZ25f6NoKyyoAOHGvEa625OafA7CiscXVDhlk\nw3PKPnYwAM8tWOvaqpbOa/fcywaOBGDyqEpXG1Jiz6N+/vuutmHxBvvY0j7IJQnRCUPHyuPzKB7k\nQ9WyZTbgp6bJh92sjYPFVsYBQg31za6tKX5eSdAYQLbFhpJlmn04WUfhMi5gLN0+dCxVWLSJ9eJQ\nsDgwKB0EB6Xi55xuFTBm1y9ItQ8YC4PIkuVcoWNJSFGuwLBc4WSdDdTKJde2xmy8bUts7vsiH9YA\nsM3VFcFtv/nHLAAW7/OEq93z7ioAHrzpVld74sevADDs2ekA7FP0ul//2w/ZhUFTXG3AzmsAePuy\ny13t8FF9AfjOg++4WtlA22ft28sGgF26rsG1nXioDWx8Yp4PB1u3YCYA/cbs5moHj7F9x+LfveZq\nSb+3Q0khAEP22snvo8iGHb62IAgMW+n700RRue33KgbYMK8dB5S5tsq4e2hZ4berXW6fc93a9qFj\nmah9KFdR/EIqDVICS0rsr8WiskK/Xm973MI+vVwtVWZ/b2TD0LGCOBAy/l0RdscN8Xk0ZNqHjtU3\n+xWTALJMsF4SLNY2fKx1LQgRy2Ta1bLZ9rWOtA0pC/fR0fqbqonI9qdHXAyIiIiIiHQp3UAM6AZi\nEREREZG8pZEBEREREclLGhnQyICIiIiISN7SyICIiIiI5B9jWk1Mkq90MSAiIiIiecfmDOhiQB8T\nEhERERHJUxoZEBEREZH8o6lFgR5yMfDOvBVMPvkGzn/q9wDcfOxFru1fZQ8DMOIP97jax255CYD7\nvvUIAM+94cNbPnrU0QA88LNzXe2rv7YhYl+57d+u9rlTPwvAq9f92tUm9LaBMut2/hQAS2de69r6\n7zQZgCmjK1ztxSdsKFgmyLoZMXkIAC2j9gTgqek+6Kx+zVIA0kGITeUQGxI0aWgfV0uvsyE762Yv\ncrW1a2zIThKwEyqLw7h69fP77TXUhgSl+w91tWwvG/ATho6tqbPhYWtr7WNjvQ9Qa05Cx5rC0DEb\nStYqAKcToWOp4MWYKihs1WZrRa0ewYeMJeFjJgz9SmpBKlTbMLGwliuIzD2a9vsNMopyBpH5tnC/\nrWtBRpoLr8oVYpXqZMSXCyRrVdvyVKzOBpx1dRBavrj65i8DsNtHz3S1T99TA8BDaw5xtXt/9zgA\n/7rySQBe++kRrm3aD23fNPXmF13tiyceAMD9X/mFq533m68AcPpDPrBs7Mf2A2D97TcCrfuQCSfb\nELMrn5/vas21VQAM22W4q4026wB4+jnfJyUhjTuV29dzxe67u7bX4mCzpYuqXK2pxu4jfN2XVtr+\ncmAcOja6r+/D0huW2+MsW+xq1cvs921tk+9zGpLArqAfTl6zpfGLsDwIHSwss31McUWxr8VhY+my\n3q6WipezhT6ILJuEjjXZvrEh+F7WxcFijS3tQ8eactQyQc0Hi7X+2tai+LF9n9tR35s7JKz974+c\n22YUJibyYdIjLgZERERERLqWafVmZL7SxYCIiIiI5B+jG4hBNxCLiIiIiOQtjQyIiIiISN4x6AZi\n0MiAiIiIiEje0siAiIiIiOQljQzoYkBERERE8pFyBoAecjGQShdQUjGQISX2dNcvete1/er5GQA8\nO2C6q714+ScB+OdPywA465Z/urZnfnMGANd/r8HVao+8AICaa7/pat89ZCcA/nTaSlc7csooAG6f\nucJut8rPqf3Rw+183KXv+ayCNxfaObT7FfkftBEH7QrAnA12buglc9e5tqZ4/u5e/Xdwtf5D7FzW\nO/cP5rKe+xwA6+escLXlDXbe52Ru76Jg0vfk+GWD/T7KhticgYIBQ/zxS22mwYYa/71JcgbW1DQC\n0Fjf7NqaG+x6La1yBuz6Hc1vDe2vxMOvk+VN1VJJbkBBPG9/MHF/Kp5IPB3MH57kBhQV+H34Of83\nnjMQ5gck66fCTAOTI3sgx6T7yXptHzemo+awaVP7set0cr9dkEvQnfvbHJ35vmwPppYfB8CV//yx\nq712yjUAPDmozNWWv3cTANOefwCApVP/5Nq+dLnNHLjo9vtc7ZUrrrf7D16zHHcxAGtuOM+Vrrjw\nSADeOO17AIwoLXRtRYd8FYC3f/CqqyU5KIfsOczVsm/8C4D3llT7beMfuh0mDQKgYNd9/fNbtAGA\ntStqXC3p/wpKy12tfMAAACbEOStDy4OckTU2Z6B6oe8Ha1fUAlDVHMzbn2k/d35ybiXxY5gzUNzH\nHqOod4lfv7ftO00vn/diSm3fHBX6PIIkQ6Ap7oeTbAGAxvg86sNasn6Lr7XEtWwQjJCNa0n2QJIt\nAJDtZM6Aq+XICNhUf92VNvdYW/PcRPJVj7gYEBERERHpWhoZAN1ALCIiIiKStzQyICIiIiL5x4BJ\na2RAFwMiIiIikneUM2DpY0IiIiIiInlKIwMiIiIikn80tSigkQERERERkbylkQERERERyUsaGegh\nFwMTR/fnmd+dTHH1oQA0zfVtY463YTe/ffRuV3v77esAOOouG6Kz/rzHXNvIZ34DwN6VPlDmwgds\niFnFyAmuNuzdhwBY2tDiah/5+jEAnP3EHMCH7wCc/IkxAKx84GpXWxSH/exW4Y9V+YmDALh3oQ0b\nW7dkqWtLwlXKBo50tT3H2HCw4b19EFDT3Lft81pQ5WrrmlsHs4QhOknoWPlgH2DUa0h/u9B7gKtV\nN9p9rI6DxgBWbbBhY9W1ttZU778fmUYbNpZt9utnc4Td5NI2RCxXwFi6wAcMpQqL4loQLBaHSiXB\nViYMAku1DxMrTkLKOggYC5eLkvWD7KpcgWG5A8Zot15HUiT73cR62zBHq6sP3dnn0jOiwz64v/70\nRgBKZjzpan/7jw0efKb/y642+Z82PHHHy2yfN+2mqa7tnOV/BKD0Ed//rL7Gtu/bz/dTVz01D2gd\n7HXSRNsHXB0HIH5yfH/XNqPeBmutmuVDxyqG7wzA5yYOdbVlN04HYH7QdwyI+52he40GoL7fWNf2\n/JMzAahdtdDVkj6jKAj26h2f+7jB9nz7l/p+onnx+wDULFnlanWrbZ9U0+KDxpqCgK5EEjpWGgcV\nlgbBkMUVNkSsqI/vL5PlVO++/nzj/j8q9GGOSdhY8tgYBJ4lAWP1Tb5vTALI6oJaJtM6YAzCYLGo\n1de21j5MrDNBXbnWyRVSlt3EvtruRyFhIj1Pj7gYEBERERHpaqlt+S7bdkIXAyIiIiKSd4wxrT5V\nkK90A7GIiIiISJ7SyICIiIiI5CXTyXv7Psw0MiAiIiIikqc0MiAiIiIieUk3EOtiQERERETykWGb\n3EBsjDkCuAFIA7dGUXR1m/brgCnxl72AQVEU9Y3bMsBbcdvCKIqO3tLz0cWAiIiIiMhWYIxJAzcB\nhwKLgZeNMfdHUfROsk4URRcE638T2CPYRX0URbt35Tn1iIuB9W+9y/2j9uSqU64F4OnRT7u2Pq89\nCMDO17zpatN+cS4A+/7wAAB2PMiHwjx87qUAHPntT7naZfc8B8Buh+7nam/+5PsAjCj1YV8cdiYA\n83/3KwAqR090TUftbIN73vvmG65WHwfPjJvgg70Yvz8Aj92zAICaFfNdUyoO2eo7dLCrTR5pQ25K\nqha72rJ37bbrl9W4WhiyA1CW9reDVPSzoWdlQ3xgTsHAYQBkynzAUE2z3ceaIDhoZbUNHWuMw8aa\nGoPQsaY4dKwlCB2LA8g6Gzxj0knoWBAmFgeM5QwiC55XKgkFa/MIkIrXKwpquQLGOgwiSwLGwjbT\n+hE2EkRmkmOx0Vpn71nq7HqbewNQR+t39o2SfBxd7ep7zT77zbMAOO32ea52d7ENtLrpDve7gdcn\nPA/AvZceBMCff+1P5ITfvAjAx4870tUeuOR0AI751hRX++F9dn8j9/a1zH02pHF5HLA46bQDXNt3\nXrB9Td0aH4644362faLvTnhx+nsAVDX7figJWxy4n/0dNmtdo2t7f/56ABrWraCt0sohbnlgHDa2\nY6Xtw4trfcBY47L5ANQsq3a1qvg51AZhX3E33Oo1m4SOlRfYx+I+xa4tWS7u64PZ0mV2OdWrt6tl\n47CxbKEPlWyqi/vJ+Ph1QRhkstwqdCxebgr67yRsLBM8h2z8JLJRHD4W5Qgd20RgWK71fFu2XS2X\nMNhM5MPAsE1GBvYB5kRRNBfAGHMHcAzwzkbW/xLw/e48Id1ALCIiIiLSdQYYY14J/p0ZtA0DFgVf\nL45r7RhjRgFjgCeDckm8zxeMMcd2xcn2iJEBEREREZGuZVqN6neh1VEU7dUF+zkRuDuKonBYblQU\nRUuMMWOBJ40xb0VR9P6WHEQXAyIiIiKSf7bNDcRLgBHB18PjWi4nAueEhSiKlsSPc40x07H3E2zR\nxYA+JiQiIiIisnW8DIwzxowxxhRh/+C/v+1KxpjxQCXwn6BWaYwpjpcHAPuz8XsNOk0jAyIiIiKS\nl7b2yEAURS3GmHOBR7FTi06LouhtY8wVwCtRFCUXBicCd0ThjAEwAbjZGJPFvqF/dTgL0QeliwER\nERERka0kiqKHgYfb1C5v8/UPcmz3PDCpq89HFwMiIiIikneMUQIx9JCLgcZMxLzaZuY8bTMFrrz+\nUdf2Us2nAXjie37e7Pt/Z+d9vvTndr3bf+RnXvrL9bUAHDn156629uPfAOCGG09ztQcuXgjAQZP9\nnNe3z1wJwIbFswHY64tfcm2DV84A4N5Za12totDekjHqU+NdbUFzGQDvz1kDQEOVnze7tNLmCwwY\n5ueynjTILkeLX/XnO9vO+b2k3s/5Xx/PSZ3Mn92vyM/RXzbIHrN82EBXS8c5Ay29Kl1tw7pmAFYF\nOQNrauzc4E31tq250be1uJyBZlfrKF8gZ25AnK2QZCyEbWEtWQ6H85IXcJI9kErnaMuRH1CUbp89\nkCuPIJlhIG3a77d1pkDr7ULhernyCBK5JjNIkWO9HPttu61pdcz2+83FdM9sCttEN80M0W1uK7b9\n1II/3udq9zfa11TtnLtcbdo9twNQlrkTgNPP/bhr+9l9/wBg8RM/dbXLzm6w653/E1dbdvAlAPzo\nZ99ytVcvssGXQ0rsr4O+XzjDtf37ZzY/IHwtHjDZ9h3mzcdc7b+zbb8XzuU/ahebYVL00QMBeHFx\nlWtbs9xmAzTV+lpBiZ3Lv2yA73PHD+0DwIgKO/d/ev0c11a90GYUVAd5K2vjefvrM+3nzQ9fxyVx\nrkl5/LovKvd5MknOQFEfn0+T6m37yVRZH1drKbJZEE0ZP4LfEC8nmQJhzkB9UgtyBppa2ucMJJkC\nUdbvNxsvZ5MMgmD9jnIGWtXaZAR0NgtmW9nez0/kw6RHXAyIiIiIiHQ1o6l0dDEgIiIiIvnpwzQy\n/kHpekhEREREJE9pZEBERERE8o4xRjcQo5EBEREREZG8pZEBEREREclLWzt0bHukiwERERERyUu6\nGNDHhERERERE8laPGBkYOnFHLnngr2Rm2zCxSSf/27Xd9Lc/A7DqjZ+52ldv/18Azr30OQD2eqfB\ntc2ssPuY+sB/Xa3P8J0B2Hnhk642v86G/px3wTGuduEjNmwsCcU66aCxrm3VvTbEbHYc0gUwMQ6v\nGXTIp1ztj/NtOM+qeYuB1sEqZQNHAjB5bH9XG1lhw3Capr/pamvfs/tYHYTXJLk35QX2CndgsQ/4\n6r1DHOYThI6ZvjbgrLrFXxGvqLGBYsvX++9XVVxrjAPOWup9wE+m0YaOZVp8ENnmho61fYQgiKyw\nKKjZ69Z0EBiWhIwlV/XhTUAF8frFQZhYUUEcdJYjiCxnLS7lCg7LHTBGu/U6kjNULFf4WBe8abEl\nM6dt7qYdTdPW6RC0zTxmT3bxydMAuOMI/xp4frgNGdzrBR+wOPLSZwC4+ZpvA3DOCt8nFH3uxwA0\n3fhtV5vc1/Z117ywwtWS8LBz9hnuaj9704YpHjiuHwDvmB1c2/K37wB8HwnwpTh0bMVvr3e192tt\nHzCw2P9KGba/7R/rB+0CwLPPvu3aNiybD0A26DuS0MW+cUgiwMRhNuRrUC+73+aZ77u2mjh0rHZF\nratVNdswrqYgsMvtP0hES8LGyuPzLakscW3FfXu3egRIlcXhj4WlrhYV2lCyxiAArMmFjmXbtdUn\ngWhBEFkSQNYS1DI5gsWSsLEkfCxXmFhnQ7pyrZcrpOyD7qOz64tsF0zPC6rsDhoZEBERERHJUz1i\nZEBEREREpCsZdM8A6GJARERERPKS0cUA+piQiIiIiEje0siAiIiIiOQfgxKI0ciAiIiIiEje0siA\niIiIiOSljqbDzhcaGRARERERyVM9YmTg7RVNfPSGhTw9ejoAfWb6IJ5drn0DgN/87BxX2+dbNwMw\nbspQAB762iWu7fPfORSAH/7taVfb89OfAOCNH17uaiNKbdgXR57ranP+YEN2+o3dDYDPTfAhXrPO\ntyFB9RkfdjNh0iC7MOlgV3vw7rkA1KyYD/gQIIDKYTbsZ5/Rla5WWmXDyZa+6cN21iy1wV81QShN\nok8crFXRz4folA3pC0DBwGGulimzwWZVjT4MZk2dDQBaWe2D0xri8LXGevuYaar3+2iy4WTZ5s6F\njoVMOgkbs9ejYcBYziCyOGwsFYSIuSCy+DGVDgPG4rYcYWJhEFmu0LGi5FhtwsfC5ZxBZCY8Fhut\ndTZYrLNvVmzuFX1H63f2o5P5+BHL7nrz6JTDbDjXncFr4LrfvgbAq8Nec7V//sD2XX/5vT2RY3/1\nH9d24ImfBeDv3z7V1b5wySEA7Pe3Ga42Zj+7j+w/rnW1pQ02UHC3Mz8JwEXPzHVttasWATDpY/u5\n2h5x9/SfR95xtSTsKwk6Axi8/54AzFpn+5PZc9e5toZ1PggtUVo5BICBg8tdbVw/G0BWXLsKgMYl\nvh/csLgKgLVx3wRQm4kDu4LMseQ1W5QKQ8fiviAOhkweAYr72uOny/x5pMptH5ot9rVsoX2uDXUt\nrtYUH78uDhGrC8LEXOhYEBbZlCNgLBPvIxs8iWwUh41FOULHcgSGZXPU2vbNUbb974+Q20dGgWHy\n4WWnFt3WZ7Ht9YiLARERERGRLqUbiAF9TEhEREREJG9pZEBERERE8pJCx7bCyIAxJm2Med0Y82D8\n9RhjzIvGmDnGmL8ZY4o2tQqK1mgAACAASURBVA8REREREel6W+NjQucD7wZfXwNcF0XRTsA64Gtb\n4RxERERERAIGY7r+X0/TrRcDxpjhwFHArfHXBjgYuDte5Tbg2O48BxERERGRtkx8A3FX/+tpuntk\n4HrgIiCZw6w/sD6KomQutsXAsFwbioiIiIhI9+q2G4iNMZ8BVkZR9Kox5qAPsP2ZwJkApqg38/7z\nON+//n4Anl9xiFvvxR9+CoAH/tDL1c6/6h8APH7jqQDcdF2ta5ty9jUArL3d5wfc9JvTAfjrhQtc\n7fADRgAwbcYyV9uweDYAH/vyyQAMXPSCa/vbu2sA6Ffk58Yfc8QkAOY0+Lm3359t12uosvNm9+q/\ng2sbPLICgN2H9Ha17Fx7jLWzl7raknp7LVWf8fNEJ3NoDyy2xy8bVObaeo8cDEDBkJGu1pzkDKz2\nmQLLa+zyyg0NrtZQazMEmuptvkBzQ40/txbbtqlsgZy5AXG+QpKzELal2rTZ9ngu/4Jg3v4kDyCd\ntLXPDygq8PtN8gNaZQrkyCNomxuQytnmn186x7sAyXq58gjcczLtl1O031dYSfaRaxTSuGO2b8ul\nq4cyt+XQaNvvbU+y4Vd/A+DZCj/X/cqV9wIw7d47XM2suhWAsy4/HICr/3G3a1v3b5uBcsFZ/vV8\n+tSfA7DsgG+62vU3XQbAS2f+yNWSTJU+X/yGPY8r33Bt6aJSAA7b1/cdvPIgAO/O8bkByeth5ESf\nvVK0xxQA/rNoPQCrl25wbY3VawEoKPHz9vceZHNhJg6rcLXhfWwfkF77XwA2zPP9cfUy2xetbfL9\nYNgnuvOIXxBlQQ5JRaHtF4rj/Zf0LXVtSc5AqrfPe0lyBlqK/HqNcTZAU5AHUN3YOl+gPsgZqIvz\nBZpafK0xrrXKFMiRPdC2tqmcgaiDjIDOZsF0hc091tY8N5GEbiDu3tmE9geONsYcCZQAfYAbgL7G\nmIJ4dGA4sCTXxlEU3QLcApAqHxzlWkdERERERD64bvuYUBRFl0ZRNDyKotHAicCTURR9BXgKOC5e\n7RTgvu46BxERERGRXIyxo/td/a+n2RY5AxcDdxhjrgReB363Dc5BRERERPJcT/zjvattlYuBKIqm\nA9Pj5bnAPlvjuCIiIiIisnFKIBYRERGRvGPomR/r6WpbI3RMRERERES2QxoZEBEREZH8Y3TPAGhk\nQEREREQkb/WIkYEdRw/hummXUX7yywD89uG7XNsrT9jwnOOm/5+rnXKKDeoZfu9PAJgy0AeSnfyX\nGQD032myqw1/+c8ALG1ocbXdLz0NgLMeeNfVkoCcsw8ZZ9e/4/uu7f04nGvffj6UZuBhRwJw5/ur\nXW31vHmAD1cpHzzGte03bgAAoyp82FbDf20A0Or/rvX7aLLnGeTUUFFor+uS0LE+w31wWdkOg+KV\nBrna+jioZ2VtEDq23oaNVcfPBaApDjhrqbcBP9lm35Zp/uChY21r6SBgLFUYBw2FIWJxYFAYbJWK\n25NQsPDqvjhHmFg6x3o+nCw4VhIAliM4LFm/dZgY7WodSYLFNvVmRFe8WeHDzDZzuy0/dCudDkLr\n4uNuqa2RZXbMaVcB8Hj6VVcb98hjAEz62b9d7aafnA3AGYtsn1D+vO/zlnzrqwAcEoQNTn3ABnUV\n9+7nal8bb/vCH8xa42pH7W2DD5+vsX3Gsrf+49oqR08E4KTJw11t8dVXAL7PA9ihxAaXjfzkeFer\n6rsjAE88/hYAG5bNdW1Jn1FSMcAfa7DtXycFoWNDyu1+m2fYwMcN833oWM1S2ydVBcFeTdn2kTRJ\n6Fh58BovjcMhSyttIGRxpe8vk+VU777+fOOwsajQ/y5piDvgxhZ/zCRszIWONflzSwLI6oJaJpMj\nYCx+DlHwXHytfZhYrv63M7VcIWXZTfTlnT2WSE9h0MgA9JCLARERERGRrmQMFOhiQB8TEhERERHJ\nVxoZEBEREZG8o48JWRoZEBERERHJUxoZEBEREZH8YxQ6BroYEBEREZE8ZD8mpA/J6DsgIiIiIpKn\nNDIgIiIiInlJHxPqIRcDhYvnMezSk+kbB/Eccvdbru2vZ38ZgNvfqXS1vY8/AYDfX3QWAGf/6WzX\ndtaNDwNwyjknuNrzl3wNgN0qSlxt1e6fB2D+VVe72qCP7A/A5ybYoJxn7nzdtSVhN7vsO8zVWnY9\nGID7f/uKq9WsmA/4ALMBIwe7tn1H23CgohWzXG3Zm3MAWLmy1u8jCKhJVBbaEJ2KOHSoz0gfMJYe\nPBKATG9/rKpGGxSzMggOWlltA8jqq32tsaHZbttUHz82uDYXgNPJ0LF0YRAsVmDDhJKAsVQYOhYv\nJ0FjthaHiAXBQUnYWCper1VwWI4wsSSILNd66TBEzNVo9Qi5g8WSbYPTzVlLNs0VZJUEkW0q5CpX\niJjpgmSszvSF22N/2dmgt+3ZiD0PAuAvv7rH1R668EEAZl1/hKvddmMxAJ+/6kkAzjn3WNf252Nv\nBeCiP57paufe8TQAEw4+xNXW/PqHAKwNgq/2uOAYAI573AZ7NVStcm0f/bQ9/s4pH1L25D/temE/\ndNBA2+8M+MT+rvbCyjoA5r5vAxPr1ix1bUmfUDZwpKsN28GGfY0f4IPTCtctAqB2wfsAbFi8wbWt\nbrSBiLUZfx5JEGP4mi2NX4StQsfisLGSJHSsb7lrS8dhY+kgdCxbZNszBf53RGOdPX5D8H1IwsYa\n41pNEGSZBJCFQWRJ2FgmeA7Z+Em0rtnlXH1uR4FhudaLsu1/f+QSBpuJSNcyxhwB3ACkgVujKLq6\nTfupwE+BJXHpxiiKbo3bTgG+G9evjKLoti09nx5xMSAiIiIi0pWM2fojA8aYNHATcCiwGHjZGHN/\nFEXvtFn1b1EUndtm237A94G9gAh4Nd523Zacky4GRERERCTvGLbJbEL7AHOiKJoLYIy5AzgGaHsx\nkMvhwONRFK2Nt30cOAK4fUtOSDcQi4iIiIhsHcOARcHXi+NaW18wxrxpjLnbGDNiM7fdLLoYEBER\nEZG8lDamy/8BA4wxrwT/ztzUebTxADA6iqKPAo8DW3xfQEf0MSERERERka6zOoqivTbStgQYEXw9\nHH+jMABRFK0JvrwVuDbY9qA2207fkhMFjQyIiIiISB5KbiDu6n+b8DIwzhgzxhhTBJwI3N/6vMzQ\n4MujgXfj5UeBw4wxlcaYSuCwuLZFNDIgIiIiIrIVRFHUYow5F/tHfBqYFkXR28aYK4BXoii6HzjP\nGHM00AKsBU6Nt11rjPkR9oIC4IrkZuIt0SMuBlZXNXLr/bP5+1nTAFh8+QTX9trhYwGYcv0fXG3N\nY1cCcNEldo78N/Y4xbXVr7sYgGvi7QC+e/xqAL5+lh/R+fGTdl7rcG7sw086GgDzzF8AeHGRn/N6\nRKmdN3/c5/dztZeX2Xm2F85a7WqN1fb/rM/wnQEYu2M/1zYpzghoef01V1s1cxkAS+qD+arjeahL\ng8m0BxbbebvLd7DzYfcO8gsKdxgNQG1Rb1dbs96e27INPjdg2XqbJdBQ1+xqTXU236C5vgaAbIvP\nIOgoXyCZRzxczlVLF7TPGUgXFMS1IA+gIJ63P8weiJ9/Okd+QFFBvP9gHnqXKRBctSfbhPPVJ9u4\nHINWba33FbaH6+Ws0frdgk1mCrTZ16Ykp7TprIKumzlhU/vqzCQNPT8pYMvMvHg8AKuGHOVq0+59\nBoDnPv5zVzv1wSsAOPdc+wbS9y8Y7toujOe3n/+Jr7va2qumAnDXT09ytWf2PR+AyX39fPkcaWeu\ne/MM26+VVAx0TScfZPvJpif/5GqvL6kGWs/bP+IAey6pSQe52tMz7Cj3miW2/2uurXJtxb1tv1cx\neICrTR5ls2KG9/F9AfNsP1w1x46gb1hU7Z9fMm9/Ei4QKAp+8MriPqOi0J9vicsZsH1uSV/fN6bK\nbb6A6VXhalGxXS/MFGiI5/6vaQr65vj/oSbOQKgLMgXq4/Vagn20NNvlbFBLsgeirH9e2Xi5o5yB\nVrVM5zJgusMHOea2OE+RxLYIHYui6GHg4Ta1y4PlS4FLN7LtNGBaV55Pj7gYEBERERHpSsZAwfaY\nqLmV6Z4BEREREZE8pZEBEREREck72yh0bLujkQERERERkTylkQERERERyUsaGdDFgIiIiIjkoSRn\nIN/pY0IiIiIiInlKIwMiIiIikncMGhmAHnIxMGxUP6763kn89W82sOuXB5zr2k5eMgOA0q/80tXe\nPuFYAL5y8GgATr/hOdc2bsoxAKy+ZqqrZSIb6DLm4u+52sPffwuAsoEjXO2iT9mgsPcu+gEASxt8\n2MyxY21gTtnBX3C1u1+zgWVr573jaknYVt8Rdl9TJgxybUPTNghs/YwZrrb2PfucVwfBNomKQh/i\nNbCXDT2rGGkDc0pG+PPO9BkCwLoGH+yyrLoRgMVr612tttoGijXW+9CxljhsLNNk18s0f4DQsbRd\nThUUulqqsHXYWPI1QCoOCQoDxpLl1kFkrcPGCnKEibUOImtfc0FkQV9QGNcKc4SU+TAxv35ymq0D\nztioXP1OsmlX90kdDf3lPI9Orvdh14W5bJt03c6fBaDo0Udc7aLJ9nV25xFHuNqzdbsBMPbAYgCm\nH/sN1/alQ8YAcNotL7pa/50mA/DxOt+fnLfa9jHnnTHZ1abNsMGGq2fbQMvhex/m2j433oaCzf7J\nM662KO4fJvbxwWWjDtkDgMWm0tWenDkPgOqlc9o959L+O9hzHFruapOG9gGgX6rR1ZoXvAtA1fwV\nAKyPzx9gQxzO1ZRtHzpWGrwAk3C0Xn2KfXscOlYch40V9/MBY+mK/gBk46AxgGyRXa5v9MdqbLHL\ndc2+H0yWa+LfDfVhWxxA1hLUkrCxTCYIHcu0ryV9bTbuf8O+N9tBEFkoymZbteXaRy6596WQMJEP\nkx5xMSAiIiIi0qV0zwCgiwERERERyUPKGbB0A7GIiIiISJ7SyICIiIiI5CWNDGhkQEREREQkb2lk\nQERERETyjkLHLI0MiIiIiIjkKY0MiIiIiEjeUeiY1SMuBhalKjm/9PM8dqsN1nls/F9d2wGXPQbA\nz3/wFVebtvdvALh+hQ3KmXPEj1zbQ3dcBcADe3/L1Y4c1w+Af9b4ALAVb9ltP3LEsa62O4sA+NNj\nNkynNEiq2vnoCQAs6b2jqz03w4ad1a5a5GrFve2xhoyy4Tz7DfchPakFLwGw8nUf0rOoygbw1LT4\nAJqi+Ad3QJEP9uoz3Ibn9BkzFIDCoaNdW6a3fV7rq4LQsRq732VVPnSsPq6FoWPNDTZ0LNvSHD92\nHDqWhI2FoWOpXLWCopyP4MPE0kE4WBI2liuILGfAWNxWnCtgLHjhF6Zse2Gw31QqCRZr/dhqH51M\npQpXa7tJKmfEV9AebxBul5ylCYqd6cfM1kzR2gpSH6LnM7DYvi7O/t+fu9rqqfZ1/N7nd3G1T/7k\nNgBeuuMSAK775fmu7cZH/gTAW1+4ydX+56L/AeD1i3zAYhLANe6ii1zthGk22Ct5PR9wwGjX1m++\n7cMefdb3YZk4d2vCTr7v6nPQUQA8sLDK1ZbPXw9A/TobGJYuKnVtvQePAmDS6H6utsuAXgAUrPH9\n35o58wFYP9/ud1Wj73PCPjGR9I1lweu5otAuJ0FjACXxckl/G3SWioPGAFK9bXBjS3FvV2uOX3mN\nGR/+mASMVTf5c6qNa/VNSfiY70uTWqbFB5e5gLGglgSRRdmw1jpsLFfAWJTZ8nCwXPvoLgouk+2C\nPiYE6GNCIiIiIiJ5q0eMDIiIiIiIdCWD6fRI/4eZRgZERERERPKURgZEREREJC99mO5B+6B0MSAi\nIiIieccAaV0L6GNCIiIiIiL5SiMDIiIiIpJ/jJ9OPJ9pZEBEREREJE/1iJGBtctX8tef3sg3b7fB\nN996+Aeu7YcX3A/AFw+c7WqNFTZQ5vRHVwLQZ/jOru3AlU8BcHdVg6ud+sdv2LY733C1JCDr7M9O\ncLUVt10NwBvxtrtV+BCb4cfZcLLfv7fG1ZbNXgC0DurqM8yey8d3tUFgu/Qvdm0Nz7wAwKp3Vrva\n0gYbcpPx+TMuRGdIif/v6zu6wu5/tA0rMgOGu7aqFrv+smofMLZkrV1eE3wfGmptQE5zrQ8OyjTa\n9TJtQm82JmfoWM5gsbhWWBR/3T5MLBV8kC+ppQt8raCgdbBYUYE/Zq6AsVzhZMkhUq1CvFpvG36e\nMNXB+jlrtA8Hy3Wvkmtr39RpW3IP1OZu2lGIWWffZNF7MdaJi14B4OpLn3C1K065AIBzlr/pauZ4\n2/+U3XQhAPv28yFeP3jDBlWZtH8NXHP4WLvfLy9wtSM+MhCAV4t8mNn8l2zYWeXoiQB88xNjXdvS\n//sVADM3NLraDnG/M/bQ8a5WO3wyAP+8e6arrV34PuD7v7KBI1xbv6E20GvyqL6uNqy8EICWmX4f\n62bbPn/D4mq7zyDgqz7TPnQsCYIsD17jFfH5loShY/0rWj2GoWNRqQ0ii4rL/bGa7bEagnCwmvhc\n6puDILS4v65utI91wfm2xOu1BOtn4oCxbBCglo3DxjItPuAsCQPLFTqWS672tttmt2AfnV1fZHtn\n7xnQb6MecTEgIiIiItLVNJuQPiYkIiIiIpK3NDIgIiIiInlHU4taGhkQEREREclTGhkQERERkfxj\njKYWRSMDIiIiIiJ5SyMDIiIiIpJ3DJpNCHrIxUDfwQM59Pyv89vTjgeg9gQ/R/YnThsJwC9PPd3V\npv792wB8+9p/AHDWBV9xbU+f8VUAJvf1c06v3N9uO/v6q1xtyG5TADhp4gBXm/4lmwNQH0/6v/tB\nI11by+SjAbjj5pdcbf2idwEoKPHzVQ8eOwyAKePsfktXznJt8196B4Bl8Zza4OfVDm9wqSy0c4n3\nG1zmahWjBwNQOHwnADIVQ13buka7jyXVPlNg8bo6+1yqfQZCY4PNGWhpqHG1TJPdJtu8eTkD6UKf\nKZAqsPOHp1rVWmcPJDkCtpZkCgR5AAVJ9oCvtc0NCPMDinPUkuVwTuFUjiyBttkDYUeRdhkEdFgz\nHWUKxDPsb6r/SdpTrWqb7rQ2NdzXmRHR7XHUdGt02Nvid8K4s+4C4P1fHuFqf/2zzR+Z8p1HXe2y\n750KwM2HHw7Ad+4837WN/M2DAHz0yM+42uprpgKwvMHPV7/vD21f+Nn733a1ujVLAZh0+KcA2K1g\nlWt74s4ZAFQ1+3nwDxxs+7MhRx7uas8utX3GO//129aushkBSZ9QPniMaxsb5wvsNriPq5WsXwhA\nzZz/utr6eesAWFVn+6YNLcEc/fGU/+FrtzR+ESZZLAClcb5Ar/4+l6Gkvz1uQd9+dh+9fd5Btsg+\nv0yB/x1RX2e/h7VBbkCSJVDd5L+/Sc5AfZJB0NQ+UyAT5CNk4yfRumaXw762o4yAXNkDvtY+iyGX\nJMdAJB/pBmJ9TEhEREREJG/1iJEBEREREZGupI8JWRoZEBERERHJUxoZEBEREZH8YyC9Pd4kt5Xp\nYkBERERE8o4+JmTpY0IiIiIiInlKIwMiIiIikpc0tahGBkRERERE8laPGBkYbTbwu8JHWHXhJwAY\nf93Nrq3u3vMA+N63fbjKXYOOBKCl6ecA/HhvHx4zddYaAL7746Nc7cy/vwVA/brlrnb0p22wWea+\n61ztmcUbANi53AZl7fzlQ13bE/PWAzBv5jJXa66tAqDv6ImuNmnCIAD2GGqDbZr+/ZxrW/7qEruP\n2mZXa8raUJryIDxrSIkN8akc64NyKnayYWYFO4wFYEOBDzpbvsYGjC1eV+9qy9bbMLG6miB0rNo+\nv+Z6HzrW0mS3ybb49dpKQoXC5bDmgsUK2oeOpQsK4q/98yuIA4MKCtPtaulcIWKpOGAsRyBZeGNQ\nshx+PrAw3rYgDD0zrYPICoO3DXIFkeWs0f6thrYfSwzvWTKu1rm3KFpt28EmnQkp66xN7asz92Dp\nDZj2albMA+Bv46a42klv3A3AWcdc72rnHV8LwE/i/4fpO5/g2qoWXgrAH3/7dVd7YOdzADhkkA8n\nXH/g1wCY+bVbXa1s4AgAph5h+7zqe3/p2l5aagMQwxCvHQ+3fUy022Gu9uBTNrhs1bzFrpb0f6WV\nQwDoP2KQa/t4HLo4uq/vE6J3bEjjutmL/PkusPtYFQcnJoGPodLgtdsnft33K/J9R+mA0vixt6v1\nGlQJQKpPf1so7+/Po8SuV9/if6cky60CxuJAsZog1K06Xq6OAxybgtCxlji4rSVXEFlwrGzc5ydB\nj+D735wBY5n2tQ8q1z42td/NPW5XnKdIVzEY3TNAD7kYEBERERHpUppNCNDHhEREREREthpjzBHG\nmFnGmDnGmEtytP+vMeYdY8ybxph/GWNGBW0ZY8yM+N/9XXE+GhkQERERkbxjpxbdysc0Jg3cBBwK\nLAZeNsbcH0XRO8FqrwN7RVFUZ4w5G7gWSD4bWh9F0e5deU4aGRARERER2Tr2AeZEUTQ3iqIm4A7g\nmHCFKIqeiqKoLv7yBWB4d56QRgZEREREJC+lt/4NxMOARcHXi4GPdbD+14B/Bl+XGGNeAVqAq6Mo\nundLT0gXAyIiIiKSd7oxgXhA/Ad74pYoim7Z3J0YY04C9gI+GZRHRVG0xBgzFnjSGPNWFEXvb8nJ\n6mJARERERKTrrI6iaK+NtC0BRgRfD49rrRhjDgEuAz4ZRVFjUo+iaEn8ONcYMx3YA9iiiwHdMyAi\nIiIi+cdAOtX1/zbhZWCcMWaMMaYIOBFoNSuQMWYP4Gbg6CiKVgb1SmNMcbw8ANgfCG88/kB6xMjA\n4nmrufjkaez7yrMA7JR93bXds9eJAHzzB0e42h4//jsAB3z5OADeOP1M17ZDSSEAFVN/7mr/Od4G\n+wzYeW9Xu/RgG6zz6mEPu9raOCzmqL13AKDgU191bbf9wwYHrZv7hqslwVqDdtzZ1Y74yGB7rDob\n0rPshddc28o5a+0+mtuHslQGAVxDBvayz2HMAFcrGbUjAC0V9txW1/t9LNlgA8YWrKlztdoN9iKz\nodYH27Q02LCxTJMPJ0uCbzoKimkVOpa2y6mCQldLFRbFtaJ2tSRYLAwYS4LFwoCxVPzqKg3ChIrj\n9iRgrDhYP5k3uCgMKYv3EYaIJYthHHlhvJ4PHwuDy9hoLddIY65ZCjo7Irm5V+odrZ/zPDq53ofd\nts6beenP3wLgL6P+4mqf+KMNQDzsrNNdbdrnTwPgrMsPB2C/n013bTsedCwAO/zrBld7o8q+7q+8\n4QuudvEjswFYG/RTkz5zPACfGWXDGV884xnXtjQO0ZoS9zkAo44+GIB3q/1P3HNv2LDF6mXt35xK\nQs12GOlDEicPrQCgT8NqV2uc86Y9t1k+/HHNOvsckj4xCWEE/5otDV68SThar36lrtZrgD330oGV\nrlbY155LunIgANkSH0iWKbaBjbV1PkysIQ4FqwkCw2riALIwdKym0S7Xx+u1NIehY3Y5GwSnZTJ2\nv1Hka20DxsLlbK7QsZxBYdlN7yOj8C+RbSGKohZjzLnAo0AamBZF0dvGmCuAV6Iouh/4KVAO3BUH\nfi6MouhoYAJwszEmi/21f3WbWYg+kB5xMSAiIiIi0pW68Z6BDkVR9DDwcJva5cHyIRvZ7nlgUlef\njz4mJCIiIiKSpzQyICIiIiJ5yGyLqUW3O912MWCMKQGeAYrj49wdRdH3jTFjsAEL/YFXgZPj0AUR\nERERka1iW31MaHvTnR8TagQOjqJoN2B34AhjzL7ANcB1URTtBKzDhimIiIiIiMhW1m0XA5FVE39Z\nGP+LgIOBu+P6bcCx3XUOIiIiIiI5bZupRbc73XrKxpi0MWYGsBJ4HBuKsD6KomQutsXYWGYRERER\nEdnKuvUG4iiKMsDuxpi+wD3A+M5ua4w5EzgTYEhJMadN2ZFPnmezAdY+eoVb79s32Lnzd/iSr63/\nx2UA3H+KnaP7kvPmurZzvm6zBC58aJarbVhs594+7oKzXa3vf+yc39PfdFkPjCi1c+dP+toUAP6z\nxl9LvfmqzQ2oX+fnyO4z3OYL7LLrQFf7+Ag7v3bLK48AsPRFf25zapoBqInntAY/h/awUv9fVTnG\nzpFdufNIVyscaY9V26s/ACtX+kyBhettbsDitb5WF+cMNNbWuFpTbRUAmaYGV+tMvkCYH5AuaJ8p\nkNTSRX7u74Ki4ni9JFMgmLc/mec/zANokylgl+3xk/yAdDBJflGO9XPlBqTibcLPDLrsgRxtyXLO\nWjBzf9IcfhQx1WZm//CrZB+5PrpoWh2r/X47Wr8rdLS/zuYSbMkZfdg/z7lgip23/ztPXuNqV33D\n9j8Nd3/d1S6ssa/Ztaf+BIClR051bf/6+08BeHDfS1xtcl+bG9Drf37sao+eYfdbUuH7pDM/MwGA\nlof+D4DnZ65ybeXx62f8QaNcrWA/O5h7z2u+r1v6ng3PbKjy2xb37gdAvxE2Z2D/cT4XZad+9tzM\nvJddbe3MJKtlvastj+fwD/vERFH8w9enwGeP9ItzSHoN8H1N2SCbIVDav8LV0pWDAEjF59gS5AzU\nN9tj1QfHrErOo7F9pkB1kDNQH2cPNLqcAb+PbLy/TEvHtaiDLIEoR0ZArvW72+Yea2uem8jm0D0D\n1lYZzIiiaD3wFLAf0NcYk/xlmzOCOd7mliiK9oqiaK/KoqJcq4iIiIiIfGDGdP2/nqbbLgaMMQPj\nEQGMMaXAocC72IuC4+LVTgHu665zEBERERGRjevOjwkNBW4zxqSxFx13RlH0oDHmHeAOY8yVwOvA\n77rxHEREREREcmr7Ed581G0XA1EUvQnskaM+F9inu44rIiIiIiKdowRiEREREck7hp75Gf+uposB\nEREREclLnZ0V78OsB0YjiIiIiIhIV9DIgIiIiIjknx46FWhX6xEXAy0jx7L+hjsY9NOnAXjyIwe4\ntgsusMufuOxeV9v7ojjIpQAAIABJREFU+BMAmH2Gfawo9KE0I35yCwD3nHqbq/UdPRGAnxw1wdVe\n+/yFACwNAmW+vM8OAJR+5gwAbv7nPNe2atarQOuwrYE77QrAsbv7kOWhzTaUZ8VzL9jHIOBnRRBo\nkxhQZP+Lhvb3ITqV42xgUOmO41wtU2mDfVbX2X0srPLBYXNX1QKwYb2v1VXbAKPmOGgMINNkw8la\nGutdLdvS1O6cEknomEn772+qwAazpQqLglpR+1ocFFYQ/98UBP9HBUXta0l4WHGr0LFUzsdWtSAX\nvDBOEysMxgST5cJ0+3CypC2MFk8Cy8Jaro4k17Bjsl5XD0l2NLyX8zw6ud6H3fbyC+CR2WsA+MnL\ng1xtv5O+CsBfPnaKq5031fZ1R1z1JAAj9/uMa9vrnTsAuGCVDxa8/EefBuC7j81xtZXvPAfAhMOP\nc7VTJtkwsFcvfBSA+XXNrm3/uN8Z98Uprja7xYYePvbqe662Ycnsds+rfPBoAIaMtuvvN6rStfVv\nWQdA0+zXXW3NuzZyZuXKWldb12zDqpqyEeADAcEHolUU+ldAeWUctBaEjpUOtMctHuRDz5LQsWyp\nDSLLFvvQsbp6e8z6IDCsKgkYa/LhWVXx9ykMIksCyJriWjYIE0sCyDIZX0uWw34229zUvtY2dCxH\niFeU7Ti4LJsjsKzt+puqiciHU4+4GBARERER6UoGo6lF0T0DIiIiIiJ5SyMDIiIiIpKXtpePjG5L\nuhgQERERkbyUj/fMtaWPCYmIiIiI5CmNDIiIiIhIXtLAgEYGRERERETylkYGRERERCTvGHy2UD7r\nERcD789bxjGnXcWax68C4KLyb7u2ym/eAMCqz1/sanN+eyIA3634LwBnnTHZtU19dAEAa+e+4Wqf\nO//rAAx+9W+udttLSwEYUVroart//RAAXqguA+DlFxe7tro1dv0+w3d2tV0nDQbgwFF9XS37+l0A\nLHneBgHNqvbBMjVxQE1pkKwzrNT+F1WO9fvoN340AEWjx7tabZkNIlu20oYOzV/rw4cWrLYhPnUb\nGl2tsbYGgKZWoWM2lKyjsJkkaAx8mFi6oH3AWEGRD/1Jx8sFRcWulgSLpQviEK8w9Ct+/ukgRKw0\nXr+oIAgiS7cOIssVOha+yJPAsFRwt1DSHoYZpeP2pC3cR85aPMgY9icuYCzHAGRSCffh1w/3kRyr\n/X5zMV3YoW1qX5294eqDnlE+dc5XPWn7tfLzfu9qDX//BgAXXuyDAvf81k0AzPv0VAAev/ta13b/\nfgcCMLlviav1vfA6AO468w5XK+7dD4BvfO4jrpZ94JcA/Pv15UDr/uejB48GoPCgE13t7zOWAbBk\n9lJXq1+3vNX+AfqPsttOmWADviYM6OXaUgts2NjqGT6sbM17NohseRD0WBUEfwEUBT94feK+YGCx\n7xPKBtm+uXxohav1GmRDx5KgMYBUhQ0ga4lDx+paItdWG/fDVcF5JMFiGxp8IFsSMFYT1BrjULIk\nYKyl2felmZYkYCzbrhb2ubkCw6I2gWG52raGzT2WgsukJ8ijXzcbpY8JiYiIiIjkqR4xMiAiIiIi\n0tX0rri+ByIiIiIieUsjAyIiIiKSd4zp2vvteipdDIiIiIhIXlICsT4mJCIiIiKStzQyICIiIiJ5\nSZ8S2oyLAWPMx4HR4TZRFP2xG86pncKy3ozY8yD+OWYvAC6+/DDXtte3/grAAad81dXePuFYAPrE\nc82P+MWfXNvdJ94IQP+dfPbAL46xc26/fOQFrrY0nkP65P1HuFrJsXbu7+vvsRkBK9992bUl8+sP\nHufn7/7insMBGNa0zNWWTX/W7v+tlQCsaPRzWScGFPn/lqH97Rz9AyYMcbVe43YBoKXfKFdbVWf3\nM39dPQBzV9W6tg3r7VzlddU+Z6A5zhfINNW7WkujXc62+OyDtsKcAZO2y6kCn8WQLi6Na0H2QGGc\nPVDot02W3WNR+7YwN6CjLIG2j+AzCAqDOdML47HAwmBMsDDdPo8gaU+iD9JhVkFcy9V55BpqDNfr\nyqHIjob0cp5HJ9f7sNseO/2Dn7d5JFP+52uuduueJwHwre/5vu6Ayx8DYOyBxwCw9yu/dW3/u8rm\nivzoF59ztQsfmgXAipnPuNpHjvwiAKfu6ufhf/G8BwCYX2fnyz8wyAPY+cv2+O82+/UfftFmA6yf\nP7Pdc+k9dEe3PGysnd//gNE2e2BA8xrX1jDzBQBWzfRZLStX2j5rdZOfm74pa+f/T17G5cFrvF+R\nXS6v9NkKZYNtzkDpwEpXKx5kMwXS/X0fmo3zBbLFvQGoq/fHrI8zAqqCvrk6PqeqOp8pkGQPVAd5\nBE1xLckXaAlyEjKZbKtH8H1ttrmpfW0T2QO+beNZBa32kcm02S7XvpQLIJKPOnUxYIz5E7AjMANI\neosI2CoXAyIiIiIiXcmgz8tD50cG9gJ2jaIo2uSaIiIiIiLSI3T2YmAmMARYtqkVRURERER6Ak0t\n2vmLgQHAO8aYlwD3wfMoio7ulrMSEREREelOJj/vn2ursxcDP+jOkxARERERka2vUxcDURQ9bYwZ\nDOwdl16Komhl952WiIiIiEj30sBAJ2+iNsYcD7wEfBE4HnjRGHNcd56YiIiIiIh0r85+TOgyYO9k\nNMAYMxB4Ari7u05MRERERKS7GHTPAHT+YiDV5mNBa9iKU7N+ZEgJL148nst+bUOxGo+93LVVPXIN\nAI+ccJSrTZ36HgDf+f7hAJz4lzf8+gvfBeDM7051tconbgLgsdeXu9qOZTYoa4/zP+tqjy23PzGv\nvbAIgPp1fv2+oycCMHnyUFf75Ki+ADQ/e6+rLfq3PbdZ1TZYpqbFB9AkgToje/n/lgG79Aeg/8Sx\nrlY4xgabbSjp5/e7woYOzVltg3vmrqxxbTVx6FjDhipXa4pDx5obfDhZZ8LGwjCxdLycLirtsFZQ\nVBxv639kCgpT8WO61dd2H3a5NAgiKyqwy6VBcFmvotbhZGHoWBIiVpgKjpkjYCwJMwrDyZL2to+t\nasHAYtLcKmCMZFtfSxaTfbReP9lHeKz2+82lK2dC6Ghfne0wt+RsUnk4q8PLd9rgxOrrP+VqP6q3\n4VbvfvlKV1ty/MUAvPnI9QD8efzFrm3KQBsUljndr3/P6bcC0Kv/Dq526Qm7AVD356tdbfqbtmuv\niF+De3xmJ9eWnnIyAH+YvsjVFr09F4CGqlWuVlIxEICBY3wQ4uGTbMjXrvG5Met517bq/9u77zi7\nqzr/46/P9EkhnRQSCCEJLaFJU5QOioJYEbGg4rLuzwIWFHR1V1kUXSm66ior1gURYRFUlCZVpYRO\n6IQA6XWSTKbPnN8f53zP99yZOyXJTGYm9/18POZxv/dzzv3e78ydezLfnO8970f9OLjmuTyIbGmj\nD+xKx8RMbXjvjkve/5Oq/Tg5auqoWBs11YeJjZo+Kf8eJvgx2XaaGGvtI3wo2eY2v1p2fRIOti78\n7DckYWLrG/zYWJeEjtWFWmPSLwsZawshZe3J91KsloWNpWNvsYCxLDCsp/CxvtrSsLEtfS4Fl8lw\no9WE+n4y8BczuwX4Tbj/PuDmgTkkERERERHZHvr6AeLzzOzdwBGhdIVz7oaBOywRERERkYGjy4S8\nvs4M4Jy7Hrh+AI9FRERERES2ox5PBszsPufcG81sE+DSJsA553Ya0KMTERERERkgmhjo5UPAzrk3\nhtvRzrmdkq/ROhEQERERkeHLKLP+/+r1Wc3eYmbPmdmLZnZ+kfZqM/ttaH/AzGYmbReE+nNm9ub+\n+Cn0NWfg132piYiIiIhIcWZWDvwQOAnYB3i/me3TqdtZwHrn3GzgMuDb4bH7AKcD+wJvAX4U9rdN\n+ro86L7pHTOrAF63rU8uIiIiIjIozC/f3d9fvTgUeNE5t8g51wJcA5zaqc+pwC/D9nXAcebXQD0V\nuMY51+ycexl4Mexvm/R4MhCmIjYB+5nZxvC1CVgJ3LitTy4iIiIisoOZaGYLkq+zk7ZdgNeS+0tC\njWJ9nHNtwAZgQh8fu8V6/ACxc+5bwLfM7FvOuQu29cm21sqnXuSyuafw9Zv8ZVWTP/vD2PapL/8z\nAHcd/Z5Y239MDQBtn7oEgDvf+83YtsshbwXgu2+dE2t3zTsTgNXNeVjKu96yBwB2ymdi7ZIfPwjA\nqqf/BkBFTR52M2Oen+E54+AZsbZznQ/WWXzbvbG2+Ok1ACwLQTVJ1hWTQ4jOLtNGx9qkedMBqJ07\nL9baJswEYGVDHnbz0nofOvbCyk0AbFrXGNsa65sBaG3IQ8faQthYFnoDPYfFZGFj5ZVJ6FiV/zmX\nFdR82Fh5dRI6FoPF8pmsLFgsuy0rz89Ls7CxEUnoWFarToLFqjqFk1Ul+8hCxNKfbx4w1jWcrDwN\nIiujoJZ0Lx4wFraLBZEVY7F/WuvbR5h6Onsvtjxasb0OtWXUtkfQ2FDOlPn6d78IwIUnvCXWvnzt\nOQDs9sX8aszXvfcDAIz+8XkAPBLCBAEuu/FzAJzy60djbd0iH7Z42BkfirV3TfLv+9u/e0usZWPR\nyVP9uDPrzPfFtvtW+3Uj/vrAq7FW95oPbsyCCAHGzNgbgL32yoO9jprpAxPH1ft/t+ofvT+2rXp8\nCQDL1+bj1JoWfxwtHflaFdl7NgtkHJ+MCaNCmNnoJHRsxBT/nFWTdo61iok+/KwjBI0BtFX5x9SH\nMXRTMvZnYWMbmvOAsQ0hbKy+OR9z60O/ttb8sdl2R7vr0tbR7sPGegsY6ygWOtbROXSso9u2gn20\nKwBMpCfmHOZc7x233Brn3MEDseOB0NtqQns5554FfmdmB3Vud849MmBHJiIiIiKyY1kKzEjuTw+1\nYn2WhEvzxwBr+/jYLdZbzsDngLOBS4q0OeDYbT0AEREREZFB4Tp679O/HgLmmNnu+D/kTwfO6NTn\nJuBM4B/Ae4C/Ouecmd0EXG1mlwLTgDnAg9t6QL1dJnR2uD1mW59IRERERGQose18MuCcazOzTwG3\nAOXAz5xzC83sG8AC59xNwJXAr83sRWAd/oSB0O9a4GmgDfikc26brwfsUwKxmb0X+ItzbpOZ/Stw\nEHChc+7RXh4qIiIiIiKBc+5m4OZOta8l203Ae7t57EXARf15PH1dWvSr4UTgjcDx+DOWH/fngYiI\niIiIbD/OXybU31/DTF9PBrIpiLcBVzjn/gRU9dBfRERERESGuD5dJgQsNbOfACcA3zazavp+IiEi\nIiIiMvQMzNKiw0pfTwZOw8cef9c5V2dmU4HzBu6wREREREQGkHPD8rKe/tankwHnXIOZvQS82cze\nDNzrnLt1YA8tV2nGpOpyPrBsfwCqx9THtgtHPgzAOc+ujbXL//xlAI64/D4AWjbnYVv//s+HAbD2\nO+fG2s2L63z/CXlQ1rwL/h8AP31sRaw994+FALSG/e28zxGx7cTDdwXgqN3GxFrD7/4bgFfuXBRr\nz9f7wJksWCcN0dl9ZKXf7/xJsTbxgLkAlM/cN9bW2kgAFq3Lfw4vrPTbr632oUL1SSBR04b1/jmT\nn0Nbiw/7SQNwisnCxrKAoex+ul1Rlf/csrCxiqrKWMtDx8q6rVUlP4csbKyqIq9lgWJVRULHstvK\nJE2rsszXKgqCyELQWRJAlYWTpcFX2XbnW8iDxdLgrmKBVlm7FdR6T74qtt9iU3A9hZRtTb5WT/vr\na0jZEM71GtJO/8t/APD30dWx9hXnV21uXPuLWLvrM2cB8B8TfZDlGYdOi22P7ucDyR66PF8FesJs\nHw3zX2ccGGuvXfJpAG5/NR8LZtT69+r+H/OJ9k2HvCu2/eR3TwKwdOHCWMvGv1GTZ8batDk+APOt\n86fG2t4TfShh+z8eAGDlgmdj25owXi9tTEK82rr+g5yFjY0L48WkEfm4MmqaDw4bOXV8XtvFj53l\nk5JAzjE+gCwNHatvaQ+3/jnXNeYBYxtDsFgWNAZQF9rrGvLxsjn0a00Cy9rCfrPb9uR7am/z/dOg\nx2z8LRow1t61Vux+tt3RQ2hkb/voS//ebGl/ERk6+nSpj5mdA1wF7By+/tfMPj2QByYiIiIiMpDM\ndfT713DT18uEzgIOc85tBjCzb+ODEP5roA5MREREREQGVl9PBox8RSHCtq4KEBEREZHhaxj+T35/\n6+vJwM+BB8zshnD/HfisARERERGRYUgfIIa+f4D4UjO7C3hjKH1U6cMiIiIiIsNbjycDZlYDfAKY\nDTwJ/Mg519bTY0REREREhjyHZgbofTWhXwIH408ETgK+O+BHJCIiIiIi20Vvlwnt45ybD2BmVwIP\nDvwhdTV+v705/b57OfsIv5rpgj9dFtu+P/cQAD5weL6u9O92fisAT938dQAOfOf7YtuHxi4H4CeX\n3R1rtWGt+aM/8YZYe2XWcaHfvbG2btHjAIyY4Nf3nn3w7Nh2xkH++aueviPWFv7Rr6/99Cv5mt6r\nw9rUVWHx9myNb4Cpc/x62ZP2nxVrVXP9GuGtE/LasvV+zevn126OtWeWbwRg0zqfH7B5Y0Nsa65f\nB0BbU96/vdn3K7Y2dJYpkG5XhPyA8qqa2FYea0nOQNjOcgQAKkJuQEWSJZDnDPjb2qQtyw2oreya\nPVCsX5YpUFmQKWChrWt+QNqvPNa6ZhRk3ay3TIHwWfqCfp2eM20vi/fTY+u632KK5QH05bF93f/2\n1JfchW0xwLvvNxd/x49F369bEGsfO/5fAfjqxZ+LtQdP8ONa9jt7xHU/iW37XfZ3AJo3rYu1D33y\ndAD2eTUfk379s4cAaGzP/yfsnYf5sWvaWZ8E4OcLV8W2hx9cAsCmZS/FWkWNX99/wh7zY+2I/X2+\nwBG7jo21mmVPALDygZAz8Hie2fLyxmYA1rTkE83tIQS0Kvll3SlkjUyp8bejQ7YAwE7Td/K1XXeO\ntcqd/dhckeQMtNX6fIGmsjzHYWNTyBJobg23+XGsDVkCa+vzPIANoVbflPfL8gXaWpOcgdaOglra\nlmUKpNku2fjbntSyHIKi2QMdffsfzDSjYKApX0CGNwd9fF/tyHqbGYipK7o8SERERERkx9LbzMD+\nZrYxbBtQG+4b4JxzOw3o0YmIiIiIDJDhGBLW33o8GXDOlffULiIiIiIybOlkoNfLhEREREREZAfV\n19AxEREREZEdh3P+q8RpZkBEREREpERpZkBERERESpM+M6CTAREREREpTVpNaJicDDz5yjpm/9M1\nHHbGhwAoO++M2LY5hOccessfYu29H/01AGN23RuA3/7L4bHtgfe+DYDHNzTF2hmH+qCaKef8W6x9\n7JbnAFj84N+7HM/U+YcCcNZRe8TaXmU+7GflH2+KtVfufc3voyHGNcRgnWk1/kc/a+cRsW3K63xQ\nzpgDDog1N30fAJY35L+sz67x4WELl26MtZWrfG1jCB1r3rA6trVu9qFnrY31sZYG33SWho6VVVb5\n24rKgvsAFSFgLAsfgzxErLI6/9WqrPa1qiK16hAiNiIJE6ut8v2yULF0O61loWTFAsay7YIgsrIi\n/cJjy5OEquwh5aFfGV3DwYoFkfVHsFe6356u4Sv2XMWevqdjKhZg1pfH9facg224hI1lvniuDzuc\n/6/3xdr0Q04E4HONt8XaZ+9fCsDXLjwJgPMeyb/RF+68AYA9T3h3rH33rXMAeOiET8baQ+v9uHfE\nhPw9e+A5pwDwbI0PUbzy9odi2+pnfRBaOl6MmzkPgN32nhRrJ+87GYBdLQ9YbFrgw85WPPACAEsW\n1cW2lSHkq7E9v1Y3y/0bU5n/5k8K48T4cLw7TR8d20bP8M85YpepsVYxeVd/vCMnxFq2vakxD8eq\nb/Hba8LYvL4xH6PXhbCxDUmtLvRrTmptYR9Z+BjkIWPt4d+ljiJhYmktCxsrHjCW1jq6bevIakWC\nxooFgvUUEqYAMZHSNCxOBkRERERE+pfTZULoA8QiIiIiIiVLMwMiIiIiUpo0M6CTAREREREpQU6X\nCYEuExIRERERKVmaGRARERGRkmNoaVHQzICIiIiISMnSzICIiIiIlKYOzQwMi5OBjrZWGtYu5c4T\nZwDwqS8ujG2XXP1xAI790WOxVvfqMwB889LzABjxq6/GtutDENhBY2ti7bBvfQKAq1/NQ3zu/PPj\nADSsXRZrE+ceAsCJR+4OwClzx8e21j99D4AX/vBErD1W5wN+6tvyX7QsUGf2KB/iNeWAybFt8qE+\nYKxy7uvy76VmIgAvrdgca08t82FjLyzLQ8c2hbCxxjoffta8aV1+bE3+sT0FjUEeNlZWUSRYLNxW\n1ozK22pHhVr+s8yCxSqS4KAsiCytVYWQsdp4m/8qZmFihUFkIbisoliImK9VpAFjYbs8CaDKAsbK\nLA0R66EWIrWKBowVCyLLu8V9FAsRs/icXfdbTE/hYFtjMMPGyoZbIth28Pt3XQjAq1+4NNaW3/Gf\nAFy68/6x9s45fryp+/i3AfjFRy6PbVnA4o8+kQcsrv3OuQD86aF8DJsSwg6P+NCBsWanfMY/1+/9\nuPnyw0/Ftsb1KwAYMWFarO2yjw8zO/3QGbF2yLQwLjx8R6wtu/dRAFY8ttLvd3MS7NXSNdyqNrxn\nJyZjwbRaP07uNH0n/33uvnNsG72rHzsrpuyafy/jfQBZWxI6trGlo+AWuoaNralvjm1rN/txsq4h\nHy8bm3xIWhow1hKC09qS7yXbbmvx+82CxiAff4sGjLX3HDrWua03Ax02pnAykR3LsDgZEBERERHp\nX86vKFTidDIgIiIiIqXHoaVF0QeIRURERERK1oCdDJjZDDO708yeNrOFZnZOqI83s9vM7IVwO26g\njkFEREREpDvmOvr9a7gZyJmBNuDzzrl9gMOBT5rZPsD5wB3OuTnAHeG+iIiIiIhsZwP2mQHn3HJg\nedjeZGbPALsApwJHh26/BO4CvjRQxyEiIiIi0pXTZwbYTh8gNrOZwIHAA8DkcKIAsAKY3M3DRERE\nREQGjk4GBv5kwMxGAdcD5zrnNqZrnDvnnJkVXdPJzM4GzgaYNn0Gd/3qs3x75kEAfPT43WO/X+3+\nQQAe+s43Yu2wMz4EwDkTfKbAD/7t5thWFRZSP/nzx8Tay/u+A4DvXnJvrK1+9n4ARk7K19Le+4h5\nAPzz63fzbQtvjW0Lr70LgKdeyNf3XxnWoa5KFm+fOcKv4T9jb58fMPXQubGtZp5fI7x18p6xtniN\nX//6iRV5psDjr9UBULc6zx6or/PbTRtX+3005P3bm30GQbG1obNsAcjzBSqqa2OtvMpnCJRXF+YN\npNtZjgBAZU3IA6jOf7Uqq8u79MtyA0bX+HXEa5O2EZ0yCCDPF6ipSJ4rrEue5QdUJj/nmEGQZA+U\nmxX09/3KutSyX9HsNtlF8eyBrM267iO9Dq9YvkBnxfoXtBd5bOdS0T6DvL7/QOcLDOf4ggvOvRiA\nS37w5Vh75A1HA9CeLHl37D2/A2DeRX8FoHH9yth27tc+CcChi/8Uaz+77G4ANrTm7/sPHu3Hrt3O\nzSdjr3jMZwncddciANYvznMGKkKuyOS9D86P4+DpABw3K89ZGbnkEQBW3H13rC2534+/L633eStr\nWtpiW3v4ttKxcWJ4v0+pyd/jY3bz+QJjd/cfLRu9a54zULWL/14qJuc5A22jJwHQWJZnn2xs9N//\n2oY852BNyBBYm93W53kAG0KtLqm1xJyB/HvIMgfakp9vtp1lCqTZLtl2e1pr7SF7IAlC6jx2d6T9\n27duzf+tyQpQvoDIjmlAVxMys0r8icBVzrn/C+WVZjY1tE8FVhV7rHPuCufcwc65g8dPmDiQhyki\nIiIipcY56Gjv/69hZiBXEzLgSuAZ59ylSdNNwJlh+0zgxoE6BhERERER6d5AXiZ0BPAh4EkzeyzU\nvgxcDFxrZmcBrwCnDeAxiIiIiIgUlV6SV6oGcjWh++h6OXPmuIF6XhERERGR3rlheVlPf1MCsYiI\niIhIidouS4uKiIiIiAwpDs0MoJkBEREREZEhwczGm9ltZvZCuB1XpM8BZvYPM1toZk+Y2fuStl+Y\n2ctm9lj4OqC359TJgIiIiIiUHIfDtbf3+9c2Oh+4wzk3B7gj3O+sAfiwc25f4C3A5WY2Nmk/zzl3\nQPh6rMjjCwyLy4San3uOxUcdzaQQXjXlmj/EtvPf+00AdjnkrbF226d8eNdt894AwDObmmPbP5/q\nQ77GfeGyWPvAjx8EYNHfb4+1LFBr5iGHx9oXTvSP3bPJh/Msvvqa2Pb0PT5g5/kkqCYzo7Yybu+5\nqw/Rmf6GPQAYe9jrY1vHbv7k7ZVNeTjO4yt9eNijr6yPtVUr6gHYtL4x1prW++Cg1s2+f1tz3pYG\n33SWBY1BHjaWBY1BHjZWGcKHKmpHxbYsWKwyCRjLgsUqksCwrDaiJu83OmxnwWIjqroGkqWhY1ko\nWWFgWGHYWBowloVcFQSRhceWJwlV2UPS0KryTuFgZRQJEysa7JVvl8Va92lYxfoX05egse769VVf\nHrs1u1fYWO8OevfpALzz1otj7cInffzKZTecE2snXeeD21++7yYgD1cEuPAgP8bc/sZ8XHt8gw/7\nOnnq6Py5vnY2AHe3Tou1K/6wAIAVT93X5dgmzPZBjwe8Lu//vgN2AWBGy7JY23TvnwF47e5nY23x\nyz4c8bVGP541tucBatnbeHzyHp8SxoRJU/IxZtws/2/bTrtPBWDkzN1iW+W0mQC0j85D7NtqfRDa\nhoY8HGx9o9/OgsbS7dUb/b8N6zbnbVkAWUsSMJZtt7Xmq45kAWNp6FgMG2stEjoWw8S6Boyl/bKV\nTYr2y/ZR5I+NYoFgCgkTGZZOBY4O278E7gK+lHZwzj2fbC8zs1XAJKBua55QMwMiIiIiUnoc0NHR\n/1/bZrJzbnnYXgFM7qmzmR0KVAEvJeWLwuVDl5lZdW9POCxmBkRERERE+teALS060cwWJPevcM5d\nkd0xs9uBKUW68cyKAAAgAElEQVQe95WCo3POmZkr0i/bz1Tg18CZzrnsLOQC/ElEFXAFflbhGz0d\nrE4GRERERET6zxrn3MHdNTrnju+uzcxWmtlU59zy8Mf+qm767QT8CfiKc+7+ZN/ZrEKzmf0c+EJv\nB6vLhERERESk9DiH62jv969tdBNwZtg+E7ixcwczqwJuAH7lnLuuU9vUcGvAO4CnentCnQyIiIiI\niAwNFwMnmNkLwPHhPmZ2sJn9NPQ5DTgS+EiRJUSvMrMngSeBicB/9PaEukxIRERERErTtn/gt185\n59YCxxWpLwA+Hrb/F/jfbh5/7JY+p04GRERERKQEOS3Biy4TEhEREREpWcNiZmBjUxu3vLieb712\nGwCzzs0/K1FRMxKAP379xFh75v2nAvD7RT6o6917TYht83/4PQA+/cfnYu3xv9wFQFtjfaztergP\nMfvE2/eJteN29mePq37yPwA8+39P5/sIAT8tHfkKUNNCiM68ibWxNuOIXQHY+U2HAmBzD4tty9pH\n+H2t2BhrDy3238PLSzfFWt3qzQA0rF0Za00bVgPQGr6HnoLGIA8bK6vMQ8eysLEsaAy6ho1VVuf9\ns7CxquryLrXKpFYbfg6jk9Cx2qoQOlZZJGAsbFdV5Oeq1WG7prxrLQsbKxYwVhhSVtalVh4ekwaL\nZbuxTrcFbaS1viVfdd5vsTPxNKSsP8LGego962tI2VDM9doRwsYydx/jM2I+84ZbYu2L5/rAxO+N\nOTnW7rvoEgBmHenHtz/8Sz52PHbaKb72ch5OeNBY/34+6uJ3x9qr+7wdgK//+uFYe/mBvwHQunkD\nAONn7R/bZr9uFgAfOTwP+5o/OoRy3ZGHPy6+xe/vtSfyRS9eCkFeG1q7TsGPCe/7acmYMHWCH3ey\noDGAsbN92NmY2f75K3fZI7a58dMBaB+9c6ytb/Jj9Ibm/H/6VobjWJUEi2VhY6tCIOXa+jyYsimE\npLU05qFjrSF0LA0ii6FjLflj20PYYzb+tqehY50CySAPBXPJZQquSDhZRw//c7mlYWNb+r+g+l9T\n2aE5Bmpp0WFFMwMiIiIiIiVqWMwMiIiIiIj0LzfkPkA8GHQyICIiIiKlx4Fr12VCukxIRERERKRE\naWZAREREREqQ0weI0cyAiIiIiEjJ0syAiIiIiJQep5kBGCYnA7vsOZ1vXvktDrr0WQDWvfx4bPv5\nj84DYOTln461H930PADHTPLr9h9zzbdi22UvVgJww29ujbX6lYsBmLL/MbF2xtv3BuDM/fI1rBuv\nvgiAhf/7IAD3h/X+IV9Le3yyXv78MdUA7Hbkrvn3cqzPF6jY70gA1tTk+39ymc8IWPBKvlb402F7\n7Yo8Z6B+jV/Lu3H9ilhr2eyzCdpb/DrXxdaGzrIFIM8SqKjKMwXKw3aWLQBJvkCNX7O8qjr/lamu\nDTkDtZVdatVJvyxfIMsWgDxfYERV9zkDaaZAlhFQkdbCdnV51/yArH9Zwbr91rUWVtFPdhvXsC+P\n/ZO2TvtK+6fTbFbssX3MF+jS1m1LN/17WYS/L/kC27KMf19zF7bUjpQtkPrqUX4MO/OYfC3/Fz5x\nOQAX/dN3Ym3i3EMA+P2X/Ti15vyPxLarbl0EwIzkvXjy532/lnd+MdbO++0TACy86/5Ya1i7DIDR\nU/0a/ru/bl5s+6ejfM7AMbvtFGsd914NwKt/vi/WXrtvCQBPb8zX3F/XUjgGjanMf/OzfIEZYYwE\nmDB7HADj9pwaa2PnzgCgcte5vjB5Zmxr32kKAHWt+XPUhZyB5Zvy41i12W+v3NiU1zrlC2xKMgia\nQ75Ac1O+49amLFMg/57aWnx7mhsQswQ63UL+IcV0bC6WKZBtF8sWKLaPzo/rjvIFRKQ7w+JkQERE\nRESkvzktLaqTAREREREpRbpMCPQBYhERERGRkqWZAREREREpPQ7NDKCZARERERGRkqWZAREREREp\nOQ6nDxCjkwERERERKUW6TAjQZUIiIiIiIiVrWMwMPFdfwTH3TuSFO68F4JuXnhfbjr3rEgAuvvTe\nWNt/jA/Iesc15wNwVdvese37P70DgHWL8uCyLMznnaceEGufPcIHhXXcdHmsPfrf/rF/X+SDwFY3\n52eTWaDO/kmIzqwQNrbrmw+JtepDTgSgbszuADy1Ig8uu3/xOgAeemltrK1d7oPINq1aHWtZ2FhL\nw8ZY6ylszMp8iFcaOpaFjVXUjIy1qpFjAKgMtwBVI3x7dY0PM6qqzX9lKkOwWFV1HhiWhY2Nqsn7\njQqPHVUkiGxUDCTL95GFjVVXJPut8LWaijSIzKdQZWFjWdBYb7XyIoFhaaBVFkSWhXOlWVdZoFZh\n/2wfacBZ1/12ViwcrC+BYN316ylsrK/7HYphYzu6Y2eOBWDM1X+Itbd9xI87NeMmx9pV3zgVgNE/\n9uPfD//n4dg2JoT4nXZ2PtaM+8JlAHzq98/E2v1/fgCAjUuej7URE6YBsPshBwPwkeNmx7ZT9pwA\nQNmCG2Pt1ZtuAeDl21+OtcfrfKDXyua2WGt3/nZUeM9Oq8kD0WaO9GPR+DnjY238nv57Hb9XHr5W\nNXMv//xT/HjZtlMeSLah3Y8d6xrzMW95CBFbUZ+Hji3f0FRwC7AqBJBtqPehYE2b84Cx5ka/nQWN\nAbSE76u1OQ8Ra2/2Y2429kIeMtYebrOQsLStMGCso0utmK0NG1PQmEhvtLQoaGZARERERKRkDYuZ\nARERERGRfuUKZ/BKlU4GRERERKQEOdBqQrpMSERERESkVGlmQERERERKkz5ArJkBEREREZFSpZkB\nERERESk9zmlJXXQyICIiIiIlyukDxMPjZKBh/ToW/O43nP/NzwNw1vM/j23f/MINAMwckQfafPiq\ncwC4ceLxAHz10jti28qn7gFgwuyDYu3kd78egK8dv0esVd3yIwAevvyPsfa3p9cAsKzJB9BkQWOQ\nB53t+aYZsTbr5MMAqH3DybG2YcJcAB5f6cPG7l2UB4z94wW//9VL8jCxDSt9rWHt0lhrrvehZ22N\n9bHW+cw2CxoDKA8BY5W1o2ItCxsrrPntLGgMuoaNVdfmP+fqUKtJalmI2NgRecBZFjY2uiCIrDBs\nbGRlGjBWHm7zn28WNlaZpGdVx1q4Lc/bitXKiwSGhXyzGDSW1rJKWZGQsvT6uizsKw32Kpa/VSyc\nLLYVCxHrWhpyYWPbI2hsR88ym/P3uwE49KM/jLXs/furb3841va6/usA/OBbtwOQ/GrzwbP8eLbL\nf/wk1s7943MA3Hz932ItC1vMgsYAdj/sCADOOmlPAN6376TYVv2oD0J77brfx9qLf34BgMdXN8Ra\nFjaWBY1BGjbm3+t7jMrHhEn7TvS38/IQsYn7+bCzmtn7xFr5Ln68bBvnx9WN1OTfSxiHCwLGNvnt\n5RvzgLHlIRBtVVJbt9H3a2rwAWNZ0BhAS6Pfb0sSoJaFjWVBY5CHjWVhYpCHjXW0tnRpy8bo9A+P\ntD3Wsn7taThZ4fjea0iZwsZEZAsMi5MBEREREZF+5RyuXTMD+gCxiIiIiEiJ0syAiIiIiJQc59DM\nAJoZEBEREREpWZoZEBEREZES5LSaEDoZEBEREZFSpMuEAF0mJCIiIiJSsjQzICIiIiIlSTMDw+Rk\nYNr0KZxz6Zf42JM+UOfCz14f2/YY6YNsPnbdl2Lt2vEnAPDl/7wVyIPGACbOPQSAd512RKx948QQ\ndvOXH8Tagu/6kJ27nlgVa53Dxg4amwfg7HX0bkAeNAYw4sh3AFAXgsYAHl3hw8buetGHif3tudWx\nLQsbq1u+MtaysLEsaAzysLFiQTFZWFEWNAZ5sFgWNAZQNXKMbwu3kIeNZUFj0DVsLAsagzxsbFQS\nJpaFjWVBY5CHjaX9OoeNZUFjftv/fGuS0LEsbKy6oFYYLJbdT2vlRQLDypP5sCxsLK11DhtLQ6+K\nBYdlwV49BY11fkz6uNRwCBrzzzGwSWA7etBY6uAPXgZAWWUeyvW7758NwF7XfC3WvnehH8+ysLGP\n/ks+1kz75pUA/L8bn421P/3Oj3tZ0BjkYWN7vP6NsXb2yXsDcPq8nQGofeTG2Pbqb/8PgBdCgBnk\nYWOvJUFdWdjYqOT9mYWNzR1dDeRBY5CHjWVBYwA1c+f572/6nrHWOWxsdUMeBLYsBIwtTcLEltT5\nILAsaAxg+QZfy4LGABrrfdhXFjaWBY1BHjaWBY1BHjaWBY1BHhjWngSHdQ4bS8fo7LrknoLGIA8b\nKza+9xQOpqAxEdlaw+JkQERERESkPznn6GjXibFOBkRERESkJGk1IX2AWERERESkZGlmQERERERK\nj3P6ADGaGRARERERKVmaGRARERGRkqSZAZ0MiIiIiEgJcs7pA8QMk5OB8RuW896bL+Rr/3k3AEdO\nHBHb3n3LpQB8v35OrH3nIp9DkK2vPWX/Y2LbR9/vcwYuOHK3WGu++iIA/n7pLbF290vrAFjdnC85\nNT6sjX/IOL/m9Z4nzoptM095EwBVr39brK0ePROABUs2xdo9IV/gged9vsCqkC0AsHHFCiDPFgBo\n3uTzBdL1rYutD11W4dcoL6/2+QIVSc5Ali9QlWQKZPkCWbYA5PkCNSPTnIHCfIHq6jRTIMsZyPtn\n+QKjk0yBUUVyBmrCwv5ZvkCaH5DlC1SXd80NKJYl0DlvAPJ8gYL8ACtsgyQjIO/WJV+gWFZAupZ/\nsX6d+6eKPWcxQy1fYKCyBUopU6CY2nFTALjj8tNjreLCjwPwnz9+KNYmh/fWhy84HoDqcy6Jbadf\n5ce6+268K9Y2LnkegNFT94i1uW88HIBPn5Sv5f+uvSb4jXuvBmDRtX+IbS/e/BIAj6zPx5+VYR3+\nLFsA8uyVGbX5WLBHGCcn7ePzBSbOnx7bJu7nj6lq1r6xVjZjLwDaxub91rf773ltyBdYkmQKLN3k\nt5ckx7Z8g68tr8trdSGPoGlznovQ1ODX+m9t8mNpli0Aeb5Ali0A+fibZgS0hZpLliXsnC/Q0ZY/\nZ6wVyw9ob+/Sr6Bd+QIiJcPMxgO/BWYCi4HTnHPri/RrB54Md191zr091HcHrgEmAA8DH3LOdQ04\nSegzAyIiIiJSkjraO/r9axudD9zhnJsD3BHuF9PonDsgfL09qX8buMw5NxtYD5zV2xPqZEBERERE\nZGg4Ffhl2P4l8I6+PtD85QPHAtdtyeOHxWVCIiIiIiL9yg3JDxBPds4tD9srgMnd9KsxswVAG3Cx\nc+73+EuD6pxz2bWPS4BdentCnQyIiIiIiPSfieEP9cwVzrkrsjtmdjswpcjjvpLecc45M3NF+gHs\n5pxbamazgL+a2ZPAhq05WJ0MiIiIiEjpGbjQsTXOuYO7f1p3fHdtZrbSzKY655ab2VRgVTf7WBpu\nF5nZXcCBwPXAWDOrCLMD04GlxR6f0mcGRERERKTkOMB1dPT71za6CTgzbJ8J3Ni5g5mNM7PqsD0R\nOAJ42jnngDuB9/T0+M50MiAiIiIiMjRcDJxgZi8Ax4f7mNnBZvbT0GdvYIGZPY7/4/9i59zToe1L\nwOfM7EX8Zwiu7O0JdZmQiIiIiJSegbtMaKs559YCxxWpLwA+Hrb/Dszv5vGLgEO35DmHxcnA8pWb\nuPg7d3P666YCcMTNv4ltH73Vh3j94Rc/j7WmDb4268hTAbjgwwfFtjNm+MCVVd85N9Ye/vHfALhn\nTUOs1bf5X45pSVDWodNGAzDnZB/Ys8tJx8a28oNOBOC1sgmx9sDiOgD++tzqWFu4yIeZrVnqw8Y2\nrlgS2xrX+9CxLGgMCsPGOsuCxiAPG6sMAWMVtaNiW9UIHzBWmYSOVdX6/tVJSFAWNpbWqsL3Xxtu\ns6AxyMPGRiVBZKOLBIzVhrC2EZXl+XPF0LHC8DGAypCelYaIZUFkxULHykP/siROKwsbSwOtyosF\nhoXbNFCrc4iYFQspS/bRU9hY0eCwIv17CgzrKWist8f29Jx9pbCxgfXizz8MwKNvPjHWfnXPqwAc\nPj4PDzztR77f4qM+CcA//fD+2PbUrbcD+RgCMGG2H/f2PeqAWDv/zX7sOnJCHobVdtPlADz3mzsB\neP7v+Zj01EYf2JWGL2Zvy0nV+Xs2CxvbfUoeYpiFjU3abyYA4/bJw8+qZu/nN6bktSxsbE1j/lxr\nGv1xLgnHsTQJHXttnR+vs6AxyMPGNm/O83WysLGmhvx7zkLGWrPbpnwfWdhYOva2hzCxNHSso7Uw\nYMy3txbUCtqyWnvfQsX6M2hsax8jIqVhWJwMiIiIiIj0t6E2MzAYdDIgIiIiIqXHQce2f+B32NMH\niEVERERESpRmBkRERESk5DiG3geIB4NmBkRERERESpRmBkRERESk9LjiK3yVGp0MiIiIiEgJcv2R\nGDzs6TIhEREREZESNSxmBqZMHsWXPvgmyi74bwBe//2/xbanbr4BgKokUOuQ094HwH+dcSAA89Y8\nGNue/ez3ALj3Ty/l+whBNuVJCtK8naoB2D8E5wDMfrsP75l04lsBaN/ryNj2zAZ/ZnnPK2ti7c5n\nVgGw+OU8RGzdynoA6le+DEDzhrx/y+YNQGGwTcbK8oCfLGysojoPJKoIYWNZsFgWNJbW0jCxbLuq\nNv8VqK7JQseSELEQMpaFiY2pzYPOsloWKgZ52NjIqnwfNRWFAWMA1SEVrDLc1iRtxULHsrCxtJaF\njRULE8tqBeFgVtiWtqdnxT0GhvXQP9WXsLHewsJ6ChvrS9BYsefszUAFjGUUNNbVddP9OPW3tXnI\nVRaweOQ1l8XaVRt9KNd/XPRXAF65/y+xLRsfZhz2tlg76YQ5AHz6TTNjbU798wCs+dmvY+256xcA\n8NQzawF4KQnsysIXq5JfuMkhZHD3kfl4Mn3WWCAPGgOYuJ8PFBu9995+H7P2jW1t43cDoHlEHtK4\nerMPAFtRnz9/Fja2fJMfo5euy39GS9b70LFVoQ+kAWP5Ppob/X5bGpOgtVZ/WUBrk99HFjQG+fjb\nloSOFQ8Yy2r5/yp2DhvrSPpvz7AxBYyJ9JFTzgBoZkBEREREpGQNi5kBEREREZH+paVFQTMDIiIi\nIiIla8BmBszsZ8DJwCrn3LxQGw/8FpgJLAZOc86t724fIiIiIiIDwTno0MzAgM4M/AJ4S6fa+cAd\nzrk5wB3hvoiIiIjIduaXFu3vr+FmwE4GnHP3AOs6lU8Ffhm2fwm8Y6CeX0REREREera9P0A82Tm3\nPGyvACZv5+cXEREREdHSosGgrSbknHNm5rprN7OzgbMBxk2exg2n/jvfOftKANYtejz2m7L/MQB8\n9P2HxNoFR4Y1rK++CIC7Lr0ltt39kp+sWN2cr8M8PqyTf8i4mljb88RZAMw85U2xVvV6v4b3mtEz\nAVjwan1su+dFnxfwwPOrY23Vko0AbFyxItYa1i71x7bJf1SiPVnLutja0FmmQHmaKVDlt7NsAchz\nFvKcgbwtyw+oSdYFr6rtmilQHdYPHzsi7zcqPHZUaMuyBXxbRcEtQE15limQZw9k+QI1RXMGQqZA\nWZIz0EMtzQgIu4hr16dt2bLo6bL2ZcWyB8j2UeSxRTIFKNK/8+M6P29P/XraX0/9i9mWJfwHKl9A\nuQK9WxrWwf/ahSfFWvU5lwDwrt8+EWv33fi/AGxc4rMCRk/dI7bNecNhAHzmbXvF2rv2Cmv433t1\nrD1/7R8AePHmPGflkfV+DFrZ7I+jPRmVx1T63/4ZSUbJHmGcLMgUmO8zELJsAchzBcqnzwWgddyu\nsW19ux8z1m7I1/5fEvJeloZMAYAl4diWb/C15XX5eFm3yecLZNkCkOcLtDblY2lL+L5am/PsgSxX\nIBt/02yXLF8gzQXIMwXSWmvXWqcxPN3HQGUKbG1/EZHM9j4ZWGlmU51zy81sKrCqu47OuSuAKwBm\n7DW/25MGEREREZEt5sC160/M7X0ycBNwJnBxuL1xOz+/iIiIiAgOp9WEGMAPEJvZb4B/AHua2RIz\nOwt/EnCCmb0AHB/ui4iIiIjIIBiwmQHn3Pu7aTpuoJ5TRERERKRPHLgOXSakBGIRERERkRI1aKsJ\niYiIiIgMpg59gFgnAyIiIiJSepxyBgBdJiQiIiIiUrKGxczA0tdWcMG5FzNiwjQAjjzrY7Hth6ft\nB8DuL90Wawvf/xkA7vnrKwA8E8JpAKpCgtNBY/OAsfkH+iDk2accFGvjTzgZgJbZR8Ta46t98M3d\nT/gQsTufyWMSlrxSB8C65etjbfPqVwFo3rAm1lobfVBZGnKTsTIf1JUFjQFU1o4CCgPGKkKtasSY\nvF8IG6sZ6R9bECYWgsOq0loIEUoDw8aO8I/NAsYgDxnL+tVW5WFiI6t8rSBMLGxnoWJpe2VSqwyv\nQ9a/WMBYGliVBYolu6CMwn7laZjYVgaMpYr17/y4VLF8rS0NGOvtscVsaa7XQAWMZRQ0tmW+8Nzv\nAfjxayNi7ZLP3wzAsofzwMTsfT/ryFMBeOeJc2LbJw73gV7TVz8aayu+cyEAz//+sVh74iU/Ti1u\nyMefxjBFXhveQJOT9//sUX5MmDxrbKztPN+Pl2nA2Mi5Puysao/5sdY2zgeRbaryj13TmIdiLd/U\nNWBsWQgdW7IuDxZbsSEEotX5tqaGJGBss/8emkNoG0BrCBjLgsYgDxbLgsYKamEc7mjNfx5ZeFc6\nRruOjoK2gn5prb1T6Fgfg8YUMCYySJxTzgCaGRARERERKVnDYmZARERERKS/6QPEmhkQERERESlZ\nmhkQERERkdKj1YQAnQyIiIiISAlyQIcSiHWZkIiIiIhIqdLMgIiIiIiUHi0tCmhmQERERESkZA2L\nmYERY8cz/92nc9kHDgTgkM1PxLbnv/RBAH594/Ox9khdU8Hj9x5dHbcP3HsCALNP2T/WJr35JAA6\n9j4q1p7e5G/ve2xFrN31rA8Ze+llHyy2dvmm2JYFjDWtXxlrLZs3+P32MWCsorrW3yYBY1mYWLGA\nsSw4LN3OgsWyoDHf5mujR+S1PEwsqVUXBoyl27WV/nhHVOahY8UCxrJgsTSILAsYq0xSwbKQsaxW\nniRsZQFjZUVCxwpCxKyHtnBbLGAslTWXFdQKO25LwFjhc3XfQQFjpWv+JS8BsPjvt3Rpm/a6N8ft\nE46fDcDnjpoFwJzGRbFt3a++CsD91y+ItYULVwPwfH0+/tS3+Q/KVSW/cNPCezwLGJu2Wz7WTJ4/\nCYAJ++4eazvt4wPGKmfNi7WOCbsBsHnkpFhb3eCDv1atDQFjG/NxeUnYXpoEjC0PAWPLk/G7abMP\nGWtq6D5grDUJGGttagAKA8ay8betJamFkLH+DBjr/Jhi93vrv639RGTLdegDxMPjZEBEREREpD85\nhy4TQpcJiYiIiIiULM0MiIiIiEjp0cwAoJkBEREREZGSpZkBERERESlBTh8gRicDIiIiIlKKHDgl\nEOsyIRERERGRUjUsZgb23KmNu4+pY+E5pwPwg7++Etue2dQMFK6bvf+YGgDmHzgZgNmnHBTbxp9w\nMgAts4+ItSdW+3Wt7350Tazd+YzPFFjySl2srVvu8wWyTIHmDXn/1sZ6oOdMAchzBSprRwGFmQIV\noVYsU6BmZJ5HUF0kSyDmC4S8gTQrYOwI/9hR1XktzxnIa7VV/jhHVuW1LC+gWKZA1lZZ3jVToLog\nZ6AwUwDy9emzjIBkF5TRNTcge2i6bn7njIBimQLF1sHvKVMgfWzs03UX3WQW9G3RfWUKSOrVh+4E\nYLfDT4y1d544B4BPHL5rrE1f/SgAK358HgD3/P6x2PbES36cWtyQjz+N4UNxtcn7bmbIGskyBQAm\nzxoLwM7z/Xg5cb89YtvIuT5ToGqP+bHWNm46AJuqxsbamka/Dv7yFQ2xtnSTH1eXhUyBJUmmwIqQ\nKbAyzRRoCJkCm/PvIcsVyLIEWpJMgfaQG5BmCsRaMg53zhSAfJzuz0yBYrWtyQdQpoDI9uOADn2A\nWDMDIiIiIiKlaljMDIiIiIiI9CvncPoAsWYGRERERESGAjMbb2a3mdkL4XZckT7HmNljyVeTmb0j\ntP3CzF5O2g7o7Tl1MiAiIiIiJcm1u37/2kbnA3c45+YAd4T7hcfs3J3OuQOccwcAxwINwK1Jl/Oy\ndufcY50f35kuExIRERGRkuPckPwA8anA0WH7l8BdwJd66P8e4M/OuYYe+vRIMwMiIiIiIv1nopkt\nSL7O3oLHTnbOLQ/bK4DJvfQ/HfhNp9pFZvaEmV1mZtW9PaFmBkRERESkJGVLDPezNc65g7trNLPb\ngSlFmr6S3nHOOTPrdurCzKYC84FbkvIF+JOIKuAK/KzCN3o6WJ0MiIiIiIhsJ86547trM7OVZjbV\nObc8/LG/qoddnQbc4JxrTfadzSo0m9nPgS/0djzD4mRg6XNL+OpR57Gh1Z+9TUuCsk7dzYdyzT5p\nTqzNOOU4AMpf9xYAllVMim13Lt3ob//4XKw98eJaANaENoBNq/zPsmHt0lhr2ezbs2CbYuEwWagY\nQHl1LQCVPQSLZaFiAFW1vn8WHOa3K0Jb19qIpDZ2RBY2Fm77GDA2ojIPRKspLwwYgzxQLAsbS4PD\nYuhYWdo/CxFLAsDoGizWOXQsDeLKNnsKGPO1wsemoVjFrn/r3L+grUj/nsLBegsY60uw2LZkeA1U\nwJiCxba/637qPxt23JT8h992x68AeOFf7oi1++/yYYtPbfRBi6ubu44/46vy9/Peo/1YsNukEbE2\neb4fCyfOy8PMxs/zY2fVrH19Yers/DjG+oCx5c35c6wNQWCvrtwUa0tDsNiyujwAbMl6v70qtG3c\nlO+kabP/d6u5Kf771SVgDKC1yT82CxZLQx3bWrrWigeMtXapFQsWi20hYKwvoWJb2r6t/UWknzk3\nFD8zcBNwJnBxuL2xh77vx88ERMmJhAHvAJ7q7QmHxcmAiIiIiEi/cvTH6j/97WLgWjM7C3gF/7//\nmNnBwGDjBTMAAA3dSURBVCeccx8P92cCM4C7Oz3+KjObhP8/x8eAT/T2hDoZEBEREREZApxza4Hj\nitQXAB9P7i8GdinS79gtfU6dDIiIiIhIyXGgBGK0tKiIiIiISMnSzICIiIiIlJ6hGTq23elkQERE\nRERKkBuKHyDe7nSZkIiIiIhIidLMgIiIiIiUHOegw2lmYFicDOxUXcHxM8exx1v2BGCXt50Q2+wA\nv/2qy8O7/m/JBgDuuHM1AE8vWhTbsmCx+lV5mFjD2mUAtGzeEGtZsFgxWbBYFiAGebBYGiJWWTOq\nS61mpA8AykLEsgAxv+1rVUk42KgQJjY6rYVgsbFJ6FgWIpYFi6WhY1mIWE2S+pXVqivykKLKkJRV\nU9E1RCwLFktDx7JgsTLSmr9Nw6v6EixWXiwILNlJX4LFivUvaO9a2upgsb6EinX3nH2lYLHSsPN5\nHwTgt/9YEmvPbPLhWeta8lCq7D2SBYsdMq4mtu02dTQAk/adGGuT9psJwNi994i1qtn7AeAm57W2\nsX5lumUNPuxrXWMe+vXqYh8sloWKASwNYWJZqBjkwWKb0mCxBh/21RL215KGiYXtLFQM8mCxdOxt\nD4FiWbBYFioGSXBYW2uXWkHoWFZr71uIWE9BYAoVE5Ed0bA4GRARERER6W/tmhnQZwZEREREREqV\nZgZEREREpOQ4QIsJ6WRAREREREqULhPSZUIiIiIiIiVLMwMiIiIiUnJ0mZCnmQERERERkRKlmQER\nERERKTnO6TMDMExOBqr33JPdb7uL+7MwsedWx7anr3gOyMPEIA8Ua1y/AoDmTetjW1/CxCAPCsvC\nxAprvYeJQR4oVp3UskCxrQ0TgzxQrDoJB8sCxbY2TCytlSeJWlmg2NaGifl+dGGdHttTmFix/gVt\nRfpvbZhYb4/t6Tn7SmFikvrZn14A8jAxgD1G+jHg2NnjYy0LFNvSMLHVDXnYVxYo9uq6JERssR8n\ntzZMDPJAsdY0WCwEivUlTAzyQLGCwLAQKDbUwsS29jEiMvToMiFdJiQiIiIiUrKGxcyAiIiIiEh/\ncjhdJoRmBkRERERESpZmBkRERESk5GhpUU8nAyIiIiJSknQyoMuERERERERKlmYGRERERKTkKGfA\nGxYnA8+8vJJDP3QZzfU+L6CtsT62FVvr2cr8et3lVbUAVCV5ABUTpnWpZXkBVSPyTIHqsOZ/VW3+\nI8ryArL8gJokDyDLARg7Is8qyPIACrME/HaWHzCyMl9bPMsGKMgPCNuVyeL31bHWfW5AZXm6zr/f\nLsgICA8tS1bMj1kCebe4Jn722HQqqdja/8XWui/r1L+gbRtyA3rKC+hLVkB3z9UXA5UVkFJuQGn4\n1pUfBqBm7rxYK5++JwBt42bE2gZXDcCaRj/mPZ2s/b805AEsWZiv5b9kvc9gybICANZt9I9prM/X\n929uLMwNaEmzApp9vywrAPK8gDQjIOYGtCa5AW2FuQGuo6NLW6pYbkDn8b23tf23dO1/ZQWIiAyT\nkwERERERkf6mzwzoMwMiIiIiIiVLMwMiIiIiUnL80qKaGtDJgIiIiIiUHOUMeLpMSERERESkRGlm\nQERERERKki4T0syAiIiIiEjJ0syAiIiIiJQcHzo22Ecx+IbFyUBZRSUjJuzCmF33AfLwr3S7IBys\npjAcbPSIvH8WADamtqpLLQsCgzwcbGRVvt8sACwL/aouT0O/ygr6QB4UlgaAdQ4FKy/rGg6WBmYV\nCwzL2suTYudQsDSQq3hgV2H/zo/p7rF9DQQrfK7tHw420KFgCgST/vKJ8lMAWPVoHiLWdN86f9uw\nItaas1CwEBLW1poHZrU2NQCF4WBZsFdbS1JrLQwCS/tloWBpW7bdkdbauwZ1FQvv6inQq69hXwoF\nE5GBpsuEdJmQiIiIiEjJGhYzAyIiIiIi/ckBHYN9EEOAZgZEREREREqUZgZEREREpAQ5fWYAnQyI\niIiISAlSArGny4REREREREqUZgZEREREpOT4mQFNDWhmQERERESkRA2LmYF5Myfwtys/MNiHISLS\n76677L8H+xBEREqTEogBzQyIiIiIiJSsYTEzICIiIiLSn/SZAU8nAyIiIiJSknSZ0CBdJmRmbzGz\n58zsRTM7fzCOQURERERkKDGz95rZQjPrMLODe+hX9G9pM9vdzB4I9d+aWVVvz7ndTwbMrBz4IXAS\nsA/wfjPbZ3sfh4iIiIiUruwyof7+2kZPAe8C7umuQy9/S38buMw5NxtYD5zV2xMOxszAocCLzrlF\nzrkW4Brg1EE4DhERERGRIcM594xz7rleuhX9W9rMDDgWuC70+yXwjt6eczA+M7AL8Fpyfwlw2CAc\nh4iIiIiUKD8zMNhHsVW6+1t6AlDnnGtL6rv0trMh+wFiMzsbODvcbR5RW/vUYB6PbBcTgTWDfRAy\n4PQ6lwa9zqVBr3Np2JbXebf+PJD+tIaWW37CKxMHYNc1ZrYguX+Fc+6K7I6Z3Q5MKfK4rzjnbhyA\n4+nRYJwMLAVmJPenh1qB8EO7AsDMFjjnuv0QhewY9DqXBr3OpUGvc2nQ61wadtTX2Tn3lkF63uO3\ncRfd/S29FhhrZhVhdqDo39idDcZnBh4C5oRPO1cBpwM3DcJxiIiIiIgMN0X/lnbOOeBO4D2h35lA\nrzMN2/1kIJypfAq4BXgGuNY5t3B7H4eIiIiIyFBiZu80syXA64E/mdktoT7NzG6GXv+W/hLwOTN7\nEf8Zgit7fU43DJLXzOzs9For2THpdS4Nep1Lg17n0qDXuTTodd6xDYuTARERERER6X+DkkAsIiIi\nIiKDb0ifDHQXtSzDk5n9zMxWmdlTSW28md1mZi+E23Ghbmb2/fDaP2FmBw3ekUtfmdkMM7vTzJ4O\ncernhLpe5x2ImdWY2YNm9nh4nb8e6rub2QPh9fxt+GAbZlYd7r8Y2mcO5vHLljGzcjN71Mz+GO7r\ndd4BmdliM3vSzB7LlsXU2F0ahuzJQC9RyzI8/QLovIzX+cAdzrk5wB3hPvjXfU74Ohv47+10jLJt\n2oDPO+f2AQ4HPhnet3qddyzNwLHOuf2BA4C3mNnhwLeBy5xzs4H1wFmh/1nA+lC/LPST4eMc/IcU\nM3qdd1zHOOcOSJYR1dhdAobsyQDdRC0P8jHJNnDO3QOs61Q+FR+XDYWx2acCv3Le/fh1c6dunyOV\nreWcW+6ceyRsb8L/AbELep13KOH1qg93K8OXA44Frgv1zq9z9vpfBxxnZradDle2gZlNB94G/DTc\nN/Q6lxKN3SVgKJ8MFIta7jVSWYadyc655WF7BTA5bOv1H+bCJQIHAg+g13mHEy4deQxYBdwGvATU\nhSXvoPC1jK9zaN+AX/JOhr7LgS8CHeH+BPQ676gccKuZPWxmZ4eaxu4SMBgJxCJFOeecmWl5qx2A\nmY0CrgfOdc5tTP9zUK/zjsE51w4cYGZjgRuAvQb5kKSfmdnJwCrn3MNmdvRgH48MuDc655aa2c7A\nbWb2bNqosXvHNZRnBrqLWpYdy8psajHcrgp1vf7DlJlV4k8ErnLO/V8o63XeQTnn6vCJl6/HXyqQ\n/SdT+lrG1zm0jwHWbudDlS13BPB2M1uMv1T3WOB76HXeITnnlobbVfgT/EPR2F0ShvLJQNGo5UE+\nJul/N+HjsqEwNvsm4MNhxYLDgQ3JVKUMUeH64CuBZ5xzlyZNep13IGY2KcwIYGa1wAn4z4fcCbwn\ndOv8Omev/3uAvzqF3Ax5zrkLnHPTnXMz8f8G/9U59wH0Ou9wzGykmY3OtoETgafQ2F0ShnTomJm9\nFX+9YjnwM+fcRYN8SLINzOw3wNHARGAl8G/A74FrgV2BV4DTnHPrwh+VP8CvPtQAfNQ5t2Awjlv6\nzszeCNwLPEl+jfGX8Z8b0Ou8gzCz/fAfJizH/6fStc65b5jZLPz/II8HHgU+6JxrNrMa4Nf4z5Cs\nA053zi0anKOXrREuE/qCc+5kvc47nvCa3hDuVgBXO+cuMrMJaOze4Q3pkwERERERERk4Q/kyIRER\nERERGUA6GRARERERKVE6GRARERERKVE6GRARERERKVE6GRARERERKVE6GRAR6Qdm1m5mj5nZQjN7\n3Mw+b2ZbPcaa2ZeT7Zlm9lT/HKmIiEhOJwMiIv2j0Tl3gHNuX3wI10n4LI2t9eXeu4iIiGwbnQyI\niPQz59wq4GzgUyGhs9zM/tPMHjKzJ8zsn8EHOZnZPWb2JzN7zsx+bGZlZnYxUBtmGq4Kuy03s/8J\nMw+3huRfERGRbaKTARGRARCSV8uBnYGzgA3OuUOAQ4B/MrPdQ9dDgU8D+wB7AO9yzp1PPtPwgdBv\nDvDDMPNQB7x7+303IiKyo9LJgIjIwDsR+LCZPQY8AEzA/3EP8KBzbpFzrh34DfDGbvbxsnPusbD9\nMDBzAI9XRERKRMVgH4CIyI7IzGYB7cAqwIBPO+du6dTnaMB1emjn+5nmZLsd0GVCIiKyzTQzICLS\nz8xsEvBj4AfOOQfcAvyLmVWG9rlmNjJ0P9TMdg8rD70PuC/UW7P+IiIiA0UzAyIi/aM2XAZUCbQB\nvwYuDW0/xV/W84iZGbAaeEdoewj4ATAbuBO4IdSvAJ4ws0eAr2yPb0BEREqP+f+0EhGR7S1cJvQF\n59zJg30sIiJSmnSZkIiIiIhIidLMgIiIiIhIidLMgIiIiIhIidLJgIiIiIhIidLJgIiIiIhIidLJ\ngIiIiIhIidLJgIiIiIhIidLJgIiIiIhIifr/tIVLbtWYmkQAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 1008x576 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "a_b4ou4TYqUN"
      },
      "source": [
        "## Masking"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "s42Uydjkv0hF"
      },
      "source": [
        "Mask all the pad tokens in the batch of sequence. It ensures that the model does not treat padding as the input. The mask indicates where pad value `0` is present: it outputs a `1` at those locations, and a `0` otherwise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "U2i8-e1s8ti9",
        "colab": {}
      },
      "source": [
        "def create_padding_mask(seq):\n",
        "  seq = tf.cast(tf.math.equal(seq, 0), tf.float32)\n",
        "  \n",
        "  # add extra dimensions to add the padding\n",
        "  # to the attention logits.\n",
        "  return seq[:, tf.newaxis, tf.newaxis, :]  # (batch_size, 1, 1, seq_len)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "A7BYeBCNvi7n",
        "colab": {}
      },
      "source": [
        "x = tf.constant([[7, 6, 0, 0, 1], [1, 2, 3, 0, 0], [0, 0, 0, 4, 5]])\n",
        "create_padding_mask(x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Z0hzukDBgVom"
      },
      "source": [
        "The look-ahead mask is used to mask the future tokens in a sequence. In other words, the mask indicates which entries should not be used.\n",
        "\n",
        "This means that to predict the third word, only the first and second word will be used. Similarly to predict the fourth word, only the first, second and the third word will be used and so on."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "dVxS8OPI9uI0",
        "colab": {}
      },
      "source": [
        "def create_look_ahead_mask(size):\n",
        "  mask = 1 - tf.linalg.band_part(tf.ones((size, size)), -1, 0)\n",
        "  return mask  # (seq_len, seq_len)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GDCI__yA043h",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 92
        },
        "outputId": "9e4af7d3-293e-4c2f-c1ca-925e9379b3ac"
      },
      "source": [
        "tf.linalg.band_part(tf.ones((3, 3)), -1, 0)"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=207892, shape=(3, 3), dtype=float32, numpy=\n",
              "array([[1., 0., 0.],\n",
              "       [1., 1., 0.],\n",
              "       [1., 1., 1.]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yxKGuXxaBeeE",
        "colab": {}
      },
      "source": [
        "x = tf.random.uniform((1, 3))\n",
        "temp = create_look_ahead_mask(x.shape[1])\n",
        "temp"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xluDl5cXYy4y"
      },
      "source": [
        "## Scaled dot product attention"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vsxEE_-Wa1gF"
      },
      "source": [
        "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/scaled_attention.png\" width=\"500\" alt=\"scaled_dot_product_attention\">\n",
        "\n",
        "The attention function used by the transformer takes three inputs: Q (query), K (key), V (value). The equation used to calculate the attention weights is:\n",
        "\n",
        "$$\\Large{Attention(Q, K, V) = softmax_k(\\frac{QK^T}{\\sqrt{d_k}}) V} $$\n",
        "\n",
        "The dot-product attention is scaled by a factor of square root of the depth. This is done because for large values of depth, the dot product grows large in magnitude pushing the softmax function where it has small gradients resulting in a very hard softmax. \n",
        "\n",
        "For example, consider that `Q` and `K` have a mean of 0 and variance of 1. Their matrix multiplication will have a mean of 0 and variance of `dk`. Hence, *square root of `dk`* is used for scaling (and not any other number) because the matmul of `Q` and `K` should have a mean of 0 and variance of 1, and you get a gentler softmax.\n",
        "\n",
        "The mask is multiplied with -1e9 (close to negative infinity). This is done because the mask is summed with the scaled matrix multiplication of Q and K and is applied immediately before a softmax. The goal is to zero out these cells, and large negative inputs to softmax are near zero in the output.\n",
        "\n",
        "Working with complete sequence, the query and key can include the sequence length dimension\n",
        "Earlier we had worked with single  single query of dimension [m, units]\n",
        "and Values = keys = encoder_outputs = [ m, Tx, units]\n",
        "because query = s_prev was not available at all times in case of Bahdanau & Luong Attention.\n",
        "the attention was calculated at each time step\n",
        "But in transformer model, there are no recurrent time steps and the entire query with all timesteps is processed at once "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "LazzUq3bJ5SH",
        "colab": {}
      },
      "source": [
        "def scaled_dot_product_attention(q, k, v, mask):\n",
        "  \"\"\"Calculate the attention weights.\n",
        "  q, k, v must have matching leading dimensions.\n",
        "  k, v must have matching penultimate dimension, i.e.: seq_len_k = seq_len_v.\n",
        "  The mask has different shapes depending on its type(padding or look ahead) \n",
        "  but it must be broadcastable for addition.\n",
        "  \n",
        "  Args:\n",
        "    q: query shape == (..., seq_len_q, depth)\n",
        "    k: key shape == (..., seq_len_k, depth)\n",
        "    v: value shape == (..., seq_len_v, depth_v)\n",
        "    mask: Float tensor with shape broadcastable \n",
        "          to (..., seq_len_q, seq_len_k). Defaults to None.\n",
        "    \n",
        "  Returns:\n",
        "    output, attention_weights\n",
        "  \"\"\"\n",
        "\n",
        "  matmul_qk = tf.matmul(q, k, transpose_b=True)  # (..., seq_len_q, seq_len_k)\n",
        "  \n",
        "  # scale matmul_qk\n",
        "  dk = tf.cast(tf.shape(k)[-1], tf.float32)\n",
        "  scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)\n",
        "\n",
        "  # add the mask to the scaled tensor.\n",
        "  if mask is not None:\n",
        "    scaled_attention_logits += (mask * -1e9)  \n",
        "\n",
        "  # softmax is normalized on the last axis (seq_len_k) so that the scores\n",
        "  # add up to 1.\n",
        "  attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1)  # (..., seq_len_q, seq_len_k)\n",
        "\n",
        "  output = tf.matmul(attention_weights, v)  # (..., seq_len_q, depth_v)\n",
        "\n",
        "  return output, attention_weights"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FiqETnhCkoXh"
      },
      "source": [
        "As the softmax normalization is done on K, its values decide the amount of importance given to Q.\n",
        "\n",
        "The output represents the multiplication of the attention weights and the V (value) vector. This ensures that the words you want to focus on are kept as-is and the irrelevant words are flushed out."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n90YjClyInFy",
        "colab": {}
      },
      "source": [
        "def print_out(q, k, v):\n",
        "  temp_out, temp_attn = scaled_dot_product_attention(\n",
        "      q, k, v, None)\n",
        "  print ('Attention weights are:')\n",
        "  print (temp_attn)\n",
        "  print ('Output is:')\n",
        "  print (temp_out)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yAzUAf2DPlNt",
        "colab": {}
      },
      "source": [
        "np.set_printoptions(suppress=True)\n",
        "\n",
        "temp_k = tf.constant([[10,0,0],\n",
        "                      [0,10,0],\n",
        "                      [0,0,10],\n",
        "                      [0,0,10]], dtype=tf.float32)  # (4, 3)\n",
        "\n",
        "temp_v = tf.constant([[   1,0],\n",
        "                      [  10,0],\n",
        "                      [ 100,5],\n",
        "                      [1000,6]], dtype=tf.float32)  # (4, 2)\n",
        "\n",
        "# This `query` aligns with the second `key`,\n",
        "# so the second `value` is returned.\n",
        "temp_q = tf.constant([[0, 10, 0]], dtype=tf.float32)  # (1, 3)\n",
        "print_out(temp_q, temp_k, temp_v)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "zg6k-fGhgXra",
        "colab": {}
      },
      "source": [
        "# This query aligns with a repeated key (third and fourth), \n",
        "# so all associated values get averaged.\n",
        "temp_q = tf.constant([[0, 0, 10]], dtype=tf.float32)  # (1, 3)\n",
        "print_out(temp_q, temp_k, temp_v)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "UAq3YOzUgXhb",
        "colab": {}
      },
      "source": [
        "# This query aligns equally with the first and second key, \n",
        "# so their values get averaged.\n",
        "temp_q = tf.constant([[10, 10, 0]], dtype=tf.float32)  # (1, 3)\n",
        "print_out(temp_q, temp_k, temp_v)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aOz-4_XIhaTP"
      },
      "source": [
        "Pass all the queries together."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6dlU8Tm-hYrF",
        "colab": {}
      },
      "source": [
        "temp_q = tf.constant([[0, 0, 10], [0, 10, 0], [10, 10, 0]], dtype=tf.float32)  # (3, 3)\n",
        "print_out(temp_q, temp_k, temp_v)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kmzGPEy64qmA"
      },
      "source": [
        "## Multi-head attention"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fz5BMC8Kaoqo"
      },
      "source": [
        "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/multi_head_attention.png\" width=\"500\" alt=\"multi-head attention\">\n",
        "\n",
        "\n",
        "Multi-head attention consists of four parts:\n",
        "*    Linear layers and split into heads.\n",
        "*    Scaled dot-product attention.\n",
        "*    Concatenation of heads.\n",
        "*    Final linear layer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JPmbr6F1C-v_"
      },
      "source": [
        "Each multi-head attention block gets three inputs; Q (query), K (key), V (value). These are put through linear (Dense) layers and split up into multiple heads. \n",
        "\n",
        "The `scaled_dot_product_attention` defined above is applied to each head (broadcasted for efficiency). An appropriate mask must be used in the attention step.  The attention output for each head is then concatenated (using `tf.transpose`, and `tf.reshape`) and put through a final `Dense` layer.\n",
        "\n",
        "Instead of one single attention head, Q, K, and V are split into multiple heads because it allows the model to jointly attend to information at different positions from different representational spaces. After the split each head has a reduced dimensionality, so the total computation cost is the same as a single head attention with full dimensionality."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "BSV3PPKsYecw",
        "colab": {}
      },
      "source": [
        "class MultiHeadAttention(tf.keras.layers.Layer):\n",
        "  def __init__(self, d_model, num_heads):\n",
        "    super(MultiHeadAttention, self).__init__()\n",
        "    self.num_heads = num_heads\n",
        "    self.d_model = d_model\n",
        "    \n",
        "    assert d_model % self.num_heads == 0\n",
        "    \n",
        "    self.depth = d_model // self.num_heads\n",
        "    \n",
        "    self.wq = tf.keras.layers.Dense(d_model)\n",
        "    self.wk = tf.keras.layers.Dense(d_model)\n",
        "    self.wv = tf.keras.layers.Dense(d_model)\n",
        "    \n",
        "    self.dense = tf.keras.layers.Dense(d_model)\n",
        "        \n",
        "  def split_heads(self, x, batch_size):\n",
        "    \"\"\"Split the last dimension into (num_heads, depth).\n",
        "    Transpose the result such that the shape is (batch_size, num_heads, seq_len, depth)\n",
        "    \"\"\"\n",
        "    x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))\n",
        "    return tf.transpose(x, perm=[0, 2, 1, 3])\n",
        "    \n",
        "  def call(self, v, k, q, mask):\n",
        "    batch_size = tf.shape(q)[0]\n",
        "    \n",
        "    q = self.wq(q)  # (batch_size, seq_len, d_model)\n",
        "    k = self.wk(k)  # (batch_size, seq_len, d_model)\n",
        "    v = self.wv(v)  # (batch_size, seq_len, d_model)\n",
        "    \n",
        "    q = self.split_heads(q, batch_size)  # (batch_size, num_heads, seq_len_q, depth)\n",
        "    k = self.split_heads(k, batch_size)  # (batch_size, num_heads, seq_len_k, depth)\n",
        "    v = self.split_heads(v, batch_size)  # (batch_size, num_heads, seq_len_v, depth)\n",
        "    \n",
        "    # scaled_attention.shape == (batch_size, num_heads, seq_len_q, depth)\n",
        "    # attention_weights.shape == (batch_size, num_heads, seq_len_q, seq_len_k)\n",
        "    scaled_attention, attention_weights = scaled_dot_product_attention(\n",
        "        q, k, v, mask)\n",
        "    \n",
        "    scaled_attention = tf.transpose(scaled_attention, perm=[0, 2, 1, 3])  # (batch_size, seq_len_q, num_heads, depth)\n",
        "\n",
        "    concat_attention = tf.reshape(scaled_attention, \n",
        "                                  (batch_size, -1, self.d_model))  # (batch_size, seq_len_q, d_model)\n",
        "\n",
        "    output = self.dense(concat_attention)  # (batch_size, seq_len_q, d_model)\n",
        "        \n",
        "    return output, attention_weights"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0D8FJue5lDyZ"
      },
      "source": [
        "Create a `MultiHeadAttention` layer to try out. At each location in the sequence, `y`, the `MultiHeadAttention` runs all 8 attention heads across all other locations in the sequence, returning a new vector of the same length at each location."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Hu94p-_-2_BX",
        "colab": {}
      },
      "source": [
        "temp_mha = MultiHeadAttention(d_model=512, num_heads=8)\n",
        "y = tf.random.uniform((1, 60, 512))  # (batch_size, encoder_sequence, d_model)\n",
        "out, attn = temp_mha(y, k=y, q=y, mask=None)\n",
        "out.shape, attn.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RdDqGayx67vv"
      },
      "source": [
        "## Point wise feed forward network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "gBqzJXGfHK3X"
      },
      "source": [
        "Point wise feed forward network consists of two fully-connected layers with a ReLU activation in between."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ET7xLt0yCT6Z",
        "colab": {}
      },
      "source": [
        "def point_wise_feed_forward_network(d_model, dff):\n",
        "  return tf.keras.Sequential([\n",
        "      tf.keras.layers.Dense(dff, activation='relu'),  # (batch_size, seq_len, dff)\n",
        "      tf.keras.layers.Dense(d_model)  # (batch_size, seq_len, d_model)\n",
        "  ])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "mytb1lPyOHLB",
        "colab": {}
      },
      "source": [
        "sample_ffn = point_wise_feed_forward_network(512, 2048)\n",
        "sample_ffn(tf.random.uniform((64, 50, 512))).shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7e7hKcxn6-zd"
      },
      "source": [
        "## Encoder and decoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yScbC0MUH8dS"
      },
      "source": [
        "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/transformer.png\" width=\"600\" alt=\"transformer\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MfYJG-Kvgwy2"
      },
      "source": [
        "The transformer model follows the same general pattern as a standard [sequence to sequence with attention model](nmt_with_attention.ipynb). \n",
        "\n",
        "* The input sentence is passed through `N` encoder layers that generates an output for each word/token in the sequence.\n",
        "* The decoder attends on the encoder's output and its own input (self-attention) to predict the next word. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QFv-FNYUmvpn"
      },
      "source": [
        "### Encoder layer\n",
        "\n",
        "Each encoder layer consists of sublayers:\n",
        "\n",
        "1.   Multi-head attention (with padding mask) \n",
        "2.    Point wise feed forward networks. \n",
        "\n",
        "Each of these sublayers has a residual connection around it followed by a layer normalization. Residual connections help in avoiding the vanishing gradient problem in deep networks.\n",
        "\n",
        "The output of each sublayer is `LayerNorm(x + Sublayer(x))`. The normalization is done on the `d_model` (last) axis. There are N encoder layers in the transformer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ncyS-Ms3i2x_",
        "colab": {}
      },
      "source": [
        "class EncoderLayer(tf.keras.layers.Layer):\n",
        "  def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
        "    super(EncoderLayer, self).__init__()\n",
        "\n",
        "    self.mha = MultiHeadAttention(d_model, num_heads)\n",
        "    self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
        "\n",
        "    self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
        "    self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
        "    \n",
        "    self.dropout1 = tf.keras.layers.Dropout(rate)\n",
        "    self.dropout2 = tf.keras.layers.Dropout(rate)\n",
        "    \n",
        "  def call(self, x, training, mask):\n",
        "\n",
        "    attn_output, _ = self.mha(x, x, x, mask)  # (batch_size, input_seq_len, d_model)\n",
        "    attn_output = self.dropout1(attn_output, training=training)\n",
        "    out1 = self.layernorm1(x + attn_output)  # (batch_size, input_seq_len, d_model)\n",
        "    \n",
        "    ffn_output = self.ffn(out1)  # (batch_size, input_seq_len, d_model)\n",
        "    ffn_output = self.dropout2(ffn_output, training=training)\n",
        "    out2 = self.layernorm2(out1 + ffn_output)  # (batch_size, input_seq_len, d_model)\n",
        "    \n",
        "    return out2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "AzZRXdO0mI48",
        "colab": {}
      },
      "source": [
        "sample_encoder_layer = EncoderLayer(512, 8, 2048)\n",
        "\n",
        "sample_encoder_layer_output = sample_encoder_layer(\n",
        "    tf.random.uniform((64, 43, 512)), False, None)\n",
        "\n",
        "sample_encoder_layer_output.shape  # (batch_size, input_seq_len, d_model)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6LO_48Owmx_o"
      },
      "source": [
        "### Decoder layer\n",
        "\n",
        "Each decoder layer consists of sublayers:\n",
        "\n",
        "1.   Masked multi-head attention (with look ahead mask and padding mask)\n",
        "2.   Multi-head attention (with padding mask). V (value) and K (key) receive the *encoder output* as inputs. Q (query) receives the *output from the masked multi-head attention sublayer.*\n",
        "3.   Point wise feed forward networks\n",
        "\n",
        "Each of these sublayers has a residual connection around it followed by a layer normalization. The output of each sublayer is `LayerNorm(x + Sublayer(x))`. The normalization is done on the `d_model` (last) axis.\n",
        "\n",
        "There are N decoder layers in the transformer.\n",
        "\n",
        "As Q receives the output from decoder's first attention block, and K receives the encoder output, the attention weights represent the importance given to the decoder's input based on the encoder's output. In other words, the decoder predicts the next word by looking at the encoder output and self-attending to its own output. See the demonstration above in the scaled dot product attention section."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "9SoX0-vd1hue",
        "colab": {}
      },
      "source": [
        "class DecoderLayer(tf.keras.layers.Layer):\n",
        "  def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
        "    super(DecoderLayer, self).__init__()\n",
        "\n",
        "    self.mha1 = MultiHeadAttention(d_model, num_heads)\n",
        "    self.mha2 = MultiHeadAttention(d_model, num_heads)\n",
        "\n",
        "    self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
        " \n",
        "    self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
        "    self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
        "    self.layernorm3 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
        "    \n",
        "    self.dropout1 = tf.keras.layers.Dropout(rate)\n",
        "    self.dropout2 = tf.keras.layers.Dropout(rate)\n",
        "    self.dropout3 = tf.keras.layers.Dropout(rate)\n",
        "    \n",
        "    \n",
        "  def call(self, x, enc_output, training, \n",
        "           look_ahead_mask, padding_mask):\n",
        "    # enc_output.shape == (batch_size, input_seq_len, d_model)\n",
        "\n",
        "    attn1, attn_weights_block1 = self.mha1(x, x, x, look_ahead_mask)  # (batch_size, target_seq_len, d_model)\n",
        "    attn1 = self.dropout1(attn1, training=training)\n",
        "    out1 = self.layernorm1(attn1 + x)\n",
        "    \n",
        "    attn2, attn_weights_block2 = self.mha2(\n",
        "        enc_output, enc_output, out1, padding_mask)  # (batch_size, target_seq_len, d_model)\n",
        "    attn2 = self.dropout2(attn2, training=training)\n",
        "    out2 = self.layernorm2(attn2 + out1)  # (batch_size, target_seq_len, d_model)\n",
        "    \n",
        "    ffn_output = self.ffn(out2)  # (batch_size, target_seq_len, d_model)\n",
        "    ffn_output = self.dropout3(ffn_output, training=training)\n",
        "    out3 = self.layernorm3(ffn_output + out2)  # (batch_size, target_seq_len, d_model)\n",
        "    \n",
        "    return out3, attn_weights_block1, attn_weights_block2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Ne2Bqx8k71l0",
        "colab": {}
      },
      "source": [
        "sample_decoder_layer = DecoderLayer(512, 8, 2048)\n",
        "\n",
        "sample_decoder_layer_output, _, _ = sample_decoder_layer(\n",
        "    tf.random.uniform((64, 50, 512)), sample_encoder_layer_output, \n",
        "    False, None, None)\n",
        "\n",
        "sample_decoder_layer_output.shape  # (batch_size, target_seq_len, d_model)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "SE1H51Ajm0q1"
      },
      "source": [
        "### Encoder\n",
        "\n",
        "The `Encoder` consists of:\n",
        "1.   Input Embedding\n",
        "2.   Positional Encoding\n",
        "3.   N encoder layers\n",
        "\n",
        "The input is put through an embedding which is summed with the positional encoding. The output of this summation is the input to the encoder layers. The output of the encoder is the input to the decoder."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jpEox7gJ8FCI",
        "colab": {}
      },
      "source": [
        "class Encoder(tf.keras.layers.Layer):\n",
        "  def __init__(self, num_layers, d_model, num_heads, dff, input_vocab_size,\n",
        "               maximum_position_encoding, rate=0.1):\n",
        "    super(Encoder, self).__init__()\n",
        "\n",
        "    self.d_model = d_model\n",
        "    self.num_layers = num_layers\n",
        "    \n",
        "    self.embedding = tf.keras.layers.Embedding(input_vocab_size, d_model)\n",
        "    self.pos_encoding = positional_encoding(maximum_position_encoding, \n",
        "                                            self.d_model)\n",
        "    \n",
        "    \n",
        "    self.enc_layers = [EncoderLayer(d_model, num_heads, dff, rate) \n",
        "                       for _ in range(num_layers)]\n",
        "  \n",
        "    self.dropout = tf.keras.layers.Dropout(rate)\n",
        "        \n",
        "  def call(self, x, training, mask):\n",
        "\n",
        "    seq_len = tf.shape(x)[1]\n",
        "    \n",
        "    # adding embedding and position encoding.\n",
        "    x = self.embedding(x)  # (batch_size, input_seq_len, d_model)\n",
        "    x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
        "    x += self.pos_encoding[:, :seq_len, :]\n",
        "\n",
        "    x = self.dropout(x, training=training)\n",
        "    \n",
        "    for i in range(self.num_layers):\n",
        "      x = self.enc_layers[i](x, training, mask)\n",
        "    \n",
        "    return x  # (batch_size, input_seq_len, d_model)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8QG9nueFQKXx",
        "colab": {}
      },
      "source": [
        "sample_encoder = Encoder(num_layers=2, d_model=512, num_heads=8, \n",
        "                         dff=2048, input_vocab_size=8500,\n",
        "                         maximum_position_encoding=10000)\n",
        "temp_input = tf.random.uniform((64, 62), dtype=tf.int64, minval=0, maxval=200)\n",
        "\n",
        "sample_encoder_output = sample_encoder(temp_input, training=False, mask=None)\n",
        "\n",
        "print (sample_encoder_output.shape)  # (batch_size, input_seq_len, d_model)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "p-uO6ls8m2O5"
      },
      "source": [
        "### Decoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ZtT7PKzrXkNr"
      },
      "source": [
        " The `Decoder` consists of:\n",
        "1.   Output Embedding\n",
        "2.   Positional Encoding\n",
        "3.   N decoder layers\n",
        "\n",
        "The target is put through an embedding which is summed with the positional encoding. The output of this summation is the input to the decoder layers. The output of the decoder is the input to the final linear layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "d5_d5-PLQXwY",
        "colab": {}
      },
      "source": [
        "class Decoder(tf.keras.layers.Layer):\n",
        "  def __init__(self, num_layers, d_model, num_heads, dff, target_vocab_size,\n",
        "               maximum_position_encoding, rate=0.1):\n",
        "    super(Decoder, self).__init__()\n",
        "\n",
        "    self.d_model = d_model\n",
        "    self.num_layers = num_layers\n",
        "    \n",
        "    self.embedding = tf.keras.layers.Embedding(target_vocab_size, d_model)\n",
        "    self.pos_encoding = positional_encoding(maximum_position_encoding, d_model)\n",
        "    \n",
        "    self.dec_layers = [DecoderLayer(d_model, num_heads, dff, rate) \n",
        "                       for _ in range(num_layers)]\n",
        "    self.dropout = tf.keras.layers.Dropout(rate)\n",
        "    \n",
        "  def call(self, x, enc_output, training, \n",
        "           look_ahead_mask, padding_mask):\n",
        "\n",
        "    seq_len = tf.shape(x)[1]\n",
        "    attention_weights = {}\n",
        "    \n",
        "    x = self.embedding(x)  # (batch_size, target_seq_len, d_model)\n",
        "    x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
        "    x += self.pos_encoding[:, :seq_len, :]\n",
        "    \n",
        "    x = self.dropout(x, training=training)\n",
        "\n",
        "    for i in range(self.num_layers):\n",
        "      x, block1, block2 = self.dec_layers[i](x, enc_output, training,\n",
        "                                             look_ahead_mask, padding_mask)\n",
        "      \n",
        "      attention_weights['decoder_layer{}_block1'.format(i+1)] = block1\n",
        "      attention_weights['decoder_layer{}_block2'.format(i+1)] = block2\n",
        "    \n",
        "    # x.shape == (batch_size, target_seq_len, d_model)\n",
        "    return x, attention_weights"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "a1jXoAMRZyvu",
        "colab": {}
      },
      "source": [
        "sample_decoder = Decoder(num_layers=2, d_model=512, num_heads=8, \n",
        "                         dff=2048, target_vocab_size=8000,\n",
        "                         maximum_position_encoding=5000)\n",
        "temp_input = tf.random.uniform((64, 26), dtype=tf.int64, minval=0, maxval=200)\n",
        "\n",
        "output, attn = sample_decoder(temp_input, \n",
        "                              enc_output=sample_encoder_output, \n",
        "                              training=False,\n",
        "                              look_ahead_mask=None, \n",
        "                              padding_mask=None)\n",
        "\n",
        "output.shape, attn['decoder_layer2_block2'].shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "y54xnJnuYgJ7"
      },
      "source": [
        "## Create the Transformer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uERO1y54cOKq"
      },
      "source": [
        "Transformer consists of the encoder, decoder and a final linear layer. The output of the decoder is the input to the linear layer and its output is returned."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PED3bIpOYkBu",
        "colab": {}
      },
      "source": [
        "class Transformer(tf.keras.Model):\n",
        "  def __init__(self, num_layers, d_model, num_heads, dff, input_vocab_size, \n",
        "               target_vocab_size, pe_input, pe_target, rate=0.1):\n",
        "    super(Transformer, self).__init__()\n",
        "\n",
        "    self.encoder = Encoder(num_layers, d_model, num_heads, dff, \n",
        "                           input_vocab_size, pe_input, rate)\n",
        "\n",
        "    self.decoder = Decoder(num_layers, d_model, num_heads, dff, \n",
        "                           target_vocab_size, pe_target, rate)\n",
        "\n",
        "    self.final_layer = tf.keras.layers.Dense(target_vocab_size)\n",
        "    \n",
        "  def call(self, inp, tar, training, enc_padding_mask, \n",
        "           look_ahead_mask, dec_padding_mask):\n",
        "\n",
        "    enc_output = self.encoder(inp, training, enc_padding_mask)  # (batch_size, inp_seq_len, d_model)\n",
        "    \n",
        "    # dec_output.shape == (batch_size, tar_seq_len, d_model)\n",
        "    dec_output, attention_weights = self.decoder(\n",
        "        tar, enc_output, training, look_ahead_mask, dec_padding_mask)\n",
        "    \n",
        "    final_output = self.final_layer(dec_output)  # (batch_size, tar_seq_len, target_vocab_size)\n",
        "    \n",
        "    return final_output, attention_weights"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "tJ4fbQcIkHW1",
        "colab": {}
      },
      "source": [
        "sample_transformer = Transformer(\n",
        "    num_layers=2, d_model=512, num_heads=8, dff=2048, \n",
        "    input_vocab_size=8500, target_vocab_size=8000, \n",
        "    pe_input=10000, pe_target=6000)\n",
        "\n",
        "temp_input = tf.random.uniform((64, 38), dtype=tf.int64, minval=0, maxval=200)\n",
        "temp_target = tf.random.uniform((64, 36), dtype=tf.int64, minval=0, maxval=200)\n",
        "\n",
        "fn_out, _ = sample_transformer(temp_input, temp_target, training=False, \n",
        "                               enc_padding_mask=None, \n",
        "                               look_ahead_mask=None,\n",
        "                               dec_padding_mask=None)\n",
        "\n",
        "fn_out.shape  # (batch_size, tar_seq_len, target_vocab_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wsINyf1VEQLC"
      },
      "source": [
        "## Set hyperparameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zVjWCxFNcgbt"
      },
      "source": [
        "To keep this example small and relatively fast, the values for *num_layers, d_model, and dff* have been reduced. \n",
        "\n",
        "The values used in the base model of transformer were; *num_layers=6*, *d_model = 512*, *dff = 2048*. See the [paper](https://arxiv.org/abs/1706.03762) for all the other versions of the transformer.\n",
        "\n",
        "Note: By changing the values below, you can get the model that achieved state of the art on many tasks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "lnJn5SLA2ahP",
        "colab": {}
      },
      "source": [
        "num_layers = 4\n",
        "d_model = 128\n",
        "dff = 512\n",
        "num_heads = 8\n",
        "\n",
        "input_vocab_size = tokenizer_pt.vocab_size + 2\n",
        "target_vocab_size = tokenizer_en.vocab_size + 2\n",
        "dropout_rate = 0.1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xYEGhEOtzn5W"
      },
      "source": [
        "## Optimizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GOmWW--yP3zx"
      },
      "source": [
        "Use the Adam optimizer with a custom learning rate scheduler according to the formula in the [paper](https://arxiv.org/abs/1706.03762).\n",
        "\n",
        "$$\\Large{lrate = d_{model}^{-0.5} * min(step{\\_}num^{-0.5}, step{\\_}num * warmup{\\_}steps^{-1.5})}$$\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iYQdOO1axwEI",
        "colab": {}
      },
      "source": [
        "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
        "  def __init__(self, d_model, warmup_steps=4000):\n",
        "    super(CustomSchedule, self).__init__()\n",
        "    \n",
        "    self.d_model = d_model\n",
        "    self.d_model = tf.cast(self.d_model, tf.float32)\n",
        "\n",
        "    self.warmup_steps = warmup_steps\n",
        "    \n",
        "  def __call__(self, step):\n",
        "    arg1 = tf.math.rsqrt(step)\n",
        "    arg2 = step * (self.warmup_steps ** -1.5)\n",
        "    \n",
        "    return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7r4scdulztRx",
        "colab": {}
      },
      "source": [
        "learning_rate = CustomSchedule(d_model)\n",
        "\n",
        "optimizer = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98, \n",
        "                                     epsilon=1e-9)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "f33ZCgvHpPdG",
        "colab": {}
      },
      "source": [
        "temp_learning_rate_schedule = CustomSchedule(d_model)\n",
        "\n",
        "plt.plot(temp_learning_rate_schedule(tf.range(40000, dtype=tf.float32)))\n",
        "plt.ylabel(\"Learning Rate\")\n",
        "plt.xlabel(\"Train Step\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YgkDE7hzo8r5"
      },
      "source": [
        "## Loss and metrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oxGJtoDuYIHL"
      },
      "source": [
        "Since the target sequences are padded, it is important to apply a padding mask when calculating the loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MlhsJMm0TW_B",
        "colab": {}
      },
      "source": [
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "    from_logits=True, reduction='none')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "67oqVHiT0Eiu",
        "colab": {}
      },
      "source": [
        "def loss_function(real, pred):\n",
        "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
        "  loss_ = loss_object(real, pred)\n",
        "\n",
        "  mask = tf.cast(mask, dtype=loss_.dtype)\n",
        "  loss_ *= mask\n",
        "  \n",
        "  return tf.reduce_mean(loss_)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "phlyxMnm-Tpx",
        "colab": {}
      },
      "source": [
        "train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
        "train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(\n",
        "    name='train_accuracy')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aeHumfr7zmMa"
      },
      "source": [
        "## Training and checkpointing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "UiysUa--4tOU",
        "colab": {}
      },
      "source": [
        "transformer = Transformer(num_layers, d_model, num_heads, dff,\n",
        "                          input_vocab_size, target_vocab_size, \n",
        "                          pe_input=input_vocab_size, \n",
        "                          pe_target=target_vocab_size,\n",
        "                          rate=dropout_rate)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ZOJUSB1T8GjM",
        "colab": {}
      },
      "source": [
        "def create_masks(inp, tar):\n",
        "  # Encoder padding mask\n",
        "  enc_padding_mask = create_padding_mask(inp)\n",
        "  \n",
        "  # Used in the 2nd attention block in the decoder.\n",
        "  # This padding mask is used to mask the encoder outputs.\n",
        "  dec_padding_mask = create_padding_mask(inp)\n",
        "  \n",
        "  # Used in the 1st attention block in the decoder.\n",
        "  # It is used to pad and mask future tokens in the input received by \n",
        "  # the decoder.\n",
        "  look_ahead_mask = create_look_ahead_mask(tf.shape(tar)[1])\n",
        "  dec_target_padding_mask = create_padding_mask(tar)\n",
        "  combined_mask = tf.maximum(dec_target_padding_mask, look_ahead_mask)\n",
        "  \n",
        "  return enc_padding_mask, combined_mask, dec_padding_mask"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Fzuf06YZp66w"
      },
      "source": [
        "Create the checkpoint path and the checkpoint manager. This will be used to save checkpoints every `n` epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hNhuYfllndLZ",
        "colab": {}
      },
      "source": [
        "checkpoint_path = \"./checkpoints/train\"\n",
        "\n",
        "ckpt = tf.train.Checkpoint(transformer=transformer,\n",
        "                           optimizer=optimizer)\n",
        "\n",
        "ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)\n",
        "\n",
        "# if a checkpoint exists, restore the latest checkpoint.\n",
        "if ckpt_manager.latest_checkpoint:\n",
        "  ckpt.restore(ckpt_manager.latest_checkpoint)\n",
        "  print ('Latest checkpoint restored!!')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0Di_Yaa1gf9r"
      },
      "source": [
        "The target is divided into tar_inp and tar_real. tar_inp is passed as an input to the decoder. `tar_real` is that same input shifted by 1: At each location in `tar_input`, `tar_real` contains the  next token that should be predicted.\n",
        "\n",
        "For example, `sentence` = \"SOS A lion in the jungle is sleeping EOS\"\n",
        "\n",
        "`tar_inp` =  \"SOS A lion in the jungle is sleeping\"\n",
        "\n",
        "`tar_real` = \"A lion in the jungle is sleeping EOS\"\n",
        "\n",
        "The transformer is an auto-regressive model: it makes predictions one part at a time, and uses its output so far to decide what to do next. \n",
        "\n",
        "During training this example uses teacher-forcing (like in the [text generation tutorial](./text_generation.ipynb)). Teacher forcing is passing the true output to the next time step regardless of what the model predicts at the current time step.\n",
        "\n",
        "As the transformer predicts each word, *self-attention* allows it to look at the previous words in the input sequence to better predict the next word.\n",
        "\n",
        "To prevent the model from peaking at the expected output the model uses a look-ahead mask."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "LKpoA6q1sJFj",
        "colab": {}
      },
      "source": [
        "EPOCHS = 20"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iJwmp9OE29oj",
        "colab": {}
      },
      "source": [
        "# The @tf.function trace-compiles train_step into a TF graph for faster\n",
        "# execution. The function specializes to the precise shape of the argument\n",
        "# tensors. To avoid re-tracing due to the variable sequence lengths or variable\n",
        "# batch sizes (the last batch is smaller), use input_signature to specify\n",
        "# more generic shapes.\n",
        "\n",
        "train_step_signature = [\n",
        "    tf.TensorSpec(shape=(None, None), dtype=tf.int64),\n",
        "    tf.TensorSpec(shape=(None, None), dtype=tf.int64),\n",
        "]\n",
        "\n",
        "@tf.function(input_signature=train_step_signature)\n",
        "def train_step(inp, tar):\n",
        "  tar_inp = tar[:, :-1]\n",
        "  tar_real = tar[:, 1:]\n",
        "  \n",
        "  enc_padding_mask, combined_mask, dec_padding_mask = create_masks(inp, tar_inp)\n",
        "  \n",
        "  with tf.GradientTape() as tape:\n",
        "    predictions, _ = transformer(inp, tar_inp, \n",
        "                                 True, \n",
        "                                 enc_padding_mask, \n",
        "                                 combined_mask, \n",
        "                                 dec_padding_mask)\n",
        "    loss = loss_function(tar_real, predictions)\n",
        "\n",
        "  gradients = tape.gradient(loss, transformer.trainable_variables)    \n",
        "  optimizer.apply_gradients(zip(gradients, transformer.trainable_variables))\n",
        "  \n",
        "  train_loss(loss)\n",
        "  train_accuracy(tar_real, predictions)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qM2PDWGDJ_8V"
      },
      "source": [
        "Portuguese is used as the input language and English is the target language."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "bbvmaKNiznHZ",
        "colab": {}
      },
      "source": [
        "for epoch in range(EPOCHS):\n",
        "  start = time.time()\n",
        "  \n",
        "  train_loss.reset_states()\n",
        "  train_accuracy.reset_states()\n",
        "  \n",
        "  # inp -> portuguese, tar -> english\n",
        "  for (batch, (inp, tar)) in enumerate(train_dataset):\n",
        "    train_step(inp, tar)\n",
        "    \n",
        "    if batch % 50 == 0:\n",
        "      print ('Epoch {} Batch {} Loss {:.4f} Accuracy {:.4f}'.format(\n",
        "          epoch + 1, batch, train_loss.result(), train_accuracy.result()))\n",
        "      \n",
        "  if (epoch + 1) % 5 == 0:\n",
        "    ckpt_save_path = ckpt_manager.save()\n",
        "    print ('Saving checkpoint for epoch {} at {}'.format(epoch+1,\n",
        "                                                         ckpt_save_path))\n",
        "    \n",
        "  print ('Epoch {} Loss {:.4f} Accuracy {:.4f}'.format(epoch + 1, \n",
        "                                                train_loss.result(), \n",
        "                                                train_accuracy.result()))\n",
        "\n",
        "  print ('Time taken for 1 epoch: {} secs\\n'.format(time.time() - start))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QfcsSWswSdGV"
      },
      "source": [
        "## Evaluate"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "y6APsFrgImLW"
      },
      "source": [
        "The following steps are used for evaluation:\n",
        "\n",
        "* Encode the input sentence using the Portuguese tokenizer (`tokenizer_pt`). Moreover, add the start and end token so the input is equivalent to what the model is trained with. This is the encoder input.\n",
        "* The decoder input is the `start token == tokenizer_en.vocab_size`.\n",
        "* Calculate the padding masks and the look ahead masks.\n",
        "* The `decoder` then outputs the predictions by looking at the `encoder output` and its own output (self-attention).\n",
        "* Select the last word and calculate the argmax of that.\n",
        "* Concatentate the predicted word to the decoder input as pass it to the decoder.\n",
        "* In this approach, the decoder predicts the next word based on the previous words it predicted.\n",
        "\n",
        "Note: The model used here has less capacity to keep the example relatively faster so the predictions maybe less right. To reproduce the results in the paper, use the entire dataset and base transformer model or transformer XL, by changing the hyperparameters above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "5buvMlnvyrFm",
        "colab": {}
      },
      "source": [
        "def evaluate(inp_sentence):\n",
        "  start_token = [tokenizer_pt.vocab_size]\n",
        "  end_token = [tokenizer_pt.vocab_size + 1]\n",
        "  \n",
        "  # inp sentence is portuguese, hence adding the start and end token\n",
        "  inp_sentence = start_token + tokenizer_pt.encode(inp_sentence) + end_token\n",
        "  encoder_input = tf.expand_dims(inp_sentence, 0)\n",
        "  \n",
        "  # as the target is english, the first word to the transformer should be the\n",
        "  # english start token.\n",
        "  decoder_input = [tokenizer_en.vocab_size]\n",
        "  output = tf.expand_dims(decoder_input, 0)\n",
        "    \n",
        "  for i in range(MAX_LENGTH):\n",
        "    enc_padding_mask, combined_mask, dec_padding_mask = create_masks(\n",
        "        encoder_input, output)\n",
        "  \n",
        "    # predictions.shape == (batch_size, seq_len, vocab_size)\n",
        "    predictions, attention_weights = transformer(encoder_input, \n",
        "                                                 output,\n",
        "                                                 False,\n",
        "                                                 enc_padding_mask,\n",
        "                                                 combined_mask,\n",
        "                                                 dec_padding_mask)\n",
        "    \n",
        "    # select the last word from the seq_len dimension\n",
        "    predictions = predictions[: ,-1:, :]  # (batch_size, 1, vocab_size)\n",
        "\n",
        "    predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n",
        "    \n",
        "    # return the result if the predicted_id is equal to the end token\n",
        "    if predicted_id == tokenizer_en.vocab_size+1:\n",
        "      return tf.squeeze(output, axis=0), attention_weights\n",
        "    \n",
        "    # concatentate the predicted_id to the output which is given to the decoder\n",
        "    # as its input.\n",
        "    output = tf.concat([output, predicted_id], axis=-1)\n",
        "\n",
        "  return tf.squeeze(output, axis=0), attention_weights"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "CN-BV43FMBej",
        "colab": {}
      },
      "source": [
        "def plot_attention_weights(attention, sentence, result, layer):\n",
        "  fig = plt.figure(figsize=(16, 8))\n",
        "  \n",
        "  sentence = tokenizer_pt.encode(sentence)\n",
        "  \n",
        "  attention = tf.squeeze(attention[layer], axis=0)\n",
        "  \n",
        "  for head in range(attention.shape[0]):\n",
        "    ax = fig.add_subplot(2, 4, head+1)\n",
        "    \n",
        "    # plot the attention weights\n",
        "    ax.matshow(attention[head][:-1, :], cmap='viridis')\n",
        "\n",
        "    fontdict = {'fontsize': 10}\n",
        "    \n",
        "    ax.set_xticks(range(len(sentence)+2))\n",
        "    ax.set_yticks(range(len(result)))\n",
        "    \n",
        "    ax.set_ylim(len(result)-1.5, -0.5)\n",
        "        \n",
        "    ax.set_xticklabels(\n",
        "        ['<start>']+[tokenizer_pt.decode([i]) for i in sentence]+['<end>'], \n",
        "        fontdict=fontdict, rotation=90)\n",
        "    \n",
        "    ax.set_yticklabels([tokenizer_en.decode([i]) for i in result \n",
        "                        if i < tokenizer_en.vocab_size], \n",
        "                       fontdict=fontdict)\n",
        "    \n",
        "    ax.set_xlabel('Head {}'.format(head+1))\n",
        "  \n",
        "  plt.tight_layout()\n",
        "  plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "lU2_yG_vBGza",
        "colab": {}
      },
      "source": [
        "def translate(sentence, plot=''):\n",
        "  result, attention_weights = evaluate(sentence)\n",
        "  \n",
        "  predicted_sentence = tokenizer_en.decode([i for i in result \n",
        "                                            if i < tokenizer_en.vocab_size])  \n",
        "\n",
        "  print('Input: {}'.format(sentence))\n",
        "  print('Predicted translation: {}'.format(predicted_sentence))\n",
        "  \n",
        "  if plot:\n",
        "    plot_attention_weights(attention_weights, sentence, result, plot)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "YsxrAlvFG8SZ",
        "colab": {}
      },
      "source": [
        "translate(\"este é um problema que temos que resolver.\")\n",
        "print (\"Real translation: this is a problem we have to solve .\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7EH5y_aqI4t1",
        "colab": {}
      },
      "source": [
        "translate(\"os meus vizinhos ouviram sobre esta ideia.\")\n",
        "print (\"Real translation: and my neighboring homes heard about this idea .\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "J-hVCTSUMlkb",
        "colab": {}
      },
      "source": [
        "translate(\"vou então muito rapidamente partilhar convosco algumas histórias de algumas coisas mágicas que aconteceram.\")\n",
        "print (\"Real translation: so i 'll just share with you some stories very quickly of some magical things that have happened .\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_1MxkSZvz0jX"
      },
      "source": [
        "You can pass different layers and attention blocks of the decoder to the `plot` parameter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "t-kFyiOLH0xg",
        "colab": {}
      },
      "source": [
        "translate(\"este é o primeiro livro que eu fiz.\", plot='decoder_layer4_block2')\n",
        "print (\"Real translation: this is the first book i've ever done.\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RqQ1fIsLwkGE"
      },
      "source": [
        "## Summary\n",
        "\n",
        "In this tutorial, you learned about positional encoding, multi-head attention, the importance of masking and how to create a transformer.\n",
        "\n",
        "Try using a different dataset to train the transformer. You can also create the base transformer or transformer XL by changing the hyperparameters above. You can also use the layers defined here to create [BERT](https://arxiv.org/abs/1810.04805) and train state of the art models. Futhermore, you can implement beam search to get better predictions."
      ]
    }
  ]
}