English_French_NMT.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "English_French_NMT.ipynb",
      "provenance": [],
      "machine_shape": "hm",
      "authorship_tag": "ABX9TyOM6TjmkxwQhYMmF+n7maBb",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mshadloo/Neural-Machine-Translation-with-Attention/blob/master/English_French_NMT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TGIiiAGO-0Xn"
      },
      "source": [
        "# Neural Machine Translation\n",
        "Machine translation is the task of automatically converting source text in one language to text in another language. In a machine translation task, the input already consists of a sequence of symbols in some language, and the computer program must convert this into a sequence of symbols in another language. Given a sequence of symbols in a source language, there is no one single best translation of that sequence to another language. This is because of the natural ambiguity and flexibility of human language. This makes the challenge of automatic machine translation difficult, perhaps one of the most difficult in artificial intelligence.\n",
        "\n",
        "Machine Translation can be thought of as a Seq2Seq learning problem. Recurrent Neural Networks are the incumbent technology for this learning problem. A typical Recurrent Neural Networks model for Seq2Seq problem consists of an encoder and a decoder which are themselves two separate neural networks combined into a single giant network. Both encoder and decoder are typically LSTM or GRU models.\n",
        "\n",
        "# Neural Machine Translation with Attention\n",
        "To translate a sentence from a language to another one, a human translator reads the sentence part by part, and generates part of translation. A neural machine translation with attention like a human translator looks at the sentence part by part. To generate each part of translation, the attention mechanism tells a Neural Machine Translation model where it should pay attention to."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j61trOvl94QA"
      },
      "source": [
        "### Project\n",
        "I implement encoder-decoder based seq2seq models with attention. The encoder can be a Bidirectional LSTM, a simple LSTM or GRU, and the decoder can be LSTM or GRU. I have a argument for encoder type (RNN model used in encoder); it can be 'bidirectional', 'lstm' or 'gru'. When this argument is set to 'bidirectional', the model has Bidirectional LSTM as enocder a simple LSTM as decoder. When it is set to 'lstm', the encoder and decoder are both simple LSTMs, and for the 'gru' value, they are GRUs. Thus, I can have different three models. \n",
        "\n",
        "To evaluate the models, I use English-French dataset provided by http:https://www.manythings.org/anki/.\n",
        "\n",
        "I run these models and save the results. The experiments show that the model with a Bidirectional LSTM as the encoder outperforms the rest.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6kzV802gXheX"
      },
      "source": [
        "### Google Colab\n",
        "To train NMT on real-world translation data you need GPUs. You can use GPUs of Google Colab.  Even on GPUs training can take days. While you are using Google Colab and if the runtime restarts during training, you will lose your trained model. Then you have to start again from the scratch, which is not optimal. Instead, you should save your model checkpoint to Google Drive and reload it next time when you start. To do so, you need to mount your Google Drive  and give permission to Google Colab to access it. Also you need to create a directory in your Google Drive for this project. \n",
        "\n",
        "\n",
        "The code below mounts Google Drive and creates the folder 'NMT_with_attention' for this project in your Google Drive. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9WBFQR6usmuO",
        "outputId": "17c967b9-51d6-4d1c-9248-4c2aa87d1f1c"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "working_dir = './drive/MyDrive/NMT_with_attention'\n",
        "import os\n",
        "if not os.path.exists(working_dir):\n",
        "      os.makedirs(working_dir)"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hj0CDCrgrqoX"
      },
      "source": [
        "Import basic libraries."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uGohmSI48h_v"
      },
      "source": [
        "import pickle\n",
        "from collections import Counter, defaultdict\n",
        "from unicodedata import normalize\n",
        "import re\n",
        "import numpy as np\n",
        "\n",
        "import os\n",
        "from keras.layers import Bidirectional, Concatenate, Permute, Dot, Input, LSTM, Multiply\n",
        "import tensorflow as tf\n",
        "from keras.optimizers import Adam\n",
        "from keras.utils import to_categorical\n",
        "from keras.models import load_model, Model\n",
        "import keras.backend as K\n",
        "\n",
        "from keras.layers import Embedding, Input, LSTM, GRU, Dense, Bidirectional, RepeatVector, Concatenate, Activation, Dot, Lambda\n",
        "from sklearn.model_selection import train_test_split\n",
        "import numpy as np\n",
        "from keras.models import load_model"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LeGnJV00_Kof"
      },
      "source": [
        "## Download The **Dataset** "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jRWKVyEr_VUY",
        "outputId": "c61d522d-c518-445a-aa83-2dec172dc375"
      },
      "source": [
        "!mkdir ./drive/MyDrive/NMT_with_attention/data\n",
        "!wget -O ./drive/MyDrive/NMT_with_attention/fra-eng.zip http:https://www.manythings.org/anki/fra-eng.zip\n",
        "!unzip  './drive/MyDrive/NMT_with_attention/fra-eng.zip' -d './drive/MyDrive/NMT_with_attention/data'\n",
        "!rm ./drive/MyDrive/NMT_with_attention/fra-eng.zip "
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-04-15 21:50:14--  http:https://www.manythings.org/anki/fra-eng.zip\n",
            "Resolving www.manythings.org (www.manythings.org)... 172.67.173.198, 104.21.55.222, 2606:4700:3036::ac43:adc6, ...\n",
            "Connecting to www.manythings.org (www.manythings.org)|172.67.173.198|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 6281268 (6.0M) [application/zip]\n",
            "Saving to: ‘./drive/MyDrive/NMT_with_attention/fra-eng.zip’\n",
            "\n",
            "./drive/MyDrive/NMT 100%[===================>]   5.99M  5.32MB/s    in 1.1s    \n",
            "\n",
            "2021-04-15 21:50:15 (5.32 MB/s) - ‘./drive/MyDrive/NMT_with_attention/fra-eng.zip’ saved [6281268/6281268]\n",
            "\n",
            "Archive:  ./drive/MyDrive/NMT_with_attention/fra-eng.zip\n",
            "  inflating: ./drive/MyDrive/NMT_with_attention/data/_about.txt  \n",
            "  inflating: ./drive/MyDrive/NMT_with_attention/data/fra.txt  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g-yLHGCwAE7N"
      },
      "source": [
        "Now you have one file:\n",
        "* fra.txt\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SICszQ1ihjav"
      },
      "source": [
        "## Download the Glove Word Embedding\n",
        "I use word feature vector data from [*nlp.standford.edu*](https://nlp.stanford.edu/data/glove.6B.zip) to initialize the embedding layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SVs6sv_1aKrs",
        "outputId": "db313a1e-0f94-4437-b615-73909dcb65f8"
      },
      "source": [
        "!mkdir ./drive/MyDrive/NMT_with_attention/embedding\n",
        "!wget -O ./drive/MyDrive/NMT_with_attention/glove.6B.zip https://nlp.stanford.edu/data/glove.6B.zip\n",
        "!unzip  './drive/MyDrive/NMT_with_attention/glove.6B.zip' -d './drive/MyDrive/NMT_with_attention/embedding'\n",
        "!rm ./drive/MyDrive/NMT_with_attention/glove.6B.zip"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-04-15 21:59:03--  https://nlp.stanford.edu/data/glove.6B.zip\n",
            "Resolving nlp.stanford.edu (nlp.stanford.edu)... 171.64.67.140\n",
            "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: http:https://downloads.cs.stanford.edu/nlp/data/glove.6B.zip [following]\n",
            "--2021-04-15 21:59:04--  http:https://downloads.cs.stanford.edu/nlp/data/glove.6B.zip\n",
            "Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 171.64.64.22\n",
            "Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|171.64.64.22|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 862182613 (822M) [application/zip]\n",
            "Saving to: ‘./drive/MyDrive/NMT_with_attention/glove.6B.zip’\n",
            "\n",
            "./drive/MyDrive/NMT 100%[===================>] 822.24M  5.24MB/s    in 2m 43s  \n",
            "\n",
            "2021-04-15 22:01:48 (5.04 MB/s) - ‘./drive/MyDrive/NMT_with_attention/glove.6B.zip’ saved [862182613/862182613]\n",
            "\n",
            "Archive:  ./drive/MyDrive/NMT_with_attention/glove.6B.zip\n",
            "  inflating: ./drive/MyDrive/NMT_with_attention/embedding/glove.6B.50d.txt  \n",
            "  inflating: ./drive/MyDrive/NMT_with_attention/embedding/glove.6B.100d.txt  \n",
            "  inflating: ./drive/MyDrive/NMT_with_attention/embedding/glove.6B.200d.txt  \n",
            "  inflating: ./drive/MyDrive/NMT_with_attention/embedding/glove.6B.300d.txt  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wQghc2PKrvdo"
      },
      "source": [
        "Set basic parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v0XrbLbU8lDa"
      },
      "source": [
        "import easydict\n",
        "args = easydict.EasyDict({\n",
        "    \"rnn_arch\":\"bidirectional\",\n",
        "    \"epochs\": 20, 'embedding_dim':200,'hidden':1024, 'checkpoint':'checkpoint','dataset_file':'fra.txt', 'result':'result', 'glove':True,'data_dir':'data',\n",
        "    \"batch_size\": 128, \"lr\":0.1, 'momentum':0.9, 'weight_decay':5e-4,'embedding_dir' :'embedding'\n",
        "})\n",
        "\n",
        "\n",
        "rnn_arch = ['gru', 'lstm', 'bidirectional']\n",
        "embed_dim = ['50','100','200','300']\n",
        "data_dir = os.path.join(working_dir, args.data_dir )\n",
        "checkpoint_dir = os.path.join(working_dir, args.checkpoint +\"_\"+args.rnn_arch )\n",
        "if not os.path.exists(checkpoint_dir):\n",
        "      os.makedirs(checkpoint_dir)\n",
        "result_dir = os.path.join(working_dir, args.result+\"_\"+args.rnn_arch )\n",
        "if not os.path.exists(result_dir):\n",
        "      os.makedirs(result_dir)\n",
        "embedding_dir = os.path.join(working_dir,args.embedding_dir)\n",
        "\n",
        "embedding = 'glove.6B.'+str(args.embedding_dim)+'d.txt'"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B8qCdie7rnoo"
      },
      "source": [
        "## Data Preprocessing\n",
        "\n",
        "The data needs some cleaning before being used to train our neural translation model.\n",
        "1. Normalizing case to lowercase.\n",
        "2. Removing punctuation from each word.\n",
        "3. Removing non-printable characters.\n",
        "4. Converting French characters to Latin characters.\n",
        "5. Removing words that contain non-alphabetic characters. \n",
        "6. Add a special token $<eos>$ at the end of target sentences\n",
        "7.  Create two dictionaries mapping from each word in vocabulary to an id, and the id to the word. \n",
        "8.  Mark all out of vocabulary (OOV) words with a special token $<unk>$\n",
        "9. Pad each sentence to a maximum length by adding special token $<pad>$ at the end of the sentence.\n",
        "10. Convert each sentence to its feature vector:\n",
        "11.  Map each token in the sentence to one-hot encoding based on its id"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Plo2W4U7qa41"
      },
      "source": [
        "def load_data(file):\n",
        "    lines = open(file, encoding='UTF-8').read().strip().split('\\n')\n",
        "    sentence_pairs = []\n",
        "    for line in lines:\n",
        "        if '\\t' not in line:\n",
        "            continue\n",
        "\n",
        "        s1, s2, _ = line.rstrip().split('\\t')\n",
        "        sentence_pairs.append([s1, s2])\n",
        "    return sentence_pairs\n",
        "\n",
        "def filter(sentence_pairs, Tx, Ty):\n",
        "  # import pdb; pdb.set_trace()\n",
        "  lengths = [ [len(s1.split()), len(s2.split())] for s1,s2 in sentence_pairs]\n",
        "  good = [ True if (l1 <=Tx) and (l2 <=Ty) else False for l1,l2 in lengths]\n",
        "  filtered = [s for i,s in enumerate(sentence_pairs) if good[i]]\n",
        "  return filtered\n",
        "\n",
        "\n",
        "def unicode_to_ascii(s):\n",
        "    s = normalize('NFD', s).encode('ascii', 'ignore')\n",
        "    return s.decode('UTF-8')\n",
        "\n",
        "\n",
        "def clean_sentence(sentence):\n",
        "    sentence = unicode_to_ascii(sentence.lower().strip())\n",
        "\n",
        "    # creating a space between a word and the punctuation following it. Ex: \"he is a boy.\" => \"he is a boy .\"\n",
        "    # Reference:- https://stackoverflow.com/questions/3645931/python-padding-punctuation-with-white-spaces-keeping-punctuation\n",
        "    sentence = re.sub(r\"([?.!,])\", r\" \\1 \", sentence)\n",
        "    sentence = re.sub(r'[\" \"]+', \" \", sentence)\n",
        "\n",
        "    # replacing everything with space except (a-z, A-Z, \".\", \"?\", \"!\", \",\")\n",
        "    sentence = re.sub(r\"[^a-zA-Z?.!,]+\", \" \", sentence)\n",
        "\n",
        "    sentence = sentence.rstrip().strip()\n",
        "    return sentence\n",
        "\n",
        "\n",
        "class LanguageVocab:\n",
        "    def __init__(self, sentences):\n",
        "        self.vocab = self.make_vocab(sentences)\n",
        "        self.vocab.update({'<eos>', '<sos>'})\n",
        "        self.word_idx = self.word_index()\n",
        "        self.idx_word = self.reverse_word_index()\n",
        "\n",
        "    def make_vocab(self, sentences, min_occurance=3):\n",
        "        token_count = Counter()\n",
        "        for sentence in sentences:\n",
        "            tokens = sentence.split()\n",
        "            token_count.update(tokens)\n",
        "        print(\"total vocab-before triming:\", len(token_count))\n",
        "        vocab = [k for k, c in token_count.items() if c >= min_occurance]\n",
        "        print(\"total vocab-after triming:\", len(vocab))\n",
        "        return set(vocab)\n",
        "\n",
        "    def word_index(self):\n",
        "        vocab = sorted(self.vocab)\n",
        "        return dict(zip(['<pad>'] + vocab + ['<unk>'], list(range(len(vocab) + 2))))\n",
        "\n",
        "    def reverse_word_index(self):\n",
        "        return {v: k for k, v in self.word_idx.items()}\n",
        "\n",
        "\n",
        "def max_length(sentences):\n",
        "    lengths = [len(s.split()) for s in sentences]\n",
        "    return max(lengths)\n",
        "\n",
        "\n",
        "def features(sentence, language_vocab, max_length):\n",
        "    tokens = sentence.split()\n",
        "\n",
        "    tokens = [token if token in language_vocab.vocab else '<unk>' for token in tokens]\n",
        "\n",
        "    tokens.extend(['<pad>'] * (max_length - len(tokens)))\n",
        "    rep = list(map(lambda x: language_vocab.word_idx[x], tokens))\n",
        "    return rep\n",
        "def preprocess_sentences(sentences):\n",
        "\n",
        "  language_vocab = LanguageVocab(sentences)\n",
        "  lang_max_length = max_length(sentences)\n",
        "  X = np.array([features(s,language_vocab, lang_max_length) for s in sentences])\n",
        "  return X, language_vocab, lang_max_length\n",
        "\n",
        "def save_pairs_dict(sentence_pairs):\n",
        "  inp_ref_dict = defaultdict(list)\n",
        "  for s1,s2 in sentence_pairs:\n",
        "    inp_ref_dict[s1].append(s2)\n",
        "\n",
        "def prepare_data(sentence_pairs, num_examples=0, Tx = 15, Ty=18):\n",
        "    clean_sentence_pairs = [[clean_sentence(s1),clean_sentence(s2)]  for s1,s2 in sentence_pairs]\n",
        "    clean_sentence_pairs = filter(clean_sentence_pairs, Tx, Ty)\n",
        "    if num_examples > 0:\n",
        "        clean_sentence_pairs = clean_sentence_pairs[0:num_examples]\n",
        "    input_sentences = [s1 for s1, s2 in clean_sentence_pairs]\n",
        "    target_sentences = [s2 for s1, s2 in clean_sentence_pairs]\n",
        "    X, inp_vocab, inp_length = preprocess_sentences(input_sentences)\n",
        "    Y, targ_vocab, targ_length = preprocess_sentences(target_sentences)\n",
        "    return X, Y, inp_vocab, targ_vocab, inp_length, targ_length"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oUzL9VH_svGm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a7771a91-b3cb-4f44-cc02-2dee343c8a55"
      },
      "source": [
        "sentence_pairs = load_data(os.path.join(data_dir, args.dataset_file))\n",
        "X, Y, inp_vocab, targ_vocab, Tx, Ty = prepare_data(sentence_pairs)\n",
        "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2)\n",
        "del X\n",
        "del Y\n",
        "inp_vocab_size = len(inp_vocab.word_idx)\n",
        "targ_vocab_size = len(targ_vocab.word_idx)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total vocab-before triming: 14081\n",
            "total vocab-after triming: 7885\n",
            "total vocab-before triming: 23464\n",
            "total vocab-after triming: 11641\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9gz0_zud9Tjh"
      },
      "source": [
        "# Define Model \n",
        "Here are some properties of the model that you may notice: \n",
        "- There are two separate RNNs in this model : pre-attention and post-attention RNNs on both sides of the attention mechanism. Pre-attention RNN is the encoder and the post-attention RNN is the decoder.\n",
        "- Encoder:\n",
        "     - The encoder goes through $T_x$ time steps\n",
        "     - Output sequence (hidden states) of the encoder is input of the attention mechanism.\n",
        "     \n",
        "- Decoder\n",
        "     - The decoder goes through $T_y$ time steps. \n",
        "- The attention mechanism computes the context variable $context^{\\langle t \\rangle}$ for each timestep in the output ($t=1, \\ldots, T_y$)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "47mjjzobBcyP"
      },
      "source": [
        "#### one_step_attention\n",
        "* The inputs to the one_step_attention at time step $t$ are:\n",
        "    * $[a^{\\<1\\>},a^{\\<2\\>}, ..., a^{\\<T_x\\>}]$: all hidden states of the  pre-attention Bi-LSTM.\n",
        "    - $s^{\\<t-1\\>}$: the previous hidden state of the post-attention LSTM \n",
        "* one_step_attention computes:\n",
        "    - $[\\alpha^{<t,1>},\\alpha^{<t,2>}, ..., \\alpha^{<t,T_x>}]$: the attention weights\n",
        "    - $context^{ \\langle t \\rangle }$: the context vector:\n",
        "    \n",
        "$$context^{<t>} = \\sum_{t' = 1}^{T_x} \\alpha^{<t,t'>}a^{<t'>}\\tag{1}$$ "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hnUYVsmx9VlE",
        "outputId": "e17d1caf-2095-45b0-ec36-18d7c92d9f46"
      },
      "source": [
        "# setting\n",
        "HIDDEN_UNITS = args.hidden\n",
        "EMBEDDING_DIM = args.embedding_dim\n",
        "encoder_units = HIDDEN_UNITS\n",
        "decoder_units = HIDDEN_UNITS\n",
        "print(HIDDEN_UNITS)\n",
        "print(EMBEDDING_DIM)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1024\n",
            "200\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W11qsG9J9bng",
        "outputId": "76fbcedb-ba74-4c3b-98ce-d89b95fb9644"
      },
      "source": [
        "# load in word vectors in a dict\n",
        "word_embedding = np.zeros((inp_vocab_size, EMBEDDING_DIM))\n",
        "if args.glove:\n",
        "    wordVec = {}\n",
        "\n",
        "    print('Loading wordVec')\n",
        "    with open(os.path.join(embedding_dir, embedding)) as f:\n",
        "        for line in f:\n",
        "            data = line.split()\n",
        "            word = data[0]\n",
        "            vec = np.asarray(data[1:], dtype='float32')\n",
        "            wordVec[word] = vec\n",
        "\n",
        "    print('Finished loading wordVec.')\n",
        "\n",
        "\n",
        "    # create word embedding by fetching each word vector\n",
        "    for tok, idx in inp_vocab.word_idx.items():\n",
        "        if idx < inp_vocab_size:\n",
        "            word_vector = wordVec.get(tok)\n",
        "            if word_vector is not None:\n",
        "                word_embedding[idx] = word_vector"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading wordVec\n",
            "Finished loading wordVec.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WwmhD2e299kX"
      },
      "source": [
        "def lstm(units,return_sequences=False, return_state=False):\n",
        "  # If you have a GPU, we recommend using CuDNNGRU(provides a 3x speedup than GRU)\n",
        "  # the code automatically does that.\n",
        "  if tf.test.is_gpu_available():\n",
        "    return tf.compat.v1.keras.layers.CuDNNLSTM(units,\n",
        "                                    return_sequences=return_sequences,\n",
        "                                    return_state=return_state,\n",
        "                                    recurrent_initializer='glorot_uniform')\n",
        "  else:\n",
        "    return LSTM(units, return_sequences=return_sequences, return_state=return_state,  recurrent_activation='sigmoid', recurrent_initializer='glorot_uniform')\n",
        "\n",
        "\n",
        "def bidirectional(units,return_sequences=False, return_state=False):\n",
        "    return  Bidirectional(lstm(units,return_sequences, return_state))\n",
        "\n",
        "\n",
        "def gru(units, return_sequences=False, return_state=False):\n",
        "  # If you have a GPU, we recommend using CuDNNGRU(provides a 3x speedup than GRU)\n",
        "  # the code automatically does that.\n",
        "  if tf.test.is_gpu_available():\n",
        "    return tf.compat.v1.keras.layers.CuDNNGRU(units,\n",
        "                                    return_sequences=return_sequences,\n",
        "                                    return_state=return_state,\n",
        "                                    recurrent_initializer='glorot_uniform')\n",
        "  else:\n",
        "    return tf.keras.layers.GRU(units,\n",
        "                               return_sequences=return_sequences,\n",
        "                               return_state=return_state,\n",
        "                               recurrent_activation='sigmoid',\n",
        "                               recurrent_initializer='glorot_uniform')\n",
        "\n",
        "\n",
        "rnn_archs = {'lstm': lstm, 'gru':gru, 'bidirectional':bidirectional}"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yHzvTi2y9_5u"
      },
      "source": [
        "def softmax(x, axis=1):\n",
        "    \"\"\"Softmax activation function.\n",
        "    # Arguments\n",
        "        x : Tensor.\n",
        "        axis: Integer, axis along which the softmax normalization is applied.\n",
        "    # Returns\n",
        "        Tensor, output of softmax transformation.\n",
        "    # Raises\n",
        "        ValueError: In case `dim(x) == 1`.\n",
        "    \"\"\"\n",
        "    ndim = K.ndim(x)\n",
        "    if ndim == 2:\n",
        "        return K.softmax(x)\n",
        "    elif ndim > 2:\n",
        "        e = K.exp(x - K.max(x, axis=axis, keepdims=True))\n",
        "        s = K.sum(e, axis=axis, keepdims=True)\n",
        "        return e / s\n",
        "    else:\n",
        "        raise ValueError('Cannot apply softmax to a tensor that is 1D')\n",
        "class AttentionLayer:\n",
        "    def __init__(self, Tx):\n",
        "        self.repeator = RepeatVector(Tx)\n",
        "        self.concatenator = Concatenate(axis=-1)\n",
        "        self.densor1 = Dense(1024, activation=\"tanh\")\n",
        "        self.densor2 = Dense(1)\n",
        "        self.activator = Activation(softmax,\n",
        "                               name='attention_weights')  # We are using a custom softmax(axis = 1) loaded in this notebook\n",
        "        self.dotor = Dot(axes=1)\n",
        "\n",
        "    def one_step_attention(self,a, s_prev):\n",
        "        \"\"\"\n",
        "        Performs one step of attention: Outputs a context vector computed as a dot product of the attention weights\n",
        "        \"alphas\" and the hidden states \"a\" of the Bi-LSTM.\n",
        "\n",
        "        Arguments:\n",
        "        a -- hidden state output of the Bi-LSTM, numpy-array of shape (m, Tx, 2*n_a)\n",
        "        s_prev -- previous hidden state of the (post-attention) LSTM, numpy-array of shape (m, n_s)\n",
        "\n",
        "        Returns:\n",
        "        context -- context vector, input of the next (post-attention) LSTM cell\n",
        "        \"\"\"\n",
        "        # Use repeator to repeat s_prev to be of shape (m, Tx, n_s) so that you can concatenate it with all hidden states \"a\"\n",
        "        s_prev = self.repeator(s_prev)\n",
        "        # Use concatenator to concatenate a and s_prev on the last axis\n",
        "        concat = self.concatenator([a, s_prev])\n",
        "        e = self.densor1(concat)\n",
        "        energies = self.densor2(e)\n",
        "        alphas = self.activator(energies)\n",
        "        context = self.dotor([alphas, a])\n",
        "\n",
        "        return context"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dbMRAjXs9ne9"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "class NMT_Model:\n",
        "\n",
        "    def stack(outputs):\n",
        "        outputs = K.stack(outputs)\n",
        "        return K.permute_dimensions(outputs, pattern=(1, 0, 2))\n",
        "\n",
        "    def __init__(self, rnn_arch, Tx, Ty, encoder_units, decoder_units, embedding_dim, input_vocab_size,\n",
        "                 target_vocab_size, word_embedding):\n",
        "        # import pdb; pdb.set_trace()\n",
        "        self.rnn_arch = rnn_arch\n",
        "        self.decoder_units = decoder_units\n",
        "\n",
        "        # attention\n",
        "        self.attentionLayer = AttentionLayer(Tx)\n",
        "\n",
        "        # encoder\n",
        "        self.input = Input(shape=(Tx,))\n",
        "        encoder_embedding = Embedding(input_vocab_size, embedding_dim, weights=[word_embedding], input_length=Tx)\n",
        "        encoder = rnn_archs[rnn_arch](encoder_units, return_sequences=True)\n",
        "        encoder_inp_embedded = encoder_embedding(self.input)\n",
        "        self.encoder_out = encoder(encoder_inp_embedded)\n",
        "\n",
        "        # decoder\n",
        "        self.decoder_embedding = Embedding(target_vocab_size, embedding_dim)\n",
        "        if rnn_arch == 'gru':\n",
        "            self.decoder = gru(units=decoder_units, return_state=True)\n",
        "        else:\n",
        "            self.decoder = lstm(units=decoder_units, return_state=True)\n",
        "        # self.decoder = rnn_archs[rnn_arch](units=decoder_units, return_state=True)\n",
        "        self.dense_decode = Dense(target_vocab_size, activation='softmax')\n",
        "\n",
        "        # concat\n",
        "        self.concat2 = Concatenate(axis=2)\n",
        "\n",
        "        self.decoder_state_0 = Input(shape=(decoder_units,))\n",
        "        if rnn_arch != 'gru':\n",
        "            self.decoder_cell_0 = Input(shape=(decoder_units,))\n",
        "        self.train_model = self.get_train_model( Ty)\n",
        "        print('train model was built')\n",
        "        self.inference_model = self.get_inference_model(Ty)\n",
        "        print('inference model was built')\n",
        "\n",
        "    def get_train_model(self, Ty):\n",
        "        decoder_inp = Input(shape=(Ty,))\n",
        "        decoder_inp_embedded = self.decoder_embedding(decoder_inp)\n",
        "\n",
        "        decoder_state = self.decoder_state_0\n",
        "        if self.rnn_arch != 'gru':\n",
        "            decoder_cell = self.decoder_cell_0\n",
        "\n",
        "        # Iterate attention Ty times\n",
        "        outputs = []\n",
        "        for t in range(Ty):\n",
        "\n",
        "            # Get context vector with encoder and attention\n",
        "            context = self.attentionLayer.one_step_attention(self.encoder_out, decoder_state)\n",
        "\n",
        "            # For teacher forcing, get the previous word\n",
        "            select_layer = Lambda(lambda x: x[:, t:t + 1])\n",
        "            prevWord = select_layer(decoder_inp_embedded)\n",
        "\n",
        "            # Concat context and previous word as decoder input\n",
        "\n",
        "            decoder_in_concat = self.concat2([context, prevWord])\n",
        "\n",
        "            # pass into decoder, inference output\n",
        "            if self.rnn_arch == 'gru':\n",
        "                pred, decoder_state = self.decoder(decoder_in_concat, initial_state=decoder_state)\n",
        "            else:\n",
        "                pred, decoder_state, decoder_cell = self.decoder(decoder_in_concat,\n",
        "                                                                 initial_state=[decoder_state, decoder_cell])\n",
        "            pred = self.dense_decode(pred)\n",
        "            outputs.append(pred)\n",
        "\n",
        "        stack_layer = Lambda(NMT_Model.stack)\n",
        "        outputs = stack_layer(outputs)\n",
        "        if self.rnn_arch == 'gru':\n",
        "            return Model(inputs=[self.input, decoder_inp, self.decoder_state_0], outputs=outputs)\n",
        "        else:\n",
        "            return Model(inputs=[self.input, decoder_inp, self.decoder_state_0, self.decoder_cell_0], outputs=outputs)\n",
        "\n",
        "    # in the inference model teacher forcing is not available\n",
        "    def get_inference_model(self, Tx):\n",
        "        decoder_inp = Input(shape=(1,))\n",
        "\n",
        "        decoder_state = self.decoder_state_0\n",
        "\n",
        "        if self.rnn_arch != 'gru':\n",
        "            decoder_cell = self.decoder_cell_0\n",
        "\n",
        "        decoder_inp_embedded = self.decoder_embedding(decoder_inp)\n",
        "        # Get context vector with encoder and attention\n",
        "        context = self.attentionLayer.one_step_attention(self.encoder_out, decoder_state)\n",
        "\n",
        "        # Concat context and previous word as decoder input\n",
        "\n",
        "        decoder_in_concat = self.concat2([context, decoder_inp_embedded])\n",
        "\n",
        "        # pass into decoder, inference output\n",
        "        if self.rnn_arch == 'gru':\n",
        "            pred, decoder_state = self.decoder(decoder_in_concat, initial_state=decoder_state)\n",
        "        else:\n",
        "            pred, decoder_state, decoder_cell = self.decoder(decoder_in_concat,\n",
        "                                                             initial_state=[decoder_state, decoder_cell])\n",
        "\n",
        "        pred = self.dense_decode(pred)\n",
        "\n",
        "        if self.rnn_arch == 'gru':\n",
        "            return Model(inputs=[self.input, decoder_inp, self.decoder_state_0], outputs=pred)\n",
        "        else:\n",
        "            return Model(inputs=[self.input, decoder_inp, self.decoder_state_0, self.decoder_cell_0], outputs=pred)\n",
        "\n",
        "    def fit(self, enc_inp, decoder_inp, targ, batch_size = 64, verbose = 0):\n",
        "\n",
        "        s_0 = np.zeros((len(enc_inp), self.decoder_units))\n",
        "        if self.rnn_arch != 'gru':\n",
        "            c_0 = np.zeros((len(enc_inp), self.decoder_units))\n",
        "            history = self.train_model.fit([enc_inp, decoder_inp, s_0, c_0], targ, batch_size=batch_size, verbose=verbose)\n",
        "\n",
        "        else:\n",
        "\n",
        "            history = self.train_model.fit([enc_inp, decoder_inp, s_0], targ, batch_size=batch_size, verbose=verbose)\n",
        "        self.inference_model.set_weights(self.train_model.get_weights)\n",
        "\n",
        "    def compile(self, opt, loss, metrics):\n",
        "        self.train_model.compile(optimizer=opt, loss=loss, metrics=metrics)\n",
        "        self.inference_model.compile(optimizer=opt, loss=loss, metrics=metrics)\n",
        "    \n",
        "    def evaluate(self, enc_inp, decoder_inp, targ, batch_size = 64, verbose = 0):\n",
        "        s_0 = np.zeros((len(enc_inp), self.decoder_units))\n",
        "        if self.rnn_arch != 'gru':\n",
        "            c_0 = np.zeros((len(enc_inp), self.decoder_units))\n",
        "            return self.train_model.evaluate([enc_inp, decoder_inp, s_0, c_0], targ, batch_size=batch_size, verbose=verbose)\n",
        "\n",
        "        else:\n",
        "\n",
        "            return self.train_model.evaluate([enc_inp, decoder_inp, s_0], targ, batch_size=batch_size, verbose=verbose)\n",
        "\n",
        "    def inference_evaluate(self, enc_inp, decoder_inp, targ, batch_size = 64 , verbose = 0):\n",
        "        s_0 = np.zeros((len(enc_inp), self.decoder_units))\n",
        "        if self.rnn_arch != 'gru':\n",
        "            c_0 = np.zeros((len(enc_inp), self.decoder_units))\n",
        "\n",
        "        Ty = targ.shape[1]\n",
        "        loss = 0.0\n",
        "        acc = 0.0\n",
        "        preds = []\n",
        "\n",
        "        for t in range(Ty):\n",
        "\n",
        "            if self.rnn_arch == 'gru':\n",
        "                pred = self.inference_model.predict([enc_inp, decoder_inp, s_0])\n",
        "                print('prediction is done')\n",
        "\n",
        "                loss_b, acc_b = self.inference_model.evaluate([enc_inp, decoder_inp, s_0], targ[:, t], batch_size=batch_size,\n",
        "                                                              verbose=verbose)\n",
        "            else:\n",
        "                pred = self.inference_model.predict([enc_inp, decoder_inp, s_0, c_0])\n",
        "                \n",
        "                loss_b, acc_b = self.inference_model.evaluate([enc_inp, decoder_inp, s_0, c_0], targ[:, t],\n",
        "                                                              batch_size=batch_size, verbose=verbose)\n",
        "            if math.isnan(loss_b):\n",
        "                import pdb;\n",
        "                pdb.set_trace()\n",
        "                loss__ = myLoss(targ[:, t], pred)\n",
        "                acc__ = my_acc(targ[:, t], pred)\n",
        "       \n",
        "            pred = np.argmax(pred, axis=-1)\n",
        "            decoder_inp = np.expand_dims(pred, axis=1)\n",
        "            preds.append(decoder_inp)\n",
        "            loss += loss_b\n",
        "            acc += acc_b\n",
        "     \n",
        "\n",
        "        return loss / Ty, acc / Ty, NMT_Model.stack(preds).numpy()\n",
        "      def save(self,train_file, infer_file):\n",
        "        self.train_model.save(train_file)\n",
        "        self.inference_model.save(infer_file)\n",
        "\n",
        "      \n",
        "\n"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yeLXdR0HVo8Z"
      },
      "source": [
        "\n",
        "  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4pbgJbeY9zYW"
      },
      "source": [
        "def myLoss(y_train, pred):\n",
        "    \n",
        "    mask = K.cast(y_train > 0, dtype='float32')\n",
        "    mask2 = tf.greater(y_train, 0)\n",
        "    non_zero_y = tf.boolean_mask(pred, mask2)\n",
        "    val = K.log(non_zero_y)\n",
        "    # import pdb; pdb.set_trace()\n",
        "    return 0.0 if K.sum(mask)== 0 else -K.sum(val) / K.sum(mask)\n",
        "\n",
        "\n",
        "def my_acc(y_train, pred):\n",
        "    # import pdb; pdb.set_trace()\n",
        "    targ = K.argmax(y_train, axis=-1)\n",
        "    pred = K.argmax(pred, axis=-1)\n",
        "    correct = K.cast(K.equal(targ, pred), dtype='float32')\n",
        "\n",
        "    mask = K.cast(K.greater(targ, 0), dtype='float32')  # filter out padding value 0.\n",
        "    correctCount = K.sum(mask * correct)\n",
        "    totalCount = K.sum(mask)\n",
        "    return 1.0 if totalCount==0 else correctCount / totalCount"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mkxNTv0X-DsL"
      },
      "source": [
        "import math\n",
        "\n",
        "\n",
        "def make_batch(X, Y, shuffle=True, batch_size=64):\n",
        "    # import pdb; pdb.set_trace()\n",
        "    idx = np.arange(len(X))\n",
        "    if shuffle:\n",
        "        np.random.shuffle(idx)\n",
        "    dataset = []\n",
        "    batchs = math.ceil(len(X) / batch_size)\n",
        "    for b in range(batchs):\n",
        "        s = b * batch_size\n",
        "        e = min(s + batch_size, len(X))\n",
        "        dataset.append([b, X[s:e], Y[s:e]])\n",
        "    return dataset\n",
        "\n",
        "\n",
        "def save_result(loss,acc,loss_test,acc_test,epochs, dir):\n",
        "  f ={}\n",
        "  f['loss'] = loss\n",
        "  f['acc'] = acc\n",
        "  f['loss_test'] = loss_test\n",
        "  f['acc_test'] = acc_test\n",
        "  name = open(os.path.join(dir,'result_'+str(epochs)+'.pkl'),'wb')\n",
        "  pickle.dump(f,name)\n",
        "  name.close()\n",
        "from keras.models import load_model\n",
        "def save_model(model, main_file, train_file, infer_file):\n",
        "  f = {}\n",
        "  f['model'] = model\n",
        "  name = open(main_file,'wb')\n",
        "  pickle.dump(f,name)\n",
        "  name.close()\n",
        "  model.save(train_file, infer_file)\n",
        "def load_model(main_file, train_file, infer_file):\n",
        "  pkl_file = open(main_file, 'rb')\n",
        "  f = pickle.load(pkl_file)\n",
        "  model = f['model']\n",
        "  pkl_file.close()\n",
        "  model.train_model = load_model(train_file)\n",
        "  model.inference_model  = load_model(infer_file)\n",
        "  return model\n"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0xsd4XHl-ETs"
      },
      "source": [
        " \n",
        "def evaluate(model, dataset, batch_size =64, verbose = 0):\n",
        "  loss, acc, data_count =0.0, 0.0, 0\n",
        "  for batch, inp, targ in dataset:\n",
        "  \n",
        "    data_count += len(inp)\n",
        "\n",
        "    decoder_inp = np.zeros((len(targ), Ty))\n",
        "    decoder_inp[:, 1:] = targ[:, :-1]\n",
        "    decoder_inp[:, 0] = targ_vocab.word_idx['<sos>']\n",
        "    targ_one_hot = np.zeros((len(targ), Ty, targ_vocab_size), dtype='float32')\n",
        "    for idx, tokVec in enumerate(targ):\n",
        "      \n",
        "        for tok_idx, tok in enumerate(tokVec):\n",
        "          if (tok > 0):\n",
        "            targ_one_hot[idx, tok_idx, tok] = 1\n",
        "                \n",
        "    \n",
        "    loss_b, acc_b = model.evaluate(inp, decoder_inp, targ_one_hot, batch_size=batch_size, verbose=verbose)\n",
        "    loss += loss_b* len(inp) \n",
        "    acc += acc_b * len(inp)\n",
        "  return loss/ data_count, acc/ data_count\n",
        "\n"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G6Tr0oPv-9Rn"
      },
      "source": [
        "test_dataset = make_batch(X_test, Y_test, shuffle= False, batch_size= args.batch_size )"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QMnQu5P5ZcUL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "84455103-f78a-4e16-8ce7-170172fd19ce"
      },
      "source": [
        "model = NMT_Model(args.rnn_arch, Tx, Ty, encoder_units, decoder_units, EMBEDDING_DIM, inp_vocab_size, targ_vocab_size, word_embedding)\n",
        "\n",
        "\n",
        "model.compile(opt='adam', loss=myLoss, metrics=[my_acc])\n",
        "\n",
        "# final final debug\n",
        "### debug\n",
        "\n",
        "EPOCHS = args.epochs \n",
        "import time\n",
        "loss_t, acc_t = [], []\n",
        "loss_e = []\n",
        "acc_e = []\n",
        "best_acc = 0\n",
        "best_model = None"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train model was built\n",
            "inference model was built\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xLR3VhvkAhqF",
        "outputId": "9cd7e47f-f5d8-44b1-ec62-ec4d30566580"
      },
      "source": [
        "\n",
        "for epoch in range(EPOCHS):\n",
        "    start = time.time()\n",
        "    print(\"epoch:\", epoch+1)\n",
        "    loss, acc, data_count = 0.0, 0.0, 0\n",
        "    dataset = make_batch(X_train, Y_train, batch_size=args.batch_size)\n",
        "    for batch, inp, targ in dataset:\n",
        "        \n",
        "        data_count += len(inp)\n",
        "\n",
        "        \n",
        "\n",
        "        decoder_inp = np.zeros((len(targ), Ty))\n",
        "        decoder_inp[:, 1:] = targ[:, :-1]\n",
        "        decoder_inp[:, 0] = targ_vocab.word_idx['<sos>']\n",
        "        targ_one_hot = np.zeros((len(targ), Ty, targ_vocab_size), dtype='float32')\n",
        "        for idx, tokVec in enumerate(targ):\n",
        "          \n",
        "            for tok_idx, tok in enumerate(tokVec):\n",
        "              if (tok > 0):\n",
        "                targ_one_hot[idx, tok_idx, tok] = 1\n",
        "                    \n",
        "       \n",
        "        history = model.fit(inp, decoder_inp, targ_one_hot, batch_size=args.batch_size, verbose=0)\n",
        "      \n",
        "        loss_b, acc_b =  history.history['loss'][0], history.history['my_acc'][0]\n",
        "        \n",
        "        loss += (loss_b * len(inp))\n",
        "        acc += (acc_b * len(inp))\n",
        "        if batch % 100 == 0:\n",
        "            print('Epoch {} Batch {} Loss {:.4f} Accuracy {:.4f}'.format(epoch + 1,\n",
        "                                                                          batch,\n",
        "                                                                          loss / data_count, acc / data_count))\n",
        "    \n",
        "    loss, acc= evaluate(model, dataset,batch_size=args.batch_size)\n",
        "    loss_test, acc_test = evaluate(model, test_dataset, batch_size=args.batch_size)\n",
        "    if acc_test > best_acc:\n",
        "      save_model(model.train_model,EPOCHS, checkpoint_dir)\n",
        "     \n",
        "      \n",
        "    loss_t.append(loss_test)\n",
        "    acc_t.append(acc_test)\n",
        "    loss_e.append(loss)\n",
        "    acc_e.append(acc)\n",
        "    save_result(loss_e, acc_e,loss_t,acc_t,EPOCHS, result_dir)\n",
        "    print('Epoch {}  Loss {:.4f} Accuracy {:.4f}'.format(epoch + 1, loss, acc))\n",
        "    print('Epoch {}  Loss on test {:.4f} Accuracy on test {:.4f}'.format(epoch + 1, loss_test, acc_test))\n",
        "    print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "epoch: 1\n",
            "Epoch 1 Batch 0 Loss 9.3639 Accuracy 0.0000\n",
            "Epoch 1 Batch 100 Loss 6.0147 Accuracy 0.1169\n",
            "Epoch 1 Batch 200 Loss 5.8025 Accuracy 0.1337\n",
            "Epoch 1 Batch 300 Loss 5.6491 Accuracy 0.1495\n",
            "Epoch 1 Batch 400 Loss 5.4951 Accuracy 0.1702\n",
            "Epoch 1 Batch 500 Loss 5.3458 Accuracy 0.1874\n",
            "Epoch 1 Batch 600 Loss 5.2021 Accuracy 0.2038\n",
            "Epoch 1 Batch 700 Loss 5.0573 Accuracy 0.2194\n",
            "Epoch 1 Batch 800 Loss 4.9165 Accuracy 0.2341\n",
            "Epoch 1 Batch 900 Loss 4.7817 Accuracy 0.2480\n",
            "Epoch 1 Batch 1000 Loss 4.6529 Accuracy 0.2610\n",
            "Epoch 1 Batch 1100 Loss 4.5309 Accuracy 0.2734\n",
            "Saved model to disk\n",
            "Epoch 1  Loss 3.0641 Accuracy 0.4179\n",
            "Epoch 1  Loss on test 3.1708 Accuracy on test 0.4115\n",
            "Time taken for 1 epoch 570.528392791748 sec\n",
            "\n",
            "epoch: 2\n",
            "Epoch 2 Batch 0 Loss 3.4238 Accuracy 0.3790\n",
            "Epoch 2 Batch 100 Loss 3.1165 Accuracy 0.4163\n",
            "Epoch 2 Batch 200 Loss 3.0457 Accuracy 0.4227\n",
            "Epoch 2 Batch 300 Loss 2.9824 Accuracy 0.4296\n",
            "Epoch 2 Batch 400 Loss 2.9322 Accuracy 0.4340\n",
            "Epoch 2 Batch 500 Loss 2.8811 Accuracy 0.4391\n",
            "Epoch 2 Batch 600 Loss 2.8324 Accuracy 0.4442\n",
            "Epoch 2 Batch 700 Loss 2.7849 Accuracy 0.4490\n",
            "Epoch 2 Batch 800 Loss 2.7422 Accuracy 0.4532\n",
            "Epoch 2 Batch 900 Loss 2.7033 Accuracy 0.4568\n",
            "Epoch 2 Batch 1000 Loss 2.6653 Accuracy 0.4605\n",
            "Epoch 2 Batch 1100 Loss 2.6295 Accuracy 0.4644\n",
            "Saved model to disk\n",
            "Epoch 2  Loss 2.1402 Accuracy 0.5168\n",
            "Epoch 2  Loss on test 2.4023 Accuracy on test 0.4927\n",
            "Time taken for 1 epoch 543.5342154502869 sec\n",
            "\n",
            "epoch: 3\n",
            "Epoch 3 Batch 0 Loss 2.4434 Accuracy 0.4603\n",
            "Epoch 3 Batch 100 Loss 2.2126 Accuracy 0.5096\n",
            "Epoch 3 Batch 200 Loss 2.1819 Accuracy 0.5119\n",
            "Epoch 3 Batch 300 Loss 2.1562 Accuracy 0.5160\n",
            "Epoch 3 Batch 400 Loss 2.1385 Accuracy 0.5179\n",
            "Epoch 3 Batch 500 Loss 2.1166 Accuracy 0.5210\n",
            "Epoch 3 Batch 600 Loss 2.0964 Accuracy 0.5239\n",
            "Epoch 3 Batch 700 Loss 2.0758 Accuracy 0.5268\n",
            "Epoch 3 Batch 800 Loss 2.0593 Accuracy 0.5288\n",
            "Epoch 3 Batch 900 Loss 2.0420 Accuracy 0.5307\n",
            "Epoch 3 Batch 1000 Loss 2.0246 Accuracy 0.5330\n",
            "Epoch 3 Batch 1100 Loss 2.0084 Accuracy 0.5353\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}