TV Script Generation with RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TV Script Generation\n",
    "\n",
    "In this project, I have tried to generate my own [Seinfeld](https://en.wikipedia.org/wiki/Seinfeld) TV scripts using RNNs. I have used part of the [Seinfeld dataset](https://www.kaggle.com/thec03u5/seinfeld-chronicles#scripts.csv) of scripts from 9 seasons.  The Neural Network will generate a new ,\"fake\" TV script, based on patterns it recognizes in this training data.\n",
    "\n",
    "## Get the Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load in data\n",
    "import helper\n",
    "data_dir = './data/Seinfeld_Scripts.txt'\n",
    "text = helper.load_data(data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset Stats\n",
      "Roughly the number of unique words: 46367\n",
      "Number of lines: 109233\n",
      "Average number of words in each line: 5.544240293684143\n",
      "\n",
      "The lines 0 to 10:\n",
      "jerry: do you know what this is all about? do you know, why were here? to be out, this is out...and out is one of the single most enjoyable experiences of life. people...did you ever hear people talking about we should go out? this is what theyre talking about...this whole thing, were all out now, no one is home. not one person here is home, were all out! there are people trying to find us, they dont know where we are. (on an imaginary phone) did you ring?, i cant find him. where did he go? he didnt tell me where he was going. he must have gone out. you wanna go out you get ready, you pick out the clothes, right? you take the shower, you get all ready, get the cash, get your friends, the car, the spot, the reservation...then youre standing around, what do you do? you go we gotta be getting back. once youre out, you wanna get back! you wanna go to sleep, you wanna get up, you wanna go out again tomorrow, right? where ever you are in life, its my feeling, youve gotta go. \n",
      "\n",
      "jerry: (pointing at georges shirt) see, to me, that button is in the worst possible spot. the second button literally makes or breaks the shirt, look at it. its too high! its in no-mans-land. you look like you live with your mother. \n",
      "\n",
      "george: are you through? \n",
      "\n",
      "jerry: you do of course try on, when you buy? \n",
      "\n",
      "george: yes, it was purple, i liked it, i dont actually recall considering the buttons. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "view_line_range = (0, 10)\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "print('Dataset Stats')\n",
    "print('Roughly the number of unique words: {}'.format(len({word: None for word in text.split()})))\n",
    "\n",
    "lines = text.split('\\n')\n",
    "print('Number of lines: {}'.format(len(lines)))\n",
    "word_count_line = [len(line.split()) for line in lines]\n",
    "print('Average number of words in each line: {}'.format(np.average(word_count_line)))\n",
    "\n",
    "print()\n",
    "print('The lines {} to {}:'.format(*view_line_range))\n",
    "print('\\n'.join(text.split('\\n')[view_line_range[0]:view_line_range[1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Implement Pre-processing Functions\n",
    "The first thing to do to any dataset is pre-processing. I will implement the following pre-processing functions below:\n",
    "- Lookup Table\n",
    "- Tokenize Punctuation\n",
    "\n",
    "### Lookup Table\n",
    "To create a word embedding, you first need to transform the words to ids.  In this function, create two dictionaries:\n",
    "- Dictionary to go from the words to an id, we'll call `vocab_to_int`\n",
    "- Dictionary to go from the id to word, we'll call `int_to_vocab`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "import problem_unittests as tests\n",
    "from collections import Counter\n",
    "\n",
    "def create_lookup_tables(text):\n",
    "\n",
    "    words = Counter(text)\n",
    "    vocab = sorted(words, key = words.get, reverse = True)\n",
    "    vocab_to_int = {word : ii for ii, word in enumerate(vocab)} \n",
    "    int_to_vocab = {ii : word for ii, word in enumerate(vocab)}\n",
    "    # return tuple\n",
    "    return (vocab_to_int, int_to_vocab)\n",
    "\n",
    "tests.test_create_lookup_tables(create_lookup_tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenize Punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def token_lookup():\n",
    "    \n",
    "    #getting the key-value pair in a dict\n",
    "    punctuation = {'.': \"||Period||\", ',': \"||Comma||\", '\"': \"||Quotation_Mark||\", ';': \"||Semicolon||\",\n",
    "                   '!': \"||Exclamation_Mark||\", '?': \"||Question_Mark||\", '(': \"||Left_Parentheses||\",\n",
    "                   ')': \"||Right_Parentheses||\", '-': \"||Dash||\", '\\n': \"||Return||\"\n",
    "                  }\n",
    "    \n",
    "    return punctuation\n",
    "\n",
    "tests.test_tokenize(token_lookup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-process all the data and save it\n",
    "\n",
    "Running the code cell below will pre-process all the data and save it to file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pre-process training data\n",
    "helper.preprocess_and_save_data(data_dir, token_lookup, create_lookup_tables)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "int_text, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Neural Network\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# Check for a GPU\n",
    "train_on_gpu = torch.cuda.is_available()\n",
    "if not train_on_gpu:\n",
    "    print('No GPU found. Please use a GPU to train your neural network.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([[ 88,  77,  66,  55,  44],\n",
       "         [ 66,  55,  44,  33,  22],\n",
       "         [ 55,  44,  33,  22,  11]]), tensor([ 33,  11,   0])]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "\n",
    "def batch_data(words, sequence_length, batch_size):\n",
    "\n",
    "    num_featues = len(words) - sequence_length  #this is how many total features we can have at given sequence length\n",
    "    X_train = np.zeros((num_featues, sequence_length), dtype = int) #num_features gives us shape of complete X_train\n",
    "    y_train = np.zeros(num_featues)  #num of labels is equal to num of rows in X_train\n",
    "    \n",
    "    #now, we will ommit the zeros with our words with this logic\n",
    "    for i in range(0, num_featues):\n",
    "        X_train[i] = words[i:i+sequence_length]\n",
    "        y_train[i] = words[i+sequence_length]\n",
    "    \n",
    "    #changing dtype\n",
    "    feature_array = np.asarray(X_train, np.int64)\n",
    "    target_array = np.asarray(y_train, np.int64)\n",
    "    data = TensorDataset(torch.from_numpy(feature_array), torch.from_numpy(target_array))\n",
    "    dataloader = DataLoader(data, batch_size = batch_size, shuffle = True)\n",
    "\n",
    "    return dataloader\n",
    "\n",
    "words = np.array([99,88,77,66,55,44,33,22,11,0])\n",
    "loader = batch_data(words, 5, 3)\n",
    "dataiter = iter(loader)\n",
    "dataiter.next()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([10, 5])\n",
      "tensor([[ 18,  19,  20,  21,  22],\n",
      "        [ 44,  45,  46,  47,  48],\n",
      "        [ 14,  15,  16,  17,  18],\n",
      "        [ 32,  33,  34,  35,  36],\n",
      "        [  0,   1,   2,   3,   4],\n",
      "        [ 35,  36,  37,  38,  39],\n",
      "        [ 16,  17,  18,  19,  20],\n",
      "        [  1,   2,   3,   4,   5],\n",
      "        [ 24,  25,  26,  27,  28],\n",
      "        [ 33,  34,  35,  36,  37]])\n",
      "\n",
      "torch.Size([10])\n",
      "tensor([ 23,  49,  19,  37,   5,  40,  21,   6,  29,  38])\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "# test dataloader\n",
    "\n",
    "test_text = range(50)\n",
    "t_loader = batch_data(test_text, sequence_length=5, batch_size=10)\n",
    "\n",
    "data_iter = iter(t_loader)\n",
    "sample_x, sample_y = data_iter.next()\n",
    "\n",
    "print(sample_x.shape)\n",
    "print(sample_x)\n",
    "print()\n",
    "print(sample_y.shape)\n",
    "print(sample_y)\n",
    "print(len(t_loader))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Build the Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, dropout=0.5):\n",
    "        \n",
    "        super(RNN, self).__init__()\n",
    "        # set class variables\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        \n",
    "        #embedding layer\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        #lstm layer\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, dropout=dropout, batch_first=True)\n",
    "        \n",
    "        # dropout layer\n",
    "        self.dropout = nn.Dropout(p=0.2)\n",
    "        #fully connected layer\n",
    "        self.fc = nn.Linear(hidden_dim, output_size)\n",
    "    \n",
    "    \n",
    "    def forward(self, nn_input, hidden):\n",
    "        \n",
    "        batch_size = nn_input.size(0)\n",
    "        \n",
    "        embeds = self.embedding(nn_input)\n",
    "        lstm_output, hidden = self.lstm(embeds, hidden)\n",
    "        \n",
    "        #stack the outputs of the lstm to pass to your fully-connected layer - kinda flattening step\n",
    "        lstm_output = lstm_output.contiguous().view(-1, self.hidden_dim)\n",
    "        \n",
    "        output = self.dropout(lstm_output)\n",
    "        output = self.fc(lstm_output)\n",
    "        output = output.view(batch_size, -1, self.output_size)\n",
    "        #getting the last batch of outputs\n",
    "        output = output[:,-1]\n",
    "        \n",
    "        # return one batch of output word scores and the hidden state\n",
    "        return output, hidden\n",
    "    \n",
    "    \n",
    "    def init_hidden(self, batch_size):\n",
    "        #hidden state of dims (n_layers, batch_size, hidden_dim)\n",
    "        \n",
    "        weight = next(self.parameters()).data\n",
    "        \n",
    "        # initialize hidden state with zero weights, and move to GPU if available\n",
    "        if train_on_gpu:\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),\n",
    "                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())\n",
    "        else:\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),\n",
    "                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())\n",
    "        return hidden\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define forward and backpropagation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def forward_back_prop(rnn, optimizer, criterion, inp, target, hidden):\n",
    "   \n",
    "    # move data to GPU, if available\n",
    "    if train_on_gpu:\n",
    "        inp, target = inp.cuda(), target.cuda()\n",
    "    \n",
    "    # perform backpropagation and optimization\n",
    "    \n",
    "    hidden = tuple([i.data for i in hidden])\n",
    "    \n",
    "    rnn.zero_grad()\n",
    "    output, hidden = rnn(inp, hidden)\n",
    "    loss = criterion(output, target)\n",
    "    loss.backward()\n",
    "    \n",
    "    #useing gradient clipping to prevent exploding gradients problem\n",
    "    nn.utils.clip_grad_norm_(rnn.parameters(), 5)\n",
    "    optimizer.step()\n",
    "    \n",
    "\n",
    "    # return the loss over a batch and the hidden state produced by our model\n",
    "    return loss.item(), hidden\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Training\n",
    "\n",
    "With the structure of the network complete and data ready to be fed in the neural network, it's time to train it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_rnn(rnn, batch_size, optimizer, criterion, n_epochs, show_every_n_batches=100):\n",
    "    batch_losses = []\n",
    "    \n",
    "    rnn.train()\n",
    "\n",
    "    print(\"Training for %d epoch(s)...\" % n_epochs)\n",
    "    for epoch_i in range(1, n_epochs + 1):\n",
    "        \n",
    "        # initialize hidden state\n",
    "        hidden = rnn.init_hidden(batch_size)\n",
    "        \n",
    "        for batch_i, (inputs, labels) in enumerate(train_loader, 1):\n",
    "            \n",
    "            # making sure to iterate over completely full batches, only\n",
    "            n_batches = len(train_loader.dataset)//batch_size\n",
    "            if(batch_i > n_batches):\n",
    "                break\n",
    "            \n",
    "            # forward, back prop\n",
    "            loss, hidden = forward_back_prop(rnn, optimizer, criterion, inputs, labels, hidden)          \n",
    "            # record loss\n",
    "            batch_losses.append(loss)\n",
    "\n",
    "            # printing loss stats\n",
    "            if batch_i % show_every_n_batches == 0:\n",
    "                print('Epoch: {:>4}/{:<4}  Loss: {}\\n'.format(\n",
    "                    epoch_i, n_epochs, np.average(batch_losses)))\n",
    "                batch_losses = []\n",
    "\n",
    "    # returns a trained rnn\n",
    "    return rnn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data params\n",
    "# Sequence Length - [5, 7, 10, 15, 20, 25]\n",
    "sequence_length = 15  # of words in a sequence\n",
    "# Batch Size - [64, 128, 256]\n",
    "batch_size = 128\n",
    "\n",
    "# data loader - do not change\n",
    "train_loader = batch_data(int_text, sequence_length, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training parameters\n",
    "# Number of Epochs\n",
    "num_epochs = 25\n",
    "# Learning Rate - [0.001 - 0.005, 0.01, 0.1]\n",
    "learning_rate = 0.001\n",
    "\n",
    "# Model parameters\n",
    "# Vocab size\n",
    "vocab_size = len(int_to_vocab)\n",
    "# Output size\n",
    "output_size = vocab_size\n",
    "# Embedding Dimension - [200, 400, 600]\n",
    "embedding_dim = 128\n",
    "# Hidden Dimension - [300, 500, 1000]\n",
    "hidden_dim = 500\n",
    "# Number of RNN Layers - [2,3]\n",
    "n_layers = 2\n",
    "\n",
    "# Show stats for every n number of batches\n",
    "show_every_n_batches = 500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training for 25 epoch(s)...\n",
      "Epoch:    1/25    Loss: 5.423726204872131\n",
      "\n",
      "Epoch:    1/25    Loss: 4.816921296596527\n",
      "\n",
      "Epoch:    1/25    Loss: 4.5948448047637935\n",
      "\n",
      "Epoch:    1/25    Loss: 4.465704329490662\n",
      "\n",
      "Epoch:    1/25    Loss: 4.394996451377868\n",
      "\n",
      "Epoch:    1/25    Loss: 4.312475248336792\n",
      "\n",
      "Epoch:    1/25    Loss: 4.285950607776642\n",
      "\n",
      "Epoch:    1/25    Loss: 4.2121966876983645\n",
      "\n",
      "Epoch:    1/25    Loss: 4.203113260269165\n",
      "\n",
      "Epoch:    1/25    Loss: 4.1777085647583005\n",
      "\n",
      "Epoch:    1/25    Loss: 4.095357491016388\n",
      "\n",
      "Epoch:    1/25    Loss: 4.106398698806763\n",
      "\n",
      "Epoch:    1/25    Loss: 4.0812316989898685\n",
      "\n",
      "Epoch:    2/25    Loss: 3.9684305444467425\n",
      "\n",
      "Epoch:    2/25    Loss: 3.881323283672333\n",
      "\n",
      "Epoch:    2/25    Loss: 3.8777885551452638\n",
      "\n",
      "Epoch:    2/25    Loss: 3.867604364871979\n",
      "\n",
      "Epoch:    2/25    Loss: 3.85935231590271\n",
      "\n",
      "Epoch:    2/25    Loss: 3.829647635936737\n",
      "\n",
      "Epoch:    2/25    Loss: 3.855135422706604\n",
      "\n",
      "Epoch:    2/25    Loss: 3.860082482814789\n",
      "\n",
      "Epoch:    2/25    Loss: 3.8412785549163817\n",
      "\n",
      "Epoch:    2/25    Loss: 3.81725435256958\n",
      "\n",
      "Epoch:    2/25    Loss: 3.8207085614204406\n",
      "\n",
      "Epoch:    2/25    Loss: 3.819502682209015\n",
      "\n",
      "Epoch:    2/25    Loss: 3.8278458976745604\n",
      "\n",
      "Epoch:    3/25    Loss: 3.701739651499887\n",
      "\n",
      "Epoch:    3/25    Loss: 3.6223127346038817\n",
      "\n",
      "Epoch:    3/25    Loss: 3.6112140641212465\n",
      "\n",
      "Epoch:    3/25    Loss: 3.622787039756775\n",
      "\n",
      "Epoch:    3/25    Loss: 3.629208384037018\n",
      "\n",
      "Epoch:    3/25    Loss: 3.6256437578201295\n",
      "\n",
      "Epoch:    3/25    Loss: 3.6306768369674685\n",
      "\n",
      "Epoch:    3/25    Loss: 3.608042973041534\n",
      "\n",
      "Epoch:    3/25    Loss: 3.631112483024597\n",
      "\n",
      "Epoch:    3/25    Loss: 3.643419871330261\n",
      "\n",
      "Epoch:    3/25    Loss: 3.6411145873069763\n",
      "\n",
      "Epoch:    3/25    Loss: 3.641309657096863\n",
      "\n",
      "Epoch:    3/25    Loss: 3.6315266065597536\n",
      "\n",
      "Epoch:    4/25    Loss: 3.5197710816213097\n",
      "\n",
      "Epoch:    4/25    Loss: 3.4263078150749204\n",
      "\n",
      "Epoch:    4/25    Loss: 3.4401624436378477\n",
      "\n",
      "Epoch:    4/25    Loss: 3.4297226567268373\n",
      "\n",
      "Epoch:    4/25    Loss: 3.4523994188308715\n",
      "\n",
      "Epoch:    4/25    Loss: 3.443440107345581\n",
      "\n",
      "Epoch:    4/25    Loss: 3.4637670612335203\n",
      "\n",
      "Epoch:    4/25    Loss: 3.462138439655304\n",
      "\n",
      "Epoch:    4/25    Loss: 3.489903239250183\n",
      "\n",
      "Epoch:    4/25    Loss: 3.4784525499343872\n",
      "\n",
      "Epoch:    4/25    Loss: 3.5121204528808594\n",
      "\n",
      "Epoch:    4/25    Loss: 3.5049839186668397\n",
      "\n",
      "Epoch:    4/25    Loss: 3.498932755947113\n",
      "\n",
      "Epoch:    5/25    Loss: 3.377569002755778\n",
      "\n",
      "Epoch:    5/25    Loss: 3.2775952835083006\n",
      "\n",
      "Epoch:    5/25    Loss: 3.291125778675079\n",
      "\n",
      "Epoch:    5/25    Loss: 3.287435998439789\n",
      "\n",
      "Epoch:    5/25    Loss: 3.3145964980125426\n",
      "\n",
      "Epoch:    5/25    Loss: 3.313120436191559\n",
      "\n",
      "Epoch:    5/25    Loss: 3.344851444721222\n",
      "\n",
      "Epoch:    5/25    Loss: 3.3494268312454225\n",
      "\n",
      "Epoch:    5/25    Loss: 3.372710421085358\n",
      "\n",
      "Epoch:    5/25    Loss: 3.379210249900818\n",
      "\n",
      "Epoch:    5/25    Loss: 3.370316672325134\n",
      "\n",
      "Epoch:    5/25    Loss: 3.393772722244263\n",
      "\n",
      "Epoch:    5/25    Loss: 3.386518869876862\n",
      "\n",
      "Epoch:    6/25    Loss: 3.270958100556097\n",
      "\n",
      "Epoch:    6/25    Loss: 3.1792452545166014\n",
      "\n",
      "Epoch:    6/25    Loss: 3.180706570148468\n",
      "\n",
      "Epoch:    6/25    Loss: 3.177193211555481\n",
      "\n",
      "Epoch:    6/25    Loss: 3.2259489402770996\n",
      "\n",
      "Epoch:    6/25    Loss: 3.2127830023765562\n",
      "\n",
      "Epoch:    6/25    Loss: 3.2426922855377196\n",
      "\n",
      "Epoch:    6/25    Loss: 3.2609983291625975\n",
      "\n",
      "Epoch:    6/25    Loss: 3.272311643600464\n",
      "\n",
      "Epoch:    6/25    Loss: 3.265612841129303\n",
      "\n",
      "Epoch:    6/25    Loss: 3.2858721432685853\n",
      "\n",
      "Epoch:    6/25    Loss: 3.2793973231315614\n",
      "\n",
      "Epoch:    6/25    Loss: 3.2865922865867616\n",
      "\n",
      "Epoch:    7/25    Loss: 3.173845962351198\n",
      "\n",
      "Epoch:    7/25    Loss: 3.084418257236481\n",
      "\n",
      "Epoch:    7/25    Loss: 3.095620177268982\n",
      "\n",
      "Epoch:    7/25    Loss: 3.1060116381645204\n",
      "\n",
      "Epoch:    7/25    Loss: 3.1186632051467895\n",
      "\n",
      "Epoch:    7/25    Loss: 3.161971022605896\n",
      "\n",
      "Epoch:    7/25    Loss: 3.1654104895591737\n",
      "\n",
      "Epoch:    7/25    Loss: 3.1419670176506043\n",
      "\n",
      "Epoch:    7/25    Loss: 3.1963283710479735\n",
      "\n",
      "Epoch:    7/25    Loss: 3.2146775126457214\n",
      "\n",
      "Epoch:    7/25    Loss: 3.203678563117981\n",
      "\n",
      "Epoch:    7/25    Loss: 3.2142691774368286\n",
      "\n",
      "Epoch:    7/25    Loss: 3.225430968761444\n",
      "\n",
      "Epoch:    8/25    Loss: 3.1166541871640705\n",
      "\n",
      "Epoch:    8/25    Loss: 3.013746265888214\n",
      "\n",
      "Epoch:    8/25    Loss: 3.0227846550941466\n",
      "\n",
      "Epoch:    8/25    Loss: 3.0435191316604615\n",
      "\n",
      "Epoch:    8/25    Loss: 3.0682913551330566\n",
      "\n",
      "Epoch:    8/25    Loss: 3.0738880767822265\n",
      "\n",
      "Epoch:    8/25    Loss: 3.0778031883239745\n",
      "\n",
      "Epoch:    8/25    Loss: 3.087958180427551\n",
      "\n",
      "Epoch:    8/25    Loss: 3.117186101436615\n",
      "\n",
      "Epoch:    8/25    Loss: 3.1220001120567322\n",
      "\n",
      "Epoch:    8/25    Loss: 3.127596525192261\n",
      "\n",
      "Epoch:    8/25    Loss: 3.159515588760376\n",
      "\n",
      "Epoch:    8/25    Loss: 3.1668424954414367\n",
      "\n",
      "Epoch:    9/25    Loss: 3.0418095652656043\n",
      "\n",
      "Epoch:    9/25    Loss: 2.944866075515747\n",
      "\n",
      "Epoch:    9/25    Loss: 2.959615846157074\n",
      "\n",
      "Epoch:    9/25    Loss: 2.9890300397872926\n",
      "\n",
      "Epoch:    9/25    Loss: 2.99966903924942\n",
      "\n",
      "Epoch:    9/25    Loss: 3.0172240529060366\n",
      "\n",
      "Epoch:    9/25    Loss: 3.0249201374053953\n",
      "\n",
      "Epoch:    9/25    Loss: 3.0264137110710143\n",
      "\n",
      "Epoch:    9/25    Loss: 3.0642929010391233\n",
      "\n",
      "Epoch:    9/25    Loss: 3.056668417453766\n",
      "\n",
      "Epoch:    9/25    Loss: 3.072351321220398\n",
      "\n",
      "Epoch:    9/25    Loss: 3.114251217842102\n",
      "\n",
      "Epoch:    9/25    Loss: 3.120476620197296\n",
      "\n",
      "Epoch:   10/25    Loss: 2.996784360785233\n",
      "\n",
      "Epoch:   10/25    Loss: 2.894531777858734\n",
      "\n",
      "Epoch:   10/25    Loss: 2.908388903141022\n",
      "\n",
      "Epoch:   10/25    Loss: 2.940441247463226\n",
      "\n",
      "Epoch:   10/25    Loss: 2.934600160598755\n",
      "\n",
      "Epoch:   10/25    Loss: 2.961125741958618\n",
      "\n",
      "Epoch:   10/25    Loss: 2.9857519626617433\n",
      "\n",
      "Epoch:   10/25    Loss: 2.9872849378585817\n",
      "\n",
      "Epoch:   10/25    Loss: 3.0033742341995238\n",
      "\n",
      "Epoch:   10/25    Loss: 3.0268198471069336\n",
      "\n",
      "Epoch:   10/25    Loss: 3.0344871549606323\n",
      "\n",
      "Epoch:   10/25    Loss: 3.0542313933372496\n",
      "\n",
      "Epoch:   10/25    Loss: 3.065020387649536\n",
      "\n",
      "Epoch:   11/25    Loss: 2.9448738403241577\n",
      "\n",
      "Epoch:   11/25    Loss: 2.852096177101135\n",
      "\n",
      "Epoch:   11/25    Loss: 2.874803240776062\n",
      "\n",
      "Epoch:   11/25    Loss: 2.886046655654907\n",
      "\n",
      "Epoch:   11/25    Loss: 2.8947686076164247\n",
      "\n",
      "Epoch:   11/25    Loss: 2.9271398310661314\n",
      "\n",
      "Epoch:   11/25    Loss: 2.94361022567749\n",
      "\n",
      "Epoch:   11/25    Loss: 2.9356246271133424\n",
      "\n",
      "Epoch:   11/25    Loss: 2.961412893295288\n",
      "\n",
      "Epoch:   11/25    Loss: 2.977890034198761\n",
      "\n",
      "Epoch:   11/25    Loss: 3.009846233844757\n",
      "\n",
      "Epoch:   11/25    Loss: 2.9900183429718017\n",
      "\n",
      "Epoch:   11/25    Loss: 3.031431785106659\n",
      "\n",
      "Epoch:   12/25    Loss: 2.9001472638487447\n",
      "\n",
      "Epoch:   12/25    Loss: 2.819765064239502\n",
      "\n",
      "Epoch:   12/25    Loss: 2.8354614934921263\n",
      "\n",
      "Epoch:   12/25    Loss: 2.8334973969459534\n",
      "\n",
      "Epoch:   12/25    Loss: 2.873412536621094\n",
      "\n",
      "Epoch:   12/25    Loss: 2.894861284732819\n",
      "\n",
      "Epoch:   12/25    Loss: 2.8817454223632812\n",
      "\n",
      "Epoch:   12/25    Loss: 2.9006946725845335\n",
      "\n",
      "Epoch:   12/25    Loss: 2.9325569076538085\n",
      "\n",
      "Epoch:   12/25    Loss: 2.938425148010254\n",
      "\n",
      "Epoch:   12/25    Loss: 2.9584497237205505\n",
      "\n",
      "Epoch:   12/25    Loss: 2.9615586318969727\n",
      "\n",
      "Epoch:   12/25    Loss: 2.9783458652496337\n",
      "\n",
      "Epoch:   13/25    Loss: 2.8682352443474612\n",
      "\n",
      "Epoch:   13/25    Loss: 2.7632153182029726\n",
      "\n",
      "Epoch:   13/25    Loss: 2.778425857543945\n",
      "\n",
      "Epoch:   13/25    Loss: 2.8134298725128173\n",
      "\n",
      "Epoch:   13/25    Loss: 2.842955076217651\n",
      "\n",
      "Epoch:   13/25    Loss: 2.8395848183631895\n",
      "\n",
      "Epoch:   13/25    Loss: 2.860767062664032\n",
      "\n",
      "Epoch:   13/25    Loss: 2.869985694885254\n",
      "\n",
      "Epoch:   13/25    Loss: 2.8936745157241823\n",
      "\n",
      "Epoch:   13/25    Loss: 2.9096081900596618\n",
      "\n",
      "Epoch:   13/25    Loss: 2.921373104095459\n",
      "\n",
      "Epoch:   13/25    Loss: 2.9440590887069704\n",
      "\n",
      "Epoch:   13/25    Loss: 2.9368180027008055\n",
      "\n",
      "Epoch:   14/25    Loss: 2.8500159906283007\n",
      "\n",
      "Epoch:   14/25    Loss: 2.7378627982139587\n",
      "\n",
      "Epoch:   14/25    Loss: 2.759641805648804\n",
      "\n",
      "Epoch:   14/25    Loss: 2.777440806388855\n",
      "\n",
      "Epoch:   14/25    Loss: 2.7840148525238035\n",
      "\n",
      "Epoch:   14/25    Loss: 2.8280431723594663\n",
      "\n",
      "Epoch:   14/25    Loss: 2.82318993139267\n",
      "\n",
      "Epoch:   14/25    Loss: 2.8596754422187805\n",
      "\n",
      "Epoch:   14/25    Loss: 2.858047308921814\n",
      "\n",
      "Epoch:   14/25    Loss: 2.868330186367035\n",
      "\n",
      "Epoch:   14/25    Loss: 2.890076735019684\n",
      "\n",
      "Epoch:   14/25    Loss: 2.9077807393074036\n",
      "\n",
      "Epoch:   14/25    Loss: 2.917652482509613\n",
      "\n",
      "Epoch:   15/25    Loss: 2.803697016217022\n",
      "\n",
      "Epoch:   15/25    Loss: 2.7324283919334413\n",
      "\n",
      "Epoch:   15/25    Loss: 2.7325531182289122\n",
      "\n",
      "Epoch:   15/25    Loss: 2.761611647605896\n",
      "\n",
      "Epoch:   15/25    Loss: 2.7832669429779053\n",
      "\n",
      "Epoch:   15/25    Loss: 2.7728698477745057\n",
      "\n",
      "Epoch:   15/25    Loss: 2.7891263241767885\n",
      "\n",
      "Epoch:   15/25    Loss: 2.8040876483917234\n",
      "\n",
      "Epoch:   15/25    Loss: 2.8381275033950804\n",
      "\n",
      "Epoch:   15/25    Loss: 2.8434978289604187\n",
      "\n",
      "Epoch:   15/25    Loss: 2.8605653276443483\n",
      "\n",
      "Epoch:   15/25    Loss: 2.8592444229125977\n",
      "\n",
      "Epoch:   15/25    Loss: 2.900561838150024\n",
      "\n",
      "Epoch:   16/25    Loss: 2.776701474952501\n",
      "\n",
      "Epoch:   16/25    Loss: 2.692239896774292\n",
      "\n",
      "Epoch:   16/25    Loss: 2.700208675861359\n",
      "\n",
      "Epoch:   16/25    Loss: 2.7349094619750978\n",
      "\n",
      "Epoch:   16/25    Loss: 2.7433988199234007\n",
      "\n",
      "Epoch:   16/25    Loss: 2.770338861942291\n",
      "\n",
      "Epoch:   16/25    Loss: 2.7860610771179197\n",
      "\n",
      "Epoch:   16/25    Loss: 2.7856276512145994\n",
      "\n",
      "Epoch:   16/25    Loss: 2.7888686661720277\n",
      "\n",
      "Epoch:   16/25    Loss: 2.814302157878876\n",
      "\n",
      "Epoch:   16/25    Loss: 2.832471616744995\n",
      "\n",
      "Epoch:   16/25    Loss: 2.8476196637153626\n",
      "\n",
      "Epoch:   16/25    Loss: 2.8484257378578186\n",
      "\n",
      "Epoch:   17/25    Loss: 2.752413369061654\n",
      "\n",
      "Epoch:   17/25    Loss: 2.6680232315063477\n",
      "\n",
      "Epoch:   17/25    Loss: 2.697887749195099\n",
      "\n",
      "Epoch:   17/25    Loss: 2.6998074336051943\n",
      "\n",
      "Epoch:   17/25    Loss: 2.7132198696136474\n",
      "\n",
      "Epoch:   17/25    Loss: 2.7291082677841185\n",
      "\n",
      "Epoch:   17/25    Loss: 2.757754333972931\n",
      "\n",
      "Epoch:   17/25    Loss: 2.78592320728302\n",
      "\n",
      "Epoch:   17/25    Loss: 2.7876637444496155\n",
      "\n",
      "Epoch:   17/25    Loss: 2.7778154859542847\n",
      "\n",
      "Epoch:   17/25    Loss: 2.8050678157806397\n",
      "\n",
      "Epoch:   17/25    Loss: 2.8244374985694884\n",
      "\n",
      "Epoch:   17/25    Loss: 2.8292524766922\n",
      "\n",
      "Epoch:   18/25    Loss: 2.7246569192077352\n",
      "\n",
      "Epoch:   18/25    Loss: 2.6446062450408934\n",
      "\n",
      "Epoch:   18/25    Loss: 2.657641568183899\n",
      "\n",
      "Epoch:   18/25    Loss: 2.682559905052185\n",
      "\n",
      "Epoch:   18/25    Loss: 2.704728938102722\n",
      "\n",
      "Epoch:   18/25    Loss: 2.7195037474632264\n",
      "\n",
      "Epoch:   18/25    Loss: 2.7354616260528566\n",
      "\n",
      "Epoch:   18/25    Loss: 2.7468639268875124\n",
      "\n",
      "Epoch:   18/25    Loss: 2.761533100605011\n",
      "\n",
      "Epoch:   18/25    Loss: 2.786409499645233\n",
      "\n",
      "Epoch:   18/25    Loss: 2.8004404368400575\n",
      "\n",
      "Epoch:   18/25    Loss: 2.7911802096366882\n",
      "\n",
      "Epoch:   18/25    Loss: 2.8045156536102294\n",
      "\n",
      "Epoch:   19/25    Loss: 2.7175861445742866\n",
      "\n",
      "Epoch:   19/25    Loss: 2.629490830898285\n",
      "\n",
      "Epoch:   19/25    Loss: 2.644871874809265\n",
      "\n",
      "Epoch:   19/25    Loss: 2.661536959171295\n",
      "\n",
      "Epoch:   19/25    Loss: 2.6747522110939026\n",
      "\n",
      "Epoch:   19/25    Loss: 2.7085666308403016\n",
      "\n",
      "Epoch:   19/25    Loss: 2.720521739959717\n",
      "\n",
      "Epoch:   19/25    Loss: 2.7261462926864626\n",
      "\n",
      "Epoch:   19/25    Loss: 2.7372481760978697\n",
      "\n",
      "Epoch:   19/25    Loss: 2.7387118229866028\n",
      "\n",
      "Epoch:   19/25    Loss: 2.769590602874756\n",
      "\n",
      "Epoch:   19/25    Loss: 2.7614417181015014\n",
      "\n",
      "Epoch:   19/25    Loss: 2.798788748264313\n",
      "\n",
      "Epoch:   20/25    Loss: 2.678337988838692\n",
      "\n",
      "Epoch:   20/25    Loss: 2.6148586761951447\n",
      "\n",
      "Epoch:   20/25    Loss: 2.624310518741608\n",
      "\n",
      "Epoch:   20/25    Loss: 2.643509036064148\n",
      "\n",
      "Epoch:   20/25    Loss: 2.663993935585022\n",
      "\n",
      "Epoch:   20/25    Loss: 2.6790711035728454\n",
      "\n",
      "Epoch:   20/25    Loss: 2.7049645166397096\n",
      "\n",
      "Epoch:   20/25    Loss: 2.70785759305954\n",
      "\n",
      "Epoch:   20/25    Loss: 2.7059489998817443\n",
      "\n",
      "Epoch:   20/25    Loss: 2.728218497276306\n",
      "\n",
      "Epoch:   20/25    Loss: 2.738645821094513\n",
      "\n",
      "Epoch:   20/25    Loss: 2.769261274814606\n",
      "\n",
      "Epoch:   20/25    Loss: 2.7636462926864622\n",
      "\n",
      "Epoch:   21/25    Loss: 2.6797141684344186\n",
      "\n",
      "Epoch:   21/25    Loss: 2.590811703205109\n",
      "\n",
      "Epoch:   21/25    Loss: 2.6034995455741883\n",
      "\n",
      "Epoch:   21/25    Loss: 2.6256689391136168\n",
      "\n",
      "Epoch:   21/25    Loss: 2.645283872127533\n",
      "\n",
      "Epoch:   21/25    Loss: 2.6562546162605285\n",
      "\n",
      "Epoch:   21/25    Loss: 2.682731101036072\n",
      "\n",
      "Epoch:   21/25    Loss: 2.6828139524459838\n",
      "\n",
      "Epoch:   21/25    Loss: 2.693458671092987\n",
      "\n",
      "Epoch:   21/25    Loss: 2.7055781598091126\n",
      "\n",
      "Epoch:   21/25    Loss: 2.725803728580475\n",
      "\n",
      "Epoch:   21/25    Loss: 2.747095009803772\n",
      "\n",
      "Epoch:   21/25    Loss: 2.748643099784851\n",
      "\n",
      "Epoch:   22/25    Loss: 2.6597692806531277\n",
      "\n",
      "Epoch:   22/25    Loss: 2.5654588098526\n",
      "\n",
      "Epoch:   22/25    Loss: 2.594270328760147\n",
      "\n",
      "Epoch:   22/25    Loss: 2.616382310628891\n",
      "\n",
      "Epoch:   22/25    Loss: 2.6440662546157836\n",
      "\n",
      "Epoch:   22/25    Loss: 2.6413160367012023\n",
      "\n",
      "Epoch:   22/25    Loss: 2.6494956216812136\n",
      "\n",
      "Epoch:   22/25    Loss: 2.6591703062057497\n",
      "\n",
      "Epoch:   22/25    Loss: 2.6672996978759764\n",
      "\n",
      "Epoch:   22/25    Loss: 2.695569216251373\n",
      "\n",
      "Epoch:   22/25    Loss: 2.7116618309020994\n",
      "\n",
      "Epoch:   22/25    Loss: 2.725586987495422\n",
      "\n",
      "Epoch:   22/25    Loss: 2.7474132409095766\n",
      "\n",
      "Epoch:   23/25    Loss: 2.6425021152378236\n",
      "\n",
      "Epoch:   23/25    Loss: 2.5631883401870725\n",
      "\n",
      "Epoch:   23/25    Loss: 2.5884375352859497\n",
      "\n",
      "Epoch:   23/25    Loss: 2.587847029685974\n",
      "\n",
      "Epoch:   23/25    Loss: 2.6097604975700377\n",
      "\n",
      "Epoch:   23/25    Loss: 2.6330496764183042\n",
      "\n",
      "Epoch:   23/25    Loss: 2.6372087824344637\n",
      "\n",
      "Epoch:   23/25    Loss: 2.649695957183838\n",
      "\n",
      "Epoch:   23/25    Loss: 2.66528467130661\n",
      "\n",
      "Epoch:   23/25    Loss: 2.6835353851318358\n",
      "\n",
      "Epoch:   23/25    Loss: 2.6917580909729004\n",
      "\n",
      "Epoch:   23/25    Loss: 2.6993147778511046\n",
      "\n",
      "Epoch:   23/25    Loss: 2.7352095308303834\n",
      "\n",
      "Epoch:   24/25    Loss: 2.611891325536281\n",
      "\n",
      "Epoch:   24/25    Loss: 2.552810824632645\n",
      "\n",
      "Epoch:   24/25    Loss: 2.584387104511261\n",
      "\n",
      "Epoch:   24/25    Loss: 2.596249171257019\n",
      "\n",
      "Epoch:   24/25    Loss: 2.59130916595459\n",
      "\n",
      "Epoch:   24/25    Loss: 2.6253699145317078\n",
      "\n",
      "Epoch:   24/25    Loss: 2.6202923293113707\n",
      "\n",
      "Epoch:   24/25    Loss: 2.63862423992157\n",
      "\n",
      "Epoch:   24/25    Loss: 2.6578328495025634\n",
      "\n",
      "Epoch:   24/25    Loss: 2.6620410323143004\n",
      "\n",
      "Epoch:   24/25    Loss: 2.675981827735901\n",
      "\n",
      "Epoch:   24/25    Loss: 2.675276035308838\n",
      "\n",
      "Epoch:   24/25    Loss: 2.6994691615104673\n",
      "\n",
      "Epoch:   25/25    Loss: 2.610744612627846\n",
      "\n",
      "Epoch:   25/25    Loss: 2.536155734539032\n",
      "\n",
      "Epoch:   25/25    Loss: 2.5606893205642702\n",
      "\n",
      "Epoch:   25/25    Loss: 2.5531453783512115\n",
      "\n",
      "Epoch:   25/25    Loss: 2.5850091309547425\n",
      "\n",
      "Epoch:   25/25    Loss: 2.5972999987602234\n",
      "\n",
      "Epoch:   25/25    Loss: 2.6019597239494323\n",
      "\n",
      "Epoch:   25/25    Loss: 2.607006297111511\n",
      "\n",
      "Epoch:   25/25    Loss: 2.6459733371734617\n",
      "\n",
      "Epoch:   25/25    Loss: 2.6550617311000826\n",
      "\n",
      "Epoch:   25/25    Loss: 2.662325550556183\n",
      "\n",
      "Epoch:   25/25    Loss: 2.6779383215904238\n",
      "\n",
      "Epoch:   25/25    Loss: 2.6812004787921904\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/torch/serialization.py:193: UserWarning: Couldn't retrieve source code for container of type RNN. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Trained and Saved\n",
      "Time taken: 5.182297595010864 hours\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "t0 = time.time()\n",
    "\n",
    "\n",
    "from workspace_utils import active_session\n",
    "with active_session():\n",
    "\n",
    "\n",
    "    # create model and move to gpu if available\n",
    "    rnn = RNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers, dropout=0.5)\n",
    "    if train_on_gpu:\n",
    "        rnn.cuda()\n",
    "\n",
    "    # defining loss and optimization functions for training\n",
    "    optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "    # training the model\n",
    "    trained_rnn = train_rnn(rnn, batch_size, optimizer, criterion, num_epochs, show_every_n_batches)\n",
    "\n",
    "    # saving the trained model\n",
    "    helper.save_model('./save/trained_rnn', trained_rnn)\n",
    "    print('Model Trained and Saved')\n",
    "    \n",
    "    \n",
    "t1 = time.time()\n",
    "print('Time taken:', (t1-t0)/3600, 'hours')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "_, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()\n",
    "trained_rnn = helper.load_model('./save/trained_rnn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate TV Script\n",
    "With the network trained and saved, I can use it to generate a new, \"fake\" Seinfeld TV script.\n",
    "\n",
    "### Generate Text\n",
    "To generate the text, the network needs to start with a single word and repeat its predictions until it reaches a set length. It takes a word id to start with, `prime_id`, and generates a set length of text, `predict_len`. Also note that it uses topk sampling to introduce some randomness in choosing the most likely next word, given an output set of word scores!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "def generate(rnn, prime_id, int_to_vocab, token_dict, pad_value, predict_len=100):\n",
    "    \n",
    "    rnn.eval()\n",
    "    \n",
    "    # create a sequence (batch_size=1) with the prime_id\n",
    "    current_seq = np.full((1, sequence_length), pad_value)\n",
    "    current_seq[-1][-1] = prime_id\n",
    "    predicted = [int_to_vocab[prime_id]]\n",
    "    \n",
    "    for _ in range(predict_len):\n",
    "        if train_on_gpu:\n",
    "            current_seq = torch.LongTensor(current_seq).cuda()\n",
    "        else:\n",
    "            current_seq = torch.LongTensor(current_seq)\n",
    "        \n",
    "        # initialize the hidden state\n",
    "        hidden = rnn.init_hidden(current_seq.size(0))\n",
    "        \n",
    "        # get the output of the rnn\n",
    "        output, _ = rnn(current_seq, hidden)\n",
    "        \n",
    "        # get the next word probabilities\n",
    "        p = F.softmax(output, dim=1).data\n",
    "        if(train_on_gpu):\n",
    "            p = p.cpu() # move to cpu\n",
    "         \n",
    "        # use top_k sampling to get the index of the next word\n",
    "        top_k = 5\n",
    "        p, top_i = p.topk(top_k)\n",
    "        top_i = top_i.numpy().squeeze()\n",
    "        \n",
    "        # select the likely next word index with some element of randomness\n",
    "        p = p.numpy().squeeze()\n",
    "        word_i = np.random.choice(top_i, p=p/p.sum())\n",
    "        \n",
    "        # retrieve that word from the dictionary\n",
    "        word = int_to_vocab[word_i]\n",
    "        predicted.append(word)     \n",
    "        \n",
    "        # the generated word becomes the next \"current sequence\" and the cycle can continue\n",
    "        current_seq = np.roll(current_seq, -1, 1)\n",
    "        current_seq[-1][-1] = word_i\n",
    "    \n",
    "    gen_sentences = ' '.join(predicted)\n",
    "    \n",
    "    # Replace punctuation tokens\n",
    "    for key, token in token_dict.items():\n",
    "        ending = ' ' if key in ['\\n', '(', '\"'] else ''\n",
    "        gen_sentences = gen_sentences.replace(' ' + token.lower(), key)\n",
    "    gen_sentences = gen_sentences.replace('\\n ', '\\n')\n",
    "    gen_sentences = gen_sentences.replace('( ', '(')\n",
    "    \n",
    "    # return all the sentences\n",
    "    return gen_sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a New Script\n",
    "It's time to generate the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:46: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jerry: shrubs.)\n",
      "\n",
      "jerry: hey, george!\n",
      "\n",
      "george: shhh!(he runs back and forth between a hug)\n",
      "\n",
      "elaine:(to jerry) what is this? a little test?\n",
      "\n",
      "jerry: oh, no.\n",
      "\n",
      "elaine: i mean, why not?\n",
      "\n",
      "jerry: well i was just admiring your skin.\n",
      "\n",
      "kramer: hey.\n",
      "\n",
      "jerry: hey, how's your father?\n",
      "\n",
      "george:(muttering) i don't know..(sees the flies) it's not safe. i don't want to see the milk anymore. i'm not going to commit suicide.\n",
      "\n",
      "jerry: i don't understand. maybe i can figure out why would i be friends to do anything...\n",
      "\n",
      "jerry: i don't think you want to do this.\n",
      "\n",
      "elaine: well, i was just leaving.\n",
      "\n",
      "jerry: well if you think about this...\n",
      "\n",
      "george: well, you know...\n",
      "\n",
      "kramer:(quietly) i said cubans, cosmo kramer.\n",
      "\n",
      "jerry: oh, no, not really.\n",
      "\n",
      "george: what do you mean?\n",
      "\n",
      "jerry: because i'm not going to punch him........ and i screamed at it.\n",
      "\n",
      "elaine: what?!\n",
      "\n",
      "jerry: well i just got it, too many misunderstandings. i can't take it. it's hard to be used to the cabin.\n",
      "\n",
      "jerry:(handing over the phone) puddy, you're gonna take this thing to me?\n",
      "\n",
      "kramer: no, i can't...\n",
      "\n",
      "jerry: i mean, what do you have to go with me to a vegetable stand?\n",
      "\n",
      "elaine: no i haven't. i have to tell you, i'm not a doctor.\n",
      "\n",
      "kramer: yeah.\n",
      "\n",
      "george: you know what, maybe you shouldn't take a look.\n",
      "\n",
      "jerry: i know, it's a big thing to meet the spray cologne.\n",
      "\n",
      "jerry: i don't know. maybe i liked it.\n",
      "\n",
      "kramer: no, i got it.\n",
      "\n",
      "jerry: you got a\n"
     ]
    }
   ],
   "source": [
    "gen_length = 400 \n",
    "prime_word = 'jerry' # name for starting the script\n",
    "\n",
    "pad_word = helper.SPECIAL_WORDS['PADDING']\n",
    "generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)\n",
    "print(generated_script)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save your favorite scripts\n",
    "\n",
    "Once you have a script that you like (or find interesting), save it to a text file!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save script to a text file\n",
    "f =  open(\"generated_script_1.txt\",\"w\")\n",
    "f.write(generated_script)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}