dlnd_tv_script_generation-zh.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 生成电视剧剧本\n",
    "\n",
    "在这个项目中，你将使用 RNN 创作你自己的[《辛普森一家》](https://zh.wikipedia.org/wiki/%E8%BE%9B%E6%99%AE%E6%A3%AE%E4%B8%80%E5%AE%B6)电视剧剧本。你将会用到《辛普森一家》第 27 季中部分剧本的[数据集](https://www.kaggle.com/wcukierski/the-simpsons-by-the-data)。你创建的神经网络将为一个在 [Moe 酒馆](https://simpsonswiki.com/wiki/Moe's_Tavern)中的场景生成一集新的剧本。\n",
    "\n",
    "## 获取数据\n",
    "我们早已为你提供了数据`./data/Seinfeld_Scripts.txt`。我们建议你打开文档来看看这个文档内容。\n",
    "\n",
    ">* 第一步，我们来读入文档，并看几段例子。\n",
    "* 然后，你需要定义并训练一个 RNN 网络来生成新的剧本！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "# load in data\n",
    "import helper\n",
    "data_dir = './data/Seinfeld_Scripts.txt'\n",
    "text = helper.load_data(data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 探索数据\n",
    "使用 `view_line_range` 来查阅数据的不同部分，这个部分会让你对整体数据有个基础的了解。你会发现，文档中全是小写字母，并且所有的对话都是使用 `\\n` 来分割的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "view_line_range = (0, 10)\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "\n",
    "print('Dataset Stats')\n",
    "print('Roughly the number of unique words: {}'.format(len({word: None for word in text.split()})))\n",
    "\n",
    "lines = text.split('\\n')\n",
    "print('Number of lines: {}'.format(len(lines)))\n",
    "word_count_line = [len(line.split()) for line in lines]\n",
    "print('Average number of words in each line: {}'.format(np.average(word_count_line)))\n",
    "\n",
    "print()\n",
    "print('The lines {} to {}:'.format(*view_line_range))\n",
    "print('\\n'.join(text.split('\\n')[view_line_range[0]:view_line_range[1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 实现预处理函数\n",
    "对数据集进行的第一个操作是预处理。请实现下面两个预处理函数：\n",
    "\n",
    "- 查询表\n",
    "- 标记符号\n",
    "\n",
    "### 查询表\n",
    "要创建词嵌入，你首先要将词语转换为 id。请在这个函数中创建两个字典：\n",
    "\n",
    "- 将词语转换为 id 的字典，我们称它为 `vocab_to_int`\n",
    "- 将 id 转换为词语的字典，我们称它为 `int_to_vocab`\n",
    "\n",
    "请在下面的元组中返回这些字典\n",
    " `(vocab_to_int, int_to_vocab)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import problem_unittests as tests\n",
    "\n",
    "def create_lookup_tables(text):\n",
    "    \"\"\"\n",
    "    Create lookup tables for vocabulary\n",
    "    :param text: The text of tv scripts split into words\n",
    "    :return: A tuple of dicts (vocab_to_int, int_to_vocab)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    \n",
    "    # return tuple\n",
    "    return (None, None)\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_create_lookup_tables(create_lookup_tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 标记符号的字符串\n",
    "我们会使用空格当作分隔符，来将剧本分割为词语数组。然而，句号和感叹号等符号使得神经网络难以分辨“再见”和“再见！”之间的区别。\n",
    "\n",
    "实现函数 `token_lookup` 来返回一个字典，这个字典用于将 “!” 等符号标记为 “||Exclamation_Mark||” 形式。为下列符号创建一个字典，其中符号为标志，值为标记。\n",
    "\n",
    "- period ( . )\n",
    "- comma ( , )\n",
    "- quotation mark ( \" )\n",
    "- semicolon ( ; )\n",
    "- exclamation mark ( ! )\n",
    "- question mark ( ? )\n",
    "- left parenthesis ( ( )\n",
    "- right parenthesis ( ) )\n",
    "- dash ( -- )\n",
    "- return ( \\n )\n",
    "\n",
    "这个字典将用于标记符号并在其周围添加分隔符（空格）。这能将符号视作单独词汇分割开来，并使神经网络更轻松地预测下一个词汇。请确保你并没有使用容易与词汇混淆的标记。与其使用 “dash” 这样的标记，试试使用“||dash||”。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def token_lookup():\n",
    "    \"\"\"\n",
    "    Generate a dict to turn punctuation into a token.\n",
    "    :return: Tokenized dictionary where the key is the punctuation and the value is the token\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "        \n",
    "    return None\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_tokenize(token_lookup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预处理并保存所有数据\n",
    "运行以下代码将预处理所有数据，并将它们保存至文件。建议你查看`helpers.py` 文件中的 `preprocess_and_save_data` 代码来看这一步在做什么，但是你不需要修改`helpers.py`中的函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "# pre-process training data\n",
    "helper.preprocess_and_save_data(data_dir, token_lookup, create_lookup_tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 检查点\n",
    "这是你遇到的第一个检点。如果你想要回到这个 notebook，或需要重新打开 notebook，你都可以从这里开始。预处理的数据都已经保存完毕。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "int_text, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建神经网络\n",
    "在本节中，你会构建 RNN 中的必要 Module，以及 前向、后向函数。\n",
    "\n",
    "### 检查 GPU 访问权限"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import torch\n",
    "\n",
    "# Check for a GPU\n",
    "train_on_gpu = torch.cuda.is_available()\n",
    "if not train_on_gpu:\n",
    "    print('No GPU found. Please use a GPU to train your neural network.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 输入\n",
    "让我们开始预处理输入数据。我们会使用 [TensorDataset](http:https://pytorch.org/docs/master/data.html#torch.utils.data.TensorDataset) 来为数据库提供一个数据格式；以及一个 [DataLoader](http:https://pytorch.org/docs/master/data.html#torch.utils.data.DataLoader), 该对象会实现 batching，shuffling 以及其他数据迭代功能。\n",
    "\n",
    "你可以通过传入 特征 和目标 tensors 来创建 TensorDataset，随后创建一个 DataLoader 。\n",
    "```\n",
    "data = TensorDataset(feature_tensors, target_tensors)\n",
    "data_loader = torch.utils.data.DataLoader(data, \n",
    "                                          batch_size=batch_size)\n",
    "```\n",
    "\n",
    "### Batching\n",
    " 通过 `TensorDataset` 和 `DataLoader` 类来实现  `batch_data` 函数来将 `words` 数据分成 `batch_size` 批次。\n",
    "\n",
    ">你可以使用 DataLoader 来分批 单词, 但是你可以自由设置 `feature_tensors` 和 `target_tensors` 的大小以及 `sequence_length`。\n",
    "\n",
    "比如，我们有如下输入:\n",
    "```\n",
    "words = [1, 2, 3, 4, 5, 6, 7]\n",
    "sequence_length = 4\n",
    "```\n",
    "\n",
    "你的第一个 `feature_tensor` 会包含:\n",
    "```\n",
    "[1, 2, 3, 4]\n",
    "```\n",
    "随后的 `target_tensor` 会是接下去的一个字符值:\n",
    "```\n",
    "5\n",
    "```\n",
    "那么，第二组的`feature_tensor`, `target_tensor` 则如下所示:\n",
    "```\n",
    "[2, 3, 4, 5]  # features\n",
    "6             # target\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "\n",
    "def batch_data(words, sequence_length, batch_size):\n",
    "    \"\"\"\n",
    "    Batch the neural network data using DataLoader\n",
    "    :param words: The word ids of the TV scripts\n",
    "    :param sequence_length: The sequence length of each batch\n",
    "    :param batch_size: The size of each batch; the number of sequences in a batch\n",
    "    :return: DataLoader with batched data\n",
    "    \"\"\"\n",
    "    # TODO: Implement function\n",
    "    \n",
    "    # return a dataloader\n",
    "    return None\n",
    "\n",
    "# there is no test for this function, but you are encouraged to create\n",
    "# print statements and tests of your own\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 测试你的 dataloader \n",
    "\n",
    "你需要改写下述代码来测试 batching 函数，改写后的代码会现在的比较类似。\n",
    "\n",
    "下面，我们生成了一些测试文本数据，并使用了一个你上面写 dataloader 。然后，我们会得到一些使用`sample_x`输入以及`sample_y`目标生成的文本。\n",
    "\n",
    "你的代码会返回如下结果(通常是不同的顺序，如果你 shuffle 了你的数据):\n",
    "\n",
    "```\n",
    "torch.Size([10, 5])\n",
    "tensor([[ 28,  29,  30,  31,  32],\n",
    "        [ 21,  22,  23,  24,  25],\n",
    "        [ 17,  18,  19,  20,  21],\n",
    "        [ 34,  35,  36,  37,  38],\n",
    "        [ 11,  12,  13,  14,  15],\n",
    "        [ 23,  24,  25,  26,  27],\n",
    "        [  6,   7,   8,   9,  10],\n",
    "        [ 38,  39,  40,  41,  42],\n",
    "        [ 25,  26,  27,  28,  29],\n",
    "        [  7,   8,   9,  10,  11]])\n",
    "\n",
    "torch.Size([10])\n",
    "tensor([ 33,  26,  22,  39,  16,  28,  11,  43,  30,  12])\n",
    "```\n",
    "\n",
    "### 大小\n",
    "你的 sample_x 应该是 `(batch_size, sequence_length)`的 大小 或者是(10, 5)， sample_y 应该是 一维的: batch_size (10)。\n",
    "\n",
    "### 值\n",
    "\n",
    "你应该也会发现 sample_y, 是 test_text 数据中的*下一个*值。因此，对于一个输入的序列 `[ 28,  29,  30,  31,  32]` ，它的结尾是 `32`, 那么其相应的输出应该是 `33`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test dataloader\n",
    "\n",
    "test_text = range(50)\n",
    "t_loader = batch_data(test_text, sequence_length=5, batch_size=10)\n",
    "\n",
    "data_iter = iter(t_loader)\n",
    "sample_x, sample_y = data_iter.next()\n",
    "\n",
    "print(sample_x.shape)\n",
    "print(sample_x)\n",
    "print()\n",
    "print(sample_y.shape)\n",
    "print(sample_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 构建神经网络\n",
    "使用 PyTorch [Module class](http:https://pytorch.org/docs/master/nn.html#torch.nn.Module) 来实现一个 循环神经网络 RNN。你需要选择一个 GRU 或者 一个 LSTM。为了完成循环神经网络。为了实现 RNN，你需要实现以下类:\n",
    " - `__init__` - 初始化函数\n",
    " - `init_hidden` - LSTM/GRU 隐藏组昂泰的初始化函数\n",
    " - `forward` - 前向传播函数\n",
    " \n",
    "初始化函数需要创建神经网络的层数，并保存到类。前向传播函数会使用这些网络来进行前向传播，并生成输出和隐藏状态。\n",
    "\n",
    "在该流程完成后，**该模型的输出是 *最后的* 文字分数结果** 对于每段输入的文字序列，我们只需要输出一个单词，也就是，下一个单词。 \n",
    "\n",
    "### 提示\n",
    "\n",
    "1. 确保 lstm 的输出会链接一个 全链接层，你可以参考如下代码 `lstm_output = lstm_output.contiguous().view(-1, self.hidden_dim)`\n",
    "2. 你可以通过 reshape 模型最后输出的全链接层，来得到最终的文字分数:\n",
    "\n",
    "```\n",
    "# reshape into (batch_size, seq_length, output_size)\n",
    "output = output.view(batch_size, -1, self.output_size)\n",
    "# get last batch\n",
    "out = output[:, -1]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, dropout=0.5):\n",
    "        \"\"\"\n",
    "        Initialize the PyTorch RNN Module\n",
    "        :param vocab_size: The number of input dimensions of the neural network (the size of the vocabulary)\n",
    "        :param output_size: The number of output dimensions of the neural network\n",
    "        :param embedding_dim: The size of embeddings, should you choose to use them        \n",
    "        :param hidden_dim: The size of the hidden layer outputs\n",
    "        :param dropout: dropout to add in between LSTM/GRU layers\n",
    "        \"\"\"\n",
    "        super(RNN, self).__init__()\n",
    "        # TODO: Implement function\n",
    "        \n",
    "        # set class variables\n",
    "        \n",
    "        # define model layers\n",
    "    \n",
    "    \n",
    "    def forward(self, nn_input, hidden):\n",
    "        \"\"\"\n",
    "        Forward propagation of the neural network\n",
    "        :param nn_input: The input to the neural network\n",
    "        :param hidden: The hidden state        \n",
    "        :return: Two Tensors, the output of the neural network and the latest hidden state\n",
    "        \"\"\"\n",
    "        # TODO: Implement function   \n",
    "\n",
    "        # return one batch of output word scores and the hidden state\n",
    "        return None, None\n",
    "    \n",
    "    \n",
    "    def init_hidden(self, batch_size):\n",
    "        '''\n",
    "        Initialize the hidden state of an LSTM/GRU\n",
    "        :param batch_size: The batch_size of the hidden state\n",
    "        :return: hidden state of dims (n_layers, batch_size, hidden_dim)\n",
    "        '''\n",
    "        # Implement function\n",
    "        \n",
    "        # initialize hidden state with zero weights, and move to GPU if available\n",
    "        \n",
    "        return None\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_rnn(RNN, train_on_gpu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 定义前向及后向传播\n",
    "\n",
    "通过你实现的 RNN 类来进行前向及后项传播。你可以在训练循环中，不断地调用如下代码来实现：\n",
    "```\n",
    "loss = forward_back_prop(decoder, decoder_optimizer, criterion, inp, target)\n",
    "```\n",
    "\n",
    "函数中需要返回一个批次以及其隐藏状态的loss均值，你可以调用一个函数`RNN(inp, hidden)`来实现。记得，你可以通过调用`loss.item()` 来计算得到该loss。\n",
    "\n",
    "**如果使用 GPU，你需要将你的数据存到 GPU 的设备上。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def forward_back_prop(rnn, optimizer, criterion, inp, target, hidden):\n",
    "    \"\"\"\n",
    "    Forward and backward propagation on the neural network\n",
    "    :param decoder: The PyTorch Module that holds the neural network\n",
    "    :param decoder_optimizer: The PyTorch optimizer for the neural network\n",
    "    :param criterion: The PyTorch loss function\n",
    "    :param inp: A batch of input to the neural network\n",
    "    :param target: The target output for the batch of input\n",
    "    :return: The loss and the latest hidden state Tensor\n",
    "    \"\"\"\n",
    "    \n",
    "    # TODO: Implement Function\n",
    "    \n",
    "    # move data to GPU, if available\n",
    "    \n",
    "    # perform backpropagation and optimization\n",
    "\n",
    "    # return the loss over a batch and the hidden state produced by our model\n",
    "    return None, None\n",
    "\n",
    "# Note that these tests aren't completely extensive.\n",
    "# they are here to act as general checks on the expected outputs of your functions\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_forward_back_prop(RNN, forward_back_prop, train_on_gpu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 神经网络训练\n",
    "\n",
    "神经网络结构完成以及数据准备完后，我们可以开始训练网络了。\n",
    "\n",
    "### 训练循环\n",
    "\n",
    "训练循环是通过 `train_decoder` 函数实现的。该函数将进行 epochs 次数的训练。模型的训练成果会在一定批次的训练后，被打印出来。这个“一定批次”可以通过`show_every_n_batches` 来设置。你会在下一节设置这个参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "\n",
    "def train_rnn(rnn, batch_size, optimizer, criterion, n_epochs, show_every_n_batches=100):\n",
    "    batch_losses = []\n",
    "    \n",
    "    rnn.train()\n",
    "\n",
    "    print(\"Training for %d epoch(s)...\" % n_epochs)\n",
    "    for epoch_i in range(1, n_epochs + 1):\n",
    "        \n",
    "        # initialize hidden state\n",
    "        hidden = rnn.init_hidden(batch_size)\n",
    "        \n",
    "        for batch_i, (inputs, labels) in enumerate(train_loader, 1):\n",
    "            \n",
    "            # make sure you iterate over completely full batches, only\n",
    "            n_batches = len(train_loader.dataset)//batch_size\n",
    "            if(batch_i > n_batches):\n",
    "                break\n",
    "            \n",
    "            # forward, back prop\n",
    "            loss, hidden = forward_back_prop(rnn, optimizer, criterion, inputs, labels, hidden)          \n",
    "            # record loss\n",
    "            batch_losses.append(loss)\n",
    "\n",
    "            # printing loss stats\n",
    "            if batch_i % show_every_n_batches == 0:\n",
    "                print('Epoch: {:>4}/{:<4}  Loss: {}\\n'.format(\n",
    "                    epoch_i, n_epochs, np.average(batch_losses)))\n",
    "                batch_losses = []\n",
    "\n",
    "    # returns a trained rnn\n",
    "    return rnn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 超参数\n",
    "\n",
    "设置并训练以下超参数:\n",
    "-  `sequence_length`，序列长度 \n",
    "-  `batch_size`，分批大小\n",
    "-  `num_epochs`，循环次数\n",
    "-  `learning_rate`，Adam优化器的学习率\n",
    "-  `vocab_size`，唯一标示词汇的数量\n",
    "-  `output_size`，模型输出的大小 \n",
    "-  `embedding_dim`，词嵌入的维度，小于 vocab_size\n",
    "-  `hidden_dim`， 隐藏层维度\n",
    "-  `n_layers`， RNN的层数\n",
    "-  `show_every_n_batches`，打印结果的频次\n",
    "\n",
    "如果模型没有获得你预期的结果，调整 `RNN`类中的上述参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Data params\n",
    "# Sequence Length\n",
    "sequence_length =   # of words in a sequence\n",
    "# Batch Size\n",
    "batch_size = \n",
    "\n",
    "# data loader - do not change\n",
    "train_loader = batch_data(int_text, sequence_length, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Training parameters\n",
    "# Number of Epochs\n",
    "num_epochs = \n",
    "# Learning Rate\n",
    "learning_rate = \n",
    "\n",
    "# Model parameters\n",
    "# Vocab size\n",
    "vocab_size = \n",
    "# Output size\n",
    "output_size = \n",
    "# Embedding Dimension\n",
    "embedding_dim = \n",
    "# Hidden Dimension\n",
    "hidden_dim = \n",
    "# Number of RNN Layers\n",
    "n_layers = \n",
    "\n",
    "# Show stats for every n number of batches\n",
    "show_every_n_batches = 500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练\n",
    "下一节，通过预处理数据来训练神经网络。如果你的loss结果不好，可以通过调整超参数来修正。通常情况下，大的隐藏层及层数会带来比较好的效果，但同时也会消耗较长的时间来训练。\n",
    "> **你应该努力得到一个低于3.5的loss** \n",
    "\n",
    "你也可以试试不同的序列长度，该参数表明模型学习的范围大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "\n",
    "# create model and move to gpu if available\n",
    "rnn = RNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers, dropout=0.5)\n",
    "if train_on_gpu:\n",
    "    rnn.cuda()\n",
    "\n",
    "# defining loss and optimization functions for training\n",
    "optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# training the model\n",
    "trained_rnn = train_rnn(rnn, batch_size, optimizer, criterion, num_epochs, show_every_n_batches)\n",
    "\n",
    "# saving the trained model\n",
    "helper.save_model('./save/trained_rnn', trained_rnn)\n",
    "print('Model Trained and Saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 问题: 你如何决定你的模型超参数？\n",
    "比如，你是否试过不同的 different sequence_lengths 并发现哪个使得模型的收敛速度变化？那你的隐藏层数和层数呢？你是如何决定使用这个网络参数的？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**答案:** (在这里写下)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 检查点\n",
    "\n",
    "通过运行上面的训练单元，你的模型已经以`trained_rnn`名字存储，如果你存储了你的notebook， **你可以在之后的任何时间来访问你的代码和结果**. 下述代码可以帮助你重载你的结果!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import torch\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "_, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()\n",
    "trained_rnn = helper.load_model('./save/trained_rnn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成电视剧剧本\n",
    "你现在可以生成你的“假”电视剧剧本啦！\n",
    "\n",
    "### 生成文字\n",
    "你的神经网络会不断重复生成一个单词，直到生成满足你要求长度的剧本。使用 `generate` 函数来完成上述操作。首先，使用 `prime_id` 来生成word id，之后确定生成文本长度 `predict_len`。同时， topk 采样来引入文字选择的随机性!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "import torch.nn.functional as F\n",
    "\n",
    "def generate(rnn, prime_id, int_to_vocab, token_dict, pad_value, predict_len=100):\n",
    "    \"\"\"\n",
    "    Generate text using the neural network\n",
    "    :param decoder: The PyTorch Module that holds the trained neural network\n",
    "    :param prime_id: The word id to start the first prediction\n",
    "    :param int_to_vocab: Dict of word id keys to word values\n",
    "    :param token_dict: Dict of puncuation tokens keys to puncuation values\n",
    "    :param pad_value: The value used to pad a sequence\n",
    "    :param predict_len: The length of text to generate\n",
    "    :return: The generated text\n",
    "    \"\"\"\n",
    "    rnn.eval()\n",
    "    \n",
    "    # create a sequence (batch_size=1) with the prime_id\n",
    "    current_seq = np.full((1, sequence_length), pad_value)\n",
    "    current_seq[-1][-1] = prime_id\n",
    "    predicted = [int_to_vocab[prime_id]]\n",
    "    \n",
    "    for _ in range(predict_len):\n",
    "        if train_on_gpu:\n",
    "            current_seq = torch.LongTensor(current_seq).cuda()\n",
    "        else:\n",
    "            current_seq = torch.LongTensor(current_seq)\n",
    "        \n",
    "        # initialize the hidden state\n",
    "        hidden = rnn.init_hidden(current_seq.size(0))\n",
    "        \n",
    "        # get the output of the rnn\n",
    "        output, _ = rnn(current_seq, hidden)\n",
    "        \n",
    "        # get the next word probabilities\n",
    "        p = F.softmax(output, dim=1).data\n",
    "        if(train_on_gpu):\n",
    "            p = p.cpu() # move to cpu\n",
    "         \n",
    "        # use top_k sampling to get the index of the next word\n",
    "        top_k = 5\n",
    "        p, top_i = p.topk(top_k)\n",
    "        top_i = top_i.numpy().squeeze()\n",
    "        \n",
    "        # select the likely next word index with some element of randomness\n",
    "        p = p.numpy().squeeze()\n",
    "        word_i = np.random.choice(top_i, p=p/p.sum())\n",
    "        \n",
    "        # retrieve that word from the dictionary\n",
    "        word = int_to_vocab[word_i]\n",
    "        predicted.append(word)     \n",
    "        \n",
    "        # the generated word becomes the next \"current sequence\" and the cycle can continue\n",
    "        current_seq = np.roll(current_seq, -1, 1)\n",
    "        current_seq[-1][-1] = word_i\n",
    "    \n",
    "    gen_sentences = ' '.join(predicted)\n",
    "    \n",
    "    # Replace punctuation tokens\n",
    "    for key, token in token_dict.items():\n",
    "        ending = ' ' if key in ['\\n', '(', '\"'] else ''\n",
    "        gen_sentences = gen_sentences.replace(' ' + token.lower(), key)\n",
    "    gen_sentences = gen_sentences.replace('\\n ', '\\n')\n",
    "    gen_sentences = gen_sentences.replace('( ', '(')\n",
    "    \n",
    "    # return all the sentences\n",
    "    return gen_sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 生成一个新剧本\n",
    "是时候生成一个剧本啦。设置`gen_length` 剧本长度，设置 `prime_word`为以下任意词来开始生成吧:\n",
    "- \"jerry\"\n",
    "- \"elaine\"\n",
    "- \"george\"\n",
    "- \"kramer\"\n",
    "\n",
    "你可以把prime word 设置成 _任意 _ 单词, 但是使用名字开始会比较好(任何其他名字也是可以哒!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# run the cell multiple times to get different results!\n",
    "gen_length = 400 # modify the length to your preference\n",
    "prime_word = 'jerry' # name for starting the script\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "pad_word = helper.SPECIAL_WORDS['PADDING']\n",
    "generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)\n",
    "print(generated_script)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 存下你最爱的片段\n",
    "\n",
    "一旦你发现一段有趣或者好玩的片段，就把它存下啦！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# save script to a text file\n",
    "f =  open(\"generated_script_1.txt\",\"w\")\n",
    "f.write(generated_script)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 这个电视剧剧本是无意义的\n",
    "如果你的电视剧剧本不是很有逻辑也是ok的。下面是一个例子。\n",
    "\n",
    "### 生成剧本案例\n",
    "\n",
    ">jerry: what about me?\n",
    ">\n",
    ">jerry: i don't have to wait.\n",
    ">\n",
    ">kramer:(to the sales table)\n",
    ">\n",
    ">elaine:(to jerry) hey, look at this, i'm a good doctor.\n",
    ">\n",
    ">newman:(to elaine) you think i have no idea of this...\n",
    ">\n",
    ">elaine: oh, you better take the phone, and he was a little nervous.\n",
    ">\n",
    ">kramer:(to the phone) hey, hey, jerry, i don't want to be a little bit.(to kramer and jerry) you can't.\n",
    ">\n",
    ">jerry: oh, yeah. i don't even know, i know.\n",
    ">\n",
    ">jerry:(to the phone) oh, i know.\n",
    ">\n",
    ">kramer:(laughing) you know...(to jerry) you don't know.\n",
    "\n",
    "\n",
    "如果这个电视剧剧本毫无意义，那也没有关系。我们的训练文本不到一兆字节。为了获得更好的结果，你需要使用更小的词汇范围或是更多数据。幸运的是，我们的确拥有更多数据！在本项目开始之初我们也曾提过，这是[另一个数据集](https://www.kaggle.com/wcukierski/the-simpsons-by-the-data)的子集。我们并没有让你基于所有数据进行训练，因为这将耗费大量时间。然而，你可以随意使用这些数据训练你的神经网络。当然，是在完成本项目之后。\n",
    "# 提交项目\n",
    "在提交项目时，请确保你在保存 notebook 前运行了所有的单元格代码。请将 notebook 文件保存为 \"dlnd_tv_script_generation.ipynb\"，并将它作为 HTML 文件保存在 \"File\" -> \"Download as\" 中。请将 \"helper.py\" 和 \"problem_unittests.py\" 文件一并打包成 zip 文件提交。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}