tutorial2-cnns-in-pytorch.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vGC2D_19Re73"
      },
      "source": [
        "# Convolutional Neural Networks! #\n",
        "\n",
        "Today we'll explore convnets on the [Cifar10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. First we start with some magic incantations (and check to make sure we're using the GPU):\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NOVeMnEcLrt4"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torchvision\n",
        "import torchvision.transforms as transforms\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "\n",
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "print(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2eYxX893R_4P"
      },
      "source": [
        "## Getting the Cifar Data\n",
        "Is pretty easy, we can just use the built in dataset functionality. This time we might use some data augmentation.\n",
        "\n",
        "**`RandomCrop(32, padding=4)`** - This means we'll take random 32x32 crops out of the image zero padded with 4 pixels per size. Since our image was 32x32 this means we first zero pad to make it 40x40 (adding 4 pixels per side) and then take a 32x32 crop out of that. This means that the network sees slightly shifted around every time so it is harder to overfit to specific pixels in specific places. This forces the network to learn more robust filters and reduces overfitting.\n",
        "\n",
        "**`RandomHorizontalFlip()`** - This means half the time we will flip the image horizontally. Same basically as above, the network sees shifted versions of the data so it's harder to overfit.\n",
        "\n",
        "**Note:** data augmentation is turned off by default. We'll try to train the network normally and then see what affect data augmentation has."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S4zbAcWsL-JM"
      },
      "outputs": [],
      "source": [
        "def get_cifar10_data(augmentation=0):\n",
        "  # Data augmentation transformations. Not for Testing!\n",
        "  if augmentation:\n",
        "    transform_train = transforms.Compose([\n",
        "      transforms.RandomCrop(32, padding=4, padding_mode='edge'), # Take 32x32 crops from 40x40 padded images\n",
        "      transforms.RandomHorizontalFlip(),    # 50% of time flip image along y-axis\n",
        "      transforms.ToTensor(),\n",
        "    ])\n",
        "  else: \n",
        "    transform_train = transforms.ToTensor()\n",
        "\n",
        "  transform_test = transforms.Compose([\n",
        "    transforms.ToTensor(),\n",
        "  ])\n",
        "\n",
        "  trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True,\n",
        "                                        transform=transform_train)\n",
        "  trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True,\n",
        "                                            num_workers=32)\n",
        "\n",
        "  testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True,\n",
        "                                      transform=transform_test)\n",
        "  testloader = torch.utils.data.DataLoader(testset, batch_size=128, shuffle=False,\n",
        "                                          num_workers=32)\n",
        "  classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n",
        "  return {'train': trainloader, 'test': testloader, 'classes': classes}\n",
        "\n",
        "data = get_cifar10_data()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jqLGlQPCTAfG"
      },
      "source": [
        "----------------\n",
        "**Handy tip:** in colab you can run commands in the underlying virtual machine (outside of python) by prefixing with an exclamation mark, like this:\n",
        "\n",
        "    !ls ./data\n",
        "\n",
        "You can use that syntax to run arbitrary commands in the underlying virtual machine. For instance you can run your homework projects here:\n",
        "\n",
        "    !git clone https://github.com/pjreddie/uwimg\n",
        "    !cd uwimg; ls; make; ./uwimg test hw0\n",
        "\n",
        "Note every time you call a command with `!` colab spawns a new shell so commands like `!cd` don't presist between lines.\n",
        "\n",
        "I wouldn't do the C homework here or anything but this can be useful for installing dependencies.\n",
        "\n",
        "-----------------"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MAAtLJ6INUYT"
      },
      "outputs": [],
      "source": [
        "!ls ./data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JFlQkN3STVag"
      },
      "source": [
        "Looks like our data is in the right folder! For CIFAR10 the training set is 50,000 images and the test set is 10,000:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xVuyEK_DNHQd"
      },
      "outputs": [],
      "source": [
        "print(data['train'].__dict__)\n",
        "print(data['test'].__dict__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Emab4ufITnty"
      },
      "source": [
        "### Visualizing Some Data ###\n",
        "Our handy visualizations from last time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4tpWI3mzNsCW"
      },
      "outputs": [],
      "source": [
        "dataiter = iter(data['train'])\n",
        "images, labels = next(dataiter)\n",
        "print(images.size())\n",
        "\n",
        "def imshow(img):\n",
        "    npimg = img.numpy()\n",
        "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
        "    plt.show()\n",
        "\n",
        "# show images\n",
        "imshow(torchvision.utils.make_grid(images))\n",
        "# print labels\n",
        "print(\"Labels:\" + ' '.join('%9s' % data['classes'][labels[j]] for j in range(8)))\n",
        "\n",
        "\n",
        "flat = torch.flatten(images, 1)\n",
        "print(images.size())\n",
        "print(flat.size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iZIOXDp4TskV"
      },
      "source": [
        "## Defining the Networks!##\n",
        "We'll try out the SimpleNet from last time. Note: the number of inputs has changed since our input images is now a 32x32 RGB image (32x32x3 = 3072)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mWXiqJ8fOTt6"
      },
      "outputs": [],
      "source": [
        "class SimpleNet(nn.Module):\n",
        "    def __init__(self, inputs=3072, hidden=512, outputs=10):\n",
        "        super(SimpleNet, self).__init__()\n",
        "        self.fc1 = nn.Linear(inputs, hidden)\n",
        "        self.fc2 = nn.Linear(hidden, outputs)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = torch.flatten(x, 1) # Takes image-like to vector-like\n",
        "        x = self.fc1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.fc2(x)\n",
        "        return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vy9ndWnSRmpg"
      },
      "source": [
        "### Defining the CNN\n",
        "\n",
        "Our convolutional neural network is pretty simple to start. It has 3 convolutional layers followed by a fully connected layer.\n",
        "\n",
        "**conv1**\n",
        " - Input: `32 x 32 x 3` image\n",
        " - 16 filters, size `3 x 3`, stride 2\n",
        " - Output: `16 x 16 x 16` image\n",
        "\n",
        "**conv2**\n",
        " - Input: `16 x 16 x 16` image\n",
        " - 32 filters, size `3 x 3`, stride 2\n",
        " - Output: `8 x 8 x 32` image\n",
        "\n",
        "**conv3**\n",
        " - Input: `8 x 8 x 32` image\n",
        " - 64 filters, size `3 x 3`, stride 2\n",
        " - Output: `4 x 4 x 64` image\n",
        "\n",
        "**fc1**\n",
        " - Input: 1024 vector\n",
        " - Output: 10 vector (unnormalized class probabilities)\n",
        "\n",
        "**Note:** after the 3rd convolutional layer we have to convert the feature map between tensor formats. It's in an image-like format (NxCxHxW) but fully-connected layers need it to be in a vector-like format (NxM)\n",
        "\n",
        "To do that we just call our normal `x = torch.flatten(x,1)` on the feature map.\n",
        "\n",
        "You can also see in the `forward` function we use the `relu` activation function after each convolutional layer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1B7bc_89Rjaq"
      },
      "outputs": [],
      "source": [
        "class ConvNet(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(ConvNet, self).__init__() # https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html\n",
        "        # Input 32x32x3 image\n",
        "        # 16 filters\n",
        "        # 3x3 filter size (they also have 3 channels)\n",
        "        # stride 2 (downsampling by factor of 2)\n",
        "        # Output image: 16x16x16\n",
        "        self.conv1 = nn.Conv2d(3, 16, 3, stride=2, padding=1)\n",
        "\n",
        "        # Input 16x16x16 image\n",
        "        # 32 filters\n",
        "        # 3x3x16 filter size (they also have 16 channels)\n",
        "        # stride 2 (downsampling by factor of 2)\n",
        "        # Output image: 8x8x32\n",
        "        self.conv2 = nn.Conv2d(16, 32, 3, stride=2, padding=1)\n",
        "\n",
        "\n",
        "        # Exercise left to the reader\n",
        "        # Output image: 4x4x64 -> 1024 neurons\n",
        "        self.conv3 = nn.Conv2d(32, 64, 3, stride=2, padding=1)\n",
        "\n",
        "        self.fc1 = nn.Linear(1024, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.conv1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.conv2(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.conv3(x)\n",
        "        x = F.relu(x)\n",
        "        x = torch.flatten(x, 1)\n",
        "        x = self.fc1(x)\n",
        "        return x\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LullDR7pT_ca"
      },
      "source": [
        "### Training Code\n",
        "\n",
        "Not much to see here... It's the same as last time I think."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VgrhxQ_NOrAH"
      },
      "outputs": [],
      "source": [
        "\n",
        "def train(net, dataloader, epochs=1, lr=0.01, momentum=0.9, decay=0.0, verbose=1):\n",
        "  net.to(device)\n",
        "  losses = []\n",
        "  criterion = nn.CrossEntropyLoss()\n",
        "  optimizer = optim.SGD(net.parameters(), lr=lr, momentum=momentum, weight_decay=decay)\n",
        "  for epoch in range(epochs):\n",
        "    sum_loss = 0.0\n",
        "    for i, batch in enumerate(dataloader, 0):\n",
        "        # get the inputs; data is a list of [inputs, labels]\n",
        "        inputs, labels = batch[0].to(device), batch[1].to(device)\n",
        "\n",
        "        # zero the parameter gradients\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # forward + backward + optimize \n",
        "        outputs = net(inputs)\n",
        "        loss = criterion(outputs, labels)\n",
        "        loss.backward()  # autograd magic, computes all the partial derivatives\n",
        "        optimizer.step() # takes a step in gradient direction\n",
        "\n",
        "        # print statistics\n",
        "        losses.append(loss.item())\n",
        "        sum_loss += loss.item()\n",
        "        if i % 100 == 99:    # print every 100 mini-batches\n",
        "            if verbose:\n",
        "              print('[%d, %5d] loss: %.3f' %\n",
        "                  (epoch + 1, i + 1, sum_loss / 100))\n",
        "            sum_loss = 0.0\n",
        "  return losses\n",
        "\n",
        "def accuracy(net, dataloader):\n",
        "  correct = 0\n",
        "  total = 0\n",
        "  with torch.no_grad():\n",
        "      for batch in dataloader:\n",
        "          images, labels = batch[0].to(device), batch[1].to(device)\n",
        "          outputs = net(images)\n",
        "          _, predicted = torch.max(outputs.data, 1)\n",
        "          total += labels.size(0)\n",
        "          correct += (predicted == labels).sum().item()\n",
        "  return correct/total\n",
        "\n",
        "def smooth(x, size):\n",
        "  return np.convolve(x, np.ones(size)/size, mode='valid')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bt80jS3zUHXw"
      },
      "source": [
        "## Train the networks! ##\n",
        "\n",
        "It's time.\n",
        "\n",
        "First we'll start with SimpleNet:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FZzpV5IfUXk7"
      },
      "source": [
        "### SimpleNet on Cifar10"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4jeCMeEGOuQX"
      },
      "outputs": [],
      "source": [
        "net = SimpleNet(inputs=3072)\n",
        "\n",
        "losses = train(net, data['train'], epochs=5, lr=.01)\n",
        "plt.plot(smooth(losses,50))\n",
        "\n",
        "print(\"Training accuracy: %f\" % accuracy(net, data['train']))\n",
        "print(\"Testing  accuracy: %f\" % accuracy(net, data['test']))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DqOKSMzGUaQi"
      },
      "source": [
        "### ConvNet on CIFAR 10"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CdS62eiRQPRx"
      },
      "outputs": [],
      "source": [
        "conv_net = ConvNet()\n",
        "\n",
        "conv_losses = train(conv_net, data['train'], epochs=15, lr=.01)\n",
        "plt.plot(smooth(conv_losses, 50))\n",
        "\n",
        "print(\"Training accuracy: %f\" % accuracy(conv_net, data['train']))\n",
        "print(\"Testing  accuracy: %f\" % accuracy(conv_net, data['test']))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KFAQJCnoET54"
      },
      "outputs": [],
      "source": [
        "plt.plot(smooth(losses,50), 'r-')\n",
        "plt.plot(smooth(conv_losses, 50), 'b-')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OILkg6E0SDvI"
      },
      "source": [
        "### Simulated Annealing\n",
        "\n",
        "It can be useful to slowly lower the learning rate over time so that the network converges to a better local optimum. Let's try it!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y4rlUxOiPrkl"
      },
      "outputs": [],
      "source": [
        "anneal_net = ConvNet()\n",
        "\n",
        "anneal_losses =  train(anneal_net, data['train'], epochs=5, lr=.1)\n",
        "anneal_losses += train(anneal_net, data['train'], epochs=5, lr=.01)\n",
        "anneal_losses += train(anneal_net, data['train'], epochs=5, lr=.001)\n",
        "\n",
        "plt.plot(smooth(anneal_losses, 50))\n",
        "\n",
        "print(\"Training accuracy: %f\" % accuracy(anneal_net, data['train']))\n",
        "print(\"Testing  accuracy: %f\" % accuracy(anneal_net, data['test']))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yNA4U0OCSsGX"
      },
      "source": [
        "### Batch Normalization!\n",
        "\n",
        "Training is better and faster with batchnorm. Let's add it in to our network:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EM9LFxWnS5_H"
      },
      "outputs": [],
      "source": [
        "class ConvBNNet(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(ConvBNNet, self).__init__() # https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html\n",
        "        self.conv1 = nn.Conv2d(3, 16, 3, stride=2, padding=1)\n",
        "        self.bn1 = nn.BatchNorm2d(16)\n",
        "        self.conv2 = nn.Conv2d(16, 32, 3, stride=2, padding=1)\n",
        "        self.bn2 = nn.BatchNorm2d(32)\n",
        "        self.conv3 = nn.Conv2d(32, 64, 3, stride=2, padding=1)\n",
        "        self.bn3 = nn.BatchNorm2d(64)\n",
        "        self.fc1 = nn.Linear(1024, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.conv1(x)\n",
        "        x = self.bn1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.conv2(x)\n",
        "        x = self.bn2(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.conv3(x)\n",
        "        x = self.bn3(x)\n",
        "        x = F.relu(x)\n",
        "        x = torch.flatten(x, 1)\n",
        "        x = self.fc1(x)\n",
        "        return x\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0ZXuotZaOiNV"
      },
      "outputs": [],
      "source": [
        "norm_net = ConvBNNet()\n",
        "\n",
        "norm_losses = train(norm_net, data['train'], epochs=15, lr=.01)\n",
        "\n",
        "plt.plot(smooth(norm_losses, 50))\n",
        "\n",
        "print(\"Training accuracy: %f\" % accuracy(norm_net, data['train']))\n",
        "print(\"Testing  accuracy: %f\" % accuracy(norm_net, data['test']))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F8YEJTuwQLA1"
      },
      "outputs": [],
      "source": [
        "plt.plot(smooth(losses,50), 'r-')\n",
        "plt.plot(smooth(conv_losses, 50), 'b-')\n",
        "plt.plot(smooth(norm_losses, 50), 'g-')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8UE8S3E3P0X9"
      },
      "outputs": [],
      "source": [
        "lr_net = ConvBNNet()\n",
        "\n",
        "lr_losses = train(lr_net, data['train'], epochs=15, lr=.1)\n",
        "\n",
        "plt.plot(smooth(lr_losses, 50))\n",
        "\n",
        "print(\"Training accuracy: %f\" % accuracy(lr_net, data['train']))\n",
        "print(\"Testing  accuracy: %f\" % accuracy(lr_net, data['test']))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RefGZ70sRw3r"
      },
      "outputs": [],
      "source": [
        "#plt.plot(smooth(losses,50), 'r-')\n",
        "#plt.plot(smooth(conv_losses, 50), 'b-')\n",
        "plt.plot(smooth(norm_losses, 50), 'g-')\n",
        "plt.plot(smooth(lr_losses, 50), 'r-')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "30q05yadSe-9"
      },
      "outputs": [],
      "source": [
        "anneal2_net = ConvBNNet()\n",
        "\n",
        "anneal2_losses =  train(anneal2_net, data['train'], epochs=5, lr=.1)\n",
        "anneal2_losses += train(anneal2_net, data['train'], epochs=5, lr=.01)\n",
        "anneal2_losses += train(anneal2_net, data['train'], epochs=5, lr=.001)\n",
        "\n",
        "\n",
        "plt.plot(smooth(anneal2_losses, 50))\n",
        "\n",
        "print(\"Training accuracy: %f\" % accuracy(anneal2_net, data['train']))\n",
        "print(\"Testing  accuracy: %f\" % accuracy(anneal2_net, data['test']))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KiwfHa_ZFui4"
      },
      "source": [
        "### Weight Decay\n",
        "\n",
        "We can try adding in some weight decay now because we are overfitting to the data quite a bit"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i8qzqE4TF0KV"
      },
      "outputs": [],
      "source": [
        "decay_net = ConvBNNet()\n",
        "\n",
        "decay_losses =  train(decay_net, data['train'], epochs=5, lr=.1  , decay = .0005)\n",
        "decay_losses += train(decay_net, data['train'], epochs=5, lr=.01 , decay = .0005)\n",
        "decay_losses += train(decay_net, data['train'], epochs=5, lr=.001, decay = .0005)\n",
        "\n",
        "\n",
        "plt.plot(smooth(decay_losses, 50))\n",
        "\n",
        "print(\"Training accuracy: %f\" % accuracy(decay_net, data['train']))\n",
        "print(\"Testing  accuracy: %f\" % accuracy(decay_net, data['test']))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GV6dB2hGVFwF"
      },
      "outputs": [],
      "source": [
        "#plt.plot(smooth(losses,50), 'r-')\n",
        "#plt.plot(smooth(conv_losses, 50), 'r-')\n",
        "#plt.plot(smooth(norm_losses, 50), 'g-')\n",
        "plt.plot(smooth(anneal2_losses, 50), 'b-')\n",
        "plt.plot(smooth(decay_losses, 50), 'm-')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VdfRaxZxWtfZ"
      },
      "source": [
        "#### Data Augmentation ####\n",
        "\n",
        "Our training accuracy is much higher than our testing accuracy which indicates overfitting. Let's add in data augmentation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "h-5etXowXEyV"
      },
      "outputs": [],
      "source": [
        "data_aug = get_cifar10_data(augmentation=1)\n",
        "data_net = ConvBNNet()\n",
        "\n",
        "data_losses =  train(data_net, data_aug['train'], epochs=5, lr=.1  , decay=.0005)\n",
        "data_losses += train(data_net, data_aug['train'], epochs=5, lr=.01 , decay=.0005)\n",
        "data_losses += train(data_net, data_aug['train'], epochs=5, lr=.001, decay=.0005)\n",
        "\n",
        "\n",
        "plt.plot(smooth(data_losses, 50))\n",
        "\n",
        "print(\"Training accuracy: %f\" % accuracy(data_net, data_aug['train']))\n",
        "print(\"Testing  accuracy: %f\" % accuracy(data_net, data_aug['test']))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "XeHcb2GUZS_W"
      },
      "outputs": [],
      "source": [
        "plt.plot(smooth(decay_losses, 50), 'r-')\n",
        "plt.plot(smooth(data_losses, 50), 'g-')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rFepNt-MZz4h"
      },
      "outputs": [],
      "source": [
        "final_net = ConvBNNet()\n",
        "\n",
        "final_losses =  train(final_net, data_aug['train'], epochs=15, lr=.1  , decay=.0005)\n",
        "final_losses += train(final_net, data_aug['train'], epochs=5, lr=.01 , decay=.0005)\n",
        "final_losses += train(final_net, data_aug['train'], epochs=5, lr=.001, decay=.0005)\n",
        "\n",
        "\n",
        "plt.plot(smooth(final_losses, 50))\n",
        "\n",
        "print(\"Training accuracy: %f\" % accuracy(final_net, data_aug['train']))\n",
        "print(\"Testing  accuracy: %f\" % accuracy(final_net, data_aug['test']))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rQ1-ePKDcQod"
      },
      "outputs": [],
      "source": [
        "plt.plot(smooth(decay_losses, 50), 'r-')\n",
        "plt.plot(smooth(data_losses, 50), 'g-')\n",
        "plt.plot(smooth(final_losses, 50), 'b-')"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}