tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MXNet Tutorial and Hand Written Digit Recognition\n",
    "\n",
    "In this tutorial we will go through the basic use case of MXNet and also touch on some advanced usages. This example is based on the MNIST dataset, which contains 70,000 images of hand written characters with 28-by-28 pixel size.\n",
    "\n",
    "This tutorial covers the following topics:\n",
    "- network definition.\n",
    "- Variable naming.\n",
    "- Basic data loading and training with feed-forward deep neural networks.\n",
    "- Monitoring intermediate outputs for debuging.\n",
    "- Custom training loop for advanced models.\n",
    "\n",
    "More examples:\n",
    "- [Neural Art](neuralart/neuralart.ipynb)\n",
    "- [Char LSTM](char-lstm/char-rnn.ipynb)\n",
    "- [Deep3D](deep3d/deep3d.ipynb)\n",
    "- [Minpy API](minpy/minpy_tutorial.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's import the modules and setup logging:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import mxnet as mx\n",
    "import numpy as np\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import logging\n",
    "\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.DEBUG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Network Definition\n",
    "Now we can start constructing our network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Variables are place holders for input arrays. We give each variable a unique name.\n",
    "data = mx.symbol.Variable('data')\n",
    "\n",
    "# The input is fed to a fully connected layer that computes Y=WX+b.\n",
    "# This is the main computation module in the network.\n",
    "# Each layer also needs an unique name. We'll talk more about naming in the next section.\n",
    "fc1  = mx.symbol.FullyConnected(data = data, name='fc1', num_hidden=128)\n",
    "# Activation layers apply a non-linear function on the previous layer's output.\n",
    "# Here we use Rectified Linear Unit (ReLU) that computes Y = max(X, 0).\n",
    "act1 = mx.symbol.Activation(data = fc1, name='relu1', act_type=\"relu\")\n",
    "\n",
    "fc2  = mx.symbol.FullyConnected(data = act1, name = 'fc2', num_hidden = 64)\n",
    "act2 = mx.symbol.Activation(data = fc2, name='relu2', act_type=\"relu\")\n",
    "\n",
    "fc3  = mx.symbol.FullyConnected(data = act2, name='fc3', num_hidden=10)\n",
    "# Finally we have a loss layer that compares the network's output with label and generates gradient signals.\n",
    "mlp  = mx.symbol.SoftmaxOutput(data = fc3, name = 'softmax')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the network we just defined with MXNet's visualization module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http:https://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"102pt\" height=\"536pt\"\n",
       " viewBox=\"0.00 0.00 102.00 536.00\" xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 532)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-532 98,-532 98,4 -4,4\"/>\n",
       "<!-- fc1 -->\n",
       "<g id=\"node1\" class=\"node\"><title>fc1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-58 -7.10543e-15,-58 -7.10543e-15,-0 94,-0 94,-58\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-32.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-17.8\" font-family=\"Times,serif\" font-size=\"14.00\">128</text>\n",
       "</g>\n",
       "<!-- relu1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>relu1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"94,-152 -7.10543e-15,-152 -7.10543e-15,-94 94,-94 94,-152\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-126.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-111.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu1&#45;&gt;fc1 -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>relu1&#45;&gt;fc1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-83.7443C47,-75.2043 47,-66.2977 47,-58.2479\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-93.8971 42.5001,-83.897 47,-88.8971 47.0001,-83.8971 47.0001,-83.8971 47.0001,-83.8971 47,-88.8971 51.5001,-83.8971 47,-93.8971 47,-93.8971\"/>\n",
       "</g>\n",
       "<!-- fc2 -->\n",
       "<g id=\"node3\" class=\"node\"><title>fc2</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-246 -7.10543e-15,-246 -7.10543e-15,-188 94,-188 94,-246\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-220.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-205.8\" font-family=\"Times,serif\" font-size=\"14.00\">64</text>\n",
       "</g>\n",
       "<!-- fc2&#45;&gt;relu1 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>fc2&#45;&gt;relu1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-177.744C47,-169.204 47,-160.298 47,-152.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-187.897 42.5001,-177.897 47,-182.897 47.0001,-177.897 47.0001,-177.897 47.0001,-177.897 47,-182.897 51.5001,-177.897 47,-187.897 47,-187.897\"/>\n",
       "</g>\n",
       "<!-- relu2 -->\n",
       "<g id=\"node4\" class=\"node\"><title>relu2</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"94,-340 -7.10543e-15,-340 -7.10543e-15,-282 94,-282 94,-340\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-314.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-299.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu2&#45;&gt;fc2 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>relu2&#45;&gt;fc2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-271.744C47,-263.204 47,-254.298 47,-246.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-281.897 42.5001,-271.897 47,-276.897 47.0001,-271.897 47.0001,-271.897 47.0001,-271.897 47,-276.897 51.5001,-271.897 47,-281.897 47,-281.897\"/>\n",
       "</g>\n",
       "<!-- fc3 -->\n",
       "<g id=\"node5\" class=\"node\"><title>fc3</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-434 -7.10543e-15,-434 -7.10543e-15,-376 94,-376 94,-434\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-408.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-393.8\" font-family=\"Times,serif\" font-size=\"14.00\">10</text>\n",
       "</g>\n",
       "<!-- fc3&#45;&gt;relu2 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>fc3&#45;&gt;relu2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-365.744C47,-357.204 47,-348.298 47,-340.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-375.897 42.5001,-365.897 47,-370.897 47.0001,-365.897 47.0001,-365.897 47.0001,-365.897 47,-370.897 51.5001,-365.897 47,-375.897 47,-375.897\"/>\n",
       "</g>\n",
       "<!-- softmax -->\n",
       "<g id=\"node6\" class=\"node\"><title>softmax</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"94,-528 -7.10543e-15,-528 -7.10543e-15,-470 94,-470 94,-528\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-495.3\" font-family=\"Times,serif\" font-size=\"14.00\">SoftmaxOutput</text>\n",
       "</g>\n",
       "<!-- softmax&#45;&gt;fc3 -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>softmax&#45;&gt;fc3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-459.744C47,-451.204 47,-442.298 47,-434.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-469.897 42.5001,-459.897 47,-464.897 47.0001,-459.897 47.0001,-459.897 47.0001,-459.897 47,-464.897 51.5001,-459.897 47,-469.897 47,-469.897\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7ffb802d67d0>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mx.viz.plot_network(mlp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variable Naming\n",
    "\n",
    "MXNet requires variable names to follow certain conventions:\n",
    "- All input arrays have a name. This includes inputs (data & label) and model parameters (weight, bias, etc).\n",
    "- Arrays can be renamed by creating named variable. Otherwise, a default name is given as 'SymbolName_ArrayName'. For example, FullyConnected symbol fc1's weight array is named as 'fc1_weight'.\n",
    "- Although you can also rename weight arrays with variables, weight array's name should always end with '_weight' and bias array '_bias'. MXNet relies on the suffixes of array names to correctly initialize & update them.\n",
    "\n",
    "Call list_arguments method on a symbol to get the names of all its inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data',\n",
       " 'fc1_weight',\n",
       " 'fc1_bias',\n",
       " 'fc2_weight',\n",
       " 'fc2_bias',\n",
       " 'fc3_weight',\n",
       " 'fc3_bias',\n",
       " 'softmax_label']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlp.list_arguments()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loading\n",
    "\n",
    "We fetch and load the MNIST dataset and partition it into two sets: 60000 examples for training and 10000 examples for testing. We also visualize a few examples to get an idea of what the dataset looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAA9CAYAAABiDehcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXlUVGea/z+1UAtbsYPsO8gSRUCICioRJ1E0dIzSbkQT\nk06O3XHSPZ2eTPc4p3v6zOnTczozmUycxMzpxBijIWIgLnGJRCW4oOKCCIKA7PsmBVRRVN3fHw73\naNSYSCGhf/dzTp0DxaWeh8u93/d5n/d53isTBAEJCQkJicmFfKIdkJCQkJD44UjiLSEhITEJkcRb\nQkJCYhIiibeEhITEJEQSbwkJCYlJiCTeEhISEpMQSbwlJCQkJiGSeEtISEhMQiTxlpCQkJiESOIt\nISEhMQlRjtcHy2SyCem7FwRB9mP0Y6J8kfyQ/JD8+Nvw49tIkfckQalUkpyczJYtW9i8efNEuyMh\nITHBjFvkLWE9bGxsyMjI4Oc//zkGg4H8/PyJdmlCCQkJYdWqVcyfP5+GhgZ2795NQUEBAwMDE+2a\nhMQj40cv3lqtll//+tfExsayb98+9u3bR1dX10S79UhJTk5m3bp1JCQk8Omnn3Lq1KkJ8cPGxobI\nyEjmz5/P9OnTcXV1pauriyNHjvD5559jMBjG1b5cLicpKYn169fz1FNP4e3tjV6vJzAwEBcXF7Zt\n2zau9iUkfkz86MU7MTGRjIwM4uLi8PPzY2BggLy8PEZGRibatUeCj48PixcvJjk5mdOnT/Pxxx9z\n8+bNR+6HQqHgiSeeYP369cTFxeHt7Y2trS2Dg4NMnToVR0dH3n//fSwWy7jYt7W15Sc/+Qlr1qwh\nPj4eOzs7SkpKGBgYICgoiKioqHGx+2PE0dGRsLAw4uPjiYqKwsnJCYDBwUGqq6s5ceIEZ8+enWAv\nJx6ZTIadnR0ODg6MjIzQ398/7gEGwNKlS5k1axZlZWUUFBRgMpmYNm0aU6ZMwdHRkZCQEJydnTGZ\nTFy+fJm33377oexMqHi7uLjg6upKc3Pzfae8/v7+2Nvbo1Ao8PHxwd3dnf9f9iBXKBTMnj2b9PR0\n+vr6yM/P58yZMxPii5+fH4sXL2bx4sXY2tpiMBjo7OxELpcTHR3NU089xQcffMDw8LDVbWu1WjIz\nM9m4cSPx8fH09fWRm5vL7t27GRkZ4cknn0Qul6NSqcbF/g9BJpOhVCrRaDQ4ODgQFxeHj48Ptra2\n7Nixg46OjjF9flJSEk8++SRxcXGEh4fj5+eHnZ0dAEajkdbWVpKSkvj00085cOAAQ0ND1viz7olG\noyElJYWpU6eiUCgYHh7m2rVrnDt3jt7e3nGzez8cHR2ZPXu2OJArFArc3Nzw8PDAaDTS3t5Oe3s7\njY2N1NbWitfM6dOnrebDokWLePXVV5kxYwb19fWkp6djNpvF2aGtrS1eXl7Y29szMjJCWVkZlZWV\nHDp06AfbmhDxdnZ2JikpiSeeeAI7OzsKCgrIzc19oCgbDAaGhoYwm83j5ptMJkOn07FixQq8vLwY\nHh7mwIEDVFZWPpJR+3Y/Hn/8cX7605/i7e1NXl4eR44cwWg0PjIfbsdkMtHQ0MA333xDT08PtbW1\ntLW14efnx9KlS/H19SUqKoqLFy9a1a5WqyUjI4NXXnmFuLg46urq2Lt3Lzt37qSkpASFQkFLSwv+\n/v4olcoJEW8HBwe8vb0JCAggKCgIR0dH3N3dsbOzIyYmBi8vL7RaLTqdjt///vcPZUOr1TJr1izW\nrl1Leno6np6eyOV31huo1WoCAgLw8PDAx8cHlUrFzp07rfEn3oGHh4cY9aenpxMZGYlCocBkMlFV\nVcVXX33FoUOHuHz5stVt346zszMBAQEolUpqa2txd3dn2bJlPPXUU8CtNJudnZ0olAMDA/T29tLa\n2kp9fT1yuRyTyYSNjQ2FhYVj9mf69Ok8//zzzJ49G41Gg7OzM9OmTbvv8UqlEh8fH5KSkiaPePv6\n+pKVlcWaNWswGAx4eXlx/PjxMUclY0Wj0RAWFsbChQvZtGkTrq6umM1moqKi+OMf/8j169fHLS3w\nbUJDQ3nuuedITU2luLiYXbt2UVVV9Uhs34umpiby8vK4fPkyg4OD1NfXYzQaWb58OYIgIJfLcXFx\nsapNmUzGnDlzePnll0lISOD69et88skn5OTkUFNTgyAIWCwWrly5QlVV1bgId0pKCoGBgfT19d0V\nxSqVSlFAAgMDCQ0NJSwsDHt7ezGVASAIAoODg2RkZDyUeDs5OZGWlsbzzz9PcnIyTk5Odwh3T08P\nvb292Nra4uzsjFarJTExkaysLL788kurRcEKhYKIiAgWLVpERkYGU6dORS6Xi+fd3d0df39/pk6d\nik6no6Ojg5aWFqvY/jYuLi4888wzpKam0t3dzbZt2+jt7cViseDo6IharUYmk6HX62lra0On04lC\nHhAQQFJSEoIg0NvbS21trdXEOyoqCo1Gc9fPuru7GRgYwM/PT3xvZGSEpqYmiouLH8rehIi3TqfD\n1dUVhUKBnZ0dU6dOJSkpiX379t117M2bNx9JNKVWq4mOjuZnP/sZmZmZqNVqPvvsM9zc3FiyZAkf\nfvgh1dXV4+4HgJubG88++yxpaWk0Njby0UcfUVRU9Ehs345MJsPX1xcPDw9cXV3x9PREo9Hg6OjI\nzJkz0Wg0ZGRkMDw8TF5eHleuXLGq/YiICF544QUSEhK4ceMGH3zwAbt27aK5ufmO40wmEyaTyaq2\nR1mzZg2LFi2iq6uLmzdvIpPJxBmiSqXC29sbLy8v5HL5HYJqNpvp6emhoaGBtrY2ampqqK+v/8H2\nPT09WbRoERs2bCAuLu6ewtDR0cHZs2cZHh4mNDSUqKgonJ2dCQwMZPbs2ezfv//hT8BtTJkyhXXr\n1rFy5Uo8PDxoamrixIkT1NTUALdSnLNmzSIkJIQlS5ZQU1PDxx9/bPX7V61Wk5GRwaZNm5g6dSpH\njhxBrVZTV1dHfn4+BoMBFxcX5HI5TU1N6PV68dp1cnIiNDQUf39/4Na5s9Z93dvbS3d3N8PDw1gs\nFrq7u6msrKShoYGmpiZcXV158cUX7zj+yy+/5ODBgw9l70exYKnT6YiPj7+neOv1ekwmkxi9jEcO\nT6PREB0dzQsvvMDatWsZGBggPz+ff/iHf8Df359XXnmFrq6uR5Jr12g0LFiwgOXLl6PVatm6dStf\nffXVhCzQBgcHk5WVRUJCAuHh4fj7+6NSqZDL5QiCgCAIdHV18cEHH/Dv//7vVv3faLVaVq9eTWpq\nKgaDgZ07d5KTk3OXcI83JpMJOzs7fHx87nuMIAiYTCZ6enro6upCr9eLedUzZ85QUVFBeXn5Q6X7\n5s6dy2uvvUZkZCRK5b1vVzs7OzQaDbW1tZhMJjw8PHBwcCAwMJBly5Zx7Ngxq5RRLly4kGXLlqHV\naikqKuLgwYPs2bNHFD8PDw82btzIL37xC2JiYli4cCF79uyxunj7+/vz4osvEhkZSUtLC8XFxbS2\ntmKxWNi/fz/79+9Hp9OhVCrvqEyzsbHBx8eHOXPmMGfOHABu3LjBJ598YhW/ysrK+Pzzz2lsbMRs\nNlNVVcXBgwcpLS3Fz8+P7Oxs8dj+/n6OHTs2prTWhIi3wWBgcHBQ/H54eJj29vZ7Huvt7Y2dnR2C\nIFBRUSGO8tZCLpcTHBzMhg0beP755xkYGGD//v38/d//Pb29vXR2drJx40ZsbGwICgpCq9XS1tZG\nV1eX1VMoMpmMmJgYnnvuOUJCQsjLy+PgwYN0dnZa1c73ZcWKFWzcuJEpU6bc8b4gCKIQGQwG9Hr9\nXfnXsSCTyZgxYwbp6em4ubmxd+9e9u3bR2Njo9VsfF8KCgqYN2/eXWkQvV5Pb28ver2ewcFBuru7\nuXjxIufPn6eurs4qC8tyuZy5c+fi6+sr5pThVvri9vPt4+PDsmXLgFsRv8lkQiaTodVq8fX1xc3N\nzSri/fTTTyOXy8nLy+Ptt9+mtLT0jnugra2NCxcuUFdXx2OPPYZcLkcme2Cj4A8mJSUFb29vjEYj\nu3fvJjc39670TF9f312/ZzKZuHHjBjdu3ODjjz+2ul9VVVX85S9/Eb+3tbXF19eXxYsXs2TJEjEX\nbzabuXLlCv/2b/82pnWBCRHvxsZG6urqsFgsWCwWGhsbKSgo+M7fMRgMVFdXo9frcXR0tFq5nE6n\nY+HChaxcuRKTyURxcTG//OUv6enpAW7lNZ2cnIiJiWHDhg2EhYWxZcsWdu3aZfVZgKurK+vXrych\nIYGysjJ27NjBpUuX7nmsXC5Hq9WiVqu5efPmuETmo6VNo2I9Ouj29vbS19eH0WjEwcGB9PR0Tp06\nxfHjx61i18HBgZ/97GdERUXR3t7O4cOH7zu1lcvl2Nvbo9PpgFs3hl6vt8r1YWdnh1arvUMo9Xo9\nLS0tnD9/npMnT1JaWkpjYyPd3d3iNWMtdDod06ZNw97eHplMxtWrVzEYDDg7O6NWq4Fb16dWq0Wl\nUqFQKFCr1WJq5ebNm3R0dFhVQLdv387WrVvvO5BeuXKFs2fP8thjj1nN5rdZsGABnp6eNDc3U1ZW\nRnNzMxaLRRzU5HI5IyMj41rY8CAcHBxISkri1VdfZcmSJeL7w8PDNDY28vXXX3PhwoUx2ZgQ8Vap\nVKhUKuBW3qe8vPyBkZtKpWLdunXEx8fz17/+lU8//XTMfshkMlJSUli9ejW2trZcvXqVd999V5xq\nabVagoODefHFF8nMzMTb2xu4NT20ZqQ5SmZmJunp6QwPD7NlyxaOHTt21zFKpRJbW1tcXV2ZMWMG\nERER5OXlUVlZaXUBP3z4MMHBwcTGxtLZ2Ul5eTmnT5+msLBQHEhTUlL453/+Z55++mmriXd8fDxx\ncXHY2dnx/vvvc+jQoXtGUjY2NmJOeDTy7Onp4fDhw+Tk5KDX68fkR3p6Om+88Qbh4eEIgsDQ0BBf\nfvkl//Vf/0VxcfG45dlHiY6OxsvLC6VSidls5o033uDkyZP4+fnh4OAA3MqJx8fHExAQgKenJ1FR\nUXh4eKBSqXBwcCAjIwOz2cwrr7wy5uh769atlJSU3BXlKpVKUTjd3d1xdXUV3x8dZKzJaOlwaGgo\n//qv/4pSqeTcuXMMDAxgb2+Pvb099fX1NDc3YzKZxCBxvLGxsUGtVqPValmyZAmvvvqqWG1isVgw\nGo1cvXqVt99+2zoNZaO5S2u/AOF+r5kzZwqfffaZYDabheHhYaGiokLYtGnTPY9dt26dcPXqVcFs\nNgs3b94UTpw4IWRlZd33s3+IH/b29sJvf/tbwWQyCXV1dcLvfvc7QSaTCXK5XNBoNEJ2drZQVlYm\n3Lx5UzCZTILZbBauXbsmLF++XPi/zWq+tx8P8sXGxkbYtm2b0N/fL7z33ntCeHj4XcfI5XJh2rRp\nwpYtW4TW1lZhZGREMBqNQlFRkRAWFmYVP+5lU6lUCnK5/J4/T0hIEHbt2iX8x3/8h9XOh7e3t3Dy\n5EnBbDYLO3bsEGJiYu7ySa1WC8nJycLOnTuFoaEhwWw2C2azWRgZGRGampqEzZs3j9mPffv2CQaD\nQbBYLMLQ0JCwZcsWITIyUlAqleLrfuflQa/v48evfvUrobW1VbBYLEJdXZ2Qmpr6nZ/p7u4uZGdn\nC4WFhcLw8LBgsVgEi8UiNDc3C//4j/94z2v2Ya8PmUwmKBQKQaVSCdOnTxcyMzOFNWvWCJ9++qlg\nNBoFi8UiXLp0SVi5cqWgUCiscj5GX++9957Q09Mj/n29vb1CX1+f0NraKvT09AhGo1H4+uuvhRdf\nfFGYMWOG4OHh8cD71Rr3y8yZM4XNmzcLV65cEW7HbDYLbW1twkcffSQkJiY+tB/ffk1Y5D1ayqNU\nKunt7b1vJD069ZHJZLS3t/PRRx+Rm5trFT9kMpk4zWpra+PUqVPodDqxXnPZsmW0trbym9/8hlde\neYWIiAhOnz5NVVWV1RcvFyxYwLRp02hqaiI/P5/r16/fdUxkZCSbNm1izZo1NDc3k5ubi6enJy4u\nLtjb21vVn1G+T9Si0+nQ6XRoNBqr1ML39vaKUW1DQ8NdUXdaWhovv/wyjz/+OO7u7shkMvr6+sRF\n5aGhIXx9fcfsh4uLizhDFAQBOzs7Fi1adPtNTXl5OadOnRqXrteQkBAxcm1qarpjnehedHZ28s03\n32A0GikpKeGll15CrVbj7u7OihUr+Pzzz7l27ZpVfNPpdDz99NNkZWURFxeHk5MTMpkMhUKBQqEA\nblUL/eY3v0Gn0/Huu+9axS7Ae++9R2JiItOnTwcQZyGj98Boj8SMGTMwm80cOHCAN954g4aGBqv5\ncC8iIyN59tlnmTp16h3vHz58mN///vdcuHDBqrO1CRHv0ZykIAjIZDJcXV1JS0u7a9VXp9OxYMEC\nccFMEARGRkbGJb8bGhrKn/70J+RyOT4+PphMJrZt28aHH36In58fSqWS6upq9uzZQ1lZmdXtx8bG\n4urqSkNDA/39/XcJppubG+np6cyePZtdu3bx+uuv4+bmxksvvcT8+fOt7s8PQaFQ4O3tTVpaGgcO\nHBjz542ub0RHR3P+/HmxwkQul5Odnc1rr71GWFgYKpUKQRAoKiriT3/6ExcuXCAsLIx/+qd/Ij4+\nnpiYmDGVL7a0tGAwGNBoNKjVapYtW8bTTz99xzEmk4nOzk6++OIL3n33XWpra8f0t9/O7Qt+tw8Y\n90MQBOrq6mhpaeHkyZN0dHTwhz/8AYVCgb+/Py+88AKvv/76mP36xS9+QVZWFmFhYTg6OmJjY3NH\nGnG0nFKlUhEVFcXmzZtZuHAhu3bt4tixY/ctTvi+lJaWkp2dzeLFi0VtaG9vp7m5GXt7e7HOOzk5\nmVmzZuHl5YWfnx+NjY3jWjGWmpoq1r7Drdrud955h7feeoubN29aPc02IeJdUVHBmTNnWLRoEU5O\nTvj4+PDGG2+Qnp6OWq3m66+/JiIiQsxJj46sbW1t49bI4+DgQGhoKFVVVezYsYPc3FxqampQqVRs\n3rwZPz8/sSxqPHKdoyWRAQEBxMTEUFpaKjZXhIaG8vLLL7N8+XJOnTrFn//8Z1QqFcuXLyczM5OK\niooxldCNRkxms/kHXdxyuRxnZ2ecnZ0pKSmxWi26xWIRz8fw8LA4+1q0aBHr168nMjISuLU4tnPn\nTvLz86mrq2NoaIigoCAiIyPp6+sbc6S1ZcsWtFotSUlJaLValEolSqXyjoFVrVbj6OhIdnY2FouF\nP/zhD1ZbyI6NjUWr1f6g3zGbzQwNDYmLeXBL1LVaLREREWPyZ/ReyMrKwtfXF5VKRXV1NXV1dbi7\nuxMcHIy9vT1Go5EbN24wODhIbGwsHh4eLFy4kOnTp1NSUsKWLVseWKDwXZhMJvGat7GxEd8zmUx3\nLFp+/PHHrFq1ipdffplf//rX/PGPf+T8+fNjOgffhVqtFss5L126xJYtW9izZ8+4baQ3YaWC+/fv\nJzg4mE2bNqFWqwkPD8fX1xe5XE5ycjIODg5YLBaxm0wQBMrLy63aKDM4OMi2bdvEKK+2tpazZ8/S\n1dVFW1sbJpOJtLQ0IiMjkcvlXL58ecxRw/04evQoS5cuJSUlhddee43Q0FCOHj2KXq9n6dKlrFy5\nkuHhYWpqaoiPjyczM5OZM2fS1tbGn//85zFdIJmZmaSmpvLee+9RWVn5vRd3pk2bxoYNG/D29qao\nqOiei4oPS01NDUNDQyxevJiKigquXbvGjBkziI2Npb+/n71797J9+3auXLlCZ2cnFouFwMBAfvrT\nn+Ls7MyFCxfGvGB5+vRpfvnLX4rXoLOzM8PDw3cs/Dk4OPDMM8+wdu1asrKy6O7u5s0337RKpYOT\nk9N9a7sfhNlsFuurZTIZarX6ju6+h0Gj0TBz5kwCAwNRKBTU1dXx1ltv0dDQwKpVqwgICACgurqa\nrVu3cvz4ccLCwsjMzGTFihUEBATg5uZGQEAAOTk5/M///M9D/49GRkbo7u7+zmO6u7upqKjAZDIR\nHR1NfHz8uIr37QwODtLY2DiuZb4T1qTT2trK9evXGRkZQalUYmNjI5Z7jUbat9PW1ibW0FoLs9lM\nfX09X3zxBYWFhej1+rvKvebOnYtOp+PQoUPivh7jQW1tLX/5y18YHBwkNTWVVatWkZaWhslkwsfH\nBy8vLwYHB3nmmWcwm814eHhQVlbGW2+9xcmTJ8eUSsrKyiI1NZXBwUHeeeed7xXFR0ZGsn79eubO\nncu1a9es1ugwyrlz5+jo6ODv/u7vqKmpYfv27eTn5+Pm5oazszM5OTmcPHlSzLEHBQWxYcMGMjMz\nGRkZscpWBgMDA1RUVIjfq9VqzGbzHefazs6O6OhoTCYTvr6+JCQkoNVqxzxwANTX1xMcHCzmre/V\nXfl9GG0iGmur+mh5qkKhYHBwkM8++4zCwkLS0tJ47LHHsLW1pbq6mu3bt5OTk0NrayuVlZVUVFRQ\nXFzM8uXLxQHY2dkZLy8vfvWrX43JJ5lMhkqlEve1uX1WbLFYqKmpoaSkhOTk5DEPXg/i8uXLNDQ0\n4OfnR2RkJGFhYeNqb8LE22g0UlRUxH//93+TkJCAxWIhISEBW1tb4NailUqlQqPRIJfLuXTpEmVl\nZePSYanX6+95s9na2vL4449ja2tLfn4+5eXl41YeNjw8LA4OBw8eJCIigpSUFGJiYtBoNHR0dNDQ\n0EBvby8NDQ2UlZVRXFxMSUnJAxeyHoS3tzeurq5kZGSQm5v7neItk8kIDQ1l7dq1ZGZmolAoOH/+\nvNU3pCotLeXq1auEh4fz3HPPERUVxdWrV6mqqqKtrY1r164hl8txc3MjKiqK1atXs3DhQpycnDh2\n7Bi7du2yen7z9k3B7O3tSUpKYv78+aSmpopTZhcXF7y9vamsrByzvaqqKpKTk1Gr1Xh7e+Pu7o6N\njc33ugbd3d2JjY0Fbv3PhoeHxyzeFotF7HaGW2md119/nenTpxMUFMTQ0BDHjx/niy++EG0NDg6K\ntdgXL14kPDyclJQUkpKSrLKAOHv2bBYuXMiUKVPIzc3l6NGj4vmRy+VoNBpxwHnYwe/7cvHiRVG8\nnZ2dmTt3LoWFhZSUlIyLvQkTb7PZTHl5Oe+88w4hISEIgkBsbKy4vWV3dzfZ2dnExsaKexKM/uxR\noFAoWLFiBVFRUXc0CI0nQ0NDnDt3joqKCtzd3amqquKll14iKiqKo0ePsnPnTvR6PZ2dnTQ1NdHT\n02MVgSopKSE2NpaQkBCWLl3K4OAgNTU1d7Q1j3aYJiYmkpqaSmpqKgMDA+zevZvdu3dbvQW6r6+P\n3bt3o1QqWbBgAc8++ywdHR3cuHGDlpYW5syZg8FgQKfTERQUJO7xffbsWd555537NjeNleDgYBIS\nEoiLi2PmzJlER0fj4uKCUqlkaGiI9vZ2q10nubm5zJs3D3t7e2xtbVm9ejVNTU1cvHjxOwXc1taW\nWbNmsXz5cvE9o9E4ZrE0GAzs27cPf39//Pz8mDNnDjY2NhiNRk6fPk1BQQFHjhy5qwvaYrHQ1dXF\n8ePHOXPmDGfPniU8PNwqayTz588nOzsbT09PfHx8SExMpKqqiqamJhQKBSkpKeIaiTV54YUX8PT0\npLGxkcOHD9Pa2srly5c5duwYQUFBTJkyhVmzZpGSkvK3J95w64Kqrq6muroamUxGYWGhuAChVCr5\nyU9+Iq62h4SE4O3tLS6sjTcKhYL58+fj6OjI4cOHaW9vf2T7iI/OBAoLC3niiSeIiIjg+vXrHDx4\ncFwi/08++YSYmBgef/xxVq9eTWBgIOfOnaO8vJz+/n5cXV0JDg5m+vTpxMfH4+HhQWlpKTk5ORQV\nFVl9y4JRTpw4QWdnJ/X19SxZsoSQkBB8fHyQy+UYjUZGRkbEaKq9vZ2vvvqKnJwcjh8/btWmDIVC\ngZ+fH4mJicyfP5/ExESx0gJuBSKtra0UFhayc+dO2trarGL35MmTfPnll0yZMgVXV1eefPJJDAYD\ne/bs4cKFC7S0tNwxG1AoFHh6ejJv3jyys7PFBpGRkRGam5s5ceLEmPwZHh7ms88+QxAEMjIycHZ2\npqGhgdLSUoqKisRU13dhMBi4ePGi1WZqo7v1KRQK5s2bR2JiIvX19bS0tIiVYy4uLly8eNFqTWRz\n585l48aN+Pn50draysyZM6murqapqQm1Wi0GMp6ennh5eVnF5r34UWxMBbfycgaDQcxhjtYvjy7Y\naLVacTPz/v7+cfdHJpMRFBREf38/O3bsoLm5+ZE/BKK1tZUvvviCurq6cX0yytmzZ3n33XdRKpXM\nmDGDNWvWkJaWxrVr1+jr68PDw4OgoCBcXFwQBIFvvvmGN998k4KCgnEdSPv7+ykuLqa5uZna2lqm\nT59OeHg4AQEBODo64uDggF6vp6qqiqNHj7Jnzx7OnTtn1f/TaPS2aNEi0tPTiYiIEAcMQRAYGBig\nqqqKI0eOkJ+fz7lz56x2TkwmE5988gmRkZGkpaVhZ2fHM888Q3BwMMXFxdTU1KDX6+nu7kYQBDw9\nPQkMDCQ9PZ2EhATgVtRbX1/PRx999NBbj95OQ0MDH374IeXl5Tg7O3Pjxg0qKiro7e2dkIekFBQU\nEBkZibOzs7hzoKurK3FxcXf4vH//fo4cOWIVm/PmzcPf3x9XV1fc3NyIiYmhr6+Puro6tFotHh4e\nVrHzIH404v1tjEYjxcXF4gbzbW1t9Pf3j8tGN9/GxsaGmJgYpk2bRmFhIWfPnh33lMm96OrqYvv2\n7Wzfvn1c7QiCwJ49exAEgWeffVZsTY+JiUGn06FSqWhraxOj7EOHDvHVV189kptVEAQaGhrYsWMH\neXl5xMTEEB0djZubG66urhiNRs6fP09hYSFNTU1Wt69UKnnyySdZuXKl2Pbd19cnPpCioqKC06dP\nc+zYsXFpArl06RL/+7//y40bN0hLSyM4OJjk5GQSExMZGhrCZDKJe3v4+fmJOd5RPy9fvsyRI0fY\nunWr1R7QPLou82OgqqqKnJwc4NYjE6dMmSKum8lkMkZGRjhz5gyHDx+22oNMOjo67poB63S6e+7n\nEhYWNuaNJdlkAAACbUlEQVR+g/vxoxXv3t5e3nzzTc6fP8/s2bMpLy+noKDgkTy/UaPRsHTpUhQK\nBXl5eRPyzMhHjcViITc3l8rKSubOnYurq6tYXmZvb09paSn79+/n8uXL4/porfsxNDTE0NAQx44d\nu+eeL+OFxWLh4sWLODg44OjoKHZz1tbWUlRUxKVLl8a8YPwg9u7dy969e1m3bh0ZGRnExMTg4eEh\nPp9Rp9NhMBgwm80YjUba2tro6enh+vXrvP/++5w5c2ZCgo9HwcjICIWFheLM7LHHHsPd3V38ucFg\n4MyZM1YVz1OnTnHgwAGxmAFuLRCPfn07ycnJLF269J6dwmNFNl7R0//tJfDIEQThjtD8YfzQ6XT8\ny7/8C/Pnz2ft2rVUVFT84FK8b/vxsL6MFcmPvy0/pkyZQkZGBjNnzsTX1xd7e3vxeaIGg0F8LuJo\n/fuDcsuT/XxMpB+rVq0SOzyTkpLEjetup7Kykq+//pozZ878oAqke/lxl1+SeN+Ni4sL77//Phcu\nXOA///M/HypqmYwXo+SH5Ifkx4/Xj29j/X1N/wYY3TDrQSVZEhISEhOFFHk/Ij8myhfJD8kPyY+/\nDT++zbiJt4SEhITE+CGlTSQkJCQmIZJ4S0hISExCJPGWkJCQmIRI4i0hISExCZHEW0JCQmISIom3\nhISExCREEm8JCQmJSYgk3hISEhKTEEm8JSQkJCYhknhLSEhITEIk8ZaQkJCYhEjiLSEhITEJkcRb\nQkJCYhIiibeEhITEJEQSbwkJCYlJiCTeEhISEpMQSbwlJCQkJiGSeEtISEhMQiTxlpCQkJiESOIt\nISEhMQn5f4+DPzsVZsBbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ffb549feed0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_mldata\n",
    "mnist = fetch_mldata('MNIST original')\n",
    "np.random.seed(1234) # set seed for deterministic ordering\n",
    "p = np.random.permutation(mnist.data.shape[0])\n",
    "X = mnist.data[p]\n",
    "Y = mnist.target[p]\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.imshow(X[i].reshape((28,28)), cmap='Greys_r')\n",
    "    plt.axis('off')\n",
    "plt.show()\n",
    "\n",
    "X = X.astype(np.float32)/255\n",
    "X_train = X[:60000]\n",
    "X_test = X[60000:]\n",
    "Y_train = Y[:60000]\n",
    "Y_test = Y[60000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create data iterators from our MNIST data. A data iterator returns a batch of data examples each time for the network to process. MXNet provide a suite of basic DataIters for parsing different data format. Here we use NDArrayIter, which wraps around a numpy array and each time slice a chunk from it along the first dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "batch_size = 100\n",
    "train_iter = mx.io.NDArrayIter(X_train, Y_train, batch_size=batch_size)\n",
    "test_iter = mx.io.NDArrayIter(X_test, Y_test, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "With the network and data source defined, we can finally start to train our model. We do this with MXNet's convenience wrapper for feed forward neural networks (it can also be made to handle RNNs with explicit unrolling). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Start training with [gpu(0)]\n",
      "INFO:root:Epoch[0] Batch [200]\tSpeed: 54861.39 samples/sec\tTrain-accuracy=0.420200\n",
      "INFO:root:Epoch[0] Batch [400]\tSpeed: 55642.10 samples/sec\tTrain-accuracy=0.667000\n",
      "INFO:root:Epoch[0] Batch [600]\tSpeed: 55592.13 samples/sec\tTrain-accuracy=0.759517\n",
      "INFO:root:Epoch[0] Resetting Data Iterator\n",
      "INFO:root:Epoch[0] Train-accuracy=0.759517\n",
      "INFO:root:Epoch[0] Time cost=1.090\n",
      "INFO:root:Epoch[0] Validation-accuracy=0.944800\n",
      "INFO:root:Epoch[1] Batch [200]\tSpeed: 55644.46 samples/sec\tTrain-accuracy=0.949450\n",
      "INFO:root:Epoch[1] Batch [400]\tSpeed: 56210.39 samples/sec\tTrain-accuracy=0.955975\n",
      "INFO:root:Epoch[1] Batch [600]\tSpeed: 56176.92 samples/sec\tTrain-accuracy=0.959400\n",
      "INFO:root:Epoch[1] Resetting Data Iterator\n",
      "INFO:root:Epoch[1] Train-accuracy=0.959400\n",
      "INFO:root:Epoch[1] Time cost=1.076\n",
      "INFO:root:Epoch[1] Validation-accuracy=0.963000\n",
      "INFO:root:Epoch[2] Batch [200]\tSpeed: 57344.71 samples/sec\tTrain-accuracy=0.966100\n",
      "INFO:root:Epoch[2] Batch [400]\tSpeed: 56675.23 samples/sec\tTrain-accuracy=0.969875\n",
      "INFO:root:Epoch[2] Batch [600]\tSpeed: 57061.83 samples/sec\tTrain-accuracy=0.971800\n",
      "INFO:root:Epoch[2] Resetting Data Iterator\n",
      "INFO:root:Epoch[2] Train-accuracy=0.971800\n",
      "INFO:root:Epoch[2] Time cost=1.057\n",
      "INFO:root:Epoch[2] Validation-accuracy=0.965200\n",
      "INFO:root:Epoch[3] Batch [200]\tSpeed: 55927.63 samples/sec\tTrain-accuracy=0.975550\n",
      "INFO:root:Epoch[3] Batch [400]\tSpeed: 55997.93 samples/sec\tTrain-accuracy=0.977175\n",
      "INFO:root:Epoch[3] Batch [600]\tSpeed: 55633.90 samples/sec\tTrain-accuracy=0.977933\n",
      "INFO:root:Epoch[3] Resetting Data Iterator\n",
      "INFO:root:Epoch[3] Train-accuracy=0.977933\n",
      "INFO:root:Epoch[3] Time cost=1.079\n",
      "INFO:root:Epoch[3] Validation-accuracy=0.968900\n",
      "INFO:root:Epoch[4] Batch [200]\tSpeed: 55834.72 samples/sec\tTrain-accuracy=0.981150\n",
      "INFO:root:Epoch[4] Batch [400]\tSpeed: 56353.62 samples/sec\tTrain-accuracy=0.980750\n",
      "INFO:root:Epoch[4] Batch [600]\tSpeed: 56360.24 samples/sec\tTrain-accuracy=0.981383\n",
      "INFO:root:Epoch[4] Resetting Data Iterator\n",
      "INFO:root:Epoch[4] Train-accuracy=0.981383\n",
      "INFO:root:Epoch[4] Time cost=1.073\n",
      "INFO:root:Epoch[4] Validation-accuracy=0.973200\n",
      "INFO:root:Epoch[5] Batch [200]\tSpeed: 41500.31 samples/sec\tTrain-accuracy=0.982700\n",
      "INFO:root:Epoch[5] Batch [400]\tSpeed: 39625.54 samples/sec\tTrain-accuracy=0.983825\n",
      "INFO:root:Epoch[5] Batch [600]\tSpeed: 39155.04 samples/sec\tTrain-accuracy=0.984033\n",
      "INFO:root:Epoch[5] Resetting Data Iterator\n",
      "INFO:root:Epoch[5] Train-accuracy=0.984033\n",
      "INFO:root:Epoch[5] Time cost=1.502\n",
      "INFO:root:Epoch[5] Validation-accuracy=0.970900\n",
      "INFO:root:Epoch[6] Batch [200]\tSpeed: 55340.21 samples/sec\tTrain-accuracy=0.986300\n",
      "INFO:root:Epoch[6] Batch [400]\tSpeed: 37988.67 samples/sec\tTrain-accuracy=0.986275\n",
      "INFO:root:Epoch[6] Batch [600]\tSpeed: 35698.22 samples/sec\tTrain-accuracy=0.986267\n",
      "INFO:root:Epoch[6] Resetting Data Iterator\n",
      "INFO:root:Epoch[6] Train-accuracy=0.986267\n",
      "INFO:root:Epoch[6] Time cost=1.453\n",
      "INFO:root:Epoch[6] Validation-accuracy=0.973400\n",
      "INFO:root:Epoch[7] Batch [200]\tSpeed: 55618.74 samples/sec\tTrain-accuracy=0.987200\n",
      "INFO:root:Epoch[7] Batch [400]\tSpeed: 53917.95 samples/sec\tTrain-accuracy=0.987275\n",
      "INFO:root:Epoch[7] Batch [600]\tSpeed: 53863.94 samples/sec\tTrain-accuracy=0.987850\n",
      "INFO:root:Epoch[7] Resetting Data Iterator\n",
      "INFO:root:Epoch[7] Train-accuracy=0.987850\n",
      "INFO:root:Epoch[7] Time cost=1.106\n",
      "INFO:root:Epoch[7] Validation-accuracy=0.973300\n",
      "INFO:root:Epoch[8] Batch [200]\tSpeed: 53650.93 samples/sec\tTrain-accuracy=0.987600\n",
      "INFO:root:Epoch[8] Batch [400]\tSpeed: 53295.25 samples/sec\tTrain-accuracy=0.989300\n",
      "INFO:root:Epoch[8] Batch [600]\tSpeed: 53178.48 samples/sec\tTrain-accuracy=0.989000\n",
      "INFO:root:Epoch[8] Resetting Data Iterator\n",
      "INFO:root:Epoch[8] Train-accuracy=0.989000\n",
      "INFO:root:Epoch[8] Time cost=1.129\n",
      "INFO:root:Epoch[8] Validation-accuracy=0.972900\n",
      "INFO:root:Epoch[9] Batch [200]\tSpeed: 54587.53 samples/sec\tTrain-accuracy=0.989150\n",
      "INFO:root:Epoch[9] Batch [400]\tSpeed: 54106.54 samples/sec\tTrain-accuracy=0.989500\n",
      "INFO:root:Epoch[9] Batch [600]\tSpeed: 55994.46 samples/sec\tTrain-accuracy=0.990117\n",
      "INFO:root:Epoch[9] Resetting Data Iterator\n",
      "INFO:root:Epoch[9] Train-accuracy=0.990117\n",
      "INFO:root:Epoch[9] Time cost=1.098\n",
      "INFO:root:Epoch[9] Validation-accuracy=0.975000\n"
     ]
    }
   ],
   "source": [
    "model = mx.model.FeedForward(\n",
    "    ctx = mx.gpu(0),      # Run on GPU 0\n",
    "    symbol = mlp,         # Use the network we just defined\n",
    "    num_epoch = 10,       # Train for 10 epochs\n",
    "    learning_rate = 0.1,  # Learning rate\n",
    "    momentum = 0.9,       # Momentum for SGD with momentum\n",
    "    wd = 0.00001)         # Weight decay for regularization\n",
    "model.fit(\n",
    "    X=train_iter,  # Training data set\n",
    "    eval_data=test_iter,  # Testing data set. MXNet computes scores on test set every epoch\n",
    "    batch_end_callback = mx.callback.Speedometer(batch_size, 200))  # Logging module to print out progress"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "After the model is trained, we can evaluate it on a held out test set.\n",
    "First, lets classity a sample image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPwAAAD8CAYAAABTq8lnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnU9sa3l1x78njh3HcRznz2NAaNph0UpVhQSqhCrRChYI\ngSpB2VCNhDSqKsSiBcQK2gV01xYJhNQFmw4V0IoWFTGdLtoyVFSFRVuo+DP8a0FiJEAz7828PDu2\nYztx8uvi+VzOPT6/ayexnWvf85Gu7rXzYt/k5etzfud3/lAIAY7jFION274Bx3GWhwvecQqEC95x\nCoQL3nEKhAvecQqEC95xCsS1BU9EbyGiHxLRj4jog/O8KcdxFgNdZx+eiEoA/hfAmwD8HMDXATwe\nQviB+De+we84t0gIgfRz17XwrwPw4xDCcyGEcwB/B+DtN7k5x3EWz3UF/0oAPxWPfzZ+znGcHHNd\nwbu77jgryHUF/3MAj4rHj+KhlXccJ8dcV/DfAPArRPQYEVUA/B6Ap+d3W47jLILN63xTCGFERH8E\n4F8BlAA8KSP0juPkk2tty830wr4t5zi3yjy35RzHWUFc8I5TIFzwjlMgXPCOUyBc8I5TIFzwjlMg\nXPCOUyBc8I5TIFzwjlMgXPCOUyBc8I5TIFzwjlMgXPCOUyBc8I5TIFzwjlMgXPCOUyBc8I5TIFzw\njlMgXPCOUyBc8I5TIFzwjlMgXPCOUyBc8I5TIFzwjlMgXPCOUyBc8I5TIFzwjlMgXPCOUyBc8I5T\nIFzwjlMgXPCOUyA2b/LNRPQcgBMAFwDOQwivm8dNOY6zGG4keAABwBtDCMfzuBnHcRbLPFx6msNr\nOI6zBG4q+ADgy0T0DSJ69zxuyHGcxXFTl/71IYTniegOgGeI6IchhK/O48Ycx5k/N7LwIYTnx+cX\nAXwRgAftHCfHXFvwRFQjot3x9Q6ANwN4dl435jjO/LmJS/8IgC8SEb/O34YQvjSXu3IcZyFQCGEx\nL0y0mBd2HGcmQggTO2ieaec4BcIF7zgFwgXvOAXipvvwzi0zDpom1xsbGyCiiWs+5Pfos8SK7Vxe\nXiKEkDrLa/0e+n1jr6+v+bG+dm6OC37F0AKX15ubm8lRLpdTjzc3N6MfCHzOEiEAnJ+fp46zs7PU\n41KphI2NDWxsbCTX8swC1oLmgz9A9CE/ZJyb4YJfIbIsNBGhXC5ja2vLPCqViilIeW1ZVCnIwWCA\nfr+PwWCQXPf7fRARLi8vUSqVoh82m5ubKWFb54uLC1xcXGA0Gk1c88/por8ZLvgVQ4ucrzc2NlAu\nl1GtVrG9vY1arZY6tre3USqVElHytXysLa4+ut0uut0uer0eut0uSqUSAODi4gJnZ2fY3NxEpVJJ\njnK5nHocs+B8jEajxFvgawDJh4H8+V3418MFv4JY62MW/NbWFnZ2drC7u4t6vZ6c6/V61PKWy+Wp\ngr+8vES73Uar1UK73cbm5sM/HRZ7v99PXou9imq1imq1mlxfXFzg8vIysd76+uzsLPE22Jqz2LV1\nd2t/PVzwK4gWPFv4zc1NVKtV1Go17O7uotlsYm9vLznK5bJ5VCqVTJebr+/fv4+tra0kHiDFzu9f\nqVSwtbWF7e3txNPgaxa35bpfXFwknoa04peXl9jYsDeTXPRXxwW/QuiItxY8u/Rs4ZvNJg4ODnBw\ncID9/f2Ue721tZWIna+nReGl2C8vLzEcDnF6eopyuZwIni389vY2dnZ2UgcLfDQaTRwXFxeJ2KVl\nH41Gyc/o4r45LvgVxLLu2qVvNBpoNps4PDzEnTt3cHR0FA3ocVDPiozzIQXJlv309BSdTicJCJZK\npZSFr9VqqWXFxcVFan0uzyxsfl8W+/n5eWLhLdH7B8HVcMGvGCxufbDYOWi3s7OTiG1vbw/NZjNT\n8FtbW1ODaqenp6mDg3gcyONYwe7ubnLIxyxsKXIpfF5iyPgC/3zsVVhbdfycMx0X/IogLbkVaa9U\nKonYOVgmg2bSkrOoZIDMej99yCVDo9FAv9/H+fl5ss6u1WrJB4111u68ftzv99Hr9bCzszPx4XJ6\nejoRwZfXckvRieOCXyFkcE7vd7MbHRN9tVpNWVC5D68FL91kKXh+n1qthkajgbOzsySCXi6Xze1A\nefDWWyxw1+/3E7H3+/1E6Hw9GAwwHA6TPAC+DiFgNBrdxn/JyuGCXyGIKLHoHHBjEbPQY2LngJv8\noOA1uWXh+f1CCKaFPz8/T4ldvpe8D77mKH3Wthwn8rDIreter4der5cEClnsvpafDRf8CsEWmUXL\n6/ZKpZISVsydl8sAaeUBpCw6i5wf85m3/XZ2dpLceRko1DEB+f7VajUzPnBxcZHK3pMHP9/pdJId\nhY2NjcRjGA6Ht/MfsoK44FcItvBsqTkiLve9s1x6K612Fpeez+zSs9ili9/v9ycy6/SRlVZ7eXmZ\nStm1ztVqNQnksdh5tyDmpThpXPArBK/hpUsvI/PWWVpZq2Am5tJbHwLsurOXwZZ9OBwmqbW8XJBn\nvo4VzfAxHA5Tufry4A8U6cZzHgBn/TnT8d/UiiDFzkLSgs+y8Cx4fi3rPO39OemGxa6DbnLXwNpJ\nkFjVeDIgpwN0g8Eg+dBgsfd6PWxtbaWy85xsXPA5xLKu2rJzCi1nscl9b05nrVarqW24674/w8Iq\nlUqmSx6rwuPraej8fpn6W6lUMBgM0O12sb29PVEB6MyGCz5HxCrhWPBS6PV6HY1GA41GA7u7u2g0\nGtjf38fe3l4iehnguso96CIVCa+f2bUGkIrkx5YJMhA4688v4wzsJcy6JHFsXPA5w/pjZqvKgudk\nFk6f5Uw6LpKp1+uJFeRKuOvcB2ALlcUuhR9CmCrGrA8S+XyW2GWg0cV+dVzwOULnx8szR+VlNRwL\nnYtjZGprrVZLRbVnfX8W+CwWWYpdCn4WMcZe36oRmCZ6Z3Zc8DlDW3W592659Pv7+zg4OMDh4WGy\ndufMtuu49Nb9aHHKijaZ0qqXIhJL4Nbrag9Bi1269VcJOjoPccHnhJhlk5F5S/BcEaer4Tgyf13B\nx1x6vUev1/Hye2Mu/SxrefmzX1xcmPkDsToAJ44LPmdo0cstLi34vb29xMIfHR1NRLdlN5urvD8w\nXZz631kWPPa98t/GrLz1O4it4V30s+OCzxGxP3QteN6GYwvPgrc8A76+zr1M+1rWOl/ntc8apc/y\ncmJRemd2XPC3jHR/ZVKNPqrVamqvXR87OzvJ62jLdxVRWEKV11mtpq2fSz8X276z4gKxltW6/ZYz\nOy74JRNze2VPOt38kd34O3fu4ODgILX1xskn/Fra6l1V7DExy040suBFnmOCjkXd5TULXna74Vx5\nPjiFl+vg+X1d9LPjgl8ilhCl4NmScwadPNfrdRwdHaWSa1jwnPJ6XaEzWtz62upFxw0ouFTW6sbD\nz3OqrTzLARlWeyseeMGptsPhcELw3u1mdlzwt4Dlbss2VbzPzut0mUm3v7+PRqMxYeGt17yKSy+t\neqyElcXHVlZf6+0zfa0LavjeZIxBC54tOx9u4W/GVMET0acA/A6AeyGEV4+fOwDw9wB+GcBzAN4Z\nQmgt8D7XhqyItKw3566zzWYzseqNRiM5W4KXr39dC69r1OVZCo8PaXktCy4tOX+gyR0A/kCQ3oUl\neH4fF/zNmMXC/zWAvwTwGfHchwA8E0L4KBF9cPz4Qwu4v7Uktr7VFp732A8PD5NMOtkjjtNn2S2W\nr38V9JpddqKRB7vWulEFP9YddfSZxc4Re/6Qs7yLqwreRT8bUwUfQvgqET2mnn4bgDeMrz8N4N/h\ngp+ZLMHrQRKHh4d42ctehsPDQ7MMVlt4/T7TsKLw0sLLElgWHzeb5JZT3HvOGnAhr3mtLS37xcXF\nhOD5feWywRI835eLfXauu4Z/JIRwd3x9F8Ajc7qftcYKrFkWXrr03Ff+zp070blt160Ht4SiRS+D\ncyw8FrpuUy3victX+eDOslLs5XI5sdL83jpKf35+7mv4OXLjoF0IIRCR/8anIIVt1YzLfvKypzwH\n7Pb29sy5cLMWx8S6zMhDj4PWhx4mqR9Xq1WMRqNU1JzddnkPWfcYC9r5Gn4+XFfwd4no5SGEF4jo\nFQDuzfOm1hU9Tlmet7e30Ww2k8SanZ2dZC+et950imkstTRmua11uTxYUNbeN/eO091k5XOcPMSC\nlyW9XL2nm3LoRpqW4GWkXvaj9225q3NdwT8N4AkAfzE+PzW3O1pDWJTsxsqhEHyu1WqpCDxXvlmC\nz+opH8uUY8FnWXDZVirrHDs4/ZeTcOTPy9V7sqiHvROZqhtz6QeDQWob0C389ZhlW+5zeBigOyKi\nnwL4MIA/B/B5IvoDjLflFnmTq4per0sB6ODbzs5OSvAchecBEnp4xLQmEFYwjgUf216LRd/50FNf\n9PSXSqWC0WiUrNX552ULLz/gdG98ea8s+qvsw7voZ2OWKP3jkS+9ac73slZYKbRa8HKyar1eT7n0\nloWPZbEx0prLM19Lweve79ZoJ31Ye/NykIRew+sKPxm1t8ZdZbn0voafD55ptwBiW2SW4HVgTlt4\ny/3V1WJZEXrLpZdba3xw8C3rWr+mfswjobVLz4LnUl0dcIy59OxBeJR+frjgl4gWPDex4F50WWt4\nmYo6iytvFcFIwfPWWqfTwcnJCTqdDrrdLjqdTupani0PQx7D4XDCpZclvbprjSyc4fvU+/9u4eeL\nC37OZJWF6mw6tvDcyEK3mtYBrth7ZAXudMqqtvAnJydot9tot9s4OTlJDv4gkI+tIZZyp+Hs7Czl\n0uugndW4wrLwVqZdTOwu+Kvhgp8j06rhZD65blXFFXDanZfr9xj8Bz9tdhu76GzFWcztdhutViv1\nnCV4a3wU/4yyqw4/1vX9+meQQuf7lzsJcg+eYw2W8F3ws+OCnzNS6NLtll1r2KWX89S50yxvX7Eb\nb0WxJdJt1yWrOkuOLXmr1UqupXXn9TpH6HkctFXsoqffyMy6rJHU2vuQ11ayDYud70uKnr0Jt/Kz\n44KfM7HmD7F5cOzWc/WbnhgTm6xiReRZ2NIF5vNgMDDdd75ut9upeeyy7tzKgbcEb225WSOprXuX\ngpd773JMtJwRr117ZzZc8HNEW3e5ZpUikS69tPB6xLLep44JBUBqdLK1l356ehp11/k5OchxFsFL\n0ct4g7XlptFNNiwLzz+L/iCSbr279FfDBT8nYmKX0eiYhWfB66KYLJdeCgX4heDZmsstN712Z8F3\nu91E7J1OZyKVll16LXjr55nFpZf3bM2ms1x6/uDq9XoYDAbJ193CXw8X/ByxItBWb3lp4Tlwt7u7\naxbGWC59rPiFBc8C0cE5a6uNz91udyJzjtf/01x6aeGl4PU+u75/HViMreHZpdelsR60uzou+DkT\nE7t26Xm9zha+Xq9P7FPHXGJL7Nqll1tuHKSTFW5WtZvOntP73DGXPsvC60w6ef+6s46VXSddevY4\nrCw/F/1suODniOXW8x+8NdNdr+GtYJ98XSC+x2659J1OB61WC8fHx3jw4EEqa067+6enp5mls3wf\nV7Hw1geW5dJnlcRKl55r6q0lgTMbLvg5M2vnmVg0X8IZa/KP3Mpl5waT1jpdWnndoYYPdp2z7tcS\nuq7hl8lCssJPLkl0IpDsqCO3FOVePB+cxSdfh6+d2XDBzxHLzWYhawumO8fUarWJpYAukuE1rHUM\nh0M8ePAADx48SCx61v66FfTSHoX2WGQxjFyO6DoAFr5MGrI+uKTIZdxAu+q6Ii62l+9MxwU/ZyzR\nA0itUfUec7fbxfb29sRoKV3/nlWz3u/30Wq1ksNKqNGdYzggJ0UT8zyIKGXZZWowj7zSguetRW3h\nteB1qa0l+qxKQGd2XPBzRgudsYpBeG3K7aFi3V75dVnwVukqB+lkMo3ccuOg17TUVCvPXdcB6KQh\nnlPPwUfLwuvfje6Xp0Vv5cpPOzvTccHPCf5jtvabAaSsmZVFxoKXeeryNUulUmopEKtq04eMxFvb\nbjp1ls86j4ADj1kWnrcZZbagFrzVEdey7pbo3ZW/OS74OSOtzuXlJTY2Niai0FLwbOE5eq/LPmVh\nCn8vR+B1PnxWLXuv15sIkFk94bLyCOQug17DN5vNiZl4VuFPLDJvid4Suwv9Zrjg54i0lDIDjogm\n9pmla86Ct4JovIYHMGHhW61WEqTjKLw+5D52rJJO3rcWu8wJ0Gt43k5kl95qoR1z6XXQTrvzcg0f\na2Pl4r86LvgFIF1xHaXXe8zSnbey2uTARV7DSwt///795ODXlGe+Hg6HE5bSElEsLZgDidYani28\nzA6MtdHW1v0qFt4FfnNc8HPGsvI6V1xmw3HBidx2k91Z+XG5XJ4obeVrPse6yvLrzYJl4XVdu8yf\nl9NwWNyxTEFp3XWSjSyM8ckyi8MFvyBkAI/Pcj4bj2Zid300GqHX66U62crz5uZm4r7LfXaOwMv9\n9eu2gNIuvd4m1FuF+rC66cqEIhmdl0KXOw0yV8CLY+aPC34BWGKXxS2DwSDVAeby8hJnZ2epunJd\nKlsqlVL17HxwWmxs7tp1BG8V/ViuekzwfFi/Fyt4yUFFjjnIDy/PlZ8vLvg5wgK3npPtmwaDQUrs\n/JwOeMmjVCqlttnkdhxb+KxstVmx3PmY0HWBDJfyxjrq6n7zOvmILbzXuy8OF/yC0Gt5aeG12Nm1\njY2hYm9Aj3eSZ+4YG9vDnoVYlF6L3coClOv1mEsvLbx26fmDy136xeKCnzPaysvAHa/hgckONbqk\n1BKVDsTpx1ZRzVWj27NaeMulzyoEkr8DS/Ds1uvgnQt+vrjgF0DMtR+NRgDSYreGRMaKZ2QqqtWo\nMlY6Ok8Ln7V+t0p6Z3HpZYmuW/jF4oJfEJbI+I/XylW3auC1aLSIp1WTXSc7LcvCZ0XodSsu+Xry\nd5Ll0vsafvG44JfEKiSOSLHrJpWxQZCWdY/lvFslwrJMWFp4ny6zGFzwDoD0lpzuziPbZ2vRWz3n\nY2fZI193pdV78G7hF4ML3knQXW1kJp016y427lkf/HxWCyvZfUfW6/safr644J0Enb+v++/Jajhp\n4WMtrHR8IatJJbeh1um1buHnS3xg2Rgi+hQR3SWiZ8Vzf0pEPyOib46Ptyz2Np1lIKvztEtvWXgd\nrNO1A7r2XUborTbUctiEHiXlzIepggfw1wC0oAOAj4cQXjs+/mX+t+YsGxmw040qdWMLaw0PTHbS\njdW+6yYg2qV3C78Yprr0IYSvEtFjxpcm92CclYYtvNVSe5agHWAPmbD6zsugnWxDrWvhPUo/X2ax\n8DHeS0TfJqIniag5tztybg0dpZ8WtNMZdtb6XQo4FrSbNizSxT4/riv4TwJ4FYDXAHgewMfmdkfO\nwrGSfawIvR5rzdtzcmik7DvPQre62cj+8rGD3f3Y5Bvn5lwrSh9CuMfXRPRXAP5pbnfkLIxY2qu0\n7NqNl80qWfSyBbUO2sn0Wdkw8/z8fCKLTifWeN+6xXMtwRPRK0IIz48fvgPAs1n/3skPsWo2bkNt\nRee5d50VuNMWXgfnpAWPbbvJKLyLfrFMFTwRfQ7AGwAcEdFPAXwEwBuJ6DV4GK3/CYD3LPQunbmh\nRc7X1nRbaeHr9XryOEvwMiIv1+ocoItNvplXLYCTzSxR+seNpz+1gHtxFowldl0VZw2bYAvPz8vk\nG8ult/Ll+ZAuve5oE8vBd+aHZ9oVkKxWVnIarLbwsoCGW1DrNtSxBBtZGGOlzsasuQt/vrjgC0RM\n6LLvvLUdxxZeduDha2nh9Qgpvf2mLXzMpWdc7PPHBV8wYk0urDU8j45iCx+ri7fW8LoFtU6bnUXw\nzvxxwRcIKXarfRVbb10Dz+v2WCceIL5+1z3rdEttT6xZLi74gqEtuiV2dtnZmmuB6551MuFGNuVk\noXN77U6nk6p7ZyvvxTHLwwVfIKysOt3VRotduvAxsQN2vzpOmeWxWCcnJ6kU2tiMemdxuOALQmzd\nrsVuWXi5Trc60uqONjJQx/3zeVZ9t9s1S2Cd5eCCLxB6/a7nxUnRWy69fB1Gt+HOsvBywoy28M5y\ncMEXiFj7aRZ8zLpLCw9kN6mUte56Dc+lsGzhve/88nHBF4hYZdw0C89HrMBFB+3Ozs5SZa9s4fV0\nW7fwy8cFXzCsKL0ldsut5603Rte/xyw8C16PwJZReg/aLQcX/BoSGwhh5cvr5BrZ6EJ3teGJOtYg\nDJlhF2tyoSfleFfa5eOCXyOy6t2JKMmR53RZTpnl8/7+Pvb29hLhV6vV1Ax7wG5hpfvWxQ7Z1MKz\n624HF/yaERtbxfXuW1tbieAbjQb29vaSc7PZRLPZTGrf2crrjjbSssvec1litzrZuOiXjwt+zbAa\nW7A7Xy6XUxaeRb6/v49ms4lGo4FGo5EIflYLz+twOa7aGnqpG1P62n35uODXCG3RteBlJxu28Pv7\n+zg4OMDh4SF2dnZSa3kWvGXhdYNKKfSYtbcsu1v45eKCXzOsajhZ/iotfKPRQLPZxOHhIY6OjlIz\n5GTgLsvCS7FnWXfZgVZPpHHBLw8X/BoRE7sUvF7Ds+Dv3LmTbM/Jw7LwMSs/bQ0fmzvnLA8X/JqR\nVe9uWXh26Y+OjiYKZmTiDWNF6Gdx50ejkXe1yQEu+DXEErvsaCNbWMm+dVa9u6x51yOjOFuOj9gE\nWE+uyQ8u+DXC6kZrtbKyWlxl1bsDvyiOkY0t+ODHx8fHSd27LJDxxJr84IJfM6at46XVj1lznbjD\nlplTZweDQVL51uv10O120ev1cHx8jFarlRI87787+cAFvyZY+++WWy+Frq2+lbTDSAvPqbJc5861\n7tLC93q9VEWckw9c8GvItO60lnXnwJy27hLuRMsWvtPpoNVqJUe73Z5w6c/Pz339niNc8GuGNWhi\nFusu6935dRjp0mvBt9ttHB8f4/j4OOlowxae1/Bu4fODC36NyHLrtXtvBewYyxrL8lcufT05OUGr\n1cLx8TFefPHFJEovo/Xu0ucLF/yakbUPb1l4ebDQuQQWmOxoE7PwL730UmqcFJfGskvv5AMX/Boh\nxz7LKbCVSiWVNitr3WVHWgATWXAys07uv/NWHAv/5OTEnPXuHW3yhQt+jZDpszovvlarYW9vLzXn\nXZe/AnYJLJ9ZxHoUtLzWde+eQpsvXPBrhKx55ww6WQEnBc8jn/UwyGnNLWKily2rrMo4Jx+44NcE\n2cJKCn53dxe7u7tJoYy28FmCl/XrHLCzDmndpYX3MVL5YyPri0T0KBF9hYi+R0TfJaL3jZ8/IKJn\niOj/iOhLRNRczu06WWgLLyviDg4OEgvPfet0NRwwKfpY5Zu07G7hV4dMwQM4B/CBEMKvA/hNAH9I\nRL8G4EMAngkh/CqAfxs/dm4ZOUlGWnjuaqNd+iwLr5tbxKy75db7Gj6/ZLr0IYQXALwwvu4S0Q8A\nvBLA2wC8YfzPPg3g3+Giv3U4Qi9demnh6/V6sqZnCy8FD2Bi3S7LXWNi57NuauGCzx8zr+GJ6DEA\nrwXwXwAeCSHcHX/pLoBH5n5nzpWQfetkk4vd3V3s7e1hf38/aUnNEfxYk0pt5WXnmphlPzs7mxB3\nrP7duT1mEjwR1QF8AcD7QwgdlXYZiMj/R5eEznHnx3L/fWtrK2XleS0vZ73z+l2OkZJCZ6suBa3r\n3632VU6+mSp4Iirjodg/G0J4avz0XSJ6eQjhBSJ6BYB7i7xJ5yFW+2k5IFJOkWHRa5HLkdCySg5I\np89yCq0cDXV6epqaC+dCXz2mRekJwJMAvh9C+IT40tMAnhhfPwHgKf29zvzRAremwMquNlrwnHVn\nCV5H5uXkGJ4RZ3Wz4e9xVoNpFv71AN4F4DtE9M3xc38M4M8BfJ6I/gDAcwDeubA7dFLEimKswZDc\nyooP+W+k4Bm28FLw3NUmJni38KvFtCj91xD3At40/9txsoi1reL9dyl26dKzlZfNKS2Xntfv7NJr\nC69HPfu459XDM+1WjFglnGXhtUuvm2DE1vDawme59G7hVwsX/IqQJXTZkTZL8LIpxrSutLE1vAft\nVhsX/AqRJXpZEivLYqXoreg+PwbSFp5derbw3W43auHdpV8dXPArxqyNKvUHweZm/L+aE2asencW\nO3emlaLnNbxn060OLvgVRVtr6+ux7rM6A4635Cw3nnvUyW607NbLrTlnNXDBrzCWi571AZCV6y4F\nLwN13M2m3W4nfeh1vzoX/Orggl9BpKCnWXqNTLDRte/SwvPee7fbTQQvp824hV9NXPAriiV6vo5Z\nfGv6qyySkWOkpEvPgpdpttyC2gW/WrjgVwircCYWeY+RNerZSrbpdDrJGl6Ww3qTytXEBb9ixNbn\n+nFM/Fnlr3oNz9F5tvCyfZVuVumsBi74FcRy12Mit1z62Gx3K0ovg3ZWJ1s+O6uBC36F0KOjrP7z\nVnEMi17Xu+tGFjxIInZY23ny7OQfF/wKIaviZBUcd7ixes7zkAm27jJ1li26rneXATnZmw5wca86\nLvgVQU6V4bJXHjAhG1ZqwcuedbIaTlt0XrfLLTe9z261r3JWCxf8CqEFX6vVkhZWfEjBVyqViRZW\nseIYOQDSKo5xsa8HLvgVQgpeN6psNBopC1+tVhMLb9W7WxF5y8LrbTctdhf/auGCXyFk3zp26dmV\n14KXa3jt0rOF10k2eg0vXfrYCGlntXDBrwjWGl63otaCt2bH6Y42szS40NtuLvTVxQW/QlhBOx42\nocdI6Sg9EG9wwSm0WUE7F/l64ILPGbFMOt5/l/vuMmjH1r1Wq6Umw7LgdUdaWSDDCTZa9J46u364\n4HNEVuac7jkvt+V4fJQUu96D1+2nZbCOM+lkvbvci3frvj644HOGTpvlTDmdVacFX6/XkyQcHhKp\n3XnLurPgW61Wqt5ddrRxC78+uOBzhBa5bDKpB0xIwbPoZT96beGtYF1WvbuXv64nLvgcoS277Ds/\ni4XnHnZ86Kky01x6WevuDS7WExd8TpBrd6s5pRY8T4CVgtffJ1tQW/vvuhpO17p70G79cMHniNjs\nOEvwVtCWHs+RAAAGw0lEQVROt52WZbFs4XWTC25S2Wq1zFp3X8OvFy74nGFZeN13PubS6yaVumZd\nu/R6DW81tvR69/XCBZ8jpMDlJJnNzc3EhedDz46rVCopi8xnnT8vE274rOvdAa+MW1dc8DlBp85q\nQW9vb6PRaKBeryfFMVbqLFtyfchyWA7GyXHPLupi4ILPETKTjl12PjiFtl6vpxJs5F67HhUlDx19\nz5oN5+JfX2KjoAEARPQoEX2FiL5HRN8loveNn/9TIvoZEX1zfLxlObe73rCFZ4vOlXB7e3toNpsT\n6bNWg4usElir9FU3uHDxrzfTLPw5gA+EEL5FRHUA/0NEzwAIAD4eQvj4wu+wIEyrhuMSWLbwWdVw\nsgstW3VOpokVxnj5azHIFHwI4QUAL4yvu0T0AwCvHH95+pgT50qwS88WngXP1XBXWcNfx8JLXOzr\nSaZLLyGixwC8FsB/jp96LxF9m4ieJKLmAu6tUFgWXrv0jUbDrIjTgrdKYNnCy+YWHLSTDSp9Eux6\nM5Pgx+78PwB4fwihC+CTAF4F4DUAngfwsYXdYYGIufSy3t0K2lkdbThQJy18LGjnUfriMFXwRFQG\n8AUAfxNCeAoAQgj3whgAfwXgdYu9zWKgk204Yq/7zsue85Y7bzW3kPXuuoWVi704TIvSE4AnAXw/\nhPAJ8fwrxD97B4BnF3N7xULPa5cJNDrVVR5W6qxMm22323jw4AHa7Ta63W5S886i90y64jAtSv96\nAO8C8B0i+ub4uT8B8DgRvQYPo/U/AfCexd1isbAmvGqhx0QvJ8BynzqudW+1WqkmF17+WkymRem/\nBtsL+OfF3E6xyRrlrC283lLTFr7f709YeKuNlQu+WHimXc6YJvqrWHjZzeb4+Hhi4IRb+OLhgs8R\nVqWbZeVnWcPLene28Ho4pFv44uGCzxmW2KcF7Wax8A8ePEiq5WTlHEfqnWLggs8RVh36NJder+F1\nvbu08LzvzpVyfO0Wvji44HOCFC4Xv8gsOU6y0a2rgIcJN8PhECcnJ0nLKg7Q8Tbc6elpNNLv+/DF\nwQWfI6TQ+/3+xCAJ7a6zy87FNHfv3sW9e/dw//59tFotdDqdZL9dLwU8jbaYuOBzBLvlLHjZZno0\nGqWaT8rZ8Nvb26hUKrh//z7u37+Pl156aULw1prfBV88XPA5QfaOHw6HKcvOwThOpqlWq8nQCT6X\ny+UkuYYTbbrdLvr9fiqF1hK7i744uOBzxOXlJc7Pzycs+/n5OQaDQWrQhB46sbm5mcqZ5+vT09PE\nwuugoFv44uGCzwmyecVwOExZ9sFgkGpoyYU1cuhEqVRKTY2RU2R4De9lsI4LPkewhZeRet2fXg6n\n0NNpeH+d+9jJa133bp2d9YcW9Z9NRP5XdEX0bLlZD/5eK/tOHhoX+noTQpjoSuWCd5w1xRL8zC2u\nHMdZfVzwjlMgXPCOUyBc8I5TIBYWtHMcJ3+4hXecAuGCd5wCsRTBE9FbiOiHRPQjIvrgMt7zKhDR\nc0T0nfFgzP/Owf18iojuEtGz4rkDInqGiP6PiL50m9N+IveXiwGjGQNQc/H7u+0BrQtfwxNRCcD/\nAngTgJ8D+DqAx0MIP1joG18BIvoJgN8IIRzf9r0AABH9NoAugM+EEF49fu6jAF4KIXx0/KG5H0L4\nUI7u7yMAOrc9YJSIXg7g5XIAKoDfBfD7yMHvL+P+3okl/P6WYeFfB+DHIYTnQgjnAP4OwNuX8L5X\nJTfDMUMIXwXwQD39NgCfHl9/Gg//SG6FyP0BOfgdhhBeCCF8a3zdBcADUHPx+8u4P2AJv79lCP6V\nAH4qHv8Mv/gB80IA8GUi+gYRvfu2bybCIyGEu+PruwAeuc2biZCrAaNiAOp/IYe/v9sY0LoMwa/C\nvt/rQwivBfBWAH84dllzy3imX95+r7kaMDp2l7+AhwNQO/Jrefj93daA1mUI/ucAHhWPH8VDK58b\nQgjPj88vAvgi8jkc8+54/cez/e7d8v2kyNOAUTEA9bM8ABU5+v3d5oDWZQj+GwB+hYgeI6IKgN8D\n8PQS3ncmiKhGRLvj6x0Ab0Y+h2M+DeCJ8fUTAJ7K+LdLJy8DRmMDUJGT399tD2hdSqYdEb0VwCcA\nlAA8GUL4s4W/6YwQ0avw0KoDDxuC/O1t3x8RfQ7AGwAc4eF688MA/hHA5wH8EoDnALwzhNDKyf19\nBMAb8dAdTQaMijXzMu/ttwD8B4Dv4Bdu+x8D+G/k4PcXub8/AfA4lvD789RaxykQnmnnOAXCBe84\nBcIF7zgFwgXvOAXCBe84BcIF7zgFwgXvOAXCBe84BeL/Aebom1naQaBHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ffb549c8610>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result: 7\n"
     ]
    }
   ],
   "source": [
    "plt.imshow((X_test[0].reshape((28,28))*255).astype(np.uint8), cmap='Greys_r')\n",
    "plt.show()\n",
    "print 'Result:', model.predict(X_test[0:1])[0].argmax()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also evaluate the model's accuracy on the entire test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 97.64 %\n"
     ]
    }
   ],
   "source": [
    "print 'Accuracy:', model.score(test_iter)*100, '%'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now, try if your model recognizes your own hand writing.\n",
    "\n",
    "Write a digit from 0 to 9 in the box below. Try to put your digit in the middle of the box."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">canvas { border: 1px solid black; }</style><div id=\"board\"><canvas id=\"myCanvas\" width=\"100px\" height=\"100px\">Sorry, your browser doesn't support canvas technology.</canvas><p><button id=\"classify\" onclick=\"classify()\">Classify</button><button id=\"clear\" onclick=\"myClear()\">Clear</button>Result: <input type=\"text\" id=\"result_output\" size=\"5\" value=\"\"></p></div><script type=\"text/JavaScript\" src=\"https://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js?ver=1.4.2\"></script><script type=\"text/javascript\">function init() {var myCanvas = document.getElementById(\"myCanvas\");var curColor = $('#selectColor option:selected').val();if(myCanvas){var isDown = false;var ctx = myCanvas.getContext(\"2d\");var canvasX, canvasY;ctx.lineWidth = 5;$(myCanvas).mousedown(function(e){isDown = true;ctx.beginPath();var parentOffset = $(this).parent().offset(); canvasX = e.pageX - parentOffset.left;canvasY = e.pageY - parentOffset.top;ctx.moveTo(canvasX, canvasY);}).mousemove(function(e){if(isDown != false) {var parentOffset = $(this).parent().offset(); canvasX = e.pageX - parentOffset.left;canvasY = e.pageY - parentOffset.top;ctx.lineTo(canvasX, canvasY);ctx.strokeStyle = curColor;ctx.stroke();}}).mouseup(function(e){isDown = false;ctx.closePath();});}$('#selectColor').change(function () {curColor = $('#selectColor option:selected').val();});}init();function handle_output(out) {document.getElementById(\"result_output\").value = out.content.data[\"text/plain\"];}function classify() {var kernel = IPython.notebook.kernel;var myCanvas = document.getElementById(\"myCanvas\");data = myCanvas.toDataURL('image/png');document.getElementById(\"result_output\").value = \"\";kernel.execute(\"classify('\" + data +\"')\",  { 'iopub' : {'output' : handle_output}}, {silent:false});}function myClear() {var myCanvas = document.getElementById(\"myCanvas\");myCanvas.getContext(\"2d\").clearRect(0, 0, myCanvas.width, myCanvas.height);}</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run hand drawing test\n",
    "from IPython.display import HTML\n",
    "\n",
    "def classify(img):\n",
    "    img = img[len('data:image/png;base64,'):].decode('base64')\n",
    "    img = cv2.imdecode(np.fromstring(img, np.uint8), -1)\n",
    "    img = cv2.resize(img[:,:,3], (28,28))\n",
    "    img = img.astype(np.float32).reshape((1, 784))/255.0\n",
    "    return model.predict(img)[0].argmax()\n",
    "\n",
    "html = \"\"\"<style type=\"text/css\">canvas { border: 1px solid black; }</style><div id=\"board\"><canvas id=\"myCanvas\" width=\"100px\" height=\"100px\">Sorry, your browser doesn't support canvas technology.</canvas><p><button id=\"classify\" onclick=\"classify()\">Classify</button><button id=\"clear\" onclick=\"myClear()\">Clear</button>Result: <input type=\"text\" id=\"result_output\" size=\"5\" value=\"\"></p></div>\"\"\"\n",
    "script = \"\"\"<script type=\"text/JavaScript\" src=\"https://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js?ver=1.4.2\"></script><script type=\"text/javascript\">function init() {var myCanvas = document.getElementById(\"myCanvas\");var curColor = $('#selectColor option:selected').val();if(myCanvas){var isDown = false;var ctx = myCanvas.getContext(\"2d\");var canvasX, canvasY;ctx.lineWidth = 5;$(myCanvas).mousedown(function(e){isDown = true;ctx.beginPath();var parentOffset = $(this).parent().offset(); canvasX = e.pageX - parentOffset.left;canvasY = e.pageY - parentOffset.top;ctx.moveTo(canvasX, canvasY);}).mousemove(function(e){if(isDown != false) {var parentOffset = $(this).parent().offset(); canvasX = e.pageX - parentOffset.left;canvasY = e.pageY - parentOffset.top;ctx.lineTo(canvasX, canvasY);ctx.strokeStyle = curColor;ctx.stroke();}}).mouseup(function(e){isDown = false;ctx.closePath();});}$('#selectColor').change(function () {curColor = $('#selectColor option:selected').val();});}init();function handle_output(out) {document.getElementById(\"result_output\").value = out.content.data[\"text/plain\"];}function classify() {var kernel = IPython.notebook.kernel;var myCanvas = document.getElementById(\"myCanvas\");data = myCanvas.toDataURL('image/png');document.getElementById(\"result_output\").value = \"\";kernel.execute(\"classify('\" + data +\"')\",  { 'iopub' : {'output' : handle_output}}, {silent:false});}function myClear() {var myCanvas = document.getElementById(\"myCanvas\");myCanvas.getContext(\"2d\").clearRect(0, 0, myCanvas.width, myCanvas.height);}</script>\"\"\"\n",
    "HTML(html+script)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Debugging\n",
    "\n",
    "DNNs can perform poorly for a lot of reasons, like learning rate too big/small, initialization too big/small, network structure not reasonable, etc. When this happens it's often helpful to print out the weights and intermediate outputs to understand what's going on. MXNet provides a monitor utility that does this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Start training with [gpu(0)]\n",
      "INFO:root:Batch:       1 fc1_backward_weight            0.000557817\t\n",
      "INFO:root:Batch:       1 fc1_weight                     0.00576876\t\n",
      "INFO:root:Batch:       1 fc2_backward_weight            0.00181161\t\n",
      "INFO:root:Batch:       1 fc2_weight                     0.0057341\t\n",
      "INFO:root:Batch:       1 fc3_backward_weight            0.00535519\t\n",
      "INFO:root:Batch:       1 fc3_weight                     0.0057601\t\n",
      "INFO:root:Epoch[0] Batch [100]\tSpeed: 35343.43 samples/sec\tTrain-accuracy=0.137700\n",
      "INFO:root:Batch:     101 fc1_backward_weight            0.0690022\t\n",
      "INFO:root:Batch:     101 fc1_weight                     0.00705788\t\n",
      "INFO:root:Batch:     101 fc2_backward_weight            0.198601\t\n",
      "INFO:root:Batch:     101 fc2_weight                     0.0152523\t\n",
      "INFO:root:Batch:     101 fc3_backward_weight            1.39443\t\n",
      "INFO:root:Batch:     101 fc3_weight                     0.054037\t\n",
      "INFO:root:Epoch[0] Batch [200]\tSpeed: 34402.10 samples/sec\tTrain-accuracy=0.425050\n",
      "INFO:root:Batch:     201 fc1_backward_weight            0.172101\t\n",
      "INFO:root:Batch:     201 fc1_weight                     0.023511\t\n",
      "INFO:root:Batch:     201 fc2_backward_weight            0.407629\t\n",
      "INFO:root:Batch:     201 fc2_weight                     0.0507858\t\n",
      "INFO:root:Batch:     201 fc3_backward_weight            1.09523\t\n",
      "INFO:root:Batch:     201 fc3_weight                     0.216835\t\n",
      "INFO:root:Epoch[0] Batch [300]\tSpeed: 34260.16 samples/sec\tTrain-accuracy=0.581567\n",
      "INFO:root:Batch:     301 fc1_backward_weight            0.121955\t\n",
      "INFO:root:Batch:     301 fc1_weight                     0.0298391\t\n",
      "INFO:root:Batch:     301 fc2_backward_weight            0.254035\t\n",
      "INFO:root:Batch:     301 fc2_weight                     0.0630359\t\n",
      "INFO:root:Batch:     301 fc3_backward_weight            0.69842\t\n",
      "INFO:root:Batch:     301 fc3_weight                     0.239144\t\n",
      "INFO:root:Epoch[0] Batch [400]\tSpeed: 45079.35 samples/sec\tTrain-accuracy=0.667450\n",
      "INFO:root:Batch:     401 fc1_backward_weight            0.148367\t\n",
      "INFO:root:Batch:     401 fc1_weight                     0.0343186\t\n",
      "INFO:root:Batch:     401 fc2_backward_weight            0.330322\t\n",
      "INFO:root:Batch:     401 fc2_weight                     0.0699758\t\n",
      "INFO:root:Batch:     401 fc3_backward_weight            0.862394\t\n",
      "INFO:root:Batch:     401 fc3_weight                     0.245399\t\n",
      "INFO:root:Epoch[0] Batch [500]\tSpeed: 52828.71 samples/sec\tTrain-accuracy=0.720200\n",
      "INFO:root:Batch:     501 fc1_backward_weight            0.168489\t\n",
      "INFO:root:Batch:     501 fc1_weight                     0.0382638\t\n",
      "INFO:root:Batch:     501 fc2_backward_weight            0.354083\t\n",
      "INFO:root:Batch:     501 fc2_weight                     0.0766841\t\n",
      "INFO:root:Batch:     501 fc3_backward_weight            0.869474\t\n",
      "INFO:root:Batch:     501 fc3_weight                     0.252072\t\n",
      "INFO:root:Epoch[0] Batch [600]\tSpeed: 51447.76 samples/sec\tTrain-accuracy=0.757567\n",
      "INFO:root:Epoch[0] Resetting Data Iterator\n",
      "INFO:root:Epoch[0] Train-accuracy=0.757567\n",
      "INFO:root:Epoch[0] Time cost=1.484\n",
      "INFO:root:Epoch[0] Validation-accuracy=0.953200\n"
     ]
    }
   ],
   "source": [
    "def norm_stat(d):\n",
    "    \"\"\"The statistics you want to see.\n",
    "    We compute the L2 norm here but you can change it to anything you like.\"\"\"\n",
    "    return mx.nd.norm(d)/np.sqrt(d.size)\n",
    "mon = mx.mon.Monitor(\n",
    "    100,                 # Print every 100 batches\n",
    "    norm_stat,           # The statistics function defined above\n",
    "    pattern='.*weight',  # A regular expression. Only arrays with name matching this pattern will be included.\n",
    "    sort=True)           # Sort output by name\n",
    "model = mx.model.FeedForward(ctx = mx.gpu(0), symbol = mlp, num_epoch = 1,\n",
    "                             learning_rate = 0.1, momentum = 0.9, wd = 0.00001)\n",
    "model.fit(X=train_iter, eval_data=test_iter, monitor=mon,  # Set the monitor here\n",
    "          batch_end_callback = mx.callback.Speedometer(100, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Under the hood: Custom Training Loop\n",
    "\n",
    "`mx.model.FeedForward` is a convenience wrapper for training standard feed forward networks. What if the model you are working with is more complicated? With MXNet, you can easily control every aspect of training by writing your own training loop.\n",
    "\n",
    "Neural network training typically has 3 steps: forward, backward (gradient), and update. With custom training loop, you can control the details in each step as while as insert complicated computations in between. You can also connect multiple networks together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_shapes {'softmax_label': (100,), 'data': (100, 784)}\n",
      "epoch: 0 iter: 100 metric: ('accuracy', 0.1427)\n",
      "epoch: 0 iter: 200 metric: ('accuracy', 0.42695)\n",
      "epoch: 0 iter: 300 metric: ('accuracy', 0.5826333333333333)\n",
      "epoch: 0 iter: 400 metric: ('accuracy', 0.66875)\n",
      "epoch: 0 iter: 500 metric: ('accuracy', 0.72238)\n",
      "epoch: 0 iter: 600 metric: ('accuracy', 0.7602166666666667)\n",
      "epoch: 1 iter: 100 metric: ('accuracy', 0.9504)\n",
      "epoch: 1 iter: 200 metric: ('accuracy', 0.9515)\n",
      "epoch: 1 iter: 300 metric: ('accuracy', 0.9547666666666667)\n",
      "epoch: 1 iter: 400 metric: ('accuracy', 0.95665)\n",
      "epoch: 1 iter: 500 metric: ('accuracy', 0.95794)\n",
      "epoch: 1 iter: 600 metric: ('accuracy', 0.95935)\n",
      "epoch: 2 iter: 100 metric: ('accuracy', 0.9657)\n",
      "epoch: 2 iter: 200 metric: ('accuracy', 0.96715)\n",
      "epoch: 2 iter: 300 metric: ('accuracy', 0.9698)\n",
      "epoch: 2 iter: 400 metric: ('accuracy', 0.9702)\n",
      "epoch: 2 iter: 500 metric: ('accuracy', 0.97104)\n",
      "epoch: 2 iter: 600 metric: ('accuracy', 0.9717)\n",
      "epoch: 3 iter: 100 metric: ('accuracy', 0.976)\n",
      "epoch: 3 iter: 200 metric: ('accuracy', 0.97575)\n",
      "epoch: 3 iter: 300 metric: ('accuracy', 0.9772666666666666)\n",
      "epoch: 3 iter: 400 metric: ('accuracy', 0.9771)\n",
      "epoch: 3 iter: 500 metric: ('accuracy', 0.9771)\n",
      "epoch: 3 iter: 600 metric: ('accuracy', 0.97755)\n",
      "epoch: 4 iter: 100 metric: ('accuracy', 0.9805)\n",
      "epoch: 4 iter: 200 metric: ('accuracy', 0.9803)\n",
      "epoch: 4 iter: 300 metric: ('accuracy', 0.9814666666666667)\n",
      "epoch: 4 iter: 400 metric: ('accuracy', 0.981175)\n",
      "epoch: 4 iter: 500 metric: ('accuracy', 0.98132)\n",
      "epoch: 4 iter: 600 metric: ('accuracy', 0.98145)\n",
      "epoch: 5 iter: 100 metric: ('accuracy', 0.9837)\n",
      "epoch: 5 iter: 200 metric: ('accuracy', 0.98365)\n",
      "epoch: 5 iter: 300 metric: ('accuracy', 0.9835333333333334)\n",
      "epoch: 5 iter: 400 metric: ('accuracy', 0.98395)\n",
      "epoch: 5 iter: 500 metric: ('accuracy', 0.984)\n",
      "epoch: 5 iter: 600 metric: ('accuracy', 0.9842166666666666)\n",
      "epoch: 6 iter: 100 metric: ('accuracy', 0.9848)\n",
      "epoch: 6 iter: 200 metric: ('accuracy', 0.98475)\n",
      "epoch: 6 iter: 300 metric: ('accuracy', 0.9858333333333333)\n",
      "epoch: 6 iter: 400 metric: ('accuracy', 0.98555)\n",
      "epoch: 6 iter: 500 metric: ('accuracy', 0.9855)\n",
      "epoch: 6 iter: 600 metric: ('accuracy', 0.9856333333333334)\n",
      "epoch: 7 iter: 100 metric: ('accuracy', 0.9842)\n",
      "epoch: 7 iter: 200 metric: ('accuracy', 0.98625)\n",
      "epoch: 7 iter: 300 metric: ('accuracy', 0.9869)\n",
      "epoch: 7 iter: 400 metric: ('accuracy', 0.9877)\n",
      "epoch: 7 iter: 500 metric: ('accuracy', 0.98774)\n",
      "epoch: 7 iter: 600 metric: ('accuracy', 0.9875333333333334)\n",
      "epoch: 8 iter: 100 metric: ('accuracy', 0.9864)\n",
      "epoch: 8 iter: 200 metric: ('accuracy', 0.9878)\n",
      "epoch: 8 iter: 300 metric: ('accuracy', 0.9886666666666667)\n",
      "epoch: 8 iter: 400 metric: ('accuracy', 0.98885)\n",
      "epoch: 8 iter: 500 metric: ('accuracy', 0.98918)\n",
      "epoch: 8 iter: 600 metric: ('accuracy', 0.9894666666666667)\n",
      "epoch: 9 iter: 100 metric: ('accuracy', 0.9884)\n",
      "epoch: 9 iter: 200 metric: ('accuracy', 0.98855)\n",
      "epoch: 9 iter: 300 metric: ('accuracy', 0.9894666666666667)\n",
      "epoch: 9 iter: 400 metric: ('accuracy', 0.98945)\n",
      "epoch: 9 iter: 500 metric: ('accuracy', 0.98972)\n",
      "epoch: 9 iter: 600 metric: ('accuracy', 0.9899333333333333)\n"
     ]
    }
   ],
   "source": [
    "# ==================Binding=====================\n",
    "# The symbol we created is only a graph description.\n",
    "# To run it, we first need to allocate memory and create an executor by 'binding' it.\n",
    "# In order to bind a symbol, we need at least two pieces of information: context and input shapes.\n",
    "# Context specifies which device the executor runs on, e.g. cpu, GPU0, GPU1, etc.\n",
    "# Input shapes define the executor's input array dimensions.\n",
    "# MXNet then run automatic shape inference to determine the dimensions of intermediate and output arrays.\n",
    "\n",
    "# data iterators defines shapes of its output with provide_data and provide_label property.\n",
    "input_shapes = dict(train_iter.provide_data+train_iter.provide_label)\n",
    "print 'input_shapes', input_shapes\n",
    "# We use simple_bind to let MXNet allocate memory for us.\n",
    "# You can also allocate memory youself and use bind to pass it to MXNet.\n",
    "exe = mlp.simple_bind(ctx=mx.gpu(0), **input_shapes)\n",
    "\n",
    "# ===============Initialization=================\n",
    "# First we get handle to input arrays\n",
    "arg_arrays = dict(zip(mlp.list_arguments(), exe.arg_arrays))\n",
    "data = arg_arrays[train_iter.provide_data[0][0]]\n",
    "label = arg_arrays[train_iter.provide_label[0][0]]\n",
    "\n",
    "# We initialize the weights with uniform distribution on (-0.01, 0.01).\n",
    "init = mx.init.Uniform(scale=0.01)\n",
    "for name, arr in arg_arrays.items():\n",
    "    if name not in input_shapes:\n",
    "        init(name, arr)\n",
    "    \n",
    "# We also need to create an optimizer for updating weights\n",
    "opt = mx.optimizer.SGD(\n",
    "    learning_rate=0.1,\n",
    "    momentum=0.9,\n",
    "    wd=0.00001,\n",
    "    rescale_grad=1.0/train_iter.batch_size)\n",
    "updater = mx.optimizer.get_updater(opt)\n",
    "\n",
    "# Finally we need a metric to print out training progress\n",
    "metric = mx.metric.Accuracy()\n",
    "\n",
    "# Training loop begines\n",
    "for epoch in range(10):\n",
    "    train_iter.reset()\n",
    "    metric.reset()\n",
    "    t = 0\n",
    "    for batch in train_iter:\n",
    "        # Copy data to executor input. Note the [:].\n",
    "        data[:] = batch.data[0]\n",
    "        label[:] = batch.label[0]\n",
    "        \n",
    "        # Forward\n",
    "        exe.forward(is_train=True)\n",
    "        \n",
    "        # You perform operations on exe.outputs here if you need to.\n",
    "        # For example, you can stack a CRF on top of a neural network.\n",
    "        \n",
    "        # Backward\n",
    "        exe.backward()\n",
    "        \n",
    "        # Update\n",
    "        for i, pair in enumerate(zip(exe.arg_arrays, exe.grad_arrays)):\n",
    "            weight, grad = pair\n",
    "            updater(i, grad, weight)\n",
    "        metric.update(batch.label, exe.outputs)\n",
    "        t += 1\n",
    "        if t % 100 == 0:\n",
    "            print 'epoch:', epoch, 'iter:', t, 'metric:', metric.get()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Operators\n",
    "\n",
    "MXNet provides a repository of common operators (or layers). However, new models often require new layers. There are several ways to [create new operators](https://mxnet.readthedocs.org/en/latest/tutorial/new_op_howto.html) with MXNet. Here we talk about the easiest way: pure python. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Start training with [gpu(0)]\n",
      "INFO:root:Epoch[0] Batch [100]\tSpeed: 32015.46 samples/sec\tTrain-accuracy=0.136000\n",
      "INFO:root:Epoch[0] Batch [200]\tSpeed: 42459.26 samples/sec\tTrain-accuracy=0.409200\n",
      "INFO:root:Epoch[0] Batch [300]\tSpeed: 42320.50 samples/sec\tTrain-accuracy=0.570833\n",
      "INFO:root:Epoch[0] Batch [400]\tSpeed: 42720.08 samples/sec\tTrain-accuracy=0.660200\n",
      "INFO:root:Epoch[0] Batch [500]\tSpeed: 42759.71 samples/sec\tTrain-accuracy=0.714940\n",
      "INFO:root:Epoch[0] Batch [600]\tSpeed: 42681.95 samples/sec\tTrain-accuracy=0.753933\n",
      "INFO:root:Epoch[0] Resetting Data Iterator\n",
      "INFO:root:Epoch[0] Train-accuracy=0.753933\n",
      "INFO:root:Epoch[0] Time cost=1.496\n",
      "INFO:root:Epoch[0] Validation-accuracy=0.953000\n",
      "INFO:root:Epoch[1] Batch [100]\tSpeed: 43306.01 samples/sec\tTrain-accuracy=0.946900\n",
      "INFO:root:Epoch[1] Batch [200]\tSpeed: 42849.48 samples/sec\tTrain-accuracy=0.950100\n",
      "INFO:root:Epoch[1] Batch [300]\tSpeed: 42720.82 samples/sec\tTrain-accuracy=0.953133\n",
      "INFO:root:Epoch[1] Batch [400]\tSpeed: 42676.13 samples/sec\tTrain-accuracy=0.955825\n",
      "INFO:root:Epoch[1] Batch [500]\tSpeed: 43642.57 samples/sec\tTrain-accuracy=0.957380\n",
      "INFO:root:Epoch[1] Batch [600]\tSpeed: 44032.33 samples/sec\tTrain-accuracy=0.959433\n",
      "INFO:root:Epoch[1] Resetting Data Iterator\n",
      "INFO:root:Epoch[1] Train-accuracy=0.959433\n",
      "INFO:root:Epoch[1] Time cost=1.397\n",
      "INFO:root:Epoch[1] Validation-accuracy=0.961300\n"
     ]
    }
   ],
   "source": [
    "# Define custom softmax operator\n",
    "class NumpySoftmax(mx.operator.NumpyOp):\n",
    "    def __init__(self):\n",
    "        # Call the parent class constructor. \n",
    "        # Because NumpySoftmax is a loss layer, it doesn't need gradient input from layers above.\n",
    "        super(NumpySoftmax, self).__init__(need_top_grad=False)\n",
    "    \n",
    "    def list_arguments(self):\n",
    "        # Define the input to NumpySoftmax.\n",
    "        return ['data', 'label']\n",
    "\n",
    "    def list_outputs(self):\n",
    "        # Define the output.\n",
    "        return ['output']\n",
    "\n",
    "    def infer_shape(self, in_shape):\n",
    "        # Calculate the dimensions of the output (and missing inputs) from (some) input shapes.\n",
    "        data_shape = in_shape[0]  # shape of first argument 'data'\n",
    "        label_shape = (in_shape[0][0],)  # 'label' should be one dimensional and has batch_size instances.\n",
    "        output_shape = in_shape[0] # 'output' dimension is the same as the input.\n",
    "        return [data_shape, label_shape], [output_shape]\n",
    "\n",
    "    def forward(self, in_data, out_data):\n",
    "        x = in_data[0]  # 'data'\n",
    "        y = out_data[0]  # 'output'\n",
    "        \n",
    "        # Compute softmax\n",
    "        y[:] = np.exp(x - x.max(axis=1).reshape((x.shape[0], 1)))\n",
    "        y /= y.sum(axis=1).reshape((x.shape[0], 1))\n",
    "\n",
    "    def backward(self, out_grad, in_data, out_data, in_grad):\n",
    "        l = in_data[1]  # 'label'\n",
    "        l = l.reshape((l.size,)).astype(np.int)  # cast to int\n",
    "        y = out_data[0]  # 'output'\n",
    "        dx = in_grad[0]  # gradient for 'data'\n",
    "        \n",
    "        # Compute gradient\n",
    "        dx[:] = y\n",
    "        dx[np.arange(l.shape[0]), l] -= 1.0\n",
    "\n",
    "numpy_softmax = NumpySoftmax()\n",
    "\n",
    "data = mx.symbol.Variable('data')\n",
    "fc1 = mx.symbol.FullyConnected(data = data, name='fc1', num_hidden=128)\n",
    "act1 = mx.symbol.Activation(data = fc1, name='relu1', act_type=\"relu\")\n",
    "fc2 = mx.symbol.FullyConnected(data = act1, name = 'fc2', num_hidden = 64)\n",
    "act2 = mx.symbol.Activation(data = fc2, name='relu2', act_type=\"relu\")\n",
    "fc3 = mx.symbol.FullyConnected(data = act2, name='fc3', num_hidden=10)\n",
    "# Use the new operator we just defined instead of the standard softmax operator.\n",
    "mlp = numpy_softmax(data=fc3, name = 'softmax')\n",
    "\n",
    "model = mx.model.FeedForward(ctx = mx.gpu(0), symbol = mlp, num_epoch = 2,\n",
    "                             learning_rate = 0.1, momentum = 0.9, wd = 0.00001)\n",
    "model.fit(X=train_iter, eval_data=test_iter,\n",
    "          batch_end_callback = mx.callback.Speedometer(100, 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}