4_Dynamic_Programming.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dynamic Programming\n",
    "\n",
    "Using dynamic programming we can solve the RL problem when all environment dynamics are known and we have a finite state set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any environment that our agent can explore needs to be described using a subclass of `Environment`. This allows us to implement a single DP algorithm once for all problems. We can then solve the individual problems by describing them using the methods from `Environment`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Environment:\n",
    "    def get_states(self):\n",
    "        \"\"\"\n",
    "        Return the set of possible states\n",
    "        \"\"\"\n",
    "        pass\n",
    "    \n",
    "    def get_possible_actions(self, state):\n",
    "        \"\"\"\n",
    "        Returns the actions that can be taken from the given state\n",
    "        \"\"\"\n",
    "        pass\n",
    "    \n",
    "    def get_state_actions(self, state, action):\n",
    "        \"\"\"\n",
    "        Return a list of (probability, new_state, reward) tuples for the\n",
    "        given state and action\n",
    "        \"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DPAgent` can find a perfect policy given an `Environment`. It works using value iteration. Initially, a random policy is generated. After that, each time `agent.learn()` is executed, we perform:\n",
    "\n",
    "* policy evaluation: The value for each state is updated, by using the values of neighbouring states\n",
    "* policy improvement: For each state we choose all actions that maximize the expected value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from random import sample\n",
    "from operator import itemgetter\n",
    "\n",
    "class DPAgent:\n",
    "    def __init__(self, env, discount_factor=1):\n",
    "        self.env = env\n",
    "        self.discount_factor = discount_factor\n",
    "        \n",
    "        states = self.states = env.get_states()\n",
    "        self.values = { state: 0 for state in states }\n",
    "        self.policy = { state: self._random_action_init(state) for state in states }\n",
    "    \n",
    "    def _random_action_init(self, state):\n",
    "        possible_actions = self.env.get_possible_actions(state)\n",
    "        \n",
    "        if len(possible_actions) == 0:\n",
    "            return set()\n",
    "        else:\n",
    "            return set(sample(possible_actions, 1))\n",
    "        \n",
    "    def learn(self, evaluate=1):\n",
    "        for _ in range(evaluate):\n",
    "            self._evaluate_policy()\n",
    "            \n",
    "        self._improve_policy()\n",
    "        \n",
    "    def _evaluate_policy(self):\n",
    "        values = self.values.copy()\n",
    "        \n",
    "        for state in self.states:\n",
    "            value = self.values[state]\n",
    "            actions = self.policy[state]\n",
    "            change = 0.\n",
    "            \n",
    "            for action in actions:\n",
    "                for prob, new_state, reward in self.env.get_state_actions(state, action):\n",
    "                    next_value = self.values[new_state]\n",
    "                    change += prob * (reward + self.discount_factor * next_value)\n",
    "            \n",
    "            values[state] = change / len(actions)\n",
    "\n",
    "        self.values = values\n",
    "    \n",
    "    def _improve_policy(self):\n",
    "        for state in self.states:\n",
    "            action = self.policy[state]\n",
    "            possible_actions = {}\n",
    "\n",
    "            for possible_action in self.env.get_possible_actions(state):\n",
    "                action_value = 0\n",
    "                \n",
    "                for prob, new_state, reward in self.env.get_state_actions(state, possible_action):\n",
    "                    value = self.values[new_state]\n",
    "                    action_value += prob * (reward + self.discount_factor * value)\n",
    "                \n",
    "                possible_actions[possible_action] = action_value\n",
    "            \n",
    "            max_value = max(possible_actions.values())            \n",
    "            self.policy[state] = set([action for action, value in possible_actions.items() if value == max_value])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GridWorld\n",
    "\n",
    "GridWorld is on of the standard examples for dynamic programming. Our agent can move in a 2d matrix. The rewards are defined in a matrix of the same size. There environment is totally deterministic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from itertools import product\n",
    "import numpy as np\n",
    "\n",
    "class GridWorld(Environment):\n",
    "    def __init__(self, reward_matrix):\n",
    "        self.reward_matrix = reward_matrix\n",
    "        \n",
    "        n, m = reward_matrix.shape\n",
    "        self.states = list(product(range(n), range(m)))\n",
    "        \n",
    "        self.max_down = n - 1\n",
    "        self.max_right = m - 1\n",
    "        \n",
    "        self._init_actions()\n",
    "        \n",
    "    def _init_actions(self):\n",
    "        self.UP = \"U\"\n",
    "        self.DOWN = \"D\"\n",
    "        self.LEFT = \"L\"\n",
    "        self.RIGHT = \"R\"\n",
    "        \n",
    "        self.actions = [self.UP, self.DOWN, self.LEFT, self.RIGHT]\n",
    "        \n",
    "    def get_states(self):\n",
    "        return self.states\n",
    "    \n",
    "    def get_possible_actions(self, state):\n",
    "        i, j = state\n",
    "        \n",
    "        actions = []\n",
    "        \n",
    "        if i > 0:\n",
    "            actions.append(self.UP)\n",
    "            \n",
    "        if i < self.max_down:\n",
    "            actions.append(self.DOWN)\n",
    "        \n",
    "        if j > 0:\n",
    "            actions.append(self.LEFT)\n",
    "        \n",
    "        if j < self.max_right:\n",
    "            actions.append(self.RIGHT)\n",
    "            \n",
    "        return actions\n",
    "    \n",
    "    def get_state_actions(self, state, action):\n",
    "        reward = self.reward_matrix[state]\n",
    "        new_state = self._get_next_state(state, action)\n",
    "        \n",
    "        return [(1, new_state, reward)]\n",
    "    \n",
    "    def _get_next_state(self, state, action):\n",
    "        i, j = state        \n",
    "        \n",
    "        if action == self.UP:\n",
    "            i -= 1\n",
    "            \n",
    "        if action == self.DOWN:\n",
    "            i += 1\n",
    "            \n",
    "        if action == self.LEFT:\n",
    "            j -= 1\n",
    "            \n",
    "        if action == self.RIGHT:\n",
    "            j += 1\n",
    "            \n",
    "        i = max(0, min(i, self.max_down))\n",
    "        j = max(0, min(j, self.max_right))\n",
    "        \n",
    "        return i, j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some helper functions for visualizing the policy/values/rewards of GridWorld."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def draw_policy(agent, n, m):\n",
    "    lookup = {\n",
    "        \"U\": u\"↑\",\n",
    "        \"D\": u\"↓\",\n",
    "        \"L\": u\"←\",\n",
    "        \"R\": u\"→\"\n",
    "    }\n",
    "    \n",
    "    vals = sorted(agent.policy.items())\n",
    "    actions = [\"\".join([lookup[act] for act in val]) for _, val in vals]\n",
    "    \n",
    "    print \"Policy\"\n",
    "    draw_matrix(np.array(actions).reshape((n, m)))\n",
    "    print\n",
    "\n",
    "def draw_values(agent, n, m):\n",
    "    vals = sorted(agent.values.items())\n",
    "    vals = [\"%.2f\" % val for _, val in vals]\n",
    "    \n",
    "    print \"Values\"\n",
    "    draw_matrix(np.array(vals).reshape((n, m)))\n",
    "    print\n",
    "    \n",
    "from math import ceil, floor\n",
    "\n",
    "def lrjust(s, size):\n",
    "    missing = size - len(s)\n",
    "    \n",
    "    if missing > 0:\n",
    "        left = int(ceil(missing / 2.))\n",
    "        right = int(floor(missing / 2.))\n",
    "        s = \" \" * left + s + \" \" * right\n",
    "    \n",
    "    return s\n",
    "\n",
    "def draw_matrix(data):    \n",
    "    result = \"\"\n",
    "    rows = []\n",
    "    \n",
    "    for row in data:\n",
    "        row_result = []\n",
    "        \n",
    "        for cell in row:\n",
    "            row_result.append(lrjust(\"%s\" % cell, 6))\n",
    "            \n",
    "        rows.append(\"|\".join(row_result))\n",
    "        \n",
    "    print (\"\\n\" + \"-\" * len(rows[0]) + \"\\n\").join(rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test cases\n",
    "\n",
    "In this simple test there's only one cell with a reward, so what we're really looking for is the shortest path to that cell.\n",
    "\n",
    "If a cell of the policy matrix contains several action, then this means that we can take any of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward\n",
      "   0  |   0  |   0  |   0  \n",
      "---------------------------\n",
      "   0  |   0  |   1  |   0  \n",
      "---------------------------\n",
      "   0  |   0  |   0  |   0  \n",
      "\n",
      "Policy\n",
      "  →↓  |  →↓  |   ↓  |  ↓←  \n",
      "---------------------------\n",
      "   →  |   →  | →↑↓← |   ←  \n",
      "---------------------------\n",
      "  →↑  |  →↑  |   ↑  |  ↑←  \n",
      "\n",
      "Values\n",
      " 0.17 | 0.33 | 0.67 | 0.33 \n",
      "---------------------------\n",
      " 0.33 | 0.67 | 1.33 | 0.67 \n",
      "---------------------------\n",
      " 0.17 | 0.33 | 0.67 | 0.33 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "R = np.array([\n",
    "        [0, 0, 0, 0],\n",
    "        [0, 0, 1, 0],\n",
    "        [0, 0, 0, 0]\n",
    "    ])\n",
    "\n",
    "print \"Reward\"\n",
    "draw_matrix(R)\n",
    "print\n",
    "\n",
    "grid_world = GridWorld(R)\n",
    "agent = DPAgent(grid_world, discount_factor=0.5)\n",
    "\n",
    "for k in range(10):\n",
    "    agent.learn()\n",
    "    \n",
    "draw_policy(agent, 3, 4)\n",
    "draw_values(agent, 3, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this new test case there's a single path without penalties to a reward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward\n",
      "   0  |   0  |   0  |   0  \n",
      "---------------------------\n",
      "   0  |  -1  |  -1  |  -1  \n",
      "---------------------------\n",
      "   0  |   0  |   0  |   0  \n",
      "---------------------------\n",
      "  -1  |  -1  |  -1  |   0  \n",
      "---------------------------\n",
      "   1  |   0  |   0  |   0  \n",
      "\n",
      "Policy\n",
      "   ↓  |   ←  |   ←  |   ←  \n",
      "---------------------------\n",
      "   ↓  |   ↓  |   ↓  |   ↓  \n",
      "---------------------------\n",
      "   →  |   →  |   →  |   ↓  \n",
      "---------------------------\n",
      "   ↓  |   ↓  |   ↓  |   ↓  \n",
      "---------------------------\n",
      "   →  |   ←  |   ←  |   ←  \n",
      "\n",
      "Values\n",
      " 0.00 | 0.00 | 0.00 | 0.00 \n",
      "---------------------------\n",
      " 0.00 | -0.99| -0.99| -0.98\n",
      "---------------------------\n",
      " 0.01 | 0.01 | 0.02 | 0.04 \n",
      "---------------------------\n",
      " -0.33| -0.67| -0.83| 0.08 \n",
      "---------------------------\n",
      " 1.33 | 0.67 | 0.33 | 0.17 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "R2 = np.array([\n",
    "        [0, 0, 0, 0],\n",
    "        [0, -1, -1, -1],\n",
    "        [0, 0, 0, 0],\n",
    "        [-1, -1, -1, 0],\n",
    "        [1, 0, 0, 0]\n",
    "    ])\n",
    "\n",
    "print \"Reward\"\n",
    "draw_matrix(R2)\n",
    "print\n",
    "\n",
    "grid_world = GridWorld(R2)\n",
    "agent = DPAgent(grid_world, discount_factor=0.5)\n",
    "\n",
    "for k in range(50):\n",
    "    agent.learn()\n",
    "    \n",
    "draw_policy(agent, 5, 4)\n",
    "draw_values(agent, 5, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Jack's car rental problem\n",
    "\n",
    "There are two locations for car rentals. At each cars can be rented and returned. The cars don't have to be returned at the location where they were rented.\n",
    "\n",
    "We want to move cars between the two locations, so that we maximize the number of cars that can be rented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from scipy.stats import poisson\n",
    "from itertools import product\n",
    "from collections import defaultdict\n",
    "\n",
    "class CarRental(Environment):\n",
    "    def __init__(self, max_cars, rent_profit, move_cost, average_requests, average_returns, max_lambda):\n",
    "        self.max_cars = max_cars\n",
    "        self.rent_profit = rent_profit\n",
    "        self.move_cost = move_cost\n",
    "        self.average_requests = average_requests\n",
    "        self.average_returns = average_returns\n",
    "        self.max_lambda = max_lambda\n",
    "        \n",
    "    def get_states(self):\n",
    "        car_range = range(0, self.max_cars + 1)\n",
    "        return list(product(car_range, car_range))\n",
    "    \n",
    "    def get_possible_actions(self, state):\n",
    "        \"\"\"\n",
    "        Returns the actions that can be taken from the given state\n",
    "        \"\"\"\n",
    "        cars_a, cars_b = state\n",
    "        return range(-cars_b, cars_a + 1)\n",
    "    \n",
    "    def get_state_actions(self, state, action):\n",
    "        result = defaultdict(int)\n",
    "\n",
    "        cars_a, cars_b = state        \n",
    "        cars_a = max(min(cars_a - action, self.max_cars), 0)\n",
    "        cars_b = max(min(cars_b + action, self.max_cars), 0)\n",
    "                        \n",
    "        for requested_a in range(0, self.max_lambda):\n",
    "            prob1 = poisson.pmf(requested_a, self.average_requests[0])\n",
    "            \n",
    "            for requested_b in range(0, self.max_lambda):\n",
    "                prob2 = poisson.pmf(requested_b, self.average_requests[1])\n",
    "                \n",
    "                rented_a = min(requested_a, cars_a)\n",
    "                rented_b = min(requested_b, cars_b)\n",
    "                reward = (rented_a + rented_b) * self.rent_profit\n",
    "                \n",
    "                for returned_a in range(0, self.max_lambda):\n",
    "                    prob3 = poisson.pmf(returned_a, self.average_returns[0])\n",
    "                    \n",
    "                    for returned_b in range(0, self.max_lambda):\n",
    "                        prob4 = poisson.pmf(returned_b, self.average_returns[1])\n",
    "                        prob = prob1 * prob2 * prob3 * prob4\n",
    "                        \n",
    "                        if cars_a < rented_a or cars_b < rented_b:\n",
    "                            print \"problem\"\n",
    "                            print cars_a, cars_b\n",
    "                            print rented_a, rented_b\n",
    "                            print requested_a, requested_b, returned_a, returned_b\n",
    "                            return []\n",
    "                        \n",
    "                        new_cars_a = min(cars_a + returned_a - rented_a, self.max_cars)\n",
    "                        new_cars_b = min(cars_b + returned_b - rented_b, self.max_cars)\n",
    "                        new_state = (new_cars_a, new_cars_b)\n",
    "                        \n",
    "                        result[(new_state, reward)] += prob\n",
    "                        \n",
    "        return [(prob, new_state, reward) for (new_state, reward), prob in result.items()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def visualize_car_rental_policy(agent):\n",
    "    policy = agent.policy\n",
    "    max_n = agent.env.max_cars\n",
    "    \n",
    "    result = \"\"\n",
    "    \n",
    "    for i in reversed(range(max_n + 1)):\n",
    "        result += \" %d |\" % i\n",
    "        for j in range(max_n + 1):\n",
    "            result += \"%+4s\" % str(list(policy[(i, j)])[0])\n",
    "            \n",
    "        result += \"\\n\"\n",
    "        \n",
    "    result += \" \" * 4\n",
    "    for j in range(max_n + 1):\n",
    "        result += \"-\" * 4\n",
    "    \n",
    "    result += \"\\n\"\n",
    "    result += \" \" * 4\n",
    "    for j in range(max_n + 1):\n",
    "        result += \"%+4s\" % str(j)\n",
    "        \n",
    "\n",
    "    print \"Car Rental Policy\\n(y-axis location 1, x-axis location 2)\\n\"\n",
    "    print result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, because of the poisson distributions, this problem takes pretty long to compute. Because of this, we use fewer cars than in the Sutton & Barto book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "env = CarRental(\n",
    "    max_cars=3,\n",
    "    rent_profit=10,\n",
    "    move_cost=2,\n",
    "    average_requests=[1, 1],\n",
    "    average_returns=[0, 2],\n",
    "    max_lambda=5\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Car Rental Policy\n",
      "(y-axis location 1, x-axis location 2)\n",
      "\n",
      " 3 |   1   1   0   0\n",
      " 2 |   1   0   0  -1\n",
      " 1 |   0   0  -1  -1\n",
      " 0 |   0  -1  -1  -2\n",
      "    ----------------\n",
      "       0   1   2   3\n"
     ]
    }
   ],
   "source": [
    "car_rental_agent = DPAgent(env, discount_factor=0.8)\n",
    "\n",
    "for _ in range(5):\n",
    "    car_rental_agent.learn()\n",
    "    \n",
    "visualize_car_rental_policy(car_rental_agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gambler's problem\n",
    "\n",
    "A gambler can put a chosen amount of money on a bet with a winning probability of `p_win`. If the gambler wins, he gains as much money as his stake, otherwise he loses the stake.\n",
    "\n",
    "The gambler wins the entire game if he manages to get to $100. We want to find an optimal policy to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class GamblerProblem(Environment):\n",
    "    def __init__(self, goal, p_win):\n",
    "        self.goal = goal\n",
    "        self.p_win = p_win\n",
    "    \n",
    "    def get_states(self):\n",
    "        return range(self.goal + 1)\n",
    "    \n",
    "    def get_possible_actions(self, capital):\n",
    "        return range(min(capital, self.goal - capital) + 1)\n",
    "    \n",
    "    def get_state_actions(self, capital, stake):\n",
    "        if capital == 0:\n",
    "            return [(1, 0, 0)]\n",
    "        \n",
    "        if capital == self.goal:\n",
    "            return [(1, 0, 100)]\n",
    "        \n",
    "        if capital + stake == self.goal:\n",
    "            possible_reward = 1\n",
    "        else:\n",
    "            possible_reward = 0\n",
    "        \n",
    "        return [\n",
    "            (self.p_win, capital + stake, possible_reward),\n",
    "            (1 - self.p_win, capital - stake, 0)\n",
    "        ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env = GamblerProblem(goal=100, p_win=0.4)\n",
    "\n",
    "gambling_agent = DPAgent(env, discount_factor=1)\n",
    "\n",
    "gambling_agent.learn(evaluate=10)\n",
    "gambling_agent.learn(evaluate=100)\n",
    "gambling_agent.learn(evaluate=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: {0},\n",
       " 1: {0, 1},\n",
       " 2: {0, 2},\n",
       " 3: {0, 3},\n",
       " 4: {0, 4},\n",
       " 5: {4},\n",
       " 6: {5},\n",
       " 7: {0, 7},\n",
       " 8: {1},\n",
       " 9: {0, 9},\n",
       " 10: {3, 10},\n",
       " 11: {0, 2, 11},\n",
       " 12: {12},\n",
       " 13: {0, 13},\n",
       " 14: {0, 14},\n",
       " 15: {15},\n",
       " 16: {16},\n",
       " 17: {8},\n",
       " 18: {0, 18},\n",
       " 19: {6},\n",
       " 20: {20},\n",
       " 21: {0, 21},\n",
       " 22: {22},\n",
       " 23: {2, 23},\n",
       " 24: {0, 24},\n",
       " 25: {0, 25},\n",
       " 26: {0, 1, 26},\n",
       " 27: {0, 2, 27},\n",
       " 28: {3},\n",
       " 29: {4},\n",
       " 30: {5},\n",
       " 31: {31},\n",
       " 32: {7},\n",
       " 33: {8},\n",
       " 34: {16},\n",
       " 35: {0, 10, 35},\n",
       " 36: {0, 36},\n",
       " 37: {13},\n",
       " 38: {38},\n",
       " 39: {0, 39},\n",
       " 40: {0, 40},\n",
       " 41: {0, 41},\n",
       " 42: {8},\n",
       " 43: {7},\n",
       " 44: {6, 44},\n",
       " 45: {0, 5, 45},\n",
       " 46: {0, 4, 46},\n",
       " 47: {3},\n",
       " 48: {0, 48},\n",
       " 49: {1},\n",
       " 50: {0, 50},\n",
       " 51: {0, 1},\n",
       " 52: {0, 2},\n",
       " 53: {0, 3},\n",
       " 54: {0, 4},\n",
       " 55: {5},\n",
       " 56: {6},\n",
       " 57: {7},\n",
       " 58: {8},\n",
       " 59: {41},\n",
       " 60: {10},\n",
       " 61: {11},\n",
       " 62: {12, 38},\n",
       " 63: {0, 12, 13},\n",
       " 64: {14},\n",
       " 65: {0, 15},\n",
       " 66: {16},\n",
       " 67: {0, 8},\n",
       " 68: {32},\n",
       " 69: {0, 6},\n",
       " 70: {0, 5, 20, 30},\n",
       " 71: {0, 4},\n",
       " 72: {3},\n",
       " 73: {0, 2},\n",
       " 74: {1},\n",
       " 75: {0, 25},\n",
       " 76: {0, 1, 24},\n",
       " 77: {0, 23},\n",
       " 78: {3},\n",
       " 79: {21},\n",
       " 80: {0, 5},\n",
       " 81: {0, 6},\n",
       " 82: {7},\n",
       " 83: {8},\n",
       " 84: {9, 16},\n",
       " 85: {0, 10, 15},\n",
       " 86: {14},\n",
       " 87: {0, 12},\n",
       " 88: {0, 12},\n",
       " 89: {0, 11},\n",
       " 90: {0, 10},\n",
       " 91: {0, 9},\n",
       " 92: {0, 1, 8},\n",
       " 93: {0, 7},\n",
       " 94: {6},\n",
       " 95: {0, 5},\n",
       " 96: {0, 4},\n",
       " 97: {0, 2, 3},\n",
       " 98: {0, 2},\n",
       " 99: {0, 1},\n",
       " 100: {0}}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gambling_agent.policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visualisation is a bit different than the one in the Sutton & Barto book. This is because it's not totally clear how to visualize the policy. For many states, there are multiple optimal actions. In the book, the visualisation chooses one action per state.\n",
    "\n",
    "This first visualisation only shows the action with the maximum stake:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEKCAYAAAD6q1UVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucXHV9//HXZ5NssmSzYVNXLgU3BCogiGwwFBXrooTS\n2BqqlrAqotALtT+0FxSw7S/RPmqF3n7+RCr8TCFYWUIVCdpo0pQsVi2yleVSSAJesiKXZCEhkpAE\nQj6/P86ZcDI7M3uZM3Nu7+fjsY+dy5lzPt9zZj5z5nO+53vM3RERkWJpSToAERFpPiV/EZECUvIX\nESkgJX8RkQJS8hcRKSAlfxGRAlLyT5CZLTWzL4e3jzazX5iZxbyMVjN72MwOi2Fe683s4pjiutHM\nPh3HvJrFzN5nZt9OOo5GMrOLzOw/k44jKq71bmavNrNHzGxaHHFlnZJ/DMxss5m9ECbvp8LEdsg4\nX+4A7v64u3d4/Cde/D5wt7tviXm+45bGhDIZ7n6Lu5+bdBwTMckv2cRO/jGzbjPbb2YHclNc693d\ntwJ3AX9Q77zyQMk/Hg680907gPnAG4G/SDakAy4FvpxwDEaCCUUypfReifUXcMQtKPkDSv5xMgB3\nfwr4FnAygJkdYWarzOxZM3vUzH634ovL9njMrNPM/tnMnghfe3v4+ENm9s7I66aa2YiZvaHCPI8G\njgF+EHmsw8xuNrOtZvZTM/vzyHMXmdl/mtnfmtk2M/uxmY3a4zKzaWFMJ0Ue6zKzXWb2S2XTngD8\nE/AmM3vezLZFnp5jZt8MfzH9l5kdE32dma0Nl7PBzH6n4lrnQDnqr8zse+EyVpnZL5nZv5jZDjP7\ngZm9JjL9/zGzn4XPDZrZmZHn/s3M/i5y/1Yz+1J0/USe229mf2hmj4Xz+rSZzTOz75vZdjPrN7Op\nlV4bef288PaNZvYFM1sdtuE7ZnZ4GOv2sFwxahtH5vWPZrYljOMBM3udmf0e8H7gE+E6XhVOe4WZ\n/Sh87H/M7Lwa8/3bMJZZ4f2Lw1ieNbNvRddrhdfeFv4S3m5mA2b2ushzM8zs7y341fxcuIwZwN3h\nJM+F8f1qhfX+ZjO7N5zvD8zsTZHn1ofb4bvh679tZnMiYf0AmBd+NorN3fVX5x/wU+Dt4e2jgf8B\nloX3vwN8HpgGvAHYCvSGzy0Fbg5vdwMvAy3h/X8D+oEOYArw1vDxjwO3Rpa9GHigSlyLgIfKHrsZ\n+DpwSLjMTcCHw+cuAvYCFxN8mV0KPBF57Xrg4vD2tcDfRJ77KLCqShwXAd8pe+xGYAQ4jWAn5F+A\nW8LnDgF+BnwwjKO03k6oMv/1wKPAXGAW8HB4/6xw3iuA5ZHp3wccGj73J8BTQGv43GHA00AvQeL8\nEXBIpXYA+8N1ORM4EdgD/Ee4XktxXFhjHbwMzIusj63AqUBrOJ/NYQwG/BVwV5X2nwMMArPC+8cD\nh0Xm++my6d8Tef53gJ2R+xcRvGcN+H8EOzLTI++1R4HXhuvuk8D3anwuPhRuy2nAPwBDkee+QFCC\nOTxc1hnhdKXPgVV6/wCdwLZwG7YAF4T3OyPvhceAY4Hp4f3PlMX1APCbSeeNpP8SDyAPfwTJ/xfh\nm/CnBMl+OnAU8FIpeYTTfgb45/B2xeQPHAHsAzoqLOsI4HmgPbz/r8DlVeJ6H/D9yP0WguR+fOSx\n3y8llfBD9mjkuTaCBPfq8H40+f8qMByZdhB4b5U4qiX/GyL3fwN4JLx9PsFxiuj0XwT+ssr81wNX\nRe7/HfBvkfu/CdxXY/ttA14fuf/bBF8+W4E3VWtHuG7OiNz/b+DjZXH8Q411sJ+Dk//1kef+F/Bw\n5P7JwLYq8Z8FbAy3iZU9Nyr5V3j9EPBbkTjvAW4N31tTI9OtJtxRiLyfdgFHj+MzcmjY3lkEyf4F\n4OQK0x20E1S+7oAPAPeUveb7wAcj74VPRp77Q2B12fTfBT4wkc94Hv9U9onPYnef4+7HuPtl7r4X\nOJLgA/tCZLph4JfHmNdR4et+Uf6EB2Wl7wLvMbPZBEnzK1Xms53gw1byKmAqQWKrFs/TkWXtDm+2\nV4jjB8ALZvY2MzueYE/rzjHaVe7pyO0XIsvpBs4IS0/bzGw7wRfZ4TXmFT2gvbvC/QNtMLPLw9LF\n9nDeHQTrpuSbBL+2Nrn7f43Rhq3jXe44jLsNUe6+nuCX2BeALWb2RTOrulwz+6CZDUXafxIHt/84\n4F3Ap9x9X+TxbuBzpe0CPEtQnx/1fjazFjP7bFheeo5gp8jD5byKYOfoJ9VirOFIgvdsVNX3MAe/\nr0pmAc9NYtm5ouQfn0oHqJ4kqGvPjDz2GuCJMeb1ePi6jirP3wxcSPCT/fvhF0IlDwLH2Cs9J54h\n+CXSHZmmexzxVLMijONC4Kvu/mKV6XyC830cGAi/TOe4e6cHPaH+aJJxHmBmbyUonb03nG8nwa+2\n6Pb7DPAIcISZXVDvMkO7CEogpThqfZFNmLtf6+5vBF5HUPb5eOmp6HRhjf4G4COR9j/Mwe1/BPgw\n8G0ze23k8Z8Bf1C2Xdrd/Z4KIb0P+C2CcuihBCU5C/+eISiRHVupKWM09clwXlHj+UwBYGZTCL7c\nHhjP9Hmm5N9A7v5zgp+kf2Nm083sFOASqve+KR00fpqg1nqdmR1qwUHdt0am+zpBr6KPEnwRVFv+\nEwQ169PD+/uB24C/NrN2M+smqHlPtjfQVwhKJO+vFQfBHuxRNv7+1d8EXmtmHwjbPs3M3mjBweN6\ntRN8AT5rwTkQ/5vIryMz+zWCMsOFBDXrz5vZETEs9wHgJDM7xcymE5T8JvqlWLEHTLhuTg8PLu8m\nSKz7w6e3APMik88Mn3sm3Dv/MGHnhCh3X0lQ019XOigNXA98snTg1sxmm9l7q8Q6i6DEuD3c+fkb\nXunW7ATlqH+woENEi5mdEb4/RsL4Kn0xQFB6+hUzu8DMppjZEoLjLd+oMn2504Gfuvvj45w+t5T8\n41HrQ9xH0OPmSeBrBHXr9eOYz4UEdf+NBB/gjx2YyH0PcHs439vHiO16ggOnJR8l+Cn8E4IDe//i\n7jfWeL1XuU34ARoKbvp3a8zjLoK9y6fNbGuN6Urz3UlwEPMCgvX2JPBZggOhY8U4ljXh36MEpYgX\nCH5pEPZoWQH8kbs/HbbpSwSJajzLrRqHuz8GfJrgQO6jwGTOe6g2/w6Cg7OlY07PAH8bPrec4Etn\nm5nd7u4bCA6+3kNQHjmJoIxYKeabSzGb2Wvc/Q6C7XBrWMp5EKjW//5mgl8KTxB0gPh+2fOXAw8R\nHCt6NpxvS1hq/Gvge2HMp5fFtI3gGM7lYTsvJ+hmvX2MdVTyfoLjR4Vn4QGQ+mYS1J6/RLAHsZ+g\nt8ijwEqCssJm4Hx331H3wgQAM/sL4LXu/sExpmsF7gPe4Q040cuCbpBPuvv/jnveInEysy5gAOip\nUaIsjLiS/00EvTNuDH96ziT4yfisu19jZlcQdMW6su6FCWG/5R8SdCOstcfd6Di6Cfb8e9y9/CCc\niKRY3WWf8KDkW0ulA3ffF+7hLyb4CU34v+qJJDJ+Fpwk9jOC7mtJJv5PE/xsv0aJXyR76t7zt+Cs\nwxsIegi8gaCv8x8TnBzUGZlum7vPqTwXERFppjgO+E4l6HnyBXefT9Cl7UomcDBMRESaa2oM8/g5\n8Li7/3d4/2sEyX+LmR3m7lvCPs0Ve3mYmb4UREQmwd0nPQBe3Xv+YQ+SxyMng7yDoFvfnQT9pCHo\nN72qxjwy+7d06dLEY1D8ycdRxPizHHse4q9XHHv+EPQd/0p4ksZPCM4OnALcZsHFP4YJxmsREZEU\niCX5u/sDwIIKT50dx/xFRCReOsO3Tr29vUmHUBfFn6wsx5/l2CH78dcrlpO86grAzJOOQUQka8wM\nT/KAr4iIZI+Sv4hIASn5i4gUkJK/iEgBKfmLiBSQkr+ISAEp+YuIFJCSv4hIASn5i4gUkJK/iEgB\nKfmLiBSQkr+ISAEp+YuIFJCSv4hIASn5i4gUkJK/iEgBKfmLiBSQkr+ISAEp+YuIFJCSv4hIASn5\ni4gU0NQ4ZmJmm4EdwH7gJXc/3cw6gZVAN7AZON/dd8SxPBERqU9ce/77gV5373H308PHrgTWufvx\nwF3AVTEtS6QpRkZGGBwcZGRkJOlQRGIXV/K3CvNaDKwIb68AzotpWSIN19+/ku7uE1i48FK6u0+g\nv39l0iGJxMrcvf6ZmP0EeA54Gbje3b9kZtvdvTMyzTZ3n1PhtR5HDCJxGRkZobv7BHbvXg+cAjxI\nW9tZDA9vpKurK+nwRAAwM9zdJvv6WGr+wFvc/Skz6wLWmtkmoDyjV83wy5YtO3C7t7eX3t7emMIS\nmbjNmzfT2jqX3btPCR85hWnTutm8ebOSvyRmYGCAgYGB2OYXy57/QTM0WwrsBH6X4DjAFjM7HFjv\n7idWmF57/pIqo/f8B5g+fTFDQ/dw4omj3sIiiah3z7/umr+ZHWJm7eHtmcA5wEPAncCHwskuAlbV\nuyyRZujq6mL58utoazuLGTPmAYtoaenmtNPOVO1fcqPuPX8zOwb4OkFZZyrwFXf/rJnNAW4DjgaG\nCbp6Plfh9drzl1TasGEDPT1vZu/eu1HtX9Im8Zq/u/8UOLXC49uAs+udv0gzjIyMsHnzZubOnXsg\nse/cuZMZM+axd69q/5I/OsNXCq9at865c+fy4oubgQfDKR/kpZeGmTt3bkKRisQn9gO+Ew5AZR9J\n0FjdOvv7V3LJJR9h2rRuXnppmOXLr6Ovb0nSYYskX/YRybKxunX29S3h7LPfPqokVKlMJJIlKvtI\noY2ntNPV1cWCBQsOJHmd/St5oLKPFN5ESjs6+1fSQmUfkTpVK+1UorN/JS+U/EUISjvjSd4Hl4mC\nPX/1AJIsUs1fZAKiZ/92dMynre0sli+/Tnv9kjmq+UthxNlDp9K81ANIminxsX1EsiDuHjrqASRZ\npz1/yb1G99BRDyBJgvb8RcZQ6qETJGaI9tDJwvxFGkHJX3Kv0WP0jJ7/AHv3/pj29vZY5i/SCEr+\nknuN7qGj8f8li1Tzl8JodG8cjf8vzaQzfEUqqJTox3si12Rp/H/JEpV9JHeS6nap8f8lS1T2kVxJ\nutulxv+XZlHZRyRiaGiIlpajqdTtshnJfyKDxIkkSclfcqO/fyUXX3wpe/a8SJIDrzX62IJIHJT8\nJRdGRka45JKPsGfP3cAGoBeYQ1vbdg28JlKBDvhKLhx8lu0SYBMzZ07hjjv6VXMXqUDJX3JhdE+b\np9i//xl6enoSjEokvWJL/mbWYmb3mdmd4f1OM1trZpvMbI2ZzY5rWSLlNM6+yMTE1tXTzP4EOA3o\ncPd3mdnVwLPufo2ZXQF0uvuVFV6nrp4SmyyNqZ+lWCV9UjGqp5kdBSwCvhR5eDGwIry9AjgvjmWJ\n1FI+zn5aafx/SVose/5m9q/AXwOzgT8L9/y3u3tnZJpt7j6nwmu15y+FkvSJaJIPiZ/kZWbvBLa4\n+/1m1ltj0qoZftmyZQdu9/b20ttbazYi2VbqmbR7t8YAkvEbGBhgYGAgtvnVvedvZp8BPgDsA9qA\nWcDXgTcCve6+xcwOB9a7+4kVXq89fykU7flLHBKv+bv7J939Ne4+D7gAuMvdLwS+AXwonOwiYFW9\nyxLJA/VMkjSIdWA3M3sbr9T85wC3AUcDw8D57v5chddoz18KSb19pB717vlrVE/JhCIkyiK0UeKT\neNlHpNGK0C2yCG2UdNGev6RaEQ6OFqGNEj/t+UuuHTxgG0S7ReZFEdoo6aPkL6lWhEsjFqGNkj5K\n/pJqRegWWYQ2Svqo5i+pU6nXSxF6whShjRIfdfWUXCldAL21NSiF6ALoIpUp+UtuqNeLyPipt4/k\nxtDQEC0tR6NeLyKNp+QvqdDfv5LFi5ewa9djqNeLSOPVPaSzSL1GRka45JKPsGfP3cAGoBeYQ1vb\ndvV6EWkQ7flL4g4+yWkJsImZM6dwxx39Otgr0iBK/pK40Sc5PcX+/c/Q09OTYFTpNTIywuDgICMj\nI0mHIhmm5C+J00lO46cB4CQu6uopqaGTnGpTV1iJSvwaviKTUSnRd3V1KYnVoGv/SpxU9pGmU+li\ncjQAnMRJZR9pKpUu6lMa/mLatG5eemlYw18UmMo+kim1zuJV8h9bX98Szj777To2InVT8pem6e9f\nycUXX8qePS8SlC6CPX+VLiZGx0YkDkr+0hQ6i1ckXXTAV5pCZ/E2h04Ak/FS8pem0Fm8jadeVDIR\ndff2MbPpwHeA1vBvlbt/0sw6gZVAN7AZON/dd1R4vXr7FIR6qjSOelEVTyou5mJmh7j7C2Y2Bfge\n8GfAu4Bn3f0aM7sC6HT3Kyu8Vsm/QHQWb2MMDg6ycOGl7NjxwwOPdXTMZ92661mwYEGCkUmjpKKr\np7u/EN6cTlBK2g4sBt4WPr4CGABGJX8pFvVUaYyDy2rqRSVji6Xmb2YtZjYEPA0MuPsjwGHuvgXA\n3Z8GXh3HskRkNA2OJxMV157/fqDHzDqANWbWC5TXcqrWdpYtW3bgdm9vL729vXGEJVIo1U4AU6kt\nHwYGBhgYGIhtfrEP72BmfwnsBi4Bet19i5kdDqx39xMrTK+av0iDlA6yt7YGZSEdZM+PxA/4mtmr\ngJfcfYeZtQFrgE8B5wDb3P1qHfAVaT71AMq3NBzwPQJYYWZGcAzhy+7+H+ExgNvM7GJgGDg/hmWJ\nyDhpCGippe7k7+4PAfMrPL4NOLve+YvI5KgHkNSiM3xFcko9gKQWjecvknPq7ZNPiR/wrZeSv4jI\nxNWb/FX2EREpICV/EZECUvIXKTCN/19cSv4iBaXx/4tNB3xFCkhn/2afDvhK06hEkB8HX1YT4BSm\nTDmS1atXa/sWhJK/jItKBPky+rKa17Bz54+57LLPafsWhMo+MiaVCPKpNOLnlClHsnPnj4F70PbN\nDpV9pOGGhoZoaTmaaImgNECYZFdf3xKGhzdy7bWXM2vWCWj7FouSv9TU37+SxYuXsGvXY7xSItAA\nYXnR1dXFokWL2LdvGG3fYonlSl6STyMjI1xyyUfYs+duYAPQC8yhrW27BgjLkdIAcJdcchbTpnXz\n0kvD2r4FoJq/VDU4OMjChZeyY8cPw0dGmDnzTG6//fOcc845icYm8dMAcNmShou5SA6NjIywffv2\nsvHgn2L//mfo6elJNjhpiK6uLiX9AlHyl1Gi133dt+9FWlt/jRkz5qkcIJIjKvvIQSp165wx422s\nWrWSnp4eJX6RlFBXT4lVpW6dra3H0NnZWTPx6+xfkWxR8pcDJtutU2f/imSPyj4ClJd7NgB/SLRb\nZ1/fknG8TmeHijSLyj5St5GREVavXs3Uqd0ECXwJsImZM6dwxx39VRM/VB4gTGeHiqSfevsUXKln\nz9Spv8zzz/+IiXbrPHiAsGDPX2eHiqRf3Xv+ZnaUmd1lZg+b2UNm9tHw8U4zW2tmm8xsjZnNrj9c\niVPpDN7du9fz/PMPAsuAM5g1q4e2trPG1a2zdHZoW9tZdHTMH/frRCRZddf8zexw4HB3v9/M2oEf\nAouBDwPPuvs1ZnYF0OnuV1Z4vWr+CRl9Bi+0t7+ea6+9nEWLFk0ogVc6O1RnjIo0TuI1f3d/2t3v\nD2/vJDhaeBTBF8CKcLIVwHn1LkviNXpM9wd5+eUnJ5z4IfgFsGDBggOvUw8gkXSLtbePmc0FBoCT\ngcfdvTPy3DZ3n1PhNdrzT1Cp5h8d0KvWAd7xUA8gkcZLzdg+Ycnnq8DH3H2nmZVn9KoZftmyZQdu\n9/b20tvbG1dYUkG0HNPXt4Szz357rOWZUg+g3btLPYCOoKXlVQwNDWlAuBxSea85BgYGGBgYiG+G\n7l73H8GXyLcJEn/psQ3AYeHtw4ENVV7r0jy33HKrt7XN8dmz53tb2xy/5ZZbY1/G1q1bva1tjsMD\nDrc6dDoc17DlSXKa8X6SysLcOem8HUvZx8xuBp5x9z+NPHY1sM3dr9YB33RoZjmmv38lF198KXv2\nvAj8V8OXJ82n8l6yEj/ga2ZvAd4PvN3MhszsPjM7F7gaWGhmm4B3AJ+td1lSn2aekNXXt4RVq1Yy\nc+avNGV50nw6wS/b6q75u/v3gClVnj673vlLPCqPz9/YE7J6enrYv//xpi2vyJKou48+wW+AvXt/\nTHt7e1OWL/XR8A4FUOp2ef75Vx0Yn78ZJ2TpBLDmSKpbbXT7zpgxD1hES0s3p512prr2ZoAGdsu5\nNIzPr94gjZOGuvuGDRvo6Xkze/fenVgMRZR4zV/SbbLj88ep/AQwicfoAfkgibr7zp07wz3/V2KY\nMuVIVq9eres7pJiSf45Ndnx+Sb9Sqeeyy/6e55/fSJLbd/SZ4tewc+ePueyyz+ns7jSrp59oHH+o\nn39DVO5rf6z6YufAwdvWHa52aPNZs05NbPuW+vu3t5/s0BaJ7QFva5vjW7dubXpMeUed/fy1559D\n9YzPL+k3uovlJ2hvP5bPf/6PGR7emMj27etbwvDwRq699nJmzToBdf9MPyX/nKleDhjf+PySfnEO\nyBenrq4uFi1axL59w6jMmH5K/jkSx/j8kn5p7kKb5tjkYOrqmSNxjs8v6ZfmLrTlsaU51qyqt6un\nkn+OpKHPt0i50rDhra1BuSqOYcNFyV/KNGJ8fpHJ0g5J46RmPH9Jh0aMzy8yWaOv7fBK7x+9N5Ol\n5J9DXV1d+mBJKowe/E29f9JCvX1EpGHU+ye9VPPPqKL0nihKO8vlrd2V2pO3NjZbvTV/De+QQUW5\ndF5R2lmuCO0uQhsbjTRcxrEe2vOfmKL0nihKO8sVod1FaGMzaEjnginKpfOK0s5yRWh3EdqYBUr+\nGdPe3s6ePT8h72OnVBq/Jo/tLFeEdldq44sv/pTt27dr/P8mUvLPkP7+lZx22pm0tHQCZ9DW9vrc\n9p4oai+RIrS7vI3Tpp3J/v3O+edfpfH/m0g1/4wYXScdYPr0xQwN3cOJJ56YdHgNU9QeIUVo98jI\nCENDQ5x3Xp/q/5Ogmn8BVL5cXy/Tpx/Lzp07GRkZYXBwMJc/mcsvAZnHtlZqU6ndQG7bC9DZ2VlW\n/z+ClpZXMTQ0lFR4xVFPV6E4/lBXz5pKXeJmzXp9xSskffGLNxSmy1weuwfWalMR2lt6/x58xbnj\nctPeRqLOrp5xJfDlwBbgwchjncBaYBOwBphd5bWNWzsZN9bl+g7+4LzyhZDHS+aNXhfZb2utNhWp\nvV/84g0+Y8ahDofkqr2NVm/yj6vscyPw62WPXQmsc/fjgbuAq2JaVmEMDQ3R0nI01S7XN3/+qYX5\nyVype+CUKUeyevXqzJZEanV5LFJ7588/lVWrVjJz5q9QhPdyatTzzRH9A7o5eM9/I3BYePtwYGOV\n1zXoezHbbrnl1jH3hipfpD2fP5mr/wrqyWx7J7bnX6T25vu9HBfSUPbxysl/W9nz26q8rjFrJsMq\nfxCOrfhBGM+XRF6U6sXt7SdXPP6RxfaW2tTRMTqpF7G9RXkvxyFLyf/ZKq/zpUuXHvhbv359Q1ZU\nltx7770+e/b88M3vDlt95szX+po1aypOv2bNGp858w2R6d07Onr83nvvbXLkE7d161a/9957x/3h\n3rp1q990000+a1bPuNdPmlRqb611ULm92d6+tdpb6b3c3n6y33TTTYX/Ali/fv1BuTLNyX9DWdln\nQ5XXNWpdZdZED/Zl9eDgZHuzZLVEEE97871981juapQ0Jf+5wEOR+1cDV4S3rwA+W+V1jVo3mVbr\n53Ec0yet3oSWtRJBHO0tyvbNY7mrEVKR/IFbgCeBvcDPgA8TdPVcR9DVcy1waJXXNnL9ZNpkSiIT\nmT5Jo0tbEy9lZKncFUesRdq+WS/vNUMqkn9dASj5F1IcpYyslEOy9islDvFv3+yU95pFyV8yK45S\nRtrLIRPpuZU3cW3fon1xjle9yV8Du0mi4hjALM2DoA0ODrJw4aXs2PHD8JERZs48k9tv/zznnHNO\norE1QxzbZu3atbz73Z9g1677DzzW0TGfdeuuPzD+URHVO7Db1DiDkYlLc+Jqhq6urrrbHcc8GmFk\nZITt27dHxq4/BXiK/fufoaenJ9ngmiSObdPT08P+/Y/zyjo8ePz/NG77LNCongnq719Jd/cJLFx4\nqcYxz5nStj3//KvYt+9FWlt/Lbfj8zeaxv9vDJV9EqLrmOZXpW07Y8bbWLVqJT09Pdq+k6Tx/w+m\n8fwzavSgbbqOaSMkMf5/pQHMWluPobOzs3AJKk5dXV0a/z9GSv4J6O9fyeLFS9i16zHyfK3WpCVV\nVivCdXiTcvC6XQkcz65d+znvvD6Vfyaqnq5CcfxRsK6eRe7610xJnwOQ9i6oWabunwFSMp6/jMPo\nyzEuATYxc+YUbrrpnzjuuHlNLU/k8ZKIJbXGym+Gvr4lDA9vZN266xke3khf35KmLDeqkds3yfdO\nX98Sjf8fh3q+OeL4oyB7/rUuxzht2qymX64vj5cIjEp6zz9pjdy+aXjv6Ozf+vf8lfyboNblGGfM\nONRbW2c3NUkVJTEWtfTSyO2bpvdO0cs/9SZ/lX2aoNblGFetWklb27E0szzR6JJIWspJ1UovccaX\nlrZGNXL7pqmXWuXyT/Yvd9k09XxzxPFHzvf8x9o7SWJPqpHLTENJoJY440trWxu1fdO4p13k8f9R\n2Se9xtuzJ4nyRCOWmaaSQCVxxpf2tsa9fdPcS62o4/8r+afURMcjj47V3qxx2+NeThxj9DdSpfgm\ne4nAtLfVffT2rWd7j74eQbrG1i/i+P9K/ilUq2fPWHsiaS0ljEfa94bjLBGkva3l6nlfpbHcU0nR\negAp+adMrZ49Y70Js5ZQKkl7D5s4SwRpb2tJPe+rNJd7KsnKF1UclPxTpp7SQhZKCeOR9ssNVi4R\nTG5dp72t7vW9r0a/Nv2llCxd3rMe9SZ/dfWMWXt7O3v2/ITouC4vv/wkixYtGnNQryyOCVOpq2NX\nVxcLFixI7SBmXV1dLFq0iH37hpnoui5vb9rbCpXfV9Hx8KsZfT0CyML1CA4e/x/G297CqeebI44/\ncrTnXyoVjMArAAAIAUlEQVQDtLUd49DmbW0nT6q+moVSgnu2j0+4T3xdZ7m90bZOmzbLW1tn12xH\ntK3TprV7a+vsTLwnSyba3ixCZZ90GF1XXe/Tp3f4I488Mql5pb2UkIfjE+7jX9d5aO/WrVt9zZo1\nY7ajUltnzDjU16xZk8v2ZlW9yV9ln5iMPvOxl+nTj2Xnzp0TnlfaSwmjB6iDrF6PoHxdVztjN+mB\n4uJQeTz80e2odBZvFq9HoPH/x1DPN0ccf+Rgz79IPQzq6caadrXKOnnY83cfux15ey/nufsnaS/7\nAOcCG4FHgSsqPN+YNdMkWesKV496urGm3XiSe5aOx9RSrR15fS/n7QutJNXJn+BKYT8CuoFpwP3A\nCWXTNGrdNFzRziqM8wzZtBlvd8gsHI8Zj0rtyGK3zvFK+xnKk5H25H8G8K3I/SvL9/6zmvzzXP6o\nJi+lj0ry3LbxyvM6yGP5J+3J/z3ADZH7HwD+b9k0jVkzDZTn8sdY8lL6qCTPbRuvPK+DvJV/6k3+\nFsyjMczsPcCvu/vvh/c/AJzu7h+NTONLly498Jre3l56e3sbFlMcBgcHWbjwUnbs+OGBx9rbX8+1\n114+rpO5sm5kZITNmzczd+7c3LU1z20brzyvg7Vr1/Lud3+CXbvuP/BYR8d81q27ngULFiQY2dgG\nBgYYGBg4cP9Tn/oU7m6TnV+jk/8ZwDJ3Pze8fyXBt9XVkWm8kTE0wsjICN3dJ7B793qCLmQP0tZ2\nFsPDG3P3YRHJkzx9ds2sruTf6H7+g8BxZtZtZq3ABcCdDV5mw3V1dbF8+XW0tZ1FR8d82trOYvny\n6zL35hEpGn12X9HQPX8AMzsX+BzBF81yd/9s2fOZ2/MvyfPPY5E8y8Nnt949/4Yn/zEDyHDyFxFJ\nStrLPiIikkJK/iIiBaTkLyJSQEr+IiIFpOQvIlJASv4iIgWk5C8iUkBK/iIiBaTkLyJSQEr+IiIF\npOQvIlJASv4iIgWk5C8iUkBK/iIiBaTkLyJSQEr+IiIFpOQvIlJASv4iIgWk5C8iUkBK/iIiBaTk\nLyJSQEr+IiIFVFfyN7P3mtn/mNnLZja/7LmrzOwxM9tgZufUF6aIiMSp3j3/h4DfBu6OPmhmJwLn\nAycCvwFcZ2ZW57JSaWBgIOkQ6qL4k5Xl+LMcO2Q//nrVlfzdfZO7PwaUJ/bFwK3uvs/dNwOPAafX\ns6y0yvobSPEnK8vxZzl2yH789WpUzf+Xgccj958IHxMRkRSYOtYEZvbvwGHRhwAH/tzdv9GowERE\npHHM3eufidl64M/c/b7w/pWAu/vV4f1vA0vd/QcVXlt/ACIiBeTukz6WOuae/wREg7gT+IqZ/SNB\nuec44N5KL6oneBERmZx6u3qeZ2aPA2cA3zSzbwG4+yPAbcAjwGrgIx7HTwwREYlFLGUfERHJlsTO\n8DWza8ITwO43s6+ZWUfkuUycIGZm55rZRjN71MyuSDqeWszsKDO7y8weNrOHzOyj4eOdZrbWzDaZ\n2Rozm510rLWYWYuZ3Wdmd4b3MxO/mc02s38N39cPm9mvZiz+q8K4HzSzr5hZa5rjN7PlZrbFzB6M\nPFY13rTlnSrxx5Y3kxzeYS1wkrufSnAewFUAZvY6MnCCmJm1ANcCvw6cBPSZ2QnJRlXTPuBP3f0k\n4E3AH4XxXgmsc/fjgbsIt0OKfYygnFiSpfg/B6x29xOBNwAbyUj8ZtYN/B7Q4+6nEBwv7CPd8d9I\n8PmMqhhvSvNOpfhjy5uJJX93X+fu+8O79wBHhbffRTZOEDsdeMzdh939JeBWgpPbUsndn3b3+8Pb\nO4ENBOt8MbAinGwFcF4yEY7NzI4CFgFfijycifjDPbS3uvuNAOH7ewcZiR/4BfAiMNPMpgJtBOfv\npDZ+d/8usL3s4Wrxpi7vVIo/zryZloHdLiY4MAzZOUGsPM6fk844RzGzucCpBG+ew9x9CwRfEMCr\nk4tsTP8IfJzgPJOSrMR/DPCMmd0Ylq1uMLNDyEj87r4d+HvgZwSfyR3uvo6MxB/x6irxZiXvRNWV\nNxua/M3s38P6YOnvofD/b0Wm+XPgJXfvb2QsEjCzduCrwMfCXwDlR/xT2QPAzN4JbAl/vdT6OZvK\n+AnKJPOBL7j7fGAXQQkiK+t/HvAnQDdwJMEvgPeTkfhryFq8QDx5M85+/qO4+8Jaz5vZhwh+xr89\n8vATwNGR+0eFj6XNE8BrIvfTGucB4c/1rwJfdvdV4cNbzOwwd99iZocDW5OLsKa3AO8ys0UEJYdZ\nZvZl4OmMxP9z4HF3/+/w/tcIkn9W1v8bge+5+zYAM/s68GayE39JtXizkndiy5tJ9vY5l+An/Lvc\nfW/kqTuBC8KeBMdQ4wSxhA0Cx5lZt5m1AhcQxJ5m/ww84u6fizx2J/Ch8PZFwKryF6WBu3/S3V/j\n7vMI1vVd7n4h8A2yEf8W4HEze2340DuAh8nI+gc2AWeY2YzwQOI7CA68pz1+Y/QJqB8Kb0fjTWve\nOSj+WPOmuyfyR3BAYhi4L/y7LvLcVcCPCA5KnpNUjONow7kEH4rHgCuTjmeMWN8CvAzcDwyF6/xc\nYA6wLmzHWuDQpGMdR1veBtwZ3s5M/AQ9fAbDbXA7MDtj8X+c4AvrQYKDpdPSHD9wC/AksJfgWMWH\ngc5q8aYt71SJP7a8qZO8REQKKC29fUREpImU/EVECkjJX0SkgJT8RUQKSMlfRKSAlPxFRApIyV9E\npICU/EVECuj/AyanuGpE7/CNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x104f50d90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEKCAYAAAD+XoUoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFOW1x/HvETCKCG4PixBJEBdEXBFxZRQVRQFXrgoK\nkrjEjSgmgUQv5KpX1Ny4XFGDoiAqCMqmESUIo6IXRQFFBxBcCBAYRAdE2Qbm3D/eQibjDMv0Ur38\nPs/TD13V3VWni57Tb596633N3RERkfywS9wBiIhI+ijpi4jkESV9EZE8oqQvIpJHlPRFRPKIkr6I\nSB5R0peMY2ZNzazMzDL282lml5vZa3HHIbKzMvaPSrKXmU00swGVrO9iZst2MJlnzAUklX0Jufvz\n7n52ivY31cx6pWLbIkr6kgrDgO6VrO8ODHf3sjTHkygjfAlZ3IGIJEpJX1JhHLCvmZ28ZYWZ7QWc\nBzwTLXc0s5lmttrMFplZ/6o2ZmZfmtnp5Zb7m9nwcsttzewdMysxs1lm1m4b22pkZi+a2Qoz+9zM\nbir32HFmNiOKaZmZ/SV66M3o31Vm9p2ZHW9mPczs7XKvLTOz35jZguj1/2Vmzczs3SiuEWZWc8ux\nMLOXoxi+ie7vHz12F3AK8Ei0r4ej9Yea2aTo+XPN7JJy++5oZp9Gz19sZrdu4/9G8p2766Zb0m/A\nYGBwueVrgZnllk8FWkb3DweWAZ2j5abAZmCXaPlL4PRyr+0PPBPdbwysBDpEy+2j5X0ricmAD4A/\nATWAXwALgTOjx98FukX3awNtKsRj5bbVA3ir3HIZMBbYA2gBrAfeiF67J/ApcEX03H2AC4CfRc9/\nARhbbltTgV7llmsD/wSujN7DkcDXwKHR4/8CTozu1wOOivv/X7fMvamlL6kyDLjEzHaNlq+I1gHg\n7m+5+6fR/U+AkUCVLfRt6Ab83d1fj7b1BiGxd6zkuccB+7n73e6+2d2/Ap4ELo0eLwWam9m+7r7W\n3d+v8PrtlXfudfcf3H0u8Anwmrsvcvc1wETg6CjGb919rLtvcPcfgHsIX4JVOQ/40t2f8eAj4CVg\nS2t/I9DSzPZ099XuPns7cUoeU9KXlHD3dwit0fPNrBkh4T6/5XEza2NmU6ISxyrCL4H9qrGrpkBX\nM/s2upUAJwGNqnhu4wrP7QfUjx7vBRwCzDOz98zs3J2MZUW5++uA4grLdQDMbHcz+5uZfRW99zeB\nvcysqi+VpkDbCnFfDjSIHr8IOBdYFJ0EbruTcUseqRl3AJLThhPKIIcCr7v71+Ueex54mFCWKTWz\nB4B9q9jOD4QSxxYNy91fTCj1XLsD8SwGvnD3Qyp70N0/JyRTzOwi4EUz24fk9yS6DTgIOM7dvzaz\nI4GZbD1hXHF/i4FCd+9QRdwfEr5cawA3AaOAA5Ics+QItfQllZ4BzgB+TbnSTqQOUBIl/DZEybac\n8q3e2cClZlbTzFoDF5d77Fmgk5mdZWa7mNluZtZuy4nRCt4H1pjZ76Pn1TCzltE2MbNuZrbl18Zq\nQvItI/xiKQMO3NkDUIU6hJb/d9GXyoAKjxcDzcotvwIcbGbdo2NQy8xaRyd3a0XXDNR1983AGsL5\nB5FKKelLyrj7IsLJ0drAhAoPXw/caWargdsJJzP/7eXl7t8BNAe+JZzEfa7cPpYAXYA/EpLzIkJL\n+iefbQ9dRc8DjiKcHF4BPAHUjZ5yNvCpmX0HPAD8R1R3XwfcDbwTlVfaVPZ2t7Nc3oOEY7KScHxe\nrfD4Q4TzId+Y2YPu/j1wFuHcw7+i20Cg/PmSL6NS0TX89AtU5Efmvu1frmY2hPCHUuzuR0Tr9ib8\nkTYFvgK6uvvq6LF+hNroJqC3u09KWfQiIrJTdqSl/zRQsZbYF5gc1UanEE6GYWaHAV0JXdbOAR7d\nxskpERFJs+0mfXefBpRUWN2FrTXaYcD50f3OwEh33xR1h1sAVPZTWEREYlDdmn59dy8GcPflbO3y\n1pjQ02CLpdE6ERHJAMk6kZsxg2OJiEjVqttPv9jMGrh7sZk1ZOtFKUuBn5d7XpNo3U+Ymb4oRESq\nwd2rfa50R1v6xr/3m54A9Izu9wDGl1t/qZntama/JHSzq3gp+4/iHoMiU279+/ePPYZMuelY6Fjo\nWFR+69vX6dEj8bbydlv6ZvY8UEAYNfGfhH7SA4HRFsb8XkTosYO7F5nZKKCIMI7J9e6uFr2ISAK+\n+goGD4Y5c2BYxcscd9J2k767V3WhxxlVPP8ewgBSIiKSBH37Qu/esH9l15nvJI29kwEKCgriDiFj\n6FhspWOxVT4fi3ffhXfegaeeSs72tntFbqqYmSo/IiLbsHkztG0LN98MV1wR1pkZnoYTuSIikmZ/\n+xvUrg3dK5t8tJrU0hcRyUDLl0OrVlBYCC1bbl2faEtfSV9EJANdfjk0bQr3VOgWk2jS14lcEZEM\nM3lyOIH75JPJ37Zq+iIiGeT77+H66+GRR0I9P9lU3hERyRClpdCpExxwQLgYqzKq6YuI5ICyMujZ\nE0pKYOxYqFlF8V01fRGRHNCvHyxcGOr5VSX8ZFDSFxGJ2ZAhMGECTJuWmjp+eSrviIjEaOlSOOoo\nmDoVDj98+8/XFbkiIlnsppvgN7/ZsYSfDCrviIjEZMwYKCqC559P3z5V3hERicGqVWF4hZEj4ZRT\ndvx16rIpIpJl3KFXL/jZz+Dxx3futeqyKSKSZW6/HT76KAymlm5K+iIiaXT33TB+fEj4deumf/9K\n+iIiafLAA2GO2zffhP32iycGJX0RkRQrK4M//xmGDw8t/EaN4otFSV9EJIXWrg1j6ixZAv/3f9Cg\nQbzx6OIsEZEUWbYM2rULvXSmTIk/4YOSvohISpSWwgUXQIcO8MwzsNtucUcUqJ++iEgK/P734Wrb\nl18Gq3av+p9SP30RkQwzcWK40nbmzOQm/GRQ0hcRSaKlS+Gqq2D06Pi6ZW6LavoiIklSWgqXXw43\n3LBz4+mkk2r6IiJJcsstMH9+qOPXqJGafaimLyKSAZ59NiT7GTNSl/CTQS19EZEEzZoFZ50V+uK3\napXafWnmLBGRGC1eDBdeCIMGpT7hJ4OSvohINU2dCm3awM03Q9eucUezY1TTFxHZSe5hxMz77gu1\n/DPOiDuiHaekLyKyEzZtgmuuCZOgvPceNG0ad0Q7R0lfRGQHbdwI3brB6tXw1luwxx5xR7TzVNMX\nEdkB69bB+eeHlv7LL2dnwocEk76Z9TOzT83sYzN7zsx2NbO9zWySmc03s9fNrF6yghURicOyZWG0\nzH32gVGjwlDJ2araSd/MmgJXA0e7+xGEUtFlQF9gsrsfAkwB+iUjUBGROPzjH3DssXD66WGI5Fq1\n4o4oMYnU9L8DNgJ7mFkZsDuwlJDk20XPGQYUEr4IRESyRmkpDBgAQ4fCc8/BaafFHVFyVDvpu3uJ\nmf0P8E9gLTDJ3SebWQN3L46es9zM6icpVhGRtJg9G3r1CjNdzZoF9XMoi1U76ZtZM+AWoCmwGhht\nZt2AimMrVDnWwoABA368X1BQQEFBQXXDERFJ2IYNcOed8MQToQ/+lVfGPx5+YWEhhYWFSdtetcfe\nMbOuwJnufnW0fAXQFjgdKHD3YjNrCEx19xaVvF5j74hIxti4Ec45B+rUgccfh0aN4o6ocnGOvTMf\naGtmu5mZAe2BImAC0DN6Tg9gfAL7EBFJOXe4+mrYc08YMyZzE34yJFLT/8jMngE+BDYDs4DBwJ7A\nKDPrBSwCsmREChHJV3feGeazLSzM7GGRk0FDK4tIXnv2Wbj9dpg+HRo2jDua7dMkKiIi1fSPf8Ct\nt4Zx8LMh4SeDkr6I5KUZM8J8tmPGwOGHxx1N+mjsHRHJO/PmQadOMGRI5k5gnipK+iKSV5YsgbPP\nhnvugc6d444m/ZT0RSRvFBdD+/Zw001w1VVxRxMPJX0RyQvffBNmuOrWDfr0iTua+KjLpojkvNWr\nQwu/fXsYODD+oRUSEecVuSIiGW/58tDCP+GE7E/4yaCkLyI5a948OPHE0FPn4YeV8EH99EUkR739\nNlx8cRgts0ePuKPJHEr6IpJzxo2Da64Jk5+ceWbc0WQWlXdEJKcMGQLXXw+vvaaEXxm19EUkJ7iH\nUs7jj4fRMg8+OO6IMpOSvohkvc8/D33vFy6EadOgceO4I8pcKu+ISNZaswb69oXjjw+3Dz5Qwt8e\ntfRFJCtNmwbdu0NBAcyZk9uzXSWTkr6IZJVNm+Duu+Gxx+DJJ+G88+KOKLso6YtI1lixAi65BGrV\ngpkzYf/9444o+6imLyJZoagI2raFU0+FSZOU8KtLLX0RyXiTJ4dZrv7yF7jyyrijyW5K+iKSscrK\nwpg5AwfC6NHQrl3cEWU/JX0RyUjFxdCzJ5SUwLvvQrNmcUeUG1TTF5GM4h7Gzjn6aGjdOgycpoSf\nPGrpi0jGmDoVbr8dVq2CESNUzkkFJX0Rid3cuXDzzfDll/DnP8Oll0KNGnFHlZtU3hGR2KxfD/37\nh26YnTuH5N+tmxJ+KqmlLyJpV1YGEybAH/4ALVvCrFnQpEncUeUHJX0RSZv16+HZZ0N/+z33DP92\n6hR3VPlFSV9EUmrNmjChyfjxMHFiGA3z8cfDSVrNWZt+5u7x7NjM49q3iKTHJ5+Eev3xx4eafadO\nKuMkysxw92p/XSrpi0hKlJWF1ny3bnDddXFHkzsSTfrqvSMiKTF0KGzcGCYol8yhlr6IJN3KlaFX\nzmuvhStrJXlU3hGRjNOrF9StCw8+GHckuSfRpK/eOyKSVFOmhPHui4rijkQqk1BN38zqmdloM5tr\nZp+a2fFmtreZTTKz+Wb2upnVS1awIpLZPvooDKHw9NOhpS+ZJ9ETuQ8Br7p7C+BIYB7QF5js7ocA\nU4B+Ce5DRLLAggVwzjkwaBCceWbc0UhVql3TN7O6wCx3P7DC+nlAO3cvNrOGQKG7H1rJ61XTF8kR\nS5bAySeHETJ//eu4o8ltcXbZ/CWw0syeNrOZZjbYzGoDDdy9GMDdlwP1E9iHiGS4WbNCf/wbb1TC\nzwaJJP2awDHAIHc/BviBUNqp2HxXc14kB7nDY4/BWWfB3XfDbbfFHZHsiER67ywBFrv7B9HyS4Sk\nX2xmDcqVd1ZUtYEBAwb8eL+goICCgoIEwhGRdFm3LnTLnDs3TGV40EFxR5S7CgsLKSwsTNr2Euqn\nb2ZvAle7+2dm1h+oHT30rbvfa2Z/APZ2976VvFY1fZEstH49dOkC++wDTz0Fu+8ed0T5JdaLs8zs\nSOBJoBbwBXAVUAMYBfwcWAR0dfdVlbxWSV8ky2zYABdeCHXqwHPPQU1d6ZN2uiJXRNKitBQuuQR2\n2QVeeAFq1Yo7ovykAddEJOWWLIEOHcLImSNHKuFnMyV9EdmmMWPg2GPhjDNg7FjYdde4I5JEqCIn\nIpVasQL69oW33grz2R5/fNwRSTKopS8i/2bdOrjnHjjsMKhXD2bOVMLPJWrpi8iP3n8/nKxt3Rqm\nT4fmzeOOSJJNSV9EgDAccrdu8MQTcP75cUcjqaKkLyKMGAG//W04UXvyyXFHI6mkpC+Sx9zh/vvh\n4Ydh8mRo1SruiCTVlPRF8lRJCfTsCcuXwzvvQNOmcUck6aDeOyJ5aMYMOOYYaNYM3n5bCT+fqKUv\nkkdWr4b+/eH558OwyBddFHdEkm5q6YvkAXd45hlo0QJ++AE+/VQJP1+ppS+S4z77DK69FtasgXHj\noE2buCOSOKmlL5KjNm6Eu+6CE08M/e7fe08JX9TSF8k5paWhlPPf/x2GUpg5Ew44IO6oJFMo6Yvk\niHXrYPjwMG5O8+YwdCicckrcUUmmUdIXyXLLlsGjj8LgwXDccWFGqxNPjDsqyVSq6YtkqbIyGDgQ\nWraEb78NQyC/8ooSvmybWvoiWaikBK68Er75Bj7+GJo0iTsiyRZq6YtkmQ8/DDNZNW8Ob76phC87\nR0lfJEu4h2GPzz4b7r0XHnhAc9XKzlN5RyQLrF0L118PH3wA06bBIYfEHZFkK7X0RTLcrFnQti1s\n2hQusFLCl0Qo6YtkqA0b4PbboUMH6NMn9MHfY4+4o5Jsp/KOSAZ6//0w1v2hh8JHH0GjRnFHJLlC\nSV8kg5SWwp13wt/+Fmaz6toVzOKOSnKJkr5IhpgzJ7TuGzWC2bPVupfUUE1fJGYLFkD37tC+PVx3\nHbz8shK+pI6SvkgM3LfW7U88MdTuFy6Eq69WOUdSS+UdkTT69tsw+uVTT4VRMX/1K3jwQdhrr7gj\nk3yhpC+SBsXF8Ne/wpNPQseOMGhQGPZ4F/3WljTTR04khUpKQh/7LXPTzpwZ+tu3a6eEL/HQx04k\nBUpL4ZFHQq1+7dowEfkjj0DTpnFHJvlO5R2RJNq8GUaNCn3tGzeGyZOhVau4oxLZSklfJAk2bIBn\nnw2TmjRoEOr3HTqoJ45kHiV9kWratAmmTIERI2D8eGjTBoYMgVNPjTsykaqZuye2AbNdgA+AJe7e\n2cz2Bl4AmgJfAV3dfXUlr/NE9y0Sh5UrwzAJjz4aJjC59NIwXELjxnFHJvnAzHD3av+GTMaJ3N5A\nUbnlvsBkdz8EmAL0S8I+RGK1YQNMmhQunjroIPjiC5g4MQx1fMstSviSPRIq75hZE6AjcDdwa7S6\nC9Auuj8MKCR8EYhkje+/D+PYf/ghFBbC1KlhAvLOnWH+fKhfP+4IRaon0Zr+A8DvgHrl1jVw92IA\nd19uZvrzkKxRVgZduoRafatWcMwxcPHF4aKq/faLOzqRxFU76ZvZuUCxu882s4JtPFWFe8kaTzwR\nLqhatUrzz0puSqSlfxLQ2cw6ArsDe5rZcGC5mTVw92IzawisqGoDAwYM+PF+QUEBBQUFCYQjkpji\nYrjjDnjjDSV8yRyFhYUUFhYmbXsJ994BMLN2QJ+o9859wDfufq+Z/QHY291/UtNX7x3JNFdeCQ0b\nwn33xR2JSNUS7b2Tin76A4FRZtYLWAR0TcE+RJJq6lR4800oKtr+c0WyWVJa+tXasVr6kiHWrYOj\nj4Z77w0ncUUyWaItfSV9yWulpXDBBbDvvjBsWNzRiGxfJlycJZKVysrCzFVmoUumSD7Q2DuSl9zh\nxhth6dJwZa1660i+UNKXvLN+PfTuHa62nTIFdt897ohE0kflHckrCxeGici//Tb0x69bN+6IRNJL\nSV/yxpgxIeH36hUmOqlXb/uvEck1Ku9IXhgxAm67DV59FVq3jjsakfioy6bkvHHj4LrrwtSFhx8e\ndzQiicnEK3JFMsbrr8M114QeOkr4IqrpS45yh6FD4YorQkv/2GPjjkgkM6ilLznn66/h2mtDT53J\nk+GII+KOSCRzqKUvOcMdXnoJjjwSmjeHGTOU8EUqUktfcsKMGdCnT5j8ZMQIaNdu+68RyUdq6UtW\nW7cOrroKzj8fevQI89oq4YtUTS19yVolJWGi8p//PExWXqdO3BGJZD619CUrLV4Mp5wCxx0Hzz6r\nhC+yo5T0JevMng0nnxyGRf7rX2EXfYpFdpj+XCSrDBkCZ54J998fhlUQkZ2jmr5khbVrw/j306fD\nW29BixZxRySSndTSl4z32mtw1FGwcSO8/74Svkgi1NKXjPXVV3DLLTBnDjz8MHTsGHdEItlPLX3J\nOKWloWbfunUYM+eTT5TwRZJFLX3JKDNmwNVXQ/36oZTTrFncEYnkFrX0JSN89lm4orZTJ/jd78KQ\nyEr4IsmnpC+xKiqC7t3hpJPgwANh3jzo1g2s2lNEiMi2KOlL2pWVhUlNOnSA004LvXE+/xz+8z9h\nr73ijk4kt6mmL2k1e3Zoye+6K/TuDePHw267xR2VSP5Q0pe0GTo01OsffBAuv1wlHJE4KOlLyq1f\nDzffHK6kLSyEli3jjkgkf6mmLyn15ZfhJO2qVaE7phK+SLyU9CVl/v53aNs2TE7+wguw555xRyQi\nKu9I0q1fD/37w/PPw5gxoaUvIplBLX1JqnffDYOjLVwIH36ohC+SadTSl6T47ju44w4YPToMjnbx\nxXFHJCKVUUtfElJWBs88A4ceCmvWhBExlfBFMpda+lJt770Xhj7etAnGjYM2beKOSES2p9otfTNr\nYmZTzOxTM5tjZjdH6/c2s0lmNt/MXjezeskLVzLB/PmhNX/RRWFEzOnTlfBFskUi5Z1NwK3u3hI4\nAbjBzA4F+gKT3f0QYArQL/EwJRPMnx+S/EknhbHuP/sMrrpKE5OLZJNq/7m6+3J3nx3d/x6YCzQB\nugDDoqcNA85PNEiJjzu8/TZ06QKnnAL77x+Sf9++ULt23NGJyM5KSk3fzH4BHAVMBxq4ezGELwYz\nq5+MfUh6uYe5ae+6C1asgD59YMQIJXqRbJdw0jezOsCLQG93/97MvMJTKi7/aMCAAT/eLygooKCg\nINFwJAkmToTbbw8Tkf/pT3DJJVCjRtxRieSnwsJCCgsLk7Y9c68yJ2//xWY1gVeAie7+ULRuLlDg\n7sVm1hCY6u4tKnmtJ7JvSb6iotCi/+ILGDgwlHRUrxfJLGaGu1d7jNpE/6SfAoq2JPzIBKBndL8H\nMD7BfUiKLVoEv/kNtGsXJjaZMwcuuEAJXyQXJdJl8ySgG3C6mc0ys5lmdjZwL3Cmmc0H2gMDkxOq\nJNv8+aH3zTHHQN26MHcu/Pa3YYITEclN1a7pu/s7QFWV3jOqu11Jra+/DkMljBwZ5qO96aYwTs7e\ne8cdmYikQ0I1/YR2rJp+2pSVweTJMGgQvPkmdOwIl10GZ50FP/tZ3NGJyM5ItKavYRhy2JdfhqGN\nBw+G3XeHG24Iwx3vsUfckYlIXJT0c8jKlWE443ffDROO/+tfcN55MGRIuIpWc9KKiMo7OeDRR+G+\n+6CkBI4+Go4/Hjp1ghNOUP96kVyj8k6ee/ppuPdemDABWrVSN0sR2Ta19LPY2LFw/fVQWAiHHBJ3\nNCKSDmrp56k33oBrrw3j4yjhi8iOUjEgC02cCJdeCqNGhQurRER2lJJ+lnnuOejZM9TwNT6diOws\nlXeyyEMPwV/+AlOmQMuWcUcjItlIST/DbdoUTtg+8ACsXh0mNPnFL+KOSkSylco7GaqsLFxJ27w5\nPPww3HYbfPyxEr6IJEYt/Qw0c2YY6rhmTXjhhXCxlYhIMqiln0EWLAj97s85J3THfPttJXwRSS4l\n/Zht3gwvvQRnnAEnnxzGtS8qgl69dHWtiCSfyjsxmj49jHxZqxb07g0XXqihjkUktZT0Y7ByJfTr\nB3//O9x/P1x+uUbAFJH0UAEhjTZuDF0vW7QIY9rPnQvduinhi0j6qKWfBqWl8Mor0LcvNGsGb70V\nEr+ISLop6afIhg0h0Y8dC6++CgcfDA8+GHrmiIjERUMrJ1lpKQwdCnfdBQceCJdcAp07Q+PGcUcm\nIrlAQytniMWLYdy4ULNv3hxGjgwzV4mIZBIl/QQUFcGLL4b5aBctgnPPDa38U0+NOzIRkcqpvLMT\n3GHWrJDkR4+GNWvgoovgggvCxOM19RUqIimWaHlHSX873GHy5DAGzquvQp06YdLxiy8OQyToqlkR\nSScl/RSaMgXuuANKSsJYOB07wkEHxR2ViOQznchNgffeC33qly6F/v3D1IQ1asQdlYhI4lScKGfB\ngtDF8qKLoHv3cKK2WzclfBHJHXmf9N1h2rSQ5E84IUw0/tln8Ktf6cSsiOSevK3pr1sHw4bBoEFh\nTJzrroMePWCffWILSURku1TT30mrVsFjj4VJxtu0Cf+edpoGPROR/JAXSX/TJpg6FZ57DiZMgPPO\nC90wDz887shERNIrJ8s77jBvHrzzTqjXv/YaNG0axq3v2hUaNUrJbkVEUk799COlpaE1P2ZMGAOn\ndu1wlezJJ0P79mE8HBGRbJfXNX13+OADePrpcMXsQQeF7pbTpinJi4hUJmVJ38zOBh4kdAsd4u73\nJmvbX30Fo0bB8OGwdi307AkzZ4YSjoiIVC0l/fTNbBfgEaAD0BK4zMwOre72fvgB3n4b7rsP2raF\n1q1h4UL43/8NF1TdcUd2J/zCwsK4Q8gYOhZb6VhspWORPKm6OKsNsMDdF7l7KTAS6LKjL96wAd54\nA/r0gSOPhPr14bbbwpj1AwbAsmUweDAUFOTGgGf6QG+lY7GVjsVWOhbJk6ryTmNgcbnlJYQvgp/Y\ntCmMRT97Nnz4YbhNnw6HHRYGOBs8GI4+GnbdNUWRiojkkVhP5B54ICxZAg0bwhFHwLHHwo03hv70\n++0XZ2QiIrkpJV02zawtMMDdz46W+wJe/mSumWX2uMoiIhkq4/rpm1kNYD7QHlgGvA9c5u5zk74z\nERHZYSkp77j7ZjO7EZjE1i6bSvgiIjGL7YpcERFJv1g6PJrZ2WY2z8w+M7M/xBFDXMysiZlNMbNP\nzWyOmd0crd/bzCaZ2Xwze93M6sUdazqY2S5mNtPMJkTLeXkcAMysnpmNNrO50efj+Hw8HmbWL3r/\nH5vZc2a2az4dBzMbYmbFZvZxuXVVvv/oeC2IPjdnbW/7aU/6yb5wKwttAm5195bACcAN0fvvC0x2\n90OAKUC/GGNMp95AUbnlfD0OAA8Br7p7C+BIYB55djzMrClwNXC0ux9BKEFfRn4dh6cJ+bG8St+/\nmR0GdAVaAOcAj5pte6D4OFr6CV24le3cfbm7z47ufw/MBZoQjsGw6GnDgPPjiTB9zKwJ0BF4stzq\nvDsOAGZWFzjF3Z8GcPdN7r6a/Dse3wEbgT3MrCawO7CUPDoO7j4NKKmwuqr33xkYGX1evgIWUMU1\nUVvEkfQru3CrcQxxxM7MfgEcBUwHGrh7MYQvBqB+fJGlzQPA74DyJ5by8TgA/BJYaWZPR+WuwWZW\nmzw7Hu5eAvwP8E9Csl/t7pPJs+NQifpVvP+K+XQp28mnOTCIQXYyszrAi0DvqMVf8Yx6Tp9hN7Nz\ngeLoV8+2fo7m9HEopyZwDDDI3Y8BfiD8pM+3z0Uz4BagKbA/ocXfjTw7Djug2u8/jqS/FDig3HKT\naF3eiH62vggMd/fx0epiM2sQPd4QWBFXfGlyEtDZzL4ARgCnm9lwYHmeHYctlgCL3f2DaPklwpdA\nvn0uWgORHEUTAAABP0lEQVTvuPu37r4ZGAucSP4dh4qqev9LgZ+Xe95282kcSX8G0NzMmprZrsCl\nwIQY4ojTU0CRuz9Ubt0EoGd0vwcwvuKLcom7/9HdD3D3ZoTPwBR3vwJ4mTw6DltEP90Xm9nB0ar2\nwKfk2eeCcFFnWzPbLToh2Z5woj/fjoPx77+Aq3r/E4BLox5OvwSaEy6GrXrDcfTTj8baf4itF24N\nTHsQMTGzk4C3gDmEn2gO/JHwHzWK8K29COjq7qviijOdzKwd0MfdO5vZPuTvcTiScFK7FvAFcBVQ\ngzw7Hmb2O0KC2wzMAn4N7EmeHAczex4oAPYFioH+wDhgNJW8fzPrB/wKKCWUiydtc/u6OEtEJH/o\nRK6ISB5R0hcRySNK+iIieURJX0Qkjyjpi4jkESV9EZE8oqQvIpJHlPRFRPLI/wOyOpUh162Q+QAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10552ae10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = [(capital, max(opts)) for capital, opts in gambling_agent.policy.items()]\n",
    "x, y = zip(*data)\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.scatter(x, y)\n",
    "plt.title(\"Policy (only the maximum stake action)\")\n",
    "plt.show()\n",
    "\n",
    "x, y = zip(*gambling_agent.values.items())\n",
    "plt.plot(x, y)\n",
    "plt.title(\"Value estimates\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This visualisation shows all optimal actions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEKCAYAAAD6q1UVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+cHXV97/HXZ5Pd7LqbhE1dQQQ2QRSigG4Qrq20LBgo\nhlvD7Q/itnpF0h9cb9G2WgFrL/G2tsJtL/WaWqQgUCpLUJEgNxrKJYu1rU2UIBQSBCRr5EeykBjZ\nkISEfO4fMyc5e3bO7tmdOWdmzryfj8d55OycMzPf78zJ58z5zGe+Y+6OiIgUS0vaDRARkcZT8BcR\nKSAFfxGRAlLwFxEpIAV/EZECUvAXESkgBX+pysyuMrNbw+fHmtnPzMwSXkebmT1qZkfW8N5eMzto\nZi3h3+vM7JIk2zPBuv/DzH6pDss9y8y2Jr3ciPWcaWab6ryOvzKzS+u5DkmOgn8BmNkWM3s5DN7P\nmdlNZvaaGmd3AHff6u5zPPkLQ34XeMDdt02lPfUUbp//OWal7ie7+7frtMrE+xR+SR5/aAXu33H3\nhUmvp8JfAZ80s5l1Xo8kQMG/GBy4wN3nAIuAdwCfSrdJh1wK3Jp2I5pQw6/edPfngU3Aexu9bpk6\nBf/iMAB3fw74JnAygJm93sxWm9mLZvZDM/vtyJnHp1y6zexLZvZMOO+d4fRHzOyCsvlmmtmImb0t\nYpnHAguAfy+btsTMHjSzXWY2bGZXTauzQTrpb8L2/cTMrjWz1vC1s8xsq5ldGbbtR2b2m+FrvwP8\nFvCJ8JfS6nD602Z2Tvj8KjO7w8xuDd/zAzN7k5ldYWbbwl9ai8vacrGZPRa+90kz+90p9ONvzOzH\n4fbYYGZnlr3WYmafDJf5s/D1Y8zsAYL9/XA4/Tcq00tmdlKYNtsZ7rNfKXvtJjNbaWb3hPP/m5kt\nKHv92rCfu8K+v6WsyQ8AFyCZp+BfMGHAXQI8GE5aBfwYOAr4DeAvzKy/yuzlR5P/CHQAC4HXAdeG\n0/8B+EDZ+y4AnnX3H0Qs7xTgR+5+sGzaKPABd58bznupmU3nSPJTwBnAqcDbwuflv3aOAuYBRwMX\nA9eb2Zvc/e+BLwPXhGmupVWW/5+BW4AjgIeAfyIIuEcDfw5cX/bebcCS8JfXh4BrzeztNfZjfdiH\nbuA24Ctm1ha+9jFgGXB+uOxLgN3uflb4+ilhH74S/u0QfCED3wC+BfQAHwG+bGZvKlvvMuCqsH9P\nAZ8J5z0POBM4IdxHFwEvls23iWB7S8Yp+BfHXWa2A/g2sA74SzM7Bvh54HJ33x8G6BuA/zrRgszs\n9cAvA7/n7j9z91fd/Z/Dl/8RuMDMusK/30/1tM4RwEvlE9z92+7+aPj8P4DbgbMi5p3MbwKfdvcX\n3f1F4NOM/VJy4E/Dfn8b+L8EgaxW/+zu94VfXF8h+CL5rLu/Gra518zmhP34prtvCZ//M3Av8Iu1\nrMTdb3P3n7r7QXe/FpgFnBi+vBz4E3d/MnzvI+6+s2z2aifnfx7odPer3f2Au68D7gEGyt7zdXf/\nfti/LwOlL6v9wGzgLWZm7v54xfmalwj2q2Scgn9xLHX3ee6+wN0vc/d9BEepO9z95bL3DQNvmGRZ\nx4Tz/azyhTCt9B3g18xsLvAeguARZSdBIDnEzM4ws/vNbLuZ/RT4PeC1tXSwwtEEv2hKhsNph9bt\n7nsneH0y5QFvD/BC2cnwPQSBtwvAzN4Tpk5eNLOdBNukpj6Z2cfDlNHOcN45ZfMeC/xoCm0ueT1Q\nWWFUud+fL3v+MmFfwi+KlcDfAtvM7DozK9+Hs4GfTqNN0mAK/sURdRT4LDDPzDrLph0HPDPJsraG\n882p8nop9fMbwL+GXwhRHgYWlM4jhG4D7gLe4O5HAF+s0vbJPAP0lv3dS9Dfkm4z6yj7+7iy1xM7\nWRqmaL4KXAP0uHs3wTmXSfsU5vf/GPh1d+8O5/1Z2bxbgTdOo1nPEnxxlKtlvwPg7ivd/R3AWwh+\nhXy87OWFQFSKTzJGwb/A3P0nwL8SpIBmmdmpBKmEamma0knj5wkC2BfM7IjwpG55GuPrBFVFHyH4\nIqi2/meAJwny8SVdBEfl+83sDIL0zbg21OB24FNm9lozey3wpxX9MuDTZtYatv0C4I7wtW3A8SSj\nLXy84O4Hzew9wHk1zjubIM3yYngC+38w9pfSDcCfmdkJAGZ2ipl1h689T/U+/Dvwspl9Itx3/QTn\nMAYna5CZvSP8dTaT4BfOXqD8nM1ZBJ8NyTgF/2KY6Eh2gKDi5lngawR58HU1LOcDwAFgM0Gw/Oih\nNwXplDvD5d45Sdu+yNhzDB8mCGi7CE7QrpqgDRP168+B7xH8uvhB+PwzZa8/R5B2epbgS+H33P2J\n8LUbgbea2Q4Lq5gmWVeU0vURowRfgl8Jz7m8D1hd4zLWho8fAk8TpF/K0zX/m+AL695we91AcBIe\ngnMc/xD24dfHNMx9P/ArBCf+XyBI43ygrP8T9XUO8PfAjrBNLwD/Cw6dC1pI8MtNMs6SuGYnzO3e\nQFA+eJCg6uCHBP9xe4EtwEXuviv2yiQXzOxTwJvdfbKTx20ElUfvnsKFXnHbdhZwq7sf14j1FYWZ\n/RXwpLtfl3ZbZHJJBf+bCa7SvCn8OdgJfBJ40d2vMbPLgW53vyL2yiTzzGwe8H2Co8nvpN2eSgr+\nIgmkfcKTfr/o7jcBhKVju4ClBHXQhP9eGHddkn0WXCT2Y2BNFgO/iARiH/lbcOXm9cBjBBd3fA/4\nA+CZsDqh9L4d7j4v1spERCQRSZzwnUlQ2fG37r4I2A1cwfiTRrpTvIhIRiQx+t5PgK3u/r3w768R\nBP9tZnaku28zs6OA7VEzm5m+FEREpsHdpz3Eeuwj/7BCY6uZvTmc9G7gUeBugjFTAD7IBOVt7p7b\nx1VXXZV6G9T+9NtRxPbnue3N0P64khp3uzQwVCvB5eYfAmYAd1hws41hpjZuioiI1FEiwd+DAcFO\nj3hpccQ0ERFJma7wjam/vz/tJsSi9qcrz+3Pc9sh/+2PK5GLvGI1wMzTboOISN6YGZ7mCV8REckf\nBX8RkQJS8BcRKSAFfxGRAlLwFxEpIAV/EZECUvAXESkgBX8RkQJS8BcRKSAFfxGRAlLwFxEpIAV/\nEZECUvAXESkgBX8RkQJS8BcRKSAFfxGRAlLwFxEpIAV/EZECUvAXESkgBX8RkQJS8BcRKaCZSSzE\nzLYAu4CDwH53P8PMuoFVQC+wBbjI3XclsT4REYknqSP/g0C/u/e5+xnhtCuA+9z9ROB+4MqE1iXS\nECMjI2zYsIGRkZG0myKSuKSCv0UsaylwS/j8FuDChNYlUneDg6vo7T2Jc8+9lN7ekxgcXJV2k0QS\nZe4efyFmPwJ+CrwKfNHdbzCzne7eXfaeHe4+L2JeT6INIkkZGRmht/ck9uxZB5wKPExHx9kMD2+m\np6cn7eaJAGBmuLtNd/5Ecv7Au9z9OTPrAe41s8eByoheNcKvWLHi0PP+/n76+/sTapbI1G3ZsoW2\ntvns2XNqOOVUWlt72bJli4K/pGZoaIihoaHElpfIkf+YBZpdBYwCv01wHmCbmR0FrHP3hRHv15G/\nZMr4I/8hZs1aysaN32XhwnEfYZFUxD3yj53zN7PXmFlX+LwTOA94BLgbuDh82weB1XHXJdIIPT09\n3HjjF+joOJv29uOBJbS09HLaaWcq9y9NI/aRv5ktAL5OkNaZCXzZ3T9rZvOAO4BjgWGCUs+fRsyv\nI3/JpE2bNtHX9wvs2/cAyv1L1qSe83f3p4G3R0zfASyOu3yRRhgZGWHLli3Mnz//UGAfHR2lvf14\n9u1T7l+aj67wlcKrVtY5f/58XnllC/Bw+M6H2b9/mPnz56fUUpHkJH7Cd8oNUNpHUjRZWefg4CqW\nL/8wra297N8/zI03foGBgWVpN1sk/bSPSJ5NVtY5MLCMxYvPGZcSikoTieSJ0j5SaLWkdnp6ejj9\n9NMPBXld/SvNQGkfKbyppHZ09a9khdI+IjFVS+1E0dW/0iwU/EUIUju1BO+xaaLgyF8VQJJHyvmL\nTEH51b9z5iyio+NsbrzxCzrql9xRzl8KI8kKnahlqQJIGin1sX1E8iDpCh1VAEne6chfml69K3RU\nASRp0JG/yCRKFTpBYIbyCp08LF+kHhT8penVe4ye8csfYt++p+jq6kpk+SL1oOAvTa/eFToa/1/y\nSDl/KYx6V+No/H9pJF3hKxIhKtDXeiHXdGn8f8kTpX2k6aRVdqnx/yVPlPaRppJ22aXG/5dGUdpH\npMzGjRtpaTmWqLLLRgT/qQwSJ5ImBX9pGoODq7jkkkvZu/cV0hx4rd7nFkSSoOAvTWFkZITlyz/M\n3r0PAJuAfmAeHR07NfCaSASd8JWmMPYq22XA43R2zuCuuwaVcxeJoOAvTWF8pc1zHDz4An19fSm2\nSiS7Egv+ZtZiZg+a2d3h391mdq+ZPW5ma81sblLrEqmkcfZFpiaxUk8z+0PgNGCOu7/XzK4GXnT3\na8zscqDb3a+ImE+lnpKYPI2pn6e2SvZkYlRPMzsGWALcUDZ5KXBL+PwW4MIk1iUykcpx9rNK4/9L\n2hI58jezrwCfAeYCHwuP/He6e3fZe3a4+7yIeXXkL4WS9oVo0hxSv8jLzC4Atrn7Q2bWP8Fbq0b4\nFStWHHre399Pf/9EixHJt1Jl0p49GgNIajc0NMTQ0FBiy4t95G9mfwG8HzgAdACzga8D7wD63X2b\nmR0FrHP3hRHz68hfCkVH/pKE1HP+7v5Jdz/O3Y8H3gfc7+4fAL4BXBy+7YPA6rjrEmkGqkySLEh0\nYDczO4vDOf95wB3AscAwcJG7/zRiHh35SyGp2kfiiHvkr1E9JReKECiL0EdJTuppH5F6K0JZZBH6\nKNmiI3/JtCKcHC1CHyV5OvKXpjZ2wDYoL4tsFkXoo2SPgr9kWhFujViEPkr2KPhLphWhLLIIfZTs\nUc5fMieq6qUIlTBF6KMkR6We0lRKN0BvawtSIboBukg0BX9pGqp6Eamdqn2kaWzcuJGWlmNR1YtI\n/Sn4SyYMDq5i6dJl7N79BKp6Eam/2EM6i8Q1MjLC8uUfZu/eB4BNQD8wj46Onap6EakTHflL6sZe\n5LQMeJzOzhncddegTvaK1ImCv6Ru/EVOz3Hw4Av09fWl2KrsGhkZYcOGDYyMjKTdFMkxBX9JnS5y\nqp0GgJOkqNRTMkMXOU1MpbBSLvV7+IpMR1Sg7+npURCbgO79K0lS2kcaTqmL6dEAcJIkpX2koZS6\niKc0/EVray/79w9r+IsCU9pHcmWiq3gV/Cc3MLCMxYvP0bkRiU3BXxpmcHAVl1xyKXv3vkKQugiO\n/JW6mBqdG5EkKPhLQ+gqXpFs0QlfaQhdxdsYugBMaqXgLw2hq3jrT1VUMhWxq33MbBbwbaAtfKx2\n90+aWTewCugFtgAXufuuiPlV7VMQqlSpH1VRFU8mbuZiZq9x95fNbAbwL8DHgPcCL7r7NWZ2OdDt\n7ldEzKvgXyC6irc+NmzYwLnnXsquXd8/NG3OnEXcd98XOf3001NsmdRLJko93f3l8OksglTSTmAp\ncFY4/RZgCBgX/KVYVKlSH2PTaqqiksklkvM3sxYz2wg8Dwy5+2PAke6+DcDdnwdel8S6RGQ8DY4n\nU5XUkf9BoM/M5gBrzawfqMzlVM3trFix4tDz/v5++vv7k2iWSKFUuwBMqbbmMDQ0xNDQUGLLS3x4\nBzP7U2APsBzod/dtZnYUsM7dF0a8Xzl/kTopnWRvawvSQjrJ3jxSP+FrZq8F9rv7LjPrANYCnwbO\nA3a4+9U64SvSeKoAam5ZOOH7euAWMzOCcwi3uvv/C88B3GFmlwDDwEUJrEtEaqQhoGUisYO/uz8C\nLIqYvgNYHHf5IjI9qgCSiegKX5EmpQogmYjG8xdpcqr2aU6pn/CNS8FfRGTq4gZ/pX1ERApIwV9E\npIAU/EUKTOP/F5eCv0hBafz/YtMJX5EC0tW/+acTvtIwShE0j7G31QQ4lRkzjmbNmjXavwWh4C81\nUYqguYy/reY1jI4+xWWXfU77tyCU9pFJKUXQnEojfs6YcTSjo08B30X7Nz+U9pG627hxIy0tx1Ke\nIigNECb5NTCwjOHhzaxc+XFmzz4J7d9iUfCXCQ0OrmLp0mXs3v0Eh1MEGiCsWfT09LBkyRIOHBhG\n+7dYErmTlzSnkZERli//MHv3PgBsAvqBeXR07NQAYU2kNADc8uVn09ray/79w9q/BaCcv1S1YcMG\nzj33Unbt+n44ZYTOzjO5887Pc95556XaNkmeBoDLlyzczEWa0MjICDt37qwYD/45Dh58gb6+vnQb\nJ3XR09OjoF8gCv4yTvl9Xw8ceIW2tl+ivf14pQNEmojSPjJGVFlne/tZrF69ir6+PgV+kYxQqack\nKqqss61tAd3d3RMGfl39K5IvCv5yyHTLOnX1r0j+KO0jQGW6ZxPw3ygv6xwYWFbDfLo6VKRRlPaR\n2EZGRlizZg0zZ/YSBPBlwON0ds7grrsGqwZ+iB4gTFeHimSfqn0KrlTZM3PmG3jppSeZalnn2AHC\ngiN/XR0qkn2xj/zN7Bgzu9/MHjWzR8zsI+H0bjO718weN7O1ZjY3fnMlSaUrePfsWcdLLz0MrADe\nyezZfXR0nF1TWWfp6tCOjrOZM2dRzfOJSLpi5/zN7CjgKHd/yMy6gO8DS4EPAS+6+zVmdjnQ7e5X\nRMyvnH9Kxl/BC11dp7By5cdZsmTJlAJ41NWhumJUpH5Sz/m7+/Pu/lD4fJTgbOExBF8At4RvuwW4\nMO66JFnjx3R/mFdffXbKgR+CXwCnn376oflUASSSbYlW+5jZfGAIOBnY6u7dZa/tcPd5EfPoyD9F\npZx/+YBeE53grYUqgETqLzNj+4Qpn68CH3X3UTOrjOhVI/yKFSsOPe/v76e/vz+pZkmE8nTMwMAy\nFi8+J9H0TKkCaM+eUgXQ62lpeS0bN27UgHBNSOm9xhgaGmJoaCi5Bbp77AfBl8i3CAJ/adom4Mjw\n+VHApirzujTObbfd7h0d83zu3EXe0THPb7vt9sTXsX37du/omOfwA4fbHbodTqjb+iQ9jfg8SbQw\ndk47bieS9jGzfwBecPc/Kpt2NbDD3a/WCd9saGQ6ZnBwFZdccil7974C/Fvd1yeNp/ReulI/4Wtm\n7wJ+CzjHzDaa2YNmdj5wNXCumT0OvBv4bNx1STyNvCBrYGAZq1evorPzTQ1ZnzSeLvDLt9g5f3f/\nF2BGlZcXx12+JCN6fP76XpDV19fHwYNbG7a+Iksj7z7+Ar8h9u17iq6uroasX+LR8A4FUCq7vOii\nKw+Nz9+IC7J0AVhjpFVWW75/29uPB5bQ0tLLaaedqdLeHNDAbk0uC+PzqxqkfrKQd9+0aRN9fb/A\nvn0PpNaGIko95y/ZNt3x+ZNUeQGYJGP8gHyQRt59dHQ0PPI/3IYZM45mzZo1ur9Dhin4N7Hpjs8v\n2VdK9Vx22V/z0kubSXP/jr9S/BpGR5/isss+p6u7syxOnWgSD1TnXxfRtfZvVC12Exi7b93haocO\nnz377ant31K9f1fXyQ4dZW37gXd0zPPt27c3vE3Njph1/jryb0JxxueX7BtfYvkJurreyOc//wcM\nD29OZf8ODCxjeHgzK1d+nNmzT0Lln9mn4N9kqqcDahufX7IvyQH5ktTT08OSJUs4cGAYpRmzT8G/\niSQxPr9kX5ZLaLPcNhlLpZ5NJMnx+dOgktCpyfL2qmxbltuaV3FLPRX8m0gWar6nqzS0dFtbkNJI\nYmhpyQbt2/pQ8Jcx6jE+f73l+UtLJqZ9Wz+ZGc9fsqEe4/PX2/jx/w9XiOSh/VKd9m12Kfg3oZ6e\nnlz9xxo/QJgqRJqF9m12qdpHUqcKkealfZtdyvnnVDNWT0T1qRn7WYtm67f2bfLi5vw1vEMOFeXW\neUXpZ6Ui9LsIfaw3snAbxzh05D81RameKEo/KxWh30XoYyNoSOeCKcqt84rSz0pF6HcR+pgHCv45\n09XVxd69P6LZx06JGr+mGftZqQj9jurjK688zc6dOzX+fwMp+OfI4OAqTjvtTFpauoF30tFxStNW\nTxS1SqQI/e7p6WH58vcD7wTeDJzBgQOvctFFV2r8/wZSzj8nxudJh5g1aykbN36XhQsXpt28uilq\nRUgz9/vwZ/lrwCsEQ47rFpBTpZx/AUTfrq+fWbPeyOjoKCMjI2zYsKFpfjKX96fyFpDN1leI7lOp\n30DT9ffwrUX7gW6g/BaQr6el5bVs3LgxreYVR5xSoSQeqNRzQqWSuNmzT4m8Q9J1113fVCVzE5UA\nNmN5YBH7295+hMNrws/y9vAuc+V3nDuhafpbT8Qs9UwqgN8IbAMeLpvWDdwLPA6sBeZWmbd+Wyfn\nJrtdXynwN8st88b393B/Jnotr4rd38O3Fm1t7fLW1tllXwjN0d96ixv8k0r73AT8csW0K4D73P1E\n4H7gyoTWVRiHfx5H365v0aK3V5TM5fsn8/j+Hi4BjCoPnDHjaNasWZPblEiR+jvRrUXvuedr3HPP\nV+nsfBPN8lnOhTjfHOUPoJexR/6bgSPD50cBm6vMV6fvxXwb//N4/NFQ9JFUPn8yT9bf6r+C+tTf\njJssdTm+v/n+LDcKWUj7eHTw31Hx+o4q89Vny+RYtZ/HUf8RavmSyLpa+1sKIl1dJ1cNInlQpP5O\nlrqs7G/eP8uNlKfg/2KV+fyqq6469Fi3bl1dNlSerF+/3ufOXRR++N1hu3d2vtnXrl0b+f61a9d6\nZ+fbyt7vPmdOn69fv77BLZ+67du3+8033+yzZ/fV1N+pvj9ritbfqM9mV9fJfvPNN0cG9Km+v0jW\nrVs3JlZmOfhvqkj7bKoyX722VW5N9WRfXk8O1pIOiJLXFEER+zvVI/lmSnfVW5aC/3zgkbK/rwYu\nD59fDny2ynz12ja5VgoUc+bU9qGf6vvTNpV0QJS8pQiK3d+JU5eVmiHd1QiZCP7AbcCzwD7gx8CH\nCEo97yMo9bwXOKLKvPXcPrm2fft2X79+fc0f9qm+P01J/LzPU7pL/Z1aqirv6a5GyETwj9UABf/C\nSeooNi/pLvU3if7mJ93VKAr+kitx0gFRsp7uUn/j9zdP6a5GUvCXXJlqJVMtspzuUn/j9zdP6a5G\nihv8Z079sjBJUjOP3hhl7FjupwLPcfDgC/T19U17mT09PZncdiMjI+zcubMw/YX67N++vj4OHtxa\ntsyx4/9ndVtkXpxvjiQeFPjIvxkH7qpF1lMXSSjft62tXd7WNrep+1uuHvu3fJmtrbO9rW1u4f7f\nVEL38M2not/HtJl/8UTt2/b2s1i9ehV9fX1N198o9di/IyMjbNy4kQsvHCjs/5tyGs8/pyYa1KsI\nKsfpr5c0xv+PGpStrW0B3d3dTRmgJrofQZL97enpobu7u6kGM0yTgn8KBgdXsXTpMnbvfoJmvldr\n2gYHV9HbexLnnntpQ28PWIT78JY0ehuP3bargBPZvfsgF144oNs/TlWcnFESDwqW80+6FE6ipV0T\nX4TzGmltY5V/BsjIeP5Sg4nGNL/55r/jhBOOb2h6ot4pkTRvuZj2ePgDA8sYHt7Mffd9keHhzQwM\nLKv7OivVc/tH31q0ManLgYFlrF69SuP/xxXnmyOJBwU58p9oUK/W1tkNr/qpd6VR2pVMRR8grJ7b\nf7oD1CVJV//GP/JX8G+AiQb1am8/wtva5jb0P1C9f66nnXIpKeoAYfXc/nEHqEtS0dM/cYO/0j4N\nMNHtGFevXkVHxxtp5E/nqJRIkuvMSiVTKfWycuXHmT37JOqRIkgztVVNPffvZLcWbWR6Kzr9k+/b\nXTZUnG+OJB40+ZH/1G/Xl+8j/ywejdUrRZB2aquaeu3f7O/bYqX3UNonu6Z6u75GVobUY51ZrmRK\nOnBlJbVVTdL7N+v7tojpPQX/jJrO7fpKg3U1auCupNdTj0G9kpTkLQLH9zV7g41V7t84+zvu+Pz1\nVsTx/xX8MyhONURWUwm1yPrRcJIpgqz3tVKcz1UW0z1RilYBpOCfMXGqIfIWUKJk/eKmJFMEWe9r\nSZzPVZbTPVHy8kWVBAX/jIlKB9SaWshDKqEWWR5v3r1aimB62zrrfXWP97nKeiovSlHG/48b/FXq\nmbCuri727v0R5eO6vPrqsyxZsmTSQa7yOCZMowb1SlJPTw9LlizhwIFhDm/rIfbte4qurq4J563s\nb9b7CtGfq/Lx8KsZfz8CSGJ8/nobO/4/1NrfwonzzZHEgyY68i+lATo6Fjh0eEfHydPKr+YhleCe\n7/MT7ofb395e2l+nTNiPPPd3quPh5/1+BEUY/x+lfbJhfF51nc+aNccfe+yxaS0r66mEZjg/4e7+\n2GOP+axZR0zaj2bo7/bt233t2rWT9iOqr+3tR/jatWubsr95FTf4K+2TkPFXPvYza9YbGR0dnfKy\nsp5KSHNQr6SNjo7S3n48k139W++rohuh1vHwo67QztP9CEqpOUDj/08kzjdHEg+a4Mi/SBUGWRjU\nK0m1lgc2w5G/++T9zftnuTI1d9111zdt+SdZT/sA5wObgR8Cl0e8Xp8t0yB5K4WLI0uDeiWp1oCX\np/MxE6nW3/b2I3L9Wa72BX3dddfn+gutmrjBf2Y9f1WYWQuwEng38CywwcxWu/vmeq63UcanP04F\nzqGz80y+9KW/Y8GCBYyMjBz6qRx1X9OJ7nVaeq2rq4vR0dFDVT+V0yb6KZ7EvVRLy3j66acjBvW6\nlc9//g9qqmaq1qepbIs4/a322sDAMmbMMC6++M/Ys2f8AGGlvg0MLGPx4nPGLCOJNteyvZK8F+7A\nwDJ+7ue6+dVf/QS7dx/ur9k84DUc/iyfSnv7+dx8899x9tlns2HDhmnvt8k+w5XTol6LWmf5+9ev\nX1+RigxSPAsW9LJ69aqK/h5O/5x33nmJbNfcifPNMdkDeCfwzbK/r6Di6J+cHvlPlP6YMaNzXFVI\nVKXIRNUjhyuHjj9UiVKqWiifVu/qlLHtaI919BTVp6lsi8n6Md1lRB8JT371bxJtrmV71aNKJfpX\nXPn+PZxoY63KAAAH5ElEQVQimehzl8RnuHJaa+tx416L+n9T+f729oVl/xfHpniuu+76sqHTmyP9\nQ5bTPsCvAdeX/f1+4P9UvKc+W6aOxv7H2e7QGX6Y+hzmjvsyGPtzuvq0UiA9vPx1DuXr6a6YVj0A\nJ5Gjjm7H9NIB0cuayraYfn9rf63Ut/nj9mEtFTFTbXNt26t+aYrxVzuvc+gKP8OlL4Hqn7tkPsOV\n06Jei1pntfd3VrT/8HytrZV9y3f6J27wr2vap1YrVqw49Ly/v5/+/v7U2lKLUuVHkCLYAJwIfAvY\nAuwELmPs+OKvAzomnVZePRIsvxOYH75nA7AAKJ82dr7yn9xj21j9fbX1s3ydh1Nbd945WPNP5uhl\nTWVbTL+/Ey1j7GtB39rb30FLy5t4+eXq65zu+qa+7ae/jMmU0lhr1qzhsss+x0svdQJvBv4S+AST\nfe6S+QxXTot6LWqd1d5/YkX7D883Y0YH+/dfM+61pLdrvQwNDTE0NJTcAuN8c0z2IEj7fKvs76ZI\n+4w/8i8/OlnX5Ef+yS8rO0f+k7cn7vqmvr3qf4Q6ft/U9rnL5pH/ZG2N/1nOCjKe9pkBPAn0Am3A\nQ8DCivfUa9vU1dgrCMdeAfn7v/+RQ69V5iknm1a5/Pb2IAXR0XHyofWUT6slBx6nOiWqHfVYVi3b\nYrJ+THcZU903cdc31e3ViOqiyn3T2npsTZ+7JD7DldOi1h21b6bb1iQ+y1kQN/hbsIz6MbPzgc8B\nLcCN7v7Zite93m2ol/IqBxhbkVBrZU+eqn1qXed0l5VWtU+112pZZx6rfSZb10QVN1mu9plKW5P4\nLKfNzHB3m/b8aQfePAd/EZG0xA3+Gt5BRKSAFPxFRApIwV9EpIAU/EVECkjBX0SkgBT8RUQKSMFf\nRKSAFPxFRApIwV9EpIAU/EVECkjBX0SkgBT8RUQKSMFfRKSAFPxFRApIwV9EpIAU/EVECkjBX0Sk\ngBT8RUQKSMFfRKSAFPxFRApIwV9EpIAU/EVECihW8DezXzez/zCzV81sUcVrV5rZE2a2yczOi9dM\nERFJUtwj/0eA/wI8UD7RzBYCFwELgfcAXzAzi7muTBoaGkq7CbGo/enKc/vz3HbIf/vjihX83f1x\nd38CqAzsS4Hb3f2Au28BngDOiLOurMr7B0jtT1ee25/ntkP+2x9XvXL+bwC2lv39TDhNREQyYOZk\nbzCzfwKOLJ8EOPAn7v6NejVMRETqx9w9/kLM1gEfc/cHw7+vANzdrw7//hZwlbv/e8S88RsgIlJA\n7j7tc6mTHvlPQXkj7ga+bGbXEqR7TgDWR80Up/EiIjI9cUs9LzSzrcA7gXvM7JsA7v4YcAfwGLAG\n+LAn8RNDREQSkUjaR0RE8iW1K3zN7JrwArCHzOxrZjan7LVcXCBmZueb2WYz+6GZXZ52eyZiZseY\n2f1m9qiZPWJmHwmnd5vZvWb2uJmtNbO5abd1ImbWYmYPmtnd4d+5ab+ZzTWzr4Sf60fN7D/lrP1X\nhu1+2My+bGZtWW6/md1oZtvM7OGyaVXbm7W4U6X9icXNNId3uBd4q7u/neA6gCsBzOwt5OACMTNr\nAVYCvwy8FRgws5PSbdWEDgB/5O5vBX4e+O9he68A7nP3E4H7CfdDhn2UIJ1Ykqf2fw5Y4+4LgbcB\nm8lJ+82sF/gdoM/dTyU4XzhAttt/E8H/z3KR7c1o3Ilqf2JxM7Xg7+73ufvB8M/vAseEz99LPi4Q\nOwN4wt2H3X0/cDvBxW2Z5O7Pu/tD4fNRYBPBNl8K3BK+7RbgwnRaODkzOwZYAtxQNjkX7Q+P0H7R\n3W8CCD/fu8hJ+4GfAa8AnWY2E+gguH4ns+139+8AOysmV2tv5uJOVPuTjJtZGdjtEoITw5CfC8Qq\n2/kTstnOccxsPvB2gg/Pke6+DYIvCOB16bVsUtcCf0xwnUlJXtq/AHjBzG4K01bXm9lryEn73X0n\n8NfAjwn+T+5y9/vISfvLvK5Ke/MSd8rFipt1Df5m9k9hfrD0eCT891fK3vMnwH53H6xnWyRgZl3A\nV4GPhr8AKs/4Z7ICwMwuALaFv14m+jmbyfYTpEkWAX/r7ouA3QQpiLxs/+OBPwR6gaMJfgH8Fjlp\n/wTy1l4gmbiZZJ3/OO5+7kSvm9nFBD/jzymb/AxwbNnfx4TTsuYZ4Liyv7PazkPCn+tfBW5199Xh\n5G1mdqS7bzOzo4Dt6bVwQu8C3mtmSwhSDrPN7Fbg+Zy0/yfAVnf/Xvj31wiCf162/zuAf3H3HQBm\n9nXgF8hP+0uqtTcvcSexuJlmtc/5BD/h3+vu+8peuht4X1hJsIAJLhBL2QbgBDPrNbM24H0Ebc+y\nLwGPufvnyqbdDVwcPv8gsLpypixw90+6+3HufjzBtr7f3T8AfIN8tH8bsNXM3hxOejfwKDnZ/sDj\nwDvNrD08kfhughPvWW+/Mf4C1IvD5+XtzWrcGdP+ROOmu6fyIDghMQw8GD6+UPbalcCTBCclz0ur\njTX04XyC/xRPAFek3Z5J2vou4FXgIWBjuM3PB+YB94X9uBc4Iu221tCXs4C7w+e5aT9Bhc+GcB/c\nCczNWfv/mOAL62GCk6WtWW4/cBvwLLCP4FzFh4Duau3NWtyp0v7E4qYu8hIRKaCsVPuIiEgDKfiL\niBSQgr+ISAEp+IuIFJCCv4hIASn4i4gUkIK/iEgBKfiLiBTQ/wcqqVOPcrrI7wAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x104f83990>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = []\n",
    "y = []\n",
    "\n",
    "for a, bs in gambling_agent.policy.items():\n",
    "    for b in bs:\n",
    "        x.append(a)\n",
    "        y.append(b)\n",
    "\n",
    "plt.scatter(x, y)\n",
    "plt.title(\"Policy (all optimal actions)\")\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}