Navigation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigation\n",
    "\n",
    "---\n",
    "\n",
    "You are welcome to use this coding environment to train your agent for the project.  Follow the instructions below to get started!\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "Run the next code cell to install a few packages.  This line will take a few minutes to run!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip -q install ./python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environment is already saved in the Workspace and can be accessed at the file path provided below.  Please run the next code cell without making any changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: BananaBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 37\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: discrete\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import random\n",
    "import torch\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# please do not modify the line below\n",
    "env = UnityEnvironment(file_name=\"/data/Banana_Linux_NoVis/Banana.x86_64\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 1\n",
      "Number of actions: 4\n",
      "States look like: [ 1.          0.          0.          0.          0.84408134  0.          0.\n",
      "  1.          0.          0.0748472   0.          1.          0.          0.\n",
      "  0.25755     1.          0.          0.          0.          0.74177343\n",
      "  0.          1.          0.          0.          0.25854847  0.          0.\n",
      "  1.          0.          0.09355672  0.          1.          0.          0.\n",
      "  0.31969345  0.          0.        ]\n",
      "States have length: 37\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents in the environment\n",
    "print('Number of agents:', len(env_info.agents))\n",
    "\n",
    "# number of actions\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Number of actions:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "state = env_info.vector_observations[0]\n",
    "print('States look like:', state)\n",
    "state_size = len(state)\n",
    "print('States have length:', state_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Note that **in this coding environment, you will not be able to watch the agent while it is training**, and you should set `train_mode=True` to restart the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.0\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=True)[brain_name] # reset the environment\n",
    "state = env_info.vector_observations[0]            # get the current state\n",
    "score = 0                                          # initialize the score\n",
    "while True:\n",
    "    action = np.random.randint(action_size)        # select an action\n",
    "    env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "    next_state = env_info.vector_observations[0]   # get the next state\n",
    "    reward = env_info.rewards[0]                   # get the reward\n",
    "    done = env_info.local_done[0]                  # see if episode has finished\n",
    "    score += reward                                # update the score\n",
    "    state = next_state                             # roll over the state to next time step\n",
    "    if done:                                       # exit loop if episode finished\n",
    "        break\n",
    "    \n",
    "print(\"Score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  A few **important notes**:\n",
    "- When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```\n",
    "- To structure your work, you're welcome to work directly in this Jupyter notebook, or you might like to start over with a new file!  You can see the list of files in the workspace by clicking on **_Jupyter_** in the top left corner of the notebook.\n",
    "- In this coding environment, you will not be able to watch the agent while it is training.  However, **_after training the agent_**, you can download the saved model weights to watch the agent on your own machine! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the agent with DQN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dqn_agent import Agent\n",
    "\n",
    "agent = Agent(state_size=state_size, action_size=action_size, seed=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: 0.61\n",
      "Episode 200\tAverage Score: 3.37\n",
      "Episode 300\tAverage Score: 7.88\n",
      "Episode 400\tAverage Score: 10.24\n",
      "Episode 500\tAverage Score: 12.65\n",
      "Episode 600\tAverage Score: 12.90\n",
      "Episode 700\tAverage Score: 13.93\n",
      "Episode 800\tAverage Score: 14.84\n",
      "Episode 847\tAverage Score: 15.02\n",
      "Environment solved in 747 episodes!\tAverage Score: 15.02\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXe8HFX5/z/P7m3pvRduIAlJ6HCB0AkJVQUpUlQEBRFFiqIY9KviF36CiohYEBAUvxQbIEjoEAgBAoQQQgiEFJKQ3uu9uWX3/P6YObtTzpk5M9vvPu/X677u7syZmbOzs89znnKeQ0IIMAzDMNVLotQdYBiGYUoLKwKGYZgqhxUBwzBMlcOKgGEYpsphRcAwDFPlsCJgGIapclgRMAzDVDmsCBiGYaocVgQMwzBVTk2pO2BC//79RWNjY6m7wTAMU1G88847G4UQA8LaVYQiaGxsxOzZs0vdDYZhmIqCiJabtGPXEMMwTJXDioBhGKbKYUXAMAxT5bAiYBiGqXJYETAMw1Q5rAgYhmGqHFYEDMMwVQ4rAoZhCk46LfDP2Z+irSNd8Gst37QLry7aUPDrdCZYETAMU3D+O281rvv3PPxh+uKCX+u4X72MC+99q+DX6UywImAYpuBsa2kHAGza1VrinjAqWBEwDMNUOawIGIYpOGT/F6Kk3WA0sCJgGKbwEIW3YUoGKwKGYYoGGwTlCSsChmGYKocVAcMwZcsX75mFs+98Hcf9ajo6Uvo5CM1tHZj48xfx+uKNxufe2tyGppuex9xPt+ajqy4u+9ts/Ob5j/N+3kLBioBhmIITN1j8+pJNeGf5Fizf1Iytdgqqio/X7cTa7bvxi2c+Mj73W59sxsadbfj9S4uidcqA5xasw29fzP95CwUrAoZhCk6hY8UJ+/xpDkLEghUBwzAVT8LWNOkIJgdxJlMGVgQMw3Qa4lgEPLeBFQHDMEUlvtQNEtjSIhBRLILYPel8sCJgGKbgEKSgLsz5E7Yk49F9PFgRMAwTyJptLdja3JbTOfLhjhcB1kScGEH2vMVh3fbd2LzLfx/TaYGFa3cUqRdqWBEwDBPIETe/hCNufqnU3QhxDVn/owWL5XmLowoO//mLOPjG533b75qxFCffPgPzVuZ/PoMprAgYhgmlpT2Vl/PkInODhXxhXU+F5D17QtuqLS0l6wMrAoZhCk4+ArMmGUHloAeiWhhBLq9iwYqAYZiikYvQSwdqAmtfLNdQ7B6p6Yg5q62U0xoKpgiIaAQRTSeiD4noAyK62t7el4ieJ6JF9v8+heoDwzDlQV6CxQHyVe6LEyzON6mIiqAMulxQi6ADwLVCiPEAJgK4gogmAJgK4EUhxBgAL9rvGYZhAgkS8nJPWl+XrmjEtQhKScEUgRBijRBijv16B4APAQwDcAaA++1m9wP4fKH6wDBMeVGoYLHc5fTPh/nqqUBTylKpuB+ydL6hosQIiKgRwEEA3gQwSAixBrCUBYCBxegDw+STf779KV75eENO57h35id4Z/mW0HaptMBNTy7A+u27c7peHIL98uZ4he6fXlmC+au2ReuL3ZWOVBo3eu6HVBLO3poqnXy7ZjpCzJJtLe244YkPsNvOxFJdfltzOz77u1dx58tL8ts5DQVXBETUHcAjAK4RQmyPcNxlRDSbiGZv2JDbD45h8s11j8zDRfe9ldM5bnxyAc6+8/XQdq8v2Yg/z/wEUx99P6frxaE9z74WKfRuefojfPZ3M6Mda0vsGYs24N6Zn+Anj3/g2Gf9d1oNofGCAg3Aw2IEv35uIf76+jI89u4qd3cc/bnjpUWYv2p7pLLauVBQRUBEtbCUwINCiEftzeuIaIi9fwiA9apjhRB3CyGahBBNAwYMKGQ3GaaskT7nqEHIfNAe283hIQ9CV378ljZLOSUc0ktksoac20pDWIygtd2tXFX6qiZRXDdRIbOGCMC9AD4UQtzm2PUEgIvs1xcBeLxQfWCYToEtKEqRXtjeUbroq9fHL0f40qVSX5N0tPUfY+wayqGPKsIUdsruWNLzhTrfNdQmUUxqCnjuowBcCOB9Ipprb/shgFsA/JOILgGwAsAXCtgHhql45Gi3FKHE9oDlIeMghPmEK688lYqg1VZO9TX+cazbIggLFss+5VcVhFkEMu6SCBj1d6nrJIpACDET+md3cqGuyzCdDemmL8VCKq15sgicPTcP4grPe9knyyKocyiCTLA4hkWQb1KKuIqzXxmLINN9f0e7FNki4JnFDFMCooxCZctOYRFAZARhGFEsAnnKUsRRvKgsAme/5P6E1zXkeN9Q61ByRfhMrAgYpgRE+W1LpVGSGEGegsVOIWc6+9fbTt4zGWx1xQjkf6drKOAy23e3x7KwTBR4h+KeOZVD2qMIVKd0WjvFmKDGioCpGFJpgcap04qWUldIooxcsy3zrwl+/dxCNE6dph3559siANSC79y73kDj1GlYtbUFjVOn4dE5K31tpGJoS/ldQ1JAO5XHgf/7HE687RU0Tp2Gn/03m2o6Z8UW7H/Dc/jdi4uM+nvny0vQOHUaXliwDqOufwpfue8tNE6dhg7HvfnD9MVonDoNzW0dyu/WuU328cqH30Xj1GmZ97pvtxhWDisCpmKQQum+mZ+UuCe5E6UmjmxaiIzCe+17qYsFtOVbEQj1Z3/rk80AgIVrralGT7y32tdOZOIAcP0HstaC84jWjjQWrd8JAPjLa8sy29dtsyaizTaYzAcAD8xaDgB4/L3VAIAZ9kRC50j9b29Y59/e0qEcwTu3eW+pqr0zzBA2QS0fsCJgmBIQbZRXQtdQAYLFQZ9dulVqEgmf5eAV9u6grN8i0OG9vKlO7uoJ4KY1gWlTiyCovbMNWwQMo6D04cDcMQ2YAlkhU6jaOEHkbUKZjUBwfESOjmsS5I8R2Pvkf+dIOjuzOLwPUe69E29Kp0pAE6lH8M5t3uPk53BudXaRYwQMo6ITaIIomSCyaSEtAt2ppTsuV7eUW8gFWARSESTJ9zXL+yAFqVOgZoPF4fc17rwB7yQvp7x3njGqRSBjDbryGGwRMEwnJVqwuPCuIV1vZIygJpmbqHALOX076e6pSRCEZ2AtBbgc0ce1CHyxB8ORRVevRaBQKAT1CN6ZSeT97uV7oblHbBEwjINyWMAjX5Sba0jnW5cWQW2eTAIhRKAfXwrMZCLhE9BSHqYVFoFqQpmOuPFvZ26/9/quGIHCneZs61UE0v3m3OyyCPLsnlPBioCpGPK9tuuKTc14ffHGvJ7TlLBEkGnz1mD77nYAjgll5G+zw27jZPrC9VinKVm9tbkNz8xfk3mvysBx0h5iESzZsBO3v/AxPt3c7Nv3xpJN+Pc7K+3P4BDaAULRHSNwny/tswiyNzEni0AASzfszGQuOVm3fTdWbbUWlfcq4rQQeOXjDVizLbvo/NKNuzBr6SbfeZyj+jc911mwZruvX06FxllDDOMgI7TypBCO/dV0fPHPb+blXFEJsgiWbNiJKx6ag2v/+R4A54SyrCD6aO12XPHQHFyvKE391b+8jbP+qC5vfc0/5uLyB+b4BbdOEXRkBbOKyb9+Bbe/sAiTbn3Zt++Ce2bhe/96D7taOxzfnVtYP/TmctcxrhiBpuicHNG7YwQRYi4KbXHCr1/BuXe94dt+xu9f014jlRa46L638Jk7ZkLewPPvnoU/K9KbTVyBbgsn2rG5woqAqRg6kWcoMFjc0mZNmFq1pcW13SmKN+9qAwBs3NmqPK8cxXpZt91qv63FbUno3DXZGEGwayjIj92RcotQ57U22Z9DkkplYwS6NM9M1pDTZRLh4YgiV9e6Fr9x75MCerPnM6gwUVTueRFOi4AVAcP46AyxgiijvMzMU4cslsrCW5wsTGh0sf3cLXYpZ4nuqIxrKBFfVKQ9cQHna2+9Hdl/dYzA7RpSZQ2Z4LXGTJ8n73emm0egwuQauoA6WwQM4yDf5YLjko9+xAsWZ5GCvGudu4BwmNCQ7ZvbPIogJFgcZhEEkRLCFYtwXsp7VrdryL3PGyx21e+JdD/jZQ15rbhImV9GisDZni0ChlFSHmogPxaJyTwCOVjOKALH6FkKcm9uuwws6lJNZQpkS1uHuz/aYLEcocdXBGnhFrVOAertp9xXk1ApggCLIOR2Oq8Tt5qnVyC7LIKQY02UjX4eAQeLGSaDM+BYSuLOTI17DlXWkHQN+XLbNSWOJbK9VCRSQOkEVZtdYsK7mlYUhEDmy7OCxU5F4HENpbKKQFdrKJWxCBxZQyF9cJW4iPn1+UtDmB9r8nVr5xFw+ijDOCjQ7yHqCDGKG0JHkFtBNxJ2pi9mXUPqGIFuAN/Fdg15YwTarCFb2uXyiVNpb7A4qK2teBIJ7czitNIiyME1ZOiP100Ei3p9Hdp5BEVwDRVyqUqmQvn632bjkD364PLj9ip1V1zkK230p4/Pd73vSAv8+tmPsK25Hau2tmDK+EG46MhGAMBfXvsE0xduwN++dhgAq9zw28v8+eZOlm7YiQvumYUnvn00BvVsULYxsfYzg2XheY+sReB1DUmhoau1L4PLP3psPrrXZ3/+eteQrQgUgu6Wp/3lwBes3o6v/fVtXHrMKMe5hUvZCpdF4LmeI0bgVdDfenCOp2/W/ov/8paBayh7oSDB+u2H5mDsoB54fclG9OpS69r3x5eXuN6bDAjmrdyKG574APsP7x3aVhd8LkaMgBUB4+P5Bevw/IJ15acI8vR7uP8Nd+56Ki1w1ytLM+9fXbQxowh+9t8Frra/enZh6Pn/9sZyrNveiifnrcElR49StonmGpIWQRa5gLs/RhBsEdQ6gr5X/31uZrasTslmCqIpdv/plSXKbWu378ZN0z7MbEun3bWAnHJNNUkLMCunIdu+vHBDeGPXcfp9T89fi6fnrzU6j6vEhabNR2t3YM6KrZizYmukfkklePbBwzGwZ71Rf3KBFQFTceR/sfH8BuOksO0IcCIHuoZ8E5es/wlFsNgXbE0Fxwh0VSp0t1QKJNM7rlJAaWfWEIKDxVIxCBE+4o7iMnFexl9rKB5B32+2TTyXlfxovzpn/8BF7vMFxwiYikF4/ueLfPtgk3bOfZBJb+JWkEJROY/Atgi8p5FKTacIdPWKdP3x+uXDUF03JfQTyryt5eHptAi1AON+b/laA1hV9M5LlBXenP2KYhnlA1YETMVQqHkE+fbBZi0C88BjEEETynSLnEQVILpbK11YprdeFZsQQri+O9c8Ak0/0wYWQZSgvSt9NE9ft0mwuLUjpdyuwjuPgEgf68k3rAiYiqFQIbN8WwQ1GYtAPxqMMirN9s+fNeQVQNkYgcYiiOgaUq0FHITaNaQvn+C1UJz1pMKuGM015AgW52lA4Rzt687Y2h7BIvC4hrTuvQLAioCpGAo1sTjfFoGchRt03ijCSJ7GKRea7Qlhuvo3Oreyd3NYIb9srXyzvipdQ2mHWBfuz+5tnlU84RZg3PR633ljnsdEEenWglbhVZZFCA1kYEXAxGLttt2ZMsm5sGZbi7KUsorM5CfHD2bB6u34eN0OrN2mLrsMAJ9s3JWZGKXCpN67bgQvhcGabS3YvrsdSzfszOz7aM12LNmwE5t2tmL9Dnf/lm3cFXpNAYHF63dkrr2tpR3rd+zGonU70NIu0zrdx8xbuQ2AuUUgBdXGnW3Ysbsdq7e2YMOOVsxZsQXtqXR2nWDbvbN4/Y7A715VkmhrczvWbrOK3S3ZsNPnMnMK1JV2oT0hwmME6bRQlr9WYn/ujTtbsWGHu1Bf3LTk9oCid3V22e6P1u4wPt+bn2zGkg07sbW5Dcs27cr7MqFBcNYQE4uJN7+IwT0bMOuHk3M6zxE3v4SRfbtixnWTwht7fhctbSmcdsermffLbvmM75CNO1sx6daX8eWJI3HT5/dTntYka6gjLVCnGKLd/sLHuPakvXHEzS+hoTaB3Q5XwPSFGzB94Su+/nWk0vjx4x9oryUF4PxV2zHlthk4duwAANb6A9PmrXG1dboTmts68MPHrLLUOt+yLlh89p2vY49+XbF8U1awfvHwkdkFXwDc//oy3GCn0w7ppZ4fobruBffMyrz+aO0O/Ob5j13tb38h+37a+2synytswJ1KCxzzy+nBjTw03fRCpPZh15fsaHWX7KivSaAtlcYLH64zPt8LH66L1D6fsEXAxGatZvGTqKwwHNV55YJJIE6O/mYv26JtE7VWvBPnIiO7Df3BzY5ZvbUGxdw+DhhVOt0cLY5CcnHcCk4lAACvLNyQXTBeCMxbtS2zb43HAhszsLvxdd9yTMgTQign6AkRPlKPFCw2bmlO0AAiSpG+d398Yj66kxOsCJiKxcS1ryvO5sQkRqD90Rv4sr3scoweg/olaQtIQXRe2ams8hFo3N2ecriGgusNhdU4UrUFgtNWwwy1KLGdoG7FjT0FZYVFuf/dG0rvmGFFwFQMuho8QcgZuN66/U5ysQiAYEGtYldrduSu6pf3SkHxDd0CJrqReRR/eGtHOhPYTYvgUW5YtpKOVFrtrhIiPG8oSo5+IQh6JqLchmJmB2n7UKgTE9F9RLSeiOY7tt1ARKuIaK79d1qhrs90PnQLlQQhLYIudblZBEGBu+ZW81xxq08RLYJARZB97Z6xqxYuUUa/rR0ph7UjAktRx52/oLcIwoPF7REycnSxkVwIem6i5P8XMztI24cCnvuvAE5RbP+NEOJA+++pAl6fKXOi5u/7sv4MDpf59kGKwKTee1Bfd3oChWE4LQLdWsBOgiwOnUWgOybKHW9PicznToe4hsJmNOvQ3Vchwr/ftohZNToXXtzcnKAYQZS7UKxJY0EUTBEIIWYACC7TyFQ1UU177w/WyDWkWdLRiUk9GN2PXkD4VvsKw2kRqD5DpJiDyyLI9nG3pk9RS2g700eTActVhs1fCDtOdd2wvkZ5fojMg/mm5CtGUA6UIkrxbSL6CoDZAK4VQujTOZhOTVTfulNALly7Aw/MWq5s9+qiDVi3vRVnHjQMP7ZLTgcpghunLdDuk+gE1tvLtuDBN9X98NKeSuOXz3yE4X26OrYJXP/o+9jW0oZvHLsXHp+7GlMmDDQ6H6C3CLzrDcxaugn/fPtT1Cajjf0WrN5uXwcIOjRujCDQNRRybBRF0NyWwh0vLYrQs3D+90n9c1NheqDoiuBOADfCGsfcCODXAL6makhElwG4DABGjhxZrP4xRSSKjxdwuwo+9/uZPt95Oi2QSBAuvPctAED/7nWZCVNBAnD+qu2h1w7yB//tDTNF8PyCdbjn1U9c21ZtbcHDb60AYKVr7mpLYdVWw0lS0K9kdfiefV3tfvHMR3h3xVaMG9zD+Nyyf4CBRSCrnkY0CfSuIRFoETTUJiJblHe+7C+bHdSHXKg0i6CoWUNCiHVCiJQQIg3gHgCHBbS9WwjRJIRoGjBgQPE6yRSNXGZOqgKo3rIN7iURzc+tCormQ1h4BddRo/u53u/KFJIzP6duJav6GrcFJFNW42baCBEcz5D9UDUJDDIH+O11euB7J43F2QcPz9vM21RaHwiPqjglFaYHiqsIiGiI4+2ZAObr2jKdn3yn/3mFdVimiG4ylyoomo91Y70j3BrNCNskgCzRrWSlq0oat66SgJlA19Ua0pFOC6XQTHsqljohosC+RKU9ldYGwuNep9IUQcFcQ0T0MIDjAfQnopUAfgrgeCI6ENZztQzANwp1fab8iR4jCN4fVcg11CbRnvJn/Kjkcz4sAm+8WaeIorhXhMYi8HZXNovqjsueT4RYBNb/qBkwukcgLfTZPET5db10pIX1nSvi61GUspNKcw0VTBEIIS5QbL63UNdjKg9pEZj+1sLChybF45w01CaxY7dCEagsAkXWUDJBsdYVkOgsgqA0Tf85s6+dffSOpuW12+NaBCJYQYkA11AQujiAEEJb6I+QX4sglRZa67EmYnBdUllqgGcWMyWkvcP6oesEopdwiyDaaFfrDjB0b5jUCXLiUwSa4yO5hqCzCLyKwPpvsryiijCLIDOhLKIIDJxHoDmGyMxlY6pP21NpbawirsKpNIuAFQFTMtpSli1uWqArbCwbuA6wYpfux68a+arcTlFTMb2n0B0fxTXktgisN3XJhM8NJRVD3FhH2FGZ+QYRp2elhD5GoLMWCGaC1vQudjgmznmJquwz164sPcCKgCkcKzY1o3HqNDwz3yot/L1/vYfGqdMy+9syFgHhkXdWonHqNKxTVDRtnDoN5931Bibd+nLg9TrSAv94e0Xm/Vf/+nZge10wcltLO6YvXO/adv7ds3zt6iIqgusffd/1XjfC/vc7K43P6YoR2EK+Nkl4Y+kmNE6dlvmTVVijxmUkbR1p3Pz0R4FtHpi1HLe/EC1XX+fOSwvg7299qtxnWQTh5zaNV7Sn0lpFEJQym49rlwusCJiCMW/VVgDAE++tBuAXcFKIERH+Mdv60S/doF6wxVnuWUcqLbQ5/arfZZAF8fT7a3zbvIqjria3n0+uxwPuALS0CGoV55XzKQpZqO2hN1eEN/Kgs8qCVuhKEBnFUXTHnzRhkOt9UImQ2tiuIeCaKWO0+68/dVzoOf5+2cRY144DKwKmYMjfuM5vLBRvcokBdqSFb0ZtEFEzgUxdO6bkQxGoYgRBlkoB5k5liOpPT1DA2s0ieJ6JyUhd99x98fCRGNk3O7s7aDnJKOsKeK892l6jQcVgzcI+TiYM7Rnr2nFgRcAUDPkzNlkwPe2wDuKSSqe1NXZUA8+g4LNqnzcYrfIfRxHuebEIFFlDuSqouERVBDXJhBUjUAjstBBaN5Y1jyD8/LpHyTQ5QfYxDkTBgXOT55wXr2c6BSJEuGeFf1ZpFMoiUAUxo8478FoQdTX++kX1hsKdKHqMQYVqZnE+FEwcoubc1wak36aF3o1FMAuo6+RoImEe1I47j4CIAgPGJmctZpSBFQFTMLKuIc1+x+t8WAQdqQBFYDDCDz2/VxEoLAJvaQcdyTzNjlXNLI6b6ZIrcSyCoKJzOkWQILO5FroReZLIeF2G+OmjwYLc5DFni4DpFMhRl941JOz/zpmp8a+XSgttqWFVhlDUWjXeDBfVyNvUIkgkKPZo04lqZnGpXENR/em1Sb1FIJCdZ+LFtMSE7lnSHav6PmpjZg0lQi2C/M2DyAelXyyT6RS8sWQTNu5sxYShPbHXACtI5rQIPtmYzQbasbsdry3ehNnORculRQDg8bmrsEe/bmioTWDcYPOA2Ssfb9DuSwtLaMoMpjC2trTj3pmfuLZ5LQKVwK2vNRMclkWQu8B+/L3VOH5vq2z1Y3NWaftVDF5bvClS+9pkAut3tOLdFVt9+56Zvxb7DeulPC6sxERtktCeElpRm0ioLYL+3eux1pO+nMxpHkFQjCD8HMW0CFgRMJFRja4vuCebZ7/sls/Y7az3ROSaA3Ddv+fh6flrXcdLGTtt3hr82SGA5blMuO35j/V9hsALH67H1X+fa3Su5xesw/ML1rm2eUevSkVg6hrKm0UAXPMP92cqlWvISa8utTh27AD8N0Dx1iRJqQQA614v36ROJSYEu2ySCUsR6ARpktSWSO+utT5FoIvjDO3VgNXb/HNeJKfuOyT3GEERv0Z2DTGRMfWvZrKGPNtXbmlxvbeCxVbrTbvacuuchrQAtjTndm5/1pD/52MaqE0YlkmIQ67nnTJ+EI4bqy/93mBg9bz305PwuwsOwp+/0gTAGm17CXO7aNdqJgoMFmeyggJcQzKW9H+XZCvh92jwj4u7KpY4XXbLZ/Da1BNw5F7ZMuIPXHJ45vWhjX3wzeP34hgB07kx9azrsoZUvmQpYwuxSIjsS64/K3/WkCpYbOgaSlDmPuRbIURJj1QRZlFEWe0y2xX/QWExBd2jEBYslufVCdIEZRVBz4bazPZu9X5FoNoG+OMUztfy/gQL8vgT4goBKwImMqbr3mZdQ+7t3pG0FSwWkc4dlYgJQkr8WUPxg8XJRFaQmB5jStRVwnzHh4xEo3xDQVlgYQprd4c6A8yqPqo/TrrcgoLF0tro1SWrCLorhL7KItCdUyKfk0DXkMFXVMwyFawImMiYu4ayAWAnQfnzhZr4KiBy/mGZxQhMXUPZGEG+FUHOsYeww6NYBPY9Vz0zqlIYrstorhMWLJZCWRsjcFy2Z4giCLqXzufJeU7pQgx63MqtOikrAiYyuVoESteQtAgK5BrKx2m9lTtVgsw0RmBZBFZb0wCzKbkKmXA9YH4zg3RS3Do+YcFiaWlos4Yc98cp/FWKwBTnOeVzIlNEVV9HeakBVgRMHtBV8cwGi92PvXckTZRVGl4lozt3VPLhcvLFCBQWgWnqpssiMEw5NSVXiyDMcooUI5AWgWJf7Do+IYF2eV6TeQROxd3DES9wXUzXD805My7EHF1DxYQVQSfl3pmf4FsPvuPatm77bhzzy5cyaXkrNjXj6F+8hLUBaXAAsHFnK4795XQsXr8TgFsQ/OrZj/A//3EvPX3iba9gd3tKaxGoBGg2RuD/HPlAiNxHYZ/7/UzXe1VQNenwT/ftVqc916qtLZm2DXm2CHINPocdHi1GoH4NxJ/vQESBVo9UhL27qu+/7ti+3fXfl7ofjteOpyu7SA9c/3XHlgOsCDopNz65AE+9787Vf2Luany6uSVTqvmBN5dj5ZYW/Gfuqkwb1Qj8uQ/WYcXmZtw7cykA9+j6D9OX4EFP+eFF63di8fqd2pnF3vfO1ai8179p2ofBH9SQMMtiiEE1SC9eN9B1p+ydEUIJIjz6zSMDj49qEfzxSwfj2hPH4qCRvQPb5aoIvEf/7WuHud5HsdKyMQJF1lCBXEP7D++Ni49sxM/P3A8PXXo4fv/Fg3D15GxJ6GSC8OSVR+O2cw9wHXdu03Bf6WgCcM9XmvDtSaON+ydLYwRZVl4r+fErjsJvzjsA3z1xrPF18gkrgipCmswdKXcwyynYVb/x7H69ma86xjmhLIysa8jg5DFIi+BR2HWn7B35nN4R7ZcO3yNrEQBo7N/Ntb9Xl1qXlRA1a+i0/YbgysljQkfSOSsCz43af7h7hm+Ur8gkqBuVMIugW30SN5y+Dw4b1RdHju6Pz+4/FN9xCNhkgrDvsF446+C3yTPzAAAgAElEQVThruPqkglcM2WsT0GdOGEQvney//lwtnJ2x2cRGAQJDhjRG2ceNBxXTdavYVBIeGZxFSFL6soFzFUZHaofubcyqMmIUJZ0APwjTNVcAalsCjWPICxGECf33iuQnZPEVIKqoTbhUnRZRRDNNRQ2ks5dEQSfL1qMwD5Gua8wFkFYQTrddaXADnJnqdp7MUofDexh8TF++onoaCL6qv16ABGNKly3mEIgszTa7Rxq+TAKl0Xg/8lmJ4ZZ701kdVpkc0u8PwhvVUmiws8jCDtrnLIM3liHa+Usxem61LoFvryPUctGhwl6k8qcQXjdFrkoliBrML5FELxUZVgNp7Dr5ppmnLG4ZdZQAa6Rb4yeQCL6KYAfALje3lQL4IFCdYopDNIiSHksgnSIRZD2tDfxDQina8jzU/CuCCVEdsJXgfSANbM4KJUxRuDSqzwSjtIHKlnTUJt0fb5We8JU1HkEYSPpuIXSJN7T55KOmrUiFftil3gOdg2FfZVhitLl8gkqHOd87XjTkfm9mF2jHDB9As8EcDqAXQAghFgNoEehOsUUBulSaPeYri7XkOIHm3UNyRiBmWtIju69P4igdXML5hoKmVkcZyUq78I0RI5ZrYqfehfPLNVWu2R2VEVQeIvATW6KQB8sjtvPsPTRMAUT5gU0/by6rKHMfJNAt5LRJYqG6RPYJqxvUgAAEXULac+UIXLU2+HJali6cWd2bQCFkPfKZhNZvWN3O9Ztb3VdR7Jo3U5f+1VbW+xzF0YRpITQVroE4k1uimoReF1D0jKKGiMI62quE4u9gjCX80mBrfpWc3E5BdYaCllvIOy6zt2mAlsdLA5QVmWmCUwVwT+J6C4AvYno6wBeAHBP4brFFAKZNdSemflo8fjc1XjAkwLqRAi3a8gkWPy1v87Gn15ZotznrTC6raXdca3QU8dizvItmbRZFfEsAn+wOFvnxv9D32doT5y23+DM+zGDrHUbjhzdz9fWy4i+XRzXKayPOyxY7KWnomqn6lzes6zc0hyxZ/KcwdVHdUri8wcNs/bnLUaQbeesrnqq/R1nwkWK05WXGjDMGhJC3EpEJwLYDmBvAD8RQjxf0J4xeUeOYFO2n8QpUOau2IoLJ+4RuMh7hBCBiyhyKa5F0LdbHTYHlLAOK0EdZ5arP2soGyz2fuanrjoGew3shiQRvnrUKPTrXoeeDbV44/oTMKRXl8B1Eub+5ESX1SAF2beO3wtfaBqBmgThmF9Od/UjiD36dcV/rzwa+9/wnHK/f95HdsO9FzXhkvtnAwBevPY49OpSi/dXbsNX//o2AGDP/t3w2BVH+fui+Fq3NLf7NxoQVsJbpyRuPms//OCUcaEWWJDy0jGgRz3e+uFkAEAfO0U48Ngy0wShioCIkgCeFUJMAcDCv4KRP0qT9DYn0l0kf19RhXUUMziuIujdtTZQEYS5s+IsSagqlZHJWPFcb8LQ7EproxzzC4b06oIwvDNk5f3s3lDjOld2f/D5utXVuMov+9GfoHfX7HGDezagW32NK8bRvaHGVdEz6LvfrVlfOgxCSLBYs682mcCAHv51EbzEiREAwMCeDZ79zniR+4EwWaqymIQ+/UKIFIBmIlKvG8dUHJmZj45tUtirJ5RZ/6NkDTmJ8sjHjhXn6FKKYxF4XUNWjXrrdaFiHUB2xKsTJmHB0jA5F3S4U0gmFW4wr1slaB6Bbn3pMMKCxfnMmsplLkDwHIRIXSo4phPKdgN4n4ieh505BABCiKt0BxDRfQA+C2C9EGJfe1tfAP8A0AhgGYBzhRBbYvWciYz8McqsBpXAUAeL3VkQhXQNxS0yl6vYzUf6KJC1CAo1QxoA5GV1sjDsfuey36kI5GvXAi0BbiUv8S2C4GBxrllT+Qrkku+FYl+ZYPr0TwPwYwAzALzj+AvirwBO8WybCuBFIcQYAC/a75liYQundpWUsjcFxQiy8w6iSbkowctUTEWQSoucslDyMaEMQHEsAk0cwrs/7HgdQW4L1apcztuuyzhSKfjYioCCU0BzFeSurKEcFqCPMyu5VJgGi+8nojoAsmDHQiFEYKRHCDGDiBo9m88AcLz9+n4AL8OaqMYUATnaVwWLhee/6zhPqYioMi6SayjmSmIdqTSSCfWi5CbEyRpSWRHSIiigHgh3DYUJqJDzm7o05HVIYSXo3jvxrvhmineZyKA+xsMwRhB+J31nI7Ir4ZaXHjCeWXw8gEUA/gDgjwA+JqJjY1xvkBBiDQDY/wfGOAcDYP6qbfjLa59EOkYK2fmrtgMIfxj//c5KTJu3JuPmuGuGVX008s+3CFlDHWmRU757vHkECkUQM6AeBV1mkkQnfGuykjvw/EHC27lPKgCnUNbNSs7n3Qh1DeWzDHcOp1J1I2PNxT9tQTAdBv0awElCiOOEEMcCOBnAbwrXLYCILiOi2UQ0e8OGDYW8VEXy2d/NxM/+uyDSMd4foytYLCeUOQTY9/71Hq54aI6rINyu1o7Iq4hFyZCIKz9TaZFTJkaccgeqxeuTSRkjKGSw2Pqvcy/otkvFJfee2zQcvzx7/0husWSCcO9FTa75EI39umLfYVZW1A2n7+Ppi/5c913chNMPGGp03cuP28txTvc8gi9PHOlqG9U19KPTxuOqE7JlpnXH33zWfrj4yEZHP4LPq/oenEr84iMb8f/O3DdSXwuFabC4VgixUL4RQnxMREH5ZzrWEdEQIcQaIhoCYL2uoRDibgB3A0BTU1MBDe3qwVtcTvXAq11D2ddxfPgmMnbinn0xa+nmyAJ0SK8GrNm2O2eLII5fWVWxVP7QCxksDhtV6u5DXU0CLe2pjAD75TlWPf69BnbD2Xe+kWkXHCwGJo8fhMnjB2W29e5ahyevPEbdXs4sVtyPE8YNwgnjBuGJ91Znzu28b3XJBNrsDLcrJu2F5xasxdINu1wWQff6Gtz0+f3wwKzshMioz8HXj93T9V43j+CCw0Zq26lQ7c5OMiOf0iwlphbBbCK6l4iOt//uQXiwWMUTAC6yX18E4PEY52Bi4vwtev2UwrHdd5xjY0dKRI8RGPww5Wg1qqJxls3IJQAX50iVCyLmoluxrhvVNSTTXf37zf36URWmu1xDRJeU51jnaFoVqM6cJ2fXUH4cN6qZxZXuGvomgA8AXAXgagALAFwedAARPQzgDQB7E9FKIroEwC0ATiSiRQBOtN8zRcIpY9Mai0BlEjhHae2pdPSsIYPHXgr0qEpGujU60iKnH1ecH79aERReE6jWkXDvV2+vTaiFkLd90J2IrghkjCD8i/XHF9x9cq71IIW96jvIVZCTRwFp24U8car9qrkX5YCpa6gGwG+FELcBmdnGgVP0hBAXaHZNNu8ek09criF4LAKZPqr4wTq3tXWk8zKPwGn2A9lAZlQlU+ssrZ3DMksUQ36rBE4xLIKwNF6dkEloLAlv+yAZFTUQG0XeeWfgOgVpgijbT8q6hlT9yVXEGvc5NEYgmzk/h9GhRcf0sX0RgHMufBdYheeYCsIpNnQWQdDMYsCyCKJO+lI99N6ZvLUxg6yyvEGuFkGcY1UysRgWQdhcBd2IOLuMpkfwe9oFjoIj3qgw6yX4WPd15ecmV5v8WwSuzKiczuQnzK1XKkyf2gYhRKZ2sP26a2G6xBQK77oDqhiBCmeWUHtKRA+EKp56b6lg6eJp74hnEViXif/riiM8grJCConsq25ZB92gPeEYUSu325jOLDYhH2sZSLIxAsrEkpSuoSLFCMJaqWIEOmVcakwVwS4iOli+IaImAC2F6RITBSEEXl20AY1Tp2Hx+h3BbR3iXghN+qjiODl/AACe/WAtptz2SqQ+3vHiIt82bw6+fL92++5I55Y/rDEDu+c0yoojsOprVRPKiqAIFG60cYOz60Tp1+S1/2u2S4b17qIsZhd0bm1f7eYm6n38EM9aVx6LIOGIcchZ3c7P7b1mXJyH9w8oUrdHv+CxsErYh80KLxWmXtVrAPyLiFbD+k6HAjivYL1ijEkL4Mn31gAA3l62BaMHun8Y1hKNfvM87dgOOLOGgn+yj8xZmXunoVAENQkM690ls0BNFP51+RHYs383TI6ooJzE+WH2bKjFQ5cejv496tHSZpVLyLciePLKo33bkhmLIPtdPfT1iTj4xuDiwA12+WXdCnFdapP47fkHYsr4QWhPpzF6YHccOKK3q03UUhwqq+nJK49WFvm7/2uH4cD/zX4Gt4smmzWUIMKAHvV48NLDsf9wfy3MfAWLrz91HI4bO0Db7popYzGkVxfsN0xdj1OleMttQRpJoCIgokMBfCqEeJuIxgH4BoCzADwDINq0VqYghAlupwvIOYJMe9fwFa5/WjLL8OWIVxAkiXDU6H745+zoiubQxr4AcivrEPf3eeTo/q73utWx4rKvQsjISzgVQd9u2VLVutiBXCqzuc1d48c5Sj1pH2uiWH0iiS9P3MN3jtrI6yvbLxxdGtW/G7rV+0WPt9y2O3vHP5o+ynPvM9fMk2to0rjgwge1yYTyHnnP46RSYwR3AZBF3o8A8ENYZSa2wJ7sxZSWtMi6fFTPlk42poX6QQ0Tph1xiwF58FoEyQTFqgCaL/I1UstVCEW5hm4QoPsKu9qKoMVT7C3KR1cV2gsil2CuN2hram3la6nOXGeHqz5mZlZ4mcUIwlxDSSHEZvv1eQDuFkI8AuARItIvqcQUDaffX/XgpYVAEorMDW+MQK5HEGIT5Msi8LoYrB95boqg2DECFfm2CFRkXEM6RaD5ihrsNZNb2tSKwKTnUZW16r6a3mpnM2ehubDCgvlyDeVaJUT2IqgoX7kQ9q0miUgqi8kAXnLsyyFrm8kXYQ+rK1MIbteQ6YQyJzr/clRUFkF9RLdDPsmX/C7GDz2REYjq/TplnrEINK4hE+LOIwgbsKiPdQpQdZBcRb4sgriVbCVKiyDCBLtiEibMHwbwChFthJUl9CoAENFoANsK3DfGgFBF4BT+DsGR9qQNZSeUBRO3dLAXb9nnBFFRRtM68jXTsxhZQ8kw15DmK+piWwTNXtdQ/rrmI0gYhuEtT2G6VGqu32W+LAJ5Z93B4nydO78EKgIhxP8johcBDAHwnMg+eQkAVxa6c0w4aRFc+0cI4KO12yGEW8i3tKfw9PtrfO1nLd0UeD1voDEu3rLPyUS8VcLKjaKkjyqCxSbIYLH3uEIaMaRyS5oe6+lXNlsq+Lh8TSjLddSu6kbS0KopNiZrFs8SQjwmhHAuUfmxEGJOYbvGmOB8nFQBKCGAU25/Faf+9lXXCPLnT32I6Qs3uNoBwNV/L07ox+caIkKtoqxzIRnSqwFXTR6Tef85T0nkYb31C8vr5H1cRXDMmP4Y2ddsjmYiNEag3n76AUORTBDOP3SEa3sh695IK++aKWPw3ROtda28k+4OHtkbE4b09B37reNHu94njGMEsbsLAPja0Y0AYPx96CDfC7OZ1geOUN+PQsJ+/gonbGThmkTm2L5ic7O2XTHwCsxEgnwZKb/+wgG49l/vBZ5HJcNO3mcQnv1gnfb9Jzef5hJ+UkD97oKD8F+7JDIAHD26P35xzv5onDrNdw2dLIrr3vq/Sw4HAOW1vGRdQ+r9cvsVk/bCH6YvAQAsu+UzAICFN57iu/eFVL+JBGWubfVptK/No986yrdNHvPTJz7IbMuW+A5RBDlqgjMPGo4zDxqe0zkAtYI1yUj6zxX++1FoWBFUOF6XjxeXwHJOKFOY11EXnMkFldnvtRKkKyMqXkHnE3ymJQRiyJOiBItDgplyq8pCVC3JKe9HuVXE9GLqVimXzBzy/Aecn6Ho3Qmk8p2yVY7LDaBKAvJMIlO9ttoB7XmaIxCHhGIeQdzRdZR1c4OIc5hqxmy+ybhIQoLFpv0vYYw+EvLzFNo1lC/UgXL5qrw0ASuCCic8a0jzWnFce57mCJjgKyiWIN/cApPgcVDNd917c6IfV4zRaDLjZ9ZZBPoJhirKbXKTDlOLoBiF/0xQTqZji4ApBKFZQ45BvrfWkKsdrLUGioXKNVTnmUcQd3SdL0WgkiddQ9xVxUiBDcsayny1BXSBlQLTrKFydnFl4hxlpglYEVQ4YY+TO1isDhxL8jVZzATvT1XlGoo7svMeF/c8qqNkLr722kUsMaGfUGZhbBGUr9x0kRlNV5hrSDWzuLzUACuCisc5sn9+wTqcf/cbrv3eNQgki9fv9LUrrkXgFdZ+V1DckZ0vIymPMYKwjJRiKAJZKkJVBhtA5ouOM4O3nOluF6oLu8eqgHgpUN3X7g1mn6HYcNZQheMU7s8vsFIknS4Dp6IIqlQqhAi0CLrWJXOaTNajoQY7dncAAP75jSNw94wlrv3JBPlG2wkCbvz8vmhu7cDNT39kfC2nS+niIxvR2mH1e0CPenxnytjAY5+66hicdser9vVV6X/AL8/eH+M1ed7eH/i9FzXFEkx//eqhWtfPafsOxuLJY3DpMaOU+51ZQ3+5+NDQa5nIpPsubiqKwrjjgoMwtFdD5v2z1xyL2cutcmffO3lv9GiowRkHDtUdju+dNBYHj+yt3V9MMllDjtv2q3P2x4NvrkDTHn1K0icd5aE6mdgIAd8cAKdA1wWLvaSECAwWn3HgsJg9tLjt3AMBWKO6w0b1hddxkUgQutYnfdsunLgH9hrQPdK1ahzLRV5y9KjMtSYM6YkvHj4y8NgJQ3viC4dYOeQqsZcgwrmHjsB+ijr4cr+TyeMHBda013H83gMxefwg5b6aZALfPXEsejbUKvc7s4YmjRsYWk7ZJFh8wrhBmLR38HnywekHDEWTXVYcAPYe3ANfOtwq9dy9vgbXnrR3oGL99gljysbCUXWjX/d6XDW5fPooYUVQ4agyKJyLwutcQ146UsEWQa5B0GyNFbXbIkmEbnU1ymOCskRUvydn0DmZyNaoMf0IsuZ+0IQgHaWslySJnDUkfdmF6U7VIhVsJdxXVgQVjjLo2+FUBGauobZU2qVAvOTq0/QGybynSyTIl5EjBXHUuizONNSaBGWubfoZgmruhw3kirEeQRhR5xGU2eC001BJ95UVQYWTTvunFrfFcA21daRdCsRLrorAW9HR645IEvlWrcpOx492LWfQ2WkRmJrjUpHEqZxZHhaBhfEM6ooYs1Ye2RhB+d9fVgSdkPYOpxWQ3R4kUNtT6cAYQb5qvGfcFl7XkMIiiFtl06kIahKJzA/RNI1UHq8SkGGnKIdskKiFLStATlUmFXRfWRGUGWu37cbudn12zqqtLS5ffliMwJkS+vYnm31tJe2pNNpS+SkxrcJbddErfBKKhWniLhnodA0lk5S5VsLwaQ+a0RxmEZTDmrRRCwiWS22ezkYl3VdWBGXGxJtfxKX3z1bu2767HUfd8hJ+8vj8zDZ1qYis8L/ukWz1zmc+WKu9bntKoK0jKCibp2Cx/X7inv1c+3s21PiuId/u2T9a1pDbIsjGCEw/gww2q9ZnPiEkA0daHaftO8ToWirGDor2eb0cONxKn1QtfK+icsRVML26qLOoyoFjY2SOFROeR1CGzFy8Ubm9udUasb/00frMtrTwj/+cimDWUr0VIDlwRG9s2NEaGExWCYt/XX4EtjW349K/ZRXX144aha8fOwqvL97kLiEtT2Bf4sKJe+Anj1slhp+88mhl/XU5utalaurwKwL7fMYrY7mtlzk/PhEH3/g8AOD6U8cFH5sgvPXDyejdtS5SnyXv/M8UdK3L7Wd56n5DMPMHkzC8j1k9/cxtqWCNMPcnJ5bNRDIvBODuCw/JzKMpR1gRVBDZ6ovZbfkoFTGsTxes3tqirWZpXdy/aXifLr5JYKMHdseQXl3Qt5tbEPpjBNkT6kauTtO6JkHGy2S6XEMJylzL1H2ftV6s6zk/i4mwGdizIbSNjn7d62Mf68RUCQCdI1gcV/EWEufPqaE2mZkRXo6UpwqtUoJG5EBWEbS0ZUcWqmOCXDwq6pMJtKfSgcFkXZVP7yA7W1/Fvd1kZSYv7jVrNf1SbHdaBETOGIGZwAtb+KXTUfl6oKyphFBBSSwCIloGYAeAFIAOIURTKfpRbpgmx+xylHoQwq8MoloEdTUJtKdEsGtI8TDXJBI+BSHf+dcEsPsboV9Oq8G6jv9olYLyBnsTGYvAMJ2yTAuDFYyq+aDFpdir/uVCKV1Dk4QQamd4lRKWHaNcVSwkWGxCXU0Cbal0YJqmSoQmE+TLxNEJ3YxwjTDMdp0jwqjKu66BfGceI7D+V41FwBSY8jcJ2DVURoQqAsV+oRh3RFYEtmsoUBEoLQLyj8g15RziWASJeHpAbxEYPu1Z1xBrAiYHKujxKZVFIAA8R0QCwF1CiLtL1I+ywjniX75pF6a9vwa1iQSOGdsf/3l3Nc48yF/4LZ32T7hqi7jSWF1NAkIA3//3PG0b4xiB/K9ZLjJajICUr8PwKwJ1n3RQjL52Bsp/3FqZcIxAz1FCiNVENBDA80T0kRBihrMBEV0G4DIAGDkyuGJkZ8E54v/aX9/Gkg27rDdPWf+Wb9rlO0ZA+CyF3RHLRZtkM+gsAu/IPzOL17dgvFlfbj/vQFzzj7m+Y6L8mHTrGphmDWVWkXLc168fMwp9upVfZoqTbxy3J3rUR/9J92iowdGj++Mbx+1ZgF5VL/261+OIPfvhyhNGl7oroZREEQghVtv/1xPRYwAOAzDD0+ZuAHcDQFNTU1WMzZzpm6pyDy2KGcdCWJVDnTS3RctX9tb4kVx5wmj87qXFAPQxAu8ecu3LYjqi//xBwzKKIOEKFpvjixFk3FURYwSObT/6zIQIPSgN1586PtZxiQThgUsPz3NvmGSC8PBlE0vdDSOKHiMgom5E1EO+BnASgPnBR1UHzvWFGxSrT6lc+GkhfK6hXREtgu71aovAdV6FECXyWwTSD+8tvhbHPHYpgpxcQ/bM4ojXjVragmEqlVJYBIMAPGb/sGsAPCSEeKYE/Sg7nBaByl2jCl4KAd9Eq3xZBC49oDnWVxYCatdQnLorJvMIVNT5ahZFvG6cyDbDVDBFVwRCiKUADij2dSuBdIgiUGX1KC2C1mgWgU4ROBWPdkKX9729oSaZuyKgmK4hvzVCvvMFwXqAqTY4fbSMSKeDFYE6fdRfHC2qRdBVEyx2KhhdGQKdgPcK4zjVmd0WgfoEqs264nWmZNdOYFXAVAesCMoI58C+oUYRI1BMDxB5iBHohGzKxCLQZg2pM3ei4I4RRDlOfx4TpNJjNcBUC1x0LiJn/vE1bNnVhpe/P0nb5vN/eA3bWtpxz1cOwZTbrGSoP3+lKVOl8/QDhuKOCw7CrKWbcP7ds/C5A4biv++txswfZM/53IJ1vvOqLIKz73zDt23avDWRPlPPLurHwG0RmCGFcL6DxWMH9cBbAespqI6TxeKiWiPyslFXRmOYSoUtgoi8u2Irlm1qDmwz99Ot+GTjLrzpEFwPvLk88/qJ91YDyArs/9rvWwOWigTiZ7F0rUvi6auPcW1zpliOHdgD917kLvd0xoFDXUForyB/5JtHAvAXcosSLH5t6gl47FtHavtNjqfzngub8NPPTcCz1xyrbe/s678uPwLPeD5z1GUk2TXEVAtsERQQp/BrVrhrvMUhglYmA4CIE4YznDhhEMZ76v031CbRnrJiCUTA5PGDXPsP2aMP5q/alnnvde0cskcfa7vnWqSxCFQyeFjvLhjWu4u2387716trLb561ChtW+9xhzb2zbyXczJM69VnS2YzTHXAFkEBcQo/kwDu7vYQiyCmr0JVbM25PrDKf9/WkfZNVFOhKzGR7/TRKHgvJV1cphYBB4uZaoMVQQFxCthmg5TO1o4QiyCu01oh/7wLynjpSItA15BEV2W0xhcsNuhnyLnjHic/hzelNex41gNMtcCKoIA4BdIuA4ugNcwiyKNkCqsv1OGpRqoTyjrXUDIv8wgiH6I8rsOuxmpsEdj/eWYxUy2wIiggLtdQHiyCfMqlLnXBiqA9JVzlrLUiVOMa8scIcksfzeW4jGvIMEZQrdVHmeqlahTBywvXY/vu9sz75rYOvPihP0XTycK1O/Dxuh3KfbOXbcaabS2Z97OWbsKC1dvxxpJNmW3PfrA289prETz9/ho8Pne1a9tzH4T0R9OXOHQNVQRuiyC6ayi/6aM6VPENn0UQM0bA6aNMtVAVimD9jt24+C9v48qH3s1s+9Fj83HJ/bO1gh4ATr59Bk76zQzlvnP+9AYm3fpy5v35d8/CaXe8igvumZXZ9qxDsHuFyjcfnIMdu93K4dF3V5l8nLzw5cP3CNw/ZcIgfKFpROb9MWMGYNzgHgCAAT2yC6x7ReuQXtbC7XHLUAPAd6aMBRA/WKyzCLx90hHzsgxTsVRF+qj0vS9evzOzbekG67UqrdMUmeUTdUWwYjGsdxes2tri277sls8EHufc73z9jCKH3zkiX3jTKaivSfq2A9HcPFdPGYOrp4wxbu/rk+d9e9QYQaavbBIw1UFVWAQq5Ag97qjTiYn/vxSEuX/ygfP+SSWgble8cbZXCUWOEdj/2TXEVAtVoQhUeeEyIyQfAsokI6gUmKw8liu6YnRe8qFwTfFeqiOia0hmvvI8AqZaqBJF4J8pGme0pxMMUat9Fot6ReG6vGMo4OMUncsXMn3Uu3KZDqnc2CJgqoWqUARyRq4zLzzOaM+7AIwkav3/YuFdoKUQmI70i2kReMlmDRneD2lBFqg/DFNuVIUikD5ip+yXSkEn3FXogsLl6hryLtlYCMwXeymdJohaYiI7s5hVAVMdVIcisH/Q63e04rp/vwcga/an0mnsbO3AWX98DQvXqlNJz73rDTROnYYJP3lWub9cg8VhZSTygal4L5QecK7b0K2uRnktaRnVK9aBVpHJGWI9wFQJVaEInMXa/jl7pbVNWgQpgTeWbMKcFVvxi2c+Uh4fVgc/F4ugsV9Xo3Y9NMtJ6jissS+pgGAAAA96SURBVC9OGDfQte3xK47CL87eL9J5wgga6d/6hQOM2kXliW8fhZ+fuR+umjwGN5+V/TwPXno4rj91HHp3rXO1/+nn9sGVJ4zGcWMHek+lJFt9lDUBUx1UhSJIqYZ2GYsg9x97LnMRpn/veKN20646xrft4iMbte2nnjbOt1bAASN647xDR0bpXihB8v2cQ4YbtYvK/sN744uHj8R3TxyLft2zk9sa+3fDN47by9e+b7c6XHvS3uYTyuTM4vKcHsIweac6FIFm0XfAihHkKqN2tca3CEx97F3r/W6eoGBwrWlgtEiUMkYQlex0MrYImOqgvKRFgVCN7NIOi0AGgeOKqmJkDXVXuIaCgp+mo99cMRXwlaQIMllDrAeYKqEqFIHKNeS0CKIu9u6lGPMIVHMCghRBsQwCU/leyvTRqPDi9Uy1UR2KwOMaSqdFJoCcSqczgjzuD78Y6aMqF1IyQNoXawRunjVUOZogkfUNMUxV0OmLzn26udm3wMjyzc1YvW03ANsisF07qbTAnBVbAAADHEHIIFZuacbsZVvy2GNzgqYJEBx58AUUaBXl8jFEKi1emIapFjq1Ipi5aCO+fO+b+PJEd6aMs3x0Ki3QYo/oX/l4A175eEOkaxz9i+k591NHl9okenapwbrtrcr9zqwgIrdPu0+3Ouw1sDsA4JDGPkbX61Ffgx0RA99SD5y8z6BIx5UzspT2EXv1K3FPGKY4dGpFsGDNNgDAO8u3att0pARaDcpI/+CUcfj1cwu1M5EPGNEb732qv46KR755hNW//5mCSbe+jO2O9QluPGMfnHHQMBCsheRVOKt91iYTaOtIY/K4gfjZGfugf/d69O9ejxnfn4QRfbsY9ef1609Au8GC9U6ICLOun4w+3WojHVfOjOjbFa9eNwlDe5vdN4apdKoiRhBUKiCVFmjvCBd+o/p3xaCeDdr9BwzvFblf/bpZ7qd+3evRo8EtSI/Yqz96NtSiR0OtK1feSYNjpmyd7SeqTSYwvE92ktrIfl2N/fM9GmrRt1tdeEMPg3s1BJagrkRG9O1atMwrhik1VaEIgiaNdaSF0cIyYXV7enWJPiIOEjSqdFEvDS6LwDpXJ3TZMwxTYEqiCIjoFCJaSESLiWhqwa5j57QoZxbbpNJpI0VQV5MIFLI9G8IVgVfu1yTdPn4nqglkXpwL0EtF1RmDtwzDFJaiKwIiSgL4A4BTAUwAcAERTSjkNdMhFkFbHiyCHg3hI/ieHqshmQhQBAYF45yuIdk/1gMMw0SlFBbBYQAWCyGWCiHaAPwdwBmFvGCwRSCMAqRhisCk9r9XWTjr43tX+jJZVlHlGmKLgGGYqJQia2gYgE8d71cCOLyQF/x0s38Bd8lN0z40OkddiGA2CSz2qK8FkO1LkEVgQn2t3zXEeoBhmKiUQhGoRJVvSE5ElwG4DABGjsxvxcwg+narw+Zdbb7tdTUJ/OqcA3DBPbOUx9UkEjh5n0F49oN1+J/PjMeMRRsxwzMn4atHNeLemZ9g78E9sHFnqysgLG/KxUc2Ymuz//oAcPlxe6G+JoEP12xHv+712GdoTyQThFRa4OvH7onfv7QYR+xZXrnvFx2xBwb0sLKeTtlnMCaPNysFzTBM8SiFIlgJYITj/XAAq72NhBB3A7gbAJqamvI+xfOo0f3w2uJNvu3fPXEsenWpxZUPv+va3q0+ib0H98NZBw/Do3NWZbYP7tmAtdt3I5kg3HVhU2b7pcfsicap01znGNm3K5655lhlf2SK55cOH4kxg3oo20w9dZxv25Kfn5Z5fW7TCN/+UvOzM/bNvP7ThYeUsCcMw+goRYzgbQBjiGgUEdUBOB/AE4W4UFBsQOdLr0mQMh4gV7/yBp5rayhzXBi1AXEEeTS7dhiGKTZFtwiEEB1E9G0AzwJIArhPCPFBIa7VrpmRC+iLoCUThLoa/z6ZzumdWSxjB8lkuAQPjDNw6WOGYUpESUpMCCGeAvBUoa8TND9AN4CvSaotAinEvYXIZFsjiyBAEWSXR2QYhikunXpmcVtAWqhObCcTCaXAlhZEh+ecMvPHJGuoNsBq4AXTGYYpFZ1aEQRbBNFiBBJvuQp5mhqDlWCCzivPw8sjMgxTbKpWEQTFCLoEzOod1kddkdLEIgiadDaqfzcAQNfaTl0QlmGYMqRTS52TJgzGhh2teHr+Wt++BAEvXXscHp+7Ggki/HnmUuzY3YGaBGHc4B4YPbA7Fq/fiSnjB+HKE0ZnjvvhaeNx5F79ceCI3li2aRdumrYAgFmMwFkbyMutXzgA5zZtxsh+XbVt8s2M70/Cxl3qtQ4YhqkeOrVFcPSY/rjzy+rc9QQR9hzQHd85cSyunjImU7o5mSAkEoTzD7Vy8kf27YoDRvTOHNdQm8Qp+w7G4F4NmOiYvGVS2iGoflCPhlpMHl/cxV1G9uuKg0eaLVrDMEznpVMrgiC8Ln25ZoH09csRfkc6vCCdKSb1gxiGYYpN1UqmoBgBACRtoa1bkYxhGKazULWKwOvKkfMD5BoBSXt/KuLSjQzDMJVGFSsC93uZvy8tgqxriBUBwzCdmypWBPp5BEBWIXhnEjMMw3Q2qkIR7NEvm/lzyj6DAQCnHzjU1UaK+0ywOGlmEVw4cQ8AwHDF/IILDrPKZ48Z2B3jh/SM13mGYZgC06nnEUhe+f6k0DbCEyOQCiEVkjV03qEjcd6h6vUSbj5rP9x81n5RusowDFN0qsIiMEGO+2XsQLqGvLWFGIZhOhusCCQZee8OFntrCzEMw3Q2WBHYSHEvY8hJwxgBwzBMpcOKwEbGCGQuEVsEDMNUC6wIbLIWgTt9NJ8lJhiGYcoRVgQ2svS0nFFcb5eMDlpDgGEYpjNQFemjJtzzlSY89u4qjOhrzQc4aEQfXHXCaHzJnifAMAzTWSFRATNnm5qaxOzZs0vdDYZhmIqCiN4RQjSFtWO/B8MwTJXDioBhGKbKYUXAMAxT5bAiYBiGqXJYETAMw1Q5rAgYhmGqHFYEDMMwVQ4rAoZhmCqnIiaUEdEGAMtjHt4fwMY8dqezwfcnGL4/wfD9CabU92cPIcSAsEYVoQhygYhmm8ysq1b4/gTD9ycYvj/BVMr9YdcQwzBMlcOKgGEYpsqpBkVwd6k7UObw/QmG708wfH+CqYj70+ljBAzDMEww1WARMAzDMAF0akVARKcQ0UIiWkxEU0vdn2JDRCOIaDoRfUhEHxDR1fb2vkT0PBEtsv/3sbcTEd1h3695RHRwaT9BcSCiJBG9S0RP2u9HEdGb9v35BxHV2dvr7feL7f2Npex3MSCi3kT0byL6yH6OjuDnJwsRfcf+bc0nooeJqKESn59OqwiIKAngDwBOBTABwAVENKG0vSo6HQCuFUKMBzARwBX2PZgK4EUhxBgAL9rvAetejbH/LgNwZ/G7XBKuBvCh4/0vAPzGvj9bAFxib78EwBYhxGgAv7HbdXZ+C+AZIcQ4AAfAuk/8/AAgomEArgLQJITYF0ASwPmoxOdHCNEp/wAcAeBZx/vrAVxf6n6V+J48DuBEAAsBDLG3DQGw0H59F4ALHO0z7TrrH4DhsITZCQCeBECwJgDVeJ8jAM8COMJ+XWO3o1J/hgLem54APvF+Rn5+Mp9vGIBPAfS1n4cnAZxcic9Pp7UIkP2SJCvtbVWJbYYeBOBNAIOEEGsAwP4/0G5WjffsdgDXAUjb7/sB2CqE6LDfO+9B5v7Y+7fZ7TsrewLYAOAvtuvsz0TUDfz8AACEEKsA3ApgBYA1sJ6Hd1CBz09nVgSk2FaVKVJE1B3AIwCuEUJsD2qq2NZp7xkRfRbAeiHEO87NiqbCYF9npAbAwQDuFEIcBGAXsm4gFVV1f+zYyBkARgEYCqAbLPeYl7J/fjqzIlgJYITj/XAAq0vUl5JBRLWwlMCDQohH7c3riGiIvX8IgPX29mq7Z0cBOJ2IlgH4Oyz30O0AehNRjd3GeQ8y98fe3wvA5mJ2uMisBLBSCPGm/f7fsBQDPz8WUwB8IoTYIIRoB/AogCNRgc9PZ1YEbwMYY0fw62AFcZ4ocZ+KChERgHsBfCiEuM2x6wkAF9mvL4IVO5Dbv2Jnf0wEsE26ADojQojrhRDDhRCNsJ6Pl4QQXwIwHcA5djPv/ZH37Ry7fVmM6AqBEGItgE+JaG9702QAC8DPj2QFgIlE1NX+rcn7U3nPT6mDFAUO5pwG4GMASwD8qNT9KcHnPxqW6TkPwFz77zRYfskXASyy//e12xOsTKslAN6HlQ1R8s9RpHt1PIAn7dd7AngLwGIA/wJQb29vsN8vtvfvWep+F+G+HAhgtv0M/QdAH35+XPfnZwA+AjAfwP8BqK/E54dnFjMMw1Q5ndk1xDAMwxjAioBhGKbKYUXAMAxT5bAiYBiGqXJYETAMw1Q5rAiYTg0RpYhoruMvsAotEV1ORF/Jw3WXEVH/GMedTEQ3EFEfInoq134wjAk14U0YpqJpEUIcaNpYCPGnQnbGgGNgTUg6FsBrJe4LUyWwImCqErusxD8ATLI3fVEIsZiIbgCwUwhxKxFdBeByWOW8FwghzieivgDugzVpqBnAZUKIeUTUD8DDAAbAmixEjmt9GVa54jpYRf++JYRIefpzHqwKuXvCql8zCMB2IjpcCHF6Ie4Bw0jYNcR0drp4XEPnOfZtF0IcBuD3sGoMeZkK4CAhxP6wFAJgzSR91972QwB/s7f/FMBMYRVnewLASAAgovEAzgNwlG2ZpAB8yXshIcQ/YNXxmS+E2A/WTNWDWAkwxYAtAqazE+Qaetjx/zeK/fMAPEhE/4FVXgGwynacDQBCiJeIqB8R9YLlyjnL3j6NiLbY7ScDOATA21Y5GnRBtkiblzGwyjMAQFchxA6Dz8cwOcOKgKlmhOa15DOwBPzpAH5MRPsguJSw6hwE4H4hxPVBHSGi2QD6A6ghogUAhhDRXABXCiFeDf4YDJMb7BpiqpnzHP/fcO4gogSAEUKI6bAWrukNoDuAGbBdO0R0PICNwlrjwbn9VFjF2QCrKNs5RDTQ3teXiPbwdkQI0QRgGqz4wC9hFUk8kJUAUwzYImA6O13skbXkGSGETCGtJ6I3YQ2ILvAclwTwgO32IVhr0G61g8l/IaJ5sILFsqzwzwA8TERzALwCq0QxhBALiOh/ADxnK5d2AFcAWK7o68GwgsrfAnCbYj/DFASuPspUJXbWUJMQYmOp+8IwpYZdQwzDMFUOWwQMwzBVDlsEDMMwVQ4rAoZhmCqHFQHDMEyVw4qAYRimymFFwDAMU+WwImAYhqly/j8lV1tTdrRoUAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8250787da0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def dqn(n_episodes=2000, eps_start=1.0, eps_end=0.01, eps_decay=0.995):\n",
    "    \"\"\"Deep Q-Learning.\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n",
    "        eps_end (float): minimum value of epsilon\n",
    "        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n",
    "    \"\"\"\n",
    "    scores = []\n",
    "    scores_window = deque(maxlen=100)\n",
    "    eps = eps_start\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name] # reset the environment\n",
    "        state = env_info.vector_observations[0]            # get the current state\n",
    "        score = 0\n",
    "        while True:\n",
    "            action = agent.act(state, eps)\n",
    "            env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "            next_state = env_info.vector_observations[0]   # get the next state\n",
    "            reward = env_info.rewards[0]                   # get the reward\n",
    "            done = env_info.local_done[0] \n",
    "            agent.step(state, action, reward, next_state, done)\n",
    "            state = next_state\n",
    "            score += reward\n",
    "            if done:\n",
    "                break\n",
    "        scores_window.append(score)\n",
    "        scores.append(score)\n",
    "        eps = max(eps_end, eps_decay*eps) #decrease epsilon\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "        if np.mean(scores_window)>=15.0:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(agent.qnetwork_local.state_dict(), 'checkpoint.pth')\n",
    "            break\n",
    "    return scores\n",
    "\n",
    "scores = dqn()\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Watch the trained agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 17.0\n"
     ]
    }
   ],
   "source": [
    "# load the weights from file\n",
    "agent.qnetwork_local.load_state_dict(torch.load('checkpoint.pth'))\n",
    "\n",
    "env_info = env.reset(train_mode=True)[brain_name] # reset the environment\n",
    "state = env_info.vector_observations[0]            # get the current state\n",
    "score = 0\n",
    "\n",
    "while True:\n",
    "    action = agent.act(state)        # select an action\n",
    "    env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "    next_state = env_info.vector_observations[0]   # get the next state\n",
    "    reward = env_info.rewards[0]                   # get the reward\n",
    "    done = env_info.local_done[0]                  # see if episode has finished\n",
    "    state = next_state                             # roll over the state to next time step\n",
    "    score += reward\n",
    "    if done:                                       # exit loop if episode finished\n",
    "        break\n",
    "    \n",
    "print(\"Score: {}\".format(score))\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}