Deep_Reinforcement_Learning_Part_2_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": " Deep Reinforcement Learning- Part-2 .ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5bUv9ftkhM_n",
        "colab_type": "text"
      },
      "source": [
        "This is the Part-2 of Deep Reinforcement Learning Notebook series.***In this Notebook I have introduced introduces the first Policy Gradient method known as REINFORCE***.\n",
        "\n",
        "\n",
        "The Notebook series is about Deep RL algorithms so itexcludes all other techniques that can be used to learn functions in reinforcement learning and also the Notebook Series is not exhaustive i.e. it contains the most widely used Deep RL algorithms only.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SeRaQFKfhivP",
        "colab_type": "text"
      },
      "source": [
        "The REINFORCE algorithm involves learning a parametrized policy that produces action probabilities given states. Agents use this policy directly to act in an environment. The key idea is that during learning, actions that resulted in good outcomes should become more probable (these actions are positively reinforced). Conversely, actions that resulted in bad outcomes should become less probable. If learning is successful, over the course of many iterations action probabilities produced by the policy shift to distribution that results in a good performance in an environment. Action probabilities are changed by following the policy gradient, hence REINFORCE \n",
        "is known as a policy gradient algorithm.\n",
        "The algorithm needs three components:\n",
        "1. a parametrized policy\n",
        "2. an objective to be maximized\n",
        "3. a method for updating the policy parameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jgHI0PI1wLB_",
        "colab_type": "text"
      },
      "source": [
        "# A Parametrized Policy\n",
        "A policy π is a function which maps states to action probabilities, which is used to sample an action a ∼ π(s). In REINFORCE, an agent learns a policy and uses this to act in an environment. A good policy is one that maximizes the cumulative discounted rewards. The key idea in the algorithm is to learn a good policy, and this means doing function approximation. Neural networks are powerful and flexible function approximators, so we can represent a policy using a deep neural network consisting of learnable parameters θ. This is often referred to as a policy network $π_θ$. We say that the policy is parametrized by θ. Each specific set of values of the parameters of the policy network represents a particular policy. Formulated in this way, the process of learning a good policy corresponds to searching for a good set of values for θ. For this reason, the policy network must be differentiable. We will see later in this notebook that the mechanism by which the policy is improved is through gradient ascent in parameter space.\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A0tHgqeWxHSf",
        "colab_type": "text"
      },
      "source": [
        "# THE OBJECTIVE FUNCTION\n",
        "An objective can be understood as an agent’s goal, such as winning a game or getting the highest score possible.that an agent acting in an environment generates a trajectory, which contains a sequence of rewards along with the states and actions. A trajectory is denoted τ = $s_0, a_0,r_0,...,s_T,a_T,r_T$ .The return of a trajectory $R_t$(τ) is defined as a discounted sum of rewards from time step t to the end of a trajectory as shown below(Equation 2.1).\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-03%20at%2011.14.35%20PM.png?raw=true)      \n",
        "                          Equation 2.1\n",
        "\n",
        "Here we can see that the sum starts from time step t, but the power that the discount factor γ is raised to starts from 0 when summing for return, hence we need to offset the power by the starting time step t using t′ – t.\n",
        "\n",
        "The natural question which might now arise is that why do we need the discounted return? Why not just simply add all returns? The Answer is the return is task-oriented i.e. in some cases you might give equal priority to every return and in the other one return is more important than others. Different values of gamma(γ) have different effects on the agent. If we keep the value of gamma less than one then the agent is short-sighted i.e. it gives more priority to initial rewards than rewards obtained later in the time. And similarly, if the value of gamma if greater than one then the agent is far-sighted i.e. it gives more priority to rewards that are obtained later in time rather than initial rewards. The value of gamma depends on the tasks of the agent. We can have gamma equal to one in case we want to give equal priority to all rewards.\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-04%20at%2012.00.27%20AM.png?raw=true)                         \n",
        "                          Equation 2.2\n",
        "\n",
        "Here the objective is the expected return overall complete trajectories generated by an agent. The above equation(Equation 2.2) says that the expectation is calculated over many trajectories sampled from a policy, that is, τ ∼ $π_θ$. This expectation approaches the true value as more samples are gathered, and it is tied to the specific policy $π_θ$ used.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k9AR2md12eCc",
        "colab_type": "text"
      },
      "source": [
        "#THE POLICY GRADIENT\n",
        "\n",
        "The policy gradient is the mechanism by which action probabilities produced by the policy are changed. If the return $R_t(τ)$ > 0, then the probability of the action $π_θ (a_t|s_t)$ is increased; conversely, if the return $R_t$(τ) < 0, then the probability of the action $π_θ (a_t|s_t)$ is decreased. Over the course of many updates, the policy will learn to produce actions that result in high $R_t(τ)$.\n",
        "\n",
        "To maximize the objective, we perform gradient ascent on the policy parameters θ.To improve on the objective1, compute the policy gradient, and use it to update the parameters as shown below:\n",
        "\n",
        "1.Compute the policy gradient\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-04%20at%2012.11.50%20AM.png?raw=true)     \n",
        "                        Equation 2.3\n",
        "\n",
        "The term $▽_θJ(π_θ)$ is known as the policy gradient. The term $π_θ(a_t|s_t)$ is the probability of the action taken by the agent at time step t. The action is sampled from the policy, $a_t$ ∼ $π_θ(s_t)$. The right-hand side of the equation states that the gradient of the log probability of the action with respect to θ is multiplied by return $R_t(τ)$. Here I have skipped the proof of above equation but you can look at https://danieltakeshi.github.io/2017/03/28/going-deeper-into-reinforcement-learning-fundamentals-of-policy-gradients/\n",
        "\n",
        "2.Update the policy parameters θ\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-04%20at%2012.13.44%20AM.png?raw=true)                           \n",
        "                            Equation 2.4\n",
        "\n",
        "α is a scalar known as the learning rate; it controls the size of the parameter update"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tZW6MwSx5mYm",
        "colab_type": "text"
      },
      "source": [
        "The REINFORCE algorithm numerically estimates the policy gradient using Monte Carlo sampling.\n",
        "Monte Carlo sampling refers to any method which uses random sampling to generate data which is used to approximate a function. In essence, it is just “approximation with random sampling”\n",
        "\n",
        "The expectation $E_{τ∼πθ}$ implies that as more trajectories τ s are sampled using a policy $π_θ$ and averaged, it approaches the actual policy gradient $▽_θJ(π_θ)$. Instead of sampling many trajectories per policy, we can sample just one as shown in equation 2.3.This is how policy gradient is implemented — as a Monte Carlo estimate over sampled trajectories."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZpJd8Rhd-vEI",
        "colab_type": "text"
      },
      "source": [
        "**Disadvantage of Reinforce Algorithm and its few remedies**\n",
        "\n",
        "Disadvantage=>\n",
        "\n",
        "When using Monte Carlo sampling, the policy gradient estimate may have high variance because the returns can vary significantly from trajectory to trajectory. This is due to three factors. First, actions have some randomness because they are sampled from a probability distribution. Second, the starting state may vary per episode. Third, the environment transition function may be stochastic.\n",
        "\n",
        "Remedy=>\n",
        "\n",
        "One way to reduce the variance of the estimate is to modify the returns by subtracting a suitable action-independent baseline as shown below(Equation 2.5)\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-04%20at%204.59.45%20AM.png?raw=true)\n",
        "\n",
        "Equation 2.5\n",
        "\n",
        "One option for the baseline is the value function V. This choice of baseline motivates the Actor-Critic algorithm(Which we discuss later in this notebook series)\n",
        "\n",
        "An alternative is to use the mean returns over the trajectory.Let $b=\\sum_{t=0}^T R_t(τ)$\n",
        "\n",
        "To see why this is useful, consider the case where all the rewards for an environment are negative. Without a baseline, even when an agent produces a very good action, it gets discouraged because the returns are always negative. Over time, this can still result in good policies since worse actions will get discouraged even more and thus indirectly increase the probabilities of better actions. However, it can lead to slower learning because probability adjustments can only be made in a single direction. Likewise, the converse happens in environments where all the rewards are positive. A more effective way is to both increase and decreases the action probabilities directly. This requires having both positive and negative returns."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BM5OAz0t36rE",
        "colab_type": "text"
      },
      "source": [
        "# Here is the implementation of Reinforce Algorithm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RsIrUXHeBiwu",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-04%20at%204.47.48%20AM.png?raw=true)\n",
        "\n",
        "First, initialize the learning rate α and construct a policy network πθ with randomly initialized weights.\n",
        "Next, iterate for multiple episodes as follows: use the policy network πθ to generate a trajectory $τ = s_0, a_0, r_0, . . . , s_T, a_T, r_T$ for an episode. Then, for each time step t in the trajectory, compute the return $R_t(τ)$. Next, use $R_t(τ)$ to estimate the policy gradient. Sum the policy gradients for all time steps, then use the result to update the policy network parameters θ"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pq9KIniXBnYZ",
        "colab_type": "text"
      },
      "source": [
        "# Below is the implementation in code of this algorithm on a simple example of Cart-Pole where agent tries to balance the pole on the cart"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "11IylqnLlfTd",
        "colab_type": "text"
      },
      "source": [
        "**Below code setups the environment required to run the game.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hmhPprax4UrV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Installing the dependencies\n",
        "!pip install gym\n",
        "!apt-get install python-opengl -y\n",
        "!apt install xvfb -y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tqh1bvYz4YVB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install gym[atari]\n",
        "!pip install pyvirtualdisplay\n",
        "!pip install piglet"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RRc3yd6e4q80",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from pyvirtualdisplay import Display\n",
        "display = Display(visible=0, size=(1400, 900))\n",
        "display.start()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8gyhug2U4rb7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# This code creates a virtual display to draw game images on. \n",
        "# If you are running locally, just ignore it\n",
        "import os\n",
        "if type(os.environ.get(\"DISPLAY\")) is not str or len(os.environ.get(\"DISPLAY\"))==0:\n",
        "    !bash ../xvfb start\n",
        "    %env DISPLAY=:1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7wJ1Lye24tYJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import gym\n",
        "import tensorflow as tf\n",
        "from keras.layers import Dense\n",
        "from keras.models import Sequential\n",
        "from keras.optimizers import Adam\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "from IPython import display\n",
        "from gym import logger as gymlogger\n",
        "from gym.wrappers import Monitor\n",
        "gymlogger.set_level(40) # error only\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "import random\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import math\n",
        "import glob\n",
        "import io\n",
        "import base64\n",
        "from IPython.display import HTML\n",
        "\n",
        "from IPython import display as ipythondisplay"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nz0mpYHflr6M",
        "colab_type": "text"
      },
      "source": [
        "\"\"\"\n",
        "Utility functions to enable video recording of gym environment and displaying it\n",
        "To enable video, just do \"env = wrap_env(env)\"\"\n",
        "\"\"\"\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V-zUBrYJ4u8x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def show_video():\n",
        "  mp4list = glob.glob('video/*.mp4')\n",
        "  if len(mp4list) > 0:\n",
        "    mp4 = mp4list[0]\n",
        "    video = io.open(mp4, 'r+b').read()\n",
        "    encoded = base64.b64encode(video)\n",
        "    ipythondisplay.display(HTML(data='''<video alt=\"test\" autoplay \n",
        "                loop controls style=\"height: 400px;\">\n",
        "                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
        "             </video>'''.format(encoded.decode('ascii'))))\n",
        "  else: \n",
        "    print(\"Could not find video\")\n",
        "    \n",
        "\n",
        "def wrap_env(env):\n",
        "  env = Monitor(env, './video', force=True)\n",
        "  return env"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y9N8b_w7ltXw",
        "colab_type": "text"
      },
      "source": [
        "This part ensures the reproducibility of the code below by using a random seed.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RK55zKBE44I5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "RANDOM_SEED=1\n",
        "N_EPISODES=500\n",
        "\n",
        "# random seed (reproduciblity)\n",
        "np.random.seed(RANDOM_SEED)\n",
        "tf.random.set_seed(RANDOM_SEED)\n",
        "\n",
        "# set the env\n",
        "env=gym.make(\"CartPole-v1\") # env to import\n",
        "#env=wrap_env(env) #To enable video, just do \"env = wrap_env(env)\"\"\n",
        "env.seed(RANDOM_SEED)\n",
        "env.reset() # reset to env"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VSHb7-IKo77W",
        "colab_type": "text"
      },
      "source": [
        "#Defining the REINFORCE Class\n",
        "At initiation, the REINFORCE object sets a few parameters. First, is the environment in which the model learns and its properties. Second, are both the parameters of the REINFORCE algorithm — Gamma (𝛾) and alpha (𝛼). Gamma, as discussed above is the decay rate of past observations and alpha is the learning rate by which the gradients affect the policy update. Lastly, it sets the learning rate for the neural network. In addition, this snippet includes the saved space for recording the observations during the game.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jrqQ4aWEo9Qv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class REINFORCE:\n",
        "\n",
        "  def __init__(self, env, path=None):\n",
        "    self.env=env #import env\n",
        "    self.state_shape=env.observation_space.shape # the state space\n",
        "    self.action_shape=env.action_space.n # the action space\n",
        "    self.gamma=0.99 # decay rate of past observations\n",
        "    self.alpha=1e-4 # learning rate in the policy gradient\n",
        "    self.learning_rate=0.01 # learning rate in deep learning\n",
        "    \n",
        "    if not path:\n",
        "      self.model=self._create_model() #build model\n",
        "    else:\n",
        "      self.model=self.load_model(path) #import model\n",
        "\n",
        "    # record observations\n",
        "    self.states=[]\n",
        "    self.gradients=[] \n",
        "    self.rewards=[]\n",
        "    self.probs=[]\n",
        "    self.discounted_rewards=[]\n",
        "    self.total_rewards=[]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NgGI99zVnMvA",
        "colab_type": "text"
      },
      "source": [
        "##Defining useful utility methods\n",
        "The REINFORCE agent object uses a couple of utility methods. The first, hot_encode_action, encodes the actions into a one-hot-encoder format (read more about what is one-hot-encoding and why is it a good idea, here(https://medium.com/@michaeldelsolewhat-is-one-hot-encoding-and-how-to-do-it-f0ae272f1179)). And the second, remember, records the observations of each step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pDJ87pkXolTY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def remember(self, state, action, action_prob, reward):\n",
        "    '''stores observations'''\n",
        "    encoded_action=self.hot_encode_action(action)\n",
        "    self.gradients.append(encoded_action-action_prob)\n",
        "    self.states.append(state)\n",
        "    self.rewards.append(reward)\n",
        "    self.probs.append(action_prob)\n",
        "  \n",
        "  def hot_encode_action(self, action):\n",
        "    '''encoding the actions into a binary list'''\n",
        "\n",
        "    action_encoded=np.zeros(self.action_shape, np.float32)\n",
        "    action_encoded[action]=1\n",
        "\n",
        "    return action_encoded"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZVoY-MWZnOfs",
        "colab_type": "text"
      },
      "source": [
        "##Creating a Neural Network Model\n",
        "I chose to use a neural network to implement this agent. The network is a simple feedforward network with a few hidden layers. The output layer incorporates a softmax activation. The softmax function takes in logit scores and outputs a vector that represents the probability distributions of a list of potential outcomes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w9r_dv2lnt59",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def _create_model(self):\n",
        "    ''' builds the model using keras'''\n",
        "    model=Sequential()\n",
        "\n",
        "    # input shape is of observations\n",
        "    model.add(Dense(24, input_shape=self.state_shape, activation=\"relu\"))\n",
        "    # add a relu layer \n",
        "    model.add(Dense(12, activation=\"relu\"))\n",
        "\n",
        "    # output shape is according to the number of action\n",
        "    # The softmax function outputs a probability distribution over the actions\n",
        "    model.add(Dense(self.action_shape, activation=\"softmax\")) \n",
        "    model.compile(loss=\"categorical_crossentropy\",\n",
        "            optimizer=Adam(lr=self.learning_rate))\n",
        "        \n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ut6scLzGnTrR",
        "colab_type": "text"
      },
      "source": [
        "##Action Selection\n",
        "The get_action method guides its action choice. It uses the neural network to generate a normalized probability distribution for a given state. Then, it samples its next action from this distribution."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3MaAw88rnzJw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def get_action(self, state):\n",
        "      '''samples the next action based on the policy probabilty distribution \n",
        "      of the actions'''\n",
        "      # transform state \n",
        "      state=state.reshape([1, state.shape[0]])\n",
        "      # get action probably\n",
        "      action_probability_distribution=self.model.predict(state).flatten()\n",
        "      # norm action probability distribution\n",
        "      action_probability_distribution/=np.sum(action_probability_distribution)\n",
        "    \n",
        "      # sample action\n",
        "      action=np.random.choice(self.action_shape,1,\n",
        "                            p=action_probability_distribution)[0]\n",
        "\n",
        "      return action, action_probability_distribution"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sw6EFyJynXcJ",
        "colab_type": "text"
      },
      "source": [
        "##Constructing the Reward\n",
        "The REINFORCE model includes a discounting parameter, 𝛾, that governs the long term reward calculation. Using gamma, the model discounts rewards from the early stages of the game. This calculation ensures that longer episodes would award a state-action pair greater than shorter ones. This function returns the normalized vector of discounted rewards."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TLlCX1UJn1vo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def get_discounted_rewards(self, rewards): \n",
        "\n",
        "    '''Use gamma to calculate the total reward discounting for rewards\n",
        "    Following - \\gamma ^ t * Gt'''\n",
        "    \n",
        "    discounted_rewards=[]\n",
        "    cumulative_total_return=0\n",
        "    # iterate the rewards backwards and and calc the total return \n",
        "    for reward in rewards[::-1]:      \n",
        "      cumulative_total_return=(cumulative_total_return*self.gamma)+reward\n",
        "      discounted_rewards.insert(0, cumulative_total_return)\n",
        "\n",
        "    # normalize discounted rewards\n",
        "    mean_rewards=np.mean(discounted_rewards)\n",
        "    std_rewards=np.std(discounted_rewards)\n",
        "    norm_discounted_rewards=(discounted_rewards-\n",
        "                          mean_rewards)/(std_rewards+1e-7) # avoiding zero div\n",
        "    \n",
        "    return norm_discounted_rewards"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ip5ud197ndpA",
        "colab_type": "text"
      },
      "source": [
        "##Updating the Policy\n",
        "Following each Monte-Carlo episode, the model uses the data collected to update the policy parameters. Recall the last step shown in the pseudo-code above. Here, training the neural network updates the policy. The network fits a vector of states to a vector of the gradients multiplied by the discounted rewards and the learning rate. This step facilitates the stochastic gradient ascent optimization.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u77cTqYcn5ml",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def update_policy(self):\n",
        "    '''Updates the network.'''\n", 
        "    # get X\n",
        "    states=np.vstack(self.states)\n",
        "\n",
        "    # get Y\n",
        "    gradients=np.vstack(self.gradients)\n",
        "    rewards=np.vstack(self.rewards)\n",
        "    discounted_rewards=self.get_discounted_rewards(rewards)\n",
        "    gradients*=discounted_rewards\n",
        "    gradients=self.alpha*np.vstack([gradients])+self.probs\n",
        "\n",
        "    history=self.model.train_on_batch(states, gradients)\n",
        "    \n",
        "    self.states, self.probs, self.gradients, self.rewards=[], [], [], []\n",
        "\n",
        "    return history"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hgk3CMaanhYI",
        "colab_type": "text"
      },
      "source": [
        "##Training the model\n",
        "This method creates a training environment for the model. Iterating through a set number of episodes, it uses the model to sample actions and play them. When such a sequence ends, the model is using the recorded observations to update the policy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ohfFPk-34_g6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "   def train(self, episodes, rollout_n=1, render_n=50):\n",
        "    '''train the model\n",
        "        episodes - number of training iterations \n",
        "        rollout_n- number of episodes between policy update\n",
        "        render_n - number of episodes between env rendering ''' \n",
        "    \n",
        "    env=self.env\n",
        "    total_rewards=np.zeros(episodes)\n",
        "\n",
        "    for episode in range(episodes):\n",
        "      # each episode is a new game env\n",
        "      state=env.reset()\n",
        "      done=False          \n",
        "      episode_reward=0 #record episode reward\n",
        "      \n",
        "      while not done:\n",
        "        # play an action and record the game state & reward per episode\n",
        "        action, prob=self.get_action(state)\n",
        "        next_state, reward, done, _=env.step(action)\n",
        "        self.remember(state, action, prob, reward)\n",
        "        state=next_state\n",
        "        episode_reward+=reward\n",
        "        #print(\"Episode_reward{}\".format(episode_reward))\n",
        "        '''\n",
        "        if episode%render_n==0: ## render env to visualize.\n",
        "          plt.imshow(env.render('rgb_array'))\n",
        "          display.clear_output(wait=True)\n",
        "          display.display(plt.gcf())'''\n",
        "        if done:\n",
        "          # update policy \n",
        "          if episode%rollout_n==0:\n",
        "            history=self.update_policy()\n",
        "\n",
        "      total_rewards[episode]=episode_reward\n",
        "    model=self.model\n",
        "    model.save('model.h5')\n",
        "    self.total_rewards=total_rewards\n",
        "\n",
        "\n",
        "plt.figure(figsize=(4, 3))\n",
        "display.clear_output(wait=True)\n",
        "Agent=REINFORCE(env)\n",
        "Agent.train(episodes=500)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}