Deep_Reinforcement_Learning_Part_1_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": " Deep Reinforcement Learning- Part-1 .ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hxUCS1EK6O5V",
        "colab_type": "text"
      },
      "source": [
        "This is the Part-1 of Deep Reinforcement Learning Notebook series. In this Notebook, I have introduced the different aspects of a deep reinforcement learning problem and gives an overview of deep reinforcement learning algorithms.\n",
        "\n",
        "The Notebook series is about Deep RL algorithms so it excludes all other techniques that can be used to learn functions in reinforcement learning and also the Notebook Series is not exhaustive i.e. it contains the most widely used Deep RL algorithms only.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GR3figikIaTu",
        "colab_type": "text"
      },
      "source": [
        "#              Introduction to Reinforcement Learning (RL)                  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MixM5PRv-LzR",
        "colab_type": "text"
      },
      "source": [
        "Reinforcement Learning (RL) is concerned with solving sequential decision-making problems.\n",
        "\n",
        "RL problems can be expressed as a system consisting of an agent and an environment. An agent is the one who performs the action and the environment can be considered as everything except agent. An environment produces information that describes the state of the system and known as the state. An agent interacts with the environment by observing the state and using this information to select an action. The environment accepts the action and transitions into the next state. It then returns the next state and a reward to the agent. When the cycle(state -> action -> reward) completes,it is said that one time step has passed.The cycle repeats until the environment terminates, for example when the problem is solved. The below image represents this entire process.![alt text](https://www.kdnuggets.com/images/reinforcement-learning-fig1-700.jpg) \n",
        "                                                                        \n",
        "  FIGURE_1:This Cycle is often referred as RL control loop           "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_NcZsddgCZqH",
        "colab_type": "text"
      },
      "source": [
        "The formal definition of policy is that a policy is a mapping from perceived states of the environment to actions to be taken when in those states. But the above definition is not quite intuitive so I define it another way i.e an agent's action-producing function is called a policy.\n",
        "\n",
        "The exchange of states and actions between agent and environment unfolds in time, hence it can be thought of as a sequential decision-making process.\n",
        "\n",
        "The objective of RL problems is the sum of the rewards received by an agent. An agent tries to maximize this objective by taking optimal actions. It learns to do this by the process trial and error method. The agent initially performs some random actions and receives feedback from the environment on these actions and uses this feedback to reinforce optimal actions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gisl2OsbGYaH",
        "colab_type": "text"
      },
      "source": [
        " The signals exchanged between an agent and the environment are ($s_{t}, a_{t}, r_{t}$), which stands for the state, action, and reward respectively, and t denotes the time step in which these signals occurred. The ($s_{t}, a_{t}, r_{t}$) tuple is called an experience. The control loop can repeat forever1 or terminate by reaching either a terminal state or a maximum time step t = T. The time horizon from t = 0 to when the environment terminates is called an episode. A trajectory is a sequence of experiences over an episode, τ = ($s_{0}, a_{0}, r_{0}$), ($s_{1}, a_{1}, r_{1}$), . . .. An agent typically needs many episodes to learn a good policy, ranging from hundreds to millions depending on the complexity of the problem.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BCczIM6QblAM",
        "colab_type": "text"
      },
      "source": [
        "#Reinforcement Learning As Markov Decision Process(MDP)\n",
        "\n",
        "\n",
        "*  How an environment transitions from one state to next using what is known as the transition function\n",
        "*  The transition function in RL is formulated as MDP, which is a mathematical framework that models sequential decision making\n",
        "*  Since using the general formulation for RL is very difficult, we represent the transition function as MDP\n",
        "\n",
        "General Formulation => \n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-02%20at%208.32.25%20AM.png?raw=true)\n",
        "\n",
        "New Transition Function => \n",
        "\n",
        "\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-02%20at%208.40.49%20AM.png?raw=true)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oGdOqFBQZwUr",
        "colab_type": "text"
      },
      "source": [
        "**RL Control Loop AS MDP Control Loop**\n",
        "\n",
        "MDP Control LooP =>\n",
        "\n",
        "                    1.Given an env(environment) and an agent:\n",
        "                    2.for episode=0.....MAx_EPISODE do\n",
        "                    3.   state=env.reset()\n",
        "                    4.   agent.reset()\n",
        "                    5.   for t=0.......T do\n",
        "                    6.       action=agent.act(state)\n",
        "                    7.       state,reward=env.step(action)\n",
        "                    8.       agent.update(action,state,reward)\n",
        "                    9.       if env.done() then\n",
        "                    10.         break\n",
        "                    11.      end if\n",
        "                    12.   end for\n",
        "                    13.end for\n",
        "\n",
        "The above Algorithm is generic to all RL problems and it defines a consistent interface between an agent and an environment. At the beginning of each episode, the environment and the agent are reset (line:3-4). On reset, the environment produces an initial state. Then they begin interacting — an agent produces an action given a state (line:6), then the environment produces the next state and reward given the action (line:7), stepping into the next time step. The agent.act-env.step cycle continues until the maximum time step T is reached or when the environment terminates. AT line:8 agent.update encapsulates an agent’s learning algorithm. Over multiple time steps and episodes, this method collects data and performs learning internally to maximize the objective.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UJNHh41V_Zyi",
        "colab_type": "text"
      },
      "source": [
        "There are mainly three primary functions to learn in reinforcement learning:\n",
        "1. a policy, π, which maps state to action, a ∼ π(s).\n",
        "2. a value function, $V^π$(s) or $Q^π$(s, a), to estimate the\n",
        "expected return $E_{τ}$ [R(τ)].\n",
        "3. the environment $model^3$, P(s′|s, a).\n",
        "\n",
        "A policy π is how an agent produces actions in the environment to maximize the objective. A policy can be stochastic. That is, it may probabilistically output a different action for the same state. We can write this as π(a|s) to denote the probability of an action a given a state s. An action sampled from a policy is written as a ∼ π(s).\n",
        "\n",
        "The value functions provide information about the objective. They help an agent understand how good states and available actions are in terms of expected future returns. They come in two forms — the $V^π(s) and Q^π(s, a)$functions.\n",
        "\n",
        "The transition function P(s′|s, a) provides information about the environment. If an agent learns this function, it can predict the next state s′ that the environment will transition into after taking action an in state s. By applying the learned transition function, an agent can “imagine” the consequences of its actions without actually touching the environment. It can then use this information to plan good actions accordingly.\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-03%20at%208.10.25%20PM.png?raw=true)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EcjYLG_VIx8W",
        "colab_type": "text"
      },
      "source": [
        "Correspondingly, there are three major families of deep reinforcement learning algorithms — policy- based, value-based, and model-based methods, which learn policies, value functions, and models respectively. There are also combined methods in which agents learn more than one of these functions.\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-03%20at%208.22.18%20PM.png?raw=true)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y5D2-xOfLehW",
        "colab_type": "text"
      },
      "source": [
        "A major advantage of policy-based algorithms is that they are a very general class of optimization methods. They can be applied to problems with any type of actions — discrete, continuous, or a mixture (multi-action). They also directly optimize for the thing an agent cares most about — the objective. Additionally, this class of methods is guaranteed to converge to a locally optimal policy, as proven by Sutton et al. with the Policy Gradient Theorem. One disadvantage of these methods is that they have high variance and are sample- inefficient(An algorithm has poor sample efficiency if it fails to learn anything useful from many samples of experience and doesn't improve rapidly).\n",
        "\n",
        "Value-based algorithms are typically more sample-efficient than policy-based algorithms. This is because they have lower variance and make better use of data gathered from the environment. However, there are no guarantees that these algorithms will converge to an optimum. In their standard formulation, they are also only applicable to environments with discrete action spaces. This has historically been a major limitation, but with more recent advances, such as QT-OPT, they can now be effectively applied to environments with continuous action spaces.\n",
        "\n",
        "Purely model-based approaches are most commonly applied to games with a target state, such as winning or losing a game of chess, or navigation tasks with a goal state s*. This is because their transition functions do not model any rewards. To use it to plan actions, some information about an agent’s objective needs to be encoded in the states themselves. Model-based algorithms are very appealing since a perfect model endows an agent with foresight — it can play out scenarios and understand the consequences of its actions without having to actually act in an environment. This can be a significant advantage in situations where it is very time-consuming or expensive to gather experiences from the environment as in robotics for example. These algorithms also tend to require many fewer samples of data to learn good policies compared to policy-based or value-based methods, since having a model makes it possible for an agent to supplement its actual experiences with imagined ones.\n",
        "However, for most problems, models are hard to come by. Many environments are stochastic, and their transition dynamics are not known. In these cases, the model must be learned. This approach is still in early development, and it faces several challenges. First, environments with a large state space and action space can be very difficult to model; doing so may even be intractable, especially if the transitions are extremely complex. Second, models are only useful when they can accurately predict the transitions of an environment many steps\n",
        "assumptions. To learn the dynamics, an agent will need to act in an environment to gather examples of actual transitions (s, a, r, s′).\n",
        "into the future. Depending on the accuracy of the model, the prediction errors will compound for every time step and quickly grow to be unreliable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_E8vasOBN61q",
        "colab_type": "text"
      },
      "source": [
        "A final important distinction between deep reinforcement learning algorithms is whether they are on-policy or off-policy. This affects how training iterations make use of data.\n",
        "An algorithm is on-policy if it learns on the policy, i.e. training can only utilize data generated from the current policy π. This implies that as training iterates through versions of policies, π1, π2, π3,... ., each training iteration only uses the current policy at that time to generate training data. As a result, all the data must be discarded after training, since it becomes unusable. This makes the on-policy methods sample inefficient, so they require more training data. In contrast, an algorithm is off-policy if it does not have this requirement. Any data collected can be reused in training. Consequently, off-policy methods are more sample efficient, this may require a large memory to store the data."
      ]
    }
  ]
}