Notebook/BankNoteAuthentication.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bank Note Authentication Using Deep Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>In this notebook, I have tried to predict whether a particular instance of a bank note is fake or real.<br>\n",
    "\n",
    "<b>Dataset</b>\n",
    "<hr>\n",
    "<ol>\n",
    "<li>This data has been taken from the UCI Machine Learning Repository</li>\n",
    "<li>Data were extracted from images that were taken from genuine and forged banknote-like specimens. For digitization, an industrial camera usually used for print inspection was used </li>\n",
    "<li>The final images have 400x 400 pixels. Due to the object lens and distance to the investigated object gray-scale pictures with a resolution of about 660 dpi were gained. Wavelet Transform tool were used to extract features from images</li>\n",
    "<li>It consists of a total of 1373 instances</li></ol>\n",
    "\n",
    "<b>About the features</b>\n",
    "\n",
    "<hr>\n",
    "<ol>\n",
    "<li>Variance : variance of Wavelet Transformed image (continuous) </li>\n",
    "<li>Skewness : skewness of Wavelet Transformed image (continuous) </li>\n",
    "<li>Curtosis : curtosis of Wavelet Transformed image (continuous) </li>\n",
    "<li>Entropy : entropy of image (continuous) </li>\n",
    "<li>Class : class (integer - 0(Fake) or 1(Real))</li> </ol>\n",
    "<br>\n",
    "The first step is to import the relevant libraries\n",
    "<br>\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step is to import the relevant libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Importing the relevant libraries\n",
    "import numpy as np\n",
    "import math\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (5.0, 4.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "In the following blocks of code, I have implemented two mathematical functions that we will be using as activation functions in our network<br>\n",
    "    <ol>\n",
    "        <li> Sigmoid Function : It is differentiable everywhere so gradiant descent approach can be used for back propagation.It also outputs probability values between 0 or 1.</li>\n",
    "        <li> Relu Function : There is a reduced likelihood of vanishing gradient with this function.It is also more computationally efficient than other functions as it needs to only select between (0,z) where z is the input to the function</li>\n",
    "    </ol>\n",
    "    \n",
    "Additionally, by methods of calculus, I have implemented two more functions {relu_backward,sigmoid_backward} that calculate the differentiation value of sigmoid and relu functions respectively. These, I have used during backpropagation for weight and bias vector updation.\n",
    "\n",
    "\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(Z):\n",
    "    \"\"\"\n",
    "    Implements the sigmoid activation in numpy\n",
    "    \n",
    "    Arguments:\n",
    "    Z -- numpy array of any shape\n",
    "    \n",
    "    Returns:\n",
    "    A -- output of sigmoid(z), same shape as Z\n",
    "    cache -- returns Z as well, useful during backpropagation\n",
    "    \"\"\"\n",
    "    \n",
    "    A = 1/(1+np.exp(-Z))\n",
    "    cache = Z\n",
    "    \n",
    "    return A, cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu(Z):\n",
    "    \"\"\"\n",
    "    Implements the RELU function.\n",
    "\n",
    "    Arguments:\n",
    "    Z -- Output of the linear layer, of any shape\n",
    "\n",
    "    Returns:\n",
    "    A -- Post-activation parameter, of the same shape as Z\n",
    "    cache -- a python dictionary containing \"A\" ; stored for computing the backward pass efficiently\n",
    "    \"\"\"\n",
    "    \n",
    "    A = np.maximum(0,Z)\n",
    "    \n",
    "    assert(A.shape == Z.shape)\n",
    "    \n",
    "    cache = Z \n",
    "    return A, cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu_backward(dA, cache):\n",
    "    \"\"\"\n",
    "    Implements the backward propagation for a single RELU unit.\n",
    "\n",
    "    Arguments:\n",
    "    dA -- post-activation gradient, of any shape\n",
    "    cache -- 'Z' where we store for computing backward propagation efficiently\n",
    "\n",
    "    Returns:\n",
    "    dZ -- Gradient of the cost with respect to Z\n",
    "    \"\"\"\n",
    "    \n",
    "    Z = cache\n",
    "    dZ = np.array(dA, copy=True) # just converting dz to a correct object.\n",
    "    \n",
    "    # When z <= 0, we set dz to 0 as well. \n",
    "    dZ[Z <= 0] = 0\n",
    "    \n",
    "    assert (dZ.shape == Z.shape)\n",
    "    \n",
    "    return dZ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid_backward(dA, cache):\n",
    "    \"\"\"\n",
    "    Implements the backward propagation for a single SIGMOID unit.\n",
    "\n",
    "    Arguments:\n",
    "    dA -- post-activation gradient, of any shape\n",
    "    cache -- 'Z' where we store for computing backward propagation efficiently\n",
    "\n",
    "    Returns:\n",
    "    dZ -- Gradient of the cost with respect to Z\n",
    "    \"\"\"\n",
    "    \n",
    "    Z = cache\n",
    "    \n",
    "    s = 1/(1+np.exp(-Z))\n",
    "    dZ = dA * s * (1-s)\n",
    "    \n",
    "    assert (dZ.shape == Z.shape)\n",
    "    \n",
    "    return dZ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "\n",
    "The neural network (with L layers) I have implemented has Relu activation functions until the (L-1)th layer and Sigmoid function at the Lth layer\n",
    "\n",
    "We can see, from the code below, that in a for loop that goes from 1 to L-1, I have initialized each layer's W using He initialization to regularise the model so that it does not suffer from high variance eventually.\n",
    "\n",
    "Since it is alright to initialise the b vector to all zeros, I have simply done that.\n",
    "\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_parameters_deep(layer_dims):\n",
    "    \"\"\"\n",
    "    Arguments:\n",
    "    layer_dims -- python array (list) containing the dimensions of each layer in our network\n",
    "    \n",
    "    Returns:\n",
    "    parameters -- python dictionary containing your parameters \"W1\", \"b1\", ..., \"WL\", \"bL\":\n",
    "                    Wl -- weight matrix of shape (layer_dims[l], layer_dims[l-1])\n",
    "                    bl -- bias vector of shape (layer_dims[l], 1)\n",
    "    \"\"\"\n",
    "    \n",
    "    np.random.seed(1)\n",
    "    parameters = {}\n",
    "    L = len(layer_dims)            # number of layers in the network\n",
    "\n",
    "    for l in range(1, L):\n",
    "        \n",
    "        \n",
    "        #He initialization\n",
    "        parameters['W' + str(l)] = np.random.randn(layers_dims[l], layers_dims[l-1])*math.sqrt(2./layers_dims[l-1])\n",
    "        parameters['b' + str(l)] = np.zeros((layers_dims[l], 1))\n",
    "        \n",
    "        assert(parameters['W' + str(l)].shape == (layer_dims[l], layer_dims[l-1]))\n",
    "        assert(parameters['b' + str(l)].shape == (layer_dims[l], 1))\n",
    "\n",
    "        \n",
    "    return parameters\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "In the following code, I have implemented linear_forward which takes as input the activation output of the previous layer (X = A[0]), the corresponding weight and bias matrix for the current layer<br>\n",
    "    <br>\n",
    "    \n",
    "We calculate Z = W.dot(A_prev) + b\n",
    "    \n",
    "    \n",
    "    \n",
    "</blockquote>\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_forward(A, W, b):\n",
    "    \"\"\"\n",
    "    Implements the linear part of a layer's forward propagation.\n",
    "\n",
    "    Arguments:\n",
    "    A -- activations from previous layer (or input data): (size of previous layer, number of examples)\n",
    "    W -- weights matrix: numpy array of shape (size of current layer, size of previous layer)\n",
    "    b -- bias vector, numpy array of shape (size of the current layer, 1)\n",
    "\n",
    "    Returns:\n",
    "    Z -- the input of the activation function, also called pre-activation parameter \n",
    "    cache -- a python dictionary containing \"A\", \"W\" and \"b\" ; stored for computing the backward pass efficiently\n",
    "    \"\"\"\n",
    "    \n",
    "    Z = W.dot(A) + b\n",
    "    \n",
    "    assert(Z.shape == (W.shape[0], A.shape[1]))\n",
    "    cache = (A, W, b)\n",
    "    \n",
    "    return Z, cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "    \n",
    "In the following code snippet, I have implemented a function that calls the previous function (linear_forward) based on whether the activation is sigmoid or Relu. It would call linear_forward with activation = \"relu\" for a total of L-1 layers before it calls linear_forward with activation = \"sigmoid\" for the Lth layer\n",
    "\n",
    "\n",
    "Additionally, linear_activation_forward returns a cache that is a combination of linear_cache which stores the Z matrix where<br> \n",
    "Z = W*Aprev + b <br>\n",
    "and activation_cache stores the value of A where A = g(Z) where g is our activation function (sigmoid/relu)\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_activation_forward(A_prev, W, b, activation):\n",
    "    \"\"\"\n",
    "    Implements the forward propagation for the LINEAR->ACTIVATION layer\n",
    "\n",
    "    Arguments:\n",
    "    A_prev -- activations from previous layer (or input data): (size of previous layer, number of examples)\n",
    "    W -- weights matrix: numpy array of shape (size of current layer, size of previous layer)\n",
    "    b -- bias vector, numpy array of shape (size of the current layer, 1)\n",
    "    activation -- the activation to be used in this layer, stored as a text string: \"sigmoid\" or \"relu\"\n",
    "\n",
    "    Returns:\n",
    "    A -- the output of the activation function, also called the post-activation value \n",
    "    cache -- a python dictionary containing \"linear_cache\" and \"activation_cache\";\n",
    "             stored for computing the backward pass efficiently\n",
    "    \"\"\"\n",
    "    \n",
    "    if activation == \"sigmoid\":\n",
    "        # Inputs: \"A_prev, W, b\". Outputs: \"A, activation_cache\".\n",
    "        Z, linear_cache = linear_forward(A_prev, W, b)\n",
    "        A, activation_cache = sigmoid(Z)\n",
    "    \n",
    "    elif activation == \"relu\":\n",
    "        # Inputs: \"A_prev, W, b\". Outputs: \"A, activation_cache\".\n",
    "        Z, linear_cache = linear_forward(A_prev, W, b)\n",
    "        A, activation_cache = relu(Z)\n",
    "    \n",
    "    assert (A.shape == (W.shape[0], A_prev.shape[1]))\n",
    "    cache = (linear_cache, activation_cache)\n",
    "\n",
    "    return A, cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "In the following code, I have implemented L_model_forward, which calls linear_activation_forward L-1 times with activation = \"relu\" and one more time for the Lth layer with activation = \"sigmoid\"\n",
    "<br>\n",
    "    <br>\n",
    "    \n",
    "Additionally, \"cache\" will store all the caches (linear cache,activation cache) for each layer. \"caches\" is a combination of all such \"cache\" which is the second parameter returned by L_model_forward\n",
    "\n",
    "\n",
    "\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def L_model_forward(X, parameters):\n",
    "    \"\"\"\n",
    "    Implements forward propagation for the [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID computation\n",
    "    \n",
    "    Arguments:\n",
    "    X -- data, numpy array of shape (input size, number of examples)\n",
    "    parameters -- output of initialize_parameters_deep()\n",
    "    \n",
    "    Returns:\n",
    "    AL -- last post-activation value\n",
    "    caches -- list of caches containing:\n",
    "                every cache of linear_relu_forward() (there are L-1 of them, indexed from 0 to L-2)\n",
    "                the cache of linear_sigmoid_forward() (there is one, indexed L-1)\n",
    "    \"\"\"\n",
    "\n",
    "    caches = []\n",
    "    A = X\n",
    "    L = len(parameters) // 2                  # number of layers in the neural network\n",
    "    \n",
    "    # Implementing [LINEAR -> RELU]*(L-1).\n",
    "    for l in range(1, L):\n",
    "        A_prev = A \n",
    "        A, cache = linear_activation_forward(A_prev, parameters['W' + str(l)], parameters['b' + str(l)], activation = \"relu\")\n",
    "        caches.append(cache)\n",
    "    \n",
    "    # Implementing LINEAR -> SIGMOID for the Lth layer\n",
    "    AL, cache = linear_activation_forward(A, parameters['W' + str(L)], parameters['b' + str(L)], activation = \"sigmoid\")\n",
    "    caches.append(cache)\n",
    "    \n",
    "    assert(AL.shape == (1,X.shape[1]))\n",
    "            \n",
    "    return AL, caches\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "\n",
    "In the following function, compute_cost, we calculate the cost of a neural network model with a binary classification output of 0 or 1. From calculus, we see that :<br>\n",
    "<br>\n",
    "cost = (1./m) * (-np.dot(Y,np.log(AL).T) - np.dot(1-Y, np.log(1-AL).T))<br>\n",
    "<br>\n",
    "is the most efficient way to calculate the cost. If ycap is our prediction and y is our true label, we see tat upon substituting y = 1 in the above function, this cost equation forces ycap to be 1 in order to minimize the loss to approximately 0 and vice versa for y = 0.\n",
    "\n",
    "\n",
    "\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost(AL, Y):\n",
    "    \"\"\"\n",
    "    Implements the cost function for a binary classification problem\n",
    "    \n",
    "    Arguments:\n",
    "    AL -- probability vector corresponding to your label predictions, shape (1, number of examples)\n",
    "    Y -- true \"label\" vector (for example: containing 0 if non-cat, 1 if cat), shape (1, number of examples)\n",
    "\n",
    "    Returns:\n",
    "    cost -- cross-entropy cost\n",
    "    \"\"\"\n",
    "    \n",
    "    m = Y.shape[1]\n",
    "\n",
    "    # Compute loss from aL and y.\n",
    "    cost = (1./m) * (-np.dot(Y,np.log(AL).T) - np.dot(1-Y, np.log(1-AL).T))\n",
    "    \n",
    "    cost = np.squeeze(cost)      # this turns [[17]] into 17\n",
    "    assert(cost.shape == ())\n",
    "    \n",
    "    return cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "\n",
    "In the following function, linear_backward, I have calculated dW and dB given dZ of that current layer and cache which stores (A_prev,W and b) from forward propagation for that layer. \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_backward(dZ, cache):\n",
    "    \"\"\"\n",
    "    Implements the linear portion of backward propagation for a single layer (layer l)\n",
    "\n",
    "    Arguments:\n",
    "    dZ -- Gradient of the cost with respect to the linear output (of current layer l)\n",
    "    cache -- tuple of values (A_prev, W, b) coming from the forward propagation in the current layer\n",
    "\n",
    "    Returns:\n",
    "    dA_prev -- Gradient of the cost with respect to the activation (of the previous layer l-1), same shape as A_prev\n",
    "    dW -- Gradient of the cost with respect to W (current layer l), same shape as W\n",
    "    db -- Gradient of the cost with respect to b (current layer l), same shape as b\n",
    "    \"\"\"\n",
    "    A_prev, W, b = cache\n",
    "    m = A_prev.shape[1]\n",
    "\n",
    "    dW = 1./m * np.dot(dZ,A_prev.T)\n",
    "    db = 1./m * np.sum(dZ, axis = 1, keepdims = True)\n",
    "    dA_prev = np.dot(W.T,dZ)\n",
    "    \n",
    "    assert (dA_prev.shape == A_prev.shape)\n",
    "    assert (dW.shape == W.shape)\n",
    "    assert (db.shape == b.shape)\n",
    "    \n",
    "    return dA_prev, dW, db"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "\n",
    "The following function linear_activation_backward calls the appropriate function (relu_backward/sigmoid_backward) based on what the activation value is (relu/sigmoid). Essentially, it makes this call to calculate dA_prev which is the differential value of the activation function of the previous layer.\n",
    "\n",
    "\n",
    "\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_activation_backward(dA, cache, activation):\n",
    "    \"\"\"\n",
    "    Implements the backward propagation for the LINEAR->ACTIVATION layer.\n",
    "    \n",
    "    Arguments:\n",
    "    dA -- post-activation gradient for current layer l \n",
    "    cache -- tuple of values (linear_cache, activation_cache) we store for computing backward propagation efficiently\n",
    "    activation -- the activation to be used in this layer, stored as a text string: \"sigmoid\" or \"relu\"\n",
    "    \n",
    "    Returns:\n",
    "    dA_prev -- Gradient of the cost with respect to the activation (of the previous layer l-1), same shape as A_prev\n",
    "    dW -- Gradient of the cost with respect to W (current layer l), same shape as W\n",
    "    db -- Gradient of the cost with respect to b (current layer l), same shape as b\n",
    "    \"\"\"\n",
    "    linear_cache, activation_cache = cache\n",
    "    \n",
    "    if activation == \"relu\":\n",
    "        dZ = relu_backward(dA, activation_cache)\n",
    "        dA_prev, dW, db = linear_backward(dZ, linear_cache)\n",
    "        \n",
    "    elif activation == \"sigmoid\":\n",
    "        dZ = sigmoid_backward(dA, activation_cache)\n",
    "        dA_prev, dW, db = linear_backward(dZ, linear_cache)\n",
    "    \n",
    "    return dA_prev, dW, db"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "\n",
    "L_model_backward runs the main function for backpropagation for the whole model, where it calls linear_activation_backward once for the Lth layer with activation = \"sigmoid\" and otherwise L-1 times with activation = \"relu\"\n",
    "\n",
    "\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def L_model_backward(AL, Y, caches):\n",
    "    \"\"\"\n",
    "    Implements the backward propagation for the [LINEAR->RELU] * (L-1) -> LINEAR -> SIGMOID group\n",
    "    \n",
    "    Arguments:\n",
    "    AL -- probability vector, output of the forward propagation (L_model_forward())\n",
    "    Y -- true \"label\" vector (containing 0 if note is fake, 1 if note is real)\n",
    "    caches -- list of caches containing:\n",
    "                every cache of linear_activation_forward() with \"relu\" (there are (L-1) or them, indexes from 0 to L-2)\n",
    "                the cache of linear_activation_forward() with \"sigmoid\" (there is one, index L-1)\n",
    "    \n",
    "    Returns:\n",
    "    grads -- A dictionary with the gradients\n",
    "             grads[\"dA\" + str(l)] = ... \n",
    "             grads[\"dW\" + str(l)] = ...\n",
    "             grads[\"db\" + str(l)] = ... \n",
    "    \"\"\"\n",
    "    grads = {}\n",
    "    L = len(caches) # the number of layers\n",
    "    m = AL.shape[1]\n",
    "    Y = Y.reshape(AL.shape) # after this line, Y is the same shape as AL\n",
    "    \n",
    "    # Initializing the backpropagation\n",
    "    dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL))\n",
    "    \n",
    "    # Lth layer (SIGMOID -> LINEAR) gradients. Inputs: \"AL, Y, caches\". Outputs: \"grads[\"dAL\"], grads[\"dWL\"], grads[\"dbL\"]\n",
    "    current_cache = caches[L-1]\n",
    "    grads[\"dA\" + str(L-1)], grads[\"dW\" + str(L)], grads[\"db\" + str(L)] = linear_activation_backward(dAL, current_cache, activation = \"sigmoid\")\n",
    "    \n",
    "    for l in reversed(range(L-1)):\n",
    "        # lth layer: (RELU -> LINEAR) gradients.\n",
    "        current_cache = caches[l]\n",
    "        dA_prev_temp, dW_temp, db_temp = linear_activation_backward(grads[\"dA\" + str(l + 1)], current_cache, activation = \"relu\")\n",
    "        grads[\"dA\" + str(l)] = dA_prev_temp\n",
    "        grads[\"dW\" + str(l + 1)] = dW_temp\n",
    "        grads[\"db\" + str(l + 1)] = db_temp\n",
    "\n",
    "    return grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "update_parameters will update W and b based on the differentiation parameters that it receives after calculation and also the learning rate(alpha). It will do so for each layer in the neural network.\n",
    "\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_parameters(parameters, grads, learning_rate):\n",
    "    \"\"\"\n",
    "    Updates parameters using gradient descent\n",
    "    \n",
    "    Arguments:\n",
    "    parameters -- python dictionary containing your parameters \n",
    "    grads -- python dictionary containing your gradients, output of L_model_backward\n",
    "    \n",
    "    Returns:\n",
    "    parameters -- python dictionary containing your updated parameters \n",
    "                  parameters[\"W\" + str(l)] = ... \n",
    "                  parameters[\"b\" + str(l)] = ...\n",
    "    \"\"\"\n",
    "    \n",
    "    L = len(parameters) // 2 # number of layers in the neural network\n",
    "\n",
    "    # Update rule for each parameter. Use a for loop.\n",
    "    for l in range(L):\n",
    "        parameters[\"W\" + str(l+1)] = parameters[\"W\" + str(l+1)] - learning_rate * grads[\"dW\" + str(l+1)]\n",
    "        parameters[\"b\" + str(l+1)] = parameters[\"b\" + str(l+1)] - learning_rate * grads[\"db\" + str(l+1)]\n",
    "        \n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "    load_data_csv : Loads data from the designated folder in \"csv\" format. It also splits the data as follows:\n",
    "    train -> 80%\n",
    "    test -> 20%\n",
    "    <br>\n",
    "    <br>\n",
    "    Eventually, we will reshape the data to the appropriate dimensions in order to pass as input to initialize_parameters_deep\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data_csv(name_of_file):\n",
    "    path = r'C:\\Users\\Aishwarya\\Desktop\\Desktop\\UdemyCoursera\\BankNoteAuthentication\\\\'\n",
    "    full_dataset = pd.read_csv(path+name_of_file)\n",
    "    \n",
    "    train, test = train_test_split(full_dataset, test_size=0.2)\n",
    "    \n",
    "    train_set_x = np.array(train.iloc[:,0:4])\n",
    "    train_set_y = np.array(train.iloc[:,4])\n",
    "    \n",
    "    test_set_x = np.array(test.iloc[:,0:4])\n",
    "    test_set_y = np.array(test.iloc[:,4])\n",
    "    \n",
    "    \n",
    "    train_set_y = train_set_y.reshape((1, train_set_y.shape[0]))\n",
    "    test_set_y = test_set_y.reshape((1, test_set_y.shape[0]))\n",
    "    \n",
    "    #print(train_set_x.shape)\n",
    "    #print(test_set_y.shape)\n",
    "    \n",
    "    return train_set_x,train_set_y,test_set_x,test_set_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x_orig, train_y, test_x_orig, test_y = load_data_csv('banknote.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples: 1097\n",
      "Number of testing examples: 275\n",
      "train_x_orig shape: (1097, 4)\n",
      "train_y shape: (1, 1097)\n",
      "test_x_orig shape: (275, 4)\n",
      "test_y shape: (1, 275)\n"
     ]
    }
   ],
   "source": [
    "# Exploring our dataset \n",
    "m_train = train_x_orig.shape[0]\n",
    "m_test = test_x_orig.shape[0]\n",
    "\n",
    "print (\"Number of training examples: \" + str(m_train))\n",
    "print (\"Number of testing examples: \" + str(m_test))\n",
    "print (\"train_x_orig shape: \" + str(train_x_orig.shape))\n",
    "print (\"train_y shape: \" + str(train_y.shape))\n",
    "print (\"test_x_orig shape: \" + str(test_x_orig.shape))\n",
    "print (\"test_y shape: \" + str(test_y.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_x's shape: (4, 1097)\n",
      "test_x's shape: (4, 275)\n"
     ]
    }
   ],
   "source": [
    "# Reshape the training and test examples \n",
    "train_x_flatten = train_x_orig.reshape(train_x_orig.shape[0], -1).T   # The \"-1\" makes reshape flatten the remaining dimensions\n",
    "test_x_flatten = test_x_orig.reshape(test_x_orig.shape[0], -1).T\n",
    "\n",
    "# Standardize data to have feature values between 0 and 1.\n",
    "train_x = train_x_flatten/255.\n",
    "test_x = test_x_flatten/255.\n",
    "\n",
    "print (\"train_x's shape: \" + str(train_x.shape))\n",
    "print (\"test_x's shape: \" + str(test_x.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "    We define our network as follows:\n",
    "    <ol>\n",
    "        <li> Input layer : 4 neurons, each corresponding to one feature of the input observation</li>\n",
    "        <li> 1st Hidden layer : 20 neurons</li>\n",
    "        <li> 2nd Hidden layer : 100 neurons</li>\n",
    "        <li> 3rd Hidden layer : 75 neurons</li>\n",
    "        <li> 4th Hidden layer : 50 neurons</li>\n",
    "        <li> 5th Hidden layer : 20 neurons</li>\n",
    "        <li> 6th Hidden layer : 10 neurons</li>\n",
    "        <li> Output Layer : 1 neuron</li></ol>\n",
    "    \n",
    "   Here, we purposely select a neural network which is very deep with so many layers, because we have very less data at hand, so in order to combat the problem of high bias, we will make the neural network more complex. Since there is very less data, we do not need to do mini-batch processing as we have enough computational memory to run around 1300 observations which would not even take much time.\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "   </blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "### CONSTANTS ###\n",
    "layers_dims = [4,20,100, 75,50,20,10,1] #  7-layer model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "    \n",
    "   L_layer_model will call the appropriate neural network functions defined above and calculate the cost of our model after every 1000 iterations. It will do so as many times as the value of num_iterations which is the epoch value that we will input for our model.\n",
    "    \n",
    "    \n",
    "   </blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def L_layer_model(X, Y, layers_dims, learning_rate = 0.0075, num_iterations = 3000, print_cost=False):\n",
    "    \"\"\"\n",
    "    Implements a L-layer neural network: [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID.\n",
    "    \n",
    "    Arguments:\n",
    "    X -- data, numpy array of shape (num_px * num_px * 3, number of examples)\n",
    "    Y -- true \"label\" vector (containing 0 if cat, 1 if non-cat), of shape (1, number of examples)\n",
    "    layers_dims -- list containing the input size and each layer size, of length (number of layers + 1).\n",
    "    learning_rate -- learning rate of the gradient descent update rule\n",
    "    num_iterations -- number of iterations of the optimization loop\n",
    "    print_cost -- if True, it prints the cost every 100 steps\n",
    "    \n",
    "    Returns:\n",
    "    parameters -- parameters learnt by the model. They can then be used to predict.\n",
    "    \"\"\"\n",
    "\n",
    "    np.random.seed(1)\n",
    "    costs = []                         # keep track of cost\n",
    "    \n",
    "    # Parameters initialization\n",
    "    parameters = initialize_parameters_deep(layers_dims)\n",
    "    \n",
    "    \n",
    "    # Loop (gradient descent)\n",
    "    for i in range(0, num_iterations):\n",
    "\n",
    "        # Forward propagation: [LINEAR -> RELU]*(L-1) -> LINEAR -> SIGMOID.\n",
    "        AL, caches = L_model_forward(X,parameters)\n",
    "        \n",
    "        \n",
    "        # Compute cost.\n",
    "        cost = compute_cost(AL,Y)\n",
    "    \n",
    "        # Backward propagation.\n",
    "        grads = L_model_backward(AL,Y,caches)\n",
    "     \n",
    "        # Update parameters.\n",
    "        parameters = update_parameters(parameters,grads,learning_rate)\n",
    "                \n",
    "        # Print the cost every 1000 training example\n",
    "        if print_cost and i % 1000 == 0:\n",
    "            print (\"Cost after iteration %i: %f\" %(i, cost))\n",
    "        if print_cost and i % 1000 == 0:\n",
    "            costs.append(cost)\n",
    "            \n",
    "    # plot the cost\n",
    "    plt.plot(np.squeeze(costs))\n",
    "    plt.ylabel('cost')\n",
    "    plt.xlabel('iterations (per 000s)')\n",
    "    plt.title(\"Learning rate =\" + str(learning_rate))\n",
    "    plt.show()\n",
    "    \n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "    \n",
    "   Now, we will run our model, and note the cost after every 1000 iterations. Plotting a graph of cost after every 1000 iterations shows us how our model reaches the appropriate optimal value.\n",
    "    \n",
    "    \n",
    "    \n",
    "   </blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost after iteration 0: 0.694189\n",
      "Cost after iteration 1000: 0.524189\n",
      "Cost after iteration 2000: 0.497194\n",
      "Cost after iteration 3000: 0.449796\n",
      "Cost after iteration 4000: 0.402558\n",
      "Cost after iteration 5000: 0.216739\n",
      "Cost after iteration 6000: 0.128143\n",
      "Cost after iteration 7000: 0.074602\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAEWCAYAAAAAZd6JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXhU5dnH8e8vCWHfCTskyA7iApFFXLBqBUvFXbGC+ra11WLf2qqlm7a2tla7vNpSrVsVXChiVVwqtoooCpKAgOyyJBDWyCo7Iff7x5zoECcLkOFMJvfnuuYyM+eZk3tG+HGeec65R2aGc865sqWEXYBzziU6D0rnnKuAB6VzzlXAg9I55yrgQemccxXwoHTOuQp4ULq4kHSmpGVh1+FcVfCgTEKS8iSdF2YNZvaemXUPs4YSkoZIKjhOv+tcSUsl7ZE0TVJmOWOzgjF7guecV2r7rZI2Stoh6QlJtYPHO0raVepmkn4UbB8iqbjU9uvi+8qTmwelOyqSUsOuAUARCfHnWFIL4F/AL4BmQC7wz3Ke8hzwEdAc+BkwWVJGsK8LgLHAuUAWcALwKwAzW2NmDUpuQB+gGHghat/ro8eY2VNV90prnoT4A+aOD0kpksZKWilpi6RJkppFbX8+6gjmXUm9o7Y9KekhSa9L2g2cExy53iZpQfCcf0qqE4w/7CiuvLHB9jskbZC0XtK3giOkLmW8jnck3SPpfWAPcIKkGyQtkfSZpFWSvhOMrQ/8G2gbdXTVtqL34ihdCiwys+fNbB/wS+BkST1ivIZuQF/gLjPba2YvAB8DlwVDrgMeN7NFZrYN+DVwfRm/dzTwrpnlHWP9rgwelDXL94GLgbOBtsA2YFzU9n8DXYGWwFzgmVLPvwa4B2gIzAgeuxIYCnQCTqLsv8xljpU0FPghcB7QJaivIqOAG4Na8oHNwHCgEXAD8GdJfc1sNzCMw4+w1lfivfhcMNXdXs7tmmBob2B+yfOC370yeLy03sAqM/ss6rH5UWMP21fwcytJzWPsazRQ+oixpaRNklZL+nPwD4Y7SmlhF+COq+8AY8ysAEDSL4E1kkaZWZGZPVEyMNi2TVJjM9sRPPyymb0f/LxPEsCDQfAg6RXglHJ+f1ljrwT+YWaLgm2/Aq6t4LU8WTI+8FrUz9MlvQmcSSTwYyn3vYgeaGZrgCYV1APQACgs9dgOImEea+yOGGPblbG95OeGwJaSByWdCbQCJkeNXUrkvV0KZBIJ0T8Rec3uKPgRZc2SCbxYciQELAEOETlSSZV0bzAV3QnkBc9pEfX8tTH2uTHq5z1E/oKXpayxbUvtO9bvKe2wMZKGSZolaWvw2i7k8NpLK/O9qMTvLssuIke00RoBnx3F2NLbS34uva/rgBfMbFfJA2a20cwWm1mxma0G7gAur/SrcF/iQVmzrAWGmVmTqFsdM1tHZFo9gsj0tzGRBQQART0/Xq2mNgDto+53qMRzPq8lWA1+AfgD0MrMmgCv80Xtseou7704TBmrzNG3bwRDFwEnRz2vPtA5eLy0RUQ+W40+2jw5auxh+wp+3mRm0UeTdYEr+PK0uzTj8P+P7gh5UCavWpLqRN3SgIeBexScsiIpQ9KIYHxDYD+RaV094LfHsdZJwA2SekqqB9x5hM9PB2oTmfYWSRoGfDVq+yaguaTGUY+V914cpvQqc4xbyWe5LwInSrosWKi6E1hgZktj7HM5MA+4K/j/cwmRz21LVq7HA9+U1EtSU+DnwJOldnMJsB2YFv1gsJDWUREdgHuBl8t681zFPCiT1+vA3qjbL4EHgCnAm5I+A2YBA4Lx44ksiqwDFgfbjgsz+zfwIJG/8CuAmcGm/ZV8/mdEFmcmEVmUuYbI6yzZvpTIqTirgql2W8p/L472dRQSWbW+J6hjAHB1yXZJD0t6OOopVwPZwdh7gcuDfWBmbwD3EXlP8oPbXaV+5XXAePtyU9m+RN7D3cAHwEIi7487SvLGvS7RSOpJ5C937dILK86FwY8oXUKQdImk9GCa+XvgFQ9Jlyg8KF2i+A6RzxhXEll9vinccpz7gk+9nXOuAn5E6ZxzFah2V+a0aNHCsrKywi7DOZdk5syZ86mZZcTaVu2CMisri9zc3LDLcM4lGUn5ZW3zqbdzzlXAg9I55yrgQemccxWIa1BKGippmaQVksbG2P5nSfOC2/Kgi4tzziWUuC3mKPJVAeOA84ECIEfSFDNbXDLGzG6NGn8LcGq86nHOuaMVzyPK/sAKM1tlZgeAiUTaeJVlJJHGBc45l1DiGZTtOLy5agFfdG8+TNDqqhPwdhnbb5SUKym3sLB0A2nnnIuveAZlrEahZV0veTUw2cwOxdpoZo+YWbaZZWdkxDwfNKaDh4p58K1PWLS+dMd955yrvHgGZQGHd6puD6wvY+zVxGHavXt/EeNn5nPb8ws4UFRc1bt3ztUQ8QzKHKCrpE6S0omE4ZTSgyR1B5ryRbPWKtOkXjq/veRElmzYybhpK6p69865GiJuQRn0EhwDTCXyxU2TzGyRpLslXRQ1dCQwMUaX5irx1d6tufiUtoybtsKn4M65o1Lt2qxlZ2fbkV7rvX3PAc7/87s0r5/OlDFnkJ7m59k75w4naY6ZZcfaViMSIzIF78PSjZ/xV5+CO+eOUI0ISoDze7XiklPb8bdpK1i4zqfgzrnKqzFBCXDX13vRtH46tz0/31fBnXOVVqOCskm9dH5XMgV/+5Owy3HOVRM1KigBzuvViktPbce4d1b6FNw5Vyk1LigB7vp6b5r7FNw5V0k1Migb16vF7y6NTMH/4lNw51wFamRQApzbsxWX9m3H395ZyccFPgV3zpWtxgYlwF3De9OiQWQKvr8oZj8O55yr2UFZMgVftukz/vKWn4junIutRgclwFd6tOKyvu15aPpKFhT4N1E4576sxgclwJ1f7+VTcOdcmTwogcZ1a3HvpSexfNMuHnzLV8Gdc4fzoAyc06Mll/drz8PTV/kU3Dl3GA/KKL8Y3ouMBrV9Cu6cO4wHZZTGdWvxu8v6sHzTLh74r0/BnXMRHpSlnNO9JVf0a8/D01cyf61PwZ1zHpQx/Xx4L1o2rMNtz89n30GfgjtX03lQxlAyBf9k8y4e8FVw52o8D8oynNO9JVdmt+fv01cyz6fgztVoHpTl+PnwXrRq5FNw52o6D8pyNKpTi3svO4kVm3fxf74K7lyN5UFZgbO7ZXBVdgceeXclH63ZFnY5zrkQxDUoJQ2VtEzSCkljyxhzpaTFkhZJejae9Rytnw3v6VNw52qwuAWlpFRgHDAM6AWMlNSr1JiuwE+AwWbWG/hBvOo5FiVT8JWFu/nzf5eHXY5z7jiL5xFlf2CFma0yswPARGBEqTHfBsaZ2TYAM9scx3qOydndMrj6tA48+u4q5voU3LkaJZ5B2Q5YG3W/IHgsWjegm6T3Jc2SNDTWjiTdKClXUm5hYWGcyq3Yz77Wk9aN6nC7T8Gdq1HiGZSK8ZiVup8GdAWGACOBxyQ1+dKTzB4xs2wzy87IyKjyQiurYfQU/D8+BXeupohnUBYAHaLutwfWxxjzspkdNLPVwDIiwZmwzuqWwcj+HXj0PZ+CO1dTxDMoc4CukjpJSgeuBqaUGvMScA6ApBZEpuKr4lhTlfjphT1p07iur4I7V0PELSjNrAgYA0wFlgCTzGyRpLslXRQMmwpskbQYmAbcbmZb4lVTVYlMwfuwqnA3f/IpuHNJT2alPzZMbNnZ2Zabmxt2GQD85F8fMzFnDZO/ezr9MpuGXY5z7hhImmNm2bG2+ZU5x+CnF/agbeO6vgruXJLzoDwGDevU4veXncSqT3fzxzeXhV2Ocy5OPCiP0RldW3DNgI48NmM1c/K3hl2Ocy4OPCirwE8v7BlMwRf4FNy5JORBWQUa1E7jvssjU/A/TPUpuHPJxoOyigzu0oJrB3bk8fdXk5vnU3DnkokHZRX6ybCetGtSl9snL2DvAZ+CO5csPCirUP1gCr760938wVfBnUsaHpRV7PTOLRg1MJMn3l9Njk/BnUsKHpRxMHZYD9o3jZyI7lNw56o/D8o4qF87jfsuO5m8LXu431fBnav2PCjjZFDn5owelMk/PljN7NU+BXeuOvOgjKMfD41Mwe+Y7FNw56ozD8o4ip6C3zd1adjlOOeOkgdlnA3q3JzrBmXy5Ad5PgV3rpryoDwOfjysBx2a1uP2yfPZc6Ao7HKcc0fIg/I4qJceORE9f8se7nvDV8Gdq248KI+TgSc05/rTs3jygzzeWrKJ4uLq1VneuZosLewCapI7hnbnnWWb+eZTuTSpV4vszKacltWM0zo148S2jUlP83+3nEtEHpTHUb30NP5182DeWrKJnLyt5OZt479LNgNQp1YKp3RoQv8gOPt2bEr92v6/x7lE4F8uFrLCz/aTm7eV2XlbycnbyuL1Oyk2SE0Rvds2IjuzGf07NSU7qxktGtQOu1znklZ5Xy7mQZlgdu0vYm7+NnLytjJ79Vbmrd3O/qJiAE7IqB854sxqRv9OzWjftC6SQq7YueTgQVmN7S86xMJ1O5i9ehu5wVHnzn2RU4xaNar9eWieltWM7q0akpLiwenc0SgvKOP6IZikocADQCrwmJndW2r79cD9wLrgob+a2WPxrKm6qZ2WSr/MZvTLbAZ0prjYWL75M3JWb2V23jZyVm/l1QUbAGhUJ43srGZkZzWlf1Yz+rRvTO201HBfgHNJIG5BKSkVGAecDxQAOZKmmNniUkP/aWZj4lVHsklJET1aN6JH60aMGpSFmVGwbS85wdHm7NVbeXtpZIGodloKJwcLRNlZTemX2ZSGdWqF/Aqcq37ieUTZH1hhZqsAJE0ERgClg9IdA0l0aFaPDs3qcWnf9gBs2bWfnLxtwcr6Vh6avpJD04wUQc82jQ6brmc09AUi5yoSz6BsB6yNul8ADIgx7jJJZwHLgVvNbG3pAZJuBG4E6NixYxxKTS7NG9Rm6ImtGXpiawB27y/iozXbIyvrq7cyMWcNT36QB0CnFvUZ3KU5I/t3pHfbxiFW7VziittijqQrgAvM7FvB/VFAfzO7JWpMc2CXme2X9F3gSjP7Snn7rWmLOfFw8FAxC9ftCKbq23h/xafsPXiIfplNGTUwk2F9Wvtnm67GCWXVW9Ig4JdmdkFw/ycAZva7MsanAlvNrNzDGg/Kqrdj70Emzyng6Vn5rP50Ny0apHPVaR24ZkAm7ZrUDbs8546LsIIyjch0+lwiq9o5wDVmtihqTBsz2xD8fAnwYzMbWN5+PSjjp7jYmLHiU8bPzOftpZsAOK9nK0YPymJwl+Z+zqZLaqGcHmRmRZLGAFOJnB70hJktknQ3kGtmU4DvS7oIKAK2AtfHqx5XsZQUcVa3DM7qlkHBtj08++EaJuas5c3Fmzghoz7XDsjksn7taVzXV85dzeInnLty7S86xOsfb2D8zHw+WrOdurVSufjUdowelEnPNo3CLs+5KuNX5rgqsXDdDsbPzOPleevZX1TMaVlNGTUoi6G9W3vnI1fteVC6KrV9zwEmzylgwqx88rfsoUWD2ozs34FrBnSkTWNf/HHVkweli4viYuPdTwqZMDOft5dtJkXi/J6tGD0ok0GdffHHVS+hXevtkltKihjSvSVDurdk7dY9PP1hPpNy1vLGoo10zqjPqIGRxR+/bNJVd35E6arUvoOHeG3BBsbPymf+2u3US0/lklPbMXpQFt1bNwy7POfK5FNvF4oFBdsZPzOfV+ZHFn/6d2rGqIGZDD2xNbVSffHHJRYPSheqbbsP8PyctTw9aw1rtu4ho2FtRvbvyDX9O9K6cZ2wy3MO8KB0CaK42Ji+vJDxM/N4Z3khKRIX9G7FqIFZDDyhmS/+uFD5Yo5LCCkp4pweLTmnR0vWbAkWf3LX8vrHG+nasgGjBmVyad/2NPAvVXMJxo8oXaj2HTzEK/PXM2FWPgsKdlA/PZVL+7Zn1KBMurXyxR93/PjU21UL89ZuZ/zMPF5dsIEDRcUM6NSM0YOy+GrvVr744+LOg9JVK1t3H2BS7lqenpVPwba9tAwWf0b64o+LIw9KVy0dKjamL9/MhJn5hy3+XDswk0En+JU/rmr5Yo6rllJTxFd6tOIrPVqRv2U3z364hn8Giz9dWjZg1MBMLu3bzq/8cXHnR5SuWtl38BCvLtjAhJl5zC/Y8fmVP6MGZdKjtbd9c0fPp94uKc1fu52nZ+UzpeTKn6xmXDso09u+uaPiQemS2rbdkbZvT3/obd/c0fOgdDVCrLZv5/VsyehBWZzubd9cBXwxx9UIpdu+PfPhGiblrmXqosh3/kQWf/w7f9yR8yNKl9T2HYx858+EWYd/58+ogZn0auuLP+4LPvV2jsh3/kyYmc/L89ex72Ax2ZlNGTUo0vatdlpq2OW5kHlQOhdlx56DQdu3fPK27KFFg3SuOq0D1wzIpF0TX/ypqTwonYuhuNiYseJTJszK560lmwA4t2crRg3M5IwuLUhJ8cWfmqS8oKzUyWaSrqjMYzHGDJW0TNIKSWPLGXe5JJMUs0jn4iElRZzVLYNHR2fz7h3ncNOQzszN38boJ2Zz7p+m89h7q9ix52DYZboEUKkjSklzzaxvRY+V2p4KLAfOBwqAHGCkmS0uNa4h8BqQDowxs3IPF/2I0sXT/qJDvLFwI+Nn5jMnfxt1aqUw4uR23D60Oy0a1A67PBdHR316kKRhwIVAO0kPRm1qBBRV8Hv7AyvMbFWwr4nACGBxqXG/Bu4Dbqtgf87FXe20VEac0o4Rp7Rj0fodPD1rDS/MLaBw134evy7bz8WsoSqaeq8HcoF9wJyo2xTgggqe2w5YG3W/IHjsc5JOBTqY2avl7UjSjZJyJeUWFhZW8Gudqxq92zbmd5f24Y4LuvP20s28uXhT2CW5kJQblGY238yeArqY2VPBz1OIHCluq2Dfsf7p/XyeLykF+DPwo4qKNLNHzCzbzLIzMjIqGu5clbru9Cx6tG7Ir6YsYs+BiiZSLhlVtnPAfyQ1ktQMmA/8Q9KfKnhOAdAh6n57IkeoJRoCJwLvSMoDBgJTfEHHJZpaqSn85uITWb9jHw+89UnY5bgQVDYoG5vZTuBS4B9m1g84r4Ln5ABdJXWSlA5cTeRoFAAz22FmLcwsy8yygFnARRUt5jgXhuysZlyZ3Z7H31vN8k2fhV2OO84qG5RpktoAVwLlfp5YwsyKgDHAVGAJMMnMFkm6W9JFR1WtcyEaO6wnDeqk8fOXFlLdzj92x6ayQXk3kcBbaWY5kk4AKpyDmNnrZtbNzDqb2T3BY3ea2ZQYY4f40aRLZM3qpzN2aA9mr97Kv+auC7scdxxVKijN7HkzO8nMbgrurzKzy+JbmnOJ58rsDvTt2ITfvr6E7XsOhF2OO04qe2VOe0kvStosaZOkFyS1j3dxziWalBTxm4v7sG3PAe6fuizsctxxUtmp9z+ILMS0JXIu5CvBY87VOL3aNuL60zvx7Ow1zFu7Pexy3HFQ2aDMMLN/mFlRcHsS8BMaXY116/ldadmwNj978WMOFfvCTrKrbFB+KulaSanB7VpgSzwLcy6RNaxTi18M78Wi9TuZMDMv7HJcnFU2KP+HyKlBG4ENwOXADfEqyrnq4Gt92nBm1xb88c3lbN65L+xyXBxVNih/DVxnZhlm1pJIcP4yblU5Vw1I4u4RJ7L/UDG/eW1J2OW4OKpsUJ4UfW23mW0FTo1PSc5VH51a1OemszszZf563l/xadjluDipbFCmSGpacie45tu/wdE54KYhnclsXo9fvLSQ/UWHwi7HxUFlg/KPwAeSfi3pbuADIj0knavx6tRK5VcX9WbVp7t59N1VYZfj4qCyV+aMBy4DNgGFwKVmNiGehTlXnQzp3pIL+7TmL2+vYO3WPWGX46pYZY8oMbPFZvZXM/tL6a9zcM7BncN7k5Yi7pqyyJtmJJlKB6VzrnytG9fh1vO7eTf0JORB6VwViu6Gvnu/d0NPFh6UzlWh6G7oD77t3dCThQelc1Usuhv6so3eDT0ZeFA6Fwcl3dB/4d3Qk4IHpXNx8Hk39LytvODd0Ks9D0rn4qSkG/rvvBt6tedB6VyclHRD3773IPd5N/RqzYPSuTiKdEPP4rnZa/hozbaKn+ASkgelc3F26/ndaNmwNj9/aSFFh4rDLscdBQ9K5+KsQe007hzem0Xrd/L0rPywy3FHIa5BKWmopGWSVkgaG2P7dyV9LGmepBmSesWzHufCcmGf1t4NvRqLW1BKSgXGAcOAXsDIGEH4rJn1MbNTiLRt+1O86nEuTN4NvXqL5xFlf2CFma0yswPARGBE9AAz2xl1tz7gZ+a6pBXdDX3GJ94NvTqJZ1C2A9ZG3S8IHjuMpO9JWknkiPL7sXYk6UZJuZJyCwsL41Ksc8dDSTf0O1/2bujVSTyDUjEe+9IRo5mNM7POwI+Bn8fakZk9YmbZZpadkeFfJ+6qL++GXj3FMygLgA5R99sD68sZPxG4OI71OJcQoruhr9ni3dCrg3gGZQ7QVVInSenA1cCU6AGSukbd/RrgfalcjfBFN3RvmlEdxC0ozawIGANMBZYAk8xskaS7JV0UDBsjaZGkecAPgeviVY9ziaSkG/q0ZYVMXeTd0BOdqtu/ZtnZ2Zabmxt2Gc4ds6JDxQz/ywx27j3If354NvVr+zdAh0nSHDPLjrXNr8xxLiRp0d3Q3/JPnRKZB6VzIcrOasZV2R14fIZ3Q09kHpTOhezHw3p4N/QE50HpXMi8G3ri86B0LgGUdEP/rXdDT0gelM4lgJJu6Du8G3pC8qB0LkF4N/TE5UHpXALxbuiJyYPSuQQS3Q19gndDTxgelM4lGO+Gnng8KJ1LMJL49YgTOeDd0BOGB6VzCSjLu6EnFA9K5xKUd0NPHB6UziWo6G7oj0z3buhh8qB0LoGVdEP/6zTvhh4mD0rnEpx3Qw+fB6VzCc67oYfPg9K5auD607Po0bohv3plEbv3F4VdTo3jQelcNVDSDX2Dd0MPhQelc9WEd0MPjwelc9VISTf0bz6Vw8J1O8Iup8bwoHSuGmlWP50nb+jPoWLjsoc+4PnctWGXVCN4UDpXzZzSoQmv3HIG/TKbcvvkBfz0xY/9yp04i2tQShoqaZmkFZLGxtj+Q0mLJS2Q9JakzHjW41yyaNGgNuP/pz83DenMsx+u4cqHZ7Ju+96wy0pacQtKSanAOGAY0AsYKalXqWEfAdlmdhIwGbgvXvU4l2zSUlP48dAe/H1UP1YV7mb4g+95A404iecRZX9ghZmtMrMDwERgRPQAM5tmZiXXZc0C2sexHueS0gW9W/PymMFkNKzN6Cc+ZNy0FRQX+xU8VSmeQdkOiP6kuSB4rCzfBP4da4OkGyXlSsotLCyswhKdSw4nZDTgpe8NZvhJbbl/6jJunDCHHXsPhl1W0ohnUCrGYzH/mZN0LZAN3B9ru5k9YmbZZpadkZFRhSU6lzzqpafxwNWncNfXe/HOss2M+OsMlm7cGXZZSSGeQVkAdIi63x5YX3qQpPOAnwEXmdn+ONbjXNKTxA2DOzHxxoHsOXCIi8e9z0sfrQu7rGovnkGZA3SV1ElSOnA1MCV6gKRTgb8TCcnNcazFuRolO6sZr37/DE5q34Qf/HMed728kANF/q2ORytuQWlmRcAYYCqwBJhkZosk3S3pomDY/UAD4HlJ8yRNKWN3zrkj1LJhHZ751gC+dUYnnpqZz9WPzGTjDv+ysqOh6tbfLjs723Jzc8Muw7lq5bUFG7h98nzqpafyl5F9GdS5edglJRxJc8wsO9Y2vzLHuRrgaye1YcqYwTSuW4trH/+QR95d6U2Aj4AHpXM1RJeWDXl5zBlc0LsVv319KTc/M5dd3tuyUjwonatBGtROY9w1ffnZhT15c/EmLvrrDD7Z5C3bKuJB6VwNI4lvn3UCz3xrADv3HmTEuPd5dcGXztxzUTwonauhBp7QnFdvOZOebRox5tmP+PWrizl4yE8hisWD0rkarHXjOjz37YFcf3oWj89YzTce/ZDNn/kpRKV5UDpXw6WnpfDLi3rzwNWn8PG6HQx/cAY5eVvDLiuheFA65wAYcUo7Xvze6dRLT2XkI7N4YsZqP4Uo4EHpnPtcj9aNmHLLGZzToyV3v7qY70+c51+Piwelc66URnVq8fdr+3H7Bd15bcF6Lh73PisLd4VdVqg8KJ1zX5KSIr53ThcmfHMAW3YfYMRf3+eNhRvDLis0HpTOuTIN7tKCV285g84tG/Ddp+dw77+XUlQDTyHyoHTOlattk7pM+s5AvjGgIw9PX8mox2fz6a6a1TrWg9I5V6Haaancc0kf7r/8JOau2cbwB2cwd822sMs6bjwonXOVdkV2B/518+nUShNX/X0mE2bm1YhTiDwonXNHpHfbxrw65kzO6NKCX7y8iB9Nms/eA4fCLiuuPCidc0escb1aPH7dafzw/G68OG8dX3vwPV6YU5C014p7UDrnjkpKivj+uV156ob+pKel8KPn5zPk/nd46oO8pDvC9K+CcM4dMzNj2rLN/G3aSnLzt9G8fjo3DM5i1KAsGtetFXZ5lVLeV0F4UDrnqtTs1Vt56J0VTFtWSIPaaXxjYEe+ObgTLRvVCbu0cnlQOueOu8Xrd/LQ9JW8tmA9aakpXN6vPd856wQym9cPu7SYPCidc6HJ37Kbv7+7ism5BRQVFzP8pLZ89+zO9GrbKOzSDuNB6ZwL3ead+3h8xmqenpXP7gOHOKd7Bjef04XTspqFXRoQ4tfVShoqaZmkFZLGxth+lqS5kookXR7PWpxz4WrZqA4/ubAnH4w9l9u+2o35BTu44uGZXP7QB7y9dFNCn7getyNKSanAcuB8oADIAUaa2eKoMVlAI+A2YIqZTa5ov35E6Vxy2HvgEP/MWcOj761m3fa99GjdkJuGdOZrfdqQlnr8z1wM64iyP7DCzFaZ2QFgIjAieoCZ5ZnZAiA5z1J1zpWpbnoq1w/uxDu3D+GPV5xMUbHxvxPn8ZU/TufpWfnsO5g452LGMyjbAWuj7hcEjx0xSTdKypWUW1hYWCXFOecSQ63UFC7r1543f3AWfx/Vj6b10/n5Sws54zuE2FsAAAl2SURBVPfTeOidlXy272DYJcY1KBXjsaOa55vZI2aWbWbZGRkZx1iWcy4RpaSIC3q35qWbT+fZbw+gZ5uG/P6NpZx+79vcP3VpqK3d0uK47wKgQ9T99oB/y7pzrlySOL1zC07v3IKPC3bw0PQV/O2dlTz23mquOq0D3z7zBDo0q3dca4pnUOYAXSV1AtYBVwPXxPH3OeeSTJ/2jfnbN/qxsnAXj0xfxXOz1/DMh2u46OS23DSkM91aNTwudcT1PEpJFwL/B6QCT5jZPZLuBnLNbIqk04AXgabAPmCjmfUub5++6u1czbVhx14ee281z81ew54DhzivZytuPqczfTs2PeZ9+wnnzrmksm33AZ6amceTH+Sxfc9BBnRqxs3ndOGsri2QYi2PVMyD0jmXlHbvL+K52Wt47L3VbNy5j95tG3HTkM4MO7ENqSlHFpgelM65pHagqJiXPlrHw9NXsurT3WQ1r8ejo7PpegSfYZYXlPFczHHOueMiPS2FK0/rEDkfc9FGJuasrdKVcQ9K51zSSE0Rw/q0YVifNlW6X/8qCOecq4AHpXPOVcCD0jnnKuBB6ZxzFfCgdM65CnhQOudcBTwonXOuAh6UzjlXgWp3CaOkQiD/CJ/WAvg0DuUcLa+nfIlWDyReTV5P+Y6mnkwzi9kZvNoF5dGQlFvWNZxh8HrKl2j1QOLV5PWUr6rr8am3c85VwIPSOecqUFOC8pGwCyjF6ylfotUDiVeT11O+Kq2nRnxG6Zxzx6KmHFE659xR86B0zrkKJHVQShoqaZmkFZLGJkA9T0jaLGlh2LUASOogaZqkJZIWSfrfkOupI2m2pPlBPb8Ks54SklIlfSTp1bBrAZCUJ+ljSfMkhf69KJKaSJosaWnwZ2lQiLV0D96XkttOST845v0m62eUklKB5cD5QAGR7xkfaWaLQ6zpLGAXMN7MTgyrjqh62gBtzGyupIbAHODisN4jRb4+r76Z7ZJUC5gB/K+ZzQqjnqi6fghkA43MbHiYtQT15AHZZpYQJ3hLegp4z8wek5QO1DOz7QlQVyqwDhhgZkd6kcphkvmIsj+wwsxWmdkBYCIwIsyCzOxdYGuYNUQzsw1mNjf4+TNgCdAuxHrMzHYFd2sFt1D/JZfUHvga8FiYdSQqSY2As4DHAczsQCKEZOBcYOWxhiQkd1C2A9ZG3S8gxBBIdJKygFOBD0OuI1XSPGAz8B8zC7Ue4P+AO4DikOuIZsCbkuZIujHkWk4ACoF/BB9PPCapfsg1lbgaeK4qdpTMQRnrS32T83OGYySpAfAC8AMz2xlmLWZ2yMxOAdoD/SWF9hGFpOHAZjObE1YNZRhsZn2BYcD3go90wpIG9AUeMrNTgd1AIqwHpAMXAc9Xxf6SOSgLgA5R99sD60OqJWEFnwW+ADxjZv8Ku54SwfTtHWBoiGUMBi4KPhOcCHxF0tMh1gOAma0P/rsZeJHIx0xhKQAKoo78JxMJzrANA+aa2aaq2FkyB2UO0FVSp+Bfl6uBKSHXlFCCxZPHgSVm9qcEqCdDUpPg57rAecDSsOoxs5+YWXszyyLy5+dtM7s2rHoAJNUPFt4IprhfBUI7i8LMNgJrJXUPHjoXCG3BNMpIqmjaDUn8vd5mViRpDDAVSAWeMLNFYdYk6TlgCNBCUgFwl5k9HmJJg4FRwMfB54IAPzWz10Oqpw3wVLBamQJMMrOEOCUngbQCXoz8G0ca8KyZvRFuSdwCPBMckKwCbgizGEn1iJzt8p0q22eynh7knHNVJZmn3s45VyU8KJ1zrgIelM45VwEPSuecq4AHpXPOVcCD0sUk6YPgv1mSrqniff801u+KF0kXS7ozTvvuF3TyWSHpweDcVCQ1k/QfSZ8E/20aPK5g3ApJCySVe3K2pP+WPNeFx4PSxWRmpwc/ZgFHFJTBeZDlOSwoo35XvNwB/O1Yd1LG63oIuBHoGtxKriQaC7xlZl2Bt/jisr5hUWNvDJ5fngnAzcdWuTtWHpQuJkklXXzuBc4MevvdGjStuF9STnBE9J1g/JCgt+WzwMfBYy8FjRsWlTRvkHQvUDfY3zPRvys42rpf0sLgKO2qqH2/E9Xz8JmoI7d7JS0OavlDjNfRDdhf0pJM0pOSHpb0nqTlwfXcJc04KvW6ovbdhkjrtZkWOSF5PHBxsHkE8FTw81OlHh8fdEqaBTSR1Ca4vRu8LwslnRmMn0LkKhMXoqS9MsdVmbHAbSV9GIPA22Fmp0mqDbwv6c1gbH/gRDNbHdz/HzPbGlyOmCPpBTMbK2lM0PiitEuBU4CTiXyBfY6kd4NtpwK9iVyv/z4wWNJi4BKgh5lZyeWPpQwG5pZ6LAs4G+gMTJPUBRh9BK+rRDsi1zqXiO5Q1crMNkCknZ2kllHPidXV6mxgqpndExy51gueu01SbUnNzWxLjNfnjgMPSnekvgqcJOny4H5jItPIA8DsUmHyfUmXBD93CMaV95f9DOA5MzsEbJI0HTgN2BnsuwAguNwyC5gF7AMek/QaEOtyxzZE2oBFm2RmxcAnklYBPY7wdZU4mg5VZT0nB3hCkSYlL5nZvKjtm4G2lP/euTjyqbc7UgJuMbNTglsnMys58tr9+SBpCJGmFoPM7GTgI6BOJfZdlv1RPx8C0sysiMjR3gtEpraxrnneG+P3lg4zo5Kvq5QCIl2pSkR3qNoUTM1Lpuibo57zpa5WQVPns4h05J4gaXTUmDrB63Ah8aB0FfkMaBh1fypwU3Dkg6Ruit2otTGwzcz2SOoBDIzadrDk+aW8C1wVfF6YQSQ4ZpdVmCJ9NBsHTTx+QGTaXtoSoEupx66QlCKpM5HGs8uO4HV9LphafyZpYPCZ6Wjg5WDzFOC64OfrSj0+Ovg8diCR6f4GSZlEel8+SqSjU9+gDgGtgbzyanHx5VNvV5EFQJGk+cCTwANEpr1zg7/EhXyxUBHtDeC7khYQCaLo7715BFggaa6ZfSPq8ReBQcB8Ikd5d5jZxiBoY2kIvCypDpEjwltjjHkX+KMk2RcdYJYB04l04vmume2T9FglX1dpNxF5X+oC/w5uEFkEmyTpm8Aa4Irg8deBC4EVwB6+6LQzBLhd0kEi36tUckTZD5gVHD27kHj3IJf0JD0AvGJm/5X0JPCqmU0OuaxKCWqfYmZvhV1LTeZTb1cT/JZgFbkaWughGT4/onTOuQr4EaVzzlXAg9I55yrgQemccxXwoHTOuQp4UDrnXAX+H/NySSk3ZcmlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "parameters = L_layer_model(train_x, train_y, layers_dims, num_iterations = 8000, print_cost = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "  \n",
    "  Now, we predict our model's performance on unseen-data(test data)\n",
    "    \n",
    "    \n",
    "    \n",
    "   </blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(X, y, parameters):\n",
    "    \"\"\"\n",
    "    This function is used to predict the results of a  L-layer neural network.\n",
    "    \n",
    "    Arguments:\n",
    "    X -- data set of examples you would like to label\n",
    "    parameters -- parameters of the trained model\n",
    "    \n",
    "    Returns:\n",
    "    p -- predictions for the given dataset X\n",
    "    \"\"\"\n",
    "    \n",
    "    m = X.shape[1]\n",
    "    n = len(parameters) // 2 # number of layers in the neural network\n",
    "    p = np.zeros((1,m))\n",
    "    \n",
    "    # Forward propagation\n",
    "    probas, caches = L_model_forward(X, parameters)\n",
    "\n",
    "    \n",
    "    # convert probas to 0/1 predictions\n",
    "    for i in range(0, probas.shape[1]):\n",
    "        if probas[0,i] > 0.5:\n",
    "            p[0,i] = 1\n",
    "        else:\n",
    "            p[0,i] = 0\n",
    "    \n",
    "    \n",
    "    print(\"Accuracy: \"  + str(np.sum((p == y)/m)))\n",
    "        \n",
    "    return p\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9644484958979032\n"
     ]
    }
   ],
   "source": [
    "pred_train = predict(train_x, train_y, parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9636363636363636\n"
     ]
    }
   ],
   "source": [
    "pred_test = predict(test_x, test_y, parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>\n",
    "    \n",
    "   We can see that our model, performs very well on both train as well as test data. This suggests, that we have low variance and low bias and we are not overfitting our model on training data.\n",
    "    \n",
    "    \n",
    "  </blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}