HOTH_2018_ML_Workshop.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "HOTH_2018_ML_Workshop.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "CI9JoDW05ozh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Cats vs Dogs\n",
        "<img src=\"https://kaggle2.blob.core.windows.net/competitions/kaggle/3362/media/woof_meow.jpg\" width=\"350\" height=\"240\"/>\n",
        "\n",
        "Distinguishing a picture of a cat from a picture of a dog is easy for humans to do. However, a computer has much more trouble with this task. Instead of trying to explicitly write logic to identify cats and dogs, let's see how we can use logistic regression to allow our machine to **learn** how to tell the two apart.\n",
        "\n",
        "#### Review: Unsupervised vs Supervised Learning\n",
        "*   Supervised learning uses existing inputs and outputs to create a mapping function from inputs to outputs\n",
        "  * Can be used for regression problems, where outputs are continuous, or classification problems, where out\n",
        "*   Unsupervised learning finds relationships and clusters in input data\n",
        "\n",
        "#### Review: Logistic Regression\n",
        "*   Example of supervised learning applied to a classification problem\n",
        "*   Simple classification algorithm that maps inputs to discrete output classes\n",
        "*   Remember, 0 is the negative class and 1 is the positive class.\n",
        "\n",
        "We use the hypothesis function:\n",
        "\n",
        "$$z^{i} = w^{T}x^{i} + b$$\n",
        "\n",
        "$${h(x)}^{i} = a^{i} = sigmoid(z^{i})$$\n",
        "\n",
        "To make an output prediction for each training example, representing the probability of the training example to be of the positive class, 1. We use the following loss function to calculate how accurate our prediction is:\n",
        "\n",
        "$$Loss = L(y^{i} , a^{i}) = -y^{i}log(a^{i}) - (1 - y^{i})log(1 - a^{i})$$\n",
        "\n",
        "Then, we apply gradient descent to optimize our weights.\n",
        "\n",
        "#### Some of the things you will learn\n",
        "* General architecture of a learning algorithm\n",
        "* initializing parameters\n",
        "* doing forward propagation\n",
        "* calculating the cost function\n",
        "* doing back propagation with gradient descent\n",
        "\n",
        "#### Instructions\n",
        "1. You will write your code between \n",
        "```python \n",
        "### START CODE HERE ###\n",
        "```\n",
        "```python \n",
        "### END CODE HERE ###\n",
        "```\n",
        "\n",
        "2. Do not use any loops unless explicitly told to do so\n",
        "\n",
        "## Packages\n",
        "First let's import some libraries\n",
        "* [numpy](http://www.numpy.org/) is used for scientific computing\n",
        "* [matplotlib](https://matplotlib.org/) is used for plotting data\n",
        "* [pytables](http://www.pytables.org/) is used to read in our data\n",
        "* [PIL](https://pillow.readthedocs.io/en/latest/) (also known as Pillow) is used for working with images"
      ]
    },
    {
      "metadata": {
        "id": "go2s6Qg66SKa",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "!pip install tables\n",
        "import tables\n",
        "from PIL import Image"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZZopwzZUngti",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **Data**\n",
        "\n",
        "The data set, \"data.h5\", contains:\n",
        "\n",
        "*   a training set of m_train images labeled as cat (y=0) or dog (y=1)\n",
        "*   a test set of m_test images labeled as cat or dog\n",
        "*   each image is of shape (num_px, num_px, 3) where 3 is for the 3 channels (RGB). Thus, each image is square (height = num_px) and (width = num_px).\n",
        "\n",
        "\n",
        "We will be using this data to create an image recognition algorithm to classify images as cat or dog. Let's load the data."
      ]
    },
    {
      "metadata": {
        "id": "4AlP91YBvklP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* * *\n",
        "### ** Run this cell if using Google Collaboratory**\n",
        "After running this cell you will be asked to choose a file to upload. Navigate to where you cloned the repository and upload the `data.h5` file. This will probably take a minute or so to finish uploading."
      ]
    },
    {
      "metadata": {
        "id": "GTtwdOzTmxun",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "\n",
        "data = files.upload()\n",
        "def keys(hf):\n",
        "    return [key for key in hf.keys()]\n",
        "with open(\"data.h5\", 'wb', 1) as f:\n",
        "    f.write(data[keys(data)[0]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WGwoqHzyQvfF",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "for fn in keys(data):\n",
        "  f = tables.open_file(fn, mode='r')\n",
        "  \n",
        "  x_train = f.root.train_inputs[()]\n",
        "  y_train = f.root.train_targets[()]\n",
        "  x_test = f.root.test_inputs[()]\n",
        "  y_test = f.root.test_targets[()]\n",
        "\n",
        "  y_train = np.asarray(y_train).reshape(1,275)\n",
        "  y_test = np.asarray(y_test).reshape(1, 49)\n",
        "  f.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pFXGqk42vw2T",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* * *"
      ]
    },
    {
      "metadata": {
        "id": "UAiG_hMUD_OQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Run this cell if using a Jupyter Notebook"
      ]
    },
    {
      "metadata": {
        "id": "_18sW9JrA0ub",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "f = tables.open_file('data.h5', mode='r')\n",
        "  \n",
        "x_train = f.root.train_inputs[()]\n",
        "y_train = f.root.train_targets[()]\n",
        "x_test = f.root.test_inputs[()]\n",
        "y_test = f.root.test_targets[()]\n",
        "\n",
        "y_train = np.asarray(y_train).reshape(1,275)\n",
        "y_test = np.asarray(y_test).reshape(1, 49)\n",
        "f.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ho0kXqN0BFMq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* * *"
      ]
    },
    {
      "metadata": {
        "id": "prKDWWdNp-lQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Here's an example of a picture in this dataset. To look at other pictures, change `index` to a number from 0 to 224."
      ]
    },
    {
      "metadata": {
        "id": "1IO1L8UwhXvH",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "index = 10\n",
        "\n",
        "plt.figure(figsize = (3, 3))\n",
        "plt.imshow(x_train[index].astype('uint8'))\n",
        "plt.title(str(y_train[0,index]) + \" - \" + (\"cat\" if y_train[0,index] == 0 else \"dog\"), size='xx-large')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ObezhUnUsqaZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **Exercise**\n",
        "\n",
        "Now we need to set the dimensions of our matrices. Find the values for:\n",
        "\n",
        "* m_train (number of training examples)\n",
        "* m_test (number of testing examples)\n",
        "* num_px (dimensions of each image)\n",
        "\n",
        "Hint: x_train is a numpy-array of shape (m_train, num_px, num_px, 3). For instance, you can access m_train by writing:\n",
        "\n",
        "\n",
        "```python\n",
        " x_train.shape[0]\n",
        "```\n",
        "\n",
        "Using this information, write lines to access m_test and num_px. (Remember, we defined x_train, y_train, x_test, and y_test, all of which you can use.)"
      ]
    },
    {
      "metadata": {
        "id": "QcFHQAFwjok1",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "### START CODE HERE ### (≈ 3 lines of code)\n",
        "m_train = x_train.shape[0]\n",
        "m_test = x_test.shape[0]\n",
        "num_px = x_train.shape[1]\n",
        "### END CODE HERE ###\n",
        "\n",
        "print (\"Number of training examples: m_train = \" + str(m_train))\n",
        "print (\"Number of testing examples: m_test = \" + str(m_test))\n",
        "print (\"Height/Width of each image: num_px = \" + str(num_px))\n",
        "print (\"Each image is of size: (\" + str(num_px) + \", \" + str(num_px) + \", 3)\")\n",
        "print (\"x_train shape: \" + str(x_train.shape))\n",
        "print (\"y_train shape: \" + str(y_train.shape))\n",
        "print (\"x_test shape: \" + str(x_test.shape))\n",
        "print (\"y_test shape: \" + str(y_test.shape))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fYBOinqc0h-2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Expected Output**\n",
        "\n",
        "Expected Output for m_train, m_test and num_px:\n",
        "\n",
        "* **m_train:**\t225\n",
        "* **m_test:**\t25\n",
        "* **num_px:**\t64\n",
        "\n",
        "\n",
        "## **Exercise**\n",
        "For convenience, you should now reshape images of shape (num_px, num_px, 3) in a numpy-array of shape (num_px $*$ num_px $*$ 3, 1).\n",
        "\n",
        "After this, our training (and test) dataset is a numpy-array where each column represents a \"flattened\" image.\n",
        "\n",
        "There should be m_train (respectively m_test) columns.\n",
        "\n",
        "**Task: ** Reshape the training and test data sets so that images of size (num_px, num_px, 3) are flattened into single vectors of shape (num_px $*$ num_px $*$ 3, 1).\n",
        "\n",
        "Hint: The following command flattens a matrix of shape (a, b, c, d) to a matrix of shape (b $*$ c $*$ d, a):\n",
        "\n",
        "\n",
        "\n",
        "```python\n",
        "X_flatten = X.reshape(X.shape[0], -1).T \n",
        "```\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "zfHWfWS65f_f",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Reshape the training and test examples\n",
        "\n",
        "### START CODE HERE ### (≈ 2 lines of code)\n",
        "\n",
        "x_train_flatten = x_train.reshape(x_train.shape[0], -1).T\n",
        "x_test_flatten = x_test.reshape(x_test.shape[0], -1).T\n",
        "\n",
        "### END CODE HERE ###\n",
        "\n",
        "print (\"train_set_x_flatten shape: \" + str(x_train_flatten.shape))\n",
        "print (\"train_set_y shape: \" + str(y_train.shape))\n",
        "print (\"test_set_x_flatten shape: \" + str(x_test_flatten.shape))\n",
        "print (\"test_set_y shape: \" + str(y_test.shape))\n",
        "print (\"sanity check after reshaping: \" + str(x_train_flatten[0:5,0]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QPHzrMwz1xZv",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "x_train_flatten /= 255.\n",
        "x_test_flatten /= 255."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8V5VskE92AqO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Expected Output:\n",
        "\n",
        "* **x_train_flatten shape:**\t(12288, 225)\n",
        "* **y_train_flatten shape:**\t(1, 225)\n",
        "* **x_test_flatten shape:**\t(12288, 25)\n",
        "* **y_test_flatten shape:**\t(1, 25)\n",
        "* **sanity check after reshaping:**\t[137. 98. 59. 146. 107.]"
      ]
    },
    {
      "metadata": {
        "id": "gocy2IlZ2_vK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **Architecture of the Learning Algorithm**\n",
        "\n",
        "Now it's time to get started on the learning algorithm itself! You will build a logistic regression algorithm.\n",
        "\n",
        "\n",
        "**Mathematical algorithm:**\n",
        "\n",
        "Given $x^{i}$ which is 1 training example of size (height $*$ width $*$ 3 , 1), \n",
        "\n",
        "$$z^{i} = w^{T}x^{i} + b$$\n",
        "\n",
        "$${h(x)}^{i} = a^{i} = sigmoid(z^{i})$$\n",
        "\n",
        "$$Loss = L(y^{i} , a^{i}) = -y^{i}log(a^{i}) - (1 - y^{i})log(1 - a^{i})$$\n",
        "\n",
        "Then to calculate the cost over all m training samples,\n",
        "\n",
        "$$Cost = J = \\frac{1}{m}\\sum _{i = 1}^{m} L(y^{i}, a^{i})$$\n",
        "\n",
        "\n",
        "The superscript $^{T}$ just means transpose. What this algorithm is doing is calculating the model predictions for each training sample, and then finding the average loss over all the samples where loss is a measure of how far off the predictions are from the correct labels.\n",
        "\n",
        "**Next steps are:**\n",
        "\n",
        "\n",
        "1.   Define model structure\n",
        "2.   Initialize parameters\n",
        "3.   Implement gradient descent: calculate loss, gradient, and update parameters.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "q5b_p0wd3nMX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Helper Functions\n",
        "\n",
        "Implement a ** sigmoid function**. Remember from **slide 24** that we need a sigmoid function in order to implement our hypothesis function since it ouputs a number between 0 and 1. Here's a reminder of what sigmoid looks like:\n",
        "$$ \\frac{1}{1 + e^{-x}} $$\n",
        "Hint: you can use `np.exp()` for *e*"
      ]
    },
    {
      "metadata": {
        "id": "a7NAxV035C7d",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def sigmoid(z):\n",
        "    \"\"\"\n",
        "    Compute the sigmoid of z\n",
        "\n",
        "    Arguments:\n",
        "    z -- A scalar or numpy array of any size.\n",
        "\n",
        "    Return:\n",
        "    s -- sigmoid(z)\n",
        "    \"\"\"\n",
        "\n",
        "    ### START CODE HERE ### (≈ 1 line of code)\n",
        "    s = 1 / (1 + np.exp(-z))\n",
        "    ### END CODE HERE ###\n",
        "    \n",
        "    return s"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fMcmfXKN2sWn",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "print(\"sigmoid(0) = \" + str(sigmoid(0)))\n",
        "print(\"sigmoid(10) = \" + str(sigmoid(9.2)))\n",
        "print(\"sigmoid(-10) = \" + str('%f' % sigmoid(-10)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "L8vqvoz05-0-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Expected Output:\n",
        "\n",
        "* **sigmoid(0) =** 0.5\n",
        "* **sigmoid(10) =** 0.9998989708060922\n",
        "* **sigmoid(-10) =** 0.000045"
      ]
    },
    {
      "metadata": {
        "id": "9hRFo5tn_IB_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Logistic-curve.svg/1200px-Logistic-curve.svg.png\" width=\"350\" height=\"240\"/>\n",
        "\n",
        "As you can see, the more negative the input to sigmoid, the closer the output is to 0 and the more positive the input, the closer the output is to 1. Becuase of this, we want our model to learn a representation that passes large positive numbers to sigmoid if the picture is a dog, and large negative numbers if the picture is a cat."
      ]
    },
    {
      "metadata": {
        "id": "FJFWcMiWRQOm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Initializing Parameters\n",
        "\n",
        "We need a function that inputs a matrix size, and then creates a weight matrix and a bias that are initialized to zero.\n",
        "\n",
        "Hint: `np.zeros(shape=(x,y))` returns a numpy array of zeros of shape (x,y)"
      ]
    },
    {
      "metadata": {
        "id": "6TiyMDhl-ptf",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def initialize_with_zeros(dim):\n",
        "    \"\"\"\n",
        "    This function creates a vector of zeros of shape (dim, 1) for w and initializes b to 0.\n",
        "    \n",
        "    Argument:\n",
        "    dim -- size of the w vector we want (or number of parameters in this case)\n",
        "    \n",
        "    Returns:\n",
        "    w -- initialized vector of shape (dim, 1)\n",
        "    b -- initialized scalar (corresponds to the bias)\n",
        "    \"\"\"\n",
        "    \n",
        "    ### START CODE HERE ### (≈ 2 line of code)\n",
        "    w = np.zeros(shape=(dim, 1))\n",
        "    b = 0\n",
        "    ### END CODE HERE ###\n",
        "\n",
        "    assert(w.shape == (dim, 1))\n",
        "    assert(isinstance(b, float) or isinstance(b, int))\n",
        "    \n",
        "    return w, b"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2tuL0hfwy9iG",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "dim = 2\n",
        "w, b = initialize_with_zeros(dim)\n",
        "print (\"w = \" + str(w))\n",
        "print (\"b = \" + str(b))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Dx_7TlbYyWeQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Forward Propagation\n",
        "\n",
        "Now it's time to use our helper functions to implement forward propagation. During forward propagation we will compute the model predictions, the cost over all of our samples and the gradients for the weights and bias. Here are the steps we will need to take:\n",
        "\n",
        "1. use the equation $w^{T}X + b$ to compute $Z$ *(the capital $X$ means that it is a matrix containing all of our $x^{i}$ samples)*\n",
        "2. use our sigmoid helper function to compute our predictions $A$\n",
        "3. use the cost function described earlier to compute our cost $J$\n",
        "4. compute the gradients using the equations given below\n",
        "\n",
        "As a reminder our cost function is $J = \\frac{1}{m}\\sum _{i = 1}^{m} -y^{i}log(a^{i}) - (1 - y^{i})log(1 - a^{i})$\n",
        "\n",
        "**Gradients:**\n",
        "\n",
        "\n",
        "$$db = \\frac{\\partial J}{\\partial b} = \\frac{1}{m}\\sum _{i = 1}^{m} a^{i} - y^{i}$$\n",
        "$$$$\n",
        "$$dw = \\frac{\\partial J}{\\partial w} = \\frac{1}{m}\\sum _{i = 1}^{m} (a^{i} - y^{i})x^{i}$$\n",
        "\n",
        "If you have taken multivariable calculus then these equations are the partial derivatives of our cost function $J$ with respect to our weights $w$ and our bias $b$. By taking their negatives we can optimize our cost function in the direction of steepest descent. If the last two scentences made no sense then fear not. You can simply implement these equations without fully understanding them. Just know that using these equations we can reduce our cost and move closer to a loss of 0 (i.e. a correct prediction).\n",
        "$$$$\n",
        "\n",
        "Hint: \n",
        "* you can use `np.dot()` to multiply two matrices together\n",
        "* you can use `np.sum()` to take the sum of all the elements in a matrix"
      ]
    },
    {
      "metadata": {
        "id": "TrLlTmWi6noP",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def propagate(w, b, X, Y):\n",
        "    \"\"\"\n",
        "    Implement the cost function and its gradient for the propagation explained above\n",
        "\n",
        "    Arguments:\n",
        "    w -- weights, a numpy array of size (num_px * num_px * 3, 1)\n",
        "    b -- bias, a scalar\n",
        "    X -- data of size (num_px * num_px * 3, number of examples)\n",
        "    Y -- true \"label\" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)\n",
        "\n",
        "    Return:\n",
        "    cost -- negative log-likelihood cost for logistic regression\n",
        "    dw -- gradient of the loss with respect to w, thus same shape as w\n",
        "    db -- gradient of the loss with respect to b, thus same shape as b\n",
        "    \n",
        "    Tips:\n",
        "    - Write your code step by step for the propagation\n",
        "    \"\"\"\n",
        "    \n",
        "    m = X.shape[1]\n",
        "    \n",
        "    # FORWARD PROPAGATION (FROM X TO COST)\n",
        "    ### START CODE HERE ### (≈ 2 lines of code)\n",
        "    A = sigmoid(np.dot(w.T, X) + b)  # compute activation\n",
        "    cost = (- 1 / m) * np.sum(Y * np.log(A) + (1 - Y) * (np.log(1 - A)))  # compute cost\n",
        "    ### END CODE HERE ###\n",
        "    \n",
        "    # BACKWARD PROPAGATION (TO FIND GRAD)\n",
        "    ### START CODE HERE ### (≈ 2 lines of code)\n",
        "    dw = (1 / m) * np.dot(X, (A - Y).T)\n",
        "    db = (1 / m) * np.sum(A - Y)\n",
        "    ### END CODE HERE ###\n",
        "\n",
        "    assert(dw.shape == w.shape)\n",
        "    assert(db.dtype == float)\n",
        "    cost = np.squeeze(cost)\n",
        "    assert(cost.shape == ())\n",
        "    \n",
        "    grads = {\"dw\": dw,\n",
        "             \"db\": db}\n",
        "    \n",
        "    return grads, cost"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mvhgpoR08jlB",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "w, b, X, Y = np.array([[1], [2]]), 2, np.array([[1,2], [3,4]]), np.array([[1, 0]])\n",
        "grads, cost = propagate(w, b, X, Y)\n",
        "print (\"dw = \" + str(grads[\"dw\"]))\n",
        "print (\"db = \" + str(grads[\"db\"]))\n",
        "print (\"cost = \" + str(cost))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7QXJQVni9QsA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Expected Output:\n",
        "* **dw = ** [[ 0.99993216] [ 1.99980262]]\n",
        "* **db = **  0.499935230625\n",
        "* **cost = ** 6.000064773192205"
      ]
    },
    {
      "metadata": {
        "id": "YJ65bokA-1jZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Backpropagation\n",
        "\n",
        "Now we want to use the gradients that we computed earlier to perform back propagation. Back propagation is process we use to update our weights bias by a small amount so that our model gets a little better better at making predictions. Here are the equations in case you forgot:\n",
        "\n",
        "$$w = w - \\alpha * dw$$\n",
        "$$w = b - \\alpha * db$$\n",
        "\n",
        "$\\alpha$ is our learning rate, which is often a small number like .01 or .001. If you remember from the slides, we want to update our weights and bias without overshooting the minimum for our cost function. The picture below illustrates this.\n",
        "\n",
        "<img src=\"https://storage.googleapis.com/aibootcamp/Week%201/assets/divergence.jpg\" width=\"350\" height=\"240\"/>"
      ]
    },
    {
      "metadata": {
        "id": "OUfiMkxY-khe",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):\n",
        "    \"\"\"\n",
        "    This function optimizes w and b by running a gradient descent algorithm\n",
        "    \n",
        "    Arguments:\n",
        "    w -- weights, a numpy array of size (num_px * num_px * 3, 1)\n",
        "    b -- bias, a scalar\n",
        "    X -- data of shape (num_px * num_px * 3, number of examples)\n",
        "    Y -- true \"label\" vector (containing 0 cat, 1 if dog), of shape (1, number of examples)\n",
        "    num_iterations -- number of iterations of the optimization loop\n",
        "    learning_rate -- learning rate of the gradient descent update rule\n",
        "    print_cost -- True to print the loss every 100 steps\n",
        "    \n",
        "    Returns:\n",
        "    params -- dictionary containing the weights w and bias b\n",
        "    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function\n",
        "    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.\n",
        "    \n",
        "    Tips:\n",
        "    You basically need to write down two steps and iterate through them:\n",
        "        1) Calculate the cost and the gradient for the current parameters. Use propagate().\n",
        "        2) Update the parameters using gradient descent rule for w and b.\n",
        "    \"\"\"\n",
        "    \n",
        "    costs = []\n",
        "    \n",
        "    for i in range(num_iterations):\n",
        "        \n",
        "        \n",
        "        # Cost and gradient calculation (≈ 1 line of code)\n",
        "        ### START CODE HERE ### \n",
        "        grads, cost = propagate(w, b, X, Y)\n",
        "        ### END CODE HERE ###\n",
        "        \n",
        "        # Retrieve derivatives from grads\n",
        "        dw = grads[\"dw\"]\n",
        "        db = grads[\"db\"]\n",
        "        \n",
        "        # update rule (≈ 2 lines of code)\n",
        "        ### START CODE HERE ###\n",
        "        w = w - learning_rate * dw\n",
        "        b = b - learning_rate * db\n",
        "        ### END CODE HERE ###\n",
        "        \n",
        "        # Record the costs\n",
        "        if i % 100 == 0:\n",
        "            costs.append(cost)\n",
        "        \n",
        "        # Print the cost every 100 training examples\n",
        "        if print_cost and i % 100 == 0:\n",
        "            print (\"Cost after iteration %i: %f\" % (i, cost))\n",
        "    \n",
        "    params = {\"w\": w,\n",
        "              \"b\": b}\n",
        "    \n",
        "    grads = {\"dw\": dw,\n",
        "             \"db\": db}\n",
        "    \n",
        "    return params, grads, costs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZZJsbo7CDH5E",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "\n",
        "params, grads, costs = optimize(w, b, X, Y, num_iterations= 100, learning_rate = 0.009, print_cost = False)\n",
        "\n",
        "print (\"w = \" + str(params[\"w\"]))\n",
        "print (\"b = \" + str(params[\"b\"]))\n",
        "print (\"dw = \" + str(grads[\"dw\"]))\n",
        "print (\"db = \" + str(grads[\"db\"]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "h5Qyd2tVDMz7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Expected Output:\n",
        "* **w = ** [[ 0.1124579 ] [ 0.23106775]]\n",
        "* **b = ** 1.55930492484\n",
        "* **dw = ** [[ 0.90158428] [ 1.76250842]]\n",
        "* **db = ** 0.430462071679"
      ]
    },
    {
      "metadata": {
        "id": "0fFyHj1fpK7E",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Predictions\n",
        "\n",
        "Once we have trained our model, we need a way to use it. For that we will create a function called predict that will take an input of training samples, a weight matrix, and a bias and then output predictions of 0 for cat or 1 for dog. Below, A refers to the Activation function, which is used to scale our inputs from 0 to 1.\n",
        "\n",
        "**Steps:**\n",
        "1. compute $\\hat{Y} = A = \\sigma(w^{T}X + b)$\n",
        "2. convert those predictions to a 0 if the activation is <= 0.5 and a 1 if it is  > 0.5\n",
        "\n",
        "Hint: If X is an array of numbers, X > 1 will return a boolean of array where every element is true if the element in X is > 1 and false otherwise. To convert a numpy array `arr` of one type to a numpy array of int type you can use `arr.astype('uint8')`"
      ]
    },
    {
      "metadata": {
        "id": "lCs8XcQZDoXk",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def predict(w, b, X):\n",
        "    '''\n",
        "    Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)\n",
        "    \n",
        "    Arguments:\n",
        "    w -- weights, a numpy array of size (num_px * num_px * 3, 1)\n",
        "    b -- bias, a scalar\n",
        "    X -- data of size (num_px * num_px * 3, number of examples)\n",
        "    \n",
        "    Returns:\n",
        "    Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X\n",
        "    '''\n",
        "    \n",
        "    m = X.shape[1]\n",
        "    Y_prediction = np.zeros((1, m))\n",
        "    w = w.reshape(X.shape[0], 1)\n",
        "    \n",
        "    # Compute vector \"A\" predicting the probabilities of a cat being present in the picture\n",
        "    ### START CODE HERE ### (≈ 1 line of code)\n",
        "    A = sigmoid(np.dot(w.T, X) + b)\n",
        "    ### END CODE HERE ###\n",
        "    \n",
        "\n",
        "    for i in range(A.shape[1]):\n",
        "        # Convert probabilities a[0,i] to actual predictions p[0,i]\n",
        "        ### START CODE HERE ### (≈ 4 lines of code)\n",
        "        Y_prediction[0, i] = 1 if A[0, i] > 0.5 else 0\n",
        "        ### END CODE HERE ###\n",
        "    \n",
        "    assert(Y_prediction.shape == (1, m))\n",
        "    \n",
        "    return Y_prediction"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fUDpqtiB1WT2",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "print(\"predictions = \" + str(predict(w, b, X)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NQR9wuGi5uQJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Model\n",
        "\n",
        "Now that we all the peices of logistic regression, lets put all together into one function. \n",
        "\n",
        "Implement the model function. Use the following notation:\n",
        "\n",
        "* `Y_prediction` for your predictions on the test set\n",
        "* `Y_prediction_train` for your predictions on the train set\n",
        "* `w`, `costs`, `grads` for the outputs of optimize()"
      ]
    },
    {
      "metadata": {
        "id": "TWuil_hQs6JA",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def model(X_train, Y_train, X_test, Y_test, num_iterations=2000, learning_rate=0.01, print_cost=False):\n",
        "    \"\"\"\n",
        "    Builds the logistic regression model by calling the function you've implemented previously\n",
        "    \n",
        "    Arguments:\n",
        "    X_train -- training set represented by a numpy array of shape (num_px * num_px * 3, m_train)\n",
        "    Y_train -- training labels represented by a numpy array (vector) of shape (1, m_train)\n",
        "    X_test -- test set represented by a numpy array of shape (num_px * num_px * 3, m_test)\n",
        "    Y_test -- test labels represented by a numpy array (vector) of shape (1, m_test)\n",
        "    num_iterations -- hyperparameter representing the number of iterations to optimize the parameters\n",
        "    learning_rate -- hyperparameter representing the learning rate used in the update rule of optimize()\n",
        "    print_cost -- Set to true to print the cost every 100 iterations\n",
        "    \n",
        "    Returns:\n",
        "    d -- dictionary containing information about the model.\n",
        "    \"\"\"\n",
        "    \n",
        "    ### START CODE HERE ###\n",
        "    # initialize parameters with zeros (≈ 1 line of code)\n",
        "    w, b = initialize_with_zeros(X_train.shape[0])\n",
        "\n",
        "    # Gradient descent (≈ 1 line of code)\n",
        "    parameters, grads, costs = optimize(w, b, X_train, Y_train, num_iterations, learning_rate, print_cost)\n",
        "    \n",
        "    # Retrieve parameters w and b from dictionary \"parameters\"\n",
        "    w = parameters[\"w\"]\n",
        "    b = parameters[\"b\"]\n",
        "    \n",
        "    # Predict test/train set examples (≈ 2 lines of code)\n",
        "    Y_prediction_test = predict(w, b, X_test)\n",
        "    Y_prediction_train = predict(w, b, X_train)\n",
        "\n",
        "    ### END CODE HERE ###\n",
        "\n",
        "    # Print train/test Errors\n",
        "    print(\"train accuracy: {} %\".format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100))\n",
        "    print(\"test accuracy: {} %\".format(100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100))\n",
        "\n",
        "    \n",
        "    d = {\"costs\": costs,\n",
        "         \"Y_prediction_test\": Y_prediction_test, \n",
        "         \"Y_prediction_train\" : Y_prediction_train, \n",
        "         \"w\" : w, \n",
        "         \"b\" : b,\n",
        "         \"learning_rate\" : learning_rate,\n",
        "         \"num_iterations\": num_iterations}\n",
        "    \n",
        "    return d"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3ihpg0hz6Myw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Run the following cell to train your model."
      ]
    },
    {
      "metadata": {
        "id": "H2vxPSxZs91q",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "d = model(x_train_flatten, y_train, x_test_flatten, y_test, num_iterations = 2500, learning_rate = 0.003, print_cost = True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VmpP-RQd_oeT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Expected Output:\n",
        "* **train accuracy: **100.0%\n",
        "* **test accuracy: ** 71.42857142857143 %"
      ]
    },
    {
      "metadata": {
        "id": "FvQoPl27CO1F",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As you can see, the training accuracy is 100%, which means that predictions for training images are correct every time. However, our test accuracy is closer to 70%, which means that predictions for images our model has not been trained on will be correct only about 70% of the time. This is called overfitting, which is when the model becomes so good at making predictions for the training dataset that it cannot generalize to data it has never seen before. There are better models that keep this from happening but that's for another time."
      ]
    },
    {
      "metadata": {
        "id": "Z3pwKIlaFY74",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Here is an image our model wrongly identified:"
      ]
    },
    {
      "metadata": {
        "id": "cQLOl6lCEyAx",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "index = 19\n",
        "plt.figure(figsize = (3, 3))\n",
        "plt.imshow(x_test[index].astype('uint8'))\n",
        "plt.title(\"Y = \" + str(y_test[0,index]) + \" - You predicted \" + str(int(d[\"Y_prediction_test\"][0, index])) + (\" (cat)\" if int(d[\"Y_prediction_test\"][0, index]) == 0 else \" (dog)\"), size='x-large')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jXxhWLYzIvyz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's see how our cost changed during training"
      ]
    },
    {
      "metadata": {
        "id": "1tXyIk3hFoYb",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "costs = np.squeeze(d['costs'])\n",
        "plt.plot(costs)\n",
        "plt.ylabel('cost')\n",
        "plt.xlabel('iterations (per hundreds)')\n",
        "plt.title(\"Learning rate =\" + str(d[\"learning_rate\"]))\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "o_eTgatUKyQi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As we would expect, the cost is decreasing and converging to 0. These means that our model is learning a representation of our data. If we wanted to, we could keep training our model to get a cost closer to 0. However, all this would do is increase the amount of overfitting in our model, which would cause the test accuracy to decrease. In machine learning we always have to keep in mind the trade-offs between training longer for better accuracy vs. overfitting our model."
      ]
    },
    {
      "metadata": {
        "id": "vjq9y5apMk-S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Things to try on your own\n",
        "\n",
        "1. Try changing the learning rate to see how a larger or smaller learning rate affects training\n",
        "2. Change the number of iterations for training\n",
        "3. Upload your own image of a cat or dog"
      ]
    },
    {
      "metadata": {
        "id": "jrrwmimrPrM3",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "file_path = \"\" #change this to where you file is saved\n",
        "\n",
        "img = Image.open(file_path).convert('RGB')\n",
        "img = img.resize((64,64))\n",
        "img_arr = np.array(img).astype(np.float32)\n",
        "img_arr_flatten = np.expand_dims(img_arr.resize(img_arr.shape[0], -1).T, axis=1)\n",
        "\n",
        "prediction = predict(d[\"w\"], d[\"b\"], img_arr_flatten)\n",
        "print(\"Prediction is \" + (\"dog\" if prediction == 1 else \"cat\"))\n",
        "plt.imshow(img)\n"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}