tutorials/image_classifier_tutorial_v1.2.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"gBYoMgZ0OWr0"},"source":["#### LLNL-MI-818588\n","# Computing Hackathon Tutorial: Deep Learning for Image Classification"]},{"cell_type":"markdown","metadata":{"id":"urjNPzxkOWr4"},"source":["In this tutorial, we will be covering the following topics:\n","\n","1. [Introduction](#intro) - 10 minutes\n","2. [Setup & Utilities](#setup) - 5 minutes\n","3. [Working with images as arrays and tensors](#images) - 15 minutes \n","4. [Machine learning background for image classification](#classifier_background) - 20 minutes\n","5. [Image classification in 5 steps with PyTorch](#classifier_script) - 20 minutes\n","6. [Packaging the classifier in an object-oriented wrapper](#classifier_class)- 20 minutes\n","7. [Wrap-up / Q & A](#conclusion) - 30 minutes\n"]},{"cell_type":"markdown","metadata":{"id":"WEgLJgrDOWr4"},"source":["<a id='intro'></a>\n","## Introduction"]},{"cell_type":"markdown","metadata":{"id":"_Dl7qjP7OWr5"},"source":["Image classification is a task in the Computer Vision domain that takes in an image as input and outputs a label for that image. Deep learning is the most effective modern method for modeling this task. In this notebook, we will overview how to perform multi-class image classification in Python using the PyTorch library. The intention is to give you a broad overview of this particular task of classification and inspire you to explore the vast fields of visual recognition and computer vision at large.\n","\n","If you are going through this notebook on your own and have questions, please do not hesitate to reach out to [Luke Jaffe(5) and Cindy Gonzales(72)](mailto:jaffe5@llnl.gov;gonzales72@llnl.gov?subject=DSI%20Deep%20Learning%20Tutorial)."]},{"cell_type":"markdown","metadata":{"id":"slI3yhdlOWr5"},"source":["### Breaking down the problem\n","\n","#### Evaluation: What should our model do when we deploy it?\n","\n","![Eval Diagram](eval_diagram.jpg)\n","\n","\n","#### Training: How do we improve the accuracy of our model?\n","\n","![Train Diagram](train_diagram.jpg)\n","\n","Gradient descent diagram from: https://blog.clairvoyantsoft.com/the-ascent-of-gradient-descent-23356390836f\n","\n","Neural network diagram from: http://neuralnetworksanddeeplearning.com/chap1.html \n","\n","We will briefly cover each of these topics, and provide links for learning about the material in greater detail."]},{"cell_type":"markdown","metadata":{"id":"UYF6hvytOWr6"},"source":["<a id='setup'></a>\n","## Setup & Utilities"]},{"cell_type":"markdown","metadata":{"id":"vo_TvA_OOWr6"},"source":["First thing's first, let's import all of the packages and setup some variables that we will need to run the code in this notebook. Read the comments of each cell if you have questions on any of the lines below."]},{"cell_type":"markdown","metadata":{"id":"J4OKpEOYOWr7"},"source":["### Setting up Jupyter Notebook"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"P5KfBOSCOWr7","executionInfo":{"status":"ok","timestamp":1655849361270,"user_tz":420,"elapsed":5,"user":{"displayName":"Brian Gallagher","userId":"02449979388063901434"}},"outputId":"ea1d121e-9c82-4c50-85c8-eda6506e69fa","colab":{"base_uri":"https://localhost:8080/"}},"outputs":[{"output_type":"stream","name":"stdout","text":["env: CUDA_VISIBLE_DEVICES=0\n"]}],"source":["# Automatically reload modules\n","%load_ext autoreload\n","%autoreload 2\n","\n","# Show matplotlib plots inline\n","%matplotlib inline\n","\n","# Select which GPU device to use (if any)\n","%env CUDA_VISIBLE_DEVICES=0"]},{"cell_type":"markdown","metadata":{"id":"vITcsqzXOWr8"},"source":["### Importing packages needed"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"otsZ8417OWr9"},"outputs":[],"source":["# System libraries\n","import os\n","import sys\n","\n","# Object-oriented utility library\n","import collections\n","\n","# Image processing library\n","import PIL\n","from PIL import Image, ImageOps\n","\n","# Numerical computing libraries\n","import numpy as np\n","import pandas as pd\n","\n","# Neural network libraries\n","import torch\n","import torchvision\n","\n","# Plotting library\n","import matplotlib.pyplot as plt\n","\n","# Progress bar library\n","import progressbar\n","\n","# Widget libraries\n","from IPython.display import display\n","import ipywidgets as widgets\n","from ipywidgets import interact"]},{"cell_type":"markdown","metadata":{"id":"ylOvP7qBOWr9"},"source":["### Setting up constants needed"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"0q_qLpsKOWr-"},"outputs":[],"source":["# Color constants\n","BLACK = (0, 0, 0)\n","WHITE = (255, 255, 255)\n","GRAY = (220,220,220)\n","GRAY2 = (231,230,230)\n","\n","# Directory where datasets will be downloaded\n","DEFAULT_DATASET_DIR = './data'"]},{"cell_type":"markdown","metadata":{"id":"2l-BI3BZOWr-"},"source":["### Setting up image display utility functions"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JkyJOuG5OWr-"},"outputs":[],"source":["def hstitch(images, spacing=5, border=5, spacing_color=BLACK, border_color=BLACK):\n","    widths, heights = zip(*(i.size for i in images))\n","\n","    total_width = sum(widths)+spacing*(len(widths)-1)\n","    max_height = max(heights)\n","\n","    new_im = Image.new('RGB', (total_width, max_height), color=spacing_color)\n","\n","    x_offset = 0\n","    for im in images:\n","        new_im.paste(im, (x_offset,0))\n","        x_offset += im.size[0] + spacing\n","\n","    if border > 0:\n","        new_im = ImageOps.expand(new_im, border=border, fill=border_color)\n","\n","    return new_im\n","\n","def vstitch(images, spacing=5, border=5, spacing_color=BLACK, border_color=BLACK):\n","    widths, heights = zip(*(i.size for i in images))\n","\n","    total_height = sum(heights)+spacing*(len(heights)-1)\n","    max_width = max(widths)\n","\n","    new_im = Image.new('RGB', (max_width, total_height), color=spacing_color)\n","\n","    y_offset = 0\n","    for im in images:\n","        new_im.paste(im, (0, y_offset))\n","        y_offset += im.size[1] + spacing\n","\n","    if border > 0:\n","        new_im = ImageOps.expand(new_im, border=border, fill=border_color)\n","\n","    return new_im\n","\n","def build_mosaic(img_list, rows, cols, clip=True, spacing=5, imborder=3, totborder=5, spacing_color=BLACK, \n","                 border_color=BLACK):\n","    if rows*cols < len(img_list) and clip:\n","        img_list = img_list[:rows*cols]\n","    elif rows*cols > len(img_list):\n","        raise Exception(\n","            '# images in list is less than width x height = {} x {} = {} < {}'.format(\n","            rows, cols, rows*cols, len(img_list)))\n","\n","    # Reshape \n","    img_mat = np.array(img_list, dtype=object).reshape(rows, cols).tolist()\n","\n","    # Border images\n","    if imborder > 0:\n","        new_img_mat = []\n","        for img_row in img_mat:\n","            new_row = []\n","            for img in img_row:\n","                img = ImageOps.expand(img, border=imborder)\n","                new_row.append(img)\n","            new_img_mat.append(new_row)\n","        img_mat = new_img_mat\n","\n","    # First create rows\n","    row_list = [hstitch(img_row, spacing=spacing, border=0, spacing_color=spacing_color, border_color=border_color) for img_row in img_mat]\n","\n","    m = vstitch(row_list, spacing=spacing, border=0, spacing_color=spacing_color, border_color=border_color)\n","\n","    if totborder > 0:\n","        m = ImageOps.expand(m, border=totborder)\n","\n","    return m"]},{"cell_type":"markdown","metadata":{"id":"aTKymxf2OWr_"},"source":["<a id='images'></a>\n","## Working with images as arrays and tensors"]},{"cell_type":"markdown","metadata":{"id":"T5NQa4QOOWsA"},"source":["### How does a computer \"see\" an image to be able to perform classification on it?\n","\n","An image is made up of pixels, which is a single number that represents the intensity of that particular area of an image. Let's consider a black-and-white image. Use the slider below to see what each pixel value represents as an image."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NqMhN0UROWsA"},"outputs":[],"source":["def display_img_tsr(x):\n","    img_tsr = torch.rand(128, 128)\n","    img_tsr.fill_(x)\n","    img_arr = img_tsr.numpy()\n","    plt.axis('off')\n","    plt.imshow(img_arr, cmap='gray', vmin=0, vmax=255)\n","    plt.show()\n","    \n","interact(display_img_tsr, x=widgets.IntSlider(min=0, max=255, step=1, continuous_update=True))"]},{"cell_type":"markdown","metadata":{"id":"AB94OGqtOWsA"},"source":["Now, let's move on to a color image. Color images are comprised of three bands-- a red, green, and blue band. A pixel value in each of the bands represents the intensity of that color in the overall image. So, a value of 0 for the red band would represent the absence of red in an image and a value of 255 would represent an intensely red image. The same is true for the blue and green bands.\n","\n","To give you some sense of what each pixel looks like at different red, green, and blue values, use the sliders below to adjust the intensity of each band independently. Some of the color combinations may suprise you!"]},{"cell_type":"code","execution_count":null,"metadata":{"scrolled":false,"id":"l5qc8n6kOWsA"},"outputs":[],"source":["def display_img_tsr(red, green, blue):\n","    fig, axs = plt.subplots(1, 4, figsize=(10, 10))\n","\n","    r = torch.zeros(128, 128, 3)\n","    g = torch.zeros(128, 128, 3)\n","    b = torch.zeros(128, 128, 3)\n","    img_tsr = torch.zeros(128, 128, 3)\n","    \n","    r[:, :, 0].fill_(red)\n","    g[:, :, 1].fill_(green)\n","    b[:, :, 2].fill_(blue)\n","    \n","    img_tsr[:, :, 0].fill_(red)\n","    img_tsr[:, :, 1].fill_(green)\n","    img_tsr[:, :, 2].fill_(blue)\n","\n","    \n","    axs[0].imshow(r/255)\n","    axs[0].axis('off')\n","    axs[0].title.set_text('Red Band')\n","    axs[1].imshow(g/255)\n","    axs[1].axis('off')\n","    axs[1].title.set_text('Green Band')\n","    axs[2].imshow(b/255)\n","    axs[2].axis('off')\n","    axs[2].title.set_text('Blue Band')\n","    axs[3].imshow(img_tsr/255)\n","    axs[3].axis('off')\n","    axs[3].title.set_text('RGB Image')\n","    plt.show()\n","    \n","interact(\n","    display_img_tsr, \n","    red=widgets.IntSlider(min=0, max=255, step=1, continuous_update=True), \n","    green=widgets.IntSlider(min=0, max=255, step=1, continuous_update=True),\n","    blue=widgets.IntSlider(min=0, max=255, step=1, continuous_update=True)\n",")"]},{"cell_type":"markdown","metadata":{"id":"Y2HLtbwvOWsB"},"source":["### Loading an image dataset for use"]},{"cell_type":"markdown","metadata":{"id":"VfGg5ARSOWsB"},"source":["To allow us to train a model in this tutorial, we have chosen to use the CIFAR dataset. Let's download the dataset as follows:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ecV-UFnxOWsB"},"outputs":[],"source":["# The torchvision library has several image datasets built-in, including CIFAR10\n","trainset = torchvision.datasets.CIFAR10(root=DEFAULT_DATASET_DIR, train=True, download=True)\n","testset = torchvision.datasets.CIFAR10(root=DEFAULT_DATASET_DIR, train=False, download=True)"]},{"cell_type":"markdown","metadata":{"id":"A_9w4M8-OWsB"},"source":["The CIFAR dataset has 10 classes with 6000 images per class. Each image is of size 32 x 32. Here is a visualization of the classes with samples of each class:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"sAGe-0eNOWsB"},"outputs":[],"source":["# These are the 10 types of objects in the CIFAR10 dataset\n","classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')\n","\n","# This loop will display 10 examples from each class in the CIFAR10 dataset\n","for i, c in enumerate(classes):\n","    print(c)\n","    imgs = []\n","    for item in trainset:\n","        if item[1] == i:\n","            imgs.append(item[0].resize((64,64))) # For visibility, double the size\n","            if len(imgs) == 10:\n","                break\n","    display(hstitch(imgs, spacing=2, border=2))"]},{"cell_type":"markdown","metadata":{"id":"6tpLYfK9OWsB"},"source":["Let's take a look at just one image from the CIFAR10 dataset..."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"hrmIYkRtOWsC"},"outputs":[],"source":["img = trainset[3000][0]\n","img.resize((64,64))"]},{"cell_type":"markdown","metadata":{"id":"hyI6tipQOWsC"},"source":["What do the red, green, and blue color channels look like for this image?"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"kmFojUhbOWsC"},"outputs":[],"source":["df = pd.DataFrame(1-np.array(img)[:, :, 0])\n","df.style.set_properties(**{'font-size':'6pt'}).background_gradient('Reds')"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"d0Q3rDDCOWsC"},"outputs":[],"source":["df = pd.DataFrame(1-np.array(img)[:, :, 1])\n","df.style.set_properties(**{'font-size':'6pt'}).background_gradient('Greens')"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"i8-cFBsdOWsC"},"outputs":[],"source":["df = pd.DataFrame(1-np.array(img)[:, :, 2])\n","df.style.set_properties(**{'font-size':'6pt'}).background_gradient('Blues')"]},{"cell_type":"markdown","metadata":{"id":"qE2GaGXoOWsC"},"source":["Understanding how an image can be interpreted as an array, we are now ready to set up a simple neural network to understand the basics of training a classifier."]},{"cell_type":"markdown","metadata":{"id":"WRU8vBLmOWsC"},"source":["<a id='classifier_background'></a>\n","## Machine Learning Background for Image Classification"]},{"cell_type":"markdown","metadata":{"id":"5R6zMiN-OWsC"},"source":["This is intended as a primer on some of the core concepts that will be used in our classifier code. If you haven't seen it before, it's ok if you don't immediately understand each part. The subject is very deep, and we have included some links to learn more later if you are interested.\n","\n","At a high level, the two topics we will discuss are neural networks and optimization."]},{"cell_type":"markdown","metadata":{"id":"A-nUSIpGOWsD"},"source":["### Neural Networks\n","\n","The model we will use to build our image classifier is a neural network. \n","\n","#### Neurons"]},{"cell_type":"markdown","metadata":{"id":"hk-L2UFSOWsD"},"source":["A neural network, also known as an artifical neural network, is a collection of artifical neurons connected in a graph, usually a directed acyclic graph (a DAG). These neurons can be thought of as a filter that is looking for relevant features, either of an image or of the previous layer in the network. Below is an example of a neural network.\n","\n","![image.png](http://neuralnetworksanddeeplearning.com/images/tikz1.png)\n","<center> Source: http://neuralnetworksanddeeplearning.com/chap1.html </center>\n","\n","Generally, neural networks take in an input and run that input through a series of groups of neurons. Each group of neurons is known as a \"layer\", where the first layer is known as the input layer (takes the input) and the last layer is known as the output layer (outputs something that can be used for classification, etc.) All layers in between the input and output layers are known as hidden layers even though they are not actually hidden. They are simply a group of neurons which are connected to neurons in the previous layer and neurons in the next layer.\n","\n","Each neuron takes in some inputs, does some calculation on those inputs, caches outputs for later parameter updates, and then pushes outputs on to the next layer. We'll go more into how they update their parameters later."]},{"cell_type":"markdown","metadata":{"id":"eyNjGMi2OWsD"},"source":["#### Convolutional Neural Networks"]},{"cell_type":"markdown","metadata":{"id":"eHvKjnbFOWsD"},"source":["Now, the picture that we have painted so far for a basic neural network just won't scale well for images. Images have a lot of pixels. In the examples that we have talked about so far (the CIFAR dataset), the input would be 32x32x3 = 3072 parameters per neuron in the first layer of the network. There are typically hundreds of neurons in one layer. In addition, most images are not as small as 32x32. Many times, the images are larger than 200x200x3 = 120,000 parameters per neuron in the first layer and hundreds of neurons in each layer which can lead to millions of parameters, if not more. This can lead to an unmanageable neural network with a huge number of parameters.\n","\n","Enter the Convolutional Neural Network (let's call these ConvNets for short). ConvNets have neurons that take a (often) 3D input-- so a neuron that takes a length, width, and depth measurement which translates well to images. As we discussed before, RGB images are three stacked arrays (one for each the red, green, and blue bands) and an image can then can be translated to a 3D array with height, width, and some number of channels. \n","\n","To display the difference, take a look at the below:\n","\n","#### Regular Neural Network\n","![image.png](https://cs231n.github.io/assets/nn1/neural_net2.jpeg)\n","<center> Source: https://cs231n.github.io/convolutional-networks/#layers </center>\n","\n","\n","#### Convolutional Neural Network\n","![image.png](https://cs231n.github.io/assets/cnn/cnn.jpeg)\n","<center> Source: https://cs231n.github.io/convolutional-networks/#layers </center>"]},{"cell_type":"markdown","metadata":{"id":"8YRRO4MrOWsD"},"source":["The uniqueness of this network and why it is so good for processing images lies in its basic pieces/parts. A ConvNet is composed of three distinct layers types: \n","1. Convolutional layer\n","2. A pooling layer\n","3. Non-linear activation\n","\n","We're going to go through the basics of these quickly and we will include resources if you want to do some studying on your own.\n","\n","#### Convolutional Layer\n","The convolutional layer takes a chunk of an image and multiplies it by a fixed matrix to detect features. Each neuron can represent a feature so you would do this repeatedly for each feature. Take a look at this [interactive tool to learn about convolutions](https://setosa.io/ev/image-kernels/).\n","\n","#### Example\n","![convolution.png](https://raw.githubusercontent.com/hiromis/notes/master/lesson6/32.png)\n","\n","\n","#### Pooling Layer\n","Next, the pooling layer takes a larger image and reduces it down in a way that still preserves the most important information. The most common type of pooling is max pooling, which takes patches from the convolved image and picks the largest value from the patch and uses that to represent the image.\n","\n","![pooling.png](https://miro.medium.com/max/512/1*ReZNSf_Yr7Q1nqegGirsMQ@2x.png)\n","<center> Source: https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2 </center>\n","\n","#### Non-Linear Layer\n","Finally, a non-linear activation allows the ConvNets to learn a non-linear function, which is very important. An [important theoretical result](https://en.wikipedia.org/wiki/Universal_approximation_theorem) is that a neural network with a single hidden layer (of arbitrary width) can represent any well-behaved function. This helps us understand why a non-linearity is necessary, but not sufficient for learning a model for a problem such as classification.\n","\n","The most common non-linear layer is the rectified linear unit layer (ReLU) which makes all negative values 0:\n","\n","$$ \\text{ReLU}(x) = \\max(0, x) $$\n","\n","\n","##### Example\n","![ReLU.png](https://i.pinimg.com/originals/8a/13/09/8a130915c66d23d37b0a5dc9fd8baf38.jpg)"]},{"cell_type":"markdown","metadata":{"id":"MP4CVaUNOWsD"},"source":["#### Why do convolutional layers work well?\n","\n","Convolutional layers are effective for image learning for a few reasons:\n","\n","#### Spatial invariance\n","ConvNets are designed to be spatially invariant. In other words, they are not sensitive to the position (translation) of objects in images. The ConvNet turns pixels into features via convolutions and pooling. The features in an image will be detected no matter their original position in the image.\n","\n","#### Hierarchy of features\n","ConvNets process features in a hierarchical way, much like the mammalian brain. Here is a really great visual to help us explain this:\n","\n","![hierarchy.png](https://miro.medium.com/max/2400/1*Ji5QhY9QXBlpNNLH4qAcNA.png)\n","<center> Source: https://towardsdatascience.com/identifying-traffic-signs-with-deep-learning-5151eece09cb </center>\n","\n","In the earlier layers, the neurons learn simple features such as edges. As we move through the network, the features become more defined, with mid-layers learning patterns and later layers learning more complex semantic constructs (such as physical objects you might recognize)."]},{"cell_type":"markdown","metadata":{"id":"RLaLyzTmOWsE"},"source":["If you are interested in learning more about how a ConvNet works, check out any of these resources: \n","1. [Stanford's CS231n: Convolutional Neural Networks for Visual Recognition class notes](https://cs231n.github.io/convolutional-networks/#layers)\n","2. [CNNs from different viewpoints](https://medium.com/impactai/cnns-from-different-viewpoints-fab7f52d159c)\n","3. [How do Convolutional Neural Networks work?](https://e2eml.school/how_convolutional_neural_networks_work.html)\n","4. The course content from Andrew Ng's deep learning specialization from Coursera is available on YouTube. Review C4W1L01 - C4W1L11. [Here's the playlist related to Convolutional Neural Networks](https://www.youtube.com/playlist?list=PLkDaE6sCZn6Gl29AoE31iwdVwSG-KnDzF)\n","5. [Interactive tool to learn about convolutions](https://setosa.io/ev/image-kernels/)\n","6. [Paper on how mammilian brains process features](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1363130/)"]},{"cell_type":"markdown","metadata":{"id":"6iEiWlFWOWsE"},"source":["### Optimization\n","\n","How do we tune the parameters of our model to classify images more accurately?\n","\n","#### Selecting a surrogate for classification: Logistic Regression\n","In a classification network, the number of outputs is equal to the number of classes in our dataset. For the CIFAR10 dataset, this is 10 classes. When the 10 raw outputs come out of the network, the output corresponding to the correct class should be higher, and the outputs corresponding to the other classes should be lower. The next step is usually to convert our outputs to a probability distribution. To do so, we can use the model of logistic regression. The logistic sigmoid function squashes $x \\in \\mathbb{R}$ to $f(x) \\in (0, 1)$."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"1HRT_zW3OWsE"},"outputs":[],"source":["# Plot the logistic sigmoid function\n","x = np.linspace(-10, 10, 100)\n","f = lambda x: 1 / (1 + np.exp(-x))\n","plt.plot(x, f(x))\n","plt.xlabel('x')\n","plt.ylabel('f(x) = logistic sigmoid')\n","plt.show()"]},{"cell_type":"markdown","metadata":{"id":"XHljA_HVOWsE"},"source":["The multiclass extension of the logistic function is called the softmax function.\n","Let the output of our network be $x \\in \\mathbb{R}^c$. Then the softmax function is defined as, \n","\n","$$ f(x)_i = \\frac{e^{x_i}}{\\sum_{j=1}^{c} e^{x_j}} $$\n","\n","Intuitively, this will squash the output of our network into a probability distribution: each class is assigned a probability between 0 and 1, and the c (=10) probabilties sum to 1."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"lCjCG1CHOWsE"},"outputs":[],"source":["# Set printing options for numpy\n","np.set_printoptions(precision=3, suppress=True)\n","\n","# Create a fake raw output of our network\n","network_output_raw = np.random.randn(10)\n","print('Raw network outputs:', network_output_raw)\n","\n","# Define the softmax function\n","def softmax_function(x):\n","    numerator = np.exp(x)\n","    denominator = numerator.sum()\n","    return numerator / denominator\n","\n","# Run the softmax function on our network output to get probabilities\n","network_output_prob = softmax_function(network_output_raw)\n","print('Network outputs converted to probabilities:', network_output_prob)\n","print('Sum of probabilities: {:.1f}'.format(network_output_prob.sum()))"]},{"cell_type":"markdown","metadata":{"id":"QbsExcQhOWsE"},"source":["Now, we want to reward the network for outputting a high probability for the correct class, and punish it for outputting a low probability for the correct class. A popular choice of \"objective\" function to accomplish this goal is the cross-entropy loss, defined as:\n","\n","$$ L_{CE}(x, \\text{true_class}) = -\\log (\\text{softmax}(x)[\\text{true_class}]) $$\n","\n","This takes the probability of the true class, as computed by the softmax function, and converts it to a quantity that we need to reduce in order to increase the probability.\n","\n","$$ \\text{prob}_{\\text{softmax}} \\in (0, 1) $$\n","$$ \\log(\\text{prob}_{\\text{softmax}}) \\in (-\\infty, 0) $$\n","$$ -\\log(\\text{prob}_{\\text{softmax}}) \\in (0, \\infty) $$\n","\n","In order to get the probability of the true class to be close to 1 for a given image, we need to get the cross-entropy loss for that class close to 0. But how do we update our model to make this happen? We can use the gradient descent algorithm.\n","\n","Note: Logistic regression is a very deep subject in machine learning theory, and there are many theoretical reasons why we use the logistic function and cross-entropy loss, aside from the fact that they produce intuitive outputs. Here is one resource on Logistic Regression and its connection to a Naive Bayes classifier: https://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf"]},{"cell_type":"markdown","metadata":{"id":"k6I7YVcaOWsE"},"source":["#### Gradient Descent "]},{"cell_type":"markdown","metadata":{"id":"QtXBKouTOWsE"},"source":["Our goal in training our network would be to minimize this loss function. Specifically, we want to find values for our weights and biases in the model that can minimize the loss function. To do this, we can use the gradient descent algorithm. \n","\n","The gradient descent update rule for a univariate function $f(x)$ is as follows:\n","$$\n","x^{t+1} = x^t - \\alpha f'(x^t)\n","$$\n","\n","To help us understand this update, let's look at a simple example function, before extending to loss minimization:\n","$$\n","f(x) = x^2\n","$$\n","$$\n","\\Rightarrow f'(x) = 2x\n","$$\n","\n","So the specific update for this $f(x)$ would be:\n","$$\n","x^{t+1} = x^t - 2 \\alpha x^t\n","$$\n","And we start the optimization procedure at some $x^0$, which can be selected at random for this example.\n","\n","Intuitively: at each timestep $t$, we evaluate the derivative of the function at the current $x$. This gives us the slope of the function. If we know the slope, we know which way is down on the function. We can then take a step of some chosen size in that new direction. Graphically:\n","\n","\n","![gradient_descent](https://miro.medium.com/max/875/1*G1v2WBigWmNzoMuKOYQV_g.png)\n","<center> Source: https://blog.clairvoyantsoft.com/the-ascent-of-gradient-descent-23356390836f </center>\n","\n","\n","To control how quickly we move down the function's surface, we will need to define a learning rate: the value multiplied by the gradient of the function to determine the magnitude of the step we will take. The learning rate is a so-called hyperparameter that you can fine-tune to get the most efficient training.  If the learning rate is too large, gradient descent could diverge (the loss value gets increasingly larger until it approaches infinity). If the learning rate is too small, the algorithm could fail to make progress or settle in a local minimum and never reach a global minimum of the function."]},{"cell_type":"markdown","metadata":{"id":"pj5KObAUOWsF"},"source":["#### Gradient descent with a model\n","\n","In practice, we need to run gradient descent on a model with parameters, not just a univariate function. In this case, we might have some function $L(w; x)$, where $x$ is the data, and $w$ is the parameters of the model. In this case, the gradient descent update could be written as\n","\n","$$\n","w^{t+1} = w^t - \\alpha \\nabla L(w^t; x)\n","$$\n","\n","We will not review the exact form of this update for cross-entropy loss, but here is one resource with more information: https://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf\n","\n","#### Stochastic Minibatch Gradient Descent\n","\n","How do we apply the gradient descent algorithm to real data? One option is compute gradients of the model parameters for the entire dataset at the same time. But in practice, due to optimization properties and memory constraints, we randomly sample \"minibatches\" from the dataset, and take a step after computing gradients for each minibatch. This is called stochastic minibatch gradient descent, abbreviated as SGD. Pseudo-code for training with SGD could look like the following:\n","```\n","# Iterate through all images in the dataset\n","for _ in num_batches(dataset):\n","    # Randomly sample a minibatch of images\n","    batch_images, batch_labels = random_sample(dataset, num_images_per_batch)\n","    \n","    # Run the model on the images to get our raw outputs\n","    model_outputs = run_model(model, batch_images)\n","    \n","    # Compute the loss for these raw outputs given the true labels\n","    batch_loss = compute_loss(model_outputs, batch_labels)\n","    \n","    # Compute gradients of the model parameters with respect to the loss\n","    batch_grads = compute_gradients(model, batch_loss)\n","    \n","    # Update the model parameters using the computed gradients and the learning rate\n","    gradient_descent(model, batch_grads, learning_rate)\n","```\n","We will see later on how to implement this in PyTorch."]},{"cell_type":"markdown","metadata":{"id":"bkE-QPMQOWsF"},"source":["#### The Backpropagation Algorithm\n","\n","The backpropagation algorithm is used to efficiently compute gradients for all parameters in the network using dynamic programming and the calculus chain rule.\n","\n","![Backprop Figure](https://www.guru99.com/images/1/030819_0937_BackPropaga1.png)\n","\n","Figure from: https://www.guru99.com/backpropogation-neural-network.html\n","\n","Resource on the backpropagation algorithm: https://page.mi.fu-berlin.de/rojas/neural/chapter/K7.pdf\n","\n","Thankfully, PyTorch (and other neural networks libraries) handle this for us, using some variation of an \"autograd\"\n","package. This handles all the calculus and dynamic programming behind the scenes, so all you have to do is specify the functions you want to compute.\n","\n","Automatic differentiation survey: https://arxiv.org/abs/1502.05767\n","\n","PyTorch autograd tutorial: https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html\n","\n","#### Gradient Descent for Large Convolutional Neural Networks\n","\n","In practice, convolutional neural networks have millions or even billions of parameters. Surprisingly, stochastic gradient descent still works on these large models, since the loss surface is more regular than you would expect. This is a property of the model architecture, the objective function, and the data itself.\n","\n","![Loss Landscape](https://myconfluence.llnl.gov/download/attachments/293571119/loss_landscape_figure.png)\n","\n","Figure from, \"Visualizing the Loss Landscape of Neural Nets\", Li, Xu, et al.: https://papers.nips.cc/paper/2018/file/a41b3bb3e6b050b6c9067c67f663b915-Paper.pdf\n","\n","A paper studying why SGD is effective: https://arxiv.org/abs/1509.01240"]},{"cell_type":"markdown","metadata":{"id":"eHvbGNfVOWsF"},"source":["#### Training, validation, and testing: does our model generalize?\n","\n","To ensure that our model will work on new data (and not just the training set), we need to carve out part of our data for \"validation\" and \"testing\". We use the validation set to tune hyperparameters (such as learning rate) during training, and we run the model on the test set once when we are done training, to report final performance statistics.\n","\n","![Training vs. Testing](https://lh6.googleusercontent.com/uuKkYCYun1Ky6C7_GwEtv0gNdaoHyx0WTXiM8jvGOQqGx75gIRVhx1to7OapyGDbOsmKyAl9Eyi5RC-atbk6AXukkA7UBXA-NutKcAdHaTGDsWSzNGyBaCBMvVu1HLuU1Wm6TxWb)\n","\n","Figure from: http://www.dailysmarty.com/posts/setting-up-test-validation-and-training-sets-of-data"]},{"cell_type":"markdown","metadata":{"id":"WXHNuWhoOWsF"},"source":["Now that we've touched on the basics, let's see how these concepts are implemented in PyTorch for training and testing an image classifier:"]},{"cell_type":"markdown","metadata":{"id":"Gqq9quUpOWsF"},"source":["<a id='classifier_script'></a>\n","## Image classification in 5 steps with PyTorch\n","#### STEP 0: SETUP\n","Not needed in this notebook, but useful if you want to move this to a standalone script."]},{"cell_type":"markdown","metadata":{"id":"eZaAdl36OWsF"},"source":["```\n","# Global imports\n","import os\n","import torch\n","import torchvision\n","import argparse\n","import progressbar\n","\n","# Get arguments from the user\n","parser = argparse.ArgumentParser()\n","parser.add_argument('--dataset_dir', type=str, default=os.path.join('..', '..', '.datasets'))\n","parser.add_argument('--dataset_name', type=str, default='fashion_mnist')\n","parser.add_argument('--batch_size', type=int, default=128)\n","parser.add_argument('--learning_rate', type=float, default=1e-3)\n","parser.add_argument('--num_epochs', type=int, default=10)\n","args = parser.parse_args()\n","```"]},{"cell_type":"markdown","metadata":{"id":"eLQGpyY0OWsG"},"source":["#### Alternate setup just for this notebook"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"U-dfd711OWsG"},"outputs":[],"source":["# Object for handling args\n","Args = collections.namedtuple('Args', [\n","    'dataset_dir',\n","    'dataset_name',\n","    'batch_size',\n","    'learning_rate',\n","    'num_epochs',\n","])\n","\n","# Instance of arg-handling object:\n","# Try modifying batch_size, learning_rate, and num_epochs\n","args = Args(\n","    dataset_dir=DEFAULT_DATASET_DIR,\n","    dataset_name='cifar10',\n","    batch_size=128,\n","    learning_rate=1e-2,\n","    num_epochs=10,\n",")"]},{"cell_type":"markdown","metadata":{"id":"26jh9g2aOWsG"},"source":["#### STEP 1: LOAD DATA"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"DluampPjOWsG"},"outputs":[],"source":["# Create a transform object to pre-process the image data\n","transform = torchvision.transforms.Compose([\n","    # ToTensor converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] \n","    # to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0].\n","    torchvision.transforms.ToTensor()\n","    # Normally, we would also normalize the data to have mean=0, SD=1\n","])\n","\n","# Create the dir to store the dataset in before downloading it\n","dataset_path = os.path.join(args.dataset_dir, args.dataset_name)\n","if not os.path.exists(dataset_path):\n","    os.makedirs(dataset_path)\n","\n","# Instantiate dataset objects for the CIFAR10 dataset (train and test)\n","trainset = torchvision.datasets.CIFAR10(dataset_path, train=True, transform=transform, download=True)\n","testset = torchvision.datasets.CIFAR10(dataset_path, train=False, transform=transform, download=False)\n","\n","# Instantiate dataloader objects to shuffle, batch, and iterate through the dataset\n","# *It is important to shuffle the training set, so the optimizer sees more variation\n","trainloader = torch.utils.data.DataLoader(trainset, batch_size=args.batch_size, shuffle=True)\n","testloader = torch.utils.data.DataLoader(testset, batch_size=args.batch_size, shuffle=False)"]},{"cell_type":"markdown","metadata":{"id":"iQ7LKn-sOWsG"},"source":["#### STEP 2: CREATE MODEL"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ouIEu6-SOWsG"},"outputs":[],"source":["# Create simple model with 2 fully-connected / linear layers\n","model = torch.nn.Sequential(\n","    # First layer takes input that is size of the unraveled image\n","    # - The layer has 100 neurons, and each neuron has 32x32x3 weights + 1 bias\n","    # - Output is 100 \"features\"\n","    torch.nn.Linear(32*32*3, 100),\n","    # We use a ReLU activation as our non-linearity\n","    torch.nn.ReLU(inplace=True),\n","    # Final linear layer takes 100 features and outputs 10 \"scores\" (1 for each class)\n","    # - We will need to convert these scores to probabilities separately by applying the softmax function\n","    torch.nn.Linear(100, 10)\n",")"]},{"cell_type":"markdown","metadata":{"id":"rVck1I-7OWsH"},"source":["#### STEP 3: CREATE OBJECTIVE AND OPTIMIZER"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"57OoB4obOWsH"},"outputs":[],"source":["# Use the cross entropy loss function/objective/criterion\n","objective = torch.nn.CrossEntropyLoss()\n","\n","# Use the SGD optimizer with user input learning rate\n","optimizer = torch.optim.SGD(model.parameters(), lr=args.learning_rate)"]},{"cell_type":"markdown","metadata":{"id":"jYWmxGyZOWsH"},"source":["#### STEP 4: TRAIN AND TEST MODEL"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"2Foyv_bVOWsH"},"outputs":[],"source":["# Function to train for one epoch\n","def run_epoch(dataloader, train_flag=True):\n","    # Put model in train or eval mode (important for BatchNorm and Dropout)\n","    model.train() if train_flag else model.eval()\n","    # Statistic accumulators\n","    total_loss, correct, total = 0, 0, 0\n","    # Use progressbar inside this block\n","    with progressbar.ProgressBar(max_value=len(dataloader)) as bar:\n","        # Iterate throught the dataset in batches\n","        for batch_idx, (batch_data, batch_labels) in enumerate(dataloader):\n","            # Zero out stored gradients during training\n","            if train_flag:\n","                optimizer.zero_grad()\n","            # Reshape data for model\n","            # - converts the data from size torch.Size([128, 3, 32, 32])\n","            # - to size torch.Size([128, 3072])\n","            batch_data = batch_data.view(batch_data.size(0), -1)\n","            # Put the image batch through the model\n","            # - output has size torch.Size([128, 10])\n","            batch_output = model(batch_data)\n","            # Compute loss function with model output and labels\n","            # - loss has size torch.Size([]) (it is a scalar)\n","            loss = objective(batch_output, batch_labels)\n","            # Update model during training\n","            if train_flag:\n","                # Backpropagate gradients (w.r.t. loss) through the computation graph during training\n","                loss.backward()\n","                # Update weights using stored gradients during training\n","                optimizer.step()\n","            # Compute loss\n","            total_loss += loss.item()\n","            # Compute accuracy\n","            _, predicted = batch_output.max(1)\n","            total += batch_labels.size(0)\n","            correct += predicted.eq(batch_labels).sum().item()\n","            # Update progress bar\n","            bar.update(batch_idx)\n","    # Compute statistics over the epoch\n","    epoch_loss = total_loss / (batch_idx + 1)\n","    epoch_acc = correct / total\n","    print('{} results: loss={:.3f}, acc={:.3f}'.format('Train' if train_flag else 'Test', epoch_loss, epoch_acc))"]},{"cell_type":"markdown","metadata":{"id":"JkKLT5LOOWsH"},"source":["#### STEP 5: THE TRAINING LOOP"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"tWU4eRvAOWsH"},"outputs":[],"source":["# Test for one epoch before training to see accuracy of the randomly initialized model\n","with torch.no_grad():\n","    run_epoch(testloader, train_flag=False)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"R_zT0Jm8OWsH"},"outputs":[],"source":["# Train and test for num_epochs\n","for epoch in range(args.num_epochs):\n","    print('Epoch {}'.format(epoch))\n","    # Train for one epoch\n","    run_epoch(trainloader, train_flag=True)\n","    # Test for one epoch: do not compute gradients when testing\n","    with torch.no_grad():\n","        run_epoch(testloader, train_flag=False)"]},{"cell_type":"markdown","metadata":{"id":"HgjgVkeCOWsI"},"source":["<a id='classifier_class'></a>\n","## Packaging the classifier in an object-oriented wrapper"]},{"cell_type":"markdown","metadata":{"id":"E-AWusPoOWsI"},"source":["### Classifier Model: Convolutional Neural Network"]},{"cell_type":"markdown","metadata":{"id":"c-xkfsmzOWsI"},"source":["First, we need to define a basic model. Read the comments below to walk you through the model."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"cBG4oGdnOWsI"},"outputs":[],"source":["# Declare CIFAR10Model class\n","class BasicModel(torch.nn.Module):\n","    \"\"\"\n","    Simple model which takes 32x32 inputs and produces\n","    score values for each of 10 classes for each element.\n","    Model adapted from:\n","    https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html\n","    \"\"\"\n","    def __init__(self):\n","        super().__init__()\n","        # Features\n","        self.layer_list = torch.nn.ModuleList([\n","            # 2d Convolution, 1 channel in, 6 channels out, kernel size 6\n","            # - Conv layer will have 6 filters of size 6x6x3 weights + 1 bias\n","            torch.nn.Conv2d(3, 6, 6),\n","            # Rectified Linear Unit activation, inplace=True saves a small amount of memory\n","            torch.nn.ReLU(inplace=True),\n","            # 2d Max Pooling, kernel size 2, stride 2\n","            # - Downsamples both width and height of our feature map by factor of 2\n","            torch.nn.MaxPool2d(2, 2),\n","            # 2d Convolution, 6 channels in, 16 channels out, kernel size 5\n","            # - Conv layer will have 16 filters of size 5x5x6 weights + 1 bias\n","            torch.nn.Conv2d(6, 16, 5),\n","            torch.nn.ReLU(inplace=True),\n","            torch.nn.MaxPool2d(2, 2),\n","        ])\n","        # Classifier\n","        self.classifier = torch.nn.Sequential(\n","            # Fully-Connected, 256 elements in, 120 elements out\n","            # - The layer has 120 neurons, and each neuron has 256 weights + 1 bias\n","            torch.nn.Linear(256, 120),\n","            torch.nn.ReLU(inplace=True),\n","            torch.nn.Linear(120, 84),\n","            torch.nn.ReLU(inplace=True),\n","            torch.nn.Linear(84, 10),\n","        )\n","\n","    def forward(self, x):\n","        \"\"\"\n","        Pass one batch of image data through the model.\n","        Return batch of score values corresponding to each class.\n","        Input size: (batch_size, 1, 32, 32)\n","        Output size: (batch_size, 10)\n","        \"\"\"\n","        # Shape of data is: (batch_size, 1, 32, 32)\n","        for layer in self.layer_list:\n","            x = layer(x)\n","        # Shape of data is: (batch_size, 16, 4, 4)\n","        x = x.view(x.size(0), -1)\n","        # Shape of data is: (batch_size, 256)\n","        x = self.classifier(x)\n","        # Shape of data is: (batch_size, 10)\n","        return x"]},{"cell_type":"markdown","metadata":{"id":"nDtT_fbJOWsI"},"source":["Next, we use that model to create a trainable classifier."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"EkbnJ8FSOWsI"},"outputs":[],"source":["class Classifier:\n","    def __init__(self, dataset_dir=DEFAULT_DATASET_DIR, dataset_name='cifar10', batch_size=128, \n","                 learning_rate=1e-3, device='cpu'):\n","        # Store parameters\n","        self.device = device\n","\n","        # Create a transform object to pre-process the image data\n","        train_transform = torchvision.transforms.Compose([\n","            torchvision.transforms.RandomCrop(32, padding=4),\n","            torchvision.transforms.RandomHorizontalFlip(),\n","            torchvision.transforms.ToTensor(),\n","            torchvision.transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),\n","        ])\n","\n","        test_transform = torchvision.transforms.Compose([\n","            torchvision.transforms.ToTensor(),\n","            torchvision.transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),\n","        ])\n","\n","        # Create the dir to store the dataset in before downloading it\n","        dataset_path = os.path.join(dataset_dir, dataset_name)\n","        if not os.path.exists(dataset_path):\n","            os.makedirs(dataset_path)\n","\n","        # Instantiate dataset objects for the CIFAR10 dataset (train and test)\n","        # - Downloads the dataset if it has not already been downloaded\n","        self.trainset = torchvision.datasets.CIFAR10(dataset_path, train=True, transform=train_transform, \n","                                                     download=True)\n","        self.testset = torchvision.datasets.CIFAR10(dataset_path, train=False, transform=test_transform, \n","                                                    download=False)\n","\n","        # Store the class names\n","        self.class_name_list = ('plane', 'car', 'bird', 'cat', 'deer',\n","                'dog', 'frog', 'horse', 'ship', 'truck')\n","\n","        # Instantiate dataloader objects to shuffle, batch, and iterate through the dataset\n","        self.trainloader = torch.utils.data.DataLoader(self.trainset, batch_size=batch_size, shuffle=True)\n","        self.testloader = torch.utils.data.DataLoader(self.testset, batch_size=batch_size, shuffle=False)\n","\n","        # Create model\n","        self.model = BasicModel()\n","\n","        # Put model on GPU if available\n","        self.model = self.model.to(self.device)\n","        if self.device == 'cuda':\n","            # Make model data parallel and put it in multiple GPUs if available\n","            self.model = torch.nn.DataParallel(self.model)\n","            # Use cuDNN\n","            torch.backends.cudnn.benchmark = True\n","\n","        # Use the cross entropy loss function/objective/criterion\n","        self.criterion = torch.nn.CrossEntropyLoss()\n","\n","        # Use the Adam optimizer with user input learning rate\n","        self.optimizer = torch.optim.Adam(self.model.parameters(), lr=learning_rate)\n","\n","        # Widgets for progressbar during training and testing\n","        # - reports loss and accuracy at every update\n","        self.widgets = [\n","            ' [', progressbar.Percentage(), '] ',\n","            progressbar.Bar(),\n","            ' [', progressbar.DynamicMessage('loss'), '] ',\n","            ' [', progressbar.DynamicMessage('acc'), '] ',\n","            ' [', progressbar.Timer(), '] ',\n","            ' [', progressbar.ETA(), '] ',\n","        ]\n","\n","    # Get some random sample images from the trainset\n","    def sample(self, n, seed=0):\n","        # Random seed\n","        np.random.seed(seed)\n","        # Create a random index to shuffle the train set\n","        rand_idx = np.arange(len(self.trainset.data))\n","        np.random.shuffle(rand_idx)\n","        # Get a list of the raw tensors stored in the trainset\n","        arr_list =  self.trainset.data[rand_idx][:n]\n","        # Convert the raw tensors to PIL Image objects\n","        img_list = [Image.fromarray(arr) for arr in arr_list]\n","        # Return the list of PIL Image objects\n","        return img_list\n","\n","    # Get some random sample images from the testset and run them through the model\n","    def sample_eval(self, n, seed=0):\n","        # Random seed\n","        np.random.seed(seed)\n","        # Create a random index to shuffle the train set\n","        rand_idx = np.arange(len(self.testset.data))\n","        np.random.shuffle(rand_idx)\n","        # Get a list of the raw tensors stored in the trainset\n","        arr_list =  self.testset.data[rand_idx][:n]\n","        label_list = np.array(self.testset.targets)[rand_idx][:n]\n","        # Convert the raw tensors to PIL Image objects\n","        img_list = [Image.fromarray(arr) for arr in arr_list]\n","        # Get a list of the raw tensors stored in the trainset\n","        batch_data = torch.stack([self.trainset.transform(img) for img in img_list], dim=0)\n","        # Put model in eval mode\n","        self.model.eval()\n","        # Run batch through model\n","        with torch.no_grad():\n","            batch_output = self.model(batch_data)\n","            # Get softmax scores for each element\n","            sm_list = torch.nn.functional.softmax(batch_output, dim=-1).cpu().numpy()\n","        # Return the list of PIL Image objects\n","        return img_list, label_list, sm_list\n","\n","    # Function to train for one epoch\n","    def train(self):\n","        \"\"\"\n","        Train for one epoch of the dataloader using the model, criterion, and optimizer.\n","        \"\"\"\n","        # Put model in train mode (important for BatchNorm and Dropout)\n","        self.model.train()\n","        # Statistic accumulators\n","        total_loss = 0.0\n","        correct = 0\n","        total = 0\n","        avg_loss, acc = None, None\n","        # Set up widgets for progress bar\n","        _mode = ['Train:']\n","        _widgets = _mode + self.widgets\n","        # Use progressbar inside this block\n","        with progressbar.ProgressBar(max_value=len(self.trainloader), widgets=_widgets) as bar:\n","            # Iterate throught the dataset in batches\n","            for batch_idx, (batch_data, batch_labels) in enumerate(self.trainloader):\n","                # Put the images and labels on GPU if available\n","                batch_data, batch_labels = batch_data.to(self.device), batch_labels.to(self.device)\n","                # Zero out stored gradients\n","                self.optimizer.zero_grad()\n","                # Put the image batch through the model\n","                batch_output = self.model(batch_data)\n","                # Compute loss function with model output and labels\n","                loss = self.criterion(batch_output, batch_labels)\n","                # Backpropagate gradients (w.r.t. loss) through the computation graph\n","                loss.backward()\n","                # Update weights using stored gradients\n","                self.optimizer.step()\n","\n","                # Compute loss\n","                total_loss += loss.item()\n","                avg_loss = total_loss / (batch_idx + 1)\n","\n","                # Compute accuracy\n","                _, predicted = batch_output.max(1)\n","                total += batch_labels.size(0)\n","                correct += predicted.eq(batch_labels).sum().item()\n","                acc = correct / total\n","\n","                # Update progress bar\n","                bar.update(batch_idx, loss=avg_loss, acc=acc)\n","\n","        return avg_loss, acc\n","\n","    # Function to test for one epoch\n","    def test(self):\n","        \"\"\"\n","        Test for one epoch of the dataloader using the model.\n","        \"\"\"\n","        # Put model in eval mode (important for BatchNorm and Dropout)\n","        self.model.eval()\n","        # Statistic accumulators\n","        total_loss = 0.0\n","        correct = 0\n","        total = 0\n","        avg_loss, acc = None, None\n","        # Set up widgets for progress bar\n","        _mode = ['Test:']\n","        _widgets = _mode + self.widgets\n","        # DO NOT compute gradients for any computation inside this block\n","        # - temporarily sets all requires_grad = False\n","        with torch.no_grad():\n","            # Use progressbar inside this block\n","            with progressbar.ProgressBar(max_value=len(self.testloader), widgets=_widgets) as bar:\n","                # Iterate throught the dataset in batches\n","                for batch_idx, (batch_data, batch_labels) in enumerate(self.testloader):\n","                    # Put the images and labels on GPU if available\n","                    batch_data, batch_labels = batch_data.to(self.device), batch_labels.to(self.device)\n","                    # Put the image batch through the model\n","                    batch_output = self.model(batch_data)\n","                    # Compute loss function with model output and labels\n","                    loss = self.criterion(batch_output, batch_labels)\n","\n","                    # Compute total loss and average loss per batch\n","                    total_loss += loss.item()\n","                    avg_loss = total_loss / (batch_idx + 1)\n","\n","                    # Compute accuracy\n","                    _, predicted = batch_output.max(1)\n","                    total += batch_labels.size(0)\n","                    correct += predicted.eq(batch_labels).sum().item()\n","                    acc = correct / total\n","\n","                    # Update progress bar\n","                    bar.update(batch_idx, loss=avg_loss, acc=acc)\n","\n","        return avg_loss, acc"]},{"cell_type":"markdown","metadata":{"id":"AYOJtYm1OWsJ"},"source":["Instantiate the classifier object that we built in classifier_module."]},{"cell_type":"code","execution_count":null,"metadata":{"scrolled":true,"id":"qI1p_muROWsJ"},"outputs":[],"source":["# Instantiate a classifier object\n","classifier = Classifier(device='cuda')"]},{"cell_type":"markdown","metadata":{"id":"NaP1S_CgOWsJ"},"source":["Visualize examples from the train set to get an idea of what they look like."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"L19dXA0DOWsJ"},"outputs":[],"source":["# Sample 100 images from the train set\n","img_list = classifier.sample(100)\n","# Combine these images in a mosaic\n","img_mosaic = build_mosaic(img_list, 10, 10, spacing=0, imborder=1, totborder=1)\n","# Display the image mosaic\n","display(img_mosaic)"]},{"cell_type":"markdown","metadata":{"id":"hcnmsNAeOWsJ"},"source":["Here, we include a utility function to visualize an image and the softmax score computed for that image by the classifier model."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JX2CGUa_OWsJ"},"outputs":[],"source":["# A function to plot an image and the model's softmax scores for that image\n","def plot_img_softmax(img, label, sm):\n","    # Get the true and predicted label name strings\n","    true_label = classifier.class_name_list[label]\n","    pred_label_idx = sm.argmax()\n","    pred_label = classifier.class_name_list[pred_label_idx]\n","    \n","    # Subplots to plot image and scores side by side\n","    fig, (img_ax, sm_ax) = plt.subplots(1, 2, figsize=(6, 3))\n","    \n","    # Plot the image\n","    img_ax.imshow(img, cmap='gray')\n","    img_ax.set_axis_off()\n","    img_ax.set_title('True label: {}\\nPredicted label: {}'.format(true_label, pred_label))\n","\n","    # Plot the softmax scores\n","    sm_ax.barh(range(len(sm)), sm)\n","    sm_ax.set_title('Softmax Probabilities')\n","    sm_ax.set_xlim(0.0, 1.0)\n","    sm_ax.set_yticks(range(len(sm)))\n","    sm_ax.set_yticklabels(classifier.class_name_list)\n","\n","    # Use tight_layout so labels don't overlap image\n","    plt.tight_layout()"]},{"cell_type":"markdown","metadata":{"id":"iE5ZblKeOWsJ"},"source":["We use this utility function to visualize 3 examples from the test set."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZUClSe6sOWsJ"},"outputs":[],"source":["# Sample 3 images from the test set and run them through the model\n","img_list, label_list, sm_list = classifier.sample_eval(3, seed=2)\n","# Display each image with softmax scores for each class\n","for img, label, sm in zip(img_list, label_list, sm_list):\n","    plot_img_softmax(img, label, sm)"]},{"cell_type":"markdown","metadata":{"id":"9IjfCaivOWsK"},"source":["This cell shows a standard pattern for training with the classifier, and retaining accuracy and loss statistics.\n","These statistics can then be visualized as a function of training epoch using matplotlib."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"r53tVGO0OWsK"},"outputs":[],"source":["# Training variables\n","num_epochs = 10\n","# Lists to hold statistics from training\n","train_loss_list, test_loss_list = [], []\n","train_acc_list, test_acc_list = [], []\n","# Train and test the model for num_epochs\n","for i in range(num_epochs):\n","    print('Epoch {}'.format(i), flush=True)\n","    # Train\n","    train_loss, train_acc = classifier.train()\n","    # Test\n","    test_loss, test_acc = classifier.test()\n","    # Save statistics from the epoch\n","    train_loss_list.append(train_loss), test_loss_list.append(test_loss)\n","    train_acc_list.append(train_acc), test_acc_list.append(test_acc)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"W-QIgSL1OWsK"},"outputs":[],"source":["# Plot loss and accuracy side by side\n","fig, (loss_ax, acc_ax) = plt.subplots(1, 2, figsize=(8, 4))\n","\n","# Plot train and test loss\n","loss_ax.plot(range(num_epochs), train_loss_list, label='Train')\n","loss_ax.plot(range(num_epochs), test_loss_list, label='Test')\n","loss_ax.legend(loc='center right')\n","loss_ax.set_xlabel('Epoch')\n","loss_ax.set_ylabel('Loss')\n","loss_ax.set_title('Loss vs. Epoch')\n","\n","# Plot train and test accuracy\n","acc_ax.plot(range(num_epochs), train_acc_list, label='Train')\n","acc_ax.plot(range(num_epochs), test_acc_list, label='Test')\n","acc_ax.legend(loc='center right')\n","acc_ax.set_xlabel('Epoch')\n","acc_ax.set_ylabel('Accuracy')\n","acc_ax.set_title('Accuracy vs. Epoch')\n","\n","# Tight layout so labels don't overlap axes\n","plt.tight_layout()"]},{"cell_type":"markdown","metadata":{"id":"lexzvbw4OWsK"},"source":["Finally, we use the example visualization function to show softmax scores for examples with the trained model.\n","Ideally, these scores will be close to 1.0 for the correct class (but not likely for this simple training procedure)."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"tf5KB707OWsK"},"outputs":[],"source":["# * Now that we have trained for a bit, we can try this step again\n","# Sample 3 images from the test set and run them through the model\n","img_list, label_list, sm_list = classifier.sample_eval(3, seed=2)\n","# Display each image with softmax scores for each class\n","for img, label, sm in zip(img_list, label_list, sm_list):\n","    plot_img_softmax(img, label, sm)"]},{"cell_type":"markdown","metadata":{"id":"f7bXpxuCOWsK"},"source":["### So, how well did we do?\n","\n","Not very! The best published model has 99.7% accuracy on this dataset: https://paperswithcode.com/sota/image-classification-on-cifar-10\n","\n","But the procedure for training that model is very similar. Researchers have tweaked each step in our process to get better results:\n","- The model will be much deeper, and have a specialized architecture e.g., ResNet: https://arxiv.org/abs/1512.03385\n","- The optimization procedure will decrease the learning rate, gradually or in steps, and could take specialized steps instead of basic SGD: https://arxiv.org/pdf/2010.01412v2.pdf\n","- The data will be heavily augmented, to try to cover all possible variations on appearance of the images\n","    - See the torchvision.transforms library: https://pytorch.org/docs/stable/torchvision/transforms.html\n","    - or the albumentations library: https://albumentations.ai/\n","- And many other things!\n","\n","\n","### What other classification datasets are there besides CIFAR10?\n","- The first dataset used for classifying images with a ConvNet was MNIST: http://yann.lecun.com/exdb/mnist/\n","    - Link to (later version of) the original paper by Yann LeCun et al.: http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf\n","- The most important benchmark dataset for image classification is ImageNet (specifically the version from the 2012 challenge): http://image-net.org/challenges/LSVRC/2012/index\n","    - AlexNet was the first effective deep learning model, trained on this dataset: http://www.cs.toronto.edu/~kriz/imagenet_classification_with_deep_convolutional.pdf\n","- torchvision supports several datasets, shown here: https://pytorch.org/vision/0.8/datasets.html\n","\n","### What visual recognition problems can we solve besides classification?\n","- Object detection: draw a rectangular box around all objects in an image\n","    - Paper for original version of an important object detection model (R-CNN): https://arxiv.org/abs/1311.2524\n","- Segmentation: classify every pixel in an image\n","    - Paper for important image segmentation model (FCN): https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf\n","- Image retrieval: return all images in a database similar to an input image\n","    - Paper for important image retrieval model: https://ieeexplore.ieee.org/document/1640964\n","- These problems applied to video\n","    - Action detection in video: https://arxiv.org/abs/1411.6031\n","- These problems in a different image domain (Infrared, Lidar)\n","- These problems with very few training examples\n","    - Method to pre-train network so it can quickly learn new task with few examples (meta-learning): https://arxiv.org/abs/1703.03400\n","- Applications such as self-driving cars: https://images.nvidia.com/content/tegra/automotive/images/2016/solutions/pdf/end-to-end-dl-using-px.pdf\n","\n","### What neural network libraries are there besides PyTorch?\n","- The most popular alternative to PyTorch is TensorFlow: https://www.tensorflow.org/\n","    - Commonly used with the Keras library: https://keras.io/\n","- There are also a number of helper / wrapper libraries for PyTorch:\n","    - fastai: https://www.fast.ai/\n","    - PyTorch Lightning: https://www.pytorchlightning.ai/\n","    - and other libraries in the PyTorch \"ecosystem\": https://pytorch.org/ecosystem/"]},{"cell_type":"markdown","metadata":{"id":"QBE7R1RnOWsK"},"source":["<a id='conclusion'></a>\n","## Q&A\n","\n","- Questions about machine learning / deep learning theory\n","- Questions about computer vision\n","- Questions about PyTorch"]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.8"},"colab":{"name":"image_classifier_tutorial_v1.2.ipynb","provenance":[]}},"nbformat":4,"nbformat_minor":0}