dl_demo05.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "demo5-test.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fc9wyyGloQwL"
      },
      "source": [
        "# Finetuning a ResNet\n",
        "\n",
        "Modern deep convnets tend to have tens (if not hundreds) of layers, with millions (if not tens of millions) of trainable parameters. More often than not several of these layers have skipped connections; the *ResNet* family of networks are an example. \n",
        "\n",
        "The flip side of having such deep network architectures is that to properly learn such networks, one requires *massive* amounts of training data. In most applications, access to such massive datasets simply isn't available; gathering and curating a dataset with a few hundred/thousand examples itself can be a challenge.\n",
        "\n",
        "What should one do in the ``small data'' setting? A possible solution:\n",
        "* start with a deep network architecture initialized with *pre-trained* weights, and\n",
        "* *fine-tune* the network weights on the (small) training dataset.\n",
        "\n",
        "In this demo we will see how to train a simple cat-vs-dog classifier using a very small training dataset of 60 images. The dataset is provided [here](https://github.com/chinmayhegde/dl-demos/blob/main/data.zip). You can unzip and save the dataset anywhere you like; I've saved it to my Google drive folder: `MyDrive\\data`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bGT3H4DjoO9n"
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from torchvision import datasets, models, transforms\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qNHnEAckpqps"
      },
      "source": [
        "\n",
        "# data transforms\n",
        "dset_transform = transforms.Compose([\n",
        "        transforms.Resize(256),\n",
        "        transforms.CenterCrop(224),\n",
        "        transforms.ToTensor(),\n",
        "        transforms.Normalize([0.485, 0.456, 0.406],\n",
        "                             [0.229, 0.224, 0.225])])\n",
        "\n",
        "\n",
        "# Use the image folder function to create datasets\n",
        "dsets = {x: datasets.ImageFolder(f\"/content/drive/MyDrive/data/{x}\", dset_transform)\n",
        "         for x in ['train', 'test']}\n",
        "\n",
        "# create data loader\n",
        "dataloaders = {x: torch.utils.data.DataLoader(dsets[x], batch_size=16,\n",
        "                                              shuffle=(x == \"train\"))\n",
        "               for x in ['train', 'test']}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iYSEZtTDJUYV"
      },
      "source": [
        "# Loading a pre-trained ResNet model\n",
        "\n",
        "Let's load a ResNet34 model from `torchvision` and examine it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tOYRfm1pBcoM"
      },
      "source": [
        "# intialize model\n",
        "model = models.resnet34(pretrained=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mVgCL8ZTtvTt",
        "outputId": "09f55e48-9227-4cbc-d857-0547a22c2304"
      },
      "source": [
        "print(model)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ResNet(\n",
            "  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
            "  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "  (relu): ReLU(inplace=True)\n",
            "  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
            "  (layer1): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (2): BasicBlock(\n",
            "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (layer2): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (downsample): Sequential(\n",
            "        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
            "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      )\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (2): BasicBlock(\n",
            "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (3): BasicBlock(\n",
            "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (layer3): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (downsample): Sequential(\n",
            "        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
            "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      )\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (2): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (3): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (4): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (5): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (layer4): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (downsample): Sequential(\n",
            "        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
            "        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      )\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (2): BasicBlock(\n",
            "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n",
            "  (fc): Linear(in_features=512, out_features=1000, bias=True)\n",
            ")\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EfWGLTlJLxzI"
      },
      "source": [
        "Hmm, lots of layers. Each `BasicBlock` is two or three conv layers with a skipped connection, with batch-norm layers thrown in for good measure. Several such residual blocks are pieced together, and in the end there is a dense layer with 1000 output neurons. This model has been trained on the well-known *ImageNet* dataset (which has over a million images with 1000 classes). As an aside, let's examine the number of trainable parameters in the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Jl9m-Z4EDFFW",
        "outputId": "e5be439e-7a37-44ff-cd51-275981c573b4"
      },
      "source": [
        "def count_parameters(model):\n",
        "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "print(count_parameters(model))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "21797672\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FdzbKTClM_Qv"
      },
      "source": [
        "Let us finetune this model for our cat-vs-dog classification problem. Since this is a binary classifier we will redefine the output (linear) layer to have 2 outputs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qkjny7j5BDRS",
        "outputId": "ac369f98-f12d-4141-ac0e-b71d796df3cc"
      },
      "source": [
        "num_ftrs = model.fc.in_features\n",
        "print(num_ftrs)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "512\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nhnOEfZoDQS6"
      },
      "source": [
        "model.fc = nn.Linear(num_ftrs, 2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Jv86CyS3DVe8",
        "outputId": "2b5b294f-e026-4083-9d05-9e5294f39291"
      },
      "source": [
        "print(model)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ResNet(\n",
            "  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
            "  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "  (relu): ReLU(inplace=True)\n",
            "  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
            "  (layer1): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (2): BasicBlock(\n",
            "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (layer2): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (downsample): Sequential(\n",
            "        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
            "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      )\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (2): BasicBlock(\n",
            "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (3): BasicBlock(\n",
            "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (layer3): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (downsample): Sequential(\n",
            "        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
            "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      )\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (2): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (3): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (4): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (5): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (layer4): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (downsample): Sequential(\n",
            "        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
            "        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      )\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (2): BasicBlock(\n",
            "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n",
            "  (fc): Linear(in_features=512, out_features=2, bias=True)\n",
            ")\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TIDpqRLJOMaJ"
      },
      "source": [
        "Observe now that the basic ResNet34 backbone remains the same; only the output layer has changed. In fact, all of the weights (except the output layer) also have been retained.  \n",
        "\n",
        "Let's do a quick model evaluation to check if there are any errors thrown during prediction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bq72K2tuDn3h",
        "outputId": "55fb64bb-3092-47a1-ad94-13a21d0dc3d0"
      },
      "source": [
        "model.eval()\n",
        "corrects = 0\n",
        "for batch_idx, (inputs,labels) in enumerate(dataloaders['test'], 1):\n",
        "  with torch.set_grad_enabled(False):\n",
        "    outputs = model(inputs)\n",
        "    _, preds = torch.max(outputs,1)\n",
        "    \n",
        "  corrects += torch.sum(preds == labels.data)\n",
        "\n",
        "print(corrects.float() / len(dataloaders['test'].dataset))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)\n",
            "  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(0.5000)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LI3eN9vcPEHb"
      },
      "source": [
        "As we can see, we get an accuracy of 50\\% on the test set -- which is exactly what we would expect since the weights of the output layer are random. \n",
        "\n",
        "We are now ready to start fine-tuning! The rest of the code below is boilerplate training; we see below that only a few epochs are enough to tune the weights to our training dataset.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UZ8n_358A6aY"
      },
      "source": [
        "# define loss function, optimizer\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)\n",
        "save_loss = {'train':[], 'test':[]}\n",
        "save_acc = {'train':[], 'test':[]}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wJHYjaYtrNHm",
        "outputId": "e970dd83-9647-4845-cdf0-673b57e70c5e"
      },
      "source": [
        "for epoch in range(5):\n",
        "\n",
        "    # Each epoch has a training and validation phase\n",
        "    for phase in ['train', 'test']:\n",
        "        if phase == 'train':\n",
        "            model.train()  # Set model to training mode\n",
        "        else:\n",
        "            model.eval()   # Set model to evaluate mode\n",
        "\n",
        "        current_loss = 0.0\n",
        "        current_corrects = 0\n",
        "\n",
        "        for batch_idx, (inputs, labels) in enumerate(dataloaders[phase], 1):\n",
        "            optimizer.zero_grad()\n",
        "\n",
        "            # Time to carry out the forward training poss\n",
        "            with torch.set_grad_enabled(phase == 'train'):\n",
        "                outputs = model(inputs)\n",
        "                _, preds = torch.max(outputs, 1)\n",
        "                loss = criterion(outputs, labels)\n",
        "\n",
        "                # backward + optimize only if in training phase\n",
        "                if phase == 'train':\n",
        "                    loss.backward()\n",
        "                    optimizer.step()\n",
        "\n",
        "            # We want variables to hold the loss/acc statistics\n",
        "            current_loss += loss.item() * inputs.size(0)\n",
        "            current_corrects += torch.sum(preds == labels.data)\n",
        "        # saving variable for plottin\n",
        "        save_loss[phase] += [current_loss / len(dataloaders[phase].dataset)]\n",
        "        save_acc[phase] += [current_corrects.float() / len(dataloaders[phase].dataset)]\n",
        "\n",
        "        # pretty print\n",
        "        print(f\"Epoch:{epoch} -- Phase:{phase} -- Loss:{save_loss[phase][-1]:.2f} -- Acc:{save_acc[phase][-1]*100:.2f}\")\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch:0 -- Phase:train -- Loss:0.84 -- Acc:51.67\n",
            "Epoch:0 -- Phase:test -- Loss:0.41 -- Acc:95.83\n",
            "Epoch:1 -- Phase:train -- Loss:0.49 -- Acc:75.00\n",
            "Epoch:1 -- Phase:test -- Loss:0.25 -- Acc:95.83\n",
            "Epoch:2 -- Phase:train -- Loss:0.26 -- Acc:91.67\n",
            "Epoch:2 -- Phase:test -- Loss:0.07 -- Acc:100.00\n",
            "Epoch:3 -- Phase:train -- Loss:0.08 -- Acc:100.00\n",
            "Epoch:3 -- Phase:test -- Loss:0.04 -- Acc:100.00\n",
            "Epoch:4 -- Phase:train -- Loss:0.05 -- Acc:100.00\n",
            "Epoch:4 -- Phase:test -- Loss:0.02 -- Acc:100.00\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 298
        },
        "id": "JGgbCwdTuy1h",
        "outputId": "75c79f5d-f52b-4109-9525-0232417c38c4"
      },
      "source": [
        "plt.plot(save_acc['train'])\n",
        "plt.plot(save_acc['test'])\n",
        "plt.legend([\"train\", \"test\"])\n",
        "plt.title(\"Accuracy\")\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'Accuracy')"
            ]
          },
          "metadata": {},
          "execution_count": 12
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU5d3/8feXEAh7gIQtARIREQTZYgCRqo/1EaiCu6AoruDW2tbayq/W7bGtfezTuq/IriJVqliwUqtWFBGSsO8xLAlrCIQ1e+7fHzNqjMFMYJIzM/m8riuXM+ecmfPlmPnknvs+5z7mnENERMJfA68LEBGR4FCgi4hECAW6iEiEUKCLiEQIBbqISIRQoIuIRAgFuohIhFCgS9gxs0/M7ICZNfa6FpFQokCXsGJmScAwwAGj6nC/DetqXyInSoEu4eYGYAkwDRj/9UIz62xmc80s18zyzOzZCutuM7P1ZnbYzNaZ2QD/cmdmp1bYbpqZPeZ/fJ6Z5ZjZb8xsNzDVzFqb2T/8+zjgf5xY4fVtzGyqme30r3/Hv3yNmV1SYbtoM9tnZv1r7ShJvaRAl3BzA/Ca/+ciM2tvZlHAP4BtQBKQAMwGMLOrgIf9r2uJr1WfF+C+OgBtgK7ABHyfl6n+512AAuDZCtvPBJoCZwDtgL/6l88AxlXYbiSwyzm3PMA6RAJimstFwoWZnQN8DHR0zu0zsw3AS/ha7PP8y0srveYDYIFz7qkq3s8B3Z1zmf7n04Ac59wDZnYesBBo6ZwrPE49/YCPnXOtzawjsANo65w7UGm7TsBGIME5d8jM3gKWOuf+94QPhkgV1EKXcDIeWOic2+d//rp/WWdgW+Uw9+sMfHWC+8utGOZm1tTMXjKzbWZ2CPgUiPV/Q+gM7K8c5gDOuZ3A58AVZhYLjMD3DUMkqDTQI2HBzJoAVwNR/j5tgMZALLAH6GJmDasI9Wyg23He9hi+LpKvdQByKjyv/PX1XqAHMMg5t9vfQl8OmH8/bcws1jmXX8W+pgO34vvMfeGc23H8f63IiVELXcLFpUAZ0Avo5//pCSzyr9sFPG5mzcwsxsyG+l83GfiVmQ00n1PNrKt/3QrgWjOLMrPhwLnV1NACX795vpm1AR76eoVzbhfwPvC8f/A02sx+VOG17wADgHvw9amLBJ0CXcLFeGCqc267c2731z/4BiXHApcApwLb8bWyrwFwzv0N+D2+7pnD+IK1jf897/G/Lh+4zr/uhzwJNAH24eu3/2el9dcDJcAGYC/w869XOOcKgLeBZGBuDf/tIgHRoKhIHTGzB4HTnHPjqt1Y5ASoD12kDvi7aG7B14oXqRXqchGpZWZ2G75B0/edc596XY9ELnW5iIhECLXQRUQihGd96HFxcS4pKcmr3YuIhKX09PR9zrn4qtZ5FuhJSUmkpaV5tXsRkbBkZtuOt05dLiIiEUKBLiISIRToIiIRQoEuIhIhFOgiIhGi2kA3sylmttfM1hxnvZnZ02aWaWarvr69l4iI1K1AWujTgOE/sH4E0N3/MwF44eTLEhGRmqr2PHTn3Kf+O60fz2hghvPNIbDEzGLNrKN/fmiR8HEkF3KWwu41UF7VzY+kskOFJWzcc5iyMk0hUhNtBozmtAHVTb9fc8G4sCgB38RDX8vxL/teoJvZBHyteLp06RKEXYucoLJS2LsWspdCzjLI/hIObK2wgXlVWVhw/p/mDgZ6XUwYWtayI4RooAfMOfcy8DJASkqK/qRL3Tm2/9vgzl4KOzKg5KhvXfP20DkVUm6BzoOgY1+IjvG23hBUVu74cP0eJi/KYtnWA7SIaci1g7ow/uwkOsU28bq8sDKolt43GIG+A98Ncr+W6F8m4o3yMsjd8N3Wd16mb51FQYc+0H+cL8QTz4LYLmBqkR/PseJS3krPYcpnW9iad4yE2Cb87uJeXHNWZ5o31i0VQkkw/m/MA+42s9n4/vAcVP+51KmCfNiR5gvw7KWwIx2KDvnWNY3zBXe/63yt7079oVHTH34/AWDPoUKmL97Ka19u52BBCf06x/LcRadz0RntaRilM55DUbWBbmZvAOcBcWaWg+/GuNEAzrkXgQXASCAT313Ub6qtYkUoL4e8zf7Wtz/AczcCDqwBtDsD+lz1beu7zSlqfdfQ+l2HmLxoC/NW7qC03HFRrw7c9qNkBnZtU/2LxVOBnOUytpr1DrgraBWJVFR02Nfi/rr1nbMMCvN962JifcHd+0rffxMGQOMW3tYbppxz/GdTLpMXbeGzzH00bRTFdYO6ctPQJLq2beZ1eRIgdYBJ6HAO9md9t/W9dx24csAg/nToNdrf+k6FtqdCA331PxmFJWW8u2IHkxdtYfPeI7Rv2ZhfD+/BdaldadU02uvypIYU6OKd4qOwc7n/zJNlvhA/ludb17glJKbA6Rf7AzwFYlp5W28EyTtSxKwl25m5ZCv7jhTTs2NL/nJ1Xy4+sxONGuqPZLhSoEvdcA7yt30b3Nlf+i7gcWW+9W27w2kjoPNZvtZ3/OlqfdeCzL1HePWzLczNyKGotJzze8Rz27BTGNKtLaaxhrCnQJfaUVIIu1Z8e953zjI4sse3LroZJA6Ec37hO/MkMQWaasCttjjn+CIrj1cXbeHfG/bSqGEDrhiQwM1Dk+neXmMOkUSBLsFxMOe7533vWgXlJb51rZPhlPO/bX236wVR+tWrbSVl5cxftYvJn2WxZsch2jZrxM9/3J1xg7sS17yx1+VJLdCnSmqutBh2r/pu6/uQ/1qyhk18Z5sMucvf+j4Lmld5P1upJQcLSnhj6Xamfb6V3YcK6RbfjD9e3ofL+icQEx3ldXlSixToUr3Du7975snOFVBW5FvXqgt0GfLted8d+kCUzo7wQvb+Y0z5fAtzlmVztLiMs7u15Y+X9+Hc0+Jp0ED94/WBAl2+q6wE9qypcN73Usjf7lsX1Rg69YPU23yt786p0KKDt/UKGdsPMHlRFv9cs5sGZozq24lbhiVzRiedFVTfKNDru6P7vtv63pEBpQW+dS06+UJ70O2+vu+OZ0JD9b2GgrJyx8K1u3llURYZ2/NpGdOQied2Y/yQJDq00sRi9ZUCvT4pL/NdqFPxvO/9Wb51DRr6ZhkceKNv8LLzIGiV6Gm58n1Hi0qZk5bNlM+3kL2/gC5tmvLwJb24KqUzzTRRVr0Xfr8BOWmwdZHXVYSXwkO+yat2ZEDxEd+yZu18re+BN/pa3536QbSmQA1Vuw4WMH3xNl7/chuHCksZ2LU1vx3Zkwt7dSBK/ePiF36Bvu1z+PBhr6sILxYFHXpD37H+vu+zILarJq0KA2t3HmTyoi28t3In5c4xondHbhmWzIAurb0uTUJQ+AX64DshdYLXVYSXBg115kkYKS93fLJpL698uoUvsvJo1iiKG4YkcdPQJDq30dS/cnzhF+hR0QoniUiFJWXMzdjBq59l8VXuUTq2imHSiNMZk9qFVk30Oy/VC79AF4kw+44UMfOLbcxaso28o8X0TmjJU2P6MbJPR6J1IwmpAQW6iEc27znsmyhr+Q6KS8v5cc923DrsFAYlt9FEWXJCFOgidcg5x+Kv8nhlURafbMylccMGXDUwkZvPSaZbfHOvy5Mwp0AXqQPFpeW8t3Inkz/bwvpdh4hr3ohfXnga4wZ3pU2zRl6XJxFCgS5Si/KPFfPal9uZvngrew8XcVr75vzvFWcyql8nTZQlQadAF6kF2/KOMuWzLcxJy6GgpIxh3eN44qq+/Kh7nPrHpdYo0EWCxDlH+rYDvLIoi4Xr9tCwgTG6XwK3nJNMz44tvS5P6gEFushJKi0r559rdzN50RZWZOfTqkk0d57nmyirXUtNlCV1R4EucoKOFJUye+l2pn6+lR35BSS1bcr/jD6DKwYm0rSRPlpS9/RbJ1JDO/MLmLZ4K298uZ3DRaWkJrXhoUt6cUHP9pooSzylQBcJ0Oqcg7yyKIv5q3cBMLJPR249J5m+nWM9rkzER4Eu8gPKyx3/3rCXyYuy+HLLfpo3bshNZydx49AkEltroiwJLQp0kSoUFJfxdkYOUz7bQta+oyTENuGBn/TkmrM60yJGE2VJaFKgi1RQUFzGC59kMnPJNg4cK6FvYiueGdufEb070FATZUmIU6CL+O0/Wswt05exIjufH/dsz23DTuGspNa6EEjChgJdBMjef4zxU5eSc6CAF64bwPDeHb0uSaTGFOhS763deZAbpy6jqKSMWbcMIjW5jdcliZwQBbrUa4sz9zFhZjotYhry2h1nc1r7Fl6XJHLCAhrlMbPhZrbRzDLN7P4q1nc1s3+b2Soz+8TMEoNfqkhwzVu5k/FTl5IQ24S5dyrMJfxVG+hmFgU8B4wAegFjzaxXpc3+DMxwzp0JPAr8MdiFigTT5EVZ/OyN5fTv0po5tw+hY6smXpckctICaaGnApnOuSznXDEwGxhdaZtewEf+xx9XsV4kJJSXO34/fx2PzV/PiN4dmHFzqm7ALBEjkEBPALIrPM/xL6toJXC5//FlQAsza1v5jcxsgpmlmVlabm7uidQrcsKKS8v5xZwVvLJoCzcM6cqz1w7QTSYkogTrSolfAeea2XLgXGAHUFZ5I+fcy865FOdcSnx8fJB2LVK9w4Ul3DxtGe+u2Ml9F/XgkVFnaCItiTiBnOWyA+hc4Xmif9k3nHM78bfQzaw5cIVzLj9YRYqcjL2HC7lp6jI27D7Mn6/qy5UDNWYvkSmQQF8GdDezZHxBPga4tuIGZhYH7HfOlQOTgCnBLlTkRGTlHmH81KXsO1zM5PEpnN+jndclidSaartcnHOlwN3AB8B6YI5zbq2ZPWpmo/ybnQdsNLNNQHvg97VUr0jAlm8/wJUvfsGxojJmTxisMJeIZ845T3ackpLi0tLSPNm3RL6PNuzhrteWE9+iMTNuTiUprpnXJYkEhZmlO+dSqlqnK0Ul4sxJy2bS3NX07NiCqTemEt+isdclidQJBbpEDOccz36Uyf/9axPDusfxwriBNG+sX3GpP/TbLhGhrNzx0Lw1zFqyncv6J/CnK86kUUPNXy71iwJdwl5hSRn3zF7OB2v3MPHcU/jNRafTQOeYSz2kQJewln+smFunp5G+/QAPXdKLm4Yme12SiGcU6BK2duYXMH7KUrblHeOZsf25+MxOXpck4ikFuoSljbsPM37KUo4WlTL95lSGdPve1EEi9Y4CXcLOkqw8bpuRRpPoKObcPoSeHVt6XZJISFCgS1h5f/Uu7nlzBZ1bN2H6zakktm7qdUkiIUOBLmFj+uKtPPzeWgZ0ac3kG1Jo3ayR1yWJhBQFuoQ85xxPfLCR5z/5ih/3bM+z1/bXPOYiVVCgS0grKSvn/rdX83ZGDmNTu/A/o8+gYZQuGBKpigJdQtbRolLufC2D/2zK5Rc/Po2fXXAqZrpgSOR4FOgSkvYdKeLmactYs+Mgj1/ehzGpXbwuSSTkKdAl5GzLO8r4KUvZfaiQl69P4ce92ntdkkhYUKBLSFmdc5Cbpi2ltNzx+m2DGdCltdcliYQNBbqEjE835XL7rHRaN23E7JtTObVdc69LEgkrCnQJCX9fnsN9f1tF9/YtmHbTWbRvGeN1SSJhR4EunnLO8dKnWTz+/gbO7taWF68fSMuYaK/LEglLCnTxTHm549F/rGPa4q1c0rcTf77qTBo31AVDIidKgS6eKCwp4945K5m/ehe3nJPMb0f21E0pRE6SAl3q3KHCEibMSGNJ1n5+O7Int/3oFK9LEokICnSpU7sPFnLj1KV8lXuEJ6/px6X9E7wuSSRiKNClzmTuPcz4KcvIP1bM1BtTOad7nNcliUQUBbrUifRt+7l5WhrRUQ14c+IQeie08rokkYijQJdat3Dtbn76xnI6xTZh+k2pdGmrm1KI1AYFutSq17/czgPvrKZPYixTxqfQtnljr0sSiVgKdKkVzjn++uFmnv73Zs7vEc9z1w2gaSP9uonUJn3CJOhKy8p54J01zF6WzVUDE/nD5X2I1k0pRGqdAl2CqqC4jLtfz+DfG/Zy9/mncu9/n6abUojUEQW6BM3+o8XcMn0ZK7Lz+Z9Le3P94K5elyRSryjQJSiy9x9j/NSl5Bwo4IXrBjK8dwevSxKpdwLq2DSz4Wa20cwyzez+KtZ3MbOPzWy5ma0ys5HBL1VC1dqdB7n8hcXsO1zEa7cOUpiLeKTaQDezKOA5YATQCxhrZr0qbfYAMMc51x8YAzwf7EIlNC3O3Mc1Ly2hYQPjrTvO5qykNl6XJFJvBdJCTwUynXNZzrliYDYwutI2Dmjpf9wK2Bm8EiVUzVu5k/FTl5IQ24S5d57Nae1beF2SSL0WSB96ApBd4XkOMKjSNg8DC83sp0Az4MdVvZGZTQAmAHTporu4h7PJi7J4bP56UpPb8MoNKbRqoptSiHgtWCcHjwWmOecSgZHATDP73ns75152zqU451Li4+ODtGupS+Xljt/PX8dj89czoncHZtycqjAXCRGBtNB3AJ0rPE/0L6voFmA4gHPuCzOLAeKAvcEoUkJDcWk59721kndX7OSGIV156JIziNJNKURCRiAt9GVAdzNLNrNG+AY951XaZjtwAYCZ9QRigNxgFireOlxYws3TlvHuip3cd1EPHhmlMBcJNdW20J1zpWZ2N/ABEAVMcc6tNbNHgTTn3DzgXuAVM/sFvgHSG51zrjYLl7qz93AhN01dxobdh/nzVX25cmCi1yWJSBUCurDIObcAWFBp2YMVHq8Dhga3NAkFWblHGD91KXlHinl1fArn9WjndUkichy6UlSOa/n2A9wyPQ0D3rhtMH07x3pdkoj8AAW6VOmjDXu467XlxLdozIybU0mKa+Z1SSJSDQW6fM+ctGwmzV1Nz44tmHpjKvEtdFMKkXCgQJdvOOd49qNM/u9fmxjWPY4Xxg2keWP9ioiEC31aBYCycsdD89Ywa8l2Lu+fwONXnEmjhrophUg4UaALhSVl3DN7OR+s3cPt53bjN8N76KYUImFIgV7PHTxWwq0zlpG27QAPXdKLm4Yme12SiJwgBXo9tjO/gPFTlrIt7xjPjO3PxWd28rokETkJCvR6auPuw4yfspSjRaVMvzmVId3ael2SiJwkBXo9tCQrj9tmpNG0URRzbh9Cz44tq3+RiIQ8BXo98/7qXdzz5go6t27CjFsGkRDbxOuSRCRIFOj1yPTFW3n4vbUM6NKayTek0LpZI69LEpEgUqDXA845nvhgI89/8hUX9mrPM2P7ExMd5XVZIhJkCvQIV1JWzv1vr+btjByuHdSFR0edQcMoXTAkEokU6BHu4XlreTsjh19eeBo//a9TdcGQSARToEew2Uu389qX25l47in87ILuXpcjIrVM370j1PLtB3jw3bUM6x7Hry863etyRKQOKNAj0N7DhdwxK4P2rRrzzNj+uvenSD2hLpcIU1xazl2vZZBfUMzcO4YS21SnJorUFwr0CPP7+etYtvUAT43pR69OugJUpD5Rl0sE+VtaNtO/2MZtw5IZ3S/B63JEpI4p0CPEqpx8fvvOGoae2pbfDNcgqEh9pECPAPuOFDFxZjrxzRvzzNgBunBIpJ5SH3qYKynzDYLuP1rM23ecTRvNzyJSbynQw9wfFqznyy37+es1femd0MrrckTEQ/puHsbmZuQw9fOt3Dw0mcv6J3pdjoh4TIEeptbsOMikuasZfEobJo3UIKiIKNDD0v6jxUycmU7bZo149toBRGsQVERQH3rYKS0r5+7XM8g9UsRbtw8hrnljr0sSkRChpl2Y+dM/N7D4qzz+cFkfzkyM9bocEQkhCvQw8u6KHbyyaAs3np3ElQM1CCoi36VADxNrdx7kN2+vIjWpDb/9SU+vyxGREBRQoJvZcDPbaGaZZnZ/Fev/amYr/D+bzCw/+KXWXwf8g6CxTRrx3HUaBBWRqlU7KGpmUcBzwIVADrDMzOY559Z9vY1z7hcVtv8p0L8Waq2XSsvK+ekby9l7qIg5tw8hvoUGQUWkaoE09VKBTOdclnOuGJgNjP6B7ccCbwSjOIEnFm7ks8x9PHZpb/p11iCoiBxfIIGeAGRXeJ7jX/Y9ZtYVSAY+Os76CWaWZmZpubm5Na213vnHqp289J8sxg3uwtVndfa6HBEJccHujB0DvOWcK6tqpXPuZedcinMuJT4+Psi7jiwbdh/ivr+tYmDX1jx48RlelyMiYSCQQN8BVGweJvqXVWUM6m45afnHipkwI52WTRrywnUDaNRQg6AiUr1AkmIZ0N3Mks2sEb7Qnld5IzM7HWgNfBHcEuuXsnLHz2avYNfBAl4YN5B2LWO8LklEwkS1ge6cKwXuBj4A1gNznHNrzexRMxtVYdMxwGznnKudUuuH/1u4kU835fLo6N4M6NLa63JEJIwENJeLc24BsKDSsgcrPX84eGXVTwtW7+L5T75ibGoXxqZ28bocEQkz6pwNEZv2HOZXf1tJ/y6xPDyql9fliEgYUqCHgIMFJUyYkUazxg15cdxAGjeM8rokEQlDCnSPlZc7fj57OTkHCnjhugG01yCoiJwgBbrHnvxwEx9vzOWhUWeQktTG63JEJIwp0D30wdrdPP1RJtekdGbcIA2CisjJUaB7JHPvYX755gr6do7lkdFnYGZelyQiYU6B7oFDhSVMmJFOk0ZRvDhuADHRGgQVkZOne4rWsfJyxy/fXMH2/cd47dZBdGzVxOuSRCRCqIVex57+aDMfrt/L7y7uxaBT2npdjohEEAV6HfrXuj08+eFmrhiQyA1DunpdjohEGAV6Hfkq9wi/fHMFfRJa8fvLemsQVESCToFeBw4X+q4EbdSwAS9eP1CDoCJSKzQoWsvKyx33zlnJ1rxjzLplEAmxGgQVkdqhFnote+7jTBau28NvR/ZkSDcNgopI7VGg16KPNuzhLx9u4rL+Cdw0NMnrckQkwinQa8mWfUe5Z/YKenVsyR8u66NBUBGpdQr0WnCkqJQJM9Jo2MB4cdxAmjTSIKiI1D4NigaZc477/raSr3KPMPOWQXRu09TrkkSknlALPche+M9XvL9mN5NG9GToqXFelyMi9YgCPYg+2biXJz7YyKi+nbh1WLLX5YhIPaNAD5JteUf52RvLOb1DS/50xZkaBBWROqdAD4JjxaVMnJlOgwbGy9drEFREvKFAP0nOOe57axWb9hzmmbH9NQgqIp5RoJ+klz/NYv6qXfx6+OkM6x7vdTkiUo8p0E/Cos25/OmfG/hJn45M/NEpXpcjIvWcAv0EZe8/xk/fWE73di343ys1CCoi3lOgn4CC4jImzEynvNzx0vUDadZY12eJiPeURDXknOP+uavYsPsQU288i6S4Zl6XJCICqIVeY69+toV3V+zkV//dg/N6tPO6HBGRbyjQa2Bx5j7++P4GRvTuwJ3ndfO6HBGR71CgByjnwDHuej2DU+Ka8cRVfTUIKiIhR4EegMKSMibOTKe03PHyDSk01yCoiISggALdzIab2UYzyzSz+4+zzdVmts7M1prZ68Et0zvOOSbNXc26XYd4akw/kjUIKiIhqtqmpplFAc8BFwI5wDIzm+ecW1dhm+7AJGCoc+6AmUXMaOG0xVv5+/Id/PLC0/iv09t7XY6IyHEF0kJPBTKdc1nOuWJgNjC60ja3Ac855w4AOOf2BrdMbyzJyuOx+eu5sFd77j7/VK/LERH5QYEEegKQXeF5jn9ZRacBp5nZ52a2xMyGV/VGZjbBzNLMLC03N/fEKq4jO/MLuOu1DJLaNuUvV/elQQMNgopIaAvWoGhDoDtwHjAWeMXMYitv5Jx72TmX4pxLiY8P3YmsCkvKuH1WOkWl5bx8QwotYqK9LklEpFqBBPoOoHOF54n+ZRXlAPOccyXOuS3AJnwBH3acczzwzhpW5Rzkr9f0o1t8c69LEhEJSCCBvgzobmbJZtYIGAPMq7TNO/ha55hZHL4umKwg1llnZi7ZxlvpOdxzQXcu7KVBUBEJH9UGunOuFLgb+ABYD8xxzq01s0fNbJR/sw+APDNbB3wM3Oecy6utomvL0i37efS9dVxwejvuuSAsv2CISD1mzjlPdpySkuLS0tI82XdVdh0s4JJnPqNlTDTv3D2Uluo3F5EQZGbpzrmUqtbpkkegqLSM22dlUFBcxhu3DVaYi4SwkpIScnJyKCws9LqUWhUTE0NiYiLR0YHnUb0PdOccD76zlpXZ+bw4bgDd27fwuiQR+QE5OTm0aNGCpKSkiJ1TyTlHXl4eOTk5JCcnB/y6ej+Xy+tLt/NmWjZ3n38qw3t39LocEalGYWEhbdu2jdgwBzAz2rZtW+NvIfU60NO37efheWs5v0c8v7jwNK/LEZEARXKYf+1E/o31NtD3HCrk9lkZJMQ24ckx/YnSlaAiEubqZaAXlZZxx6x0jhaV8tL1KbRqokFQEQlMfn4+zz//fI1fN3LkSPLz82uhom/Vy0B/5L11ZGzP589X9aVHBw2CikjgjhfopaWlP/i6BQsWEBv7vRlRgqreneXyxtLtvP7ldu44rxsj+2gQVCScPfLeWtbtPBTU9+zVqSUPXXLGcdfff//9fPXVV/Tr14/o6GhiYmJo3bo1GzZsYNOmTVx66aVkZ2dTWFjIPffcw4QJEwBISkoiLS2NI0eOMGLECM455xwWL15MQkIC7777Lk2aNDnp2utVCz1j+wEeenctw7rH8av/7uF1OSIShh5//HG6devGihUreOKJJ8jIyOCpp55i06ZNAEyZMoX09HTS0tJ4+umnycv7/kXzmzdv5q677mLt2rXExsby9ttvB6W2etNC33u4kDtmpdO+VWOeGatBUJFI8EMt6bqSmpr6nXPFn376af7+978DkJ2dzebNm2nbtu13XpOcnEy/fv0AGDhwIFu3bg1KLfUi0ItLy7nrtQwOFZQy986ziW3ayOuSRCRCNGv27W0pP/nkEz788EO++OILmjZtynnnnVflueSNGzf+5nFUVBQFBQVBqaVeBPpj89exbOsBnhnbn54dW3pdjoiEsRYtWnD48OEq1x08eJDWrVvTtGlTNmzYwJIlS+q0togP9Dlp2cz4YhsTf3QKl/Tt5HU5IhLm2rZty9ChQ+nduzdNmu+/4RcAAAh0SURBVDShfftvp9kePnw4L774Ij179qRHjx4MHjy4TmuL6NkWV2bnc9VLX5Ca1IZpN51Fw6h6NQYsEpHWr19Pz549vS6jTlT1b/2h2RYjNuFyDxcxcWY67Vr4BkEV5iIS6SKyy6WkrJy7Xs8gv6CYt+84m9bNNAgqIpEvIgP99/PXs3TLfp4a048zOrXyuhwRkToRcf0Qb6fnMG3xVm45J5nR/RK8LkdEpM5EVKCvzjnI//v7aoac0pZJI073uhwRkToVMYGed6SI22elE9e8Mc9eq0FQEal/IiL1SsvKufv15ew7UsRL1w+kbfPG1b9IROQEnOj0uQBPPvkkx44dC3JF34qIQP/j+xv4IiuPP17eh94JGgQVkdoTyoEe9me5vLN8B69+toUbz07i8gGJXpcjInXp/fth9+rgvmeHPjDi8eOurjh97oUXXki7du2YM2cORUVFXHbZZTzyyCMcPXqUq6++mpycHMrKyvjd737Hnj172LlzJ+effz5xcXF8/PHHwa2bMA/0NTsOcv/cVaQmt+G3P6kfV46JiLcef/xx1qxZw4oVK1i4cCFvvfUWS5cuxTnHqFGj+PTTT8nNzaVTp07Mnz8f8M3x0qpVK/7yl7/w8ccfExcXVyu1hW2g7z9azMSZ6bRu2ojnrh1AtAZBReqfH2hJ14WFCxeycOFC+vfvD8CRI0fYvHkzw4YN49577+U3v/kNF198McOGDauTesIy0EvLyvnpGxnkHinibxOHEN9Cg6AiUvecc0yaNImJEyd+b11GRgYLFizggQce4IILLuDBBx+s9XrCsln7xAcb+Twzj8cu7U3fzrV7jz4RkYoqTp970UUXMWXKFI4cOQLAjh072Lt3Lzt37qRp06aMGzeO++67j4yMjO+9tjaEXQv9vZU7eenTLG4Y0pWrUzp7XY6I1DMVp88dMWIE1157LUOGDAGgefPmzJo1i8zMTO677z4aNGhAdHQ0L7zwAgATJkxg+PDhdOrUqVYGRcNu+tzFmfuYtngrz147gEYNw/ILhoicBE2fe/zpc8OuhX72qXGcfWrtjBCLiIQzNXFFRCKEAl1Ewo5XXcV16UT+jQEFupkNN7ONZpZpZvdXsf5GM8s1sxX+n1trXImISABiYmLIy8uL6FB3zpGXl0dMTEyNXldtH7qZRQHPARcCOcAyM5vnnFtXadM3nXN312jvIiI1lJiYSE5ODrm5uV6XUqtiYmJITKzZdCaBDIqmApnOuSwAM5sNjAYqB7qISK2Ljo4mOTnZ6zJCUiBdLglAdoXnOf5llV1hZqvM7C0zq/IEcTObYGZpZpYW6X9dRUTqWrAGRd8DkpxzZwL/AqZXtZFz7mXnXIpzLiU+Pj5IuxYREQgs0HcAFVvcif5l33DO5TnnivxPJwMDg1OeiIgEKpA+9GVAdzNLxhfkY4BrK25gZh2dc7v8T0cB66t70/T09H1mtq2G9X4tDth3gq+tTaqrZlRXzYVqbaqrZk6mrq7HW1FtoDvnSs3sbuADIAqY4pxba2aPAmnOuXnAz8xsFFAK7AduDOB9T7jPxczSjnfpq5dUV82orpoL1dpUV83UVl0BXfrvnFsALKi07MEKjycBk4JbmoiI1ISuFBURiRDhGugve13AcaiumlFdNReqtamumqmVujybPldERIIrXFvoIiJSiQJdRCRChHSgBzDLY2Mze9O//kszSwqRujyZfdLMppjZXjNbc5z1ZmZP++teZWYDQqSu88zsYIXjVet30zWzzmb2sZmtM7O1ZnZPFdvU+fEKsC4vjleMmS01s5X+uh6pYps6/zwGWJdns8GaWZSZLTezf1SxLvjHyzkXkj/4znn/CjgFaASsBHpV2uZO4EX/4zH4ZnwMhbpuBJ714Jj9CBgArDnO+pHA+4ABg4EvQ6Su84B/1PGx6ggM8D9uAWyq4v9jnR+vAOvy4ngZ0Nz/OBr4EhhcaRsvPo+B1OXJ59G/718Cr1f1/6s2jlcot9C/meXROVcMfD3LY0Wj+XbemLeAC8zMQqAuTzjnPsV3YdfxjAZmOJ8lQKyZdQyBuuqcc26Xcy7D//gwvqubK086V+fHK8C66pz/GBzxP432/1Q+o6LOP48B1uUJM0sEfoJvOpSqBP14hXKgBzLL4zfbOOdKgYNA2xCoCwKYfdIDgdbuhSH+r83vm9kZdblj/1fd/vhadxV5erx+oC7w4Hj5uw9WAHuBfznnjnu86vDzGEhd4M3n8Ung10D5cdYH/XiFcqCHs4Bmn5RvZABdnXN9gWeAd+pqx2bWHHgb+Llz7lBd7bc61dTlyfFyzpU55/rhm6Av1cx618V+qxNAXXX+eTSzi4G9zrn02t5XRaEc6NXO8lhxGzNrCLQC8ryuy4Xu7JOBHNM655w79PXXZuebZiLazOJqe79mFo0vNF9zzs2tYhNPjld1dXl1vCrsPx/4GBheaZUXn8dq6/Lo8zgUGGVmW/F1y/6Xmc2qtE3Qj1coB/o3szyaWSN8gwbzKm0zDxjvf3wl8JHzjzB4WVelftaAZp+sI/OAG/xnbwwGDrpvZ8n0jJl1+Lrv0MxS8f1e1moQ+Pf3KrDeOfeX42xW58crkLo8Ol7xZhbrf9wE3y0pN1TarM4/j4HU5cXn0Tk3yTmX6JxLwpcRHznnxlXaLOjHK6DJubzgApvl8VVgppll4ht0GxMiddV49slgMLM38J0BEWdmOcBD+AaJcM69iG+CtZFAJnAMuClE6roSuMPMSoECYEwd/GEeClwPrPb3vwL8P6BLhbq8OF6B1OXF8eoITDffPYYbAHOcc//w+vMYYF2efB6rUtvHS5f+i4hEiFDuchERkRpQoIuIRAgFuohIhFCgi4hECAW6iEiEUKCLiEQIBbqISIT4//FKgpfMQpA9AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1__s46hPeUg"
      },
      "source": [
        "There we go! That took only 4 epochs of finetuning.\n",
        "\n",
        "This was a very small dataset (of only 60 training images) so it is not that surprising that we were able to fit the data so easily. Try training your own classifier with a slightly larger set of data points, and see if you can get similar results."
      ]
    }
  ]
}