TensorFlow: Advanced Techniques/C1_W4_Lab_2_resnet-example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/https-deeplearning-ai/tensorflow-3-public/blob/main/Course%201%20-%20Custom%20Models%2C%20Layers%20and%20Loss%20Functions/Week%204%20-%20Models/C1_W4_Lab_2_resnet-example.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ungraded Lab: Implementing ResNet\n",
    "\n",
    "In this lab, you will continue exploring Model subclassing by building a more complex architecture. \n",
    "\n",
    "[Residual Networks](https://arxiv.org/abs/1512.03385) make use of skip connections to make deep models easier to train. \n",
    "- There are branches as well as many repeating blocks of layers in this type of network. \n",
    "- You can define a model class to help organize this more complex code, and to make it easier to re-use your code when building the model.\n",
    "- As before, you will inherit from the [Model class](https://keras.io/api/models/model/) so that you can make use of the other built-in methods that Keras provides."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CmI9MQA6Z72_"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "from tensorflow.keras.layers import Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement Model subclasses\n",
    "\n",
    "As shown in the lectures, you will first implement the Identity Block which contains the skip connections (i.e. the `add()` operation below. This will also inherit the Model class and implement the `__init__()` and `call()` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-FIkYUttchv5"
   },
   "outputs": [],
   "source": [
    "class IdentityBlock(tf.keras.Model):\n",
    "    def __init__(self, filters, kernel_size):\n",
    "        super(IdentityBlock, self).__init__(name='')\n",
    "\n",
    "        self.conv1 = tf.keras.layers.Conv2D(filters, kernel_size, padding='same')\n",
    "        self.bn1 = tf.keras.layers.BatchNormalization()\n",
    "\n",
    "        self.conv2 = tf.keras.layers.Conv2D(filters, kernel_size, padding='same')\n",
    "        self.bn2 = tf.keras.layers.BatchNormalization()\n",
    "\n",
    "        self.act = tf.keras.layers.Activation('relu')\n",
    "        self.add = tf.keras.layers.Add()\n",
    "    \n",
    "    def call(self, input_tensor):\n",
    "        x = self.conv1(input_tensor)\n",
    "        x = self.bn1(x)\n",
    "        x = self.act(x)\n",
    "\n",
    "        x = self.conv2(x)\n",
    "        x = self.bn2(x)\n",
    "        x = self.act(x)\n",
    "\n",
    "        x = self.add([x, input_tensor])\n",
    "        x = self.act(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From there, you can build the rest of the ResNet model. \n",
    "- You will call your `IdentityBlock` class two times below and that takes care of inserting those blocks of layers into this network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YnMkmeecxw28"
   },
   "outputs": [],
   "source": [
    "class ResNet(tf.keras.Model):\n",
    "    def __init__(self, num_classes):\n",
    "        super(ResNet, self).__init__()\n",
    "        self.conv = tf.keras.layers.Conv2D(64, 7, padding='same')\n",
    "        self.bn = tf.keras.layers.BatchNormalization()\n",
    "        self.act = tf.keras.layers.Activation('relu')\n",
    "        self.max_pool = tf.keras.layers.MaxPool2D((3, 3))\n",
    "\n",
    "        # Use the Identity blocks that you just defined\n",
    "        self.id1a = IdentityBlock(64, 3)\n",
    "        self.id1b = IdentityBlock(64, 3)\n",
    "\n",
    "        self.global_pool = tf.keras.layers.GlobalAveragePooling2D()\n",
    "        self.classifier = tf.keras.layers.Dense(num_classes, activation='softmax')\n",
    "\n",
    "    def call(self, inputs):\n",
    "        x = self.conv(inputs)\n",
    "        x = self.bn(x)\n",
    "        x = self.act(x)\n",
    "        x = self.max_pool(x)\n",
    "\n",
    "        # insert the identity blocks in the middle of the network\n",
    "        x = self.id1a(x)\n",
    "        x = self.id1b(x)\n",
    "\n",
    "        x = self.global_pool(x)\n",
    "        return self.classifier(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Model\n",
    "\n",
    "As mentioned before, inheriting the Model class allows you to make use of the other APIs that Keras provides, such as:\n",
    "- training\n",
    "- serialization\n",
    "- evaluation\n",
    "\n",
    "You can instantiate a Resnet object and train it as usual like below:\n",
    "\n",
    "**Note**: If you have issues with training in the Coursera lab environment, you can also run this in Colab using the \"open in colab\" badge link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6dMHKPz_dIc8"
   },
   "outputs": [],
   "source": [
    "# utility function to normalize the images and return (image, label) pairs.\n",
    "def preprocess(features):\n",
    "    return tf.cast(features['image'], tf.float32) / 255., features['label']\n",
    "\n",
    "# create a ResNet instance with 10 output units for MNIST\n",
    "resnet = ResNet(10)\n",
    "resnet.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# load and preprocess the dataset\n",
    "dataset = tfds.load('mnist', split=tfds.Split.TRAIN, data_dir='./data')\n",
    "dataset = dataset.map(preprocess).batch(32)\n",
    "\n",
    "# train the model.\n",
    "resnet.fit(dataset, epochs=1)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "ResNetExample.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}