Semantic_Segmentation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/JebaMercy14/AI-Spartificial-Course/blob/main/Semantic_Segmentation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# <center> Starting with Semantic Segmentation"
      ],
      "metadata": {
        "id": "ns5OL2MIOBCz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# <center> Transpose Convolution"
      ],
      "metadata": {
        "id": "katgEikoOs2C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Transposed Convolutions are used to upsample the input feature map to a desired output feature map using some learnable parameters.\n",
        "\n"
      ],
      "metadata": {
        "id": "9M1Q0NmUTJmz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Why do we need transpose convolution?\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "One of the best ways for us to gain some intuition is by looking at examples from Computer Vision that use the transposed convolution. Most of these examples start with a series of regular convolutions to compress the input data into an abstract spatial representation, and then use transposed convolutions to decompress the abstract representation into something of use.\n",
        "\n",
        "\n",
        "![](https://miro.medium.com/max/1400/1*1OabPemOWCLrpCwIUmIsCg.png)\n",
        "\n",
        "\n",
        "A convolutional auto-encoder is tasked with recreating its input image, after passing intermediate results through a ‘bottleneck’ of a limited size. Uses of auto-encoders include compression, noise removal, colourisation and in-painting. Success depends on being able to learn dataset specific compression in the convolution kernels and dataset specific decompression in the transposed convolution kernels. Why stop there though?\n",
        "\n",
        "\n",
        "## Super Resolution\n",
        "![](https://miro.medium.com/max/1400/1*eTMo62iQxKp9aR3b5iG14w.png)\n",
        "\n",
        "\n",
        "With ‘super resolution’ the objective is to upscale the input image to higher resolutions, so transposed convolutions can be used as an alternative to classical methods such as bicubic interpolation. Similar arguments to convolutions using learnable kernels over hand crafted kernels apply here.\n",
        "\n",
        "## Semantic Segmentation\n",
        "\n",
        "\n",
        "Semantic segmentation is an example of using transposed convolution layers to decompress the abstract representation into a different domain (from the RGB image input). We output a class for each pixel of the input image, and then just for visualisation purposes, we render each class as a distinct colour (e.g. the person class shown in red, cars in dark blue, etc.).\n",
        "\n",
        "\n",
        "![](https://miro.medium.com/max/1400/1*KICInky28yGdU9T45kIL5Q.jpeg)"
      ],
      "metadata": {
        "id": "DYzUi4PGc5HL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "1. The CNN layers we have seen so far, such as convolutional layers and pooling layers typically reduce (downsample) the spatial dimensions (height and width) of the input, or keep them unchanged. \n",
        "\n",
        "2. In semantic segmentation that classifies at pixel-level, it will be convenient if the spatial dimensions of the input and output are the same. For example, the channel dimension at one output pixel can hold the classification results for the input pixel at the same spatial position.\n",
        "\n",
        "3. To achieve this, especially after the spatial dimensions are reduced by CNN layers, we can use another type of CNN layers that can increase (upsample) the spatial dimensions of intermediate feature maps. \n",
        "\n",
        "4. In this session, we will introduce transposed convolution, which is also called fractionally-strided convolution, for reversing downsampling operations by the convolution."
      ],
      "metadata": {
        "id": "efVG-Tz9RDYE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Now let's understand how transpose convolution works \n",
        "\n"
      ],
      "metadata": {
        "id": "RFhD-P9Qep2p"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### The basic operation that goes in a transposed convolution is explained below:\n",
        "### 1. Consider a 2x2 encoded feature map which needs to be upsampled to a 3x3 feature map"
      ],
      "metadata": {
        "id": "7Y19GH5Ccxvz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://miro.medium.com/max/97/1*BMJnnOKPhK8hoFP6sQ9edQ.png) \n",
        "\n",
        "Input Feature map\n",
        "\n",
        "![](https://miro.medium.com/max/252/1*VxtMdM-DsGwIa51GyDx-XQ.png)\n",
        "\n",
        "Output feature shape\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "HePsk9b6Tj8h"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2. We take a kernel of size 2x2 with unit stride and zero padding.\n",
        "\n",
        "![](https://miro.medium.com/max/102/1*e6UnrcsFRaOidCq7mwJpTA.png)\n",
        "\n",
        "kernel of size 2*2"
      ],
      "metadata": {
        "id": "K0X3rUS2ZPlU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3. Now we take the upper left element of the input feature map and multiply it with every element of the kernel\n",
        "\n",
        "![](https://miro.medium.com/max/429/1*7hVid7EAqCPkG6sEjHMI5w.png)\n",
        "\n"
      ],
      "metadata": {
        "id": "dVBnkoFhZhM8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4. Similarly, we do it for all the remaining elements of the input feature map\n",
        "\n",
        "![](https://miro.medium.com/max/700/1*yxBd_pCiEVVwEQFmc-Heog.png)"
      ],
      "metadata": {
        "id": "9rKOfEFyaW_D"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5. As you can see, some of the elements of the resulting upsampled feature maps are over-lapping. To solve this issue, we simply add the elements of the over-lapping positions\n",
        "\n",
        "![](https://miro.medium.com/max/700/1*faRskFzI7GtvNCLNeCN8cg.png)\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "63TJ9CHObYnW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 6. The resulting output will be the final upsampled feature map having the required spatial dimensions of 3x3\n",
        "\n",
        "![](https://miro.medium.com/max/790/1*Lpn4nag_KRMfGkx1k6bV-g.gif)"
      ],
      "metadata": {
        "id": "Bw17loKScMYq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def trans_conv(X, K):\n",
        "    h, w = K.shape\n",
        "    Y = np.zeros((X.shape[0] + h - 1, X.shape[1] + w - 1))\n",
        "    for i in range(X.shape[0]):\n",
        "        for j in range(X.shape[1]):\n",
        "            Y[i: i + h, j: j + w] += X[i, j] * K\n",
        "    return Y"
      ],
      "metadata": {
        "id": "ipLCMGRQucrc"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "X = np.array([[0.0, 1.0], [2.0, 3.0]])\n",
        "K = np.array([[0.0, 1.0], [2.0, 3.0]])\n",
        "trans_conv(X, K)"
      ],
      "metadata": {
        "id": "mZOhZAQ8udyi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7561550e-1762-43b5-f525-a585fa073ebe"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 0.,  0.,  1.],\n",
              "       [ 0.,  4.,  6.],\n",
              "       [ 4., 12.,  9.]])"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Experimenting with Transpose convolution with tensorflow"
      ],
      "metadata": {
        "id": "kELqsxvCYqd9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf"
      ],
      "metadata": {
        "id": "la_Cuz45Vgdq"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LJH58u2kNzHM",
        "outputId": "dad6384a-0f6e-4a2e-d228-072ab9b66aeb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[[[[-0.08466379  0.25475454  0.01260146]\n",
            "   [ 0.01814774  0.17642874 -0.24472183]\n",
            "   [ 0.24477501 -0.24257861  0.1865685 ]]\n",
            "\n",
            "  [[-0.38995063  0.64350474 -0.01756809]\n",
            "   [-0.15111329  0.33942872 -0.19101146]\n",
            "   [ 0.15556449 -0.5329158  -0.03335661]]\n",
            "\n",
            "  [[-0.48634544  0.17914513 -0.01003093]\n",
            "   [-0.24455942  0.29074305  0.02210162]\n",
            "   [-0.01392412 -0.15220435 -0.09843332]]]], shape=(1, 3, 3, 3), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "kz = 2 # kernel_size\n",
        "st = 1 # strides\n",
        "\n",
        "input = tf.random.uniform((1, 2, 2, 2)) #batch size, width, height, channels\n",
        "\n",
        "trans_conv1 = tf.keras.layers.Conv2DTranspose(filters = 3, kernel_size = kz, strides= st , padding='valid')\n",
        "\n",
        "output = trans_conv1(input)\n",
        "\n",
        "print(output)\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "kz = 3 # kernel_size\n",
        "st = 1 # strides\n",
        "\n",
        "input = tf.random.uniform((1, 2, 2, 2)) #batch size, width, height, channels\n",
        "\n",
        "trans_conv1 = tf.keras.layers.Conv2DTranspose(filters = 3, kernel_size = kz, strides= st , padding='valid')\n",
        "\n",
        "output = trans_conv1(input)\n",
        "\n",
        "print(output)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XnUQVuY8XwmY",
        "outputId": "1c62435b-e274-4b1f-97e1-560f65e7e99f"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[[[[ 0.34643492  0.03597212 -0.22168972]\n",
            "   [ 0.54429567  0.21926317 -0.35629812]\n",
            "   [ 0.32950836  0.13011386 -0.06064356]\n",
            "   [ 0.25266448 -0.12815812  0.09157193]]\n",
            "\n",
            "  [[ 0.2795857   0.19144295 -0.32391468]\n",
            "   [ 0.12711976  0.42421287 -0.45251486]\n",
            "   [ 0.032432    0.03403628 -0.35516438]\n",
            "   [ 0.1431707  -0.26054835  0.04130229]]\n",
            "\n",
            "  [[-0.13268091  0.20225747  0.07178836]\n",
            "   [-0.33900785  0.1753172  -0.49738213]\n",
            "   [-0.00076675 -0.14027904 -0.19886103]\n",
            "   [ 0.11437131 -0.21555634  0.26918483]]\n",
            "\n",
            "  [[ 0.11986202  0.06037656 -0.02993785]\n",
            "   [ 0.13260365 -0.08890825 -0.16591592]\n",
            "   [ 0.05599908 -0.12997448  0.09801738]\n",
            "   [ 0.08851447 -0.07894183  0.22635822]]]], shape=(1, 4, 4, 3), dtype=float32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "kz = 3 # kernel_size\n",
        "st = 2 # strides\n",
        "\n",
        "input = tf.random.uniform((1, 2, 2, 2)) #batch size, width, height, channels\n",
        "\n",
        "trans_conv1 = tf.keras.layers.Conv2DTranspose(filters = 3, kernel_size = kz, strides= st , padding='valid')\n",
        "\n",
        "output = trans_conv1(input)\n",
        "\n",
        "print(output)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z6I64TYzYD1J",
        "outputId": "b3267c93-465f-42b4-eaa3-d697950d6337"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[[[[-0.04079768 -0.2164548   0.03892717]\n",
            "   [-0.20397754  0.11503612  0.06166063]\n",
            "   [-0.21081898  0.1511766  -0.05352715]\n",
            "   [-0.21922427 -0.26045385 -0.2923513 ]\n",
            "   [-0.03880234  0.12246458  0.34634694]]\n",
            "\n",
            "  [[ 0.1620264  -0.11607978  0.0633934 ]\n",
            "   [-0.02376665 -0.1149589  -0.17441308]\n",
            "   [ 0.05175866  0.57124513 -0.21088901]\n",
            "   [ 0.12215492 -0.26720235 -0.18595803]\n",
            "   [-0.35696712  0.35010314  0.17473221]]\n",
            "\n",
            "  [[-0.08597484 -0.31014818 -0.03328104]\n",
            "   [-0.02016504  0.23105477 -0.2745966 ]\n",
            "   [-0.32460612 -0.13623601 -0.04326191]\n",
            "   [-0.01131815  0.12397291 -0.2083188 ]\n",
            "   [-0.33467287  0.5957557   0.1206312 ]]\n",
            "\n",
            "  [[ 0.24684122  0.01942601  0.03383207]\n",
            "   [ 0.03213599 -0.20927843 -0.22424164]\n",
            "   [-0.11069179  0.30995268 -0.00111128]\n",
            "   [-0.0080029  -0.16603892 -0.21934828]\n",
            "   [-0.2856409   0.37978572 -0.15692893]]\n",
            "\n",
            "  [[ 0.02462343 -0.22991769 -0.07112646]\n",
            "   [ 0.30649418  0.22015527 -0.26680908]\n",
            "   [-0.2808435   0.16450796 -0.1484199 ]\n",
            "   [ 0.30339748  0.27581352 -0.25625068]\n",
            "   [-0.28657618  0.2921787  -0.14594324]]]], shape=(1, 5, 5, 3), dtype=float32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Difference between Conv2DTranspose and UpSampling2D in keras\n",
        "\n",
        "Transpose: https://keras.io/api/layers/convolution_layers/convolution2d_transpose/\n",
        "\n",
        "UpSampling: https://keras.io/api/layers/reshaping_layers/up_sampling2d/"
      ],
      "metadata": {
        "id": "XNUJwMkA6vMD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Reference: https://www.matthewzeiler.com/mattzeiler/deconvolutionalnetworks.pdf"
      ],
      "metadata": {
        "id": "e7Ne9N2UN_XG"
      },
      "execution_count": 7,
      "outputs": []
    }
  ]
}