Tutorial 2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Area Processing (Tutorial 2)\n",
    "***\n",
    "# Table of Contents\n",
    "1.   [Imports](#Imports)\n",
    "2.   [Image Analysis](#Images-Analysis)\n",
    "3.   [Exercise 1 - Sliding Window](#Exercise-1---Sliding-Window)\n",
    "4.   [Exercise 2 - Convolution on RoI](#Exercise-2---Convolution-on-RoI)\n",
    "5.   [Exercise 3 - Convolution on the Whole Image](#Exercise-3---Convolution-on-the-Whole-Image)\n",
    "6.   [Exercise 4 - Different Convolution Kernels](#Exercise-4---Different-Convolution-Kernels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports\n",
    "\n",
    "Only 4 libraries are needed for this project:\n",
    "* opencv (cv2) - For image processing\n",
    "* numpy - For its arrays\n",
    "* matplotlib - Plotting histograms\n",
    "* os - File traversal\n",
    "* tqdm.notebook - tqdm progress bars, but for ipynb files\n",
    "* Classes - Custom classes written by me for this assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import os\n",
    "from tqdm.notebook import tqdm\n",
    "from Classes import Window, Sobel, Gaussian, Bilinear\n",
    "\n",
    "# A slightly modified generateHistograms function from tutorial 1\n",
    "\n",
    "def generateHistogram(image, title, detail):\n",
    "    hist = cv2.calcHist(image, [0], None, [255], [0, 255])\n",
    "    plt.ylabel('Pixels')\n",
    "    plt.xlabel('Intensity')\n",
    "    plt.title(title)\n",
    "    plt.plot(hist)\n",
    "    plt.xlim([0, 256])\n",
    "    plt.savefig(detail + '/'+ title + ' histogram.png')\n",
    "    plt.clf()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image Analysis\n",
    "add pixels\n",
    "### aiden dp.png\n",
    "\n",
    "This is a picture of me in the upper barakka gardens in Valletta. The focus is myself with the background of the Grand\n",
    "Harbour. The picture is lighted by natural lighting and includes a lot of non linear features. The image dimensions are 400x600.\n",
    "\n",
    "<img src=\"images/aiden dp.png\" width=400>\n",
    "\n",
    "### cursed.jpg\n",
    "\n",
    "The image I will be using for this lab is one of my failed attempts of creating a pie chart for one of my statistics units\n",
    "from last year. It features a sun like object in the centre with black lines coming out from the centre, it is coloured in\n",
    "full RGB while the background is gray. The image dimensions are 1920x1080.\n",
    "\n",
    "<img src=\"images/cursed.jpg\" width=400>\n",
    "\n",
    "### dog chaos.png\n",
    "\n",
    "This is a blurry picture of two of my friends and another friend's dog. They are in a dimly lit, tiled floor room with a\n",
    "white wall and some furniture behind one of my friends. The image dimensions are 1536x2048.\n",
    "\n",
    "<img src=\"images/dog chaos.png\" width=400>\n",
    "\n",
    "### jake car.png\n",
    "\n",
    "This is a picture of my classmate Jake standing next to a red car. The photo was taken in a street during daytime. The\n",
    "background features several home facades, 2 other cars and a person. The image dimensions are 813x813.\n",
    "\n",
    "<img src=\"images/jake car.png\" width=400>\n",
    "\n",
    "### jake close up.jpg\n",
    "\n",
    "This is a picture of my classmate Jake with a beige background. The image dimensions are 320x593.\n",
    "\n",
    "<img src=\"images/jake close up.jpg\">\n",
    "\n",
    "### jake sitting.jpg\n",
    "\n",
    "This is a picture of my classmate Jake sitting on the floor of a bathroom. The room is well lit and tiled all over. The image dimensions are 1536x2048.\n",
    "\n",
    "<img src=\"images/jake sitting.jpg\" width=400>\n",
    "\n",
    "### jojo ben.jpg\n",
    "\n",
    "This is a picture of my classmate Ben walking towards the Hal Ghaxaq church with a bag of fried chicken in his hand. The\n",
    "photo was taken during the night so the lighting comes from old street lamps. The image dimensions are 1536x2048.\n",
    "\n",
    "<img src=\"images/jojo ben.jpg\" width=400>\n",
    "\n",
    "#### I have permission by all the people shown to use these images for this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Here I load the images into a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c60a80725dc2465d8e3afbf1b74b1a48",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading Images:   0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "raw_images = {}\n",
    "for file in tqdm(os.listdir(\"images\"), desc='Loading Images'):\n",
    "    raw_images[file] = cv2.imread(\"images/\" + file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1 - Sliding Window\n",
    "\n",
    "### Window Class\n",
    "\n",
    "For this exercise I wrote the window class where I define some properties for the window.\n",
    "\n",
    "```python\n",
    "def __init__(self, image, n, s):\n",
    "    self.x_boundary = image.shape[1] + n\n",
    "    self.y_boundary = image.shape[0] + n\n",
    "    self.top_left = (0, 0)\n",
    "    self.bot_right = (n, n)\n",
    "    self.previousBotY = n\n",
    "    self.height = n\n",
    "    self.stride = (s, s)\n",
    "    try:\n",
    "        self.channels = image.shape[2]\n",
    "    except:\n",
    "        self.channels = 1\n",
    "```\n",
    "\n",
    "To create a Window, the image (min 2d numpy array), n (length or width) and s(stride or step) are required.\n",
    "\n",
    "Using these parameters I define:\n",
    "* The x and y boundaries for the window.\n",
    "* The starting top left and bottom right location as a tuple of two positions, where [0] is x and [1] is y.\n",
    "* Previous y position, this is used to check if the window has changed it's y position.\n",
    "* Height\n",
    "* Stride\n",
    "* Number of channels\n",
    "\n",
    "Each window is a square. The object is intended to be used for the image passed in initialisation.\n",
    "\n",
    "### Using Window for Ex 1\n",
    "\n",
    "For this exercise I use 4 functions from the class:\n",
    "\n",
    "```Python\n",
    "def getPos(self):\n",
    "    return self.top_left, self.bot_right\n",
    "```\n",
    "\n",
    "getPos returns the current position of the Window\n",
    "\n",
    "```python\n",
    "def forwardPos(self):\n",
    "    # Case when you need to go down and start new line\n",
    "    if (self.bot_right + self.stride)[0] >= (self.x_boundary - self.height):\n",
    "        return (0, self.top_left[1] + self.stride[1]), (self.height, self.bot_right[1] + self.stride[1])\n",
    "    # Generic move right case\n",
    "    else:\n",
    "        return (self.top_left[0] + self.stride[0], self.top_left[1]), \\\n",
    "               (self.bot_right[0] + self.stride[0], self.bot_right[1])\n",
    "```\n",
    "\n",
    "forwardPos returns the would be position of the next move. There are two cases:\n",
    "1. Next step stays in X boundary and so the new positions are just changed by adding stride\n",
    "2. Special case when next step would exceed X boundary so x positions are reset to 0, n and y positions are incremented\n",
    "by stride\n",
    "\n",
    "```python\n",
    "def forwardMove(self):\n",
    "    # Change positions\n",
    "    self.top_left, self.bot_right = self.forwardPos()\n",
    "    return self.top_left, self.bot_right\n",
    "```\n",
    "\n",
    "forwardMove changes the window's position to the return of forwardPos\n",
    "\n",
    "```python\n",
    "def inBoundary(self, new_top_left=None, new_bot_right=None):\n",
    "    # Use current position if no new positions are passed\n",
    "    if new_top_left is None:\n",
    "        new_top_left = self.top_left\n",
    "    if new_bot_right is None:\n",
    "        new_bot_right = self.bot_right\n",
    "    # Check if parameters are in boundary of the image given in initialisation\n",
    "    return new_bot_right[0] <= self.x_boundary and new_bot_right[1] <= self.y_boundary and \\\n",
    "           new_top_left[0] >= 0 and new_top_left[1] >= 0\n",
    "```\n",
    "\n",
    "inBoundary returns whether given positions, or the current positions are inBoundary of the image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Code Explanation\n",
    "\n",
    "First I initialise a Window for 'aiden dp.png' n=100, s=50. The reason for it being quite a \"large\" window is so that\n",
    "the sliding window demonstration can go fast.\n",
    "\n",
    "A rectangle is drawn over the initial positions and saved.\n",
    "\n",
    "rectangles are drawn in red.\n",
    "\n",
    "The starting and future positions are read using getPos and forwardPos. It is expected that win is initialised in boundary.\n",
    "\n",
    "Then I loop while win is in its boundary.\n",
    "\n",
    "* Every iteration I draw a rectangle on the image using cv2.rectangle.\n",
    "* If show is on I display this using imshow.\n",
    "* Then the positions are moved forwardMove and the future positions are taken again using forwardPos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "# On/Off switch\n",
    "show = False\n",
    "\n",
    "win = Window(raw_images[\"aiden dp.png\"], 100, 50)\n",
    "start_point, end_point = win.getPos()\n",
    "\n",
    "image = cv2.rectangle(raw_images[\"aiden dp.png\"].copy(), start_point, end_point, (0, 0, 255))\n",
    "cv2.imwrite(\"Output/aidenrectangle.png\", image, [cv2.IMWRITE_PNG_COMPRESSION, 0])\n",
    "\n",
    "new_tl, new_br = win.forwardPos()\n",
    "while win.inBoundary(new_br):\n",
    "    image = cv2.rectangle(raw_images[\"aiden dp.png\"].copy(), start_point, end_point, (0, 0, 255))\n",
    "    if show:\n",
    "        cv2.imshow(\"Sliding Window\", image)\n",
    "        cv2.waitKey(int(1/35*1000))\n",
    "    start_point, end_point = win.forwardMove()\n",
    "    new_tl, new_br = win.forwardPos()\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "A rectangle drawn using cv2.rectangle with the bounds taken from the Window object.\n",
    "\n",
    "<img src=\"Output/aidenrectangle.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2 - Convolution on RoI\n",
    "\n",
    "### Kernel Class\n",
    "\n",
    "For this exercise I wrote the window class where I define some properties for the window.\n",
    "\n",
    "```python\n",
    "def __init__(self, kernel, weight):\n",
    "    self.kernel = kernel\n",
    "    self.weight = weight\n",
    "```\n",
    "\n",
    "To create a Kernel, its kernel (numpy array) and weight  are required.\n",
    "\n",
    "Using these parameters I define:\n",
    "* The kernel which will be used to multiply and sum over a given area.\n",
    "* The weight which will be multiplied to the result of the kernel pass.\n",
    "\n",
    "Like the Window class, each Kernel is a square.\n",
    "\n",
    "\n",
    "### Using Kernel for Remaining Exercises\n",
    "\n",
    "For the remaining exercises I use the 2 functions from the class:\n",
    "\n",
    "```Python\n",
    "def filter(self, roi, axis=0, channels=1):\n",
    "    ret = []\n",
    "    if axis == 2:\n",
    "        for i in range(channels):\n",
    "            if channels == 1:\n",
    "                _filter = self.kernel * roi\n",
    "            else:\n",
    "                _filter = self.kernel * roi[:, :, i]\n",
    "            sum_of_filter1 = _filter.sum()\n",
    "            if channels == 1:\n",
    "                _filter = self.kernel.T * roi\n",
    "            else:\n",
    "                _filter = self.kernel.T * roi[:, :, i]\n",
    "            sum_of_filter2 = _filter.sum()\n",
    "            ret.append((((sum_of_filter1 ** 2) + (sum_of_filter2 ** 2)) ** (1 / 2)) * self.weight)\n",
    "\n",
    "        return np.array(ret)\n",
    "    else:\n",
    "        kernel = self.kernel if axis == 0 else self.kernel.T\n",
    "        for i in range(channels):\n",
    "            if channels == 1:\n",
    "                _filter = kernel * roi\n",
    "            else:\n",
    "                _filter = kernel * roi[:, :, i]\n",
    "            ret.append(_filter.sum() * self.weight)\n",
    "\n",
    "        return np.array(ret)\n",
    "```\n",
    "\n",
    "The filter function takes a roi, the axis of operation, and number channels of roi. There are two main parts and both\n",
    "function similarly.\n",
    "\n",
    "#### roi and kernel must have the same shape, otherwise there will be a shape error in the multiplication stage\n",
    "\n",
    "When axis is 2:\n",
    "\n",
    "1. Loop over each channel.\n",
    "2. Multiply roi's pixels in the channel with the kernel.\n",
    "3. Sum these values.\n",
    "4. Multiply roi's pixels in the channel with the kernel's transpose.\n",
    "5. Sum these values.\n",
    "6. Get the magnitude of both these values by squaring, adding then getting their square root. (vector magnitude)\n",
    "7. Multiply this result with kernel's weight.\n",
    "8. Append each result of every channel to a list.\n",
    "9. Convert ret to a numpy array and return.\n",
    "\n",
    "When axis is not 2:\n",
    "\n",
    "1. Loop over each channel.\n",
    "2. Multiply roi's pixels in the channel with the kernel.\n",
    "3. Sum these values.\n",
    "4. Multiply this result with the appropriate kernel's weight.\n",
    "5. Append each result of every channel to a list\n",
    "6. Convert ret to a numpy array and return\n",
    "\n",
    "In any case the return of filter is a pixel.\n",
    "\n",
    "\n",
    "```Python\n",
    "def filterImage(self, image, stride=1, window=None, axis=0):\n",
    "    new_image = []\n",
    "    line = []\n",
    "    if window is None:\n",
    "        moving_kernel = Window(image, self.kernel.shape[0], stride)\n",
    "    else:\n",
    "        image = window.getImageInBoundary(image)\n",
    "        moving_kernel = Window(image, self.kernel.shape[0], stride)\n",
    "\n",
    "    new_tl, _ = moving_kernel.forwardPos()\n",
    "    while moving_kernel.inBoundary(new_tl):\n",
    "        roi = moving_kernel.getImageInBoundary(image)\n",
    "        if moving_kernel.changedY():\n",
    "            new_image.append(line)\n",
    "            line = []\n",
    "\n",
    "        line.append(self.filter(roi, axis, moving_kernel.channels))\n",
    "\n",
    "        moving_kernel.forwardMove()\n",
    "        new_tl, _ = moving_kernel.forwardPos()\n",
    "\n",
    "    return np.array(new_image)\n",
    "```\n",
    "\n",
    "The filterImage function takes an image, stride, a window if it is expected to function on a RoI, and the axis of\n",
    "operation.\n",
    "\n",
    "new_image and line are 2 lists i will use for this function. new_image will be the output of passing the kernel over the\n",
    "image/RoI and line will be used to represent a  line of pixels.\n",
    "\n",
    "The first thing I do is check whether window was defined or not. If it is defined then the function is expected to work\n",
    "on a RoI, defined by window, on image and not the entire image. Hence, if it is defined I use the getImageInBoundary from\n",
    "the Window Class (explained below) and assign image to it. In any case, a moving_kernel is defined for image, the kernel\n",
    "and stride.\n",
    "\n",
    "Similar to the way I move the rectangle in Ex 1 using the Window move functions, here I define a RoI using moving_kernel\n",
    "and getImageInBoundary for image. This gives me a nxn copy of the image in moving_kernel's boundary. n here is the width\n",
    "and height of the Kernel.\n",
    "\n",
    "Then I check if the y position of moving_kernel has changed, if it did then I append line to new_image and reset line.\n",
    "This should happen in the first iteration.\n",
    "\n",
    "After this I append the filtered roi using the filter function to line.\n",
    "\n",
    "The next steps relate to the moving of the kernel.\n",
    "\n",
    "Finally new_images is returned as a numpy array.\n",
    "\n",
    "### Using Window for Remaining Exercises\n",
    "\n",
    "For this exercise I use 2 new functions (the other 4 functions are explained above) from the class:\n",
    "\n",
    "```Python\n",
    "def changedY(self):\n",
    "    if self.previousBotY == self.bot_right[1]:\n",
    "        return False\n",
    "    else:\n",
    "        self.previousBotY = self.bot_right[1]\n",
    "        return True\n",
    "```\n",
    "\n",
    "changedY checks whether the y value of the bottom right corner of the window has changed or not.\n",
    "\n",
    "```python\n",
    "def getImageInBoundary(self, image):\n",
    "    new_image = []\n",
    "    for i in range(self.top_left[1], self.bot_right[1]):\n",
    "        if i >= image.shape[0]:\n",
    "            continue\n",
    "        new_image.append(image[i][self.top_left[0]: self.bot_right[0]])\n",
    "\n",
    "    if self.channels == 1:\n",
    "        return np.resize(np.array(new_image), (self.height, self.height))\n",
    "    else:\n",
    "        return np.resize(np.array(new_image), (self.height, self.height, self.channels))\n",
    "```\n",
    "\n",
    "getImageInBoundary returns a numpy array of the pixels of image in the boundary of the Window. This is done by looping\n",
    "from the y value of the top left corner to the y value of the bottom right corner. Then for each iteration, using list\n",
    "slicing I append the x values from left to right.\n",
    "\n",
    "The return is sized appropriate to the image's ndims size.\n",
    "\n",
    "### Kernels\n",
    "\n",
    "```python\n",
    "class Sobel:\n",
    "    def __init__(self, weight):\n",
    "        self.kernel = Kernel(np.array([[-1, 0, 1],\n",
    "                                       [-2, 0, 2],\n",
    "                                       [-1, 0, 1]]),\n",
    "                             weight)\n",
    "\n",
    "    def filterImage(self, image, stride=1, window=None, axis=2):\n",
    "        return self.kernel.filterImage(image, stride, window, axis)\n",
    "\n",
    "\n",
    "class Gaussian:\n",
    "    def __init__(self, size, weight):\n",
    "        fwhm = size // 2\n",
    "        x = np.arange(0, size, 1, float)\n",
    "        y = x[:, np.newaxis]\n",
    "        x0 = y0 = size // 2\n",
    "        self.kernel = Kernel(np.exp(-4 * np.log(2) * ((x - x0) ** 2 + (y - y0) ** 2) / fwhm ** 2), weight)\n",
    "\n",
    "    def filterImage(self, image, stride=1, window=None, axis=0):\n",
    "        return self.kernel.filterImage(image, stride, window, axis)\n",
    "\n",
    "\n",
    "class Bilinear:\n",
    "    def __init__(self, weight):\n",
    "        self.kernel = Kernel(np.array([[1, 2, 1],\n",
    "                                       [2, 4, 2],\n",
    "                                       [1, 2, 1]]),\n",
    "                             weight)\n",
    "\n",
    "    def filterImage(self, image, stride=1, window=None, axis=0):\n",
    "        return self.kernel.filterImage(image, stride, window, axis)\n",
    "```\n",
    "\n",
    "The kernels used in this tutorial are then defined as above. The sobel and bilinear kernels are hard coded, while the\n",
    "gaussian kernel uses the code provided in the lecture notes to generate a kernel for the given size. Weight is explained\n",
    "above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Explanation\n",
    "\n",
    "I initialise a sobel kernel, I give a neutral weight because I don't think it needs one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sobel = Sobel(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Explanation\n",
    "\n",
    "I loop over every image and get a roi using getImageInBoundary of shape (300,300).\n",
    "Then I filter this roi using the sobel(x+y) kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "beb75bf9945c4733bb239a2633243869",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for image in tqdm(raw_images):\n",
    "    win = Window(raw_images[image], 300, 1)\n",
    "    roi = win.getImageInBoundary(raw_images[image])\n",
    "    cv2.imwrite(\"Output/RoI/\"+ image +\"_before_filter.png\", roi, [cv2.IMWRITE_PNG_COMPRESSION, 0])\n",
    "    filtered = sobel.filterImage(image=raw_images[image], window=win, axis=2)\n",
    "    cv2.imwrite(\"Output/RoI/\"+ image +\"_after_filter.png\", filtered, [cv2.IMWRITE_PNG_COMPRESSION, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Results\n",
    "Input | Filtered Output\n",
    "-|-\n",
    "<img src=\"./Output/RoI/aiden dp.png_before_filter.png\" >      | <img src=\"./Output/RoI/aiden dp.png_after_filter.png\" > \n",
    "<img src=\"./Output/RoI/cursed.jpg_before_filter.png\" >        | <img src=\"./Output/RoI/cursed.jpg_after_filter.png\" > \n",
    "<img src=\"./Output/RoI/dog chaos.png_before_filter.png\" >     | <img src=\"./Output/RoI/dog chaos.png_after_filter.png\" > \n",
    "<img src=\"./Output/RoI/jake car.png_before_filter.png\" >      | <img src=\"./Output/RoI/jake car.png_after_filter.png\" > \n",
    "<img src=\"./Output/RoI/jake close up.jpg_before_filter.png\" > | <img src=\"./Output/RoI/jake close up.jpg_after_filter.png\" > \n",
    "<img src=\"./Output/RoI/jake sitting.jpg_before_filter.png\" >  | <img src=\"./Output/RoI/jake sitting.jpg_after_filter.png\" > \n",
    "<img src=\"./Output/RoI/jojo ben.jpg_before_filter.png\" >       | <img src=\"./Output/RoI/jojo ben.jpg_after_filter.png\" > \n",
    "\n",
    "### Comments\n",
    "\n",
    "From the above images it can be seen that the Sobel (x+y) is able to bring out both x and y features in each image. Also in the case of cursed.jpg, since in the RoI there are no features to be extracted, so the output image is black. Further analyses of each image will be done in below sections. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Explanation\n",
    "\n",
    "I copy the \"aiden dp.png\" image, apply grayscale to it then get a roi and filter it as above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "image = raw_images[\"aiden dp.png\"].copy()\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)\n",
    "win = Window(image, 300, 1)\n",
    "roi = win.getImageInBoundary(image)\n",
    "cv2.imwrite(\"Output/RoI/aidendp_before_filter.png\", roi, [cv2.IMWRITE_PNG_COMPRESSION, 0])\n",
    "generateHistogram(roi, \"aidendp_before_filter\", \"Output/RoI\")\n",
    "filtered = sobel.filterImage(image=image, window=win)\n",
    "cv2.imwrite(\"Output/RoI/aidendp_after_filter.png\", filtered, [cv2.IMWRITE_PNG_COMPRESSION, 0])\n",
    "generateHistogram(cv2.imread(\"Output/RoI/aidendp_after_filter.png\"), \"aidendp_after_filter\", \"Output/RoI\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Results\n",
    "\n",
    "For this result I also include a histogram. I did not do this for the other exercises since I think it would be quite\n",
    "tedious to generate a histogram for every new image, when for the most part, visually the features created by the filter\n",
    "are apparent.\n",
    "\n",
    "Input | Filtered Output \n",
    "-|-\n",
    "<img src=\"./Output/RoI/aidendp_before_filter.png\" >      | <img src=\"./Output/RoI/aidendp_after_filter.png\" > \n",
    "\n",
    "### Comments\n",
    "\n",
    "By applying a Grayscale filter to the image before applying the sobel(x+y) filter a lot of visual noise becomes less defined. For example, in the coloured RoI, my face has a lot of cyan edges, in the grayscale output, this noise is white. The same example applies for the wave lines, where in teh colopured output the edges are orange, and in the grayscale version white. From this I would assume that for feature extraction it might be best to keep the input images in their original colouring.\n",
    "\n",
    "Input Histogram | Filtered Output Histogram\n",
    "-|-\n",
    "<img src=\"./Output/RoI/aidendp_before_filter histogram.png\" >      | <img src=\"./Output/RoI/aidendp_after_filter histogram.png\" >\n",
    "\n",
    "### Comments\n",
    "\n",
    "The filtered RoI has a higher consentration of black pixels, this makes finding edges and features (whiter) pixels easier. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3 - Convolution on the Whole Image\n",
    "\n",
    "In the next 2 exercises I pass the images through all the filters. Then in the markdown sections I discuss the result of\n",
    "the filter. Then I will discuss how the filter may effect feature extraction from the input image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Explanation\n",
    "\n",
    "I loop over every image and apply the sobel(x+y) filter on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0bc03ec22b644c32bf50fe336964f063",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for image in tqdm(raw_images):\n",
    "    filtered = sobel.filterImage(image=raw_images[image], axis=2)\n",
    "    cv2.imwrite(\"Output/Full Image/Sobel2/\"+ image +\"_after_filter.png\", filtered, [cv2.IMWRITE_PNG_COMPRESSION, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "Original | Filtered Output \n",
    "-|-\n",
    "<img src=\"./images/aiden dp.png\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel2/aiden dp.png_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/cursed.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel2/cursed.jpg_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/dog chaos.png\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel2/dog chaos.png_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jake car.png\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel2/jake car.png_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jake close up.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel2/jake close up.jpg_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jake sitting.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel2/jake sitting.jpg_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jojo ben.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel2/jojo ben.jpg_after_filter.png\" width=\"300\">\n",
    "\n",
    "### Comments\n",
    "\n",
    "In general the filter was able to bring out the focus of each image quite well. Like with \"jake close up.jpg\", younger Jake's hairline, glass, nose and mouth are outlined sufficiently well. However with higher quality input like \"ben jojo.jpg\" the filter was able to outline everything from the church's facade to the tarmac pieces on the road.\n",
    "\n",
    "I also noticed how lighting effected the edges in \"ben jojo.jpg\". For example the church facade and statue where lit using white LEDs while the street is lit with the old orange bulbs. The edges' colour for the respective objects in the image match with their lighting, and I think this information can be used for feature extraction.\n",
    "\n",
    "In other cases the edges' colour came more from the object's colour like the edges on Jake's face and arms in \"jake sitting.jpg\" or the car in \"jake car.png\".\n",
    "\n",
    "I also can see a clear difference between the clean, generated \"cursed.jpg\" and blurry, real life photo of \"dog chaos.png\". Whereas for the former there is no noise in the background or between sections of the pi chart, in the latter there is a lot of noise on my two friends, the dog and some of the tiling.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Exercise 4 - Different Convolution Kernels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Explanation\n",
    "\n",
    "I initialise a bilinear kernel, I give it a weight of 1/8 because otherwise the image would be too bright."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "bileaner = Bilinear(1/8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Explanation\n",
    "\n",
    "I initialise a gaussian kernel, I give it a weight of 1/4 because otherwise the image would be too bright."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "gaussian = Gaussian(5, 1/4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Explanation\n",
    "\n",
    "I collect the filters into a dict to make the last step easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "filters = {\"Sobel\":sobel,\n",
    "           \"Bilinear\":bileaner,\n",
    "           \"Gaussian\":gaussian}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Explanation\n",
    "\n",
    "I loop over every image and apply all the filters to them. However for the sobel kernel, instead of using the x+y here I\n",
    "use x and y seperately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5e4015666f0417a84e6e87cc30413dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for image in tqdm(raw_images):\n",
    "    for filter in filters:\n",
    "        if filter == \"Sobel\":\n",
    "            filtered = filters[filter].filterImage(image=raw_images[image], axis=0)\n",
    "            cv2.imwrite(\"Output/Full Image/Sobel0/\"+ image +\"_after_filter.png\", filtered, [cv2.IMWRITE_PNG_COMPRESSION, 0])\n",
    "            filtered = filters[filter].filterImage(image=raw_images[image], axis=1)\n",
    "            cv2.imwrite(\"Output/Full Image/Sobel1/\"+ image +\"_after_filter.png\", filtered, [cv2.IMWRITE_PNG_COMPRESSION, 0])\n",
    "        else:\n",
    "            filtered = filters[filter].filterImage(image=raw_images[image])\n",
    "            cv2.imwrite(\"Output/Full Image/\"+ filter + \"/\" + image +\"_after_filter.png\", filtered, [cv2.IMWRITE_PNG_COMPRESSION, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Results\n",
    "\n",
    "# Sobel X\n",
    "Original | Filtered Output\n",
    "-|-\n",
    "<img src=\"./images/aiden dp.png\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel0/aiden dp.png_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/cursed.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel0/cursed.jpg_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/dog chaos.png\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel0/dog chaos.png_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jake car.png\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel0/jake car.png_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jake close up.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel0/jake close up.jpg_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jake sitting.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel0/jake sitting.jpg_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jojo ben.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel0/jojo ben.jpg_after_filter.png\" width=\"300\">\n",
    "\n",
    "### Comments\n",
    "\n",
    "To avoid repeating myself from the previous section, the same can be said however for all the images the features being shown are perpendicular to the x axis.\n",
    "\n",
    "\n",
    "# Sobel Y\n",
    "Original | Filtered Output\n",
    "-|-\n",
    "<img src=\"./images/aiden dp.png\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel1/aiden dp.png_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/cursed.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel1/cursed.jpg_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/dog chaos.png\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel1/dog chaos.png_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/jake car.png\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel1/jake car.png_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/jake close up.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel1/jake close up.jpg_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jake sitting.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel1/jake sitting.jpg_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/jojo ben.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Sobel1/jojo ben.jpg_after_filter.png\" width=\"300\"> \n",
    "\n",
    "### Comments\n",
    "\n",
    "The same can be said here, however the features being shown are perpendicular to the y axis instead. However I noticed that in the \"dog chaos.png\" my friend's striped white shirt was still being somewhat prominent after the filter. In this case I would consider it as noise since whoever want to use this filter, (I assume) wouldn't want features perpendicular to the x axis.\n",
    "\n",
    "\n",
    "# Bilinear\n",
    "Original | Filtered Output\n",
    "-|-\n",
    "<img src=\"./images/aiden dp.png\" width=\"300\"> | <img src=\"./Output/Full Image/Bilinear/aiden dp.png_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/cursed.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Bilinear/cursed.jpg_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/dog chaos.png\" width=\"300\"> | <img src=\"./Output/Full Image/Bilinear/dog chaos.png_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/jake car.png\" width=\"300\"> | <img src=\"./Output/Full Image/Bilinear/jake car.png_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/jake close up.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Bilinear/jake close up.jpg_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jake sitting.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Bilinear/jake sitting.jpg_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/jojo ben.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Bilinear/jojo ben.jpg_after_filter.png\" width=\"300\"> \n",
    "\n",
    "### Comments\n",
    "\n",
    "The bilinear filter brightened each image, but this happened because I did not set a proper weight in initialisation. Other than that I noticed how the car in \"jake car.png\" had some blue noise added. After some research I learned that this is a normal outcome from the filter. I also saw that the filter is used for resizing images to half or double their size while retaining their quality.\n",
    "\n",
    "\n",
    "# Gaussian\n",
    "Original | Filtered Output\n",
    "-|-\n",
    "<img src=\"./images/aiden dp.png\" width=\"300\"> | <img src=\"./Output/Full Image/Gaussian/aiden dp.png_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/cursed.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Gaussian/cursed.jpg_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/dog chaos.png\" width=\"300\"> | <img src=\"./Output/Full Image/Gaussian/dog chaos.png_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/jake car.png\" width=\"300\"> | <img src=\"./Output/Full Image/Gaussian/jake car.png_after_filter.png\" width=\"300\"> \n",
    "<img src=\"./images/jake close up.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Gaussian/jake close up.jpg_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jake sitting.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Gaussian/jake sitting.jpg_after_filter.png\" width=\"300\">\n",
    "<img src=\"./images/jojo ben.jpg\" width=\"300\"> | <img src=\"./Output/Full Image/Gaussian/jojo ben.jpg_after_filter.png\" width=\"300\"> \n",
    "\n",
    "### Comments\n",
    "\n",
    "With the gaussian filter I managed to get a good weight so the image does not get too bright after filtration. It's de focusing effect can be seen throughout however I think it is most prevelant in \"aiden dp.png\". I think that the de-focusing effect can be changed by changing the size of the kernel. As for feature extraction I am inclined to say that the filter would be used to remove noise from the input image. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}