DigitClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_lxIB_QCV8iO"
   },
   "source": [
    "# Handwritten Digits Classification\n",
    "\n",
    "### Sidclay da Silva\n",
    "\n",
    "### June 2022\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0tcFKvgIW6ss"
   },
   "source": [
    "### Introduction\n",
    "\n",
    "In this notebook we develop a Python model for image classification of handwritten digits using a convolutional neural network (CNN) based on TensorFlow framework.\n",
    "\n",
    "Explanation of concepts and logic is provided to help understanding the steps along the notebook, but there is no detailed explanation about the code, some Phyton and arrays knowledge is required to follow it.\n",
    "\n",
    "The environment uses Python 3.7, TensorFlow 2.8 and TensorFlow Datasets 4.0.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cmuD_juU12BW"
   },
   "source": [
    "### MNIST Database\n",
    "\n",
    "The images used in this project are from the well known MNIST Database, a collection 70,000 handwritten digit images. For details please check the original work of Yann LeCun, Corinna Cortes and Christopher Burges at http://yann.lecun.com/exdb/mnist/.\n",
    "\n",
    "The images database is available in several formats and sources on the internet, TensorFlow already provides it as dataset, details at https://www.tensorflow.org/datasets/catalog/mnist.\n",
    "\n",
    "Our source will be the Extendet MNIST (EMNIST) Python package (https://pypi.org/project/emnist/) from Aaron Hosford. It provides the image database as Numpy array, this gives us a good opportunity to see how to prepare arrays to be used as input tensors in TensorFlow, a usefull knowledge to be replicated in future projects when dealing with our own data.\n",
    "\n",
    "The first thing to do is to have EMNIST package installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZjsvZN-VZNDL"
   },
   "outputs": [],
   "source": [
    "# install emnist package\n",
    "! pip install emnist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AFQ5CMyU3CXx"
   },
   "source": [
    "Next we import the required libraries, and from EMNIST we import functions extract_training_samples() and extract_test_samples(), as their names suggest, they are used to extract the training and test sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "executionInfo": {
     "elapsed": 3020,
     "status": "ok",
     "timestamp": 1655136463860,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "baVEvaPCbTHS"
   },
   "outputs": [],
   "source": [
    "# import packages\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "from emnist import extract_training_samples, extract_test_samples\n",
    "\n",
    "# set Numpy precision for printing\n",
    "np.set_printoptions(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "APlOKtwL6Aqt"
   },
   "source": [
    "### Loading Data\n",
    "\n",
    "The EMNIST package contains the complete EMNIST Database, which includes different subsets of handwritten digits and letters, for details please check The EMNIST Dataset official webpage at https://www.nist.gov/itl/products-and-services/emnist-dataset.\n",
    "\n",
    "To extract the desired subset samples, we use the respective function, training or test, having the subset name as parameter, e.g. for MNIST training subset use extract_training_samples('mnist').\n",
    "\n",
    "The EMNIST/MNIST subset contains a collection of 60,000 training and 10,000 test images with shape (28,28) in grayscale format, all of them with their respective labels. When extracted the data is loaded into two Numpy arrays, one receives images data and the other labels data. The images are labeled with one out of ten different classes, from 0 to 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1162,
     "status": "ok",
     "timestamp": 1655136469167,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "38NttYkWZRBX",
    "outputId": "eb96b94f-57fa-4458-c1cf-45067838b5b9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training \n",
      "  Image shape: (60000, 28, 28) \n",
      "  Label shape: (60000,)\n",
      "\n",
      "Test \n",
      "  Image shape: (10000, 28, 28) \n",
      "  Label shape: (10000,)\n",
      "\n",
      "Classes\n",
      "  10\n"
     ]
    }
   ],
   "source": [
    "# load training and test data from emnist/mnist\n",
    "img_train, lbl_train = extract_training_samples('mnist')\n",
    "img_test, lbl_test = extract_test_samples('mnist')\n",
    "\n",
    "# print objects information\n",
    "print('Training',\n",
    "      '\\n  Image shape:', img_train.shape,\n",
    "      '\\n  Label shape:', lbl_train.shape)\n",
    "print('\\nTest',\n",
    "      '\\n  Image shape:', img_test.shape,\n",
    "      '\\n  Label shape:', lbl_test.shape)\n",
    "print('\\nClasses\\n ', len(np.unique(lbl_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "atmGxqCE_-gU"
   },
   "source": [
    "Let's have a quick look at one training and one test sample, their labels are defined as image title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 216
    },
    "executionInfo": {
     "elapsed": 1093,
     "status": "ok",
     "timestamp": 1655136473602,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "iZMGp_z_7ddc",
    "outputId": "9ac43f31-a6e2-43ba-ab62-2c1c3cbaef3e"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAADHCAYAAAAAoQhGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAU8ElEQVR4nO3df4zV1ZnH8c8DTquAhB8uhFIUrNSEVouGsKZLW7daW1yLYjaIbaxJm2KpGtmtWtKy/qJt2Gyta1ZiQqsRK9q1LSgxrbvUYGErdkt/UZDVyk/BgRGUFhBQ4Nk/5pKMnufL3Jn781zer8TM3GfO9875zjzz+OV7zvccc3cBAPLTp9EdAAD0DgUcADJFAQeATFHAASBTFHAAyBQFHAAyRQEHgExRwJuYmY01s4Nm9kij+wJUi5lNN7P1ZrbfzDaY2cca3adcndToDuC45kv6TaM7AVSLmX1K0r9KukrS/0oa0dge5Y0C3qTMbLqkPZKek3RWg7sDVMudku5y9+dLr7c3sjO54xZKEzKzgZLukvTPje4LUC1m1lfSBEl/Y2Yvm9k2M7vPzE5pdN9yRQFvTnMlPeDu2xrdEaCKhktqk/SPkj4mabyk8yTNaWSnckYBbzJmNl7SxZLuaXRfgCo7UPr4H+7e7u67JH1P0qUN7FPWuAfefC6UNFrSVjOTpAGS+prZOHc/v4H9Airi7m+Y2TZJXZdAZTnUChjLyTYXM+snaWCX0M3qLOgz3f21hnQKqBIzu0vSZEn/IOltSUslPevu/9LQjmWKK/Am4+5vSnrz2Gsz2yfpIMUbLWKupNMkvSTpoKTHJX27oT3KGFfgAJApBjEBIFMUcADIFAUcADJFAQeATFVUwM3sM2b2Yumx2NnV6hTQaOQ2ctDrWSildQ1ekvQpSdvUuWre1e7+wnGOYcoLasrdrdL3ILfRjKLcruQKfKKkl919o7u/JelHki6v4P2AZkFuIwuVFPCRkl7p8npbKfYOZjbDzFab2eoKvhdQT+Q2slDzJzHdfYGkBRL/zERrIbfRaJUU8O2SRnV5/X6xOPsJo2/fvknsyJEjDehJTZDbmYryUmqp3HyHSm6h/EbSWDMbY2bvkTRdnQvTALkjt5GFXl+Bu/thM7tB0n9J6ivpQXdfV7WeAQ1CbiMXdV3MivuEraNZb6FUYxphb5DbzaGVb6FUexohAKCBKOAAkCk2dJDUp0/8/7FPfOITSax///5h22XLliWxQ4cOVdaxOuvXr18Smzp1atj2xhtvTGL33ntv2Hbx4sVJLLefDZrP6NGjk9hFF10Utn322WeTWE9uq2zZsiWMN3o/Ba7AASBTFHAAyBQFHAAyRQEHgEwxiClp6NChYXzmzJlJ7Kyzzgrbbt26NYmtWbOmso7VyMknnxzGb7nlliT2hS98IWwbDSDddNNNYdtVq1Ylsc2bNxd3ECjDtddem8Rmz46Xbo/yrWgAcu/evUnsqquuKvt964krcADIFAUcADJFAQeATFHAASBTFHAAyBSzUCQNGTIkjH/kIx9JYmeeeWbYds6cOUls+vTpYdujR4/2oHeViWac3HrrrWHb2267LYmZxYv7RY8h33///WHb9vb243UROK6iWVPnn39+EtuzZ0/Y9p577klib7zxRtj2i1/8YhI76aTmLJVcgQNApijgAJApCjgAZIoCDgCZqujOvJltlrRX0hFJh919QjU6VUvR2t/jxo0L2w4aNCiJFW3ZdM4555T1vaT6DmK+733vS2JTpkwJ2xYNWEZ2796dxH7+85+HbXNc+zvH3G4F0d/Xpz/96bDtJz/5ySS2fv36sO0vfvGLJFa0xvf27duTWJTvzaAaQ6t/7+67qvA+QLMht9HUuIUCAJmqtIC7pP82s9+a2YxqdAhoEuQ2ml6lt1Amuft2MxsmaZmZ/Z+7r+jaoJT8/AEgN+Q2ml5FV+Duvr30sUPSEkkTgzYL3H0Cg0DICbmNHPT6CtzM+kvq4+57S59fIumuqvWsRk455ZQkNmnSpLDt4MGDy37fffv29bpPtXTNNdcksaJZN5Fdu+IxvK9//etJrKOjo/yONbFcczsn0d+hJF155ZVJ7K674h/9zp07k9hXv/rVsO2mTZuSWNGGDs8991wYb0aV3EIZLmlJaerZSZIedfenq9IroLHIbWSh1wXc3TdKSld7AjJHbiMXTCMEgExRwAEgU825yG0NXXHFFWXFpHgN4IMHD4Ztn3rqqSR2+PDhHvau94rWK77sssuSWNH6ypGtW7eG8ZUrV5b9HjixtbW1JbFp06aFbaN19UeNGhW2feSRR5JY0S7xRQOWueMKHAAyRQEHgExRwAEgUxRwAMgUBRwAMtWys1CGDRsWxu++++6y20bWrl0bxn/4wx+W/R61ULQZw8CBA8t+jwMHDiSxxx57LGxbNDsFJ7ZoQ4Zo6YZ58+aFxw8dOjSJ/fKXvwzbRo/YN+vGC7XCFTgAZIoCDgCZooADQKYo4ACQqZYYxHzve9+bxCZPnhy2jQZJihw5ciSJzZ8/P2wb7WRdK9GA5RlnnBG27cma5u3t7Uls+fLlYdt6LhOA5jN8+PAwPnfu3CQ2ZcqUJNanT3ztuGjRoiR25513hm2LHps/kXAFDgCZooADQKYo4ACQKQo4AGSq2wJuZg+aWYeZre0SG2Jmy8zsz6WP5Y+UAU2C3EbuypmF8pCk+yQ93CU2W9Iz7j7PzGaXXqfblNdJtOD77Nmzw7bRo75FduzYkcR+//vfh21HjBiRxP7yl7+EbaPNF4YMGZLEinaPHz9+fBKbNGlS2Pa0004L45Ennngiia1fv77s4zP0kJo8t5tBtNTEV77ylbBttDlKNPPrO9/5Tnj8fffdl8Q6Ojq66+IJq9srcHdfIen1d4Uvl7Sw9PlCSfGWNkATI7eRu97eAx/u7scmDe+QFE8KBfJDbiMbFT/I4+5uZoUbzpnZDEkzKv0+QL2R22h2vb0C32lmIySp9LHwJpW7L3D3Ce4+oZffC6gnchvZ6O0V+FJJ10qaV/r4ZNV6VCVFj+r2RDR48+Mf/zhsu2/fviS2bt26sG3//v2T2Ic//OEkNmjQoPD4U089NYlV49H2aND17bffrvh9M9P0uV0rJ598chi/8cYbk9iXv/zlsG00GL9x48Yk9sADD4TH79y583hdxLuUM43wMUmrJJ1tZtvM7EvqTO5PmdmfJV1ceg1khdxG7rq9Anf3qwu+dFGV+wLUFbmN3PEkJgBkigIOAJmigANApsy9cJpr9b/ZcebUViIa+b7uuuvCtrfddlsSizaEKFK0+3s9f46VOnjwYBi/4IILktiaNWtq3Z2qcvf4F1RjtcrtWhk9enQSizZjkKTPfe5zSezQoUNh2yVLliSxaEOGDRs2hMdHf19Hjx4N20Z/czn9HfZUlNtcgQNApijgAJApCjgAZIoCDgCZaolBzMiAAQPC+NKlS5NYNKAjxTtvFz1uHCka8IwGZXbv3p3EitZBjnagLzrfSNHAZDSIWTTg2awYxHynogH6G264IYndcccdYdtoWYqnn346bBvF33rrrSS2f//+8PhoXf1t27aFbaPlK1599dWw7UsvvZTEclsmgkFMAGghFHAAyBQFHAAyRQEHgExVvCNPs4oGOCTpkksuSWJtbW1h22hA5eKLLw7bRk+mFW0oHK2P/Ktf/SqJRU+YStIf//jHMB6J1gl/8sl4ievcBizxTtGAZTRYKUmXXXZZWcdL0k9+8pMkVjQAePPNNyex6O9gy5Yt4fEjR44M45FocHTTpk1h22hN86LB/Jye5uQKHAAyRQEHgExRwAEgUxRwAMhUOXtiPmhmHWa2tkvsDjPbbmZ/KP13aW27CVQfuY3clTML5SFJ90l6+F3xe9z9u1XvUY1FszKKdnTvyW7akZ48Sh/FBg8e3KP3jUSzBfbs2VP28S3uIbVQbo8aNSqJRevfS/GMkwMHDoRtx4wZk8TOPffcsG201MSRI0eS2MCBA8Pjo9wcNGhQ2DaasVI082vWrFlJ7JZbbgnb7tq1K4w3o26vwN19haTX69AXoK7IbeSuknvgN5jZmtI/Q+NLRSBP5Day0NsCfr+kD0gaL6ld0t1FDc1shpmtNrPVvfxeQD2R28hGrwq4u+909yPuflTS9yVNPE7bBe4+wd0n9LaTQL2Q28hJrx6lN7MR7t5eejlV0trjtW8l0YBMNZx0UvqriB53Lmpb5PHHH09iixYtKr9jJ5gccrto/foFCxYksX79+oVto8fIhw0bFradODH9f9hrr70Wto3ybf78+Uls3bp14fHRhIKiwfwf/OAHSezSS+NJQx/96EeT2NChQ8O2OQ1idlsJzOwxSRdKOs3Mtkm6XdKFZjZekkvaLCneAh5oYuQ2ctdtAXf3q4Nw+XPpgCZFbiN3PIkJAJmigANApijgAJCplt3QITenn356Evv4xz9e9vFFywFEMwCKZhAgD0WzkKJHy998882wbbR7e9Hslihf5syZE7ZdunRpWcf3ZNOEonzdsGFDEouWpJDi2TjR35wkvfjii2X3rdG4AgeATFHAASBTFHAAyBQFHAAyxSBmA0Trec+cOTOJTZ48uez3fP31eFXUaPfvnHbdRmVOOeWUMB4NkA8YMCBsu3jx4iQWDVZKUkdHRw96l4rWEz/77LPDtlOmTEliffrE16TRQGh7e3vQMi9cgQNApijgAJApCjgAZIoCDgCZooADQKaYhdIAZ5xxRhK78sork1jRI9PRphIrV64M2xbNTkHriR4jb2trC9tGubV79+6wbfTY/BtvvBG2jb5ftCFDNOtKkqZOnZrExo4dG7aNZpHMnTs3bLtw4cIkFs3Qyg1X4ACQKQo4AGSKAg4AmaKAA0CmytnUeJSkhyUNV+dGrwvc/V4zGyLpPyWNVufmr9PcPR7ZOEEVPcZ8/fXXJ7ExY8YkseiRe0l65ZVXkti8efPCttGAJzrlmtt79uwJ488//3wSK3oMPVKUb5///OeTWNH689F7nHPOOUmsaJmIaHC1aMD09ttvT2JLliwJ2xati567cq7AD0v6mruPk3SBpOvNbJyk2ZKecfexkp4pvQZyQm4ja90WcHdvd/fflT7fK2m9pJGSLpd0bG7OQklX1KqTQC2Q28hdj+aBm9loSedJ+rWk4e5+bCLmDnX+MzQ6ZoakGb3vIlB75DZyVPYgppkNkPRTSbPc/a9dv+ad65OGa5S6+wJ3n+DuEyrqKVAj5DZyVVYBN7M2dSb4Inc/tjjwTjMbUfr6CEmVLQQMNAC5jZxZd4v7W+ew8kJJr7v7rC7xf5O0293nmdlsSUPc/dZu3uuE2kng3HPPDeMPP/xw2W0j0a7Zn/3sZ8O2L7/8ctnv2wrcPZ5KEcg1t4tmi0Q5tGrVqrBttHFCkWjGyd69e8O2GzduTGJnnnlmEtu0aVN4/PLly5PYc889F7b92c9+lsQOHToUtm0FUW6Xcw/87yRdI+lPZvaHUuwbkuZJetzMviRpi6Rp1eooUCfkNrLWbQF39/+RVHRVc1F1uwPUD7mN3PEkJgBkigIOAJliPfAqiXbDjtZRlqQPfehDZb1n0YDMt771rSQWPV6P1lQ08eCFF15IYkV5cfrppyex6FF8Serfv38Se+qpp8K2jz76aBI7ePBgEtuxY0d4fNEj+ohxBQ4AmaKAA0CmKOAAkCkKOABkigIOAJliFkqVRLNQooXsJalv375lvef27dvD+BNPPJHEWvkRYpSnu2Uxuop2dP/mN78Ztn311VeTWNEskmjGCWqHK3AAyBQFHAAyRQEHgExRwAEgUwxi1tC+ffvCeDTYFO0eX7TD9v79+yvrGE4YRY+mRwPpbW1tYdvNmzdXs0uoIq7AASBTFHAAyBQFHAAyRQEHgEx1W8DNbJSZLTezF8xsnZndVIrfYWbbzewPpf8urX13geoht5G7cmahHJb0NXf/nZmdKum3Zras9LV73P27tetePqLR/sWLF4dt+/Xrl8SiGSsrVqyovGM4npbK7SgHp02L92P+4Ac/mMSiTR6keJmIo0eP9rB3qIVyNjVul9Re+nyvma2XNLLWHQNqjdxG7np0D9zMRks6T9KvS6EbzGyNmT1oZoMLjplhZqvNbHVFPQVqiNxGjsou4GY2QNJPJc1y979Kul/SBySNV+dVzN3Rce6+wN0nuPuEKvQXqDpyG7kqq4CbWZs6E3yRuy+WJHff6e5H3P2opO9Lmli7bgK1QW4jZ9bdGsJmZpIWSnrd3Wd1iY8o3UOUmf2TpL919+ndvFf5Cxa3gGjw53jxd2OH7p5zdyu37Ymc21EOdv44UtEyD6i/KLfLKeCTJK2U9CdJx4aevyHpanX+E9MlbZZ03bGkP857ZZXklaKA118PC/gJm9sU8Pz0qoBXU25JXikKeP31pIBXU265TQHPT5TbPIkJAJmigANApijgAJApNnSooaLHjXkMGY1GDrYGrsABIFMUcADIFAUcADJFAQeATNV7EHOXpC2lz08rvW41nFfjnNHA730st3P4OfVWq55bDucV5nZdn8R8xzc2W92Kq7hxXie2Vv45teq55Xxe3EIBgExRwAEgU40s4Asa+L1rifM6sbXyz6lVzy3b82rYPXAAQGW4hQIAmap7ATezz5jZi2b2spnNrvf3r6bShrcdZra2S2yImS0zsz+XPoYb4jYzMxtlZsvN7AUzW2dmN5Xi2Z9bLbVKbpPX+ZxbXQu4mfWVNF/SZEnjJF1tZuPq2Ycqe0jSZ94Vmy3pGXcfK+mZ0uvcHJb0NXcfJ+kCSdeXfk+tcG410WK5/ZDI6yzU+wp8oqSX3X2ju78l6UeSLq9zH6rG3VdIev1d4cvVuc+iSh+vqGunqsDd2939d6XP90paL2mkWuDcaqhlcpu8zufc6l3AR0p6pcvrbaVYKxneZf/EHZKGN7IzlTKz0ZLOk/Rrtdi5VVmr53ZL/e5bJa8ZxKwh75zik+00HzMbIOmnkma5+1+7fi33c0Pv5f67b6W8rncB3y5pVJfX7y/FWslOMxshSaWPHQ3uT6+YWZs6k3yRuy8uhVvi3Gqk1XO7JX73rZbX9S7gv5E01szGmNl7JE2XtLTOfai1pZKuLX1+raQnG9iXXrHO7ckfkLTe3b/X5UvZn1sNtXpuZ/+7b8W8rvuDPGZ2qaR/l9RX0oPu/u26dqCKzOwxSReqczWznZJul/SEpMclna7O1emmufu7B4SamplNkrRS0p8kHdt76xvqvF+Y9bnVUqvkNnmdz7nxJCYAZIpBTADIFAUcADJFAQeATFHAASBTFHAAyBQFHAAyRQEHgExRwAEgU/8PI00BloMewsUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show sample images with labels\n",
    "# training data\n",
    "plt.subplot(1,2,1)\n",
    "plt.imshow(img_train[0], cmap='gray')\n",
    "plt.title(lbl_train[0])\n",
    "\n",
    "# test data\n",
    "plt.subplot(1,2,2)\n",
    "plt.imshow(img_test[-1], cmap='gray')\n",
    "plt.title(lbl_test[-1]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QnOLHhINA9tF"
   },
   "source": [
    "### Creating TensorFlow Datasets\n",
    "\n",
    "TensorFlow datasets are used to feed the network (CNN), in our case we should import our Numpy arrays into datasets.\n",
    "\n",
    "The standard tensor shape for feeding images to TensorFlow models has four dimensions (batch_size, heigh, width, no_channels). In grayscale images the pixels value defines its intensity in a range of 256 shades of gray from 0 (black) to 255 (white), the number of channels dimension is not used since there is only one of them.\n",
    "\n",
    "Images in RGB format use the red, green and blue channels to create color palettes, they require three channels. In this case the pixel value in each channel defines the filter intensity, also fom 0 to 255, and the combination of the three channels defines the color, so there are 16 milion possible colors for each pixel ($256^3 = 16.777.216$).\n",
    "\n",
    "MNIST images are in grayscale without the channels dimension, we need to expand them with the fourth dimension to be complient with TensorFlow standard. After that we import images and labels into TensorFlow datasets setting them as a tupple (image, label), which defines the dataset elements.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1226,
     "status": "ok",
     "timestamp": 1655136479549,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "EFbCmnXjfFTH",
    "outputId": "d96ec80b-77ad-4e1f-8036-fbb07b07fd33"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training \n",
      "  Object type: <class 'tensorflow.python.data.ops.dataset_ops.TensorSliceDataset'> \n",
      "  Image spec: TensorSpec(shape=(28, 28, 1), dtype=tf.uint8, name=None) \n",
      "  Label spec: TensorSpec(shape=(), dtype=tf.uint8, name=None)\n",
      "\n",
      "Test \n",
      "  Object type: <class 'tensorflow.python.data.ops.dataset_ops.TensorSliceDataset'> \n",
      "  Image spec: TensorSpec(shape=(28, 28, 1), dtype=tf.uint8, name=None) \n",
      "  Label spec: TensorSpec(shape=(), dtype=tf.uint8, name=None)\n"
     ]
    }
   ],
   "source": [
    "# add 4th dimension to image arrays (TF standard)\n",
    "img_train = np.expand_dims(img_train, axis=3)\n",
    "img_test = np.expand_dims(img_test, axis=3)\n",
    "\n",
    "# create TF Dataset from Numpy Arrays\n",
    "data_train = tf.data.Dataset.from_tensor_slices((img_train, lbl_train),)\n",
    "data_test = tf.data.Dataset.from_tensor_slices((img_test, lbl_test))\n",
    "\n",
    "# print objects information\n",
    "print('Training',\n",
    "      '\\n  Object type:', type(data_train),\n",
    "      '\\n  Image spec:', data_train.element_spec[0],\n",
    "      '\\n  Label spec:', data_train.element_spec[1])\n",
    "\n",
    "print('\\nTest',\n",
    "      '\\n  Object type:', type(data_test),\n",
    "      '\\n  Image spec:', data_test.element_spec[0],\n",
    "      '\\n  Label spec:', data_test.element_spec[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OJWvD_qmvNUl"
   },
   "source": [
    "### Preparing Data\n",
    "\n",
    "The data type was originally defined as integer *tf.uint8*, which comprehends the grayscale range between 0 and 255 ($2^8=256$).\n",
    "\n",
    "The images will be casted to float *tf.float32*, this allows higher precision during the calculations. After that they will be rescaled to the range between 0.0 and 1.0, dividing the pixels values by 255.0, their maximum possible value, this helps to avoid overflow during weights calculations.\n",
    "\n",
    "The labels will be casted to integer *tf.int32*, but there is no need for rescaling labels.\n",
    "\n",
    "All of this is done by a user defined cast and rescale function, which receives image and label as parameters and returns the processed data. It is applied to the dataset using Python map() function, it calls our cast_rescale() function passing the dataset elements (image, label) as parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "executionInfo": {
     "elapsed": 393,
     "status": "ok",
     "timestamp": 1655136484348,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "8xVFSuHox1eL"
   },
   "outputs": [],
   "source": [
    "def cast_rescale(image, label):\n",
    "    '''Cast and rescale the input'''\n",
    "    return tf.cast(image, tf.float32)/255., tf.cast(label, tf.int32)\n",
    "\n",
    "# cast and rescale data\n",
    "data_train = data_train.map(cast_rescale, num_parallel_calls=tf.data.AUTOTUNE)\n",
    "data_test = data_test.map(cast_rescale, num_parallel_calls=tf.data.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jxkv6I3tz-UA"
   },
   "source": [
    "### Creating Data Iterators\n",
    "\n",
    "One way to run model training sessions is using data batches, slices of the dataset fed to the network on an iterative way. After each batch processing the model parameters are updated using a gradient decent method.\n",
    "\n",
    "Data iterators are set with the batch selection policy, and they are handy because each time a batch is required they are provided accordingly without the need of manual selection. Data iterators can be created by user defined classes and/or functions, but TensorFlow already provides the tools we need to set them.\n",
    "\n",
    "We will build our data iterators first caching the data into memory, then defining the shuffle method, in our case we set its buffer size to the number of entries in our dataset, doing this each training epoch makes use of the whole dataset. The buffer size can be set to half or a third of dataset size, then the dataset is used partially in each epoch, taking each half or third portion of it sequentially epoch after epoch.\n",
    "\n",
    "We also set it to reshuffle at each iteration, meaning that for each epoch the batches are redefined again. It is also possible to set the shuffling seed for reproducibility.\n",
    "\n",
    "The shuffling step is done only for training dataset, it is not required for tests.\n",
    "\n",
    "Then we set the batch size, which is of relevant importance. As mentioned earlier, after each batch processing the model parameters are updated by a gradient decent method. The batch size also defines the number of entries used to calculate the gradient, it should be sufficient to generate a robust gradient and to allow the model convergence, but it should also be sufficient to allow the iterative process with enough number of batches, having more batches it is possible to update parameters more frequently.\n",
    "\n",
    "Finally we define how TensorFlow should prefetch data, which is how many elements to prepare in advance to feed the network while it is processing the current one. It avoids waiting time to read and feed the data, but it requires more memory, when set to tf.data.AUTOTUNE it is dynamically adjusted.\n",
    "\n",
    "For details about TensorFlow Datasets please check https://www.tensorflow.org/api_docs/python/tf/data/Dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 415,
     "status": "ok",
     "timestamp": 1655136494705,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "WRXzJpqXzJAu",
    "outputId": "2ba8ff43-0ad9-493a-83a5-1d9cdef85c62"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training \n",
      "  Object type: <class 'tensorflow.python.data.ops.dataset_ops.PrefetchDataset'> \n",
      "  Image spec: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name=None) \n",
      "  Label spec: TensorSpec(shape=(None,), dtype=tf.int32, name=None)\n",
      "\n",
      "Test \n",
      "  Object type: <class 'tensorflow.python.data.ops.dataset_ops.PrefetchDataset'> \n",
      "  Image spec: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name=None) \n",
      "  Label spec: TensorSpec(shape=(None,), dtype=tf.int32, name=None)\n"
     ]
    }
   ],
   "source": [
    "# define the batch size\n",
    "batch_size = 64\n",
    "\n",
    "# prepare training data for input\n",
    "data_train = data_train.cache()\n",
    "data_train = data_train.shuffle(buffer_size=len(data_train),\n",
    "                                seed=1532,\n",
    "                                reshuffle_each_iteration=True)\n",
    "data_train = data_train.batch(batch_size)\n",
    "data_train = data_train.prefetch(buffer_size=tf.data.AUTOTUNE)\n",
    "\n",
    "# prepare test data for input\n",
    "data_test = data_test.cache()\n",
    "data_test = data_test.batch(batch_size)\n",
    "data_test = data_test.prefetch(buffer_size=tf.data.AUTOTUNE)\n",
    "\n",
    "# print objects information\n",
    "print('Training',\n",
    "      '\\n  Object type:', type(data_train),\n",
    "      '\\n  Image spec:', data_train.element_spec[0],\n",
    "      '\\n  Label spec:', data_train.element_spec[1])\n",
    "\n",
    "print('\\nTest',\n",
    "      '\\n  Object type:', type(data_test),\n",
    "      '\\n  Image spec:', data_test.element_spec[0],\n",
    "      '\\n  Label spec:', data_test.element_spec[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e7qS8F5f0X0N"
   },
   "source": [
    "### Understanding Convolution Operation\n",
    "\n",
    "For the task of image classification we use convolutional neural networks (CNN), these networks are able to capture the correlation between the image pixels thus extract features to understand its content. It is performed by a convolution operation, performed by a sliding window, also called kernel, over the image. This operation outputs a new tensor with the image pixels cross-correlation, this output is called features map.\n",
    "\n",
    "The operation starts from the top left position, first moving the kernel to the right, then to the bottom of the image. For each position the kernel passes it operates the calculation between its elements and the elements of the overlaped portion of the image.\n",
    "\n",
    "It is not a formal way to describe it, but the example below helps to understand it. Suppose we have an input image $X$ with shape (3,3,1) and a convolutional kernel $K$ with shape (2,2,1):\n",
    "\n",
    "$X = \\begin{bmatrix} 0 & 1 & 2 \\\\ 3 & 4 & 5 \\\\ 6 & 7 & 8 \\end{bmatrix},\n",
    "K = \\begin{bmatrix} 1 & 3 \\\\ 5 & 7 \\end{bmatrix}$\n",
    "\n",
    "First we set $K$ over the top left position of $X$ and perform the elementwise multiplication:\n",
    "\n",
    "$\\begin{bmatrix} 1 \\times 0 & 3 \\times 1 \\\\ 5 \\times 3 & 7 \\times 4 \\end{bmatrix} = \n",
    "\\begin{bmatrix} 0 & 3 \\\\ 15 & 28 \\end{bmatrix}$\n",
    "\n",
    "Next we sum up the elements: $0 + 3 + 15 + 28 = 46$\n",
    "\n",
    "When sliding $K$ over $X$ from left to right and from top to bottom, the outputs are:\n",
    "\n",
    "$1 \\times 0 + 3 \\times 1 + 5 \\times 3 + 7 \\times 4 = 46$\n",
    "\n",
    "$1 \\times 1 + 3 \\times 2 + 5 \\times 4 + 7 \\times 5 = 62$\n",
    "\n",
    "$1 \\times 3 + 3 \\times 4 + 5 \\times 6 + 7 \\times 7 = 94$\n",
    "\n",
    "$1 \\times 4 + 3 \\times 5 + 5 \\times 7 + 7 \\times 8 = 110$\n",
    "\n",
    "The features map, output of the convolution of $K$ over $X$, is then:\n",
    "\n",
    "$O = \\textrm{Conv}(X, K) = \\begin{bmatrix} 46 & 62 \\\\ 94 & 110 \\end{bmatrix}$\n",
    "\n",
    "In our example the slide step had size equals to one, this means we moved one pixel each time, this is called stride. Strides can be set to higher values than one.\n",
    "\n",
    "The features map has smaller dimensions compared to the input image, to avoid this effect we can expand the input with paddings. They are pixels with value zero added to the border of the original image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cVbiKSnYZSCZ"
   },
   "source": [
    "### Convolution with Multi-channel Objects\n",
    "\n",
    "The convolution kernel must always have the same number of channels as the image to be processed, the operation is performed per channel. Let's see an example with a two channels image $X$ with shape (3,3,2) and kernel $K$ (2,2,2).\n",
    "\n",
    "$X = \\left[{\\begin{bmatrix} 0 & 1 & 2 \\\\ 3 & 4 & 5 \\\\ 6 & 7 & 8 \\end{bmatrix}\\;\n",
    "\\begin{bmatrix} 2 & 4 & 6 \\\\ 1 & 3 & 5 \\\\ 7 & 8 & 9 \\end{bmatrix}}\\right]$\n",
    "\n",
    "$K = \\left[{\\begin{bmatrix} 1 & 3 \\\\ 5 & 7 \\end{bmatrix}\\;\n",
    "\\begin{bmatrix} 2 & 4 \\\\ 6 & 8 \\end{bmatrix}}\\right]$\n",
    "\n",
    "For channel 1 we have: $O_1 = \\textrm{Conv}(X_1,K_1) = \\begin{bmatrix} 46 & 62 \\\\ 94 & 110 \\end{bmatrix}$\n",
    "\n",
    "For channel 2 we have: $O_2 = \\textrm{Conv}(X_2,K_2)  = \\begin{bmatrix} 50 & 90 \\\\ 113 & 146 \\end{bmatrix}$\n",
    "\n",
    "Next to find the features map, output of the convolution of $K$ over $X$, we sum up elementwise the channels operation outputs:\n",
    "\n",
    "$O = O_1 + O_2 = \\begin{bmatrix} 46+50 & 62+90 \\\\ 94+113 & 110+146 \\end{bmatrix} = \n",
    "\\begin{bmatrix} 96 & 152 \\\\ 207 & 256 \\end{bmatrix}$\n",
    "\n",
    "No matter how many channels the image contains, the convolution operation outputs a single features map.\n",
    "\n",
    "Usually in convolutional neural networks several features maps are used to have different characteristics extracted from the input image, in this case the convolution operator should not contain a single kernel, but a group of kernels. \n",
    "\n",
    "When dealing with tensors, this can be done just including one dimension at its first position, instead of shape (heigh, width, no_channels), the kernel should have shape (no_filters, height, width, no_channels). In this case for each filter one convolution operation is performed providing one features map each.\n",
    "\n",
    "To output three features maps in our example above we would have kernel $K$ with shape (3,2,2,2).\n",
    "\n",
    "$K = \\left[\n",
    "\\left[\\begin{bmatrix} 1 & 3 \\\\ 5 & 7 \\end{bmatrix}\\; \\begin{bmatrix} 2 & 4 \\\\ 6 & 8 \\end{bmatrix}\\right]\\\\\n",
    "\\left[\\begin{bmatrix} 0 & 9 \\\\ 4 & 5 \\end{bmatrix}\\; \\begin{bmatrix} 5 & 3 \\\\ 0 & 4 \\end{bmatrix}\\right]\\\\ \n",
    "\\left[\\begin{bmatrix} 2 & 3 \\\\ 6 & 5 \\end{bmatrix}\\; \\begin{bmatrix} 1 & 7 \\\\ 9 & 6 \\end{bmatrix}\\right]\n",
    "\\right]$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G4U2e5JrLkUA"
   },
   "source": [
    "### Pooling Operation\n",
    "\n",
    "Pooling operation is used to aggregate information from the input at the same time to reduce its dimensionality, usually pooling layers are allocated after a convolutional layer.\n",
    "\n",
    "There are two types of pooling layers, maximum pooling and average pooling. The operation is also done sliding a window, the pooling window, over the intput, but the operation is the extraction of the higher value, in case of max pooling, or the calculation of the average value, in case of average pooling.\n",
    "\n",
    "Pooling windows do not hold any value, only their shape or size.\n",
    "\n",
    "Suppose we have the same input $X$ with shape (3, 3) and a max pooling window $M$ with shape (2, 2).\n",
    "\n",
    "$X = \\begin{bmatrix} 0 & 1 & 2 \\\\ 3 & 4 & 5 \\\\ 6 & 7 & 8 \\end{bmatrix},\n",
    "M = \\begin{bmatrix}  m_{11} & m_{12}  \\\\ m_{21}  & m_{22} \\end{bmatrix}$\n",
    "\n",
    "When $M$ is positioned over the top left position of $X$ it operates the function $\\textrm{max}(0,1,3,4)=4$.\n",
    "\n",
    "When sliding $M$ over $X$ from left to right and from top to bottom, the outputs are:\n",
    "\n",
    "$\\textrm{max}(0, 1, 3, 4)=4$\n",
    "\n",
    "$\\textrm{max}(1, 2, 4, 5)=5$\n",
    "\n",
    "$\\textrm{max}(3, 4, 6, 7)=7$\n",
    "\n",
    "$\\textrm{max}(4, 5, 7, 8)=8$\n",
    "\n",
    "The output of the max pooling operation of $M$ over $X$ is then:\n",
    "\n",
    "$O = \\textrm{MaxPool}(X,M) = \\begin{bmatrix}  4 & 5  \\\\ 7  & 8 \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dL77tJxnzUhw"
   },
   "source": [
    "### Defining Model\n",
    "\n",
    "We will build a TensorFlow Sequential model, which holds its layers in sequence and the output from a layer is passed as input to the next.\n",
    "\n",
    "Our model will process the input image gradually increasing the number of feature maps (filters) using convolutional layers, and reducing the dimensionality using pooling layers, from input shape (28,28,1) to (1,1,256). Convolutional neural networks extract images features on local perspective (pixel level) within intial layers, and on global perpective (image level) within deep layers.\n",
    "\n",
    "After extracting image features we reduce the number of feature maps to the same number of classes in our problem, they are 10, this is also achieved with a convolutional layer with output shape (1,1,10). Finally we use a flatten layer to reshape it to a tensor shape (1,10).\n",
    "\n",
    "The output of the network contains the odds for the input image to belong to each of the ten classes, the class with the higher odds is considered as the best prediction for the input image.\n",
    "\n",
    "We will also use batch normalization between convolutional and pooling layers, they normalize the output of the convolutional layers helping to avoid overfitting, they also allow the usage of more aggressive learning rates.\n",
    "\n",
    "The main portion of the network has a standard layout:\n",
    "1. Convolution\n",
    "1. Batch normalization\n",
    "1. Max pooling\n",
    "\n",
    "To define the model we will have a function which receives as parameter a tupple with the layers configuration (number of output filters, convolution kernel size, pooling window size). It will add the layers in a sequence according to the given tupple, then it adds the flatten layer and returns the model.\n",
    "\n",
    "TensorFlow does not require to set the layer input shape, as it infers by itself, for convolutional layers only the output number of feature maps (filter) and the kernel size are required. Optionally strides and padding can be also set to manage the output dimensionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "executionInfo": {
     "elapsed": 322,
     "status": "ok",
     "timestamp": 1655136503285,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "bpaW8DbAFoVB"
   },
   "outputs": [],
   "source": [
    "def model(layers_config):\n",
    "    '''Defines a TensorFlow Sequential model'''\n",
    "    model = keras.models.Sequential(name='DigitClassification')\n",
    "\n",
    "    for i, (filters, kernel, pool) in enumerate(layers_config, start=1):\n",
    "        # add convolutional layer\n",
    "        model.add(layers.Conv2D(filters=filters, kernel_size=kernel,\n",
    "                                strides=1, padding='same',\n",
    "                                name='conv_'+str(i)))\n",
    "        # add batch normalization layer\n",
    "        model.add(layers.BatchNormalization(name='bnrm_'+str(i)))\n",
    "        # add activation function layer\n",
    "        model.add(layers.Activation('relu', name='actv_'+str(i)))\n",
    "        \n",
    "        if pool > 0:\n",
    "            # add max pooling layer\n",
    "            model.add(layers.MaxPool2D(pool_size=pool, strides=2,\n",
    "                                       name='pool_'+str(i)))\n",
    "    \n",
    "    # add flatten layer\n",
    "    model.add(layers.Flatten(name='flatten'))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A-zkECIkUuRP"
   },
   "source": [
    "There will be four convolutional layers for feature extraction, each of them followed by the batch normalization and activation. Then the classes odds calculation and the flatten layers.\n",
    "\n",
    "1. Conv(3x3), 32 filters | Batch normalization | Activation | MaxPool(2x2)\n",
    "1. Conv(3x3), 64 filters | Batch normalization | Activation | MaxPool(2x2)\n",
    "1. Conv(3x3), 128 filters | Batch normalization | Activation | MaxPool(2x2)\n",
    "1. Conv(3x3), 256 filters | Batch normalization | Activation | MaxPool(2x2)\n",
    "1. Conv(1x1), 10 filters | Batch normalization | Activation\n",
    "1. Flatten\n",
    "\n",
    "Bellow we define the layers configuration, notice that the last position (10,1,0) is the odds calculation, the convolution kernel has size one and there is no pooling. \n",
    "\n",
    "Next we instantiate the model with defined layers configuration and build it with the expected input tensor shape. After that we use model summary to check its architecure and correct any mistake."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 750,
     "status": "ok",
     "timestamp": 1655136507686,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "3ou3vtZyYmAs",
    "outputId": "78549927-daee-4635-f42f-cc81396cef70"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"DigitClassification\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " conv_1 (Conv2D)             (1, 28, 28, 32)           320       \n",
      "                                                                 \n",
      " bnrm_1 (BatchNormalization)  (1, 28, 28, 32)          128       \n",
      "                                                                 \n",
      " actv_1 (Activation)         (1, 28, 28, 32)           0         \n",
      "                                                                 \n",
      " pool_1 (MaxPooling2D)       (1, 14, 14, 32)           0         \n",
      "                                                                 \n",
      " conv_2 (Conv2D)             (1, 14, 14, 64)           18496     \n",
      "                                                                 \n",
      " bnrm_2 (BatchNormalization)  (1, 14, 14, 64)          256       \n",
      "                                                                 \n",
      " actv_2 (Activation)         (1, 14, 14, 64)           0         \n",
      "                                                                 \n",
      " pool_2 (MaxPooling2D)       (1, 7, 7, 64)             0         \n",
      "                                                                 \n",
      " conv_3 (Conv2D)             (1, 7, 7, 128)            73856     \n",
      "                                                                 \n",
      " bnrm_3 (BatchNormalization)  (1, 7, 7, 128)           512       \n",
      "                                                                 \n",
      " actv_3 (Activation)         (1, 7, 7, 128)            0         \n",
      "                                                                 \n",
      " pool_3 (MaxPooling2D)       (1, 3, 3, 128)            0         \n",
      "                                                                 \n",
      " conv_4 (Conv2D)             (1, 3, 3, 256)            295168    \n",
      "                                                                 \n",
      " bnrm_4 (BatchNormalization)  (1, 3, 3, 256)           1024      \n",
      "                                                                 \n",
      " actv_4 (Activation)         (1, 3, 3, 256)            0         \n",
      "                                                                 \n",
      " pool_4 (MaxPooling2D)       (1, 1, 1, 256)            0         \n",
      "                                                                 \n",
      " conv_5 (Conv2D)             (1, 1, 1, 10)             2570      \n",
      "                                                                 \n",
      " bnrm_5 (BatchNormalization)  (1, 1, 1, 10)            40        \n",
      "                                                                 \n",
      " actv_5 (Activation)         (1, 1, 1, 10)             0         \n",
      "                                                                 \n",
      " flatten (Flatten)           (1, 10)                   0         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 392,370\n",
      "Trainable params: 391,390\n",
      "Non-trainable params: 980\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# define layers configuration ((no_filters, kernel_size, pool_size))\n",
    "layers_config = ((32,3,2),(64,3,2),(128,3,2),(256,3,2),(10,1,0))\n",
    "\n",
    "# instantiate and build model\n",
    "cnn = model(layers_config)\n",
    "cnn.build(input_shape=(1,28,28,1))\n",
    "\n",
    "# check model architecture\n",
    "cnn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Tmlr8cl80hX-"
   },
   "source": [
    "### Model Callbacks\n",
    "\n",
    "Almost there!\n",
    "\n",
    "Before we start training the model, we will define three TensorFlow Callbacks, they are used to control the learning process and to record information about the model.\n",
    "\n",
    "1. Reduce Learning Rate on Plateau: it reduces the learning rate to enhance the model parameters update. It will monitor the validation loss, when it does not improve at a minimum delta the factor will be applied to the learning rate.\n",
    "1. Model Checkpoint: it saves the model for future usage, like during predictions. It will save the best version of the model, also monitoring the validation loss.\n",
    "1. CSV Logger: creates the model log in a CSV file, it can be used to get knowledge about the learning dynamics allowing improvements.\n",
    "\n",
    "There are much more callback otions, for details about TensorFlow Callbacks please check https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "executionInfo": {
     "elapsed": 318,
     "status": "ok",
     "timestamp": 1655136514896,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "sG8wbVFIVAGh"
   },
   "outputs": [],
   "source": [
    "# define folder and files to store model check point and log\n",
    "folder = '/content/drive/MyDrive/Colab Notebooks/Handwriting/'\n",
    "ckpfile = folder+'DigitClassification_ckp'\n",
    "logfile = folder+'DigitClassification_log.csv'\n",
    "\n",
    "# define reduce learning rate callback\n",
    "Reduce_LR = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5,\n",
    "                                              patience=2, verbose=0,\n",
    "                                              mode='auto', min_delta=5e-4,\n",
    "                                              cooldown=0, min_lr=(1e-5))\n",
    "\n",
    "# define model checkpoint callback\n",
    "Checkpoint = keras.callbacks.ModelCheckpoint(filepath=ckpfile,\n",
    "                                             monitor='val_loss', verbose=0,\n",
    "                                             save_best_only=True,\n",
    "                                             save_weights_only=False,\n",
    "                                             mode='auto')\n",
    "\n",
    "# define model logger callback\n",
    "Logger = keras.callbacks.CSVLogger(filename=logfile, separator=',',\n",
    "                                   append=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LO3yG8ybg-dU"
   },
   "source": [
    "### Training Model\n",
    "\n",
    "We will set the learning rate to 0.02 and the number of epochs to 20. For compiling we will use SGD optimizer, cross-entropy loss, since it is a classification problem, and sparse categorical accuracy as metric, it calculates how often predictions match integer labels.\n",
    "\n",
    "Actually it is normal to try several different setups during the training, I spent some time tuning it to achive the results below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 173233,
     "status": "ok",
     "timestamp": 1655136850212,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "31-3bDhCcZUK",
    "outputId": "578cdfac-7665-4722-de4d-aca64f89cfac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "938/938 [==============================] - ETA: 0s - loss: 0.3073 - sparse_categorical_accuracy: 0.9704INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 12s 8ms/step - loss: 0.3073 - sparse_categorical_accuracy: 0.9704 - val_loss: 0.1229 - val_sparse_categorical_accuracy: 0.9864 - lr: 0.0200\n",
      "Epoch 2/20\n",
      "931/938 [============================>.] - ETA: 0s - loss: 0.1092 - sparse_categorical_accuracy: 0.9881INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 8ms/step - loss: 0.1092 - sparse_categorical_accuracy: 0.9881 - val_loss: 0.0786 - val_sparse_categorical_accuracy: 0.9909 - lr: 0.0200\n",
      "Epoch 3/20\n",
      "932/938 [============================>.] - ETA: 0s - loss: 0.0720 - sparse_categorical_accuracy: 0.9910INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 8ms/step - loss: 0.0719 - sparse_categorical_accuracy: 0.9911 - val_loss: 0.0515 - val_sparse_categorical_accuracy: 0.9924 - lr: 0.0200\n",
      "Epoch 4/20\n",
      "928/938 [============================>.] - ETA: 0s - loss: 0.0538 - sparse_categorical_accuracy: 0.9932INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 7ms/step - loss: 0.0537 - sparse_categorical_accuracy: 0.9932 - val_loss: 0.0424 - val_sparse_categorical_accuracy: 0.9922 - lr: 0.0200\n",
      "Epoch 5/20\n",
      "933/938 [============================>.] - ETA: 0s - loss: 0.0427 - sparse_categorical_accuracy: 0.9949INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 8ms/step - loss: 0.0426 - sparse_categorical_accuracy: 0.9949 - val_loss: 0.0396 - val_sparse_categorical_accuracy: 0.9914 - lr: 0.0200\n",
      "Epoch 6/20\n",
      "938/938 [==============================] - 6s 6ms/step - loss: 0.0355 - sparse_categorical_accuracy: 0.9955 - val_loss: 0.0412 - val_sparse_categorical_accuracy: 0.9910 - lr: 0.0200\n",
      "Epoch 7/20\n",
      "937/938 [============================>.] - ETA: 0s - loss: 0.0292 - sparse_categorical_accuracy: 0.9971INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 7ms/step - loss: 0.0292 - sparse_categorical_accuracy: 0.9971 - val_loss: 0.0301 - val_sparse_categorical_accuracy: 0.9944 - lr: 0.0200\n",
      "Epoch 8/20\n",
      "934/938 [============================>.] - ETA: 0s - loss: 0.0261 - sparse_categorical_accuracy: 0.9971INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 8ms/step - loss: 0.0261 - sparse_categorical_accuracy: 0.9971 - val_loss: 0.0260 - val_sparse_categorical_accuracy: 0.9939 - lr: 0.0200\n",
      "Epoch 9/20\n",
      "937/938 [============================>.] - ETA: 0s - loss: 0.0231 - sparse_categorical_accuracy: 0.9977INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 8ms/step - loss: 0.0231 - sparse_categorical_accuracy: 0.9977 - val_loss: 0.0238 - val_sparse_categorical_accuracy: 0.9940 - lr: 0.0200\n",
      "Epoch 10/20\n",
      "938/938 [==============================] - 5s 6ms/step - loss: 0.0211 - sparse_categorical_accuracy: 0.9978 - val_loss: 0.0258 - val_sparse_categorical_accuracy: 0.9942 - lr: 0.0200\n",
      "Epoch 11/20\n",
      "938/938 [==============================] - 5s 6ms/step - loss: 0.0182 - sparse_categorical_accuracy: 0.9984 - val_loss: 0.0239 - val_sparse_categorical_accuracy: 0.9943 - lr: 0.0200\n",
      "Epoch 12/20\n",
      "933/938 [============================>.] - ETA: 0s - loss: 0.0153 - sparse_categorical_accuracy: 0.9989INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 7ms/step - loss: 0.0153 - sparse_categorical_accuracy: 0.9989 - val_loss: 0.0206 - val_sparse_categorical_accuracy: 0.9948 - lr: 0.0100\n",
      "Epoch 13/20\n",
      "937/938 [============================>.] - ETA: 0s - loss: 0.0141 - sparse_categorical_accuracy: 0.9990INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 8ms/step - loss: 0.0141 - sparse_categorical_accuracy: 0.9990 - val_loss: 0.0202 - val_sparse_categorical_accuracy: 0.9946 - lr: 0.0100\n",
      "Epoch 14/20\n",
      "938/938 [==============================] - 5s 6ms/step - loss: 0.0138 - sparse_categorical_accuracy: 0.9991 - val_loss: 0.0205 - val_sparse_categorical_accuracy: 0.9951 - lr: 0.0100\n",
      "Epoch 15/20\n",
      "938/938 [==============================] - 5s 6ms/step - loss: 0.0121 - sparse_categorical_accuracy: 0.9994 - val_loss: 0.0211 - val_sparse_categorical_accuracy: 0.9946 - lr: 0.0050\n",
      "Epoch 16/20\n",
      "931/938 [============================>.] - ETA: 0s - loss: 0.0122 - sparse_categorical_accuracy: 0.9993INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 7ms/step - loss: 0.0122 - sparse_categorical_accuracy: 0.9993 - val_loss: 0.0196 - val_sparse_categorical_accuracy: 0.9944 - lr: 0.0050\n",
      "Epoch 17/20\n",
      "938/938 [==============================] - 5s 6ms/step - loss: 0.0120 - sparse_categorical_accuracy: 0.9992 - val_loss: 0.0198 - val_sparse_categorical_accuracy: 0.9945 - lr: 0.0050\n",
      "Epoch 18/20\n",
      "938/938 [==============================] - 5s 6ms/step - loss: 0.0108 - sparse_categorical_accuracy: 0.9996 - val_loss: 0.0196 - val_sparse_categorical_accuracy: 0.9947 - lr: 0.0050\n",
      "Epoch 19/20\n",
      "936/938 [============================>.] - ETA: 0s - loss: 0.0114 - sparse_categorical_accuracy: 0.9994INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 8ms/step - loss: 0.0114 - sparse_categorical_accuracy: 0.9994 - val_loss: 0.0195 - val_sparse_categorical_accuracy: 0.9945 - lr: 0.0025\n",
      "Epoch 20/20\n",
      "936/938 [============================>.] - ETA: 0s - loss: 0.0107 - sparse_categorical_accuracy: 0.9995INFO:tensorflow:Assets written to: /content/drive/MyDrive/Colab Notebooks/Handwriting/DigitClassification_ckp/assets\n",
      "938/938 [==============================] - 7s 8ms/step - loss: 0.0107 - sparse_categorical_accuracy: 0.9995 - val_loss: 0.0192 - val_sparse_categorical_accuracy: 0.9943 - lr: 0.0025\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f2ca22f3810>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define learning rate and number of epochs\n",
    "learn_rate = 0.02\n",
    "num_epochs = 20\n",
    "\n",
    "# compile model\n",
    "cnn.compile(optimizer = keras.optimizers.SGD(learn_rate),\n",
    "            loss = keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "            metrics = [keras.metrics.SparseCategoricalAccuracy()])\n",
    "\n",
    "# train model\n",
    "cnn.fit(data_train, epochs=num_epochs, validation_data=data_test,\n",
    "        callbacks=[Reduce_LR, Checkpoint, Logger])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5vaqynaFK3EZ"
   },
   "source": [
    "We can see from the console output the callbacks working, the learning rate reduction was triggered when no improvement was made in validation loss after two epochs, and that the model was saved only when there was validation loss improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1x-_9zohB4Uo"
   },
   "source": [
    "### Analysing Log\n",
    "\n",
    "We will use the log file just recorded to have a better understanding of the training dynamics.\n",
    "\n",
    "First let's check what it contains, the columns are in the same position as the console output TensorFlow provided above.\n",
    "\n",
    "1. Epoch\n",
    "1. Training loss\n",
    "1. Training accuracy\n",
    "1. Validation loss\n",
    "1. Validation accuracy\n",
    "1. Learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 316,
     "status": "ok",
     "timestamp": 1655136892910,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "oxKFn_aRVCCQ",
    "outputId": "c2f3f0e7-05cb-4508-ec03-c4cab888725c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.000e+00, 3.073e-01, 2.000e-02, 9.704e-01, 1.229e-01, 9.864e-01],\n",
       "       [1.000e+00, 1.092e-01, 2.000e-02, 9.881e-01, 7.860e-02, 9.909e-01],\n",
       "       [2.000e+00, 7.194e-02, 2.000e-02, 9.911e-01, 5.154e-02, 9.924e-01],\n",
       "       [3.000e+00, 5.372e-02, 2.000e-02, 9.932e-01, 4.236e-02, 9.922e-01],\n",
       "       [4.000e+00, 4.261e-02, 2.000e-02, 9.949e-01, 3.964e-02, 9.914e-01],\n",
       "       [5.000e+00, 3.549e-02, 2.000e-02, 9.955e-01, 4.117e-02, 9.910e-01],\n",
       "       [6.000e+00, 2.923e-02, 2.000e-02, 9.971e-01, 3.010e-02, 9.944e-01],\n",
       "       [7.000e+00, 2.606e-02, 2.000e-02, 9.971e-01, 2.597e-02, 9.939e-01],\n",
       "       [8.000e+00, 2.310e-02, 2.000e-02, 9.977e-01, 2.378e-02, 9.940e-01],\n",
       "       [9.000e+00, 2.110e-02, 2.000e-02, 9.978e-01, 2.578e-02, 9.942e-01],\n",
       "       [1.000e+01, 1.818e-02, 2.000e-02, 9.984e-01, 2.389e-02, 9.943e-01],\n",
       "       [1.100e+01, 1.535e-02, 1.000e-02, 9.989e-01, 2.062e-02, 9.948e-01],\n",
       "       [1.200e+01, 1.409e-02, 1.000e-02, 9.990e-01, 2.022e-02, 9.946e-01],\n",
       "       [1.300e+01, 1.384e-02, 1.000e-02, 9.991e-01, 2.047e-02, 9.951e-01],\n",
       "       [1.400e+01, 1.211e-02, 5.000e-03, 9.994e-01, 2.112e-02, 9.946e-01],\n",
       "       [1.500e+01, 1.224e-02, 5.000e-03, 9.993e-01, 1.955e-02, 9.944e-01],\n",
       "       [1.600e+01, 1.202e-02, 5.000e-03, 9.992e-01, 1.981e-02, 9.945e-01],\n",
       "       [1.700e+01, 1.078e-02, 5.000e-03, 9.996e-01, 1.961e-02, 9.947e-01],\n",
       "       [1.800e+01, 1.140e-02, 2.500e-03, 9.994e-01, 1.950e-02, 9.945e-01],\n",
       "       [1.900e+01, 1.070e-02, 2.500e-03, 9.995e-01, 1.916e-02, 9.943e-01]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the model log\n",
    "model_log = np.loadtxt(logfile, dtype=np.float32, delimiter=',', skiprows=1)\n",
    "model_log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tddoFiBbIgLe"
   },
   "source": [
    "Now we use the log data to plot the results, the first plot shows the improvement of accuracy, and the gap between training and validation. The second plot shows the improvements in validation loss, which was our target during monitoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 350
    },
    "executionInfo": {
     "elapsed": 712,
     "status": "ok",
     "timestamp": 1655136904068,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "LuLZMgujinrt",
    "outputId": "0db74a5a-cc8c-4764-c101-439cd07f17c8"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt0AAAFNCAYAAADcudMsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXxU5dn/8c+VEJYIBAREtgAmgKLghihxQ60WFcTduiB2o/q01udRa61ra9W21tbWurT0pxZcq2gtIO77AioqKKggKjsREMMeIMn1++M+wSFkmUBmTpbv+/Wa18ycba6hdeabe65zH3N3REREREQkdTLiLkBEREREpLFT6BYRERERSTGFbhERERGRFFPoFhERERFJMYVuEREREZEUU+gWEREREUkxhW4RERGJhZm5meVHj/9uZtcms+0OvM65ZvbcjtYpUhcUuqXRM7NXzOwbM2sRdy0iIo2JmT1jZjdUsnykmRWaWbNkj+XuF7r7b+ugpl5RQN/62u7+oLsft7PHruS1hprZ4ro+rjROCt3SqJlZL+BwwIGT0vi6SX/RiIg0YOOA88zMKiwfBTzo7iUx1CRSLyl0S2N3PjAN+BcwunyhmfUwsyfMbIWZfW1mdySs+7GZfWJma83sYzM7IFq+zU+bZvYvM7sxejzUzBab2S/NrBC4z8zam9nk6DW+iR53T9h/VzO7z8yWRuufjJbPMrMRCdtlmdlKM9s/Zf9KIiI75kmgA2FwAwAzaw8MB8ab2WAzm2pmRWa2zMzuMLPmlR0o8TM1ev6LaJ+lZvaDCtueaGYfmNkaM1tkZr9OWP1adF9kZuvMbIiZXWBmbyTsX2Bm75rZ6ui+IGHdK2b2WzN7M/oeeM7MOtb2H8bM9oqOVWRms83spIR1J0TfL2vNbImZXR4t7xh9VxSZ2Soze93MlNUaCf0PKY3d+cCD0e27ZtbZzDKBycACoBfQDXgEwMzOAH4d7deWMDr+dZKvtTuwK9ATGEP47+u+6HkusBG4I2H7+4FsYG9gN+C2aPl44LyE7U4Alrn7B0nWISKSFu6+EXiU8JlZ7kzgU3efCZQC/wd0BIYAxwD/U9NxzWwYcDlwLNAH+E6FTdZHr9kOOBG4yMxOjtYdEd23c/fW7j61wrF3BZ4Cbif8wfBn4Ckz65Cw2TnA9wmfzc2jWpJmZlnAJOC56BgXAw+aWb9ok3uAn7h7G2Af4KVo+WXAYqAT0Bm4ivBLrTQCCt3SaJnZYYTA+6i7vwd8TvggHQx0BX7h7uvdvdjdy0dAfgTc4u7vejDP3Rck+ZJlwPXuvsndN7r71+7+uLtvcPe1wE3AkVFtXYDjgQvd/Rt33+Lur0bHeQA4wczaRs9HEQK6iEh9NA443cxaRs/Pj5bh7u+5+zR3L3H3+cA/iD4Ha3AmcJ+7z3L39YTBkK3c/RV3/8jdy9z9Q+DhJI8LIaR/5u73R3U9DHwKjEjY5j53n5vwR8V+SR673CFAa+D37r7Z3V8iDPacHa3fAvQ3s7bRd8D7Ccu7AD2j74XX3V2hu5FQ6JbGbDTwnLuvjJ4/FC3rASyootewByGc74gV7l5c/sTMss3sH2a2wMzWEH7ybBeNtPcAVrn7NxUP4u5LgTeB08ysHSGcP7iDNYmIpFQ0aLESONnM8ggDGw8BmFnfqF2iMPocvJkw6l2TrsCihOfbDH6Y2cFm9nLUvrcauDDJ45Yfu+JgygLCr57lChMebyAE6NroCixy97IqXuM0wq+YC8zsVTMbEi3/IzAPeM7MvjCzK2v5ulKPKXRLo2RmrQgjJUdGH/aFhJ849wW+AnKrONlxEZBXxWE3ENpByu1eYX3F0YjLgH7Awe7elm9/8rTodXaNQnVlxhFaTM4Aprr7kiq2ExGpD8YTRrjPA55196+i5XcTRpH7RJ+DVxE+A2uyjDA4US63wvqHgIlAD3fPAf6ecNyaRoaXEn4FTZQL1OXn7FKgR4V+7K2vEf2aOpLQevIkYTQdd1/r7pe5+x6E9sZLzeyYOqxLYqTQLY3VyYRewv6EnwX3A/YCXo/WLQN+b2a7mFlLMzs02u//AZeb2YEW5JtZ+YfzDOAcM8uM+g1r+imzDaGPuyjqIby+fIW7LwOeBu6KTrjMMrMjEvZ9EjgAuITwZSYiUp+NJ/Rd/5iotSTSBlgDrDOzPYGLkjzeo8AFZtbfzLJJ+PxMOO4qdy82s8GE1sFyKwjtfntUcewpQF8zO8fMmpnZWYTvislJ1rad6Htk6w14hzBQc0X0+T6U0L7yiJk1tzBveI67byH8+5RFxxkefe8YsJrwPVZW6YtKg6PQLY3VaEJP3kJ3Lyy/EU5kPJvw4ZcPLCSctHIWgLs/Rui9fghYSwi/u0bHvCTarwg4N1pXnb8ArQg/u04DnqmwfhShf+9TYDnwv+Uroj7Cx4HewBO1fO8iImkV9Wu/BexCGIEudzkhEK8F/gn8O8njPU34DH2J0G7xUoVN/ge4wczWAtcRjRRH+24gfI6/Gc0CckiFY39NmF3lMsKJ8lcAwxNaEWurG2GAJfHWg/B9cTzhO+Au4Hx3/zTaZxQwP2q5uZDwnQLhpNEXgHXAVOAud395B+uSesbUny9SP5nZdUBfdz+vxo1FRESkXtMFPETqoagd5YeE0RARERFp4FLaXmJm95rZcjObVcV6M7PbzWyemX1o0UVIonWjzeyz6JZ4UZMDzeyjaJ/bo74nkUbDzH5MONHyaXd/rabtRUREpP5LaXtJdGLYOmC8u+9TyfoTCBPGnwAcDPzV3Q+ORvmmA4MIZyG/Bxzo7t+Y2TvAz4G3CSdD3B71fomIiIiI1EspHemORulWVbPJSEIgd3efRpjDuAvwXeB5dy+fx/h5YFi0rm000b4TzpY+ucqji4iIiIjUA3HPXtKNbSe/Xxwtq2754kqWi4iIiIjUW432REozGwOMAdhll10O3HPPPWOuSESk9t57772V7t4p7jrSqWPHjt6rV6+4yxARqbXqPrPjDt1L2PaKU92jZUuAoRWWvxIt717J9ttx97HAWIBBgwb59OnT66pmEZG0MbOKl6tu9Hr16oU+s0WkIaruMzvu9pKJwPnRLCaHAKujK/U9CxwXXamvPXAc4bKyy4A1ZnZINGvJ+cB/Y6teRERERCQJKR3pNrOHCSPWHc1sMeEyrlkA7v53wuwjJxCuNrUB+H60bpWZ/RZ4NzrUDe5efkLm/wD/Ilzp7+noJiJSqRUrYMYM+OCDb++XLoUOHaBTJ+jYseb7du2gpslJ3WHTJiguDrfKHh9yCDRvnp73LSIi9UtKQ7e7n13Degd+WsW6e4F7K1k+Hdhu+kERSb+NG+GTT2DePMjOhl13/fbWvj1kZaWvFneYPz+E6sSAvSShAS03F/bfH449FlatgpUrobAQZs0K4XzjxsqP3axZCOkdO0JGxvahurgYNm+uucYlS6Br1zp5uyIi0sDE3dMtIg3A5s0wZ04Ip7Nnf3v/+ech7FalTZtvA3hiIE+8tW4NmZkhzCbeV7Ys8d4dPv1021Hs1avD62ZkwF57wdChIWTvvz/su28IztXZsCGE75Urq74HaNECWrYMt8THFZ9XXLfrrnXyP4eIiDRACt0islVJSRi1rhiu586F0tKwTWYm9O0L++0H550He+8dnm/aFEaPq7vNmgXffANffx1eqy60agUDB8LZZ4dwvd9+MGBAWF5b2dnQs2e4iYiI1CWFbpFGqqQkBNyagnDibeHCb9skzCAvL4TqU08N9/vsEwJ2ixY7V5s7rF8fXnPdOigrC6G+/D7xcVX37qG+vn3DHwIiIiL1mUK3SD1XXFx1SK4uVK9ZU/UxzcLJgYltHnl5cMopIVjvsw/suWcY+U0Fs9BW0rp1ao4vIiJS3yh0i9QTW7bAtGnw/PPw4ouwYEEIz1Wd3AfhBL/EnukuXaB//9C7XFUP9a67Qk6ORodFRETSSaFbJCbu4eTE55+H556DV14JrRYZGTB4MAwbtu1MIFWdhFjTVHYiIiISP4VukTRasQJeeCEE7eefh8WLw/L8fBg1Kkxld9RRofVDRGrhscfCX6LHHBN3JSIilVLoFkmh4mJ4441vR7NnzAjL27cP2eDYY8Otd+946xRp8K69NpyMoNAtIvWUQrdIHduyBZ55BsaNg6eeCsE7KwsOPRRuuimE7AMOUE+1SJ3KywsTx4uI1FMK3SJ1wD2MYo8bBw89FNpIOnWCH/0Ijj8ejjhCM3WIpFR+Prz2WviPUSc6iEg9lBF3ASIN2bJlcOut4WqHBxwAd98dAvbEieGS33/7G5xwggK3NGxmNszM5pjZPDO7spL1R5jZ+2ZWYmanJyzfz8ymmtlsM/vQzM5KWZF5eeFM5BUrUvYSIiI7QyPdIrVUXAz//W8Y1X722XCxloMPhrvugrPO0qW+pXExs0zgTuBYYDHwrplNdPePEzZbCFwAXF5h9w3A+e7+mZl1Bd4zs2fdvajOC83LC/effw677VbnhxcR2VkK3SJJcIe33gpB+9FHYfVq6NEDrrwSzj8f+vWLu0KRlBkMzHP3LwDM7BFgJLA1dLv7/GhdWeKO7j434fFSM1sOdAJSF7rnzYMhQ+r88CIiO0uhW5q84uIQoouKwn3Fx199Bf/5T/guz86G006D0aPD1H4ZatCSxq8bsCjh+WLg4NoexMwGA82B1Jzt2Lt36OXWyZQiUk8pdEujt2xZOLlx2rRvw3Ti/ebN1e+fkQFHHgnXXBMCt/qzRWrHzLoA9wOj3b2sim3GAGMAcnNza/8iLVqEn58UukWknlLolkZp48bQdz1+/Ld91/n50LFjuET6HnuES6G3a7ftfWWPW7fWiLY0aUuAHgnPu0fLkmJmbYGngKvdfVpV27n7WGAswKBBg3yHKs3LCz9JiYjUQwrd0mio71okJd4F+phZb0LY/h5wTjI7mllz4D/AeHefkLoSI/n58OSTKX8ZEZEdodAtDd78+WFEe/z48MtydjacfnoI2uq7Ftk57l5iZj8DngUygXvdfbaZ3QBMd/eJZnYQIVy3B0aY2W/cfW/gTOAIoIOZXRAd8gJ3n5GSYvPywpSBa9ZA27YpeQkRkR2l0C0N0tq18NhjIWi/+mpYdtRR4UrQ6rsWqVvuPgWYUmHZdQmP3yW0nVTc7wHggZQXWC5x2sD990/by4qIJEOhW9Ju7NhwmfQWLaBly29vic+rerxpEzz+ODzxROjb7tMHbrwRzjsPevaM+52JSKzy88O9QreI1EMK3ZI27vCb34Rb797QrFkI0cXF395qmkkEwsmNo0eH28EH64rPIhJJHOkWEalnFLolLdzhqqvg97+H738f/vlPyMzcfruyshC8i4u3D+SbNkFJSbjcesuW6X8PIlLPtWkDnTppBhMRqZcUuiXl3OHSS+Evf4ELL4Q776z65MaMjG9bSUREai0/XyPdIlIvaV4HSamyMvjpT0PgvuQSuOsuzSYiIimUl6fQLSL1kuKPpExpKYwZA3ffDVdcAbfdpv5rEUmxvDxYtCj0o4mI1CMK3ZISJSVwwQVwzz1w3XWhl1uBW0RSLj8/9LR9+WXclYiIbEOhW+rcli1wzjnwwANw001hthIFbhFJC81gIiL1lE6klDq1aROcdRb8979w661w2WVxVyQiTYpCt4jUUwrdUmc2bgxXg3z6abjjjnACpYhIWnXqFKYO1LSBIlLPKHRLnVi/Hk4+GV58MVxx8sc/jrsiEWmSzDSDiYjUSwrdstPWroXhw+GNN+Bf/4Lzz4+7IhFp0vLyYNasuKsQEdmGTqSUnVJUBMcdB2++CQ89pMAtIvVAfj588UWYt1REpJ5Q6JYdtmoVfOc78N578Nhj4QRKEZHY5eWFaZQWL467EhGRrRS6pdZKS+GFF+Doo8MvuP/5D5xyStxViYhENIOJiNRDCt2SFHd4++1wKffu3eHYY2HBApg4EU48Me7qREQS5OeHe81gIiL1SEpDt5kNM7M5ZjbPzK6sZH1PM3vRzD40s1fMrHvCuj+Y2azodlbC8n+Z2ZdmNiO67ZfK99DUffwxXHNN+A475BD4xz+goAAmTIClS0M/t4hIvdKtGzRvrpFuEalXUjZ7iZllAncCxwKLgXfNbKK7f5yw2a3AeHcfZ2ZHA78DRpnZicABwH5AC+AVM3va3ddE+/3C3SekqvambsECeOQRePhhmDkTMjJC7/a114Y2kpycuCsUEalGZib07q3QLSL1SiqnDBwMzHP3LwDM7BFgJJAYuvsDl0aPXwaeTFj+mruXACVm9iEwDHg0hfU2aStWhJMhH3oozEQCMGQI3H47nHkmdO4cb32SGmVexvSl05k0ZxJT5k0h0zI5sc+JjOg3gv133x8zi7tEkR2Tl6f2EhGpV1IZursBixKeLwYOrrDNTOBU4K/AKUAbM+sQLb/ezP4EZANHsW1Yv8nMrgNeBK50902peQuN39NPh2D9/PPhBMl99oGbb4bvfS8MFEnjs37zep7/4nkmz53MU589ReG6QjIsg4IeBZSWlfKbV3/Dr1/9NV3bdGV4n+GM6DeCY3ofQ6usVnGXLpK8/Hx47bVwQor+eBSReiDui+NcDtxhZhcArwFLgFJ3f87MDgLeAlYAU4HyCVd/BRQCzYGxwC+BGyoe2MzGAGMAcnNzU/suGqiHHoLzzoMePeCKK+Dss2HAgLirktKyUt5b9h4lZSX0zOlJlzZdyLCdO/1i0epFTJ47mUlzJ/HSly+xqXQTbVu0ZVj+MEb0HcHx+cfTIbsDAMvXL2fKZ1OYNHcSD816iLHvj6VVs1Z8Z4/vMKLvCE7seyJd23Sti7cqkjp5ebBuXfgZb7fd4q5GRCSloXsJ0CPhefdo2VbuvpQw0o2ZtQZOc/eiaN1NwE3RuoeAudHyZdHum8zsPkJw3467jyWEcgYNGuR185YajwkTwoVshg6FyZMhOzvuipq2tZvW8tznzzFp7iSmfDaFFRtWbF2XlZFFj5we5Obk0jOnZ7i167n1eY+cHrRs1nKb45W3jZQH7RmFMwDIa5/HRYMuYkS/ERyWexjNM5tvV8tuu+zGBftdwAX7XcCmkk28uuBVJs2ZxKS54QZwYJcDGdF3RINpQynzMgrXFbJw9UIWFC1g3eZ1dG/bfeu/Y3aW/gNodMqnDZw3T6FbROoFc09NHjWzZoSgfAwhbL8LnOPusxO26QiscvcyM7uJMMp9XXQSZjt3/9rMBgIPAfu5e4mZdXH3ZRa+5W8Dit19u5lREg0aNMinT5+ekvfZEE2cCKedBgcfDM88A61bx11R0zS/aP7WMPvK/FfYUraF9i3bc3yf4xneZzg5LXNYULQgBMXVC8KtaAFL1y7F2fa/291b7741hLds1pLnv3h+a9vIoT0OZXjf4YzoO4I9O+65wwHZ3Zm1fNbWID9t8TQcp2ubrpyQfwL9O/WnZ7vwR0FuTi4dszumLYxvKtnEojWLKv33Wrh6IYvWLGJz6eYq9++U3Sn8+yXUn/jHTYdWHWL7w8LM3nP3QbG8eEzq5DN7zhzYc08YPx5GjaqbwkREalDdZ3bKRrqjgPwz4FkgE7jX3Web2Q3AdHefCAwFfmdmTmgv+Wm0exbwevQltwY4LzqpEuBBM+sEGDADuDBV76ExeuYZOOMMOOAAmDJFgTudSstKeXvJ21tD66zlswDo16Eflxx8CSP6jaCgRwHNMqr/z3JL6RYWr1m8TagsD5kzCmewetNqhvYaul3byM4yMwZ0HsCAzgP41eG/2qYN5dGPH2XNpjXbbJ+dlb01vCaG2PLn3dp22+69lpaVsmbTGlZvWk1RcRGri1dX+bhoUxGLVi9i4eqFLFu3bJvjGEbXNl3JzcnloG4HcXr/07epoXXz1ixavWi7f8OPV3zM0589zcaSjdscb5esXbaG8ty234bz8lDetU3XGv93kzTr1Sv0cmsGExGpJ1I20l2faKQ7ePFFGD4c9torPG7fPp46bpt6G68tfI0T8k9geN/hdGnTJZ5C0qCytpFMy+SInkcwou8IhvcdTp8OfeIuc6e5O6s2rqr0D4Hylo7ElhmATMukW9tutGnehtWbVrO6eDVrN6+t8bWys7LJaZFDu5bt6Na22zYhuDwYd2/bvdLWmWTfy9cbv2ZB0bb1J76flRtWbvdeEttVEtuAyluAdrSFRSPdO6FnTzj8cHjggZ0/lohIEmIZ6Zb65fXX4aSToE+fMFNJXIF76dql/OrFX5FhGTz5aZghclDXQVtnyaiP/cHFJcWs2rgqjK4WF20NiJU9rriscF3hNm0jI/qOYFj+MNq1bBf326pTZkaH7A50yO7AAV0OqHSbDVs2VDq6vH7LenJa5GwN0jktq3+clZmV8vfSMbsjHbM7cmDXAyvdZv3m9SxcvfDbPy4SAvprC15jyZollHrpNvt0yu7E9DHTyc3Rid1pk5+vkW4RqTcUupuAadPghBMgNzcE7g51022wQ/745h8pKSvhs4s/Y/2W9Vt7msunqevWptvW/uOjex+d8mnqykdoE8NTxVHa5euXV3uMTMskp2UUDKOAmLdrHjktcujapivD8ocl1TbS2GVnZdOvYz/6dewXdyk7bZfmu7BXp73Yq9Nela4vKSth6dql346QR/+/2m0XndCXVnl58OSTNW8nIpIGTTsFNAHvvQfDhsHuu4eWkjgvcvPVuq/4+3t/Z9S+o+jdPkwCvs9u+2zTHzx57mQe/OhB/vHeP2jVrBXH5h3L8D7Da92Gsqlk07b9v8VFrNiwYtuWgSgMrd+yfpt9WzVrtbVNYb/O+5Gbk0unXTptDdUVA3Z2Vna9G52XeDXLaEZuTi65ObkczuFxl9N05eWFKQPXrIG2beOuRkSaOIXuRmzmTDj22NBK8tJL0DXmqZX/NPVPbC7dzFWHXbXduuqmqZs4ZyIQ2lCOzz+e5pnNtzmZbrsT7IqL2FRa9fWSOrTqQM92PenboS/H7nHsNr3APXN6pnXWDRFJofz8cP/557D//vHWIiJNnkJ3I/XxxyFw77JLCNw9etS8Tyqt3LCSu969i7P3ObvGEwdbNGvBcXnHcVzecdx+/O3bTFN342s34vjWk+nKR53bt2xPr3a9aNei3TYj0YmPO2R3IDcnl9bNNWWLSJNQPle3QreI1AMK3Y3Q3LlwzDGQmRkCd324nPttU29jw5YNXH341bXar+I0dRu2bCArIyvlJ9OJSCOQeIEcEZGYKXQ3Ml98AUcfDaWl8MorYbaSuH2z8Rv+9s7fOGPvM6o88SxZunKgiCStTZtwNUrNYCIi9YBCdyOycGEI3Bs3wssvQ//+cVcU/PXtv7J281quOfyauEsRkaYmL0+hW0TqhYy4C5C6sXRpCNxFRfDcczBwYNwVBauLV/PXt//KyXuezIDOA+IuR0Samrw8tZeISL2g0N0ILF8eeri/+ipc5v3Ayq/nEYs73rmDouIirj3i2rhLEZGmKD8fFi+GTVXPaCQikg4K3Q1cSQmceSYsWABTpsAhh8Rd0bfWblrLn6f9mRP7nFjlVQpFRFIqLw/c4csv465ERJo4he4G7qqr4NVX4R//gMPr2TU47p5+N6s2rtIot4jERzOYiEg9odDdgD3xBPzxj3DRRTBqVNzVbGvDlg38aeqfOC7vOA7ufnDc5YhIU5V4gRwRkRgpdDdQc+fCBRfAQQfBbbfFXc32xr43luXrl2uUW0Ti1bFjmDpQoVtEYqbQ3QCtXw+nnQbNm8OECdCiRdwVbau4pJhb3ryFo3odxWG5h8VdjojUATMbZmZzzGyemV1ZyfojzOx9Mysxs9MrrBttZp9Ft9Hpqxow0wwmIlIvaJ7uBsYdLrwQZs8OM5Xk5sZd0fbuef8elq1bxoOnPhh3KSJSB8wsE7gTOBZYDLxrZhPd/eOEzRYCFwCXV9h3V+B6YBDgwHvRvt+ko3YghO6PPkrby4mIVEYj3fXIR199xOsLXmfh6oWUlJVUus3dd8MDD8BvfgPHHZfmApOwqWQTv3/z9xyWexhDew2NuxwRqRuDgXnu/oW7bwYeAUYmbuDu8939Q6Cswr7fBZ5391VR0H4eGJaOorfKzw+zl5SWpvVlRUQSaaS7nrjznTu5+OmLcRyATMuke9vu5Obk0rNdT3rm9KR0VS5/vL0nR5zak/+7Iheof5dEHzdzHIvXLOaek+7BzOIuR0TqRjdgUcLzxUCyZ0hXtm+3OqorOXl5sGULLFoEvXql9aVFRMopdMfM3bnmpWu4+Y2bGdlvJBcNuoiFqxeyYPUCFqxewMLVC3l9wes8vOZhSr0UzobXgDa/h47ZHemZ05PcnFz26rgXPxn0E3Jz4us32VK6hd+98TsGdxvMsXscG1sdItLwmNkYYAxAbl33zZVPG/j55wrdIhIbhe4YlZSVMGbSGO6bcR9jDhjDnSfeSbOM7f8nKS2F44aV8MbMpdz14EJadg6BfEFRuP905adMmjuJW966hVEDR3HlYVfSt0PftL+fBz58gPlF87nj+Ds0yi3SuCwBeiQ87x4tS3bfoRX2faXiRu4+FhgLMGjQIN+RIquUOG3gMcfU6aFFRJKl0B2TDVs2cOZjZ/LUZ09x/ZHXc/2R11cZVH/9a3jphWb885+5/PDYXGD7GUEWrV7EH9/6I/98/5+MmzmOM/qfwVWHX8XAzgNT+0YiJWUl3PzGzRzQ5QBO6HNCWl5TRNLmXaCPmfUmhOjvAeckue+zwM1m1j56fhzwq7ovsRrduoXpnjSDiYjESCdSxmDlhpUcM/4Ynp73NHefeDe/HvrrKgP35Mlw443wgx/Aj35U9TF75PTg9uNvZ8H/LuCKgiuY8tkU9v37vox4eATTFk9L0Tv51iOzHmHeqnlce8S1GuUWaWTcvQT4GSFAfwI86u6zzewGMzsJwMwOMrPFwBnAP8xsdrTvKuC3hOD+LnBDtCx9MjOhd2/N1S0isTL3uv0Vrz4aNGiQT58+Pe4yAFhQtIDvPvBd5hfN5+HTHuaUvU6pctsvvoADDwwtiG+9Ba1aJf8632z8hjveuYO/vP0XVm1cxdG9j+bqw6/mqF5H1XkoLi0rZZ+79yErI4sZF84gw/S3nEhdMbP33H1Q3HWkU0o+s4cPh0/heSgAACAASURBVMWLYcaMuj2uiEiC6j6zlY7S6MOvPmTIPUP4av1XvHD+C9UG7o0b4fTo8hKPP167wA3QvlV7rj3yWhb87wJuPfZWPl7xMceMP4aCewuYPHcydfnH1oSPJ/Dpyk+59ohrFbhFpH4qv0BOExhoEpH6SQkpTV6Z/wqH33c4GZbB699/vcYrNf7sZ/DBB3D//bDHHjv+uq2bt+aygsv48pIvueuEu1i2dhkjHh7B/v/Yn0dnP0pp2c7NW1vmZdz4+o3s1XEvTut/2k4dS0QkZfLywuV8ly+PuxIRaaIUutNgwscT+O4D36Vbm25M/eFU9tltn2q3v+ceuPdeuPrq8ItoXWjZrCUXHXQRn138Gf8a+S+KS4o5a8JZ9L+rP7e8eQvTFk9jS+mWWh/3v5/+l1nLZ3H14VdrlFtE6q/EGUxERGKg2UtSrPyiN0N6DGHS2ZPYtdWu1W7//vvw05/Cd74TrjpZ17Iysxi932jOG3geT3zyBH948w/88oVfApCdlc2Q7kM4PPdwjuh5BAd3P5jsrKovwOPu/Pa139Jn1z6ctc9ZdV+siEhdKZ+re948KCiItxYRaZIUulPE3bn25Wu56fWbOKnfSTxy2iO0yqq+MXvVKjjtNOjUCR56KJxwnyqZGZmcsfcZnLH3GRSuK+SNhW/w2oLXeH3h6/zm1d/gOFkZWQzqOmhrCD8091DatWy39RhPffYUHxR+wH0j76t0fnERkXqjVy8w00i3iMRGSSkFSspK+Mmkn3DvjHv50f4/4u7hd9cYSt1h9GhYsgRefz0E73TZvfXunN7/dE7vH87cLCou4q1Fb20N4bdNu41b3roFwxjQeQBH5B7B4T0P549v/ZHe7Xpz7oBz01esiMiOaNECcnMVukUkNgrddWzDlg2cNeEsJs+dzHVHXFftHNyJJk4Mc3L/6U9w8MFpKLQa7Vq244Q+J2y9yM3GLRt5e8nbvL7gdV5b+Br3zbiPO969A4Cxw8eSlZkVZ7kiIskpn8FERCQGCt117NqXruWpuU9x94l3c+GgC5PaZ8sWuOIK6NcPLr44xQXugFZZrRjaayhDew0FYEvpFj4o/IDPV33OmXufGW9xIiLJysuD//wn7ipEpIlS6K5DW0q3cP+H93PqXqcmHbgB/v53mDs3jHZnNYBB46zMLAZ3G8zgboPjLkVEJHn5+bByJaxeDTk5cVcjIk2M5nirQ0/Pe5oVG1Ywet/RSe9TVBRmKTnqqLqbHlBERCpRPoOJ+rpFJAYK3XVo3MxxdMruxLD8YUnvc9NNYdaSP/0pnFgvIiIpotAtIjFS6K4jqzauYtKcSZw74NykTyz88ku4/XY4/3zYf/8UFygi0tQpdItIjFIaus1smJnNMbN5ZnZlJet7mtmLZvahmb1iZt0T1v3BzGZFt7MSlvc2s7ejY/7bzJqn8j0k65FZj7ClbAuj90u+teTKK8Nc3DfdlMLCREQkaNMGdttNM5iISCxSFrrNLBO4Ezge6A+cbWb9K2x2KzDe3QcCNwC/i/Y9ETgA2A84GLjczNpG+/wBuM3d84FvgB+m6j3UxriZ4xiw2wD27bxvUttPnQqPPgqXXw7duqW4OBERCfLyNNItIrFI5Uj3YGCeu3/h7puBR4CRFbbpD7wUPX45YX1/4DV3L3H39cCHwDALE14fDUyIthsHnJzC95CUT1d+yjtL3mH0vqOTmpPbHS69FHbfPUwVKCIiaZKfr9AtIrFIZejuBixKeL44WpZoJnBq9PgUoI2ZdYiWDzOzbDPrCBwF9AA6AEXuXlLNMQEwszFmNt3Mpq9YsaJO3lBVxs8cT6Zlcu7A5K7M+NhjMG0a/Pa30Lp1SksTEZFEeXmweDEUF8ddiYg0MXGfSHk5cKSZfQAcCSwBSt39OWAK8BbwMDAVKK3Ngd19rLsPcvdBnVJ4TfXSslLu//B+vpv/XXZvvXuN22/aFHq5BwyA738/ZWWJiEhl8vLCz41ffhl3JSLSxKQydC8hjE6X6x4t28rdl7r7qe6+P3B1tKwour/J3fdz92MBA+YCXwPtzKxZVcdMt5fnv8ziNYs5f+D5SW3/t7+Fz/pbbw0nUYqISBppBhMRiUkqQ/e7QJ9otpHmwPeAiYkbmFlHMyuv4VfAvdHyzKjNBDMbCAwEnnN3J/R+nx7tMxr4bwrfQ43GzRxHToscRu5ZsV19eytXwo03wrBhcNxxaShORES2lZ8f7jWDiYikWcpCd9R3/TPgWeAT4FF3n21mN5jZSdFmQ4E5ZjYX6AyUT56XBbxuZh8DY4HzEvq4fwlcambzCD3e96TqPdRk7aa1PPHJE5y191m0bNayxu1vuAHWrg2j3CIiEoOOHcPUgRrpFpE0a1bzJjvO3acQerMTl12X8HgC385EkrhNMWEGk8qO+QVhZpTYPf7J42zYsiGpubnnzoW774Yf/Qj23jsNxYmIyPbMNG2giMQi7hMpG7RxM8eRv2s+Q7oPqXHbX/4SWrYMo90iIhKj/Hy1l4hI2il076D5RfN5Zf4rnD/w/Brn5n71VXjyyTBrSefOaSpQREQql5cH8+dDaa0mxRIR2SkK3Tvo/pn3AzBq31HVbldWBpddBt27w//9XzoqExGRauXlwZYtsGhRzduKiNQRhe4d4O6M/3A8Q3sNpVe7XtVu+9BD8N57cPPNkJ2dnvpERKQamsFERGKg0L0Dpi6eyrxV8xi9b/UnUG7cCFddBQceCOcmd7FKERFJNc3VLSIxSOnsJY3VuBnjyM7K5rS9Tqt2u9tuC79e3n8/ZOjPGxGR+qFbN2jeXKFbRNJKUbCWNm7ZyL9n/5tT9zqVNi3aVLndV1/B734HI0fCkUemsUAREaleZibssYfaS0QkrRS6a2nS3Ems3rS6xtaS66+H4mK45ZY0FSYiIsnTXN0ikmYK3bU0buY4urftzlG9jqpym9mz4Z//hIsugr5901iciIgkpzx0u8ddiYg0EQrdtVC4rpBn5z3LqIGjyMzIrHK7X/wiXGX4uuuq3EREROKUnw/r14deQBGRNFDoroUHP3yQUi/l/H3Pr3KbF16Ap5+Ga66Bjh3TWJyIiCRPM5iISJopdCfJ3Rk3cxyDuw1mz457Vrnd5Mmwyy5w8cVpLE5ERGpHoVtE0kyhO0kzv5rJR8s/qvEEysLCMBtVixZpKkxERGqvV68wl6tmMBGRNFHoTtK4GeNontmc7+3zvWq3KyyE3XdPU1EiIrJjWrSAHj000i0iaaPQnYQtpVt48KMHGdF3BLu22rXabRW6RUQaCE0bKCJppNCdhGfmPcOKDSuqPYGy3LJlCt0iIg1Cfr7aS0QkbRS6kzD+w/F0yu7E8fnHV7vdhg2wZo1Ct4hIg5CXB19/DatXx12JiDQBCt01WLVxFRPnTOScAeeQlZlV7bbl070qdIuINACawURE0kihuwb/nvVvNpdurnHWEgj93KDQLSKNj5kNM7M5ZjbPzK6sZH0LM/t3tP5tM+sVLc8ys3Fm9pGZfWJmv0p37VUqD91qMRGRNFDorsG4meMYsNsA9tt9vxq3LQ/dXbqkuCgRkTQys0zgTuB4oD9wtpn1r7DZD4Fv3D0fuA34Q7T8DKCFuw8ADgR+Uh7IY6eRbhFJI4XuasxZOYe3l7zN+fuej5nVuL1GukWkkRoMzHP3L9x9M/AIMLLCNiOBcdHjCcAxFj44HdjFzJoBrYDNwJr0lF2DNm1gt90UukUkLRS6qzF+5ngyLINzB5yb1PaFheFaC506pbgwEZH06gYsSni+OFpW6TbuXgKsBjoQAvh6YBmwELjV3VeluuCk5eWpvURE0kKhuwplXsb9H97Pd/O+S5c2yfWLFBaGwJ2ZmeLiREQajsFAKdAV6A1cZmZ7VNzIzMaY2XQzm75ixYr0VZefr5FuEUmLpEK3mT1hZieaWZMJ6S9/+TKL1ixK6gTKcrowjog0UkuAHgnPu0fLKt0maiXJAb4GzgGecfct7r4ceBMYVPEF3H2suw9y90Gd0vlzYV4eLF4MGzem7zVFpElKNkTfRfjg/MzMfm9m/VJYU70wbuY4clrkcFK/k5LeR6FbRBqpd4E+ZtbbzJoD3wMmVthmIlA+SnE68JK7O6Gl5GgAM9sFOAT4NC1VJ6P8ZMovvoi3DhFp9JIK3e7+grufCxwAzAdeMLO3zOz7Zlb95NUN0LrN63j8k8c5c+8zaZXVKun9FLpFpDGKerR/BjwLfAI86u6zzewGMysfmbgH6GBm84BLgfJpBe8EWpvZbEJ4v8/dP0zvO6jGvvuG+/ffj7cOEWn0miW7oZl1AM4DRgEfAA8ChxFGNoamori4rN20ljP6n8EP9v9B0vu4K3SLSOPl7lOAKRWWXZfwuJgwPWDF/dZVtrze6N8/zGLy1lswalTc1YhII5ZU6Daz/wD9gPuBEe6+LFr1bzObnqri4tKlTRf+dfK/arVPURFs3qzQLSLSoGRmwiGHhNAtIpJCyfZ03+7u/d39dwmBGwB33+6EmKZIc3SLiDRQBQXw0Uewpn5MHy4ijVOyobu/mbUrf2Jm7c3sf1JUU4O0LPpTRKFbRKSBKSgIPYJvvx13JSLSiCUbun/s7kXlT9z9G+DHqSmpYdJIt4hIA3XwwWCmFhMRSalkQ3emJVwH3cwygeapKalhUugWEWmgcnJgn31g6tS4KxGRRizZ0P0M4aTJY8zsGODhaJlECguhRYvw2S0iIg1MQUEI3WVlcVciIo1UsqH7l8DLwEXR7UXgilQV1RAVFkKXLuEXShERaWCGDAknUn78cdyViEgjlezFccrc/W53Pz26/cPdS2vaz8yGmdkcM5tnZldWsr6nmb1oZh+a2Stm1j1h3S1mNtvMPjGz28vbW6Lt5pjZjOi2W23ecKpojm4RkQasoCDcq69bRFIkqdBtZn3MbIKZfWxmX5Tfatgnk3AlsuOB/sDZZta/wma3AuPdfSBwA/C7aN8C4FBgILAPcBBwZMJ+57r7ftFteTLvIdUUukVEGrD8fOjYUaFbRFIm2faS+4C7gRLgKGA88EAN+wwG5rn7F+6+GXgEGFlhm/7AS9HjlxPWO9CScLJmCyAL+CrJWmOh0C0iDYGZXWJmbS24x8zeN7Pj4q4rdmZhtFuhW0RSJNnQ3crdXwTM3Re4+6+BE2vYpxuwKOH54mhZopnAqdHjU4A2ZtbB3acSQviy6Pasu3+SsN99UWvJtYmzqsRlyxZYuVKhW0QahB+4+xrgOKA9MAr4fbwl1RMFBfDZZ7BiRdyViEgjlGzo3mRmGcBnZvYzMzsFaF0Hr385cKSZfUBoH1kClJpZPrAX0J0Q1I82s8Ojfc519wHA4dFtVGUHNrMxZjbdzKavSPEH6IoV4boKCt0i0gCUD1ScANzv7rMTljVt5X3d06bFW4eINErJhu5LgGzg58CBwHnA6Br2WQL0SHjePVq2lbsvdfdT3X1/4OpoWRFh1Huau69z93XA08CQaP2S6H4t8BChjWU77j7W3Qe5+6BOnTol+TZ3jOboFpEG5D0ze44Qup81szaA5skDGDQImjVTi4mIpESNoTs6IfKsKAAvdvfvu/tp7l7TUMC7QB8z621mzYHvARMrHLtjNIIO8Cvg3ujxQsIIeDMzyyKMgn8SPe8Y7ZsFDAdmJfleU0ahW0QakB8CVwIHufsGwjkz34+3pHqiVSs44ACFbhFJiRpDdzQ14GG1PbC7lwA/A54FPgEedffZZnaDmZ0UbTYUmGNmc4HOwE3R8gnA58BHhL7vme4+iXBS5bNm9iEwgzBy/s/a1lbXFLpFpAEZAsxx9yIzOw+4Blgdc031x5Ah8M474WQdEZE61CzJ7T4ws4nAY8D68oXu/kR1O7n7FGBKhWXXJTyeQAjYFfcrBX5SyfL1hPaWeqU8dHfuHG8dIiJJuBvY18z2BS4D/h9hRqojq92rqSgogL/+FWbMgIMOirsaEWlEku3pbgl8DRwNjIhuw1NVVEOzbBm0awctW8ZdiYhIjUrc3QlTtN7h7ncCbWKuqf7QRXJEJEWSGul2d/X7VUNzdItIA7LWzH5FmPnp8Oi8mqyYa6o/uneHHj1C6L7kkrirEZFGJKnQbWb3ES5Ysw13/0GdV9QAKXSLSANyFnAOYb7uQjPLBf4Yc031iy6SIyIpkGx7yWTgqej2ItAWWJeqohoahW4RaSjcvRB4EMgxs+FAsbuPj7ms+qWgABYtCjcRkTqSbHvJ44nPzexh4I2UVNQAFRZCly5xVyEiUjMzO5Mwsv0K4aI4fzOzX0Qntgt829c9dWpoNRERqQPJzl5SUR9gt7ospKFaty7cNNItIg3E1YQ5upcDmFkn4AUqmUmqydp33zBn91tvwZlnxl2NiDQSyfZ0r2Xbnu5C4JcpqaiB+eqrcK/QLSINREZ54I58TfKthk1DVlaYLlB93SJSh5JtL9F0UlXQhXFEpIF5xsyeBR6Onp9FhespCKHF5NZbYePGMOotIrKTkhrdMLNTzCwn4Xk7Mzs5dWU1HArdItKQuPsvgLHAwOg21t31y2VFBQVQUgLTp8ddiYg0Esn+pHi9u2+9TLC7FwHXp6akhkWhW0QaGnd/3N0vjW7/ibueemnIkHCvFhMRqSPJnkhZWTjf0ZMwG5XCQsjMhA4d4q5ERKRqlZybs3UV4O7eNs0l1W8dO0LfvgrdIlJnkg3O083sz8Cd0fOfAu+lpqSGpbAQdtstBG8RkfpK5+bsgIICmDwZ3MEs7mpEpIFLtr3kYmAz8G/gEaCYELybPF0YR0SkkRoyBFauhHnz4q5ERBqBZGcvWQ9cmeJaGiSFbhGRRqr8IjlvvQV9+sRbi4g0eMnOXvK8mbVLeN4+mnKqyVu2TKFbRKRR6t8f2rZVX7eI1Ilk20s6RjOWAODu36ArUlJWFi6Oo9AtItIIZWSEFpOpU+OuREQagWRDd5mZ5ZY/MbNeVH4WfJOyalWYxlWhW0SkkSoogFmzYPXqmrcVEalGsrOXXA28YWavEqaXOhwYk7KqGgjN0S0i0sgVFITZS95+G447Lu5qRKQBS2qk292fAQYBcwiXDr4M2JjCuhqE8tDdpUu8dYiISIoMHhzaTNTXLSI7KamRbjP7EXAJ0B2YARwCTAWOTl1p9Z9GukVEGrm2bWHAAIVuEdlpyfZ0XwIcBCxw96OA/YGi6ndp/BS6RUSagCFDYNo0KC2NuxIRacCSDd3F7l4MYGYt3P1ToF/qymoYCgshOxtat467EhERSZmCAli7FmbPjrsSEWnAkg3di6N5up8Enjez/wILUldWw1B+YRxdHVhEGjszG2Zmc8xsnpltd7E0M2thZv+O1r8dzXJVvm6gmU01s9lm9pGZtUxn7Tut/CI5mjpQRHZCsidSnuLuRe7+a+Ba4B7g5FQW1hDoapQi0hSYWSZwJ3A80B8428z6V9jsh8A37p4P3Ab8Idq3GfAAcKG77w0MBbakqfS6sccesNtu6usWkZ2S7Ej3Vu7+qrtPdPfNqSioIVHoFpEmYjAwz92/iD77HwFGVthmJDAuejwBOMbMDDgO+NDdZwK4+9fu3rCao83CaLdCt4jshFqHbvmWQreINBHdgEUJzxdHyyrdxt1LgNVAB6Av4Gb2rJm9b2ZXpKHeuldQAPPmwfLlcVciIg2UQvcO2rwZvv5aoVtEpAbNgMOAc6P7U8zsmIobmdkYM5tuZtNXrFiR7hprpr5uEdlJCt07qHywQ6FbRJqAJUCPhOfdo2WVbhP1cecAXxNGxV9z95XuvgGYAhxQ8QXcfay7D3L3QZ06dUrBW9hJBx4IWVlqMRGRHabQvYOWLQv3Ct0i0gS8C/Qxs95m1hz4HjCxwjYTgdHR49OBl9zdgWeBAWaWHYXxI4GP01R33WnZEg44QKFbRHaYQvcO0oVxRKSpiHq0f0YI0J8Aj7r7bDO7wcxOija7B+hgZvOAS4Ero32/Af5MCO4zgPfd/al0v4c6UVAA06eH/kIRkVpK6jLwsj2FbhFpStx9CqE1JHHZdQmPi4Ezqtj3AcK0gQ1bQQHcdhvMmAGDB8ddjYg0MBrp3kHlobtz53jrEBGRNCk/mVItJiKyAxS6d1BhIXToAM2bx12JiIikRdeu0LOnQreI7BCF7h2kObpFRJqgggJ4801wj7sSEWlgFLp3kEK3iEgTVFAAS5fCokU1bysikiClodvMhpnZHDObZ2ZXVrK+p5m9aGYfmtkrZtY9Yd0tZjbbzD4xs9ujywljZgea2UfRMbcuTzeFbhGRJmjIkHCvFhMRqaWUhW4zywTuBI4H+gNnm1n/CpvdCox394HADcDvon0LgEOBgcA+wEGEuV0B7gZ+DPSJbsNS9R6q4q7QLSLSJA0cCNnZujKliNRaKke6BwPz3P0Ld98MPAKMrLBNf+Cl6PHLCesdaAk0B1oAWcBXZtYFaOvu06KLLowHTk7he6jUunWwYYNCt4hIk5OVFaYL1Ei3iNRSKkN3NyCx6W1xtCzRTODU6PEpQBsz6+DuUwkhfFl0e9bdP4n2X1zDMVNOc3SLiDRhBQXwwQewfn3clYhIAxL3iZSXA0ea2QeE9pElQKmZ5QN7Ad0JofpoMzu8Ngc2szFmNt3Mpq9YsaJOi1boFhFpwgoKoLQ0XJ1SRCRJqQzdS4AeCc+7R8u2cvel7n6qu+8PXB0tKyKMek9z93Xuvg54GhgS7d+9umMmHHusuw9y90GdOnWqq/cEKHSLiDRphxwS7tViIiK1kMrQ/S7Qx8x6m1lz4HvAxMQNzKyjmZXX8Cvg3ujxQsIIeDMzyyKMgn/i7suANWZ2SDRryfnAf1P4Hiql0C0i0oR16AB77qnQLSK1krLQ7e4lwM+AZ4FPgEfdfbaZ3WBmJ0WbDQXmmNlcoDNwU7R8AvA58BGh73umu0+K1v0P8P+AedE2T6fqPVRl2TJo1gx23TXdrywiIvXCkCFhBhNdJEdEktQslQd39ynAlArLrkt4PIEQsCvuVwr8pIpjTidMIxibwkLo3Bky4u6IFxGReBQUwH33wWefQd++cVcjIg2AYuMO0BzdIiJNXEFBuFeLiYgkSaF7Byh0i4g0cXvuCe3aKXSLSNIUuneAQreISBOXkQFHHQWPPgpLKp1ES0RkGwrdtVRaCsuXQ5cucVciIiKx+sMfYPNm+MlPdEKliNRIobuWvv46BG+NdIuINHF9+sDNN8NTT8H998ddjYjUcwrdtaQ5ukVEZKuf/xwOOwwuuQSWLo27GhGpxxS6a0mhW0REtsrIgHvvhU2bYMwYtZmISJUUumtJoVtERLahNhMRSYJCdy2Vh+7OneOtQ0RE6pGLL4ZDD1WbiYhUSaG7lgoLoXXrcBMREQEgMzNcobK4WLOZiEilFLprSXN0i4hIpcrbTCZPhgceiLsaEalnFLprSaFbRESq9POfhzaTn/9cbSYisg2F7lpatkyhW0REqpCZGWYzUZuJiFSg0F1LGukWEZFq9e2rNhMR2Y5Cdy0UF0NRkUK3iIjUILHNZNmyuKsRkXpAobsWvvoq3Ct0i4hItdRmIiIVKHTXgi6MIyIiSevbF266CSZNUpuJiCh010Z56O7SJd46RESkgbjkEigoUJuJiCh014ZGukVEpFZ00RwRiSh010JhIZhBp05xVyIiIg1GYpvJgw/GXY2IxEShuxYKC6FjR8jKirsSERFpUC65BIYMUZuJSBOm0F0LmqNbRER2SHmbycaNajMRaaIUumtBoVtERHZYv35w441qMxFpohS6a0GhW0REdsr//u+3bSZvvRV3NSKSRgrdSXJX6BYRkZ1U3maSkRGuWDlkCDz2GJSUxF2ZiKSYQneS1qwJMz4pdIuIyE7p1w/mz4fbb4fly+HMMyE/H267LXzZiEijpNCdpPKTzRW6RaQpMrNhZjbHzOaZ2ZWVrG9hZv+O1r9tZr0qrM81s3Vmdnm6aq7XWreGiy+GuXPhiSegRw+49FLo3h0uuwwWLKib1ykthfffhz//Gc46C6ZNq5vjikitKXQnSRfGEZGmyswygTuB44H+wNlm1r/CZj8EvnH3fOA24A8V1v8ZeDrVtTY4mZlwyinw+uvwzjswfDj89a+QlxdC8ttv1+54paXwwQdh1HzkyDDP7YEHhiD/n//AmDFhGxFJO4XuJCl0i0gTNhiY5+5fuPtm4BFgZIVtRgLjoscTgGPMzADM7GTgS2B2muptmA46CB56CL78Mox6P/ssHHJI6P1+/PHKw3JZGcyYsW3IPuCAsP8nn8AZZ4SZUhYvhgcegI8+gvHj0//eRIRmcRfQUCh0i0gT1g1YlPB8MXBwVdu4e4mZrQY6mFkx8EvgWECtJcno0QNuuQWuvTacdPmXv8Dpp0Pv3uEiO4ceCm+8Aa+8Aq+9Bt98E/bLzw/bHXUUHHkkdOu27XHPOCO0mVxzTegj32WXtL81kaZMoTtJhYXQvDm0bx93JSIiDcqvgdvcfV008F0pMxsDjAHIzc1NT2X1XZs2YWrB/9/evYdHVd37H39/DWAAEQJBQECgR09BQAMEpOUiglLkKMgdj/QIVq08evxRH6l4qaCVnzeqHttKhUqrVlHEIrQ/rFQNUlv0cClQFCyosQYIBgqI5VLQ7++PNaFDyCQBZmcmyef1PPuZyd5r7/nOnpmV76xZe60bb4SFC0PCPGnSv7affTaMGAH9+oUku1Wrso9nBjNmQJ8+oWX8rrsiDV9Ejqaku4KKhwss43+GiEh1tQVoHfd3q9i60soUmFktoCGwk9AiPtLMHgIaAV+Z2QF3/0n8zu4+C5gFkJubq+ka42VkwPDhYVmxAj78EHr3Lj/JLk3v3qEP+YMPwnXXQbNmyY9XREqlPt0VpDG6RaQGWwGcY2btzKwOMBZYVKLMIuDq2P2RwJse9HH3tu7eFngM+L8lE245Dt27w9ixJ5ZwF3vggTAG7rRpSQtLRMqnpLuClHSLSE3l7oeBZKV1QQAAG+hJREFUm4DXgA3APHd/z8zuNbMhsWJPEfpwbwZuAY4ZVlDSxL//O3z3uzB7NmzcmOpoRGoMdS+poMJCuKDkZUMiIjWEuy8GFpdYd3fc/QPAqHKOMS2S4OT4TZ0aRjG57bbQX1xEIhdpS3cFJlNoY2ZvmNk6M1tqZq1i6y8yszVxy4HYkFOY2S/N7OO4bTlRPgcIozQVFamlW0REqommTeH222HRInjrrVRHI1IjRJZ0V3AyhRnAM+5+HnAvcD+Au+e5e4675wD9gX3Akrj9Jhdvd/c1UT2HYkVFYShUJd0iIlJtTJoU+obfemv4JycikYqypbsikymcC7wZu59XynYIF+S86u77Iou0HBqjW0REqp26dWH6dFi5El58MdXRiFR7USbdpU2mUGKkftYCw2P3hwENzKxJiTJjgbkl1k2PdUl51MxOTVbAiSjpFhGRamncOMjJCV1NDhxIdTQi1VqqRy+5FbjQzP4MXEgY5/XIPLdm1gLoTLhivtjtQHugO9CYMNPZMczsejNbaWYri4qKTipIJd0iIlItnXIKPPwwfPIJ/EQjOYpEKcqku9zJFNx9q7sPd/cuwJ2xdbvjiowGFrj7obh9tsXGfj0I/ILQjeUY7j7L3XPdPbdp06Yn9US2bQu3mkNARESqnYsvhksvDV1Ndu5MdTQi1VaUSXe5kymYWbaZFcdwOzCnxDGupETXkljrNxbmE74CWB9B7EcpLITTT4d69aJ+JBERkRR46CH4/HO4775URyJSbUWWdFdwMoV+wAdm9legGTC9eH8za0toKS85ltFzZvYX4C9ANhB5DaGJcUREpFrr1AkmTICf/jRMMy8iSRfp5DgVmExhPjA/wb75HHvhJe7eP7lRlk9Jt4iIVHv33gtz58Idd2g0E5EIpPpCyipBSbeIiFR7Z54ZxuyeNw/eeSfV0YhUO0q6K6CwEFq0SHUUIiIiEZs8OYwacOut4J7qaESqFSXd5di3L1xbopZuERGp9k47LXQz+eMf4ZVXUh2NSLWipLsc27eHWyXdIiJSI1xzDXToALfdBocOlV9eRCpESXc5NDGOiIjUKLVqhSEEN22CJ59MdTQi1YaS7nIo6RYRkRrnP/4DLroI7rkH9uxJdTQi1YKS7nIo6RYRkRrHLEwPv2MHPPBAqqMRqRaUdJejsBBOOQVOciZ5ERGRqqVbN7jqKnjsMfj001RHI1LlKekuR2FhSLgzMlIdiYiISCWbPj0MHXjHHRpCUOQkKekuhybGERGRGqtNG7jlFvjVr2DUKCgqSnVEIlVWpNPAVwfbtinpFinNoUOHKCgo4MCBA6kOpVrIzMykVatW1K5dO9WhiBzthz+ERo3gBz+AZctg5kwYMSLVUYlUOUq6y1FYCOeem+ooRNJPQUEBDRo0oG3btphZqsOp0tydnTt3UlBQQLt27VIdjsjRMjLg+98PI5pcfTWMHAlXXgk//jE0aZLq6ESqDHUvKYO7upeIJHLgwAGaNGmihDsJzIwmTZroVwNJbx07wvLloeV7/nzo1Al+85tURyVSZSjpLsOuXWEyLiXdIqVTwp08OpdSJdSuDXfdBStWwBlnwJAhMH487N6d6shE0p6S7jJojG6R9LV7926eeOKJE9p38ODB7C4nSbj77rt5/fXXT+j4ItXe+eeHxPuuu8JFlp06wWuvpToqkbSmpLsMxUl3ixapjUNEjlVW0n348OEy9128eDGNGjUqs8y9997LxRdffMLxiVR7deqEribLl0PDhjBoEFx/Pezdm+rIRNKSku4yqKVbJH1NmTKFDz/8kJycHCZPnszSpUvp06cPQ4YM4dzY1c9XXHEF3bp1o2PHjsyaNevIvm3btmXHjh3k5+fToUMHrrvuOjp27MjAgQPZv38/AOPHj2f+/PlHyk+dOpWuXbvSuXNnNm7cCEBRURGXXHIJHTt25Nprr6VNmzbs2LGjks+ESIp17w6rVoWLLZ96Cjp3hjffTHVUImlHo5eUQUm3SMVMmgRr1iT3mDk5YSK8RB544AHWr1/PmtgDL126lNWrV7N+/fojI4DMmTOHxo0bs3//frp3786IESNoUmK0hU2bNjF37lxmz57N6NGjefnllxk3btwxj5ednc3q1at54oknmDFjBj//+c+555576N+/P7fffju/+93veOqpp5J3AkSqksxMePBBuOKKMMLJgAFw441hXf36qY5OJC2opbsMhYWhHjn99FRHIiIV0aNHj6OG3Hv88cc5//zz6dmzJ59++imbNm06Zp927dqRk5MDQLdu3cjPzy/12MOHDz+mzNtvv83YsWMBGDRoEFlZWUl8NiJV0De+Eb6BT5oETzwB550H8+ZB7BekSvHVV2ERSTNq6S5D8XCBGlRApGxltUhXpvpxLWpLly7l9ddfZ/ny5dSrV49+/fqVOiTfqaeeeuR+RkbGke4licplZGSU22dcpEarVw8efRSGDYNrroExY0Lr1YgRMG4cXHhhGPs7mQ4ehN//Hl5+GRYuDMe/7DIYOhQGDgwxiaSYWrrLoDG6RdJXgwYN2FvGBVt79uwhKyuLevXqsXHjRt55552kx9CrVy/mzZsHwJIlS9i1a1fSH0OkyurbFz74ICTDw4aFsb0HDICzzoLJk0OLuPuJH3/fPvj1r+Gqq6BpU7j8cliwICTbAweG+8OGhQl8hgwJ/c0/+yx5z0/kOCnpLoOSbpH01aRJE3r16kWnTp2YPHnyMdsHDRrE4cOH6dChA1OmTKFnz55Jj2Hq1KksWbKETp068dJLL9G8eXMaNGiQ9McRqbIyMuDii+GXvwz/VF94Abp1Cz+PdekSLrq8/3745JOKHW/v3nCMUaNCoj1iRBiqcPRoePXVkFQ/8ww89xwUFcHrr8N118HatXDtteGfeu/e8PDD8Ne/RvrURUoyP5lvmVVEbm6ur1y58rj3O+OM8HmeOTOCoESquA0bNtChQ4dUh5FSBw8eJCMjg1q1arF8+XImTpx45MLOE1HaOTWzVe6ee7KxViUnWmdLFbJjB7z0Uhjj+09/Cuv69g2t1qNGQfz1Ebt3w6JFoevIa6+FriTNm4dW7BEjQneVWuX0lnUPiffChWH585/D+vbtQxeUoUPhggvgFLVFyskpq85Wn+4EDh0KX5LV0i0iifztb39j9OjRfPXVV9SpU4fZs2enOiSRqiE7GyZODMtHH8Hzz4cE/Lvfhf/+bxg8GPr0CV1T3ngj/FNu1QpuuCEk2t/85vH1CzcLQyLl5MDUqfC3v4VEfuFC+NGPwigrzZqFbii33BKScZEkU9KdQHG3LyXdIpLIOeecw5+LW8xE5MR87WthZss774TVq0PXkLlz4ZVXwrZJk0Ki3b178lqizzoLbropLLt3h64pCxeG5P+pp2DCBJg2LST6Ikmi31ES0BjdIiIilcgs9Pd+5BH49FPIz4fNm+Ghh6Lt+tGoEVx5Zegr/vHHcPPN8OyzcM45YcKfv/89mseVGkdJdwJKukVERFKkVi1o06byx+xt2jQMd/jBB+HizBkzQmv7/feH0VJEToKS7gSUdIuIiNRQbdvC00+Hiy/79oU77oCzz4Ynnwz9y0VOgJLuBIqT7mbNUhuHiIiIpEjnzuGCyz/8IbR433ADdOwYZtnUrJdynJR0J1BYGLp5ZWamOhIRSYbTTjsNgK1btzJy5MhSy/Tr14/yhqp77LHH2Bf3M/PgwYPZvXt38gIVkfTTu3dIvBctgjp1wiybPXqEccBFKkhJdwKFhdCiRaqjEJFkO/PMM5k/f/4J718y6V68eDGNGjVKRmhpzcwGmdkHZrbZzKaUsv1UM3sxtv1dM2sbW3+Jma0ys7/EbvtXduwiSWEWZr1cuzZM9lNUBJdcEpZ0Glf+q69g+/YwEsxvfgM/+xn84AdwzTXwrW+FlvomTcJFq5Mnw+LFYdIhiZyGDExAs1GKpLcpU6bQunVrbrzxRgCmTZtGrVq1yMvLY9euXRw6dIj77ruPoUOHHrVffn4+l112GevXr2f//v1MmDCBtWvX0r59e/bv33+k3MSJE1mxYgX79+9n5MiR3HPPPTz++ONs3bqViy66iOzsbPLy8mjbti0rV64kOzubRx55hDlz5gBw7bXXMmnSJPLz87n00kvp3bs3f/rTn2jZsiULFy6kbt26lXeyTpKZZQA/BS4BCoAVZrbI3d+PK/YdYJe7n21mY4EHgTHADuByd99qZp2A14CWlfsMRJIoIwOuvjq0ds+cCdOnh+EML7ggzKqXlXXs0qjRsesS1QHuod/4P/4RLt7ct+9f90uu270btmwJS0FBuN227dh+56ecEloSW7aEr3899FPfsAEefzxcLJqREZ7DRRdB//5hHPR69aI/lzWMku4ECgvD+09Eyjfpd5NYU3jiMzGWJqd5Do8Neizh9jFjxjBp0qQjSfe8efN47bXXuPnmmzn99NPZsWMHPXv2ZMiQIViCERBmzpxJvXr12LBhA+vWraNr165Htk2fPp3GjRvz5ZdfMmDAANatW8fNN9/MI488Ql5eHtnZ2Ucda9WqVfziF7/g3Xffxd254IILuPDCC8nKymLTpk3MnTuX2bNnM3r0aF5++WXGjRuXhLNUaXoAm939IwAzewEYCsQn3UOBabH784GfmJm5e/xA5u8Bdc3sVHc/GH3YIhHKzITvfQ++850wzOHSpWHSnbVrYdeu8luPTz01JN8NG8Lhw0cn1l9+WfE46tcP44m3bBlm52zZ8tilWbPSZ+3cvz/MCJqXF5aHHw4jtdSpE75E9O8fEvGePUO8clKUdCeglm6R9NalSxc+++wztm7dSlFREVlZWTRv3pzvfe97LFu2jFNOOYUtW7awfft2mif4MC9btoybb74ZgPPOO4/zzjvvyLZ58+Yxa9YsDh8+zLZt23j//feP2l7S22+/zbBhw6hfvz4Aw4cP5w9/+ANDhgyhXbt25OTkANCtWzfy8/OTdBYqTUvg07i/C4ALEpVx98NmtgdoQmjpLjYCWF1awm1m1wPXA5x11lnJi1wkaqefHibSKenwYdizJyTgxcvu3cf+vXs31K4dWpbr1z/6trx1DRtCgwYnPrRi3bowYEBYAL74At5+G958MyThP/wh3HNP+ILRq1dIwLt3P/ZLQvFtaevib4sfMzPz6KXkutLK1K4dvjiUdlvWtrp1w2uUmVn5Q1CWEGnSbWaDgP8BMoCfu/sDJba3AeYATYG/A+PcvcDMLgIejSvaHhjr7q+YWTvgBUJlvgr4trv/M5lxf/FFWJR0i1RMWS3SURo1ahTz58+nsLCQMWPG8Nxzz1FUVMSqVauoXbs2bdu25cCBA8d93I8//pgZM2awYsUKsrKyGD9+/Akdp9ipcS1EGRkZR3VjqSnMrCOhy8nA0ra7+yxgFkBubq5XYmgi0ahVK/SdbtIk1ZFU3GmnwaBBYYHwhWDZspCAv/lmmDm0LJmZ//pSEP9lISsrtLibhdb1AwdCorVjR7hfvC5+SbZatULy3bBhuC1eSv4dv65HDzjzzOSFkLQjlVDBPoAzgGfc/enYxTX3E5LoPCAndpzGwGZgSWyfB4FH3f0FM/sZoR/hzGTG/o9/wDe+ESajEpH0NWbMGK677jp27NjBW2+9xbx58zjjjDOoXbs2eXl5fPLJJ2Xu37dvX55//nn69+/P+vXrWbduHQCff/459evXp2HDhmzfvp1XX32Vfv36AdCgQQP27t17TPeSPn36MH78eKZMmYK7s2DBAp599tlInncKbAFax/3dKrautDIFZlYLaAjsBDCzVsAC4L/c/cPowxWRpGjUCIYMCQuEJPm990JyHZ9U168fWpQzMpLzuO7wz38enYwfOhRa2ONvy1t36FA4xuefH73s2RNut20LEyEVrztY4ke4F18MkyQlSZQt3RXpA3gucEvsfh7wSinHGQm86u77LHTM7A/8Z2zb04Q+hElNups1C12cRCS9dezYkb1799KyZUtatGjBVVddxeWXX07nzp3Jzc2lffv2Ze4/ceJEJkyYQIcOHejQoQPdunUD4Pzzz6dLly60b9+e1q1b06tXryP7XH/99QwaNIgzzzyTvLy8I+u7du3K+PHj6dGjBxAupOzSpUtV7EpSmhXAObFfGrcAY/lXPVxsEXA1sJxQb7/p7m5mjYD/B0xx9z9WYswikmzZ2aHfeNTMQh/yyu5HfvBg6ItfnJwnuaubuUfzK56ZjQQGufu1sb+/DVzg7jfFlXkeeNfd/8fMhgMvA9nuvjOuzJvAI+7+WzPLBt5x97Nj21oTEvJOZcWSm5vr5Y29KyLHZ8OGDXTo0CHVYVQrpZ1TM1vl7rkpCik+jsHAY4TugnPcfbqZ3QusdPdFZpYJPAt0IXQXHOvuH5nZXcDtwKa4ww10988SPZbqbBGpqsqqs1N9IeWthCvcxwPLCC0oRy7ZNbMWQGfCEFPHRRfliIgkj7svBhaXWHd33P0DwKhS9rsPuC/yAEVE0lyUk+OU2wfQ3be6+3B37wLcGVsXP7XbaGCBuxcPOLkTaBTrL1jqMeOOPcvdc909t2nTpif/bERERERETlCUSfeRPoBmVofQB3BRfAEzyzaz4hhuJ4xkEu9KYG7xHx76wuQR+gtC6D+4MILYRURERESSJrKk290PAzcRuoZsAOa5+3tmdq+ZxS6DpR/wgZn9FWgGTC/ePzaFcGvgrRKHvg24xcw2E4YNfCqq5yAiZYvqmpCaSOdSRKR6i7RPdwX6AM4nzFxW2r75lDJVcGw0lB5JDVREjltmZiY7d+6kSZMmCWd8lIpxd3bu3ElmZmaqQxERkYik+kJKEamiWrVqRUFBAUVFRakOpVrIzMykVatWqQ5DREQioqRbRE5I7dq1adeuXarDEBERqRKivJBSRERERERQ0i0iIiIiEjkl3SIiIiIiEYtsGvh0YmZFwCepjiNONrAj1UHEUTzlS7eY0i0eSL+Yqks8bdy9Rs3wpTq7XOkWD6RfTIqnfOkWU7rFAycWU8I6u0Yk3enGzFa6e26q4yimeMqXbjGlWzyQfjEpHkmWdHvt0i0eSL+YFE/50i2mdIsHkh+TupeIiIiIiERMSbeIiIiISMSUdKfGrFQHUILiKV+6xZRu8UD6xaR4JFnS7bVLt3gg/WJSPOVLt5jSLR5Ickzq0y0iIiIiEjG1dIuIiIiIRExJdwTMrLWZ5ZnZ+2b2npn9n1LK9DOzPWa2JrbcXQlx5ZvZX2KPt7KU7WZmj5vZZjNbZ2ZdI4zl63HPfY2ZfW5mk0qUifwcmdkcM/vMzNbHrWtsZr83s02x26wE+14dK7PJzK6OMJ6HzWxj7DVZYGaNEuxb5uub5JimmdmWuNdmcIJ9B5nZB7H31JQI43kxLpZ8M1uTYN+kn6NEn/dUvo/k+KjOrlAsqrMrHo/q7PLjqZl1trtrSfICtAC6xu43AP4KnFuiTD/gt5UcVz6QXcb2wcCrgAE9gXcrKa4MoJAwtmWlniOgL9AVWB+37iFgSuz+FODBUvZrDHwUu82K3c+KKJ6BQK3Y/QdLi6cir2+SY5oG3FqB1/VD4GtAHWBtyc9BsuIpsf1HwN2VdY4Sfd5T+T7SkpzXsEQZ1dn/elzV2WXHozq7nHhKbK8xdbZauiPg7tvcfXXs/l5gA9AytVFVyFDgGQ/eARqZWYtKeNwBwIfuXumTYbj7MuDvJVYPBZ6O3X8auKKUXb8F/N7d/+7uu4DfA4OiiMfdl7j74dif7wCtTvZxTjamCuoBbHb3j9z9n8ALhHMbWTxmZsBoYO7JPs5xxJPo856y95EcH9XZx011dhnxqM6ueDw1rc5W0h0xM2sLdAHeLWXzN8xsrZm9amYdKyEcB5aY2Sozu76U7S2BT+P+LqBy/vGMJfEHrrLPEUAzd98Wu18INCulTKrO1TWElq3SlPf6JttNsZ9P5yT4GS4V56gPsN3dNyXYHuk5KvF5T+f3kSSgOrtCVGdXnOrsstWoOltJd4TM7DTgZWCSu39eYvNqwk9z5wM/Bl6phJB6u3tX4FLgRjPrWwmPWSYzqwMMAV4qZXMqztFRPPyelBZD/JjZncBh4LkERSrz9Z0J/BuQA2wj/DyYDq6k7BaTyM5RWZ/3dHofSWKqs8unOrviVGdXSI2qs5V0R8TMahNezOfc/dclt7v75+7+Rez+YqC2mWVHGZO7b4ndfgYsIPyUFG8L0Dru71axdVG6FFjt7ttLbkjFOYrZXvwTbez2s1LKVOq5MrPxwGXAVbHK4BgVeH2Txt23u/uX7v4VMDvBY1X2OaoFDAdeTFQmqnOU4POedu8jSUx1doWpzq4A1dnlq4l1tpLuCMT6KD0FbHD3RxKUaR4rh5n1ILwWOyOMqb6ZNSi+T7jQY32JYouA/7KgJ7An7qeWqCT8llvZ5yjOIqD4iuSrgYWllHkNGGhmWbGf6QbG1iWdmQ0Cvg8Mcfd9CcpU5PVNZkzx/UaHJXisFcA5ZtYu1jo2lnBuo3IxsNHdC0rbGNU5KuPznlbvI0lMdfZxUZ1dDtXZFVbz6mxP8lWzWhygN+FniXXAmtgyGLgBuCFW5ibgPcLVwe8A34w4pq/FHmtt7HHvjK2Pj8mAnxKuXv4LkBtxTPUJFXLDuHWVeo4I/zy2AYcIfbO+AzQB3gA2Aa8DjWNlc4Gfx+17DbA5tkyIMJ7NhD5kxe+ln8XKngksLuv1jTCmZ2PvkXWEiqpFyZhifw8mXBn+YbJiKi2e2PpfFr934spGfo7K+Lyn7H2kJWmvoerso2NSnV2xeFRnlxNPbP0vqWF1tmakFBERERGJmLqXiIiIiIhETEm3iIiIiEjElHSLiIiIiERMSbeIiIiISMSUdIuIiIiIRExJt0jEzKyfmf021XGIiEjFqN6WKCjpFhERERGJmJJukRgzG2dm/2tma8zsSTPLMLMvzOxRM3vPzN4ws6axsjlm9o6ZrTOzBbGZqTCzs83sdTNba2arzezfYoc/zczmm9lGM3uueNY2ERE5caq3pSpR0i0CmFkHYAzQy91zgC+BqwgzsK10947AW8DU2C7PALe5+3mEWb6K1z8H/NTdzwe+SZiFC6ALMAk4lzDLVq/In5SISDWmeluqmlqpDkAkTQwAugErYo0ZdYHPgK+AF2NlfgX82swaAo3c/a3Y+qeBl8ysAdDS3RcAuPsBgNjx/tfdC2J/rwHaAm9H/7RERKot1dtSpSjpFgkMeNrdbz9qpdkPSpTzEzz+wbj7X6LPnojIyVK9LVWKupeIBG8AI83sDAAza2xmbQifkZGxMv8JvO3ue4BdZtYntv7bwFvuvhcoMLMrYsc41czqVeqzEBGpOVRvS5Wib20igLu/b2Z3AUvM7BTgEHAj8A+gR2zbZ4T+gwBXAz+LVc4fARNi678NPGlm98aOMaoSn4aISI2heluqGnM/0V9dRKo/M/vC3U9LdRwiIlIxqrclXal7iYiIiIhIxNTSLSIiIiISMbV0i4iIiIhETEm3iIiIiEjElHSLiIiIiERMSbeIiIiISMSUdIuIiIiIRExJt4iIiIhIxP4/hIn1b3mvl0AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# define figure\n",
    "fig = plt.figure(figsize=(12,5))\n",
    "\n",
    "# plot training and validation accuracy\n",
    "ax = fig.add_subplot(1,2,1)\n",
    "ax.plot(model_log[:,0]+1, model_log[:,3], color='b', label='training ')\n",
    "ax.plot(model_log[:,0]+1, model_log[:,5], color='g', label='validation')\n",
    "ax.set_title('Accuracy')\n",
    "ax.set_xlabel('epoch')\n",
    "ax.set_ylabel('accuracy')\n",
    "ax.set_ylim(bottom=None, top=1)\n",
    "ax.legend(loc='lower right')\n",
    "\n",
    "# plot validation loss\n",
    "ax = fig.add_subplot(1,2,2)\n",
    "plt.plot(model_log[:,0]+1, model_log[:,4], color='r', label='validation')\n",
    "ax.set_title('Validation Loss')\n",
    "ax.set_xlabel('epoch')\n",
    "ax.set_ylabel('loss')\n",
    "ax.set_ylim(bottom=None, top=None)\n",
    "\n",
    "# show figure\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ws8LX56FJXji"
   },
   "source": [
    "Both plots show that huge improvement was make until epoch 12, then it was still improving, but at a slower pace, completely normal behaviour. The accuracy plot also shows the gap between training and validation, anyway validation achieved pretty high accuracy of 99.43%, what is probably higher than a human reading handwritten digits.\n",
    "\n",
    "It could still be improved if we had some more epochs, or even better a more robust model, but in this case the cost of processing could not worth it.\n",
    "\n",
    "To finish it, below we select the best epoch to check its details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 386,
     "status": "ok",
     "timestamp": 1655136917611,
     "user": {
      "displayName": "Sidclay da Silva",
      "userId": "08174217452753453351"
     },
     "user_tz": 180
    },
    "id": "tVVt-j1Bmu57",
    "outputId": "1b93459e-e88d-427d-a459-3fdf37657189"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Epoch\t 20 / 20 \n",
      "Train. Accuracy\t 0.9995 \n",
      "Valid. Accuracy\t 0.9943 \n",
      "Valid. Loss\t 0.0192\n"
     ]
    }
   ],
   "source": [
    "# select the epoch with minimum validation loss\n",
    "best_epoch = model_log[model_log[:,4]==model_log[:,4].min()].reshape(6,)\n",
    "\n",
    "# print best epoch details\n",
    "print('Best Epoch\\t', int(best_epoch[0]+1), '/', num_epochs,\n",
    "      '\\nTrain. Accuracy\\t',round(best_epoch[3],4),\n",
    "      '\\nValid. Accuracy\\t',round(best_epoch[5],4),\n",
    "      '\\nValid. Loss\\t',round(best_epoch[4],4),)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "authorship_tag": "ABX9TyOs8goB102RWB59602T1vt+",
   "collapsed_sections": [],
   "mount_file_id": "11a-aM-9xszDIOJLow9kuCDtVKNHCBw5I",
   "name": "DigitClassification.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}