11_training_deep_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 11 – Training Deep Neural Networks**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code and solutions to the exercises in chapter 11._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/ageron/handson-ml2/blob/master/11_training_deep_neural_networks.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "%load_ext tensorboard\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"deep\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vanishing/Exploding Gradients Problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logit(z):\n",
    "    return 1 / (1 + np.exp(-z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure sigmoid_saturation_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABO30lEQVR4nO3dd3gUVdvA4d9JIR1CKEE6SEeKNCkKEYSAqIiAglRREVBfpAgqgoC8qIAUP5RXFASJiCJFerGEjhIwEVCI0kvoBFNJ2fP9MZuYsiEhbDKb5Lmva67szpydeXYyu8+emTPnKK01QgghhKNxMjsAIYQQwhZJUEIIIRySJCghhBAOSRKUEEIIhyQJSgghhEOSBCWEEMIhSYISd0UpFayUmmd2HJCzWJRSh5VSk/IppLTbXayUWp8P2wlQSmmlVOl82NYQpdQZpZTFjH2aIZZBSqloM2MQ9qfkPiiRFaVUGWAy8ChwDxAJHAbe11pvs5bxAxK11lFmxZkiJ7EopQ4D32mtJ+VRDAHAz0AZrfXVNPNLYHzeIu24rVPAPK31zDTzigF+wCWdhx9upVRJ4DIwCvgOiNJa50uCUEppoJfW+rs08zwAH6315fyIQeQPF7MDEA5tJeAJPA/8DZQF2gGlUgpora+bE1pmjhRLRlrrm/m0nQTgYj5sqgrG98d6rXVEPmzvtrTWcUCc2XEIO9NayyRTpgnwBTTwSDblgjF+xac89wfWYnxZnAaew6h1TUpTRgPDgO+BWCAceBioCGwBYoBQoEmGbT0FHAJuAWeB8VjPAmQRS1nrNlJiGZwxFhvv517ray5a4zgIPJahTDFgmnWdt4ATwH+Aqtb3lnZabH3NYowvc4CXgEuAS4b1LgO+z0kc1veablvW+QHW56XvYL+dAt4GPgX+Ac4Br99mHw2y8T6rApOAwzbKRqd5Psn6P+gNHAeigDVp47WWG5gm5ktp9uOpDNs9ZWs7afbz30CC9e+LGZZrYAiwwrqPTwD9zP7syfTvJNegRFairdMTSin3O3jdEoxf1+2BbkA/6/OM3gaWA42AEOBrYCHwCXA/cAHjSx0ApVRTjC+SVUAD4A3gTeCV28SyGKgBPAI8CQzA+CK9HW9gE9DRGttKYJVSqk6G9zgA4/RWXYwaZiTGl38Pa5n6GKdFR9jYxrcYPwAeSfP+vDD2V1AO43gKI5FMsW7nHltv5g7220iMhNAE+ACYrpRqZWudwDdAZ+vjFtZtn82irC1VgWeA7kAnjP/3f9PE/BJGsvwCaIhxivmIdXFz698XrdtNeZ6OUqo7MA+YA9wHzAU+UUo9nqHoRIwfAo2s72uRUsrW8SrMYHaGlMlxJ4wv2+tAPLAXmAk8kKFMMNZaC1Ab41dpyzTLKwHJZK5BvZfm+X3WeaPSzAsgTU0A+Ar4KcO2JwHnsoillvX1bdIsr5Ixlhzuh33A29bHNa3r7ZxF2XRxp5m/GGsNyvp8NbA0zfN+wE3APSdxWJ+fAsbcbvs53G+ngK8zlPkr7bZsxNLMup2qGdabkxpUPFAizbzxwN9pnp/DuM6Z1bY10DOb7ewGFtn4H+y6zXHoglGjl1qUg0xSgxJZ0lqvBMoDj2P8mm8N7FNKvZXFS+oAFowaUco6zmLUhjL6Pc3jS9a/h2zMK2v9WxfjSyetXUAFpVRxG+uva43l1zSxnM4illRKKS+l1HSl1B9KqRvWlmHNgMrWIvdb1/vz7daTA0HAk0opT+vzvhiNN+JzGEdO5XS//Z6hzAX+3ff2dlqnvyaXui2lVFmgAvDjXW4jq/ddL8O81PettU4CrpB371vcIUlQ4ra01vFa621a6yla69YYp+EmWVuLZaTuYNWJaTdzm3kpx6hKMy9TmHcZS1ozgV7ABIwGIY0xklzK+83tejNaDyQB3axfyo/w7+m9nMSRUzndb4k2lt3p94OFzPvH1Ua5223LXvs3Zb3ZzbPH+xZ5RP4R4k79gXEqxNZ1qT8xjqmmKTOUUhUxamH22O6DGeY9iHGqylaz8pRYUq9RKKUq5yCWB4EvtdYrtda/Y5xuujfN8oPW9T6cxesTrH+db7cRrfUtjObZfTGux1wEtt9BHCnbuu12uPP9djeuAP5KqbRJpvGdrEBrfQk4D3S4TbFEsn/ff2L7ff9xJ/EIc0mCEjYppUoppX5SSvVTSjVUSlVTSvUCxgI/aq3/yfgarfUxjFZ4/1NKtVRKNca40B1L1r/ic+pDoJ1SapJSqpZSqi8wGphuq7A1ls3Ap0qpVtZYFpN9U+RwoLtSqolSqgFGrSY1GWut/8Jo5PC5UqqHdb88pJTqby1yGuO9dlVKlVFKed9mW0FAIDAUWKa1tuQ0DqtTwENKqQq3uTH3jvbbXQrGuAfrLaXUvUqp54GeuVjPf4HXlFIjrTE3VkqNTrP8FNBBKVXOej+WLTOA/kqpl5VSNZVSr2L8GMiL9y3yiCQokZVojIvyIzB+2R/BaFq9DOMXf1YGYfzaD8Zobv4Vxg2d8XcTjNb6IMYprx5Ybxa2TrfrOWIQcBL4CVhnjf1UNpsaZY13J8Z1t33Wx2kNsK7rI+AoRuIrYY3zPPAOxpfspWzi24FRW6hH+tN7OY1jIkYjlOMYtZdMcrnfckVr/SfG7QNDMK7tdMQ4Zu50PfOBlzFa6h3G+KFRP02R0Rg12LPAb1msYw3wKkbrxD8wjuPhWut1dxqPMI/0JCHylPWX/QWgj7XRhRBC5Ij0JCHsSinVHvDBaJFXFqMmcRXjV7AQQuSYXU/xKaVeUUqFKKVuKaUW36bcQKXUAaXUP0qpc9bmtJIsCwdXYCpGglqHcc2nrdY6xtSohBAFjl1P8SmlnsJoahoIeGitB2VRbhjGueVfgDIY1ypWaK3ft1swQgghCjS71lq01qsAlFLNMPpVy6rc/DRPzyulviLrZrtCCCGKIEc5rdaWf/vaSkcpNQSjVRAeHh5NK1WqlJ9x5YjFYsHJSRpE5oTsq+ydPXsWrTWVK99ppxFFk5nHVLJOxllld0uW43DUz194ePhVrXWZjPNNT1BKqecwunB5wdZyrfUCYAFAs2bNdEhIiK1ipgoODiYgIMDsMAoE2VfZCwgIIDIyktDQULNDKRDy85iKToim36p+TA6YTKNyjfJlm/bkqJ8/pdRpW/NNTaVKqScx7snootMM7iaEEI4mITmBHt/2YF34Ok7ftPl9KuzMtBqUUqoz8BnQVWt9KLvyQghhFou2MHDNQLYe38rCJxbyRO0nzA6pSLBrgrI2FXfB6CfL2TqOUJK1l+C05dpj9DDQXWv9a+Y1CSGEY9BaM2LTCJYfXs77Hd5n8P2DzQ6pyLD3Kb63Me57eQNjfJs44G2lVGWlVLS1s04wemguAWy0zo9WSm2ycyxCCHHXEpITCL8ezqiWoxjbZqzZ4RQp9m5mPgljQDJbvNOUkyblQgiHZ9EW3FzcWN9nPc5OzqTvqF3kNcdrbyiEEA5g5R8reXDRg1yLvYarsytOSr4u85vscSGEyODnkz/z7KpnUUrh4ephdjhFliQoIYRI42DEQbot70ZNv5qs67MOT1dPs0MqsiRBCSGE1V/X/qJzUGdKepRkS78t+Hn4mR1SkSYJSgghrFycXLjX71629ttKheIVzA6nyDO9qyMhhDBbTEIMHq4eVCtZjT2D90hrPQchNSghRJEWlxhH568689K6lwAkOTkQSVBCiCIryZLEM989w+4zu+l4b0ezwxEZyCk+IUSRpLXmxXUvsi58HZ88+glP13/a7JBEBlKDEkIUSeN/Gs/i0MW80+4dhjUfZnY4wgapQQkhiqQO1TqQkJzAO+3eMTsUkQVJUEKIIuVU5Cmq+lalQ/UOdKjewexwxG3IKT4hRJGxPnw9tf6vFqv+XGV2KCIHJEEJIYqEXWd20WtFLxqVa0TH6tJiryCQBCWEKPQOXTrE418/TuUSldn47EZ83HzMDknkgCQoIUShFhkfSWBQIJ6unmztt5UyXmXMDknkkDSSEEIUar7uvrzd9m3aVmlLFd8qZocj7oAkKCFEoRR1K4qTkSdp6N+Q4c2Hmx2OyAU5xSeEKHRuJd2i+zfdCVgcQGR8pNnhiFySGpQQolBJtiTTb3U/fjz5I0ueXIKvu6/ZIYlckhqUEKLQ0FrzysZX+O6P7/iw04cMaDTA7JDEXZAEJYQoNJYdWsb/DvyPcW3GMarVKLPDEXdJTvEJIQqNp+s/za3kWzzX+DmzQxF2YNcalFLqFaVUiFLqllJqcTZlRyqlLiqlbiqlFiml3OwZixCi6Nj01yYuRV/C1dmVwfcPlkEHCwl7n+K7AEwFFt2ukFIqEHgD6ABUBaoDk+0cixCiCAi5HkK35d0Y+8NYs0MRdqa01vZfqVJTgYpa60FZLF8GnNJav2V93gH4Smtd7nbr9fHx0U2bNk037+mnn2b48OHExsby6KOPZnrNoEGDGDRoEFevXqVnz56Zlg8bNoxnnnmGs2fP0r9//0zLR48ezeOPP86xY8d46aWXMi1/++23cXFxwdfXl9deey3T8mnTptG6dWv27NnDW2+9lWn5nDlzaNy4MT/88ANTp07NtPzTTz+ldu3arFu3jg8//DDT8qVLl1KpUiW++eYb5s+fn2n5d999R+nSpVm8eDGLFy/OtHzjxo14enryySef8O2332ZaHhwcDMDMmTNZv359umUeHh5s2rQJgHfffZcff/wx3fJSpUqxcuVKAN5880327t1LZGQkvr6+AFSsWJGgoCAAXnvtNUJDQ9O9vlatWixYsACAIUOGEB4enm5548aNmTNnDgD9+vXj3Llz6Za3atWK9957D4AePXpw7dq1dMs7dOjAhAkTAOjSpQtxcXHplj/22GOMGTMGgICAgEz7Jq+OvdDQUJKSkvj666+zPfYeeeQRQkNDi+yxt//8flovaI1brBuNf2uMS5Jx1cLWsZdWUT32Uj5/9vjes+ext3379gNa62YZy5l1Dao+8H2a52GAv1KqlNY63X9SKTUEGALg6upKZGRkuhWFh4cTHBxMfHx8pmUAR48eJTg4mJs3b9pcfuTIEYKDg7l8+bLN5YcOHcLHx4czZ87YXB4WFkbt2rX5+++/bS4/ePAgCQkJHD582ObykJAQIiMjCQsLs7n8l19+ISIigkOHDtlcvnfvXo4fP86RI0dsLt+9ezclSpTg6NGjNpfv2LEDd3d3wsPDbS5P+ZI4fvx4puVxcXGpy0+ePJlpucViSV2esv+Sk5NTy7m6uqYuP3fuXKbXX7hwIXX5hQsXMi0/d+5c6vJLly5lWn7mzJnU5VeuXOGff/5Jt/zkyZOpy69fv86tW7fSLT9+/Hjqclv7Jq+OvaSkJLTWOTr2XFxciuyx9+XGL3n1t1dxT3an8o7KRN+KTl1u69hLq6geeymfv7v93gsNDSMx0Y0jR85w6VJxkpM9sVjc0doDi8WNzz6L4vvvj3LyZALh4Y+jtRsWi7FMazeGD/ekWLHLXL5cjbNn3wdaZdoGmFeDOg68rLXebH3uCiQA1bTWp7Jab7NmzXRISIjd471bwcHBNn/liMxkX2UvICCAyMjITL/qRXqdgzrz28XfmFV/Fn279DU7nAIhODiYdu0CiIuD69dtTzduwD//QFSUMaV9nHayb+pQDlWDigaKp3me8jjKhFiEEAVQ0FNBRERFcO3Pa9kXLuTi4+HSJWO6eNGY0j6+etVIPhcvtiIqCjJU2HLFwwN8fP6dPD2Nebam2y3z8IDAQNvbMCtBHQEaASknnhsBlzKe3hNCiLRiEmL4cO+HvPHgG5T2LE1pz9IE/xlsdlh5Kj4ezp2DM2fST2fPGn8vXgQbZ+GyYDSWLlYMSpUCP7/MU8mSULx4+uST8bmPD7jcRfZISEjgq6++omfP/rjcZkV2TVBKKRfrOp0BZ6WUO5CktU7KUPRLYLFS6isgAngbWGzPWIQQhUticiK9VvRiy/EttK3SloCqAWaHZBdaw7Vr8Ndf6acTJ4wEdOlS9utwcQF/fyhXzvbfMmWMhHTs2F66dm2FpyeY1RL/5MmTPP744xw5coQOHTpQuXLlLMvauwb1NvBOmuf9gMlKqUXAH0A9rfUZrfVmpdR04GfAA1iZ4XVCCJHKoi0MXjuYTX9v4tPHPi2QyUlrI+EcPgyHDhl/w8ONZHS7GpCzM1SsCJUrZ54qVYLy5Y1aj1MObhq6ceMWXl52e0t3bMWKFQwePJjY2Fg8PDyyvV/NrglKaz0JmJTFYu8MZWcBs+y5fSFE4aO1ZszWMQT9HsTUh6cypOkQs0PKVmws/PabMaUko8OHjQYHtvj4QM2a6ad774UqVeCee4wkVZDFx8fz8ssvs3z5cmJjYwFQSuGUTVaVro6EEA7t3D/nWPTbIv7T4j+89VDme2rMlpgIR47A/v3w66/G38OHITk5c9kyZaBBA7jvPmOqUwdq1YKyZc075ZbXwsPD6dq1K+fPn093v5fWOn9rUEIIYW+VSlTit5d+o4pvFYfowigqCnbvhu3bYedOOHgQMtxni5MTNGwITZsaf1OSkr+/OTGbZenSpQwdOpS4uDhs3dIkNSghRIH0/dHvCb8WzuttXqdayWqmxRETYySjn34y/h48CBZL+jL33gvNm0OLFsbf++/H1Gs9ZouJieHFF1/k+++/Tz2ll5HUoIQQBdKO0zt45rtnaFyuMf954D+4ueRfX9JaG9eNNm+GLVtg1y5ISPh3ubMzPPAAtGsHbdtCy5ZGCzlhOHz4MI899hiXLl0iPj7+tmUlQQkhCpSwi2E8/vXjVC9ZnQ3PbsiX5HTrllFDWr0a1q+HiIh/lyll1Iw6djSSUqtW4O2d9bqKspUrV9KvX79sExMYNSg5xSeEKDBO3DhBYFAgxd2Ks6XfFkp55l3VJDoaNm0yktKGDelb2N1zj9G7QWCgkZikhpQzxYsXx9fXl+joaKKjo7MtLwlKCFFg7Dm7B4u2sLXfViqVqGT39ScmGqftgoLg+++NXhpSNGwI3bvDk09Co0aFt1VdXurYsSNnzpxhyZIlvPXWW8TExMg1KCFEwZbyZdWvYT8er/U4JdxL2HHd8MsvRlJavtzotSFF69ZGUure3WjoIO6eq6srL7zwAn/99RcfffTRbctKghJCOLT4pHh6fNuDV1u8Sucane2WnK5dgyVLYMECOHbs3/n160P//tCnj9Ebg7C/a9euMW/evHTXoooVK4arqysxMTGp87I7xWfvEXWFECLHkixJ9FnZh41/beR63PW7Xp/WsGcPDBgAFSrA6NFGcrrnHhgzBkJDjRZ648ZJcspL06ZNIznDncpOTk68+eab+Pr64unpSXJycrY1KElQQghTaK0Ztn4Ya46uYW7nuTzb4Nlcrys+Hj7/3Lh21KYNLF1qNA3v3BnWrDH6wJsxQ64t5YcrV64wf/78dIMwurq60q9fP8aPH8+FCxeYMmUK9evXx83t9i005RSfEMIUb//0Np//9jlvP/Q2/3ngP7lax40bEBRUmd69/+31u2xZeP55ePFFqGbe/b1F1tSpU7FkuJPZ2dmZyZMnA+Dh4cHo0aMZPXp0tuuSBCWEyHdaay7HXGZIkyFMeXjKHb/+1CmYPRsWLoSYmOqAUTsaMwaeftoY70jkv0uXLvHZZ5+lqz0VK1aM5557jvLly9/x+iRBCSHyVWJyIq7Orix4fAEWbbmj/vVOn4apU2HxYkiyjjLXrNl1pk3z45FH5PSd2aZMmZLp2pOzszMTJ07M1frkGpQQIt9s+msT9T+pz4kbJ1BK4eyUs3Ekzp+H4cONYSg+/9zoC69vX6PRw4wZv9OxoyQns0VERPDFF1+QkKZfKDc3N55//nnKlSuXq3VKghJC5Iu9Z/fS49seeBfzprRn6Ry95uJFGDHCuEdp/nyj1tS3L/z5p3FfU6NGeRy0yLFJkybZbLk3YcKEXK9TTvEJIfLckctH6LqsK+V9yrOp7yaKuxW/bfm4OOMa03vvGV0SAfTqBZMmQb16eR+vuDPnzp3jyy+/zFR7Gjp0KGXLls31eiVBCSHy1JmbZwgMCsTNxY2t/bfi7531oEhaw7ffGvcpnT5tzHv8ceO6U8OG+RSwuGMTJ060ee3prbfuboBJOcUnhMhT3sW8aVSuEVv6baF6yepZltu/Hx56CHr3NpJTgwbwww+wdq0kJ0d25swZvv76axITE1Pnubu7M3z4cEqXztmp3KxIDUoIkSeiE6JxcXLBz8OPDc9uyLLcjRtGjemzz4znZcoYNabnnzfGXhKObcKECVn2GnG3JEEJIewuITmBp755Co1mS78tOKnMJ2u0hm++gddeM26ydXU1Ho8fDyXs11esyEOnTp3i22+/zVR7GjFiBH5+fne9fklQQgi7smgLA9cMZNuJbSx6YpHN5HTypNFsfPNm4/mDD8Knn0oDiIJm/PjxJKXckGbl7OzM2LFj7bJ+uQYlhLAbrTUjNo1g+eHlfPDIBzx3/3Pplicnw8yZRo/imzeDr6/R2/j27ZKcCprjx4+zatWqdAnKw8ODkSNH4uvra5dt2DVBKaX8lFKrlVIxSqnTSimbvT8qw1Sl1Hml1E2lVLBSqr49YxFC5L+Ze2Yyb/88RrcazeutX0+37MQJY8j01183mpH37m3cz/Tii5DNqAvCAb355pvpTu2Bce1pzJgxdtuGvU/xfQwkAP5AY2CDUipMa30kQ7lewGDgQeA0MBVYCjSxczxCiHz0aM1HuRRziekdp6d2YaS10fvDyJEQE2MMffH55/DooyYHK3ItJiaGVatWpWsc4eHhwdixYylhxwuIdvvdopTyAnoAE7TW0VrrXcBaoL+N4tWAXVrrE1rrZCAIkAq+EAXUn1f+RGtN/bL1mdlpZup1p0uX4IknYMgQIzk9/bQxHpMkp4LNy8uLHTt28MADD+Dl5QUY155Gjhxp1+3YswZVC0jWWoenmRcGtLNRdjnwjFKqFnASGAhstrVSpdQQYAiAv78/wcHBdgzZPqKjox0yLkck+yp7kZGRJCcnF5j9dPDGQd449AYvVX+JHhV7pM7fvbsUM2bU5ubNYnh7JzJixF906HCZQ4fsu305pnLO3vvq/fffJywsjE8//ZSHH36YAwcO2G3dgHFR0x4T8BBwMcO8F4FgG2WLAXMBDSRhJKlq2W2jadOm2hH9/PPPZodQYMi+yl67du10o0aNzA4jR0LOh2jvad76vk/u09djr2uttU5I0HrUKK2Nk3tad+ig9dmzeReDHFM556j7CgjRNr7z7XlpMhrI2MFWcSDKRtl3gOZAJcAdmAz8pJTytGM8Qog89Ne1v+jyVRdKeZRic9/NlPQoyZkz0LYtzJoFLi7GKLZbt0LFimZHKwoieyaocMBFKVUzzbxGQMYGEinzv9Fan9NaJ2mtFwMlketQQhQICckJPLrsUTSarf23UqF4BTZsgPvvh337oFIl2LHDGEBQWuiJ3LLboaO1jgFWAVOUUl5KqTZAN4zWeRntB3oppfyVUk5Kqf6AK/C3veIRQuSdYs7FmNlxJpv6bqJ6iVq88QY89hhcv240gPjtN2jVyuwoRUFn7982wwEP4DLwNTBMa31EKVVZKRWtlKpsLfcBRgOKUCASGAn00FpH2jkeIYQdxSbGsv3UdgC61elGdfdmdOkCH3xg9Jv3/vuwbh2UKmVyoKJQsOt9UFrr68CTNuafAbzTPI8HXrZOQogCIDE5kWe+e4Ytf2/h7//8TdS5ynTrBsePQ9mysGKFcf1JOJaAgADuu+8+5s2bZ3Yod0zODgshsqW15sV1L7I+fD1zO88lbEdlWrY0klOTJhASUriS05UrVxg+fDhVq1bFzc0Nf39/OnTowLZt23L0+uDgYJRSXL16NY8j/dfixYvx9vbONH/VqlW89957+RaHPUlnsUKIbI37YRxLwpYwqd1kbmwbxstvG43Ie/eGhQvBs5C1v+3RowexsbEsXLiQGjVqcPnyZbZv3861a9fyPZaEhASKFSuW69fbo1dxs0gNSghxWz+c+IEZe2bwUqMRHP10AuPHG/OnTYNlywpfcoqMjGTnzp28//77dOjQgSpVqtC8eXPGjBlD7969AQgKCqJ58+b4+PhQtmxZevXqxfnz5wFjCIqHH34YgDJlyqCUYtCgQYBxuu2VV15Jt71Bgwbx2GOPpT4PCAhg2LBhjBkzhjJlytCmTRsAZs2aRcOGDfHy8qJChQq88MILREZGAkaN7bnnniMmJgalFEopJk2aZHObVatWZerUqbz00ksUL16cihUrMmPGjHQxhYeH065dO9zd3alduzYbN27E29ubxYsX22Uf55QkKCHEbXWo1oFFHb/n8MzZLF+u8PaG77+HN98Ea3d7hYq3tzfe3t6sXbuW+Ph4m2USEhKYPHkyYWFhrF+/nqtXr9KnTx8AKlWqxMqVKwE4cuQIERERzJ07945iCAoKQmvNzp07+fLLLwGjI9Y5c+Zw5MgRli1bxq+//sqrr74KQOvWrZkzZw6enp5EREQQERFx205bZ8+eTYMGDTh48CDjxo1j7Nix7N27FwCLxUL37t1xcXFh3759LF68mMmTJ3Pr1q07eg/2IKf4hBA2bf57M1VKVME9ui4fDH6CY8egQgXYuLFwD8Hu4uLC4sWLefHFF1mwYAH3338/bdq0oVevXjzwwAMADB48OLV89erVmT9/PnXr1uXcuXNUrFgx9bRa2bJlczXsebVq1fjwww/TzXvttddSH1etWpXp06fTrVs3lixZQrFixShRogRKKcqVK5ft+jt16pRaq3r11Vf56KOP+PHHH2nVqhXbtm3j2LFjbN26lQoVKgBGQkupyeUnqUEJITLZdWYX3b/pzuD5n9CyJRw7ZiSlffsKd3JK0aNHDy5cuMC6devo0qULe/bsoWXLlkybNg2AgwcP0q1bN6pUqYKPjw/NmjUD4MyZM3bZftOmTTPN++mnn+jYsSMVK1bEx8eHp556ioSEBC5evHjH62+Y4Z9Yvnx5Ll++DMDRo0cpX758anICaN68OU4m3HEtCUoIkc6hS4d4bNlj+J0dwO8ffMTly/DII0bPEEWpyyJ3d3c6duzIxIkT2bNnD88//zyTJk3i5s2bBAYG4unpydKlS9m/fz+brUMDJyQk3HadTk5OKf2Rpso4phKQ2kN4itOnT9O1a1fq1q3LihUrOHDgAIsWLcrRNm1xdXVN91wphcViAYwWm8pBzt1KghJCpDoVeYrAoEA4MISLn/2P2FjFwIGwYQPYcZifAqlevXokJSURGhrK1atXmTZtGm3btqVOnTqptY8UKa3u0o6XBEajiYiIiHTzwsLCst12SEgICQkJzJ49m1atWlGrVi0uXLiQaZsZt5cbdevW5fz58+nWHxISkprA8pMkKCFEqinbpxD5wxBufjcdi0UxcSJ88QXcRSvnAufatWu0b9+eoKAgfv/9d06ePMmKFSuYPn06HTp0oF69eri5uTFv3jxOnDjBhg0bmDBhQrp1VKlSBaUUGzZs4MqVK0RHRwPQvn17Nm3axNq1azl27BijRo3i7Nmz2cZUs2ZNLBYLc+bM4eTJk3z99dfMmTMnXZmqVasSHx/Ptm3buHr1KrGxsbl6/x07dqR27doMHDiQsLAw9u3bx6hRo3Bxccn3mpUkKCEEYNzXVGrPAuI2T0Ip+OQTmDy5cLbUux1vb29atmzJ3LlzadeuHfXr1+ett97i2Wef5ZtvvqFMmTIsWbKENWvWUK9ePSZPnsysWbPSraNChQpMnjyZ8ePH4+/vn9ogYfDgwalTmzZt8Pb2pnv37tnG1LBhQ+bOncusWbOoV68en3/+OTNnzkxXpnXr1gwdOpQ+ffpQpkwZpk+fnqv37+TkxOrVq7l16xYtWrRg4MCBjB8/HqUU7u7uuVpnrtkag8NRJxkPquCTfZW9/B4PKj4xXo/a+Lp+7oV4DVo7O2v91Vf5tvm7JsdUzuV2X4WGhmpAh4SE2DcgK7IYD0qamQtRhCVbknl2xUBWTesGh91wdzf61Etz36goglavXo2Xlxc1a9bk1KlTjBo1ikaNGtGkSZN8jUMSlBBFlNaal1a/xqp3+sFfj+HjY/RE3q6d2ZEJs0VFRTFu3DjOnj1LyZIlCQgIYPbs2fl+DUoSlBBF1PjN/2Xh2CfgREdKlYLNm8F6O48o4gYMGMCAAQPMDkMSlBBF0YVrN5n1n/ZwojX+/poff1TUr292VEKkJwlKiCImOhr6PFWCW3+35p57ND/9pKhTx+yohMhMmpkLUYR8H/YTtR44zo4dUL48BAdLchKOS2pQQhQRP/15gKee8MRy5l4qVLAQHOxEjRpmRyVE1iRBCVEEhJwMJzBQYznbkgqVktgR7EL16mZHJcTtSYISopA7duE8bTrcJOlscypUSmTXDleqVjU7KiGyJ9eghCjE4uKgd09PEk42x/+eBHZul+QkCg5JUEIUUvHxmqeegtC9JfEvp9kRXIxq1cyOSoick1N8QhRCsfGJVH/wAJcOtKR0afjpR0WtWmZHJcSdsWsNSinlp5RarZSKUUqdVko9e5uy1ZVS65VSUUqpq0qp3HW9K4RIJyHRQt0OB7l0oCWexeP54QeoV8/sqIS4c/Y+xfcxkAD4A32B+UqpTPenK6WKAduAn4ByQEUgyM6xCFHkJCVpGnU+yJk9D+DmFc/2H91p1MjsqITIHbslKKWUF9ADmKC1jtZa7wLWAv1tFB8EXNBaz9Jax2it47XWv9srFiGKIosFWncP4+hPzXB1v8WPW9ykbz1RoNnzGlQtIFlrHZ5mXhhgq2/klsAppdQmoDlwGHhVa30oY0Gl1BBgCIC/vz/BwcF2DNk+oqOjHTIuRyT7KnuRkZEkJyff0X7SGj7+uAb71zfGyTWeD977g8TEfygKu1qOqZwraPvKngnKG7iZYd5NwMdG2YrAw8ATwI/ACOB7pVQdrXVC2oJa6wXAAoBmzZrpgIAAO4ZsH8HBwThiXI5I9lX2fH19iYyMvKP99M7kRFaudMXVFdatcyMwMH/H7TGTHFM5V9D2lT2vQUUDxTPMKw5E2SgbB+zSWm+yJqSZQCmgrh3jEaJIGDU1nCmTXFFKs2wZBAYWsTHaRaFlzwQVDrgopWqmmdcIOGKj7O+AtuO2hSiSPvj0FLMnGh3qzfwohp49TQ5ICDuyW4LSWscAq4ApSikvpVQboBuw1EbxIKClUuoRpZQz8BpwFfjTXvEIUdh98d0F3ni5PGgnxk6MZNQr3maHJIRd2buZ+XDAA7gMfA0M01ofUUpVVkpFK6UqA2itjwH9gP8BNzAS2RMZrz8JIWzb+PN1nu/rC8nFGDTsOu9P8jU7JCHszq49SWitrwNP2ph/BqMRRdp5qzBqXEKIO3DkCPTrURKdoHi05xUWziuDkstOohCSvviEKECO/X2LRzpauHFD8cQT8P3XZXCST7EopOTQFqKAOB+RRNOHrnExwomH2lpYvhxcpDdNUYhJghKiAIiM1DRsc4GYi+WpWOsy69Y64eFhdlRC5C1JUEI4uLg4aNjuFNdPVsavwlUO7CxLiRJmRyVE3pMEJYQDS0qCVp1Pc/b3anj63SBkZynKljU7KiHyhyQoIRyUxQIvvABhO6rg7hPL3uDiVKsmzfVE0SEJSggHpDUMGHaZJUvA0xN+3upJwwbOZoclRL6SNkBCOKBz0f34fUFZnFySWLXKhZYtzY5IiPwnNSghHMypq524fmIMKAvzP48hMNDsiIQwhyQoIRzI/CVXOH3kDQCmzLjOkIHSXE8UXZKghHAQW7dqXn7eF3CiVJU5TBhd2uyQhDCVJCghHMAvv8BTTyl0sitlqgdRocRis0MSwnTSSEIIk4X+nsAjgZqYGDf694fTpxdyM+PY1MCiRYu4cuUK9evXp27dulStWhVnZ2nZJwovSVBCmOjEyWRaPxxF3M1SPPTITRYuLEHHjrbH8ty3bx+LFi3Cy8uL5ORkEhISqFChAvXr16dZs2apiatmzZq4ubnl8zsRwv4kQQlhkosXNU0evEbc9bJUbXiOLWsr4uqadfkpU6YQFBTEP//8kzrv1KlTnDp1ik2bNuHl5QVAbGwsZcuWpU6dOjRt2pRXX32VypUr5/XbEcLu5BqUECa4eRPuf+giNy+Upey95/lte8VsO38tV64cw4YNw93dPdMyi8VCVFQUUVFRJCcnExERwc8//8yHH37IpUuX8uhdCJG3JEEJkc9iY6F952gu/n0Pxe+5SOiue/D1zdlr33777Rxfd/L09GTs2LE0b94898EKYSJJUELko8RE6NULDu7zpnS5eA7sKsU95XL+MSxZsiRjx47FI5vqlpOTEzVq1GDq1Kl3G7IQppEEJUQ+sVigc4+LbNwIpUrBjp/cqVH9NhedsjB69GiKFSt22zJubm6UKlWKqKio3IYrhOkkQQmRD7SG3oMv89O6cji5xbBxk4W6dXO3Li8vLyZNmpTaKMKWuLg4du/eTa1atdixY0cuoxbCXJKghMgHL79+jRVLyoLLLZavjKNF87v76A0bNgxPT8/blklISODq1at07tyZcePGkZiYeFfbFCK/SYISIo+9895N5n9YCpyS+HjRVXp1vfsujNzc3Pjggw8y1aJsXZuKi4tj3rx5NG3alBMnTtz1toXIL3ZNUEopP6XUaqVUjFLqtFLq2Ry85iellFZKyT1ZotD54guY8pbR4eukWWcZ3r+C3dY9YMAA/Pz8Up97eHjw5ptv4u3tnamlX2xsLEeOHKFhw4YEBQXZLQYh8pK9a1AfAwmAP9AXmK+Uqp9VYaVUX+RmYVFIrV5tjIgL8ObUS7wzoppd1+/s7Mzs2bPx8vLC09OTN954gwkTJnD48GEaNGiQ6RSgxWIhJiaGl156iV69eqW74VcIR2S3BKWU8gJ6ABO01tFa613AWqB/FuVLAO8AY+0VgxCOYu26JHo+nYTFAhMnwrTx/nmynaeeeopKlSpx3333MX78eACqVKnC/v37GTVqlM1TfrGxsaxfv57atWvz66+/5klcQtiD0tp2v193vCKl7gf2aK090swbA7TTWj9uo/zHwN/AauAk4Kq1TrJRbggwBMDf37/p8uXL7RKvPUVHR+Pt7W12GAVCUdhX+0NK8Mab9bEkFaNpl53MeD0ZpXL++tdee43k5GT+7//+L0flr127hpubm839eujQISZOnEhMTIzNRhJubm706dOHfv36FdiOZ4vCMWUvjrqvHn744QNa62aZFmit7TIBDwEXM8x7EQi2UbYZEIpxeq8qoAGX7LbRtGlT7Yh+/vlns0MoMAr7vtq+XWsXt1satG7ebb+2WO58He3atdONGjWyW0w3btzQTzzxhPb09NTWz1q6ydPTUzdv3lyfPXvWbtvMT4X9mLInR91XQIi28Z1vz2tQ0UDxDPOKA+nuFFRKOQGfACO0jRqTEAXVvn3QsXMCSbeKUbfjXvaubHpHNae84uvry5o1a/j444/x8vLCySn9xz42NpaDBw9Sr149Vq5caVKUQmRmzwQVDrgopWqmmdcIOJKhXHGMGtQ3SqmLwH7r/HNKqYfsGI8Q+ebAAejcWZMQV4zKD+4ibOMDODs7QHayUkoxaNAgQkNDqV27dqYGFMnJyURFRTFgwAAGDBhATEyMSZEK8S+7JSitdQywCpiilPJSSrUBugFLMxS9CZQHGlunR63zmwK/2CseIfLL779Dp05w86aiW/dE/tjWHFcXx7zFsEaNGoSGhjJ06NAsG1CsWLGCunXrEhoamv8BCpGGvT9FwwEP4DLwNTBMa31EKVVZKRWtlKpsPeV4MWUCrlhfe0lrnWDneITIU3/+Ce3aJ3D9OjzaNYlvl7vi5e7YgwUWK1aMDz/8kPXr1+Pn55epX7/4+HjOnj1L69atmTFjBhaLxaRIRVFn1wSltb6utX5Sa+2lta6stV5mnX9Ga+2ttT5j4zWntNZKrkeJgiY8HNo9nEjktWJ41tnFp19Gkk0frg6lffv2hIeHExAQYLNfv7i4OCZPnky7du24ePGiCRGKos4xz0OIXAkICOCVV14xO4wi4ehReLBtElcuuVLs3j0c+LEyFf3uvguj/FaqVCk2b97M9OnT8fT0RGVo1RETE8O+ffuoU6cOGzZsMClKUVQV+QR15coVhg8fTtWqVXFzc8Pf358OHTqwbdu2HL0+ODiYhx9+mKtXr+ZxpP9avHixzXsZVq1axXvvvZdvcRRVR45A23bJXLnkgkv1Heza5kud8gV3SHWlFMOHD2f//v1Ur14907WppKQkbt68ydNPP83QoUOJj483KVJR1BT5BNWjRw9+/fVXFi5cSHh4OOvXr6dLly5cu3Yt32NJSLi7S3B+fn74+PjYKRphy6FD8PDDcOWyMx61drF1kxvNq9UzOyy7qFevHocPH2bgwIE2e0qPjY3lyy+/5L777uOPP/4wIUJR5Ni6OcpRJ3vfqHvjxg0N6G3btmVZZunSpbpZs2ba29tblylTRvfs2VOfO3dOa631yZMnM930OHDgQK21cbPlyy+/nG5dAwcO1F27dk193q5dOz106FA9evRoXbp0ad2sWTOttdYffvihbtCggfb09NTly5fXzz//vL5x44bW2rjRLuM233nnHZvbrFKlin733Xf1kCFDtI+Pj65QoYKePn16upiOHTum27Ztq93c3HStWrX0hg0btJeXl/7iiy9ys0uz5ag3CuZEaKjWpUpZNGgdGKj1zaiEPNmOvW/UzY0NGzboEiVKaBcXl0zHm1JKe3p66nnz5mlLbu5EtrOCfEzlN0fdV+TDjboFjre3N97e3qxduzbL0xYJCQlMnjyZsLAw1q9fz9WrV+nTpw8AlSpVSr2x8ciRI0RERDB37tw7iiEoKAitNTt37uTLL78EjOG658yZw5EjR1i2bBm//vorr776KgCtW7dmzpw5eHp6EhERQUREBGPGjMly/bNnz6ZBgwYcPHiQcePGMXbsWPbu3QsYnYd2794dFxcX9u3bx+LFi5k8eTK3bt26o/dQFBw8CO3ba65dU1RvcZQ1a6C4952PhltQPProoxw7doxWrVplakChtSY2NpaxY8fSqVOnfD29LYoYW1nLUae86Orou+++0yVLltRubm66ZcuWevTo0Xrfvn1Zlv/zzz81kNotTEqN5sqVK+nK5bQG1aBBg2xj3LRpky5WrJhOTk7WWmv9xRdfaC8vr0zlbNWgevfuna5MjRo19Lvvvqu11nrz5s3a2dk5tUaotda7d+/WgNSg0ti1S+sSJYyaE7W+19OD5+bp9hyhBpUiOTlZz5gxI8tuklxdXbWfn5/+8ccfTYuxIB5TZnHUfYXUoGzr0aMHFy5cYN26dXTp0oU9e/bQsmVLpk2bBsDBgwfp1q0bVapUwcfHh2bNjP4Mz5zJ1GI+V5o2bZpp3k8//UTHjh2pWLEiPj4+PPXUUyQkJOSqqW/Dhg3TPS9fvjyXL18G4OjRo5QvX54KFf4do6h58+aZusIpyrZsgY4djZtwqbuS1+f8yuvt/mN2WPnGycmJMWPGsHv3bipVqoS7u3u65YmJiVy/fp3HHnuMkSNH3vV1VCHSkm8iwN3dnY4dOzJx4kT27NnD888/z6RJk7h58yaBgYF4enqydOlS9u/fz+bNm4HsGzQ4OTmldIybylZv0hlPn5w+fZquXbtSt25dVqxYwYEDB1i0aFGOtmmLq2v601BKqdQbL7XWmZoVi3999x08/jjExQGNF/Hcf7fwQed3zQ7LFI0bN+bo0aM8/fTTNhtQxMXFsWDBAho3bsxff/1lQoSiMJIEZUO9evVISkoiNDSUq1evMm3aNNq2bUudOnVSax8pUu7CT05OTje/TJkyREREpJsXFhaW7bZDQkJISEhg9uzZtGrVilq1anHhwoVM28y4vdyoW7cu58+fT7f+kJAQ6TkAWLgQnnkGEhOhS79j9HpzKwu6fVKkE7qnpydLlixh6dKl+Pj42By199ixY4wYMcKkCEVhU6QT1LVr12jfvj1BQUH8/vvvnDx5khUrVjB9+nQ6dOhAvXr1cHNzY968eZw4cYINGzYwYcKEdOuoUqUKSik2bNjAlStXiI6OBoy79Ddt2sTatWs5duwYo0aN4uzZs9nGVLNmTSwWC3PmzOHkyZN8/fXXzJkzJ12ZqlWrEh8fz7Zt27h69SqxsbG5ev8dO3akdu3aDBw4kLCwMPbt28eoUaNwcXEp0l/Es2YZI+FaLDB5Mmz4sjbf9PoaFycZ/BmMQRL/+OMPmjRpkqk25e7uzuzZs02KTBQ2RTpBeXt707JlS+bOnUu7du2oX78+b731Fs8++yzffPMNZcqUYcmSJaxZs4Z69eoxefJkZs2alW4dFSpUYNCgQYwfPx5/f//UnhwGDx6cOrVp0wZvb2+6d++ebUwNGzZk7ty5zJo1i3r16vH5558zc+bMdGVat27N0KFD6dOnD2XKlGH69Om5ev9OTk6sXr2aW7du0aJFCwYOHMj48eNRSmW61lAUWCzw5pswerTx3OPxN2jaewNKUaQTti0VK1Zk7969vPHGG6k39np5eTFnzhxq165tcnSi0LDVcsJRJxmwMO+FhoZqQIeEhOTJ+h11X8XHa/3ss1qD1s7OFu319HBdfW51HREVke+xOFIrvpz45ZdfdLly5XTXrl1NuS/KUY8pR+So+4osWvHJOYsibvXq1Xh5eVGzZk1OnTrFqFGjaNSoEU2aNDE7tHwTGQndu0NwMHh5W/DoMxDnWtvY2m835bzLmR2ew2vRogWnTp3CyclJaprCriRBFXFRUVGMGzeOs2fPUrJkSQICApg9e3aR+aI5cwYefdToX8+/nAXPgb245vsD2/tt516/e80Or8Bwc3PsIUZEwSQJqohLGUG1KAoNha5d4cIFqFsXNm5UfH6iLh2r/4fG5RqbHZ4QRZ4kKFEkrVsHzz4L0dHQtq2FeV+ep2qVSkytOtXs0IQQVkW6FZ8oerSGadOgWzcjOfV5VlN++GA6rmjO9bjrZodX5FWtWjVTq1VRdEkNShQZsbEweDB88w0oBf/9r+bS/SP56NclTGs/DT8PP7NDLBIGDRrE1atXWb9+faZl+/fvtzm6ryiaikQNas6cOTz33HMcOnTI7FCESc6cgQcfNJKTjw98/z2oh97no1/n8toDr/HGg2+YHaLA6IHFVldK+U36FHQMhT5BxcfHM2HCBJYuXcoDDzxAkyZNWLlyZaZ+8kThtWsXNG8Ov/0G994L+/ZBUo3VvPXTW/Rr2I8PAz8sMq0WHV3GU3xKKRYsWECvXr3w8vKievXqBAUFpXvNlStX6N27NyVLlqRkyZJ07do1XX+Ax48fp1u3bpQrVw4vLy+aNGmSqfZWtWpVJk2axODBg/H19aVv3755+0ZFjhT6BPXNN98ARl95cXFx/Pbbb/Tu3Zu4uDiTIxN5zWKBGTMgIAAuX4ZHHoFff4V69aDTvZ2YEjCFRU8swkkV+o9BgTZlyhS6detGWFgYzzzzDIMHD+b06dOA0f/fqFGjcHd3Z/v27ezdu5d77rmHRx55JLULsOjoaLp06cK2bdsICwujR48ePPXUUxw9ejTddmbNmkWdOnUICQlJHc1AmKvQfzLff//91P7xwPhF9sQTTzjEaQSRd65fNxpCjB0LyclG90WbNsGJ+BCibkXhVcyLCe0m4OpceAcdLCz69+9Pv379qFGjBu+++y4uLi7s3LkTgOXLl6O15osvvqBhw4bUqVOHTz/9lOjo6NRaUqNGjRg6dCgNGjSgRo0ajB8/niZNmvDdd9+l2067du0YO3YsNWrUoGbNmvn+PkVmhTpB7d+/P9O4TZ6enowdO9akiER++OUXuP9+WL8efH2N600zZ8KhK7/Rfkl7hm4YanaI4g6kHdPMxcWFMmXKpI4qcODAASIiIvDx8UkdIbtEiRLcuHGD48ePAxATE8PYsWOpV68eJUuWxNvbm5CQkEzfDSljvQnHYddWfEopP2Ah0Am4CryptV5mo9xA4D9ATeAfYBnwltY6yZ7xzJgxI9NQ7hUqVKBFixb23IxwEFrDRx/B668bw2Q0bw7ffgtVq8Lf1/+m81ed8XX35YNHPjA7VHEHbjemmcVioUaNGmzYsCHT6/z8jFaZY8aMYfPmzcycOZOaNWvi6enJgAEDMjWEkNaDjsfezcw/BhIAf6AxsEEpFaa1PpKhnCfwGvALUAZYC4wB3rdXIFevXmXdunXpxjby9vZm3LhxckG8ELp40RgiI+V7asQImD4dihWDiKgIAoMCSbYks3XQVioWr2husMJumjRpwtKlSyldujS+vr42y+zatYsBAwbQo0cPwGg4dfz4cWrVqpWPkYrcsNspPqWUF9ADmKC1jtZa78JIPP0zltVaz9da79RaJ2itzwNfAW3sFQvAggULbCaiPn362HMzwgGsXAn33WckJ19fYyTcOXOM5ATwwroXuBR9iY19N1KndB0zQxVW//zzD6GhoemmU6dO3fF6+vbti5+fH926dWP79u2cPHmSHTt2MHr06NSWfLVq1WL16tUcPHiQQ4cO0a9fv0xnVoRjsmcNqhaQrLUOTzMvDGiXg9e2BTLWsgBQSg0BhgD4+/sTHByc7cqSk5OZPn16upZ6Li4uBAYG8ssvv+QgnDsTHR2do7iEffdVdLQzH31Uk23bjB7HmzW7ztixRylVKoG0m+jv15+Onh2J/SuW4L/ss+28FBkZSXJycqE9pi5evMjOnTu5//77081v27Ztau0m7Xs/cuQIpUuXTn2escx///tfli1bxpNPPklMTAylSpWicePG/PHHH5w/f55evXoxY8aM1HHZevbsSb169bh48WLqOmxttzAqcN9VtsbgyM0EPARczDDvRSA4m9c9B5wDSme3jZyOB7V27Vrt4+OjgdTJ3d1dnzhxIucDlNwBRx1jxRHZa1/9+KPWlSoZ4zd5eGg9b57WaYciSkxO1J8f+FwnW5Ltsr38VNDGgzKbfP5yzlH3FVmMB2XPVnzRQPEM84oDUVm9QCn1JMZ1py5a66v2CuS9994jKir9Zh944AGqVatmr00Ik1y/blxr6tABzp6FFi2MG3BfftnovgiMH10vrXuJF9a9wLbj28wNWAiRa/ZMUOGAi1Iq7Q0Ejcj61F1n4DPgca213fogCg8PJzQ0NN08b29v3nhDurIpyLSGr76COnVg4ULj+tKUKbB7N2QcYfytH99iUegiJradSGCNQHMCFkLcNbtdg9JaxyilVgFTlFIvYLTi6wa0zlhWKdUeo2FEd631r/aKAYx+9xITE9PN8/b2plOnTvbcjMhHx4/D8OGwdavxvF07+N//jGSV0ay9s3h/9/sMbTqUSQGT8jVOIYR92ftG3eGAB3AZ+BoYprU+opSqrJSKVkpVtpabAJQANlrnRyulNt3txmNiYliyZAlJSf/eTuXp6cmoUaNwcirU9yQXSnFxMHWq0UJv61YoWdKoPf38s+3kdP6f84z/aTw96/Vk3qPz5HYCIQo4u94HpbW+DjxpY/4ZwDvN84ftud0US5cuzfSlZLFYeOGFF/JicyKPaG30Oj5unNELOUDfvjBrFpQtm/XrKhSvwM7ndtKgbAOcnZzzJ1ghRJ4pNNUKrTXTp08nJiYmdZ6TkxM9evSgZMmSJkYm7sSvvxrDYvTpYySnhg3hxx8hKCjr5LTn7B6Whi0FoFn5Zri5uOVjxEKIvFJoBizcvXt3av9cKdzd3RkzZoxJEYk7cfIkTJxoJCIwktF//wvPPQfOt6kMHb58mK7LulLWqyy96vfC3cU9fwIWQuS5QpOgPvjgg3S1J4B7772Xxo0bmxOQyJGzZ41EtHAhJCUZrfNGjoS33oLiGW9ayOB05GkCgwLxcPFgc9/NkpyEKGQKRYKKiIhg27b097tI03LHFhEB06bBggWQkABOTtC/P0yeDDm5Xe1KzBU6BXUiNjGWHYN2UK2k3OMmRGFTKBLU/PnzM81zdnamZ8+eJkQjbufyZTdGjjSaicfHGzfXPvMMvPMO1K2b8/WsObqGMzfPsK3/Nhr4N8i7gIUQpinwCSoxMZH/+7//49atW6nz3NzcGDp0KMVSegsVpgsLM0a3Xb78AZKTjXlPPQWTJkGDXOSXF5u+SKd7O1HFt4pd4xRCOI4C34pvzZo16e57SvHKK6+YEI1IS2vYtg06dYLGjY2eILRW9OkDBw8avZDfSXJKtiQzbP0wfj1v3NstyUmIwq1AJajk5GSCgoLSDeGecUh3gICAACpWlDF/zHLjhjFwYP36RnLatg28vOC11+Crr35h2TJjxNs7obVmxOYR/O/A/9h9ZneexC2EcCwF6hRfXFwcAwcOxM3Njf79+9OlSxf+/PPPdGVSBiUU+Utr4x6mTz+F5cuNXiAA7rkHXn0Vhg41eoIIDs7dODzv7niXj/d/zOutX2dkq5F2jFwI4agKVIJydnbGy8uLqKgoFi1axJdffpmp3z0/Pz8CAgLMCbAIunjRSEhLlkDaPno7djSS0uOPQ4YRu+/Y/P3zeSf4HQY2GijDtQtRhBS4BJVyvSkpKSnTtScvLy9eeeUV6YMtj0VHw5o1xk2127aBxWLML1XKuLF2yBCoWfO2q8gxrTXbTmzjsVqP8dnjn8n/VogipEAlKBcXFxISErJcnpyczIQJE/jll18YO3YsLVq0yMfoCrfoaNi0CVatgrVrITbWmO/qatSS+vY1/rrb8V5ZrTVKKb7t9S2JyYm4Ot9lVUwIUaAUqEYSTk5Ot/0FHR8fz61bt1i1ahUPPfQQgwYNyr/gCqGrV2HRIiPxlC4NTz9tnM6LjYU2bWD+fOOG2zVroFcv+yankAshPPTFQ0REReDi5IKHq4f9Vi6EKBAKVA0KjNN4N2/evG0ZpVTq6T6Rc8nJEBICW7bA5s3wyy//nr5TykhK3bsb9y/l5eDE4dfC6fJVF7yLeaPRebchIYRDK3AJysfH57YJys3NjfLlyxMcHEzlypWzLCeMlncnT0JwsJGUfvjBGFI9haur0dihe3fo1g3Klcv7mM7/c55OSzuhUGztt5XyPuXzfqNCCIdU4BJUiRIlOHfunM1lnp6e3H///WzcuJHi2fU0WgRpDceOwfbtsGOH8ff8+fRlqlWDzp0hMBAefjj7Dlvt6XrcdQKDArked53gQcHULGWnlhZCiAKpwCUoPz8/m/M9PT3p2bMnn3/+Oa532665kLh0CfbvN+5P2r/fmK5dS1/Gzw8eeggeecRISjVqGKfzzBCXGEcx52Ks6b2GJvc0MScIIYTDKHAJqlSpUpnmeXh4MH78eN58880i2QxZa6MmdOgQ/P77v8koZTTatPz9oV07aNvW+FuvntGTuJmSLEkoFBWKVyBkSAhOqkC13RFC5JECl6DKZhhW1dPTk8WLF9OrVy+TIso/Whs1oD//NJLR4cP//o2MzFze2xuaNYPmzY2pRQuoXNm8GpItFm3h+bXPE5cYx/KeyyU5CSFSFbgEdc899wBGk3MfHx+2bNnCAw88YHJU9hUZCX/99e8UHv7vY1uJCIxTdQ0aGFPTpkYyql379qPROoJx28bxZdiXTAmYIslJCJFOgUtQfn5+ODk5UbFiRbZv307VqlXNDumOJCbChQvG6bespn/+yfr13t5G4klJRvfdZ/wtV86xakY5MWP3DGbunckrzV/h7bZvmx2OEMLBFLgEVb9+fTp27Mjy5cvx9fU1OxzAuH/o2jW4fNnom+7iRaOBgq3Hly//e29RVjw9jcYKNWtmnvz9C14ismVx6GLG/jCW3vf1Zm6XuUXy2qEQ4vYKXILq0KEDHTp0sOs6tTZGd42KMqZ//vn3740bxr1B168bSSjlcdopMrIdOof3kyoFFSoY14LSTpUq/fvYz69wJKHbqelXk6frP82SJ5fIqT0hhE12TVBKKT9gIdAJuAq8qbVelkXZkcA4wANYCQzTWt+yVTZFUhKcPm0M5WBrio3NellMzL8JKGMiiooidZTXXL5zfH2hbFnjVFu5ckZNJ+3flMf+/nffu3dBdiPhBgBtKrehTeU2JkcjhHBk9q5BfQwkAP5AY2CDUipMa30kbSGlVCDwBtAeuACsBiZb52UpLAzy6pJTsWLGTak+Pv9OxYsbtZmMU6lS6Z+HhgbToUNA3gRWiPx+6XcG7B/Ah6U+ZEjTIWaHI4RwcErn9NxUditSygu4AdyntQ63zlsKnNdav5Gh7DLglNb6LevzDsBXWuvbdqaj1P3azW0LTk63rFMCTk63cHaOT/M47bJ463PjsYtLLM7OcTg7x+LsHIOLS8rjWJycMg8bn1ORkZEOcz3MUcW5xxHaJBRt0TT5rQnut+zYs2whExoaSlJSEs2aNTM7lAJBPn8556j7avv27Qe01pkOeHvWoGoBySnJySoMaGejbH3g+wzl/JVSpbTW6fo6UEoNAYYAuLq6UqdOp7sOVGvjdGFS7nNSOsnJyURm1f5bkOiWyN/N/yZZJVNtRzXiY+OJJ3cj6xYFSUlJaK3lmMoh+fzlXEHbV/ZMUN5Axl5cbwI+OSib8tgHSJegtNYLgAUAzZo10yEhIXYJ1p6Cg4NlFN8sJCYn0nJhS1yuuhDcP5hbHW/JvspGQEAAkZGRhKYdolhkST5/Oeeo+yqrVrz2TFDRQMauRYsDUTkom/LYVllRgLk6uzK06VAqFq9Iq0qtCD4ebHZIQogCwp7te8MBF6VU2i6oGwFHbJQ9Yl2WttyljKf3RMGVbEnm8OXDALzY9EW61OxickRCiILGbglKax0DrAKmKKW8lFJtgG7AUhvFvwSeV0rVU0qVBN4GFtsrFmEurTXDNwynxWctOBV5yuxwhBAFlL3vkByOcV/TZeBrjHubjiilKiulopVSlQG01puB6cDPwGnr9I6dYxEmmfjzRBYcXMDIliOp6lvV7HCEEAWUXe+D0lpfB560Mf8MRsOItPNmAbPsuX1hvo9++YipO6fywv0vMLX9VLPDEUIUYNLHjLCbHad3MGLzCLrX6c78x+ZL/3pCiLtS4PriE46rTaU2zAmcw0vNXsLFSQ4tIcTdkRqUuGsHLhzg3D/ncHZyZkTLEbi7SC8RQoi7JwlK3JU/r/xJYFAgA1YPMDsUIUQhIwlK5NrZm2cJDArExcmFz5/43OxwhBCFjFwoELlyLfYagUGB3Lx1k+2DtlO9ZHWzQxJCFDKSoESuvL7tdU7cOMGWfltoXK6x2eEIIQohSVAiV2YFzqJ/w/60q2qrs3ohhLh7cg1K5JhFW5i7by5xiXH4uvvycLWHzQ5JCFGISYISOaK1ZuTmkby25TVW/rnS7HCEEEWAJCiRI9N2TuOjXz9iZMuR9G3Q1+xwhBBFgCQoka0FBxbw9s9v069hP2Z2mildGAkh8oUkKHFb/9z6hwk/T6BLjS4semIRTkoOGSFE/pBWfOK2irsVZ9dzuyjvUx5XZ1ezwxFCFCHyc1jY9FvEb0zbOQ2tNTVL1cSrmJfZIQkhihhJUCKTv6//TeevOvPpgU+5EX/D7HCEEEWUJCiRTkRUBJ2WdsKiLWzttxU/Dz+zQxJCFFFyDUqkioyPpPNXnbkcc5mfB/5M7dK1zQ5JCFGESYISqXaf2c3f1/9mzTNraF6hudnhCCGKOElQIlXXWl058Z8T+Hv7mx2KEELINaiiTmvN8A3DWXN0DYAkJyGEw5AEVcS9+eObzA+ZT+jFULNDEUKIdCRBFWGz9s7ig90fMKzZMN5p947Z4QghRDp2SVBKKT+l1GqlVIxS6rRS6tnblB2olDqglPpHKXVOKTVdKSXXwvLZ0rCljN46mp71evJ/Xf5P+tcTQjgce9WgPgYSAH+gLzBfKVU/i7KewGtAaeABoAMwxk5xiBwKuxRG+2rtCeoehLOTs9nhCCFEJnddc1FKeQE9gPu01tHALqXUWqA/8EbG8lrr+WmenldKfQXIyHf5xKItOCknZnScQUJyAm4ubmaHJIQQNtnj1FotIFlrHZ5mXhiQ07HA2wJHslqolBoCDLE+jVZKHctVlHmrNHDV7CAKCNlXOVNaKSX7KWfkmMo5R91XVWzNtEeC8gZuZph3E/DJ7oVKqeeAZsALWZXRWi8AFtxNgHlNKRWitW5mdhwFgeyrnJH9lHOyr3KuoO2rbK9BKaWClVI6i2kXEA0Uz/Cy4kBUNut9Engf6KK1dsSMLoQQwkTZ1qC01gG3W269BuWilKqptf7LOrsRtz9t1xn4DOiqtT6U83CFEEIUFXfdik9rHQOsAqYopbyUUm2AbsBSW+WVUu2Br4AeWutf73b7DsKhT0E6GNlXOSP7KedkX+VcgdpXSmt99ytRyg9YBHQErgFvaK2XWZdVBv4A6mmtzyilfgYeAuLTrGKn1rrLXQcihBCi0LBLghJCCCHsTbo6EkII4ZAkQQkhhHBIkqDsTClVUykVr5QKMjsWR6SUclNKLbT22RillPpNKSXXH63upF/LokyOo9wpaN9PkqDs72Ngv9lBODAX4CxGTyMlgAnAt0qpqmYG5UDupF/LokyOo9wpUN9PkqDsSCnVG4gEfjQ5FIeltY7RWk/SWp/SWlu01uuBk0BTs2MzW5p+LSdoraO11ruAlH4tRRpyHN25gvj9JAnKTpRSxYEpwGizYylIlFL+GP05ZnljdxGSVb+WUoPKhhxHt1dQv58kQdnPu8BCrfVZswMpKJRSrhg3bS/RWh81Ox4HkOt+LYsyOY5ypEB+P0mCyoHs+iNUSjUGHgFmmxyq6XLQd2NKOSeM3kYSgFdMC9ix5Kpfy6JMjqPsFeTvJxnJNgdy0B/ha0BV4Ix1ZFpvwFkpVU9r3SSv43Mk2e0rAGXspIUYDQEe1Von5nVcBUQ4d9ivZVEmx1GOBVBAv5+kJwk7UEp5kv6X7xiMA2KY1vqKKUE5MKXU/4DGwCPWQS6FlVJqOaAxhqBpDGwEWmutJUllIMdRzhTk7yepQdmB1joWiE15rpSKBuId/Z9vBqVUFeAl4BZw0fqLDuAlrfVXpgXmOIZj9Gt5GaNfy2GSnDKT4yjnCvL3k9SghBBCOCRpJCGEEMIhSYISQgjhkCRBCSGEcEiSoIQQQjgkSVBCCCEckiQoIYQQDkkSlBBCCIckCUoIIYRD+n+lmHMcQBZOsAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "z = np.linspace(-5, 5, 200)\n",
    "\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [1, 1], 'k--')\n",
    "plt.plot([0, 0], [-0.2, 1.2], 'k-')\n",
    "plt.plot([-5, 5], [-3/4, 7/4], 'g--')\n",
    "plt.plot(z, logit(z), \"b-\", linewidth=2)\n",
    "props = dict(facecolor='black', shrink=0.1)\n",
    "plt.annotate('Saturating', xytext=(3.5, 0.7), xy=(5, 1), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.annotate('Saturating', xytext=(-3.5, 0.3), xy=(-5, 0), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.annotate('Linear', xytext=(2, 0.2), xy=(0, 0.5), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.grid(True)\n",
    "plt.title(\"Sigmoid activation function\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.2, 1.2])\n",
    "\n",
    "save_fig(\"sigmoid_saturation_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Xavier and He Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Constant',\n",
       " 'GlorotNormal',\n",
       " 'GlorotUniform',\n",
       " 'HeNormal',\n",
       " 'HeUniform',\n",
       " 'Identity',\n",
       " 'Initializer',\n",
       " 'LecunNormal',\n",
       " 'LecunUniform',\n",
       " 'Ones',\n",
       " 'Orthogonal',\n",
       " 'RandomNormal',\n",
       " 'RandomUniform',\n",
       " 'TruncatedNormal',\n",
       " 'VarianceScaling',\n",
       " 'Zeros',\n",
       " 'constant',\n",
       " 'deserialize',\n",
       " 'get',\n",
       " 'glorot_normal',\n",
       " 'glorot_uniform',\n",
       " 'he_normal',\n",
       " 'he_uniform',\n",
       " 'identity',\n",
       " 'lecun_normal',\n",
       " 'lecun_uniform',\n",
       " 'ones',\n",
       " 'orthogonal',\n",
       " 'random_normal',\n",
       " 'random_uniform',\n",
       " 'serialize',\n",
       " 'truncated_normal',\n",
       " 'variance_scaling',\n",
       " 'zeros']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[name for name in dir(keras.initializers) if not name.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7ff1b2321690>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=\"he_normal\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7ff180a32410>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "init = keras.initializers.VarianceScaling(scale=2., mode='fan_avg',\n",
    "                                          distribution='uniform')\n",
    "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nonsaturating Activation Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Leaky ReLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def leaky_relu(z, alpha=0.01):\n",
    "    return np.maximum(alpha*z, z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure leaky_relu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAqV0lEQVR4nO3de3wU9b3/8dcHAkIICfBAUGohxwtWwIIasdZbWqxawQsCiqJAVfByROwRq9RLqXgXtYpVvEBRoQoCilV/PiyeBkU9SFQ8FVqocEBBroUEQm6QfH9/fBddQkJ2N9nMXt7Px2MfzM4OM++dTPaTmfnu92vOOURERBJNs6ADiIiI1EYFSkREEpIKlIiIJCQVKBERSUgqUCIikpBUoEREJCGpQElEzMyZ2eCgcyQzMxtpZiVNtK0m+XmZ2Slm9r9mVmlmBfHeXj1ZckPvOy/IHNJ4VKBSgJlNN7M3g84RDTObEPowcWZWbWbfmtlMM/thlOspMLMn63htjZmNq2PbX8aaPcJctRWIWcDhjbydun72hwJ/acxt1eFx4AvgCOCiJtgeUOfP/Rv8+17aVDkkvlSgJEgr8B8ohwGXAMcCswNNFEfOuTLn3OYm2tZG51xFE2zqSOC/nXPfOOe2NcH26uScqwq97z1B5pDGowKVBsysh5m9ZWY7zWyzmb1sZoeEvX6imb1rZlvNbIeZLTKzk+tZ562h5c8I/Z/BNV7/hZntNrPOB1jNntAHyrfOuQ+A54CfmFl22HrOM7NPzazczP7PzO41s5Yx7oqImFlzM5sa2l6Zmf3LzH5jZs1qLDfCzP5uZhVmtsnMpofmrwkt8mroTGpNaP53l/jMrHvotWNrrHN0aL+2qC+HmU0ARgD9w85G80Ov7XMGZ2bHmtmC0Hq2hc68csJen25mb5rZWDNbb2bbzexPZpZZxz7KNTMH5ADTQtsbaWb5oemONZfde+ktbJl+ZrbYzErNrNDMjq+xjZ+Y2X+b2S4zKzaz98ysS2g/nwH8Z9j7zq3tEp+ZnR7aRnnoZ/RY+PETOhN7yszuC+33zWY2qebPWoKhH0KKM7NDgfeBL4G+wJlAFvBG2C9hW+Al4LTQMkuBt8M/ZMLWZ2Y2CRgDnOGcWwi8DFxZY9ErgTedc5sizHkI/hJRVeiBmZ0NzASeBHqG1jkYuC+SdTZAM2A9cDFwDHA78FvgV2F5rwGeAf4E/Bg4F1gWevnE0L+j8GeIe59/xzm3EigEhtV4aRgwyzm3O4Ick/BnnAtC2zkU+KjmtkJF5h2gBP/zHQj8FJhWY9HTgF74Y+SS0HJja64vZO/ltFLgptD0rDqWrcv9wG3A8cC/gZlmZqHMvYG/AV8BpwA/Cb3XjFCmj/H7fu/7/qaW9/0D4P8BnwPHAVcBl4a2G24YsAe/T24IvZ9LonwvEg/OOT2S/AFMxxeD2l67G3ivxrz2gAP61vF/DNgAXB42z+F/af8ErARyw17Lw/+C/yBs/WXAgANknoAvRCX4DzkXejwetsz7wJ01/t+Fof9joecFwJN1bGMNMK6ObX8Z5T5+AFgQ9nwd8MABlnfA4BrzRgIlYc/HAmvD3ssPgWrg5Chy1PqzD98+vlAWA23DXs8PLXNk2Hq+ATLClnkufFt15CkBRtay3o5h83JD8/JqLHN22DKnhOYdFno+E/ifA2x3v597Ldu5F1/gmtX4GVQAmWHr+bjGev4KPN+Q30k9GuehM6jUdwJwupmV7H3w/V+bRwCYWScze8bMVppZMbAT6AR0rbGuSfgPl1Odc2v2znTOFQJ/x19uArgM2I7/6/VAVgF98GcYtwOf4c8QwrPfXiP7n4E2wCHEkZldG7rstCW03V8T2h9m1gn4AfBeAzfzMtAFf+YCfr+tds59HEmOKBwD/K9zbmfYvI/wxbBH2Lzlbt/7N9/ij4N4+d8a2yJse8fR8P17DL74VIfNWwS0xN87qy3H3izxfN8SIRWo1NcMeAtfCMIfRwF7W3+9gC8Sv8Zf5uiDP0Ooea/nr/jCcG4t23me7y89XQlMd85V1ZOt0jn3lXNumXPuPvwHxR9rZP99jdw/DmXfUs+6AXbg75HU1A5/RlErM7sE+AP+rOLs0Haf4vv9YRFsu17ON5hYwPeX+YbhzxwizREpw59Z1BojbHp3La9F+xmxtxiE76MWdSwbvr29OfZurzH2cVO+b4mDjKADSNx9hr+Hsdb5+xq1ORW40Tn3FoD5hg2H1rLc28A8Qjf/nXMvhL02A3jYzG7A31MYGkPWicAKM5vsnPs0lP1HzrmvYlgX+FaCJ9Qy//jQa3U5FVjsnPuuGbOZHbF32jm3yczWA/3wRbs2u4HmEWScAUw2s2fxrRgHRZojpDKC7SwHrjSztmFnUT/Ffwj/I4KM0dj7h8OhYdN9YljPZ8DPD/B6pO/7YjNrFnYWdWro/66KIZM0Mf2VkDqyzaxPjUcu/owkB5hlZieZ2eFmdqaZPWtmbUP/dyVwufnWficCr+B/iffjnHsTGAJMMbPhYfOLgVeBR4D3nXP/ivYNOOdWA2/gCxX4+2eXmdndZtbLzH5kZoPN7KEa/7VjLe+9C/AYcLaZ3Rl6bz3N7F7gZPyZSV1WAseb2S/N7CgzuxPfaizcvcBNZvZr8y3y+pjZzWGvrwH6mdkhZtb+ANt6DX+GMRX4pMZ+iyTHGqCXmR1tZh3NrLazlZnALuBF8635Tsc38JjXgOJfl6/wl5AnhPbLWcAdMaznYeC40HHaO/T+rjazvZc31wB9Qy33OtbR6u4p/CXUp8zsGDPrj7+H96RzrjSGTNLUgr4JpkfDH/hLQK6Wx5zQ60cBc/D3hcrwZw+TgZah13sDi0OvrQKuwLf6mxC2jX1u+gPnhZYfHjbv9NBywyPIPIFaGirg/7J3wE9Dz88CPsA3pNiBb/l2Q9jyBXW890k1/v82fEuxAuD0erK1xBeM7UBRaPouYE2N5a7C/5VeCWwEptXYP//Cn0mtCc0bSVgjibBlXwxlHhNtDuBg4F38fUMH5Nfx8zoWf0+nLLS+6UBOjWPozRrbr/VnVGOZfRpJhP0Ml4a29THQn9obSdTZkCI071R8Q5my0PtfABwaeq17aN17G9jk1rGO0/HHdgWwCf9Hy0E1jp+ajS322xd6BPPY23pIpMFC90yeAbo4/YUqIg2ke1DSYKHv2eTiW+A9p+IkIo1B96CkMfwG3x/bNr6/fyQi0iC6xCciIglJZ1AiIpKQ4nYPqmPHji43Nzdeq2+QXbt20aZNm6BjJC3tv9isWLGCqqoqevToUf/Csh8dd7Gra99t3gzffANm8KMfQWatXQPH36effrrVOXdwzflxK1C5ubkUFhbGa/UNUlBQQH5+ftAxkpb2X2zy8/MpKipK2N+LRKfjLna17bv33oOzz/bTr7wCF1/c9Ln2MrO1tc3XJT4RkTSzerUvSFVVMH58sMXpQFSgRETSSEkJXHghbNsG/fvDxARud6sCJSKSJpyDkSPh73+Ho4+GmTOheSQ9RgZEBUpEJE3cey/MnQvZ2TB/PuTU1td/AlGBEhFJA/Pnw513+hZ7L7/sz6ASXVQFKtSjcrmZzYhXIBERaVxr1mRy+eV++r774NzaRnRLQNGeQf0RWBKPICIi0vi2b4c77uhFSQlccgncemvQiSIXcYEys6H4Lu8bOgyziIg0gaoqGDoU1q/PpE8fmDbNX+JLFhF9UdfMsvGDx/XDj4FT13KjgdEAnTt3pqCgoBEiNr6SkpKEzZYMtP9iU1RURFVVlfZdjHTcRW/KlMN5992uZGdXcOutn/HJJxVBR4pKpD1JTASmOue+sQOUX+fcs8CzAHl5eS5Rv/Wtb6Q3jPZfbNq1a0dRUZH2XYx03EVn5kyYNQsyMuD3v1/O0KEnBx0pavUWKDPrA5wJHBf3NCIi0mCffgpXX+2nH38cevQoDjZQjCI5g8rHD0b3dejsKQtobmY9nHPHxy+aiIhEa9Mm31NEeTmMGgXXXQcLFwadKjaRFKhngVfCno/DF6zr4hFIRERiU1kJgwbBunXw05/Ck08mV6OImuotUKHhu78bwtvMSoBy59yWeAYTEZHo3HgjfPgh/OAHvseIli2DTtQwUQ+34ZybEIccIiLSAFOmwDPPwEEHweuvwyGHBJ2o4dTVkYhIkvvgAxgzxk8/9xzk5QWbp7GoQImIJLGvv/b3nfbsgZtvhiuuCDpR41GBEhFJUqWlvsXeli3wi1/AAw8EnahxqUCJiCQh5/x3nT7/HI44wg/bnhF1q4LEpgIlIpKEHn7YD5uRleWH0ujQIehEjU8FSkQkybzzDtx2m59+6SXo2TPYPPGiAiUikkRWrvQ9lDsHEyb4e1CpSgVKRCRJ7NgBF1wAxcUwcKAfITeVqUCJiCSB6moYNgz++U/o1QteeAGapfgneIq/PRGR1HDXXfDmm9C+ve8pom3boBPFnwqUiEiCe/VVuPdef8Y0e7ZvVp4OVKBERBLYF1/AyJF+etIkOPPMQOM0KRUoEZEEtXWrb6VXWgrDh8NNNwWdqGmpQImIJKDdu+Hii2HNGjjxRN9TeTKP7RQLFSgRkQR0883wt7/5YTNeew1atQo6UdNTgRIRSTDTpsHkyX7AwXnz/ACE6UgFSkQkgfzP/8B11/npp56Ck08ONk+QVKBERBLEt9/CRRdBZSXccANcdVXQiYKlAiUikgDKy333RRs2QH4+PPpo0ImCpwIlIhIw5+Daa+GTT6BbN/9l3BYtgk4VPBUoEZGAPfGE71svM9N3Y3TwwUEnSgwqUCIiAXrvPd+kHOBPf4I+fQKNk1BUoEREArJ6tf8yblUV/Pa3flq+pwIlIhKAkhI/ttO2bdC/P0ycGHSixKMCJSLSxKqrfQewX34JRx8NM2em/thOsdAuERFpYvfeC3PnQk4OzJ/v/5X9qUCJiDSh+fP94INm8Oc/+zMoqZ0KlIhIE1m2DC6/3E/ffz+ce26weRKdCpSISBPYvt2P7VRSAkOHwm9+E3SixKcCJSISZ3v2+KL01Vdw3HEwdWr6je0UCxUoEZE4Gz8e3n0XOnb0YztlZgadKDmoQImIxNHMmTBpEmRkwJw5vq89iYwKlIhInBQWwtVX++knnoAzzgg2T7JRgRIRiYNNm/zwGeXlMGqU761coqMCJSLSyCorYdAgWLcOTjkFnnxSjSJioQIlItLIxoyBDz+EH/zA33dq2TLoRMlJBUpEpBFNmQLPPgutWvmxnQ45JOhEyUsFSkSkkbz/vj97AnjuOcjLCzZPslOBEhFpBF9/DYMH+y/l3nzz910aSewiKlBmNsPMNpjZDjNbaWZXxzuYiEiyKC313Rht2QJnnQUPPBB0otQQ6RnU/UCucy4bOB+4x8xOiF8sEZHk4BxcdRV8/jkccQS88or/Uq40XEQFyjm3zDlXsfdp6HFE3FKJiCSJhx/2RSkryw+l0b590IlSR8R13syeAkYCrYHPgbdrWWY0MBqgc+fOFBQUNErIxlZSUpKw2ZKB9l9sioqKqKqq0r6LUSIed4sXd2D8+GMB49Zb/86WLf8mwSICibnvImHOucgXNmsOnAzkAw8653bXtWxeXp4rLCxscMB4KCgoID8/P+gYSUv7Lzb5+fkUFRWxdOnSoKMkpUQ77lauhL59obgYfv97Pwhhokq0fVeTmX3qnNuvzWNUrficc1XOuUXAYcB1jRVORCSZFBfDBRf4fy+6CO64I+hEqSnWZuYZ6B6UiKSh6mrfhPyf/4ReveCFF6CZvrATF/XuVjPrZGZDzSzLzJqb2dnApcB/xz+eiEhiuesuePNN6NDBN4rIygo6UeqKpJGEw1/Om4IvaGuBm5xz8+MZTEQk0bz6Ktx7rz9jmjULDj886ESprd4C5ZzbAmgUExFJa198ASNH+ulHHoEzzww0TlrQlVMRkXps3eobRZSWwogRMHZs0InSgwqUiMgB7N4NQ4bA2rW+WfmUKRrbqamoQImIHMDNN0NBgR82Y948P4yGNA0VKBGROkybBpMn+wEH583zAxBK01GBEhGpxccfw3Wh7giefhpOPjnYPOlIBUpEpIb1630PEZWVcMMNcOWVQSdKTypQIiJhyst9cdq4EfLz4dFHg06UvlSgRERCnINrr4VPPoFu3fwXc1u0CDpV+lKBEhEJefxx37deZqbvxqhjx6ATpTcVKBERYMECGDfOT0+fDr17BxpHUIESEWH1arjkEqiqgt/+1n8xV4KnAiUiaa2kxHdjtG0bDBgAEycGnUj2UoESkbRVXQ3Dh8OXX8LRR8OMGRrbKZHoRyEiaeuee+C11yAnxzeKyMkJOpGEU4ESkbQ0fz787ne+49eXX/ZnUJJYVKBEJO0sW+aHbQe4/3745S+DzSO1U4ESkbSybZtvFFFSAkOHwm9+E3QiqYsKlIikjT174NJLYdUqOO44mDpVYzslMhUoEUkb48fDu+/CwQfD66/7HiMkcalAiUhamDEDJk2CjAyYMwe6dg06kdRHBUpEUl5hIVx9tZ9+4gk4/fRg80hkVKBEJKVt3AgDB0JFBYwe7Xsrl+SgAiUiKauyEgYPhnXr4JRT/PDtahSRPFSgRCQlOedHw/3wQzjsMJg7F1q2DDqVREMFSkRS0pQp8Nxz0KqV786oc+egE0m0VKBEJOW8/z7ceKOffu45yMsLNo/ERgVKRFLK2rX+vtOePX4Awr1dGknyUYESkZRRWupb7G3ZAmedBQ88EHQiaQgVKBFJCc7BVVfB55/DkUfCK69A8+ZBp5KGUIESkZTw0EO+KGVl+W6M2rcPOpE0lAqUiCS9t9/2/eyB79KoZ89g80jjUIESkaS2YgVcdpm/xHf33X4oDUkNKlAikrSKi31BKi6Giy6C228POpE0JhUoEUlKVVUwbJg/g+rVC154AZrpEy2l6McpIknprrvgrbegQweYP983jpDUogIlIkln9my47z7fjHz2bDj88KATSTyoQIlIUvniC/jVr/z0pEnQr1+weSR+VKBEJGls3eobRZSWwogRMHZs0IkknlSgRCQp7NljDBni+9rr29f3Vq6xnVJbvQXKzA4ys6lmttbMdprZ52b2y6YIJyKy11NPHUFBARxyiB8+o1WroBNJvEVyBpUBfAOcAeQAdwKzzSw3jrlERL4zdSq89tphtGwJ8+ZBly5BJ5KmkFHfAs65XcCEsFlvmtn/AScAa+ITS0TE+/hjuO46P/3003DyycHmkaZTb4Gqycw6A92BZbW8NhoYDdC5c2cKCgoami8uSkpKEjZbMtD+i01RURFVVVXad1HYsqUl1157Art3H8SAAf/H4YevRbsvesn6OxtVgTKzFsBM4AXn3D9rvu6cexZ4FiAvL8/l5+c3RsZGV1BQQKJmSwbaf7Fp164dRUVF2ncRKi+H00+HbdvgZz+DsWO/1r6LUbL+zkbcis/MmgEvAZXADXFLJCJpzzm45hpYsgRyc/2XcTMyXNCxpIlFdAZlZgZMBToD5zrndsc1lYiktccfhxdfhMxMP7ZTx45BJ5IgRHqJ72ngGOBM51xZHPOISJpbsABuvtlPT58OvXsHGkcCFMn3oLoB1wB9gI1mVhJ6DIt3OBFJL6tWwcUXQ3W1HzpjyJCgE0mQImlmvhbQ97VFJK5KSuDCC2H7dhgwwA8+KOlNXR2JSOCqq2H4cPjyS/jRj/yw7RrbSXQIiEjg7rnHd1+Uk+PHdsrJCTqRJAIVKBEJ1Ouvw+9+5zt+feUV6N496ESSKFSgRCQwy5bBFVf46QcegHPOCTaPJBYVKBEJxLZtfmynkhK49FK45ZagE0miUYESkSa3Zw8MHeqblR93HDz/vMZ2kv2pQIlIk7vtNvjrX+Hgg/09qMzMoBNJIlKBEpEm9dJL8MgjkJEBc+dC165BJ5JEpQIlIk2msBBGjfLTkyfDaacFm0cSmwqUiDSJjRt9TxEVFTB6NFx7bdCJJNGpQIlI3FVUwKBBsH49nHKKP3sSqY8KlIjElXMwZgx89BEcdpi/79SyZdCpJBmoQIlIXE2ZAs89B61a+e6MOncOOpEkCxUoEYmbhQvhxhv99PPPQ15esHkkuahAiUhcrF0Lgwf7L+WOGwfDNIKcREkFSkQaXWmpb7G3dSucdZbvZ08kWipQItKonIMrr4SlS+HII30P5c2bB51KkpEKlIg0qoceglmzICvLj+3Uvn3QiSRZqUCJSKN5+20YP95Pz5wJPXoEm0eSmwqUiDSKFSv8sBnOwd13w/nnB51Ikp0KlIg0WHGxH9tpxw7fY8TttwedSFKBCpSINEhVlW9CvmIFHHssTJ8OzfTJIo1Ah5GINMhdd8Fbb0GHDn5sp6ysoBNJqlCBEpGYzZ4N993nm5HPng2HHx50IkklKlAiEpOlS+FXv/LTjzwC/foFGkdSkAqUiERtyxbfU0RpKYwc+X1/eyKNSQVKRKKyezcMGeL72uvbF55+GsyCTiWpSAVKRKLyX//leyk/9FA/fEarVkEnklSlAiUiEZs6FZ580g84OG8edOkSdCJJZSpQIhKRjz6C667z01OmwE9+EmweSX0qUCJSr3Xr4KKL/P2nG2/8vvWeSDypQInIAZWX++K0aRP87GcwaVLQiSRdqECJSJ2cg9GjYckSyM31X8Zt0SLoVJIuVKBEpE5/+AO89BJkZvqxnTp2DDqRpBMVKBGp1YIFMG6cn54+HX7840DjSBpSgRKR/axaBRdfDNXVfuiMIUOCTiTpSAVKRPaxc6cf22n7djjvPD/4oEgQVKBE5DvV1TBiBCxbBsccAzNmaGwnCU5Eh56Z3WBmhWZWYWbT45xJRAIycaLvvignx4/tlJ0ddCJJZxkRLvctcA9wNtA6fnFEJCivvw4TJvgzpldege7dg04k6S6iAuWcmwdgZnnAYXFNJCJNbtkyuOIKP33//XDOOcHmEQHdgxJJe9u2+UYRJSVw6aVwyy1BJxLxIr3EFxEzGw2MBujcuTMFBQWNufpGU1JSkrDZkoH2X2yKioqoqqpKqH1XVWXcdtuxrFrVgaOO2snw4Z+zcGF10LFqpeMudsm67xq1QDnnngWeBcjLy3P5+fmNufpGU1BQQKJmSwbaf7Fp164dRUVFCbXvbr4ZCgvh4INhwYK2dO16etCR6qTjLnbJuu90iU8kTb30Ejz6KGRkwNy50LVr0IlE9hXRGZSZZYSWbQ40N7NWwB7n3J54hhOR+FiyBEaN8tOTJ8NppwWbR6Q2kZ5B3QGUAbcBl4em74hXKBGJn40bYeBAqKiAa66Ba68NOpFI7SJtZj4BmBDXJCISdxUVMGgQrF8Pp54KTzwRdCKRuukelEiacA5uuMEP3X7YYTBnDrRsGXQqkbqpQImkiaefhuefh1atfK8RnTsHnUjkwFSgRNLAwoUwdqyfnjoVTjgh2DwikVCBEklxa9fC4MGwZ4/vJeKyy4JOJBIZFSiRFFZaChdeCFu3wtln+372RJKFCpRIinIOrrwSli6FI4+El1+G5s2DTiUSORUokRT14IMwaxZkZcH8+dC+fdCJRKKjAiWSgt56C377Wz89cyb06BFsHpFYqEA1ETNjzpw5QceQNLBihW8I4ZwfIff884NOJBIbFaiQkSNHMmDAgKBjiDRIcbEf22nHDt9jxO23B51IJHYqUCIpoqoKhg3zZ1DHHgvTp4NZ0KlEYqcCFYHly5fTv39/2rZtS6dOnbj00kvZuHHjd68vWbKEs846i44dO5Kdnc2pp57Kxx9/fMB1Pvjgg3Ts2JHFixfHO76kiTvv9PeeOnTwjSKysoJOJNIwKlD12LBhA6effjq9evXik08+YcGCBZSUlHD++edTXe1HHt25cydXXHEFH3zwAZ988gl9+vTh3HPPZevWrfutzznHuHHjmDx5MgsXLuSkk05q6rckKWjWLP8dp+bNYfZs+I//CDqRSMM16oi6qejpp5+md+/ePPjgg9/Ne/HFF+nQoQOFhYX07duXn//85/v8n8mTJzN37lzeeecdLr/88u/mV1VVceWVV/Lhhx+yaNEicnNzm+ptSApbuhR+9Ss//eij0K9foHFEGo0KVD0+/fRT3n//fbJquV6yatUq+vbty+bNm7nzzjv529/+xqZNm6iqqqKsrIyvv/56n+XHjRtHRkYGixcvplOnTk31FiSFbdniG0WUlcHIkTBmTNCJRBqPClQ9qqur6d+/P5MmTdrvtc6h7qBHjBjBpk2beOyxx8jNzeWggw6iX79+VFZW7rP8L37xC15++WXefvttRo4c2RTxJYXt3g1DhsDXX8NJJ/neytUoQlKJClQ9jj/+eGbPnk23bt1o0aJFrcssWrSIJ554gv79+wOwadMmNmzYsN9y5557LhdddBFDhgzBzBgxYkRcs0tq+/WvfS/lhx4K8+b5YTREUokaSYTZsWMHS5cu3efRv39/iouLueSSS1i8eDGrV69mwYIFjB49mp07dwLQvXt3ZsyYwfLly1myZAlDhw6lZR0jwQ0YMIBXX32Va6+9lhdffLEp356kkOefhz/+0Q84OG8edOkSdCKRxqczqDAffPABxx133D7zBg0axIcffsj48eM555xzKC8vp2vXrpx11lkcdNBBAEybNo3Ro0dzwgkn0KVLFyZMmMCWLVvq3M6AAQOYPXs2F198MQDDhw+P35uSlPPRR3D99X56yhT4yU+CzSMSLypQIdOnT2f69Ol1vn6gbop69+693/eZrrjiin2eO+f2eX7eeedRVlYWfVBJa+vWwUUX+ftPN974fes9kVSkS3wiSaKsDAYOhE2b4Oc/h1ra7YikFBUokSTgHFxzDRQWQm6u/2JuHW12RFKGCpRIEvjDH+CllyAz03dj1LFj0IlE4i/lC9SKFSuYNm1a0DFEYvbXv8K4cX76hRfgxz8ONo9IU0nZRhLOOaZOncrYsWOprq6mffv2DBw4MOhYIlFZtQouuQSqq+GOO2Dw4KATiTSdlDyDKioq4oILLmDs2LGUlpZSXl7OiBEjWLduXdDRRCK2c6fvxmj7djjvPPj974NOJNK0Uq5Affzxxxx99NG8++67lJaWfje/tLSUgQMHftcDuUgiq66G4cNh2TI45hiYMQOapdxvq8iBpcwhX1VVxYQJE+jXrx+bN2+moqJin9czMjLYtGkT5eXlASUUidzEifD669CunW8UkZ0ddCKRppcSBWr9+vWcfPLJPPzww7V++TUzM5OBAweyfPlyMjMzA0goErnXXoMJE/wZ08svw1FHBZ1IJBhJ30hi/vz5DB8+nNLSUvbs2bPPa82aNaN169Y888wzDBs2LKCEIpH78kt/aQ/ggQfgnHOCzSMSpKQtUGVlZYwZM4Y///nPdZ41HX744bzxxhv8h4YXlSSwbZtvFFFSApde+n3TcpF0lZSX+JYvX06vXr3qLE6tW7fm+uuv57PPPlNxkqSwZw8MHQqrV8Pxx/veyjW2k6S7pDqDcs4xZcoUxo0bR1lZ2X4dsLZo0YKsrCzmzJmz3zDsIons1lv9F3I7dfL3oHSrVCSJCtT27dsZNmwY77///j7Nx/dq06YNffv2Zfbs2XRUPzCSRF58ER59FDIyYM4c6No16EQiiSEpLvEtWrSI7t27895777Fr1679Xm/dujUTJ07kvffeU3GSpLJkCYwe7aeffBJOOy3YPCKJJKHPoPbs2cOECRN49NFHa73X1KpVKw4++GD+8pe/0Lt37wASisRu40Y/fEZFhe+p/Jprgk4kklgCPYOqrKzks88+q/W1b775hpNOOonHHnuszlZ6gwYN4h//+IeKkySdigoYNAjWr4dTT4Unngg6kUjiCbRAPfLII5x44oksWbJkn/lz586lZ8+efPHFF/vdb2rWrBlZWVlMmzaNGTNm0KZNm6aMLNJgzsF//qcfuv2HP/T3nVq2DDqVSOIJ7BLfjh07uO+++6iuruaCCy5gxYoVZGRkcP311zN79uxaG0JkZmZy1FFHMX/+fLp16xZAapGGe+opmDoVWrXyLfY6dw46kUhiiugMysw6mNlrZrbLzNaa2WUN3fBDDz1EVVUVANu2bWPQoEH06NGDWbNm1VqcWrduzZgxYygsLFRxkqRVUpLBTTf56alT4YQTAo0jktAiPYP6I1AJdAb6AG+Z2RfOuWWxbPTf//73PveWKioqWLRoUa33mlq2bElWVhZz584lPz8/ls2JJISiIlizpg1VVXDLLXBZg//ME0ltVvPLrvstYNYG2A70cs6tDM17CVjvnLutrv/Xtm1bd0Idfx5+9dVXbNiwod6hL5o1a0Z2djY9evSgRYsWB34nUSgqKqJdu3aNtr50o/23v+pq3xtEXY9du2Dz5qUAdOjQh1691FNEtHTcxS7R993ChQs/dc7l1ZwfyRlUd6Bqb3EK+QI4o+aCZjYaGA2+V4eioqL9VrZ7926+/fbb/XqBqGVdHHLIIXTs2LHW7z41RFVVVa3ZJDKpuP+qq42qqtgfkWrRoprDDiuiuDiObyZFpeJx11SSdd9FUqCygJq/TsVA25oLOueeBZ4FyMvLc4WFhfutbNSoUXz11VdUVlbWucHs7GwWLVrEscceG0G86BUUFOhyYQMk2v6rqoIdO/wltKIiKC7+frq25zXnFRf7M6CGaNUKcnL8+E3hj73z2reHefPyqawsYunSpQ3bWJpKtOMumST6vrM6LidEUqBKgJrDpWUDO6MNsXbtWmbMmHHA4gTfn2XFq0BJYtm9O/qiEj5vx46GZ2jTpvbCUtfz8Hk5Ob5A1eedd6CeQ19EwkRSoFYCGWZ2lHPuX6F5vYGoG0iMHz9+vzGbalNWVsbQoUNZsWIFnTp1inYz0sTKy2M/eykqgloabUYtJ+fAReRAhSY7GxrxFqeINJJ6C5RzbpeZzQPuNrOr8a34LgB+Gs2GVq5cyWuvvRZRgQL/PalRo0Yxf/78aDYjUXLOF4hIz1aKiuDrr4+nuvr75xUVDcvQrFnkZyu1PW/bFpo3b1gGEUk8kTYzvx6YBmwG/g1cF20T81tuuYXdu3fvN39vzxDV1dWUl5dz6KGH0rNnT/r27cuZZ54ZzSbSUnU17NwZ/WWx8Oehr6NFYd8rvi1a+Hss0VwWC3+0aaMWbSKyv4gKlHNuG3BhrBtZtmwZb7zxBllZWQCUl5fTpUsXevXqxYknnkivXr3o2bMnRx55ZKM2J08Ge/bse4M/2ktlxcX+LKghWreO7rLY6tWf8bOfHf/d81atVGBEpPE1SVdHWVlZ3HPPPRxzzDH07NmTI444goyMhO5IPWKVldGdrdR8XlLS8Axt28Z+/yUnJ/p+4AoKdnDMMQ3PLSJyIE1SJbp168btt9/eFJuKinP73uCPpdDU0vlFVMz2LRyRXhbbOy872w90JyKSapL6o805fwYS7WWxDRv6UlHhn9dyWywqGRnRN0sOf2Rl+UYCIiKyr0ALVHV1w+6/FBXF+gXLzO+mWrb0N/hj+f5Lu3aQman7LyIi8RC3ArVpE9x114ELTWN9wTLay2IrVizm7LNPivgLliIi0vTiVqDWrYOJE+tfLjs79vsvOTmxfcGyrKxMY/CIiCS4uBWoTp3g+usPXGj0BUsREalL3ArUD38Iv/tdvNYuIiKpTu3HREQkIalAiYhIQlKBEhGRhKQCJSIiCUkFSkREEpIKlIiIJCQVKBERSUgqUCIikpBUoEREJCGpQImISEIy19DxwutasdkWYG1cVt5wHYGtQYdIYtp/sdO+i532XewSfd91c84dXHNm3ApUIjOzQudcXtA5kpX2X+y072KnfRe7ZN13usQnIiIJSQVKREQSUroWqGeDDpDktP9ip30XO+272CXlvkvLe1AiIpL40vUMSkREEpwKlIiIJCQVKBERSUgqUICZHWVm5WY2I+gsycDMDjKzqWa21sx2mtnnZvbLoHMlMjPrYGavmdmu0H67LOhMyUDHWuNI1s84FSjvj8CSoEMkkQzgG+AMIAe4E5htZrlBhkpwfwQqgc7AMOBpM+sZbKSkoGOtcSTlZ1zaFygzGwoUAe8FHCVpOOd2OecmOOfWOOeqnXNvAv8HnBB0tkRkZm2AQcCdzrkS59wi4A3gimCTJT4daw2XzJ9xaV2gzCwbuBu4OegsyczMOgPdgWVBZ0lQ3YEq59zKsHlfADqDipKOtegk+2dcWhcoYCIw1Tn3TdBBkpWZtQBmAi845/4ZdJ4ElQUU15hXDLQNIEvS0rEWk6T+jEvZAmVmBWbm6ngsMrM+wJnAYwFHTTj17buw5ZoBL+HvrdwQWODEVwJk15iXDewMIEtS0rEWvVT4jMsIOkC8OOfyD/S6md0E5AJfmxn4v3Kbm1kP59zx8c6XyOrbdwDmd9pU/E3/c51zu+OdK4mtBDLM7Cjn3L9C83qjy1QR0bEWs3yS/DMubbs6MrNM9v2rdhz+h3mdc25LIKGSiJlNAfoAZzrnSgKOk/DM7BXAAVfj99vbwE+dcypS9dCxFptU+IxL2TOo+jjnSoHSvc/NrAQoT5YfXJDMrBtwDVABbAz9dQZwjXNuZmDBEtv1wDRgM/Bv/IeEilM9dKzFLhU+49L2DEpERBJbyjaSEBGR5KYCJSIiCUkFSkREEpIKlIiIJCQVKBERSUgqUCIikpBUoEREJCGpQImISEL6/1A09DkKDp8PAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, leaky_relu(z, 0.05), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([0, 0], [-0.5, 4.2], 'k-')\n",
    "plt.grid(True)\n",
    "props = dict(facecolor='black', shrink=0.1)\n",
    "plt.annotate('Leak', xytext=(-3.5, 0.5), xy=(-5, -0.2), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.title(\"Leaky ReLU activation function\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.5, 4.2])\n",
    "\n",
    "save_fig(\"leaky_relu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['deserialize',\n",
       " 'elu',\n",
       " 'exponential',\n",
       " 'gelu',\n",
       " 'get',\n",
       " 'hard_sigmoid',\n",
       " 'linear',\n",
       " 'relu',\n",
       " 'selu',\n",
       " 'serialize',\n",
       " 'sigmoid',\n",
       " 'softmax',\n",
       " 'softplus',\n",
       " 'softsign',\n",
       " 'swish',\n",
       " 'tanh']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[m for m in dir(keras.activations) if not m.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LeakyReLU', 'PReLU', 'ReLU', 'ThresholdedReLU']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[m for m in dir(keras.layers) if \"relu\" in m.lower()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train a neural network on Fashion MNIST using the Leaky ReLU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
    "X_train_full = X_train_full / 255.0\n",
    "X_test = X_test / 255.0\n",
    "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.LeakyReLU(),\n",
    "    keras.layers.Dense(100, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.LeakyReLU(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 1.6314 - accuracy: 0.5054 - val_loss: 0.8886 - val_accuracy: 0.7160\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 2s 892us/step - loss: 0.8416 - accuracy: 0.7247 - val_loss: 0.7130 - val_accuracy: 0.7656\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 2s 879us/step - loss: 0.7053 - accuracy: 0.7637 - val_loss: 0.6427 - val_accuracy: 0.7898\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 2s 883us/step - loss: 0.6325 - accuracy: 0.7908 - val_loss: 0.5900 - val_accuracy: 0.8066\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 2s 887us/step - loss: 0.5992 - accuracy: 0.8021 - val_loss: 0.5582 - val_accuracy: 0.8200\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 2s 881us/step - loss: 0.5624 - accuracy: 0.8142 - val_loss: 0.5350 - val_accuracy: 0.8238\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 2s 892us/step - loss: 0.5379 - accuracy: 0.8217 - val_loss: 0.5157 - val_accuracy: 0.8304\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 2s 895us/step - loss: 0.5152 - accuracy: 0.8295 - val_loss: 0.5078 - val_accuracy: 0.8284\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 2s 911us/step - loss: 0.5100 - accuracy: 0.8268 - val_loss: 0.4895 - val_accuracy: 0.8390\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 2s 897us/step - loss: 0.4918 - accuracy: 0.8340 - val_loss: 0.4817 - val_accuracy: 0.8396\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try PReLU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.PReLU(),\n",
    "    keras.layers.Dense(100, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.PReLU(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 1.6969 - accuracy: 0.4974 - val_loss: 0.9255 - val_accuracy: 0.7186\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 2s 990us/step - loss: 0.8706 - accuracy: 0.7247 - val_loss: 0.7305 - val_accuracy: 0.7630\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 2s 980us/step - loss: 0.7211 - accuracy: 0.7621 - val_loss: 0.6564 - val_accuracy: 0.7882\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 2s 985us/step - loss: 0.6447 - accuracy: 0.7879 - val_loss: 0.6003 - val_accuracy: 0.8048\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 2s 967us/step - loss: 0.6077 - accuracy: 0.8004 - val_loss: 0.5656 - val_accuracy: 0.8182\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 2s 984us/step - loss: 0.5692 - accuracy: 0.8118 - val_loss: 0.5406 - val_accuracy: 0.8236\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5428 - accuracy: 0.8194 - val_loss: 0.5196 - val_accuracy: 0.8314\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 2s 983us/step - loss: 0.5193 - accuracy: 0.8284 - val_loss: 0.5113 - val_accuracy: 0.8316\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 2s 992us/step - loss: 0.5128 - accuracy: 0.8272 - val_loss: 0.4916 - val_accuracy: 0.8378\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 2s 988us/step - loss: 0.4941 - accuracy: 0.8314 - val_loss: 0.4826 - val_accuracy: 0.8398\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ELU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def elu(z, alpha=1):\n",
    "    return np.where(z < 0, alpha * (np.exp(z) - 1), z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure elu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmCUlEQVR4nO3deZgU1d328e8PBtlBEB0XRIwK0RAlYZInatSJ4VEwGowaXNCIxkAgvErUROVFH1/Do9FgglFBMRjC4oK4giyuLaJERRgCKCCILKLsDQzbMDPn/eP04NCzNlMzVT19f66rr+mp6q769ZmavruqT50y5xwiIiJR0yDsAkRERMqjgBIRkUhSQImISCQpoEREJJIUUCIiEkkKKBERiSQFlIiIRJICSkREIkkBJWnHzMaa2dR6tJ4GZva4mW02M2dmubW9zkpqqZPXnFhXGzNbb2Yn1MX6UmVmk83s5rDryGSmkSTqNzMbC1xbzqwPnHM/Ssxv55y7sILnx4BFzrlBSdP7Ao8451oEWnD11t0av+3G02k9laz/QuAFIBf4HNjinCuozXUm1hsj6XXX1WtOrOsv+G3vutpeVznrPhu4FegGHA1c55wbm/SY7wLvAMc757bVdY0CWWEXIHXiDeCapGm1/gZYW+rqzaIO35ROBL5yzr1fR+urUF29ZjNrBtwAXFQX6ytHC2ARMC5xK8M5t9DMPgeuBh6tw9okQYf4MsNe59zXSbcttb1SM+thZu+a2VYz22JmM83s5FLzzcxuMbPPzGyvma01s/sS88YC5wC/Sxz2cmbWsWSemU01s/6JQ0RZSet9ysxerk4d1VlPqeU0NrMRiXXuMbN/m9mPS82PmdlIM7vXzDaZ2QYzG25mFf6fJdb/N6BDYt1flFrWI8mPLamnOus6mPZN9TUf7OsGLgCKgffKaZNuZvamme02s+VmdraZ9TazMo89WM65ac65Ic65yYk6KvIKcGVQ65XUKKCkNjUHRgA/xB++2gZMMbNDEvPvBe4E7gO+A/wSWJOYdxMwB/gncFTiVjKvxCTgUKB7yQQzaw70AiZUs47qrKfEA8DlwPXA94CFwAwzO6rUY/oAhcAZwCBgcOI5FbkJuAdYm1j3Dyp5bLKq1lXT9oXqvebq1JLsLOBjl/Qdg5n9AHgXeBs4Ffg38P+A/5t4LSQ9foiZ5VdxO6uSOqryIfBDM2tag2XIQdIhvszQw8zyk6Y96py7rTZX6px7vvTvZnYdsB3/D58H/B4Y7Jx7MvGQ5fg3TZxz28ysANjlnPu6guVvNbNp+DfHGYnJv8C/UU6pTh3OudlVrSfxnObAAOAG59yriWm/Bc4FfgcMTTz0E+fcXYn7y8zsN8BPgacreA3bzGwHUFTZ+itQ4brMrAUH0b5mdjCvOeXXDRwHfFXO9AeBKc65YYn1PYX/W85yzr1VzuMfw39QqcyXVcyvzDqgEf57qhU1WI4cBAVUZpgF9EuaFq/tlZrvnfUn4L+Aw/F77A2ADvjvwBoDb9ZwNROAsWbWzDm3Cx9Wk51ze6pZR3WdgH+j2n+YyTlXZGZzgFNKPe4/Sc9bBxyRwnpSUdm6TqHm7Vvd11xVLeVpCqwvPcHMjsTvWf2k1OQC/N+qzN5Top4tQG0ert6d+Kk9qBAooDLDLufc8oN87nagdTnTD8UfKqvMFPyn1/6Jn4XAJ8AhgFXyvFRMTSy3l5m9iT/cd14KdVRXSb3ldXstPW1fOfMO5lB6MWXbqFHS75WtK4j2re5rrqqW8mwC2iRNK/l+8qNS0zoDS51zs8st0GwIMKSS9QD0dM69W8VjKtI28XPjQT5fakABJVVZClxgZpb0fcH3E/PKZWaH4d9wfuecezsx7ft8s819AuzFHwb6rILFFAANKyvOObfXzCbj95zaAV/juwZXt45qrQd/eKwA+DG+Kzhm1hA4HXiqiucejI3474VKOw34oprPD6J9a/M1zwf6Jk07FB9sxYl1tcR/91TZoc/aPsTXBVjnnFtf5SMlcAqozNA4cfiktCLnXMmnwlZm1jVpftw59wUwCv+l98Nm9gSwB98D60p8Z4SKbMV/Sv6Nma0BjgH+gt97wTm3w8weAu4zs734w5CHAd2cc6MSy/gC/31VRyAff35QeT2uJuC70h8PPJX0mErrqO56nHM7zWwU8Gcz2wSsxH/Hkw2MrKQdDtZbwAgz+zn+g0B/4FiqGVAH275Jy6jN1zwTuN/MDnPObU5My8Pvtd1hZhPxf6evgBPN7CTnXJmgPdhDfInv6E5M/NoA34uyK/5vv7rUQ8/im+83pY6pF19m6I7/Ry99m19q/lmJ30vfhgM45z4HzgZOAl7D92q6Avilc25aRStMvMFfju+JtQh/Hsmd+E/1Je4A7k9M/xR4Hmhfav5w/Cf4T/B7FBV9ZzQL/yn5FA7svVfdOqq7ntvwn9b/iX8zPRXo4Zwr78v+mnqy1O09fIC8mOIygmjfWnnNzrmFfLMtlUxbid9jGgAsAHbgt91FQNDniOXwzbbeFN9TcD6+RyUAZtYE3+nmiYDXLdWkkSREJBRm1gN4CDjFOVcUdj3JzOx3QC/nXPJ3mlJHtAclIqFwzs3A79G2r+qxIdkH/J+wi8hk2oMSEZFI0h6UiIhEkgJKREQiKfRu5u3atXMdO3YMu4wydu7cSfPmzcMuI+2o3VKzdOlSioqKOOWU5IEZpDLptJ05B8uXw/btcMgh8O1vQ6PkU67rQJTb7OOPP97knDs8eXroAdWxY0fmzp0bdhllxGIxcnNzwy4j7ajdUpObm0s8Ho/k/0CUpct2VlwMV10F8+bBEUfA7Nlw0knh1BLlNjOzVeVN1yE+EZFa4BzcdBM8+yy0bAnTp4cXTulKASUiUguGDYNHHvGH9V5+Gb7//bArSj8KKBGRgD32GNx1FzRoAE8/DT/5SdXPkbICDSgzm2BmX5nZdjNbZmY3BLl8EZGomzwZBg7090eNgksuCbeedBb0HtR9QEfnXCvg58AwM+sW8DpERCLpzTehTx///dOwYdAv+SpskpJAA8o5t9g5VzIIp0vcTghyHSIiUfTxx3DxxVBQADfeCEOqukqVVCnwbuZmNhJ/nZem+NGBy4x4bWb9SFzhNTs7m1gsFnQZNZafnx/JuqJO7ZaaeDxOUVGR2ixFUdvO1qxpyo03fo/8/EM499z19Or1Ke+8U/Xz6lLU2qw6amUsvlIXNcsF7nfOJV9tc7+cnBwXxXNAonzOQJSp3VJTch5UXl5e2KWklShtZ+vWwRlnwKpVcP758Morvude1ESpzZKZ2cfOuZzk6bXSi885V5S4RHN7/LVdRETqna1bfSitWgX/9V/w/PPRDKd0VdvdzLPQd1AiUg/t2gUXXQSLFsHJJ8Orr0JERxJKW4EFlJkdYWZXmFkLM2toZufjLwv+VlDrEBGJgn37oHdveO89aN8eZs6Eww4Lu6r6J8hOEg5/OO8xfPCtAgY7514OcB0iIqEqLoYbbvB7TG3bwmuvwbHHhl1V/RRYQDnnNgLnBLU8EZEouu02GDcOmjWDadP84T2pHRrqSESkmv7yFxg+HLKy4IUXfMcIqT0KKBGRavjnP+GPf/T3x43zvfekdimgRESq8Mor8Jvf+PsPPQRXXhluPZlCASUiUol334XLL4eiIhg61A9jJHVDASUiUoH//Mef67Rnjx/49Z57wq4osyigRETKsXKl/55p2zZ/yYyRI8Es7KoyiwJKRCTJ+vVw3nnw9df+YoMTJ0LDhmFXlXkUUCIipWzfDj17wvLl8L3vwUsvQZMmYVeVmRRQIiIJe/b4azrNnw8nngjTp0OrVmFXlbkUUCIi+F56ffrA22/DkUf6IYyys8OuKrMpoEQk4zkHAwf60SFat/aDvx5/fNhViQJKRDLeXXfB6NH+u6YpU+DUU8OuSEABJSIZ7u9/h2HDfC+9SZPgrLPCrkhKKKBEJGM99RTcdJO//49/+JNyJToUUCKSkWbMgGuv9fcfeAD69g21HCmHAkpEMs4HH8Cll0JhIdx6K/zhD2FXJOVRQIlIRvn0U7jgAti1y+9B3X9/2BVJRRRQIpIx1qzxQxht2QIXXghPPAEN9C4YWfrTiEhG2LzZh9PatXDmmfDss9CoUdhVSWUUUCJS7+Xnw89+BkuWQJcu/lynZs3CrkqqooASkXqtoAAuu8x3jDjuOD9KRJs2YVcl1aGAEpF6q7jYdx+fORMOP9yPr3f00WFXJdWlgBKResk5GDwYnn4aWrTwI5N36hR2VZIKBZSI1Ev33gsPPwyHHAIvvwzduoVdkaRKASUi9c7o0TB0qL9E+8SJcO65YVckB0MBJSL1yvPPw4AB/v7Ikb6DhKQnBZSI1Btvvw1XXeU7R9xzD/z2t2FXJDWhgBKRemHePOjVy3crHzTIH+KT9KaAEpG099ln0KMH7NgBV1wBDz3kv3+S9KaAEpG0tm6dH8Jo40b/81//0vh69YX+jCKStuJxv+f0xRfwwx/6DhKHHBJ2VRIUBZSIpKXdu/0VcBcuhM6d4dVX/Qm5Un8EFlBm1tjMxpjZKjPbYWbzzaxnUMsXESlRVGRcfjnMng3HHOOHMGrXLuyqJGhB7kFlAWuAc4DWwJ3AJDPrGOA6RCTDOQfDh3diyhQ/6Otrr0GHDmFXJbUhK6gFOed2AneXmjTVzFYC3YAvglqPiGS222+HGTOOolkzf1jvlFPCrkhqS619B2Vm2UAnYHFtrUNEMsvw4fDAA9CwYTGTJ8Ppp4ddkdSmwPagSjOzRsBE4F/OuSXlzO8H9APIzs4mFovVRhk1kp+fH8m6ok7tlpp4PE5RUZHarBpmzMjm/vtPBmDw4AU0bboNNVv1peP/pjnngl2gWQPgKaAV0Ms5t6+yx+fk5Li5c+cGWkMQYrEYubm5YZeRdtRuqcnNzSUej5OXlxd2KZE2dSpcfDEUFcGIEXDaadrOUhXl/00z+9g5l5M8PdBDfGZmwBggG7i0qnASEanK7Nnwy1/6cBoyBG66KeyKpK4EfYhvFHAy0N05tzvgZYtIhlm40J/rtGcP3HADDBsWdkVSl4I8D+o4oD/QFfjazPITtz5BrUNEMscXX8D55/vRIn7xCxg1SuPrZZogu5mvArT5iEiNbdjgx9X76is45xx46inIqpUuXRJlGupIRCJl+3bo2dOPUN61q79ce5MmYVclYVBAiUhk7N3rD+fNmwcnnAAzZkDr1mFXJWFRQIlIJBQVwdVXw1tvwZFH+iGMsrPDrkrCpIASkdA5B7/7HUyeDK1a+T2nb30r7KokbAooEQnd3XfD449D48YwZQqcdlrYFUkUKKBEJFSPPAL33OOvgvvss3D22WFXJFGhgBKR0DzzDNx4o7//xBPQq1e49Ui0KKBEJBSvvQa/+pX//unPf4brrw+7IokaBZSI1LkPP4RLLoF9++Dmm+GPfwy7IokiBZSI1KklS+CCC2DnTrjmGvjLXzSEkZRPASUidWbtWj+E0ebNPqTGjPGdI0TKo01DROrE5s0+nNasgTPOgOeeg0aNwq5KokwBJSK1budOuPBC+PRT+M53/LlOzZqFXZVEnQJKRGrVvn1w2WXw739Dhw4wcya0bRt2VZIOFFAiUmuKi+G66/zQRe3a+a7lxxwTdlWSLhRQIlIrnPNdyCdOhBYtYPp06Nw57KoknSigRKRW/PnP8NBDviPEiy9CTk7YFUm6UUCJSOD+8Q8YMsSf3zRhAnTvHnZFko4UUCISqBdfhP79/f1HH4XevcOtR9KXAkpEAhOLwZVX+s4Rd98NAwaEXZGkMwWUiARi/nz4+c/9ZdsHDoS77gq7Ikl3CigRqbHly6FHD9ixwx/S+/vfNb6e1JwCSkRq5Kuv4PzzYcMG3xli3Dho2DDsqqQ+UECJyEGLx6FnT/j8c9+N/IUX/GXbRYKggBKRg7J7t78C7oIF0KkTTJsGLVuGXZXUJwooEUlZYaHvrTdrFhx9tB/C6PDDw65K6hsFlIikxDl/ntPLL0ObNj6cjjsu7KqkPlJAiUhKhgyBJ5+Epk1h6lR/+QyR2qCAEpFq++tf/Rh7DRvC5Mn+woMitUUBJSLVMn483HKLvz92rL9ku0htUkCJSJVefdVf1wngb3+Dq68Otx7JDAooEanU++/DL38JRUVwxx0weHDYFUmmUECJSIUWLYKf/cyf8/TrX8P//m/YFUkmCTSgzGyQmc01s71mNjbIZYtI3Vq1yg9hFI/DxRfDY49pfD2pW1kBL28dMAw4H2ga8LJFpI5s3AjnnQfr1sE558DTT0NW0O8WIlUIdJNzzr0AYGY5QPsgly0idWPHDt9Db9kyOO00f0JukyZhVyWZKJTPRGbWD+gHkJ2dTSwWC6OMSuXn50eyrqhTu6UmHo9TVFQUmTYrKDDuuONU5s1rw9FH7+auu+Yzf35B2GWVoe0sdenYZqEElHNuNDAaICcnx+Xm5oZRRqVisRhRrCvq1G6pOfTQQ4nH45Fos6IiP77evHmQnQ2zZjXlhBOieSautrPUpWObqRefiOAc3HgjPPcctGoFM2bACSeEXZVkOgWUiHDPPTBypL+W0yuvQNeuYVckEvAhPjPLSiyzIdDQzJoAhc65wiDXIyLBGTkS7r4bGjSAZ57xvfZEoiDoPaihwG7gduDqxP2hAa9DRAIyaRIMGuTvjx7tz3cSiYqgu5nfDdwd5DJFpHa88YYfU885uO8+P1KESJToOyiRDPTRR35vad8++P3v4bbbwq5IpCwFlEiGWbrUn4i7c6ffgxo+XEMYSTQpoEQyyJdf+iGMNm2Cnj39lXEb6F1AIkqbpkiG2LLFD/66ejWcfro/56lRo7CrEqmYAkokA+zaBRdeCIsXwymnwNSp0Lx52FWJVE4BJVLP7dvnLzg4Zw506AAzZ0LbtmFXJVI1BZRIPVZcDNdfD9OmQbt28Npr0F7XGZA0oYASqaecg1tvhQkT/OG8adOgc+ewqxKpPgWUSD31wAPwt7/5jhAvvgg/+EHYFYmkRgElUg+NGQO33+7Pb5owAf77v8OuSCR1CiiReuall6BfP3//kUegd+9QyxE5aAookXpk1iy44grfOeJ//gcGDgy7IpGDp4ASqScWLICLLoK9e2HAAB9QIulMASVSD3z+uR8lYvt2f87Tww9rfD1JfwookTT39dd+fL316+GnP4Xx46Fhw7CrEqk5BZRIGtu2zQ/6umIFdOvmu5M3bhx2VSLBUECJpKk9e6BXL8jLg06dYPp0aNky7KpEgqOAEklDhYVw5ZXwzjtw9NF+fL3DDw+7KpFgKaBE0oxzvpfeSy/BoYf6cOrYMeSiRGqBAkokzQwdCv/4BzRt6i+b0aVL2BWJ1A4FlEgaGTEC7r3X99J77jk488ywKxKpPQookTQxcSL8/vf+/pNPws9+Fm49IrVNASWSBqZPh759/f0HH4Rf/SrUckTqhAJKJOLmzIFLL/U99267DW6+OeyKROqGAkokwhYv9ofydu/2V8a9776wKxKpOwookYhavdqPr7d1K/z85/D44xpfTzKLAkokgjZt8uPrffklnHUWPPMMZGWFXZVI3VJAiURMfj5ccAEsXQqnngqvvOLPeRLJNAookQgpKIBLLoGPPoLjj4cZM/xoESKZSAElEhHFxb77+OuvwxFHwGuvwVFHhV2VSHgUUCIR4BzcdBM8+6wfkXzGDDjxxLCrEgmXAkokAoYNg0cegUMO8d85fe97YVckEr5AA8rM2prZi2a208xWmdlVQS5fpD7avLkxd90FDRrA009Dbm7YFYlEQ9AdVx8FCoBsoCvwqpktcM4tDng9IvXCxo2wdq3vovfYY76DhIh45pwLZkFmzYGtQBfn3LLEtPHAl8652yt6XsuWLV23bt0CqSFI8XicQ9V9KmVqt+rbsgUWLswD4Pjju9KhQ7j1pBNtZ6mLcpu98847HzvncpKnB7kH1QkoKgmnhAXAOckPNLN+QD+ARo0aEY/HAywjGEVFRZGsK+rUbtWTn5/F5583ByArq5hWreKo2apP21nq0rHNggyoFsC2pGnbgJbJD3TOjQZGA+Tk5Li5c+cGWEYwYrEYufoyIGVqt6rNnQvnnut77h11VC5HHBEnLy8v7LLSiraz1EW5zayCMbyC7CSRD7RKmtYK2BHgOkTSWl4e9OgBO3bAFVfASSeFXZFIdAUZUMuALDMr/S93GqAOEiLAhx/CT34CmzfDhRfCuHEa/FWkMoEFlHNuJ/ACcI+ZNTezM4FewPig1iGSrmbPhu7dIR6Hiy+GyZOhUaOwqxKJtqBP1B0INAU2AE8DA9TFXDLdW2/5y2aUHNabNAkaNw67KpHoC/Q8KOfcFuDiIJcpks6eew6uuQb27oVrr4UxY6Bhw7CrEkkPGupIpBY4B8OHQ+/ePpwGDoQnn1Q4iaRCASUSsMJCGDQI/vAH//v99/tx9hrov00kJbpGp0iAtm2DPn3g1Vf9wK/jxsHll4ddlUh6UkCJBGTRIj+W3mefQdu28NJL/nLtInJwdNBBJACTJsGPfuTD6bTT/BVxFU4iNaOAEqmBvXvh5pv9YbydO/3hvfffh299K+zKRNKfDvGJHKRPP4WrrvLDF2VlwV//6jtHaHQIkWAooERS5Bw8/rjfc9q92+8tTZzoD/GJSHB0iE8kBatX+3H0Bgzw4XTttTB/vsJJpDYooESqobgYHn0UvvMdmDYNWrf2l2cfOxZaJY/hLyKB0CE+kSosXgy//a0f8BV8V/JHHoGjjgq3LpH6TntQIhWIx2HwYN9tfPZsOPJIPwr5888rnETqggJKJElRETzxhL+Y4EMP+U4RAwbAJ5/ApZeGXZ1I5tAhPpEE5/zoD0OH+jACOOccH1KnnRZqaSIZSXtQkvGcgzff9D3xLrnEh1PHjvDMM/D22wonkbBoD0oylnMwfTrcey+8956flp0Nd94Jv/mNH+xVRMKjgJKMU1gIL7wA993nR4EAP7jrLbfATTdB8+ahliciCQooyRhbtvjOD48+CmvW+GlHHgm33gr9+0OLFuHWJyIHUkBJveYc/Pvf/lLrTz3lR38A6NTJdyG/7jpo0iTUEkWkAgooqZe+/hrGj/eXWV+y5JvpPXrAjTfC+efrCrciUaeAknpj507f6WHcOD8cUVGRn56dDb/6Ffz619C5c7g1ikj1KaAkre3Y4S+vPnmyD6WSQ3hZWXDxxXD99X6vqVGjUMsUkYOggJK08+WXMHMmvPIKzJjhLxpY4kc/gt69/YUDjzgivBpFpOYUUBJ5e/f685RmzPC3hQu/mWfmL61+2WX+JNv27cOrU0SCpYCSyNm7Fz76CGbN8rfZs/33SyWaN4dzz4WePf1hPA3cKlI/KaAkdOvX+0D64AN4913fLbz0YTuALl38d0k9esCPfwyNG4dTq4jUHQWU1KkNG/whurlzfSh99JG/Sm2yLl3g7LP97ayz4Oij675WEQmXAkpqxc6d/kJ/CxfCokX+58KFPqCStWgB3brBD37g945+/GM47LC6r1lEokUBJQdt715YuRI++8zfli+HDz88lc2bYdUqP4pDspYt/d5R167wwx/6W+fO0LBhnZcvIhGngJJyOQfbtvkx69as8YfhSt9ftcr/LC5OfmZbwJ+H9O1vw3e/629duvifxx3ne96JiFRFAZVhCgth40bfMWHDBv+z5LZhgx8iaO1aHz75+ZUvq0EDOP54f+XZk06CE0+E3bv/wyWXnMrxx+tyFSJSMwqoNOScHzFh+3bYutWP0r11a9X3N22CzZvLP/RWnmbNoEMHOPZYf0u+X14IxWJbNJyQiAQikIAys0FAX+C7wNPOub5BLDddFRf7ANmz58CflU3Lz/e3HTsq/1lyK3torXrM4PDD/SgL2dnf3Er/3r69D6E2bXQ4TkTCE9Qe1DpgGHA+0DSVJ+7dC8uW+YE9S9+Ki8tOC2p6YSHs2wcFBQf+LH3/yy9P5uGHy06v6H5BwTeBs29fQK1aiSZNfIeDtm19kJTcKvu9XTt/y9J+s4ikAXPVPd5TnYWZDQPap7IHZdbSQbekqb2BgcAu4IJyntU3cdsEXFbO/AHA5cAa4Jpy5t8CXAQsBfqXM38o0B3IAwaXM/9e4AzgfWBIOfNH0LRpVxo2fIOCgmE0aMABty5dHuewwzqzdesUPvvsQRo08L3YSm433DCeDh2OJS/vWV5/fdQB8xo2hMmTJ3Pkke0YO3YsY8eOLbP2adOm0axZM0aOHMmkSZPKzI/FYgAMHz6cqVOnHjCvadOmTJ8+HYA//elPvPnmmwfMP+yww3j++ecBuOOOO5gzZ84B8xs1asTrr78OwODBg8kruWRtQqdOnRg9ejQA/fr1Y9myZQfM79q1KyNGjADg6quvZu3atQfMP/3007nvvvsAuPTSS9m8efMB83/6059y5513AtCzZ092l4wem3DhhRdy6623ApCbm0uy3r17M3DgQHbt2sUFF5Td9vr27Uvfvn3ZtGkTl11WdtsbMGAAl19+OWvWrOGaa8pue7fccgsXXXQRS5cupX///uTl5VFYWEhOTg4AQ4cOpXv37uTl5TF48OAyz7/33ns544wzeP/99xkypOy2N2LECLp27cobb7zBsGHDysx//PHH6dy5M1OmTOHBBx8sM3/8+PEce+yxPPvss4waNarM/MmTJ9OuXfjbXp8+ffjyyy8PmN++fXsmTJgAaNsrb9s777zzGDJkyP5tL1mY294777zzsXMuJ/k5oXyWNrN+QD//W3MOOaQ4cSjJYQYtW+6hTZsdwE7Wri1MPOebw03t2uWTnb2ZoqLNLFu2b//0kmUce2ycY475ir1717NgQQFmrtR8+Pa3N9Kx4yp27lzLnDl7MHP7l2/mOPPMVbRvP4/8/JXMnLlz//SSx/ziF0vo1KkRK1d+wgsvbN8/vUEDR4MGMGjQXE46Kc7HHy9g/Ph4mdffv/8HdOjwFe+/v5AdO8rOP+GEORxxxAqWLl0MxPfv+ZX44IP3aN26NUuWLCEeL/v8WbNm0aRJE5YtW1bu/JI3iRUrVpSZv3v37v3zV65cWWZ+cXHx/vmrV68uM79Nmzb7569du7bM/HXr1u2fv27dujLz165du3/++vXry8xfvXr1/vkbN25k+/btB8xfuXLl/vlbtmxhb9KQFCtWrNg/v7y2WbZsGbFYjD179pQ7f8mSJcRiMbZt21bu/MWLFxOLxdiwYUO58xcuXEjLli33t11hYSHOuf2PXbBgAVlZWSxfvrzc58+bN4+CggIWLVpU7vy5c+cSj8dZsGBBufM/+OADvvrqKxYuXFju/Dlz5rBixQoWL15c7vz33ovGtldQUFBmfqNGjbTtVbLt7dmzh1gsVu7/LYS/7ZUn9D2onJwcN3fu3MBqCEosFiv3U45UTu2WmtzcXOLxeJlP+1I5bWepi3KbmVm5e1BVXlPUzGJm5iq4za6dckVEJNNVeYjPOZdbB3WIiIgcIKhu5lmJZTUEGppZE6DQOVcYxPJFRCTzVHmIr5qGAruB24GrE/eHBrRsERHJQIHsQTnn7gbuDmJZIiIiENwelIiISKAUUCIiEkkKKBERiSQFlIiIRJICSkREIkkBJSIikaSAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhEkgJKREQiSQElIiKRpIASEZFIUkCJiEgkKaBERCSSFFAiIhJJCigREYkkBZSIiESSAkpERCJJASUiIpGkgBIRkUhSQImISCQpoEREJJIUUCIiEkkKKBERiSQFlIiIRJICSkREIkkBJSIikaSAEhGRSFJAiYhIJNU4oMyssZmNMbNVZrbDzOabWc8gihMRkcwVxB5UFrAGOAdoDdwJTDKzjgEsW0REMlRWTRfgnNsJ3F1q0lQzWwl0A76o6fJFRCQz1TigkplZNtAJWFzJY/oB/QCys7OJxWJBl1Fj+fn5kawr6tRuqYnH4xQVFanNUqTtLHXp2GbmnAtuYWaNgOnACudc/+o8Jycnx82dOzewGoISi8XIzc0Nu4y0o3ZLTW5uLvF4nLy8vLBLSSvazlIX5TYzs4+dcznJ06v8DsrMYmbmKrjNLvW4BsB4oAAYFGj1IiKScao8xOecy63qMWZmwBggG7jAObev5qWJiEgmC+o7qFHAyUB359zugJYpIiIZLIjzoI4D+gNdga/NLD9x61PTZYuISOYKopv5KsACqEVERGQ/DXUkIiKRpIASEZFICvQ8qIMqwGwjsCrUIsrXDtgUdhFpSO2WOrVZ6tRmqYtymx3nnDs8eWLoARVVZja3vBPHpHJqt9SpzVKnNktdOraZDvGJiEgkKaBERCSSFFAVGx12AWlK7ZY6tVnq1GapS7s203dQIiISSdqDEhGRSFJAiYhIJCmgREQkkhRQ1WRmJ5nZHjObEHYtUWZmjc1sjJmtMrMdZjbfzHqGXVcUmVlbM3vRzHYm2uuqsGuKMm1bNZOO72EKqOp7FPgo7CLSQBawBjgHaA3cCUwys45hFhVRj+Iv8JkN9AFGmdl3wi0p0rRt1UzavYcpoKrBzK4A4sCbIZcSec65nc65u51zXzjnip1zU4GVQLewa4sSM2sOXArc6ZzLd87NBl4Brgm3sujStnXw0vU9TAFVBTNrBdwD3BJ2LenIzLKBTsDisGuJmE5AkXNuWalpCwDtQVWTtq3qSef3MAVU1f4EjHHOrQm7kHRjZo2AicC/nHNLwq4nYloA25KmbQNahlBL2tG2lZK0fQ/L6IAys5iZuQpus82sK9Ad+FvIpUZGVW1W6nENgPH471gGhVZwdOUDrZKmtQJ2hFBLWtG2VX3p/h5W4yvqpjPnXG5l881sMNARWG1m4D/1NjSzU5xz36/t+qKoqjYDMN9YY/Bf/l/gnNtX23WloWVAlpmd5Jz7LDHtNHS4qlLatlKWSxq/h2moo0qYWTMO/JR7K/6PPcA5tzGUotKAmT0GdAW6O+fyQy4nsszsGcABN+DbaxpwhnNOIVUBbVupSff3sIzeg6qKc24XsKvkdzPLB/akwx82LGZ2HNAf2At8nfjUBtDfOTcxtMKiaSDwJLAB2Ix/01A4VUDbVurS/T1Me1AiIhJJGd1JQkREoksBJSIikaSAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhE0v8HODFknLoJseMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, elu(z), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [-1, -1], 'k--')\n",
    "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
    "plt.grid(True)\n",
    "plt.title(r\"ELU activation function ($\\alpha=1$)\", fontsize=14)\n",
    "plt.axis([-5, 5, -2.2, 3.2])\n",
    "\n",
    "save_fig(\"elu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementing ELU in TensorFlow is trivial, just specify the activation function when building each layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7ff1b25afad0>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"elu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SELU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This activation function was proposed in this [great paper](https://arxiv.org/pdf/1706.02515.pdf) by Günter Klambauer, Thomas Unterthiner and Andreas Mayr, published in June 2017. During training, a neural network composed exclusively of a stack of dense layers using the SELU activation function and LeCun initialization will self-normalize: the output of each layer will tend to preserve the same mean and variance during training, which solves the vanishing/exploding gradients problem. As a result, this activation function outperforms the other activation functions very significantly for such neural nets, so you should really try it out. Unfortunately, the self-normalizing property of the SELU activation function is easily broken: you cannot use ℓ<sub>1</sub> or ℓ<sub>2</sub> regularization, regular dropout, max-norm, skip connections or other non-sequential topologies (so recurrent neural networks won't self-normalize). However, in practice it works quite well with sequential CNNs. If you break self-normalization, SELU will not necessarily outperform other activation functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.special import erfc\n",
    "\n",
    "# alpha and scale to self normalize with mean 0 and standard deviation 1\n",
    "# (see equation 14 in the paper):\n",
    "alpha_0_1 = -np.sqrt(2 / np.pi) / (erfc(1/np.sqrt(2)) * np.exp(1/2) - 1)\n",
    "scale_0_1 = (1 - erfc(1 / np.sqrt(2)) * np.sqrt(np.e)) * np.sqrt(2 * np.pi) * (2 * erfc(np.sqrt(2))*np.e**2 + np.pi*erfc(1/np.sqrt(2))**2*np.e - 2*(2+np.pi)*erfc(1/np.sqrt(2))*np.sqrt(np.e)+np.pi+2)**(-1/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def selu(z, scale=scale_0_1, alpha=alpha_0_1):\n",
    "    return scale * elu(z, alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure selu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmLUlEQVR4nO3de3gV1dn+8e9DwhkkCJiKILxWUakHxPxsUVtjoR4QRMWKCiq1CmKxYsETglJBUUSLVkGwWCoHBUVFDtpXbeOrxVqhUC0qeAAEj4AECOck6/fH2kjYCSE7mWRm731/rmsudvZMZp5Mhn1nZtasZc45REREoqZW2AWIiIiURQElIiKRpIASEZFIUkCJiEgkKaBERCSSFFAiIhJJCiiRAzCzKWY2rwa2k2tmzsya18C2+pnZ52ZWbGYjqnt7B6ilr5kVhFmDRJMCShJiZi3MbLyZrTKznWb2jZm9bma/KLFMXuyDNn56psQyzswuLmP9bWPzcsqYl2dmj1bjz7a/gLgR6BPwtlaZ2ZC4txcChwIbgtxWGdtuCjwGPAAcBoytzu3Fbbus3/tM4IiaqkGSR2bYBUjSmQ00AH4NfAIcApwBNItb7s/A0Lj3tld7ddXAObephrazC/i6BjbVBv9/f55z7qsa2F65nHPbSdJjQ6qXzqCkwswsC/gpcJtz7nXn3Grn3LvOubHOuWfiFt/mnPs6bqrWD3oz+6GZzTGzr81sq5n928y6xS1Tx8zuNbPVsTPAz8zst2bWFvh7bLF1sb/0p8S+5/tLfGbWP3bWmBm33hlmNqcidZhZHj4kHthzdhl7v9QZnJldZGbvx2pdY2Z3mJmVmL/KzIaZ2UQz22xma83s5nL2UV9gSezLz2Lba2tmI8zsv/HLlrz0tmcZM7vUzD41sy1m9mL8GaeZXVWi5m9K7MdVsUWejW13VVnbKbGfPzGzXbF/r42b72KXKZ+N7ePPzCzQs1wJnwJKElEQm843s3phF1OGRsDLwC+AE/Fne8+b2TEllvkLcCXwO+BY/JlgPrAG6Blb5kf4S203lrGNWUAW0GXPG2bWEOgBTKtgHRcBa4G7Y9s5tKwfxsxOBp4FngeOB24DbgcGxi16E/A+0BG4HxhjZp3KWif+cto5sdenxLa9Zj/LlqUt0Au4EDgLOAm4p0TN/YGJ+DPoE4CuwLLY7P8X+/fa2Hb3fL0PM7sQeBQYBxwHPAyMN7PucYveCczB7+OZwJNm1iaBn0WizjmnSVOFJ/yH+HfADuBt/P2LH8ctkwfsYm+g7ZmuL7GMAy4uY/1tY/NyypiXBzyaYL3/BIbFXh8VW/c5+1k2Nza/edz7U/CXw/Z8/QIwtcTXfYBNQL2K1BH7ehUwpLztA9OBv8UtMwJYG7eep+OW+bjktsqoJSe2nbZx6/1v3HJ9gYK4ZXYATUq8dwfwSYmv1wL3lbPtUr/3MrbzD+DJMn4Hb8WtZ3SJrzOBbUCfsP+PaApu0hmUJMQ5NxtoCXTHnyWcCvzTzOLvN80EOsRN06uzNjNraGZjzOwDM9sYu2yUAxweW+QkoJi9l/IqaxpwgZk1iH3dG3jOObejgnVU1LH4D+uS3gIOM7ODSrz3XtwyX+LvDVaH1W7fS7Xfb8vMDsE3uni9itvY38/dPu69739u51whsI7q+7klBGokIQmLfRC/GpvuNrM/ASPMbKzzN/oBNjnnPqnE6vd8+DUpY15WifllGYu/fDUEfxaxDXgKqBObb/v5vkTNAwqBHmb2Ov5y31kJ1FFRhj9TKEvJ93eXMS/RPz6LKb1/apexXHnbCmr/7lnvgd4L4ueWCNMvU4LwAf6PnSrfl3LObQTWAyeXfD92xnAksLycbz8deMo5N9s59x7+ctMPS8z/N/6YP3M/378nXDMOUONO4Dn8mVMvfMu7NxKoY8+2yt0Ofr+eHvfe6fhLfFsO8L2JWgdkl2yAgT/rrTDn3DfAF0DnchbbzYF/7g8p++f+IJF6JPnpDEoqzMya4W/aP4m/vLIFf+nqFuB159zmEos3MLMfxK1il3PuuxJftzWzDnHLfAY8BNxmZl/i73M1A4bjg+vZckpcAVwYa023G7iLEqHpnPvYzGYBfzKzG/GB1Qp/L2YqsBr/V/h5ZjYX2O6c298DpNOA14D/AWY454orWkfMKuCnZjYN2OmcW1/GNh4E3jX/IO0MfKOCwZRuvh+EPOBgYKj559VygVLPqVXAPcAfzOwbYD7+kYTOzrkHY/NXAZ3N7A38z72xjHU8gG/ptxj4X/zZaG984xJJJ2HfBNOUPBNQF7gXeBfYiL909TE+UA4usVwe/oM+foq/yV3W1A3/F/YN+BAswJ+BPEOJm/r7qa8NPjS2xr5nCP5y3JS4n2EM/i/9ncCnwMAS84cDX+EveU2JvTeFEo0kYu8Z/sPWAcdXoo6fAP/BNzpwsfdyiWukgf9Qfh9/xrUG3yjBSsxfRenGFnmU05iEMhpJxN7vjw/prbH9fSOlG0mU25Ai9t6v8Wc7e57rerLEvO6xY2Y3sKqcdVyHf85ud+zfa+Pml9XYotS+0JTck8V+sSIiIpGie1AiIhJJCigREYkkBZSIiESSAkpERCIp9GbmzZs3d23btg27jFK2bt1Kw4YNwy4j6Wi/JWb58uUUFRXRvn18JwlSnqgeZwUFsGIFOAetWkF2dtgV7RXVfQawePHi9c65FvHvhx5Qbdu2ZdGiRWGXUUpeXh65ublhl5F0tN8Sk5ubS35+fiT/D0RZFI+zFSugUycfTgMHwiOPgAXZt0YVRXGf7WFmq8t6X5f4RESqaN066NoVvvsOunWDceOiFU7JSgElIlIF27dDjx7w6afQsSM8/TRkHKgzJ6kQBZSISCUVF8NVV8Hbb0Pr1jBvHjRqFHZVqUMBJSJSSUOHwrPPQuPGMH8+HFrm0JNSWYEGlJlNM7OvYkNPrzCza4Jcv4hIVDzxBNx/v7+c99xzcPzxYVeUeoI+gxqN74DyIOB8YFRs2GoRkZTx17/CgAH+9eOPw1lnlb+8VE6gAeWcW+b8WDmwt3fq+HFwRESS1nvvwS9/CUVFcPvtcI2uE1WbwJ+DMrPx+O7z6wNLgAVlLNMP6AeQnZ1NXl5e0GVUWUFBQSTrijrtt8Tk5+dTVFSkfZagsI6z9evrcP31HdmypR5nnvktXbp8QLL86pLx/2a1DLdhZhlAJ/z4Nvc75+KHZv5eTk6Oi+JDilF+qC3KtN8Ss+dB3aVLl4ZdSlIJ4zgrKICf/QyWLIHTToPXXoN6VR5DuuZE+f+mmS12zuXEv18trficc0XOubfwo5UOqI5tiIjUlMJCuPRSH05HHgkvvphc4ZSsqruZeSa6ByUiScw5uPFG34y8WTNYsACaNw+7qvQQWECZ2SFmdqmZNTKzDDM7G7gM+FtQ2xARqWl/+AOMHw916vgzp6OOCrui9BFkIwmHv5z3OD74VgODnHNzAtyGiEiNeeEFGDLEv/7LX+D008OtJ90EFlDOuXXAGUGtT0QkTO+8A717+0t8997r70FJzVJXRyIicVauhO7dfUewv/413HZb2BWlJwWUiEgJGzf6oTPWrYNf/AImTNDQGWFRQImIxOzaBT17wkcfwXHH+Y5ga9cOu6r0pYASEcHfa7r2Wvj73+EHP/DNyps0Cbuq9KaAEhEB7r4bnnoKGjTw4zodfnjYFYkCSkTS3tSpMGIE1KoFzzwDJ2sMhkhQQIlIWsvL8y31AMaN8633JBoUUCKStj78EC68EHbvhkGD4IYbwq5ISlJAiUha+vZbOO88yM+HHj1g7NiwK5J4CigRSTvbt8P55/sHcnNyYPp0P3S7RIsCSkTSSnExXHGF78qoTRuYOxcaNgy7KimLAkpE0sqtt8Ls2f4Zp/nz/TNPEk0KKBFJGxMm+HtNmZk+pH70o7ArkvIooEQkLSxYAAMH+tdPPAGdO4dbjxyYAkpEUt7SpdCrl7//NGwY9O0bdkVSEQooEUlpa9f65uQFBXD55b5LI0kOCigRSVmbN/tw+vJL+NnP4MknNXRGMlFAiUhKKiz0l/Xeew/atfPDt9etG3ZVkggFlIikHOd8g4hXXoHmzX0DiYMPDrsqSZQCSkRSztixMHGiP2N66SX44Q/DrkgqQwElIinl2Wfhllv866lToVOncOuRylNAiUjKePtt340RwP33wy9/GW49UjUKKBFJCZ9+6juA3bkT+vWDm28OuyKpKgWUiCS9776Drl1h/Xo45xx47DE1J08FCigRSWo7d8IFF8CKFXDCCTBzpu9rT5KfAkpEkpZzcPXV8Oab0LKl7538oIPCrkqCooASkaR1110wY4Yfz2nePGjVKuyKJEgKKBFJSlOmwMiRUKsWzJoFJ50UdkUSNAWUiCSdxYuzuPZa//rRR30DCUk9CigRSSoffAB33XUchYUweDAMGBB2RVJdFFAikjS+/tqfLW3dmknPnjBmTNgVSXVSQIlIUti6Fbp3h9Wr4dhjNzN1qr//JKkrsF+vmdU1s8lmttrMtpjZEjM7N6j1i0j6KiqC3r1h0SL4n/+Be+55n/r1w65KqluQf39kAmuAM4AmwHBglpm1DXAbIpKGhgyBOXMgK8s/69S06e6wS5IaEFhAOee2OudGOOdWOeeKnXPzgJXAyUFtQ0TSz6OPwrhxULu2H3Tw2GPDrkhqSrV1CGJm2UA7YFkZ8/oB/QCys7PJy8urrjIqraCgIJJ1RZ32W2Ly8/MpKirSPtuPhQubMXz4cYBx880fAt+Ql6fjrDKScZ+Zcy74lZrVBl4GPnXO9S9v2ZycHLdo0aLAa6iqvLw8cnNzwy4j6Wi/JSY3N5f8/HyWLl0adimRs3gx/OxnsG0bjBjhe43YQ8dZ4qK8z8xssXMuJ/79wNvAmFktYCqwCxgY9PpFJPV9/jl06+bD6cor4c47w65IwhDoJT4zM2AykA10dc7pTqaIJGTTJjjvPP/M05lnwhNPaOiMdBX0PagJwLFAF+fc9oDXLSIpbvduPwruf/8LxxwDs2dDnTphVyVhCfI5qDZAf6AD8LWZFcSm3kFtQ0RSl3O+26JXX4VDDoEFC6Bp07CrkjAFdgblnFsN6ERcRCrlvvtg8mSoVw9eesk/kCvpTR2FiEjonnkGhg7195qmT4cf/zjsiiQKFFAiEqq33oK+ff3rsWPhootCLUciRAElIqH5+GPo0QN27oTrr4ebbgq7IokSBZSIhGL9ej90xnff+X8ffljNyWVfCigRqXE7dsAFF8Ann/ih2mfOhMxq63hNkpUCSkRqVHEx/OpX8I9/QKtWMG8eNGoUdlUSRQooEalRw4b5VnuNG/uhM1q2DLsiiSoFlIjUmD/9CUaPhowMePZZOOGEsCuSKFNAiUiNePVVuO46/3r8eDj77HDrkehTQIlItXv/fbj4Yj90+623Qr9+YVckyUABJSLV6ssvfe/kmzfDJZfAvfeGXZEkCwWUiFSbggLo3h3WrIFOnWDKFKilTx2pIB0qIlItiorg8svh3/+GH/4Q5syB+vXDrkqSiQJKRALnHAwaBHPnwsEH+6EzWrQIuypJNgooEQncww/Do4/6wQZffBHatQu7IklGCigRCdSLL8Lvfudf//nP8NOfhlqOJDEFlIgE5t13/X0n52DUKP9apLIUUCISiFWrfIu97dvh6qv9AIQiVaGAEpEqy8/3Q2Z88w107gyPP66hM6TqFFAiUiW7dkHPnvDhh9C+PTz3HNSuHXZVkgoUUCJSac5B//7wt7/BD37gm5NnZYVdlaQKBZSIVNo99/jeIRo08M88tWkTdkWSShRQIlIp06fD8OH+XtOMGZCTE3ZFkmoUUCKSsP/7P99SD+APf4AePcKtR1KTAkpEErJ8OVxwgW8c8dvfwo03hl2RpCoFlIhU2Lp1vjn5xo1w/vnw0ENhVySpTAElIhWyfbsPpc8+g5NP9vedMjLCrkpSmQJKRA6ouBiuvBL++U84/HDfYq9hw7CrklSngBKRA7r9dv8A7kEHwfz5cOihYVck6UABJSLlmjgRxoyBzEyYPRuOOy7siiRdKKBEZL9eeQV+8xv/euJE6NIl3HokvSigRKRM//kP/PKXfuj2O+7Y+9yTSE1RQIlIKV98AeedBwUFcNllMHJk2BVJOgo0oMxsoJktMrOdZjYlyHWLSM3YsgW6dfMhdfrpflRcDZ0hYcgMeH1fAqOAs4H6Aa9bRKpZYSH06gVLl8JRR/nh2+vWDbsqSVeBBpRz7nkAM8sBWgW5bhGpXs75rotefhmaNfNDZzRrFnZVks6CPoOqEDPrB/QDyM7OJi8vL4wyylVQUBDJuqJO+y0x+fn5FBUVRWKfzZrVigkTjqR27WJGjFjK2rWbWbs27KrKpuMsccm4z0IJKOfcJGASQE5OjsvNzQ2jjHLl5eURxbqiTvstMVlZWeTn54e+z2bP9sO0A0ybVotLLukYaj0HouMsccm4z9SKTyTN/fOf0KePv8Q3ejRccknYFYl4CiiRNPbZZ74D2B074Npr4dZbw65IZK9AL/GZWWZsnRlAhpnVAwqdc4VBbkdEqm7jRv+s07p1cNZZ8Nhjak4u0RL0GdQwYDtwG9An9npYwNsQkSrauRMuugg++giOPx6efRZq1w67KpF9Bd3MfAQwIsh1ikiwnPOX8/LyfK/k8+f7XspFokb3oETSzO9/D1On+vGc5s2D1q3DrkikbAookTTy1FM+oGrVgpkzoWO0W5NLmlNAiaSJv/8drrnGv37kEd9AQiTKFFAiaeDDD+HCC2H3brjppr1jPIlEmQJKJMV98w107QqbNvmQeuCBsCsSqRgFlEgK27bNP4i7ahWccgpMmwYZGWFXJVIxCiiRFFVU5Lsw+te/oG1beOklaNAg7KpEKk4BJZKibrkFXngBmjTxzzplZ4ddkUhiFFAiKWj8eHjoId87xPPPQ/v2YVckkjgFlEiKmT8fbrjBv37iCfj5z8OtR6SyFFAiKWTJEj9ke3Ex3HknXHVV2BWJVJ4CSiRFrFnjH77dutU3jhgxIuyKRKpGASWSAjZv9uH01Vdwxhnwpz9p6AxJfgookSS3e7cfBff99+Hoo33Lvbp1w65KpOoUUCJJzDnfbdFf/wotWsCCBdC0adhViQRDASWSxMaM8S316tXzD+IecUTYFYkERwElkqRmzYLbbvP3mqZNg5/8JOyKRIKlgBJJQgsXwpVX+tdjxkDPnuHWI1IdFFAiSeaTT6BHD9i5E667DgYPDrsikeqhgBJJIhs2+KEz1q+Hc8+FP/5RzckldSmgRJLEzp1+PKePP4YTT/RDtmdmhl2VSPVRQIkkAefg6qvhzTfhsMN8f3uNG4ddlUj1UkCJJIE774QZM6BRIx9Ohx0WdkUi1U8BJRJxTz4Jo0b5kXBnzfKX90TSgQJKJMJeew369/evH3vMN4wQSRcKKJGIWrbMP99UWAg337w3qETShQJKJIK+/to3J9+8GS6+GO67L+yKRGqeAkokYrZuhW7d4PPPffdFTz0FtfQ/VdKQDnuRCCkqgssvh8WLfcevc+ZA/fphVyUSDgWUSIQMHux7JW/a1A+dccghYVckEh4FlEhEPPIIPPww1K7tBx08+uiwKxIJlwJKJAJeegkGDfKvn3zSD9suku4CDSgzO9jMXjCzrWa22swuD3L9Iqlo27YMLrvMd2d0993Qp0/YFYlEQ9BdTT4G7AKygQ7AfDP7j3NuWcDbEUkJO3fCypUNKSyEvn1h2LCwKxKJDnPOBbMis4bARuA459yK2HtTgS+cc7ft7/saN27sTj755EBqCFJ+fj5ZWVlhl5F0tN8S849/LKWwELKyOnDCCRo6o6J0nCUuyvvsjTfeWOycy4l/P8gzqHZA0Z5wivkPUOpqupn1A/oB1K5dm/z8/ADLCEZRUVEk64o67beK27ixDoWF/nXLlpvZtKk43IKSiI6zxCXjPgsyoBoBm+Le2wSUGhTAOTcJmASQk5PjFi1aFGAZwcjLyyM3NzfsMpKO9lvFrF8PxxwDkEurVttYtuxfYZeUVHScJS7K+8z2c+kgyEYSBcBBce8dBGwJcBsiKWHkSD86blYWNGu2K+xyRCIpyIBaAWSa2VEl3jsRUAMJkRI++wwmTPD3m448MuxqRKIrsIByzm0FngfuNrOGZnYa0AOYGtQ2RFLBHXfA7t1wxRXQsGHY1YhEV9AP6l4P1Ae+BZ4GBqiJuche//oXPPMM1K3rL/OJyP4F+hyUc+474IIg1ymSKoqL4be/9a8HDYLDDw+1HJHIU1dHIjVk6lR45x049FB/mU9EyqeAEqkBmzfDrbf612PGQONSD1+ISDwFlEgNGDkSvvkGOnWC3r3DrkYkOSigRKrZRx/BuHG+Wfkf/6jujEQqSgElUo2cg5tugsJCuOYaiGC3kyKRpYASqUazZsErr0CTJnDPPWFXI5JcFFAi1WT9erjhBv/6gQegRYtw6xFJNgookWpy002wbh2ceaa/vCciiVFAiVSDl1+GadOgXj2YNEkNI0QqQwElErAtW6B/f/965Eh1CCtSWQookYDdfjusWeNb7A0aFHY1IslLASUSoFdegcceg8xMmDzZ/ysilaOAEgnIt99C377+9e9/DyeeGGo5IklPASUSAOfg17/23RmdccbefvdEpPIUUCIBGD8e5s3zQ7hPnQoZGWFXJJL8FFAiVbRsGQwZ4l8/8QS0bh1uPSKpQgElUgUFBdCrF+zYAVdfDRdfHHZFIqlDASVSSXvuOy1bBsccAw8/HHZFIqlFASVSSWPH+s5gGzeGF1+ERo3CrkgktSigRCrhtdfgttv866eegqOPDrcekVSkgBJJ0OrVcOmlUFwMd9wBF1wQdkUiqUkBJZKAzZvh/PNhwwY45xz/QK6IVA8FlEgF7d7tW+m99x60awfTp+t5J5HqpIASqQDnfA/lr77qBx58+WU4+OCwqxJJbQookQoYORL+/GeoX9/3GHHEEWFXJJL6FFAiBzB5Mtx1F9SqBc88A6ecEnZFIulBASVSjhkz4Npr/etHHvENJESkZiigRPbjuefgyiv9/ad77oHf/CbsikTSiwJKpAxz58Jll0FREQwfDkOHhl2RSPpRQInEmT/fNycvLPS9lOtZJ5FwKKBESnj6ad8zxK5dMHAgjBkDZmFXJZKeFFAiMY8/Dr17+zOnW27xjSIUTiLhUUCJAPfdBwMG+AYRo0fD/fcrnETCFkhAmdlAM1tkZjvNbEoQ6xSpCYWFcMMNcPvtPpDGj9/bS7mIhCszoPV8CYwCzgbqB7ROkWq1aZMfDfevf4U6deAvf/G9lItINAQSUM655wHMLAdoFcQ6RarTypXQrRt88IHvW++FF+C008KuSkRKCuoMKiFm1g/oB5CdnU1eXl4YZZSroKAgknVFXTLst8WLsxg1qj35+XVo02Yro0e/z+7dOwij7Pz8fIqKiiK/z6ImGY6zqEnGfRZKQDnnJgGTAHJyclxubm4YZZQrLy+PKNYVdVHeb8XFcO+9cOedvjHE2WfDzJkNadLkJ6HVlJWVRX5+fmT3WVRF+TiLqmTcZwdsJGFmeWbm9jO9VRNFilTV+vVw3nm+VwjwITV/PjRpEm5dIrJ/BzyDcs7l1kAdItXmb3+Dq66CtWv9GE7Tp/vRcEUk2oJqZp5pZvWADCDDzOqZWSiXD0X22L4dbroJOnf24fTjH8OSJQonkWQR1IO6w4DtwG1An9jrYQGtWyRhixfDySfDuHF+WPYRI+DNN+Hww8OuTEQqKqhm5iOAEUGsS6QqCgp8GI0b53siP+YYmDoVcnLCrkxEEqWujiRlzJ0L7dvDgw/6VnqDBsG//61wEklWuk8kSe+TT/ywGHPm+K87doRJk/wlPhFJXjqDkqS1cSP87nf+rGnOHGjUyF/ae+cdhZNIKtAZlCSd7dth4kQYNQo2bPCdvP7qV/7rli3Drk5EgqKAkqSxcydMngz33ANffunfO+MMeOghf1lPRFKLAkoib9s2mDLFj9H0+ef+vQ4d4O67fYevGrdJJDUpoCSyNmzw4zM98ojvqgjgRz+C3/8eLrwQaukOqkhKU0BJ5CxZ4odfnzbNnz2Bbyp+660+mDIywq1PRGqGAkoiYds2mDXLB9M77+x9/5xz4JZbIDdXl/JE0o0CSkLjnH+QdupUP5ptfr5/PyvLd+7avz8ce2yYFYpImBRQUuNWrICnn4YZM/zrPU45Ba67zg/D3qBBePWJSDQooKRGfPyxf5h25kxYtGjv+4cc4gPpqqv0cK2I7EsBJdWisBAWLvT9482dC8uX753XuDFcdBFcfjn8/OeQqaNQRMqgjwYJzMqVMH/+oUycCP/7v/Ddd3vnZWVB166+Fd5550H9+qGVKSJJQgEllbZmjR9j6fXX/ai1q1YBHP39/KOOgu7d/XTaaVC7dliVikgyUkBJhezc6Z9PevttPy1cCF98se8yTZvCcceto1evFnTpAkcfXfa6REQqQgElpWzbBu+95wNpz/Tee7Br177LNWkCnTr5+0idO8OJJ8Kbby4jNzc3lLpFJLUooNLY9u2+8cKHH/rpgw/8tHw5FBeXXr59ex9Ie6ZjjlF3QyJSfRRQKW7HDn9vaOVK+OwzP330kQ+kVav8w7LxMjLg+OPhpJP81LGjPztq0qSmqxeRdKaASmLFxb4T1S++2Hf6/PO9YRR/n6ikjAzfkOHYY/ed2rdXKzsRCZ8CKmKc813+rFu377R+PXz77b5B9OWXsHt3+evLyIDDD4cjjtg77QmlI4+EOnVq5McSEUmYAqoaFBfDli0+aPLzYdOmva/LmjZu9ENL7AmiwsKKb6tpUzjssH2n1q33hlHr1noQVkSSU9p8dDnnW6Ht2OGbTO/YsXcq6+slS7L55BP/9datUFBQ8X+3b69arY0bQ4sWZU8tW+4NopYt1WediKSu0APqq69g+HB/1rB79/7/LW/e/pbZE0h7QicxVetGu2FDf3aTlXXgqUkTaN7cTy1aQN26Vdq0iEhKMFdWM66aLMAaO4jvJfQS4HpgG9C1jO/qG5vWAxeXMX8A0AtYA1xRYlu+WXTDhoNp0qQ7tWotZ926/tSqxT5T+/bDqFv3eBo1+op33x1ERoZ/PyPDT5deei8nnXQqq1cv5Kmnhpaa//DD4+jYsQOvvfYao0aNKlXdxIkTOfroo5k7dy4PPvhgqflTp06ldevWzJw5kwkTJpSa/9xzz9G8eXOmTJnClClTSs1fsGABDRo0YPz48cyaNavU/Ly8PADGjh3LvHnz9plXv359Xn75ZQBGjhzJ66+/vs/8Zs2aMXv2bABuv/123n777X3m165dm1dffRWAQYMGsXTp0n3mt2vXjkmTJgHQr18/VpTszhzo0KED48aNA6BPnz6sXbt2n/mdOnVi9OjRAPTs2ZMNGzbsM79z584MHz4cgHPPPZftcaez3bp1Y8iQIQBlPq91ySWXcP3117Nt2za6di197PXt25e+ffuyfv16Lr649LE3YMAAevXqxZo1a7jiiitKzR88eDDdu3dn+fLl9O/fn6VLl1JYWEhOTg4Aw4YNo0uXLixdupRBgwaV+v57772XU089lYULFzJ06NBS88eNG0eHDql/7PXu3Zsv4loAtWrVimnTpgE69so69s466yyGDh36/bEXL8xj74033ljsnMuJ/57Qz6Dq1PGXqsz2Th07+gc/i4v9cN8l55nBWWf5/twKCmDEiNLz+/SBCy7w93RuvHFv8OwxeLDvfmf5cj/mULxhwyAz80OysrIo4/fEOefAqaf63hRefLH0fD0bJCJSdaGfQeXk5LhFJcdfiIi8vDz1iFAJ2m+Jyc3NJT8/v9Rf+1I+HWeJi/I+M7Myz6D0t76IiESSAkpERCJJASUiIpGkgBIRkUhSQImISCRVOaDMrK6ZTTaz1Wa2xcyWmNm5QRQnIiLpK4gzqEz8E7FnAE2A4cAsM2sbwLpFRCRNVflBXefcVmBEibfmmdlKfPcQq6q6fhERSU+B9yRhZtlAO2BZOcv0A/oBZGdnf9/9SZQUFBREsq6o035LTH5+PkVFRdpnCdJxlrhk3GeB9iRhZrWBl4FPnXNldCJUmnqSSC3ab4lRTxKVo+MscVHeZ5XuScLM8szM7Wd6q8RytYCpwC5gYKDVi4hI2jngJT7nXO6BljEzAyYD2UBX59wBxnkVEREpX1D3oCbgB1Dq4pyr4nB9IiIiwTwH1QboD3QAvjazgtjUu6rrFhGR9BVEM/PVgAVQi4iIyPfU1ZGIiESSAkpERCIp9BF1zWwdsDrUIsrWHFgfdhFJSPstcdpnidM+S1yU91kb51yL+DdDD6ioMrNFZT04JuXTfkuc9lnitM8Sl4z7TJf4REQkkhRQIiISSQqo/ZsUdgFJSvstcdpnidM+S1zS7TPdgxIRkUjSGZSIiESSAkpERCJJASUiIpGkgKogMzvKzHaY2bSwa4kyM6trZpPNbLWZbTGzJWZ2bth1RZGZHWxmL5jZ1tj+ujzsmqJMx1bVJONnmAKq4h4D3g27iCSQCawBzgCaAMOBWWbWNsyiIuox/ACf2UBvYIKZ/SjckiJNx1bVJN1nmAKqAszsUiAfeD3kUiLPObfVOTfCObfKOVfsnJsHrARODru2KDGzhkBPYLhzrsA59xbwEnBFuJVFl46tykvWzzAF1AGY2UHA3cDgsGtJRmaWDbQDloVdS8S0A4qccytKvPcfQGdQFaRjq2KS+TNMAXVgI4HJzrk1YReSbMysNjAd+Itz7qOw64mYRsCmuPc2AY1DqCXp6NhKSNJ+hqV1QJlZnpm5/UxvmVkHoAvwh5BLjYwD7bMSy9UCpuLvsQwMreDoKgAOinvvIGBLCLUkFR1bFZfsn2FVHlE3mTnncsubb2aDgLbA52YG/q/eDDNr75zrWN31RdGB9hmA+Z01GX/zv6tzbnd115WEVgCZZnaUc+7j2HsnostV5dKxlbBckvgzTF0dlcPMGrDvX7lD8L/sAc65daEUlQTM7HGgA9DFOVcQcjmRZWbPAA64Br+/FgCnOucUUvuhYysxyf4ZltZnUAfinNsGbNvztZkVADuS4RcbFjNrA/QHdgJfx/5qA+jvnJseWmHRdD3wJPAtsAH/oaFw2g8dW4lL9s8wnUGJiEgkpXUjCRERiS4FlIiIRJICSkREIkkBJSIikaSAEhGRSFJAiYhIJCmgREQkkhRQIiISSf8fcCYlA/nY5hsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, selu(z), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [-1.758, -1.758], 'k--')\n",
    "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
    "plt.grid(True)\n",
    "plt.title(\"SELU activation function\", fontsize=14)\n",
    "plt.axis([-5, 5, -2.2, 3.2])\n",
    "\n",
    "save_fig(\"selu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the SELU hyperparameters (`scale` and `alpha`) are tuned in such a way that the mean output of each neuron remains close to 0, and the standard deviation remains close to 1 (assuming the inputs are standardized with mean 0 and standard deviation 1 too). Using this activation function, even a 1,000 layer deep neural network preserves roughly mean 0 and standard deviation 1 across all layers, avoiding the exploding/vanishing gradients problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer 0: mean -0.00, std deviation 1.00\n",
      "Layer 100: mean 0.02, std deviation 0.96\n",
      "Layer 200: mean 0.01, std deviation 0.90\n",
      "Layer 300: mean -0.02, std deviation 0.92\n",
      "Layer 400: mean 0.05, std deviation 0.89\n",
      "Layer 500: mean 0.01, std deviation 0.93\n",
      "Layer 600: mean 0.02, std deviation 0.92\n",
      "Layer 700: mean -0.02, std deviation 0.90\n",
      "Layer 800: mean 0.05, std deviation 0.83\n",
      "Layer 900: mean 0.02, std deviation 1.00\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "Z = np.random.normal(size=(500, 100)) # standardized inputs\n",
    "for layer in range(1000):\n",
    "    W = np.random.normal(size=(100, 100), scale=np.sqrt(1 / 100)) # LeCun initialization\n",
    "    Z = selu(np.dot(Z, W))\n",
    "    means = np.mean(Z, axis=0).mean()\n",
    "    stds = np.std(Z, axis=0).mean()\n",
    "    if layer % 100 == 0:\n",
    "        print(\"Layer {}: mean {:.2f}, std deviation {:.2f}\".format(layer, means, stds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using SELU is easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7ff190832b10>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"selu\",\n",
    "                   kernel_initializer=\"lecun_normal\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a neural net for Fashion MNIST with 100 hidden layers, using the SELU activation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"selu\",\n",
    "                             kernel_initializer=\"lecun_normal\"))\n",
    "for layer in range(99):\n",
    "    model.add(keras.layers.Dense(100, activation=\"selu\",\n",
    "                                 kernel_initializer=\"lecun_normal\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train it. Do not forget to scale the inputs to mean 0 and standard deviation 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "pixel_means = X_train.mean(axis=0, keepdims=True)\n",
    "pixel_stds = X_train.std(axis=0, keepdims=True)\n",
    "X_train_scaled = (X_train - pixel_means) / pixel_stds\n",
    "X_valid_scaled = (X_valid - pixel_means) / pixel_stds\n",
    "X_test_scaled = (X_test - pixel_means) / pixel_stds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1719/1719 [==============================] - 12s 6ms/step - loss: 1.3556 - accuracy: 0.4808 - val_loss: 0.7711 - val_accuracy: 0.6858\n",
      "Epoch 2/5\n",
      "1719/1719 [==============================] - 9s 5ms/step - loss: 0.7537 - accuracy: 0.7235 - val_loss: 0.7534 - val_accuracy: 0.7384\n",
      "Epoch 3/5\n",
      "1719/1719 [==============================] - 9s 5ms/step - loss: 0.7451 - accuracy: 0.7357 - val_loss: 0.5943 - val_accuracy: 0.7834\n",
      "Epoch 4/5\n",
      "1719/1719 [==============================] - 9s 5ms/step - loss: 0.5699 - accuracy: 0.7906 - val_loss: 0.5434 - val_accuracy: 0.8066\n",
      "Epoch 5/5\n",
      "1719/1719 [==============================] - 9s 5ms/step - loss: 0.5569 - accuracy: 0.8051 - val_loss: 0.4907 - val_accuracy: 0.8218\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now look at what happens if we try to use the ReLU activation function instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
    "for layer in range(99):\n",
    "    model.add(keras.layers.Dense(100, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1719/1719 [==============================] - 11s 5ms/step - loss: 2.0460 - accuracy: 0.1919 - val_loss: 1.5971 - val_accuracy: 0.3048\n",
      "Epoch 2/5\n",
      "1719/1719 [==============================] - 8s 5ms/step - loss: 1.2654 - accuracy: 0.4591 - val_loss: 0.9156 - val_accuracy: 0.6372\n",
      "Epoch 3/5\n",
      "1719/1719 [==============================] - 8s 5ms/step - loss: 0.9312 - accuracy: 0.6169 - val_loss: 0.8928 - val_accuracy: 0.6246\n",
      "Epoch 4/5\n",
      "1719/1719 [==============================] - 8s 5ms/step - loss: 0.8188 - accuracy: 0.6710 - val_loss: 0.6914 - val_accuracy: 0.7396\n",
      "Epoch 5/5\n",
      "1719/1719 [==============================] - 8s 5ms/step - loss: 0.7288 - accuracy: 0.7152 - val_loss: 0.6638 - val_accuracy: 0.7380\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not great at all, we suffered from the vanishing/exploding gradients problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_4 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization (BatchNo (None, 784)               3136      \n",
      "_________________________________________________________________\n",
      "dense_212 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 300)               1200      \n",
      "_________________________________________________________________\n",
      "dense_213 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 100)               400       \n",
      "_________________________________________________________________\n",
      "dense_214 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 271,346\n",
      "Trainable params: 268,978\n",
      "Non-trainable params: 2,368\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('batch_normalization/gamma:0', True),\n",
       " ('batch_normalization/beta:0', True),\n",
       " ('batch_normalization/moving_mean:0', False),\n",
       " ('batch_normalization/moving_variance:0', False)]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn1 = model.layers[1]\n",
    "[(var.name, var.trainable) for var in bn1.variables]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#bn1.updates #deprecated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 1.2287 - accuracy: 0.5993 - val_loss: 0.5526 - val_accuracy: 0.8230\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5996 - accuracy: 0.7959 - val_loss: 0.4725 - val_accuracy: 0.8468\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5312 - accuracy: 0.8168 - val_loss: 0.4375 - val_accuracy: 0.8558\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4884 - accuracy: 0.8294 - val_loss: 0.4153 - val_accuracy: 0.8596\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4717 - accuracy: 0.8343 - val_loss: 0.3997 - val_accuracy: 0.8640\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4420 - accuracy: 0.8461 - val_loss: 0.3867 - val_accuracy: 0.8694\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4285 - accuracy: 0.8496 - val_loss: 0.3763 - val_accuracy: 0.8710\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4086 - accuracy: 0.8552 - val_loss: 0.3711 - val_accuracy: 0.8740\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4079 - accuracy: 0.8566 - val_loss: 0.3631 - val_accuracy: 0.8752\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3903 - accuracy: 0.8617 - val_loss: 0.3573 - val_accuracy: 0.8750\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes applying BN before the activation function works better (there's a debate on this topic). Moreover, the layer before a `BatchNormalization` layer does not need to have bias terms, since the `BatchNormalization` layer some as well, it would be a waste of parameters, so you can set `use_bias=False` when creating those layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, use_bias=False),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.Dense(100, use_bias=False),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 1.3677 - accuracy: 0.5604 - val_loss: 0.6767 - val_accuracy: 0.7812\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.7136 - accuracy: 0.7702 - val_loss: 0.5566 - val_accuracy: 0.8184\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.6123 - accuracy: 0.7990 - val_loss: 0.5007 - val_accuracy: 0.8360\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5547 - accuracy: 0.8148 - val_loss: 0.4666 - val_accuracy: 0.8448\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5255 - accuracy: 0.8230 - val_loss: 0.4434 - val_accuracy: 0.8534\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4947 - accuracy: 0.8328 - val_loss: 0.4263 - val_accuracy: 0.8550\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4736 - accuracy: 0.8385 - val_loss: 0.4130 - val_accuracy: 0.8566\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4550 - accuracy: 0.8446 - val_loss: 0.4035 - val_accuracy: 0.8612\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4495 - accuracy: 0.8440 - val_loss: 0.3943 - val_accuracy: 0.8638\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4333 - accuracy: 0.8494 - val_loss: 0.3875 - val_accuracy: 0.8660\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Clipping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All Keras optimizers accept `clipnorm` or `clipvalue` arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(clipvalue=1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(clipnorm=1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reusing Pretrained Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reusing a Keras model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split the fashion MNIST training set in two:\n",
    "* `X_train_A`: all images of all items except for sandals and shirts (classes 5 and 6).\n",
    "* `X_train_B`: a much smaller training set of just the first 200 images of sandals or shirts.\n",
    "\n",
    "The validation set and the test set are also split this way, but without restricting the number of images.\n",
    "\n",
    "We will train a model on set A (classification task with 8 classes), and try to reuse it to tackle set B (binary classification). We hope to transfer a little bit of knowledge from task A to task B, since classes in set A (sneakers, ankle boots, coats, t-shirts, etc.) are somewhat similar to classes in set B (sandals and shirts). However, since we are using `Dense` layers, only patterns that occur at the same location can be reused (in contrast, convolutional layers will transfer much better, since learned patterns can be detected anywhere on the image, as we will see in the CNN chapter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_dataset(X, y):\n",
    "    y_5_or_6 = (y == 5) | (y == 6) # sandals or shirts\n",
    "    y_A = y[~y_5_or_6]\n",
    "    y_A[y_A > 6] -= 2 # class indices 7, 8, 9 should be moved to 5, 6, 7\n",
    "    y_B = (y[y_5_or_6] == 6).astype(np.float32) # binary classification task: is it a shirt (class 6)?\n",
    "    return ((X[~y_5_or_6], y_A),\n",
    "            (X[y_5_or_6], y_B))\n",
    "\n",
    "(X_train_A, y_train_A), (X_train_B, y_train_B) = split_dataset(X_train, y_train)\n",
    "(X_valid_A, y_valid_A), (X_valid_B, y_valid_B) = split_dataset(X_valid, y_valid)\n",
    "(X_test_A, y_test_A), (X_test_B, y_test_B) = split_dataset(X_test, y_test)\n",
    "X_train_B = X_train_B[:200]\n",
    "y_train_B = y_train_B[:200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(43986, 28, 28)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_A.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200, 28, 28)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_B.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 0, 5, 7, 7, 7, 4, 4, 3, 4, 0, 1, 6, 3, 4, 3, 2, 6, 5, 3, 4, 5,\n",
       "       1, 3, 4, 2, 0, 6, 7, 1], dtype=uint8)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_A[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 0., 0., 0., 0., 1., 1., 1., 0., 0., 1., 1., 0., 0., 0., 0.,\n",
       "       0., 0., 1., 1., 0., 0., 1., 1., 0., 1., 1., 1., 1.], dtype=float32)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_B[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A = keras.models.Sequential()\n",
    "model_A.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_A.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
    "model_A.add(keras.layers.Dense(8, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "                optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.9249 - accuracy: 0.6994 - val_loss: 0.3896 - val_accuracy: 0.8662\n",
      "Epoch 2/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.3651 - accuracy: 0.8745 - val_loss: 0.3288 - val_accuracy: 0.8827\n",
      "Epoch 3/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.3182 - accuracy: 0.8897 - val_loss: 0.3013 - val_accuracy: 0.8991\n",
      "Epoch 4/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.3048 - accuracy: 0.8954 - val_loss: 0.2896 - val_accuracy: 0.9021\n",
      "Epoch 5/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2804 - accuracy: 0.9029 - val_loss: 0.2773 - val_accuracy: 0.9061\n",
      "Epoch 6/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2701 - accuracy: 0.9075 - val_loss: 0.2735 - val_accuracy: 0.9066\n",
      "Epoch 7/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2627 - accuracy: 0.9093 - val_loss: 0.2721 - val_accuracy: 0.9081\n",
      "Epoch 8/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2609 - accuracy: 0.9122 - val_loss: 0.2589 - val_accuracy: 0.9141\n",
      "Epoch 9/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2558 - accuracy: 0.9110 - val_loss: 0.2562 - val_accuracy: 0.9136\n",
      "Epoch 10/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2512 - accuracy: 0.9138 - val_loss: 0.2544 - val_accuracy: 0.9160\n",
      "Epoch 11/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2431 - accuracy: 0.9170 - val_loss: 0.2495 - val_accuracy: 0.9153\n",
      "Epoch 12/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2422 - accuracy: 0.9168 - val_loss: 0.2515 - val_accuracy: 0.9126\n",
      "Epoch 13/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2360 - accuracy: 0.9181 - val_loss: 0.2446 - val_accuracy: 0.9160\n",
      "Epoch 14/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2266 - accuracy: 0.9232 - val_loss: 0.2415 - val_accuracy: 0.9178\n",
      "Epoch 15/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2225 - accuracy: 0.9239 - val_loss: 0.2447 - val_accuracy: 0.9195\n",
      "Epoch 16/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2261 - accuracy: 0.9216 - val_loss: 0.2384 - val_accuracy: 0.9198\n",
      "Epoch 17/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2191 - accuracy: 0.9251 - val_loss: 0.2412 - val_accuracy: 0.9175\n",
      "Epoch 18/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2171 - accuracy: 0.9254 - val_loss: 0.2429 - val_accuracy: 0.9158\n",
      "Epoch 19/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2180 - accuracy: 0.9252 - val_loss: 0.2330 - val_accuracy: 0.9205\n",
      "Epoch 20/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2112 - accuracy: 0.9274 - val_loss: 0.2333 - val_accuracy: 0.9200\n"
     ]
    }
   ],
   "source": [
    "history = model_A.fit(X_train_A, y_train_A, epochs=20,\n",
    "                    validation_data=(X_valid_A, y_valid_A))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A.save(\"my_model_A.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_B = keras.models.Sequential()\n",
    "model_B.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_B.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
    "model_B.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_B.compile(loss=\"binary_crossentropy\",\n",
    "                optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "7/7 [==============================] - 0s 30ms/step - loss: 1.0360 - accuracy: 0.4975 - val_loss: 0.6314 - val_accuracy: 0.6004\n",
      "Epoch 2/20\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.5883 - accuracy: 0.6971 - val_loss: 0.4784 - val_accuracy: 0.8529\n",
      "Epoch 3/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.4380 - accuracy: 0.8854 - val_loss: 0.4102 - val_accuracy: 0.8945\n",
      "Epoch 4/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.4021 - accuracy: 0.8712 - val_loss: 0.3647 - val_accuracy: 0.9178\n",
      "Epoch 5/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.3361 - accuracy: 0.9348 - val_loss: 0.3300 - val_accuracy: 0.9320\n",
      "Epoch 6/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.3113 - accuracy: 0.9233 - val_loss: 0.3019 - val_accuracy: 0.9402\n",
      "Epoch 7/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.2817 - accuracy: 0.9299 - val_loss: 0.2804 - val_accuracy: 0.9422\n",
      "Epoch 8/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.2632 - accuracy: 0.9379 - val_loss: 0.2606 - val_accuracy: 0.9473\n",
      "Epoch 9/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.2373 - accuracy: 0.9481 - val_loss: 0.2428 - val_accuracy: 0.9523\n",
      "Epoch 10/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.2229 - accuracy: 0.9657 - val_loss: 0.2281 - val_accuracy: 0.9544\n",
      "Epoch 11/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.2155 - accuracy: 0.9590 - val_loss: 0.2150 - val_accuracy: 0.9584\n",
      "Epoch 12/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.1834 - accuracy: 0.9738 - val_loss: 0.2036 - val_accuracy: 0.9584\n",
      "Epoch 13/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1671 - accuracy: 0.9828 - val_loss: 0.1931 - val_accuracy: 0.9615\n",
      "Epoch 14/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.1527 - accuracy: 0.9915 - val_loss: 0.1838 - val_accuracy: 0.9635\n",
      "Epoch 15/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1595 - accuracy: 0.9904 - val_loss: 0.1746 - val_accuracy: 0.9686\n",
      "Epoch 16/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1473 - accuracy: 0.9937 - val_loss: 0.1674 - val_accuracy: 0.9686\n",
      "Epoch 17/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1412 - accuracy: 0.9944 - val_loss: 0.1604 - val_accuracy: 0.9706\n",
      "Epoch 18/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1242 - accuracy: 0.9931 - val_loss: 0.1539 - val_accuracy: 0.9706\n",
      "Epoch 19/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1224 - accuracy: 0.9931 - val_loss: 0.1482 - val_accuracy: 0.9716\n",
      "Epoch 20/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1096 - accuracy: 0.9912 - val_loss: 0.1431 - val_accuracy: 0.9716\n"
     ]
    }
   ],
   "source": [
    "history = model_B.fit(X_train_B, y_train_B, epochs=20,\n",
    "                      validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_7\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_7 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_224 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "dense_225 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "dense_226 (Dense)            (None, 50)                5050      \n",
      "_________________________________________________________________\n",
      "dense_227 (Dense)            (None, 50)                2550      \n",
      "_________________________________________________________________\n",
      "dense_228 (Dense)            (None, 50)                2550      \n",
      "_________________________________________________________________\n",
      "dense_229 (Dense)            (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 275,801\n",
      "Trainable params: 275,801\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model_B.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A = keras.models.load_model(\"my_model_A.h5\")\n",
    "model_B_on_A = keras.models.Sequential(model_A.layers[:-1])\n",
    "model_B_on_A.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A_clone = keras.models.clone_model(model_A)\n",
    "model_A_clone.set_weights(model_A.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = False\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\",\n",
    "                     optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                     metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "7/7 [==============================] - 1s 83ms/step - loss: 0.6155 - accuracy: 0.6184 - val_loss: 0.5843 - val_accuracy: 0.6329\n",
      "Epoch 2/4\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.5550 - accuracy: 0.6638 - val_loss: 0.5467 - val_accuracy: 0.6805\n",
      "Epoch 3/4\n",
      "7/7 [==============================] - 0s 8ms/step - loss: 0.4897 - accuracy: 0.7482 - val_loss: 0.5146 - val_accuracy: 0.7089\n",
      "Epoch 4/4\n",
      "7/7 [==============================] - 0s 8ms/step - loss: 0.4899 - accuracy: 0.7405 - val_loss: 0.4859 - val_accuracy: 0.7323\n",
      "Epoch 1/16\n",
      "7/7 [==============================] - 0s 28ms/step - loss: 0.4380 - accuracy: 0.7774 - val_loss: 0.3460 - val_accuracy: 0.8661\n",
      "Epoch 2/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.2971 - accuracy: 0.9143 - val_loss: 0.2603 - val_accuracy: 0.9310\n",
      "Epoch 3/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.2034 - accuracy: 0.9777 - val_loss: 0.2110 - val_accuracy: 0.9554\n",
      "Epoch 4/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.1754 - accuracy: 0.9719 - val_loss: 0.1790 - val_accuracy: 0.9696\n",
      "Epoch 5/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.1348 - accuracy: 0.9809 - val_loss: 0.1561 - val_accuracy: 0.9757\n",
      "Epoch 6/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.1172 - accuracy: 0.9973 - val_loss: 0.1392 - val_accuracy: 0.9797\n",
      "Epoch 7/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.1137 - accuracy: 0.9931 - val_loss: 0.1266 - val_accuracy: 0.9838\n",
      "Epoch 8/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.1000 - accuracy: 0.9931 - val_loss: 0.1163 - val_accuracy: 0.9858\n",
      "Epoch 9/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.0834 - accuracy: 1.0000 - val_loss: 0.1065 - val_accuracy: 0.9888\n",
      "Epoch 10/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.0775 - accuracy: 1.0000 - val_loss: 0.0999 - val_accuracy: 0.9899\n",
      "Epoch 11/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.0689 - accuracy: 1.0000 - val_loss: 0.0939 - val_accuracy: 0.9899\n",
      "Epoch 12/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.0719 - accuracy: 1.0000 - val_loss: 0.0888 - val_accuracy: 0.9899\n",
      "Epoch 13/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.0565 - accuracy: 1.0000 - val_loss: 0.0839 - val_accuracy: 0.9899\n",
      "Epoch 14/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.0494 - accuracy: 1.0000 - val_loss: 0.0802 - val_accuracy: 0.9899\n",
      "Epoch 15/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.0544 - accuracy: 1.0000 - val_loss: 0.0768 - val_accuracy: 0.9899\n",
      "Epoch 16/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.0472 - accuracy: 1.0000 - val_loss: 0.0738 - val_accuracy: 0.9899\n"
     ]
    }
   ],
   "source": [
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,\n",
    "                           validation_data=(X_valid_B, y_valid_B))\n",
    "\n",
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = True\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\",\n",
    "                     optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                     metrics=[\"accuracy\"])\n",
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,\n",
    "                           validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, what's the final verdict?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63/63 [==============================] - 0s 723us/step - loss: 0.1408 - accuracy: 0.9705\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.1408407837152481, 0.9704999923706055]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63/63 [==============================] - 0s 705us/step - loss: 0.0682 - accuracy: 0.9935\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.0681881457567215, 0.9934999942779541]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B_on_A.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We got quite a bit of transfer: the error rate dropped by a factor of 4.5!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.538461538461503"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(100 - 97.05) / (100 - 99.35)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Faster Optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Momentum optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nesterov Accelerated Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AdaGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adagrad(lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RMSProp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.RMSprop(lr=0.001, rho=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adam Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adamax Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adamax(lr=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nadam Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Nadam(lr=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Rate Scheduling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Power Scheduling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```lr = lr0 / (1 + steps / s)**c```\n",
    "* Keras uses `c=1` and `s = 1 / decay`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.01, decay=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5980 - accuracy: 0.7933 - val_loss: 0.4031 - val_accuracy: 0.8598\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 2s 954us/step - loss: 0.3829 - accuracy: 0.8636 - val_loss: 0.3714 - val_accuracy: 0.8720\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 2s 943us/step - loss: 0.3491 - accuracy: 0.8771 - val_loss: 0.3746 - val_accuracy: 0.8738\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 2s 954us/step - loss: 0.3277 - accuracy: 0.8814 - val_loss: 0.3502 - val_accuracy: 0.8798\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 2s 934us/step - loss: 0.3172 - accuracy: 0.8856 - val_loss: 0.3453 - val_accuracy: 0.8780\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 2s 919us/step - loss: 0.2922 - accuracy: 0.8940 - val_loss: 0.3419 - val_accuracy: 0.8820\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 2s 921us/step - loss: 0.2870 - accuracy: 0.8973 - val_loss: 0.3362 - val_accuracy: 0.8872\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 2s 925us/step - loss: 0.2720 - accuracy: 0.9032 - val_loss: 0.3415 - val_accuracy: 0.8830\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 2s 929us/step - loss: 0.2730 - accuracy: 0.9004 - val_loss: 0.3297 - val_accuracy: 0.8864\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 2s 928us/step - loss: 0.2585 - accuracy: 0.9068 - val_loss: 0.3269 - val_accuracy: 0.8888\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 2s 932us/step - loss: 0.2529 - accuracy: 0.9100 - val_loss: 0.3280 - val_accuracy: 0.8878\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 2s 954us/step - loss: 0.2485 - accuracy: 0.9101 - val_loss: 0.3343 - val_accuracy: 0.8822\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 2s 964us/step - loss: 0.2420 - accuracy: 0.9148 - val_loss: 0.3266 - val_accuracy: 0.8890\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2373 - accuracy: 0.9144 - val_loss: 0.3299 - val_accuracy: 0.8890\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2363 - accuracy: 0.9154 - val_loss: 0.3255 - val_accuracy: 0.8874\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2309 - accuracy: 0.9181 - val_loss: 0.3217 - val_accuracy: 0.8910\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2235 - accuracy: 0.9211 - val_loss: 0.3248 - val_accuracy: 0.8914\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2247 - accuracy: 0.9194 - val_loss: 0.3202 - val_accuracy: 0.8934\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2235 - accuracy: 0.9218 - val_loss: 0.3243 - val_accuracy: 0.8906\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2227 - accuracy: 0.9225 - val_loss: 0.3224 - val_accuracy: 0.8900\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2193 - accuracy: 0.9230 - val_loss: 0.3221 - val_accuracy: 0.8912\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2163 - accuracy: 0.9227 - val_loss: 0.3195 - val_accuracy: 0.8948\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 2s 997us/step - loss: 0.2127 - accuracy: 0.9252 - val_loss: 0.3208 - val_accuracy: 0.8908\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2076 - accuracy: 0.9273 - val_loss: 0.3226 - val_accuracy: 0.8902\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 2s 999us/step - loss: 0.2104 - accuracy: 0.9250 - val_loss: 0.3225 - val_accuracy: 0.8924\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAzyklEQVR4nO3deXxU1f3/8dcnCyEkbGEnEkBEZFFEq6JopWKLrbbyVbuqdS2trb+21qpYta6trVq/XbQLX6UuRVutoFZUXBAVrAtVEdkRBWRfA4EEQvj8/rg3dJhMkhvIzCSZ9/PxmAcz954593OvMZ+ce849x9wdERGRxpaV7gBERKRlUoIREZGkUIIREZGkUIIREZGkUIIREZGkUIIREZGkUIIRaUbM7EIzK0tS3R+a2U0N/M4nZvbT2j5LZlOCkWbHzB4wMw9flWa21MzuMrOCdMdWHzPra2Z/M7NPzWynma0ysylmNizdsTWSY4A/pjsIaRpy0h2AyH56CTgfyAVOAu4DCoDL0hlUNTPLdffK+G3Ai8BHwNeAlUAx8HmgKOVBJoG7r093DNJ0qAUjzdVOd1/j7ivc/RFgIjAGwMzyzOy3ZrbWzCrM7E0zO7H6i2b2lpldE/N5Ytga6h5+bmNmu8xsRPjZzOxqM/vIzMrNbI6ZnRfz/T7h979pZtPMrBz4boKYBwP9gB+4+xvuviz892Z3fzmmvnZm9iczWx3GP9/Mvh5bkZmNCm9pbTezV8ysb9z+L5vZf8Lvf2xmvzCzVjH7u5rZU+H5LDOzi+ODDc/pnLhtdd4CS3DLzM1srJk9Hsa6NPbahWWOM7N3w1jfM7Mvhd8bWdtxpHlQgpGWopygNQNwB/B14GJgGDAHeN7MeoT7pwOfi/nuycAGYGT4eQRQCbwdfr4NuAT4ATAIuB34i5mdHhfD7QS3hwYBTyaIcT2wBzjbzBLePTAzA54LY7oorOsnwK6YYnnAteH5HQ90AP4cU8dogoR7D0FSuxg4B/hlTB0PAIcApxIk5m8DfRLF1Ah+DjwFDAX+AUwws95hrIXAM8AC4GjgauDOJMUhqebueunVrF4Evxyfifl8LEGC+AfBbbJdwLdj9mcT3Ja6Lfz8RaCM4BZxf2Ab8AvgL+H+XwAvhu8LCJLXSXEx/BZ4NnzfB3Dgygix/wDYHh7/VeBWYHDM/s8TJKGBtXz/wvBYA2K2nRuec1b4+TXghrjvjQmPacChYR0jYvb3BqqAm2K2OXBOXD2fAD9twGcHbo/5nAPsAM4LP38X2ATkx5T5Vvi9ken+WdPrwF5qwUhzdZqZlZlZBfBvgl+q/4/gFlQuMLO6oLtXhWUGhZteJ2gFHEPQanmdoE9nZLh/JEErh/A7rQlaQGXVL4K+nn5xMc2qL2h3vxfoTvBLdAZwJvC+mZ0fFhkGrHb3+XVUs9PdF8Z8XhWec4fw89HAdXHxPkKQLLsDAwmSWHULDXdfFtaTDB/EHGc3QUuua7jpMOBDdy+PKf9WkuKQFFMnvzRXrwFjCW5lrfKwQz3mNliiacKDP6ndy8zsXYLbZIOBVwgSUG8z60+QeK4Ov1P9R9iXgeVx9VXGfd4eJXB33wY8DTxtZtcDUwlaMg8TtDDqszu+yrhYs4CbgccTfHd9xGNU1xtfNjdRwXrEXyfnv7Eaif9bSQugBCPN1Q53X5Jg+xKC20UnAksBzCyboK/ikZhy0wkSzEDgt+5eYWZvAdexb//LPGAn0NvdpzX2Sbi7m9kC4Khw07tADzMbWE8rpi7vAofVcn0ws/kEv+CPAd4It5UAPeOKrgd6xHyvW+znRjIf+LaZ5ce0Yo5t5GNImijBSIvi7tvN7E/Ar8xsA/AxcAXQjX2fz5gOXEnQ6ng3Ztt1wCvVLSJ332ZmdwF3hR3wrwGFwHBgj7uPjxqbmR1J0LJ4mCBx7SLozL8YeDQs9jLBLaInzOwKYBFBZ3yBuz8Z8VC3AM+Y2TLgMYIWzxDgWHe/2t0XmtnzBAMVxhL0Md0d/htrGvADM3uDoH/ml0BF1PONaCLBIIr/M7NfEiS5n4X71LJp5tQHIy3RNQS/WP8KvA8cAZzm7qtjyrxO8Avs9bCPBoJbZdn8t/+l2g3ATcBPgbkEz7KcTZC8GuJTglbVz4E3w9iuBO4i6D/C3fcQDEKYCfyN4C/83wGtalaXmLtPBU4naKG9Hb7Gse8tvgvD+KcB/yJo3X0SV9WVYbzTgX8SPGu0LmocEWMtI7j9OBh4j2AE2U3h7sZOZpJi5q4/EkSk6TCzM4HJQFd335DueGT/6RaZiKSVmV1A0FJaQXAr77fAv5Rcmr+U3iIzsyIzmxw+0bvMzL5VR9krzGyNmZWa2QQzy4vZd7mZzbJgLqcHEnx3lJktMLMd4VPOvZN0SiJy4LoR9EstBO4leND0vDq/Ic1CSm+RmdmjBEntEuBIYApwgrvPjSs3GngIOIVgbP5k4E13HxfuP4tgHP9ogge0Loz5bmeCh+ouJbi3fCvBQ3LDk3luIiKyr5QlGAtmut0MDHH3ReG2h4GV1YkjpuwjwCfu/rPw8yhgort3jyt3G3BQXIIZC1zo7ifEHHcDMMzdFyTr/EREZF+p7IM5FKiqTi6h2QTDNOMNJpi7KLZcNzPr5O4b6znO4LA8sHfY6kfh9n0STJiMxgJk5bc7Oqd91737+rTTADuAPXv2kJWlaxFP1yUxXZeaWvo1WbRo0QZ375JoXyoTTCFQGretFGgboWz1+7ZAfQmmkOABsXqPEz7DMB4gr0d/73HBbwEo7pDPzHGn1HOYzDB9+nRGjhyZ7jCaHF2XxHRdamrp1yR83iqhVKbVMqBd3LZ2BBMN1le2+n2isgdynBrycrK4avSAKEVFRKQOqUwwi4CccK6nakMJHlyLNzfcF1tubYTbYzW+G/bB9KvlOPswYEjPdowZVhzhMCIiUpeUJRh33w5MAm4xswILFnM6k2B4YryHgEvMbJCZdQSuJ5iiHQAzyzGz1gRPXWebWeuY9TUmA0PM7OywzM+BD+rr4O/TLouLRvRl9qelrN2qB4hFRA5Uqnuevg/kE0w38ShwmbvPNbOScFrxEgB3f55g0ahXgGXh68aYeq4nmDdpHMF4+fJwGx4s2Xo2wZoem4HjgG9ECe7CE/pQ5c7D/671lqKIiESU0if53X0T4bK2cduXE3TOx267m2ACvkT13MR/5ytKtP8lgnUmGqSkUxs+P7AbE99axuWnHELr3OyGViEiIqGWO3ZuP118Yl8276jkyfdWpjsUEZFmTQkmznF9ixjUox0TZn6MJgIVEdl/SjBxzIyLT+zLorVlzFwSZdCaiIgkogSTwJeH9qBzYSsmzGzoch8iIlJNCSaBvJxszhvem2kL1rF0fVm6wxERaZaUYGpx7nG9aZWdxQNvfJLuUEREmiUlmFp0aZvHV47syeOzPqV0R2W6wxERaXaUYOpw0Yg+lFdW8Y9Zy+svLCIi+1CCqcPgnu0ZfnARD76xjN1Ve9IdjohIs6IEU4+LR/Rl5ZZyXpi3Nt2hiIg0K0ow9Rg1sBslRW2YMENDlkVEGkIJph7ZWcaFJ/Rh1rLNzF6xJd3hiIg0G0owEXztmF60zcvhr3rwUkQkMiWYCArzcvjaMb145oPVWitGRCQiJZiILjyhD3u0VoyISGRKMBH1KmrD5wcFa8VUVFalOxwRkSZPCaYBLh4RrBUzWWvFiIjUSwmmAY7tW8Tgnu2YMENrxYiI1EcJpgHMjItH9GXxujJmLNmQ7nBERJo0JZgGOmNoDzoX5unBSxGReijBNFBeTjbnD+/NKwvX85HWihERqZUSzH44d3hJsFbMzE/SHYqISJOVk+4AmqPOhXkc2as9f3tzGX97cxk9O+Rz1egBjBlWnO7QRESaDCWY/fDkeyuZ/Wkp1ePIVm4p59pJcwCUZEREQrpFth/unLqQnbv3XR+mvLKKO6cuTFNEIiJNjxLMfli1pbxB20VEMpESzH7o2SG/QdtFRDKREsx+uGr0APJzs/fZlp1lXDV6QJoiEhFpetTJvx+qO/LvnLqQVVvKaZOXzfadVfTrUpjmyEREmg4lmP00Zljx3kSztaKSU+6azo1Pf8gTl52AmaU5OhGR9NMtskbQrnUuV592GO8u36KZlkVEQkowjeScow5iaK8O3P7cAsp27k53OCIiaacE00iysoybvzKY9dt28oeXF6c7HBGRtFOCaURH9urAV48+iAkzP9ZEmCKS8VKaYMysyMwmm9l2M1tmZt+qo+wVZrbGzErNbIKZ5UWtx8y+ZmbzzWybmc0zszFJPK19XH3aYbTOyeaWf83TomQiktFS3YK5F9gFdAPOBf5kZoPjC5nZaGAcMAroAxwM3BylHjMrBv4G/ARoB1wFPGJmXZNzSvvq0jaPH53an1cXrefl+etScUgRkSYpZQnGzAqAs4Eb3L3M3WcATwPnJyh+AXC/u891983ArcCFEes5CNji7s95YAqwHeiXxNPbN/gT+nBI10JueWYeFZVVqTqsiEiTksrnYA4Fqtx9Ucy22cDJCcoOBp6KK9fNzDoBJfXUMwuYb2ZfAaYAXwZ2Ah/EH8TMxgJjAbp06cL06dP347QS+5+SKu6cVcF1D73Ml/u1arR6U62srKxRr0tLoeuSmK5LTZl8TVKZYAqB0rhtpUDbCGWr37etrx53rzKzh4BHgNYEt9K+6u7b4w/i7uOB8QADBgzwkSNHNuB06jYSmFP+H55dtJ4rzz6u2c5TNn36dBrzurQUui6J6brUlMnXJJV9MGUEfSKx2gHbIpStfr+tvnrM7FTgDoLf8a0IWjb3mdmR+x/6/rnu9IHscef25xak+tAiImmXygSzCMgxs/4x24YCcxOUnRvuiy231t03RqjnSOA1d5/l7nvc/R3gLeDUxjmN6HoVteF7J/fjX7NX8ebSjak+vIhIWqUswYS3qCYBt5hZgZmNAM4EHk5Q/CHgEjMbZGYdgeuBByLW8w5wUnWLxcyGASeRoA8mFb53cj+KO+Rz09Nz2V21p/4viIi0EKkepvx9IB9YBzwKXObuc82sxMzKzKwEwN2fJ7jN9QqwLHzdWF894XdfBW4C/mlm24AngF+6+wspOL8a8ltlc/3pA1mwZhuPvL08HSGIiKRFSmdTdvdNwJgE25cTdN7HbrsbuLsh9cTsvwe45wBCbVSnDenOCf068ZsXFnHGET0pKmi+o8pERKLSVDEpYGbc9JXBlO3czV0vLEx3OCIiKaEEkyKHdmvLt4/vzaNvL+fDlfGjrEVEWh4tOJZCPz71UB57ZwVn/fENKqv20LNDPleNHrB34TIRkZZECSaFXlmwjl1Ve6isCibBXLmlnGsnzQFQkhGRFke3yFLozqkL9yaXauWVVdw5Vf0yItLyKMGk0Kot5Q3aLiLSnCnBpFBt85E113nKRETqogSTQleNHkB+bnaN7Wcdpf4XEWl5lGBSaMywYm4/63CKO+RjQI/2relckMtjs1awoWxnusMTEWlUGkWWYmOGFe8zYmzeqq38zx9n8uO/v8+DFx9LdpalMToRkcajFkyaDerZjlvOHMyMJRv4/cuL0x2OiEijUYJpAr72mV6cdVQxv5+2mNcXr093OCIijSJygjGzL5rZM2Y2z8x6hdsuNbNRyQsvM5gZt40ZQv+uhfz47++zprQi3SGJiBywSAnGzM4FHgMWA32B3HBXNnB1ckLLLG1a5fDHc4+mvLKKyx95l0qtHSMizVzUFszVwHfc/Qpgd8z2NwlWkJRGcEjXQm4/63BmLdvMXXq6X0SauagJpj/w7wTby4B2jReOnHlkMecNL+Evry3lxXlr0x2OiMh+i5pgVgGHJtj+WeCjxgtHAG44YxCHF7fnysfeZ8WmHekOR0Rkv0RNMOOB35vZiPBzLzO7gGBZ4z8lJbIMlpeTzb3fOgoHvj/xXXburkp3SCIiDRYpwbj7HcAk4EWgAHgF+DPwZ3e/N3nhZa6STm34zVeHMmdlKbc9Mz/d4YiINFjkYcrufh3QGTgWGA50cfcbkhWYwBcGd2fsZw/m4TeX8fTsVekOR0SkQSJNFWNmE4Afufs2YFbM9gLgD+5+cZLiy3hXjR7Au8s2c+Vj73PbM/NYv22nVsIUkWYhagvmAiDRnPL5wLcbLxyJl5udxRlDe1BZ5azbthPnvythPvneynSHJyJSqzoTjJkVmVknwICO4efqVxfgDEBjaZPs/177uMY2rYQpIk1dfbfINgAevuYl2O/AjY0dlOxLK2GKSHNUX4L5HEHrZRpwNrApZt8uYJm7q/c5yXp2yGdlgmSilTBFpCmrM8G4+6sAZtYXWOHumiArDa4aPYBrJ82hvHLf52GO69sxTRGJiNQv0igyd18GYGY9gRKgVdz+1xo/NKlWPVrszqkLWbWlnB4dWlPUphWT3lvFcQd34uvHlKQ5QhGRmqIOU+4JPEIwNYwT3DbzmCI1F5qXRhW/EubO3VV856H/MG7SHFrnZnPmkRqyLCJNS9Rhyr8FqoBBwA7gJOCrwHzgtKREJnXKy8nmL+cdzTF9ivjJY7N5Ye6adIckIrKPqAnmZOAad19A0HJZ7+6TgGuAW5MVnNQtv1U2Ey48hsOL23P5I+/x2iKthikiTUfUBJNPMGQZgpFkXcP384AjGjsoia4wL4cHLzqWfl0LGfvwLN5aujHdIYmIANETzALgsPD9+8D3zKw38ANAj5OnWfs2uTx8ybEUd8jnkgdn8f6KLekOSUQkcoL5HdA9fH8L8AVgKfB94GdJiEsaqHNhHhMvHU7HglwumPA281dvTXdIIpLhok7XP9HdHwjfvwv0AY4BStz98agHC6eYmWxm281smZl9q46yV5jZGjMrNbMJZpYXtR4za2NmfzSzDeH3M2IYdff2rXnk0uG0aZXN+fe/xZJ1ZekOSUQyWOTp+mO5+44w0Ww3s3EN+Oq9BDMAdAPOBf5kZoPjC5nZaGAcMIogmR0M3NyAesYDRcDA8N8rGhBjs9arqA1/u/Q4AM677y2tiCkiaVPvczBm1hk4DqgEXnb3KjPLJeh/uZbgGZhfRaingGC6mSHuXgbMMLOngfMJkkmsC4D73X1u+N1bgYnAuPrqMbMBwFeAg9y9+j7Rf+qLryXp16WQhy85jm+Mf5Mz751Bq+ws1m7VNP8iklp1JhgzOwGYArQnGJ78jpldCEwGcgmGKE+IeKxDgSp3XxSzbTbBEOh4g4Gn4sp1C2d2LqmnnuOAZcDNZnY+sBq4yd2fSHB+Y4GxAF26dGH69OkRT6V5OKXYmbxk997PK7eUc/Xj7zNv/jxO6JkbqY6ysrIWd10ag65LYrouNWXyNamvBXMrMBW4DbgY+DHwDEFH/8Pu7rV/tYZCoDRuWynQNkLZ6vdtI9RzEDAEeALoCRwPTDGzee6+z9rD7j6e4HYaAwYM8JEjRzbgdJq+696cBuzeZ9uuPTBleTY/+9bISHVMnz6dlnZdGoOuS2K6LjVl8jWprw9mKHCru38IXE/QirnW3R9qYHIBKAPaxW1rB2yLULb6/bYI9ZQT3M67zd13hRN2vkIw8i2jaJp/EUmn+hJMEbAego59gmli3tvPYy0Ccsysf8y2ocDcBGXnhvtiy611940R6vlgP+NrcWqbzr9jQauE20VEGlOUUWTVK1l2ImjBtItb2bIoyoHcfTswCbjFzArMbARwJvBwguIPAZeY2SAz60jQenogYj2vAcuBa80sJ9w/kuBWX0a5avQA8nP3nYfUDDZt38WEGR/T8EaoiEh0URLMPIJWzDqC/o93ws/rCaaPacgEWN8nmHZmHfAocJm7zzWzEjMrM7MSAHd/HriD4NbWsvB1Y331hN+tJEg4XyLom/k/4NvhPGoZZcywYm4/63CKO+RjQHGHfH591uGMHtyNW56Zx41Pz2V3lZb4EZHkiLKiZaNx903AmATblxMkr9htdwN3N6SemP1zCTr3M178NP8A5xzdi18/v4C/vLaU5Zt28IdvDqNt62ijykREooq0oqW0LFlZxrVfGkifzgVc/+SHfPXP/+b+C4+hWEswi0gj2q8n+aVl+OaxJTx40bGs3FLOmffMZLYmyRSRRqQEk+FO7N+ZSZedQOvcLL4+/t88/+HqdIckIi2EEozQv1tbnvzBCAb2aMdlE9/lL69+pBFmInLA6p2LTDJD58I8Hv3OcH76+Gxuf24B0xeuY9mmHazaUkHxm9M0h5mINJgSjOzVOjeb339jGLt2V/HCvHV7t6/cUs61k+YAKMmISGSREoyZ1TahpQMVwBLgH+6+qrECk/TIyjLmrqo5e095ZRV3Tl2oBCMikUVtwXQBTgL2AB+G24YARjAV/lkET9af5O7vN3aQklqaw0xEGkPUTv6ZwHMEa6x81t0/SzBr8bPAC0Bvgmn9f5OUKCWlapvDrFVOFqtLlWREJJqoCeZHwC3hhJfA3skvfwFc4e67gF8DRzZ6hJJyieYwy8029uxxTvvt6zw7R0OZRaR+URNMIdAjwfbu/HeKl61o0ECLEDuHGQRzmN15zlBe+MnJ9OnUhu9PfJerHp9N2c7d9dQkIpksakKYDNxvZlcTTHbpwLEEE1JOCsscSzCVvrQA1XOYxS+W9M/LTuB3Ly3mj9OX8PYnm/jt149kWEnH9AUqIk1W1BbM9wimu/8b8BGwNHz/PMHMxgDzge80doDStORmZ/HT0QP4+9jj2V3lnPPnf/P7lxdrVmYRqSFSgnH3He7+PYIFyIYBRwFF7n5ZuD4L7v6+RpBljmP7FvHcj0/ijCN6cPeLi/jG+DdZsWlH/V8UkYzRoD6TMJloxUgBoF3rXH73jWF8bkBXbnjyQ774u9c5c1gPpi9Yz6otFfTskK8ZAEQyWNQHLVsTjCQbBXQlruXj7kc0fmjSXIwZVszRvTvy7QlvMfHNFXu3awYAkcwWtQXzR+B/gMeBNwg6+UX26lXUhp27a/bDaAYAkcwVNcGMAb7q7i8lMRZp5lZvqUi4XTMAiGSmqKPIdgAr6i0lGa22GQAc+OWz8/XcjEiGiZpg7gB+YmZaP0ZqlWgGgNa5WRzXtyPjX1vKqN9M5+nZq7TWjEiGiHqL7PMEk12eZmbzgMrYne7+lcYOTJqf6n6WO6cuZNWW8n1Gkb27fDM/f+pDfvjoezz61nJuPnMwh3Zrm+aIRSSZoiaYDQRP84vUqXoGgHhHlXTkqR+cyCNvL+euqQv50u9e56IRffjRqYdSmKcZhkRaokj/Z7v7RckORFq+7Czj/OG9Of3wHtzx/ALum/ExT72/iutOH8iePc5dLyyq0fIRkeZLfzpKyhUVtOJXZx/BN44t4edPfciP/v4+WQZ7wq4ZPT8j0jLU2mlvZh+YWcfw/Zzwc8JX6sKVluTIXh2Y/P0RdMjP3ZtcqlU/PyMizVddLZgngJ3h+3+mIBbJQNlZRml5ZcJ9en5GpHmrNcG4+82J3os0tp4d8llZSzK5+4WFXHLSwbTPz01xVCJyoPRci6Rdoudn8nKyGHpQe34/bQkn/Xoaf3h5sR7UFGlmok52WUSwPHJtk122a/zQJFPU9fzM3FWl/O+Li/nNi4uYMPNjvntyP759fG/atNL4FJGmLur/pfcTrAMzHliFJruURlbb8zODe7bnvgs+w+wVW/jflxbxq+cWcN/rS7ls5CGce1wJz3+4JmFiEpH0i5pgRgGfd/e3khmMSG2G9urAAxcdy3+WbeLuFxdx6zPz+N1LCymv3ENlVfD3joY3izQtUftg1gFlyQxEJIqjexcx8dLhPPqd4VRU+t7kUk3Dm0WajqgJ5jrgFjMrTGYwIlEd368TlVU1158BDW8WaSqiJpjrgS8A68xsvh60lKagruUBrnxsNvNWbU1tQCKyj6gJ5p/AXcCvgb8TPIQZ+4rEzIrMbLKZbTezZWb2rTrKXmFma8ys1MwmmFleQ+sxsxvNzM3s1KgxSvNR2/Dmk/p35rkPV/Ol37/ON8e/yUvz1rInfqoAEUm6ejv5zSwXKADudfdlB3i8e4FdQDfgSGCKmc1297lxxxwNjANOIRi1Nhm4OdwWqR4z6wecA6w+wJiliapreHNpeSV/f3s5D77xCZc+NIuDOxdw0Yg+nH30QbRplcOT763U6DORJKs3wbh7pZldBvzxQA5kZgXA2cAQdy8DZpjZ08D5/DdxVLsAuL86YZjZrcBEYFwD6rkHuOZA45amrbbhze3zc/nuyf24+MS+PPfhGu6f8TE3PDWXu15YxGf6dGTm4g1U7A76cDT6TCQ5og5TfoGgNTHhAI51KFDl7otits0GTk5QdjDwVFy5bmbWCSiprx4z+yqwy92fNbNaAzKzscBYgC5dujB9+vQGnVAmKCsra/bXpR3w40HOkp6tmfpJJS/PX1ejTHllFbc+NZsOpYsj1dkSrksy6LrUlMnXJGqCeRn4pZkdAfwH2B67090nRaijECiN21YKJFrWML5s9fu29dUTjnT7JcGghDq5+3iCh0cZMGCAjxw5sr6vZJzp06fTUq7L54DvAH3HTUn4pPCmCo98ri3pujQmXZeaMvmaRE0w94T//jDBPgeyE2yPV0bwx2SsdsC2CGWr32+LUM/NwMPu/nGEmCQD1Ta5pgM/mPguXzumFyce0pnsrNpbvyJSv0ijyNw9q45XlOQCsAjIMbP+MduGAnMTlJ0b7ostt9bdN0aoZxTww3AE2hqgF/CYmV0TMU5p4WobfXZy/8688dEGLpjwNif9ehp3v7iIFZt2pClKkeYvZTMGuvt2M5tE8MDmpQSjv84ETkhQ/CHgATObSDAK7HrggYj1jAJi53Z/B/gJ8Fwjn5I0U3WNPtu5u4qX5q3jH7NW8Idpi/nDtMWM6NeZrx3Tiy8M6rZ37rOVW8opfnOaRp+J1CFygglnVD6NoJO9Vew+d78lYjXfJxgosA7YCFzm7nPNrASYBwxy9+Xu/ryZ3QG8AuQTPGtzY331hLFsjIu7CtgcjjgTAWoffZaXk83pR/Tg9CN6sHJLOY/PWsHjsz7lh4++R35uFruqnKo9mvtMJIqo0/UPB6YQrHDZBVgJ9Ag/fwJESjDuvgkYk2D7coLO+9htdwN3N6SeWsr2iVJOJF5xh3x+fOqh/PCU/sz8aANjH5q1N7lUq577TAlGpKaoT/LfSfAcSjFQQTBkuQSYRfB0v0iLlZVlnNS/CxWViec+W7mlnPteX8rqUs2BJhIraoI5ArjH3R2oAvLcfS3Bg4w3JSk2kSaltrnPcrON26bM5/jbp/HVP7/Bg298wrptFSmOTqTpidoHsyvm/VqgNzCfYMhwz8YOSqQpumr0AK6dNIfyyqq92/Jzs7n9rMMZ2qsDz8xexTMfrObGp+dy87/mMvzgTpxxRE++OKQ7ry5ar6lpJONETTDvAscQDBGeDtxmZt2A8wDNpiwZIXb02cot5RTHJYr/N6o//29Ufxat3bY32fxs8hyumzwHM6juvtHgAMkUDVkPZlX4/npgPfAHoCPhVCsimWDMsGJmjjuFB04rYOa4UxImiEO7teUnXxjAy1eezJQfnkhBXg7xkzmXV1Zx+3PzUxS1SHpEasG4+6yY9+uBLyYtIpEWwswY3LM923fuTrh/7dadnHr3q5w6sBunDuzKsJKOmj1AWpQGPWhpZp8B+gHPhA88FgA73T3x/0EiUuvUNO3zc+jerjX3vb6UP7/6EUUFrfjcgK58flBXTurfhYI8LSsgzVvU52C6AU8T9MM40B9YSvCcSgXwo2QFKNLc1TY44OavDGHMsGK2VlTy2qL1vDRvLS/NX8sT735Kq+ws+nYpYOn6Miqr9GCnNE9RWzD/C6wBOgHLY7Y/TtAXIyK1qGtqGoB2rXM544ienHFET3ZX7WHWss28PH8tE2Z+kvDBzl8/v0AJRpqFqAlmFDDK3TfHra/yEcEDlyJSh9qmpomXk53F8IM7MfzgTtz3euIJwVeXVvDN8W9yYv/OnNS/M4N7tlffjTRJURNMPvs+C1OtC8EtMhFpZLX13RTm5bClvJI7py7kzqkL6dAmlxH9gmRzYv/OHNSxjfpupEmImmBeAy4EfhZ+djPLJniS/+UkxCWS8Wrru7ltTNB3s37bTmYu2cDrizcwY8l6psxZDUDnwlZs3lGpSTkl7aImmKuBV83sGCAP+A3BssbtgRFJik0ko9XXd9Olbd7eW2/uzpJ1Zby2eAN3PL8gYd/NLf+ay0n9O9OpMC/l5yKZKepzMPPM7HDgMoIZlFsTdPDf6+6rkxifSEaL2ndjZvTv1pb+3dpy2zPzEpbZtKOSo297iUO6FnJs3yKO61vEcX070b19671ldGtNGlPk52DcfQ37rsmCmfU2s8fc/WuNHpmI7Jfa+m46F+ZxyYl9efvjjfzr/VU88lYwILSkqA3H9i0iN9uY9O5Kdu4OZo3WrTU5UAe6omUH4OxGiENEGkltfTfXnz6QMcOKuWxkP6r2OPNXb+Wtjzfx1tKNvDx/LZt3VNaoq7yyijs0LFr2U8qWTBaR1Kiv7wYgO8sYUtyeIcXtueTEvuzZ4/T72bN4gvpWlVYw5t6ZHFXSkWElHRhW0oHiDvnEPrJQfWtNS0lLLCUYkRYoat9Ntawsq3NYdKvsLB55exkTZgbP5nRpm8dRJR0YVtKRsopK7pvx8d4F2XRrTaopwYgIUP+w6MqqPSxcs433lm/m3eVbeG/5ZqbOXZuwLt1aE6gnwZjZ0/V8v10jxiIiaVTfrbXc7Ky9t9XOPz74zqbtuzjq1hcT1reqtIIv/2EGQ4rbc3j4GtC9La1y/rtKiEattWz1tWA2RtifeD4LEWl2GnprraigFcV13Fprl5/DlA9W8ejbwYi1VtlZDOjeliHF7dm9Zw9Pv79Ko9ZasDoTjLtflKpARKR5qu/WmruzfNMO5qwsZc7KUj5cWcqUD1axtaLmKh/llVXc+sy8eh8IVcuneVAfjIgckPqWkjYzencqoHenAs44oicA7s7B1yYetbZx+y6Ovu0lurbN47Ae7RjYoy0Du7djYI92HNylgCkfrN4noanl03QpwYjIAau+tTZ9+nRGjhxZb3mz2ketdS5sxfdO7sf81duYv3orf/1oI7uqgttorbKz2OPO7gRT4dw5daESTBOjBCMiaVH7A6GD9kkUlVV7WLp+O/NXb2X+mq385dWlCetbuaWccU98EEyZ07WQQ7u1pVu7vITP6+jWWmoowYhIWkR5IBSC0WsDurdlQPe2jKGYZ2avTtjyaZWdxQvz1vL3d1bs3da2dc7eZFNRWcWzc9bsbQ3p1lryKcGISNo0dNQa1N7yuf2swxkzrJiNZTtZtLaMxeu2sXhtGYvWbuOFeWvZtL3mklbllVXc8NSH5GQbB3cu5OAuBbTOza5RTi2f/aMEIyLNSn0tn06FeRxfmMfx/Trt872+46YkHFSwrWI3lz/yHgBm0LN9Pgd3KaBfl0L6dSlgdWkFE2Z8TIWGUzeYEoyINDv70/KpbVBBz/atue+CY/hofRlL129n6YYyPlpfxmOzVrBjV1WCmoKWz01Pz6VH+9b06VxA17b79vWA5mcDJRgRyRC13Vq7+rTDGNSzHYN67jsxibuzdutOht+eeNHeLeWVfH38m3vr6d2pDb07taFP5wI2b9/Fk++vYleGt3qUYEQkI0QdVFDNzOjevnWtMxV0a5fHnecMZdnG7XyycQefbNjOknVlvLJg/d6BBLHKK6u4bvIcNpTt5KCObSgpakOvonzats7dp1xL6u9RghGRjNGYgwqu/eJAPntoF6DLPuWr9jiH1LL0wfZdVdw2Zf4+2zq2yaWkqA0HFbWhorKK1xatp7Iq+HZDWj5NMTEpwYiI1KGhLZ/sOpY+KO7Qmik/PInlm3awYlN58O/mHazYtIO5K0v5ZOOOGt8pr6zi6n9+wLQF6yjumE9xh3yKO+ZzUPhvm1Y5PPneyiY5u0FKE4yZFQH3A18ANgDXuvsjtZS9ArgGyAeeAC5z95311WNmw4FbgaOBKmA68EN3X528MxORlqyhLZ/aWj1XjT6MDm1a0aFNK444qEON79U20m1X1R7eX7GFZ+esrjGLQVFBK7ZVVO5t9VQrr6ziV88t4CtDe5KVte8AhFjJbPmkugVzL7AL6AYcCUwxs9nuPje2kJmNBsYBpwCrgMnAzeG2+urpCIwHpgK7gXuAvwKnJfPERESq1Tc/W21qb/nk89rVn6Nqj7NuWwUrN5ezcks5n4b/PvLW8oT1rdlawYAbnqN7+9b0aJ9Pz/at6dEh/Ld9PgvWbuWeaUv2a7G46sTUqvshR9dWJmUJxswKgLOBIe5eBswI15s5n/8mjmoXAPdXJx4zuxWYCIyrrx53fy7uuPcArybx1EREamjo/GxQV8tnABDcfuvRPp8e7fP5TMz3Xl24PmFiap+fyzePLWF1aTmrt1Qwa9lm1s5ZXaO1E6u8sorrn/yQzTt20b1da7q1b033dq3p2jaPnOxgLZ/4W3K1MffaD9SYzGwY8Ia758ds+ylwsrt/Oa7sbOCX7v6P8HNnYD3QGSiJWk+478fAN9x9eIJ9Y4GxAF26dDn6scceO+DzbGnKysooLCxMdxhNjq5LYrouNTX0mryxqpInFlWyscLp1No4+9BcTuiZW+93HvhwF7tiBq+1yoILh7Sq8d097mzd5WyqcG75d0XkuAxol2d0zDNWlu0hbPSw+sEfs3P14oT34FJ5i6wQKI3bVgq0jVC2+n3bhtRjZkcAPwfOTBSQu48nuJ3GgAEDPOpfGZmkIX99ZRJdl8R0XWpq6DUZCfysgccYCQzaj76U++dPq3UwwtOXn8iarRWs3VrBmtKdwfvSCtZsreCTResjxZXKBFNGzSWW2wHbIpStfr8taj1mdgjwHPAjd399P2MWEWkWGnMI9lWjD6NTYR6dCvMY3LN9je+N+FXixBQvq94SjWcRkGNm/WO2DQXmJig7N9wXW26tu2+MUo+Z9QZeAm5194cbKX4RkRZlzLBibj/rcIo75GMEgwmqJw2ty1WjB5CfYFLQeClrwbj7djObBNxiZpcSjP46EzghQfGHgAfMbCKwGrgeeCBKPWZWDEwD7nX3PyfznEREmrv9afnEjpKr6/mPVLZgAL5P8FzLOuBRgmdb5ppZiZmVmVkJgLs/D9wBvAIsC1831ldPuO9S4GDgxrDOMjMrS8G5iYhkjDHDipk57hR2rVnyn9rKpPQ5GHffBIxJsH05Qed97La7gbsbUk+472aCZ2ZERCSNUt2CERGRDKEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSZHSBGNmRWY22cy2m9kyM/tWHWWvMLM1ZlZqZhPMLC9qPWY2yswWmNkOM3vFzHon87xERKSmVLdg7gV2Ad2Ac4E/mdng+EJmNhoYB4wC+gAHAzdHqcfMOgOTgBuAImAW8I/knI6IiNQmZQnGzAqAs4Eb3L3M3WcATwPnJyh+AXC/u891983ArcCFEes5C5jr7o+7ewVwEzDUzA5L3tmJiEi8nBQe61Cgyt0XxWybDZycoOxg4Km4ct3MrBNQUk89g8PPALj7djP7KNy+IPYgZjYWGBt+3GlmHzb4rFq+zsCGdAfRBOm6JKbrUlNLvya1dkGkMsEUAqVx20qBthHKVr9vG6GeQmB9lOO4+3hgPICZzXL3z9R9CplH1yUxXZfEdF1qyuRrkso+mDKgXdy2dsC2CGWr32+LUE9DjiMiIkmSygSzCMgxs/4x24YCcxOUnRvuiy231t03Rqhnn++GfTb9ajmOiIgkScoSjLtvJxjddYuZFZjZCOBM4OEExR8CLjGzQWbWEbgeeCBiPZOBIWZ2tpm1Bn4OfODuC+IPEmf8gZ1hi6XrkpiuS2K6LjVl7DUxd0/dwcyKgAnA54GNwDh3f8TMSoB5wCB3Xx6W/QlwDZAPPAF8z9131lVPzHFOBe4h6Hx6C7jQ3T9JyUmKiAiQ4gQjIiKZQ1PFiIhIUijBiIhIUmR8gmnI/GiZxMymm1mFmZWFr4XpjikdzOxyM5tlZjvN7IG4fRk5511t18TM+piZx/zMlJnZDWkMNaXMLM/M7g9/j2wzs/fM7Isx+zPu5yXjEwwR50fLUJe7e2H4GpDuYNJkFXAbwaCSvTJ8zruE1yRGh5ifm1tTGFe65QArCGYVaU/ws/FYmHgz8ucllU/yNzkx85oNcfcyYIaZVc9rNi6twUmT4O6TAMzsM8BBMbv2znkX7r8J2GBmh0UYEt+s1XFNMlr4CMVNMZueMbOPgaOBTmTgz0umt2Bqmx9NLZjA7Wa2wcxmmtnIdAfTxNSY8w6onvMu0y0zs0/N7K/hX+4Zycy6EfyOmUuG/rxkeoJpyPxomeYagmUSigkeFPuXmfVLb0hNin52atoAHEPw/NnRBNdiYlojShMzyyU49wfDFkpG/rxkeoLRvGW1cPe33H2bu+909weBmcCX0h1XE6KfnTjh8hmz3H23u68FLge+YGbx16lFM7MsgplFdhFcA8jQn5dMTzANmR8t0zlg6Q6iCdGcd/Wrfoo7Y35uzMyA+wkGDZ3t7pXhroz8ecnoBNPA+dEyhpl1MLPRZtbazHLM7Fzgs8DUdMeWauH5twaygezqa8L+z3nX7NV2TczsODMbYGZZ4dpNvwemu3v8raGW7E/AQODL7l4esz0zf17cPaNfBEMGnwS2A8uBb6U7pnS/gC7AOwTN9y3Am8Dn0x1Xmq7FTQR/ice+bgr3nUqwiF05MB3ok+5403lNgG8CH4f/L60mmLS2e7rjTeF16R1eiwqCW2LVr3Mz9edFc5GJiEhSZPQtMhERSR4lGBERSQolGBERSQolGBERSQolGBERSQolGBERSQolGJEWKlyb5Zx0xyGZSwlGJAnM7IHwF3z86810xyaSKhm9HoxIkr1EsLZQrF3pCEQkHdSCEUmene6+Ju61CfbevrrczKaES+guM7PzYr9sZoeb2UtmVm5mm8JWUfu4MheY2Zxw+eK18cs6A0Vm9ni4JPjS+GOIJJMSjEj63Aw8DRxJsObOQ+EqkZhZG+B5grmsjgX+BziBmGWKzey7wF+AvwJHECynED8778+Bpwhm8v0HMCET1oKXpkFzkYkkQdiSOI9g4sNY97r7NWbmwH3u/p2Y77wErHH388zsO8BdwEHuvi3cPxJ4Bejv7kvM7FPgb+6ecHnv8Bi/cvdrw885wFZgrLv/rfHOViQx9cGIJM9rwNi4bVti3v87bt+/gdPD9wMJpnOPXZDqDWAPMMjMthKsNvpyPTF8UP3G3Xeb2Xqga6ToRQ6QEoxI8uxw9yX7+V3jvwt2xWvI4m+VcZ8d3RqXFNEPmkj6DE/weX74fh4w1Mxi12w/geD/2fkeLEm8EhiV9ChF9pNaMCLJk2dm3eO2Vbn7+vD9WWb2DsHiU+cQJIvjwn0TCQYBPGRmPwc6EnToT4ppFf0C+F8zWwtMAdoAo9z9N8k6IZGGUIIRSZ5TCVZ2jLUSOCh8fxNwNsHSwuuBi9z9HQB332Fmo4HfAm8TDBZ4CvhRdUXu/icz2wVcCfwa2AQ8m6RzEWkwjSITSYNwhNdX3f2f6Y5FJFnUByMiIkmhBCMiIkmhW2QiIpIUasGIiEhSKMGIiEhSKMGIiEhSKMGIiEhSKMGIiEhS/H9sdrylM8/uCQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "learning_rate = 0.01\n",
    "decay = 1e-4\n",
    "batch_size = 32\n",
    "n_steps_per_epoch = math.ceil(len(X_train) / batch_size)\n",
    "epochs = np.arange(n_epochs)\n",
    "lrs = learning_rate / (1 + decay * epochs * n_steps_per_epoch)\n",
    "\n",
    "plt.plot(epochs, lrs,  \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.01])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Power Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exponential Scheduling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```lr = lr0 * 0.1**(epoch / s)```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay_fn(epoch):\n",
    "    return 0.01 * 0.1**(epoch / 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay(lr0, s):\n",
    "    def exponential_decay_fn(epoch):\n",
    "        return lr0 * 0.1**(epoch / s)\n",
    "    return exponential_decay_fn\n",
    "\n",
    "exponential_decay_fn = exponential_decay(lr0=0.01, s=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 1.1122 - accuracy: 0.7363 - val_loss: 0.8947 - val_accuracy: 0.7496\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.7354 - accuracy: 0.7825 - val_loss: 0.6059 - val_accuracy: 0.8122\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5973 - accuracy: 0.8175 - val_loss: 0.8195 - val_accuracy: 0.7754\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.6040 - accuracy: 0.8148 - val_loss: 0.6135 - val_accuracy: 0.8398\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5462 - accuracy: 0.8323 - val_loss: 0.5075 - val_accuracy: 0.8490\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4479 - accuracy: 0.8555 - val_loss: 0.4538 - val_accuracy: 0.8502\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4225 - accuracy: 0.8622 - val_loss: 0.4792 - val_accuracy: 0.8524\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3873 - accuracy: 0.8678 - val_loss: 0.5517 - val_accuracy: 0.8448\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.3635 - accuracy: 0.8767 - val_loss: 0.5312 - val_accuracy: 0.8600\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.3353 - accuracy: 0.8840 - val_loss: 0.4671 - val_accuracy: 0.8660\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3108 - accuracy: 0.8927 - val_loss: 0.4885 - val_accuracy: 0.8670\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2895 - accuracy: 0.8987 - val_loss: 0.4698 - val_accuracy: 0.8636\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2660 - accuracy: 0.9071 - val_loss: 0.4558 - val_accuracy: 0.8820\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2442 - accuracy: 0.9153 - val_loss: 0.4325 - val_accuracy: 0.8774\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2375 - accuracy: 0.9177 - val_loss: 0.4703 - val_accuracy: 0.8800\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2196 - accuracy: 0.9231 - val_loss: 0.4657 - val_accuracy: 0.8870\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2013 - accuracy: 0.9312 - val_loss: 0.5023 - val_accuracy: 0.8760\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1938 - accuracy: 0.9331 - val_loss: 0.4782 - val_accuracy: 0.8856\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1774 - accuracy: 0.9394 - val_loss: 0.4815 - val_accuracy: 0.8898\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1703 - accuracy: 0.9418 - val_loss: 0.4674 - val_accuracy: 0.8902\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.1611 - accuracy: 0.9462 - val_loss: 0.5116 - val_accuracy: 0.8930\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1530 - accuracy: 0.9481 - val_loss: 0.5326 - val_accuracy: 0.8934\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.1436 - accuracy: 0.9519 - val_loss: 0.5297 - val_accuracy: 0.8902\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1326 - accuracy: 0.9560 - val_loss: 0.5526 - val_accuracy: 0.8930\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1308 - accuracy: 0.9560 - val_loss: 0.5699 - val_accuracy: 0.8928\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.LearningRateScheduler(exponential_decay_fn)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1kklEQVR4nO3deXhU5dnH8e+dBEgIS9gMEAVFEAUUcUOlVhQV21rFpbXVWm21tLV2sa1r3XCvtmpbl0rdXre644ZiRYyKKyiChFVlk32RQEICIdzvH+cEx2GSnGBmJmR+n+uaKzPnPPPMfQ5D7pzzbObuiIiINLasdAcgIiLNkxKMiIgkhRKMiIgkhRKMiIgkhRKMiIgkhRKMiIgkhRKMSAqY2VlmVtbA9xSb2e3Jiin8jPlm9uck1HuKmTVoDET8OdqecyZNixKMJJWZPWBmnuDxXrpjS5bw+E6J2/w40CsJn3WOmU0xszIzKzWzaWZ2bWN/Tpok5ZxJ6uSkOwDJCOOBM+K2bUpHIOni7hVARWPWaWY/B/4JnA+8BrQE+gOHNObnpEsyzpmklq5gJBU2uvuyuMcaADM73MyqzGxoTWEz+5WZrTOzXuHrYjP7t5n9w8y+DB83m1lWzHs6mNn/hfsqzGy8mfWP2X9W+Ff+MDObbmblZva6me0WG6iZfd/MPjSzSjObZ2bXmVnLmP3zzewyM7s7jPELM7sgdn/49MnwSmZ+7OfHlNvdzJ4zs2VhLB+Z2XENPK/HA8+4+93u/qm7z3D3J939j3HH9D0zez88L6vN7AUzy40pklvb8YTvb29mo81shZmtN7M3zOyAuDI/NbMFZrbBzF4ECuP2X2Vm0+O21XkLLME5uyr8t/uRmX0WxvKsmXWOKZNjZrfGfE9uNbO7zKy4/tMpjU0JRtLK3d8AbgYeMrOOZrYn8Hfgt+7+eUzR0wm+r4cAvwRGAn+I2f8AMBg4ATgI2ACMM7O8mDKtgEuAn4f1FAD/rtlpZsOBR4DbCa4Efg6cAlwfF/b5wCfAfsBfgZvMrOaq4cDw5y+AbjGv47UBXgaOBgYCTwPPhMcf1TLgoJpEnIiZHQs8B7wK7A8cAbzB1//v13o8ZmbAWKAIOA4YBLwJTDCzbmGZwQTnfzSwL/ACcHUDjqMhdgVOBU4EjgnjuS5m/5+Bs4BzgIMJjvO0JMUi9XF3PfRI2oPgF89moCzu8deYMi2AScAzwEfA43F1FANzAIvZdhnwRfi8D+DAt2P2twdKgXPC12eFZfrGlDmd4FZdVvj6TeDyuM8eEcZr4ev5wH/jyswFLot57cApcWXOAsrqOVfvxdVTDNxeR/luwLvh580FHgZ+CrSIKfM28FgdddR5PMCR4fHnxZX5GLgwfP4o8Grc/nuCXy9bX18FTK/rnER4fRVQCbSP2fYX4NOY10uBi2NeGzALKE73/4VMfOgKRlLhTYK/bGMfN9fsdPcqgr8yjwN2IrhCifeeh78xQu8CRWbWDtgL2BJuq6mzlOCv8n4x79no7rNjXi8hSG4F4ev9gb+Et9LKwtszjwL5QNeY902Li21JGHdkZpZvZjeZ2YzwVk4ZcADQI2od7r7U3Q8B9gZuI/hlejfwgZm1DosNImifqUtdx7M/0BpYGXdeBgC7h2X2Iubch+JfN5YF4b/tNrGaWXuCf6cPanaG35lJSYpF6qFGfkmFDe7+aT1lam5nFABdgLUNqN/q2BeblDbXsi8r5uco4MkE9ayMeV6VoJ6G/rH2N+BYgls6cwlu6T1I0FDfIO4+HZgO3GFm3wLeAn5IcPUYRV3HkwUsBw5L8L514c+6zn+NLQnKtYgYX6wo515TxDcRuoKRtDOzXQnaPX5D0FbwiJnF//EzOGwPqHEwsMTd1wEz+Kp9pqbOdgR/2c9oQCgfAXt60GAe/4hPTnWpArLrKfMt4EF3f9rdpwFf8NUVwTdRc7xtwp9TgGHfoL6PCBrstyQ4JytiPvPguPfFv14JFMb9G+77DeLaRnhls4ygDQ7Y2oZUWzuYJJmuYCQVWplZ17ht1e6+0syyCdoO3nD3u83sKYJbW1cCl8eU7w7cZmZ3EiSOC4BrAdx9rpk9B9xtZiMJrn6uI/gL+9EGxHk18KKZLQCeILjiGQAc5O4XNqCe+cAwM3uD4LbclwnKzAFODOOuIjje3ATlamVmdxHcIppAkKC6EbRNbQD+Fxa7DnjBzD4lOBdG0Dh+t7tviPAx4wnacZ4zswsJ2jO6Elx9jXf3twi6Sr9jZpcATwFDCRrhYxUDHYFLzeyxsEz8WKHG8A/gQjObQ5D4fklwXpYm4bOkHrqCkVQ4iuA/eOxjSrjvUqA3cDaAu68GzgQuDm/31HiE4KrgfeA/wL3ArTH7f0Zw7/358Gdr4FgPxlJE4u6vAN8j6Gn1Qfi4GFgY/VAB+FNYxyK+Os54fwRWENzOepmggf+tBn7OqwQ9554gSFhjwu1Hu/scAHd/ieCX/XfCWN4IY9sS5QPCNozvEiSx/wCzw8/rS5DccPf3CP79fk3QnnMSQYN8bD0zw/0jwzJHs23vvMbwN+Ah4H6CcwrBealMwmdJPWp6xog0WeEYhunufl66Y5Edj5l9BLzt7r9NdyyZRrfIRKTZMLOewHCCK7UcgiumgeFPSTElGBFpTrYQjAW6maAJYAbwHXefnNaoMpRukYmISFKokV9ERJJCt8hCBQUF3rt373SH0eSUl5eTn5+f7jCaHJ2XxHRettXcz8mHH364yt27JNqnBBMqLCxk8mTdpo1XXFzM0KFD0x1Gk6PzkpjOy7aa+zkJx40lpFtkIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFClNMGbW0czGmFm5mS0ws9PqKHu+mS0zs1Izu8/MWsXsO8/MJpvZRjN7IMF7h5nZLDPbYGavm1nP+mKbv24LQ26cwLNTFm/38YmIyFdSfQVzB7AJKAROB+4ys/7xhcxsOHAxMAzYFegFjIopsgS4FrgvwXs7A88AlwMdgcnA41GCW7y2gkue+URJRkSkEaQswZhZPnAycLm7l7n7ROB54IwExc8E7nX3Enf/ErgGOKtmp7s/4+7PAqsTvPckoMTdn3T3SuAqYKCZ7Rklzoqqam5+ZXb0AxMRkYRSuWTyHkC1u8+J2TYVODxB2f7Ac3HlCs2sk7snSirx751a88Ldy83ss3D7rNiCZjYSGAnQsmvvrdsXr62guLi4vuPJCGVlZToXCei8JKbzsq1MPiepTDBtgNK4baVA2whla563JfFVS/x7V0b5HHcfDYwGaNWtj9dsLyrIa9ZraDdEc19PfHvpvCSm87KtTD4nqWyDKQPaxW1rB6yPULbmeaKy3+RztmHAH47qE6WoiIjUIZUJZg6QY2axv70HAiUJypaE+2LLLY9we2yb94ZtP7vX8jlf07lNSxz4bGV5hI8REZG6pCzBuHs5Qe+uq80s38yGACcADyUo/iBwtpn1M7MOwGXAAzU7zSzHzHKBbCDbzHLNrOZ23xhggJmdHJa5Apjm7rOow67tsph82dGcesAu/Oetz/nki/i7eSIi0hCp7qZ8LpAHrAD+C/za3UvMrIeZlZlZDwB3HwfcBLwOLAgfV8bUcxlQQdCV+Sfh88vC964k6K12HfAlMBj4UdQAL/3eXnTKb8mFT0+jqnrLNzlWEZGMltIE4+5r3H2Eu+e7ew93fzTcvtDd27j7wpiyt7h7obu3c/efufvGmH1XubvFPa6K2T/e3fd09zx3H+ru86PG2D6vBdeMGMDMpeu4+43PGufARUQykKaKSWB4/658b+9u/PO1T/l0RaS+ASIiEkcJphZXHd+f1q2yufCpaVRv8frfICIiX6MEU4subVtxxXH9+GjhWh56d366wxER2eEowdThxEFFHL5HF256ZTaL1mxIdzgiIjsUJZg6mBnXnTgAAy4d8wnuulUmIhKVEkw9du7Qmou+sydvzV3F0x9plmURkaiUYCL4yeCeHNCzA9e8OIMV6yvTHY6IyA5BCSaCrCzjr6fsQ0VVNVc+V++MMyIighJMZLt3acPvh/Xh5enLePmTpekOR0SkyVOCaYCR3+5Fv27tuPy5Eko3VKU7HBGRJk0JpgFaZGdx0yn78OWGTVw7dka6wxERadKUYBpoQFF7fvntXjz54Re8NTd+XTMREamRyhUtm43fDevDk5MXcdb9k9iyxelekMcFw/syYlBRukMTEWkylGC2w7jpyyit3Lx1jrLFayu45JlPAJRkRERCukW2HW5+ZTabNn99rZiKqmpufmV2miISEWl6lGC2w5K1FQ3aLiKSiZRgtkP3grxatuemOBIRkaZLCWY7XDC8L3ktsrfZPrhXpzREIyLSNCnBbIcRg4q44aS9KSrIw4Ciglz26tqWF6ctZcaSdekOT0SkSVAvsu00YlDR13qMrS7byLH/eIvf/vcjXvjtt2jdUqdWRDKbrmAaSac2rbjt1H35fFU5V7+gUf4iIkowjWhI7878+vDdeWzSIl6YuiTd4YiIpJUSTCM7/+g9GNSjgEuf+UTLLItIRlOCaWQtsrP4548GAfC7x6ZQVb2lnneIiDRPSjBJsEvH1lx/0t5MWbiW28bPSXc4IiJpoQSTJN8f2J0fHrAzdxZ/xjufrkp3OCIiKacEk0RXHd+f3Trn84fHP2Z12cZ0hyMiklJKMEnUumUO//rxINZuqOKCp6bh7ukOSUQkZZRgkqx/9/Zc+t09mTBrBQ+8Mz/d4YiIpIwSTAqceeiuHLXXTtzw0iymLy5NdzgiIimR0gRjZh3NbIyZlZvZAjM7rY6y55vZMjMrNbP7zKxV1HrM7IdmNtPM1pvZDDMbkcTDqpeZcdMpA+mQ34Lf/XcK5Rs3pzMcEZGUSPUVzB3AJqAQOB24y8z6xxcys+HAxcAwYFegFzAqSj1mVgQ8DPwRaAdcADxqZjsl55Ci6ZjfklvDqWQOvG48u108liE3TuDZKYvTGZaISNKkLMGYWT5wMnC5u5e5+0TgeeCMBMXPBO519xJ3/xK4BjgrYj07A2vd/WUPjAXKgd2TeHiRrFi3kZwsY8OmapyvllpWkhGR5iiVU/7uAVS7e+zIw6nA4QnK9geeiytXaGadgB711DMZmGlmxwNjge8DG4Fp8R9iZiOBkQBdunShuLh4Ow4rumuKN7B5y9d7klVUVXPNc1MpKJ2b1M/eXmVlZUk/LzsinZfEdF62lcnnJJUJpg0Q38JdCrSNULbmedv66nH3ajN7EHgUyCW4lfYDdy+P/xB3Hw2MBujbt68PHTq0AYfTcGvGjU28vdJJ9mdvr+Li4iYbWzrpvCSm87KtTD4nqWyDKSNoE4nVDlgfoWzN8/X11WNmRwE3AUOBlgRXNveY2b7bH3rjqG2p5W5aallEmqHICcbMvmNmL4a9snYJt51jZsMiVjEHyDGzPjHbBgIlCcqWhPtiyy1399UR6tkXeNPdJ7v7FnefBLwPHBUxzqSpbanlPQsTXcSJiOzYIiUYMzsdeAKYC+wGtAh3ZQMXRqkjvEX1DHC1meWb2RDgBOChBMUfBM42s35m1gG4DHggYj2TgMNqrljMbBBwGAnaYFJt26WW8xiyeycmzF7JmClfpDs8EZFGFbUN5kLgF+7+mJmdE7P9PeDqBnzeucB9wApgNfBrdy8xsx7ADKCfuy9093FmdhPwOpAHPA1cWV89AO7+hpldBTxlZoXASuB6d/9fA+JMmvillquqt3DGve9z0dOf0KtzGwbuUpC+4EREGlHUBNMHeDfB9kTtIbVy9zXAiATbFxI03sduuwW4pSH1xOy/Hbg9alzp1CI7iztP35/jb5/IyIcm88J532KndmqTEZEdX9Q2mCUE3YzjfRv4rPHCyUwd81vyn58ewPrKzYx86EMqq6rTHZKIyDcWNcGMBv4ZtncA7GJmZxL01rorKZFlmL26teOWHw7k40Vr+cuY6Zp5WUR2eJFukbn7TWbWHniVYGzJ6wSDF//m7nckMb6McuyAbvx+WB/+8dpc9urWlnMO65XukEREtlvkgZbu/hczuw7oR3DlM8Pdy5IWWYb6/bA+zF62nutfmskehW359h5d0h2SiMh2idpN+T4za+vuG8LxJR+4e1nYTfi+ZAeZSbKyjL//cCB7FLblvEc/Yt6qbSYgEBHZIURtgzmToLtwvDzgp40XjgDkt8rhPz89gOws4xcPTmZ9ZVW6QxIRabA6E0y47konwIAO4euaRxfgOGB5KgLNNLt0bM2dp+/P/FXl/OGxj6neokZ/Edmx1HcFs4pgMKMTDIRcGfNYBtwD3JnMADPZIbt34srv9+O1WSv4+/9mpzscEZEGqa+R/wiCq5cJBGuwrInZtwlY4O5LkhSbAD85uCczlq7nzuLP+O8HC1m7oYruBXlcMLzv12YEEBFpaupMMO7+BoCZ7QYscvctKYlKtjIz9u9RwOOTFvLlhqAtpmahMkBJRkSarKjjYBYAmFl3ggW/Wsbtf7PxQ5Mat46fS3wTTEVVNTe/MlsJRkSarEgJJkwsjxJMDeMEt81if+VtOwe9NJolaysatF1EpCmI2k35NqCaYJDlBoLp738AzASOTUpkslVtC5V1ba9JMUWk6YqaYA4HLnL3WQRXLivd/RngIuCaZAUngdoWKmuX14JNm9UsJiJNU9QEk0fQZRmCnmQ7hc9nAPs0dlDydYkWKvvRgbswe9l6/vzkVLZojIyINEFR5yKbBewJzAc+Bn5lZouA3wCLkxKZfE38QmUAPTvl89dxs+jQugVXHd8fM0tTdCIi24qaYP4BdA2fXw2MA35MMKPymUmISyL41eG9WFO+kf+8NY9ObVrxu2F90h2SiMhWUbspPxLz/CMz25Xgimahu6+q9Y2SVGbGJd/Zi9Xlm7jl1Tl0yG/JGQf3THdYIiJA9DaYrwlnVf4IKDezixs5JmmArCzjryfvw7A9d+KK56YzdtrSdIckIgJESDBm1tnMvmdmx5hZdrithZn9gaBN5s/JDVHq0yI7i9tP248DenbgD49PYeJcXVSKSPrVN5vyocBc4AXgZeBtM9sTmAacR9BFuUeyg5T65bXM5p4zD2T3Lm0Y+dBkpi5am+6QRCTD1XcFcw3wCkFX5H8ABwEvAjcAfdz9dnffkNwQJar2eS148OcH0TG/JWfd/wGfrtCCoyKSPvUlmIHANe4+HbiMYJDlJe7+oLtr8EUTtFO7XB4+ezDZWcZP732fpaWaTkZE0qO+XmQdCdZ+wd03mNkGYErSo5JvZNfO+Tzws4P40ej3OOH2t8nOMpaVVmqafxFJqSi9yDrErGzpQLu4lS07JjlG2Q4Ditpz5qE9WbF+I0tLK3G+mub/2SkaGysiyRclwdSsZLkCaANM4qtVLVeFP6UJenbKtmvB1UzzLyKSbFFWtJQdlKb5F5F0irSipeyYuhfksThBMtE0/yKSCts1kl92DLVN85+bk8X6yqo0RCQimSSlCSbsFDDGzMrNbIGZnVZH2fPNbJmZlZrZfWbWKmo9ZtbazO40s1Xh+zNySedE0/yfeUhPFn1ZwU/u/YDSDUoyIpI8UWdTbix3AJuAQmBfYKyZTXX3kthCZjYcuBg4ElgCjAFGhdui1DOa4Nj2Ili/Zt+kHVETl2ia/yG9O/ObRz/itHve4+GzB9Mhv2WaohOR5ixlVzBmlg+cDFzu7mXuPhF4HjgjQfEzgXvdvcTdvySYUeCsKPWYWV/geGCku69092p3/zDJh7dDOaZ/V0b/9ADmrijjx/95j1VlG9Mdkog0Q6m8gtkDqHb3OTHbphIsxxyvP/BcXLnCcCxOj3rqGQwsAEaZ2RnAUuAqd386/kPMbCQwEqBLly4UFxdvz3HtkAz4/b4t+cdH6zn+1te48MBcCnK3/XujrKwso85LVDoviem8bCuTz0mkBGNm99Wyy4FK4FPgcXffduDFV9oApXHbSoG2EcrWPG8boZ6dgQHA00B34BCCW2gz3H3m14J3H01wO42+ffv60KFD6wi/+RkK7L/fan7+wCT+MT2LR38xmG7t875Wpri4mEw7L1HovCSm87KtTD4nUW+RdQFOAkYAvcPHiHBbX+BCYLaZ7VtHHWVAu7ht7YD1EcrWPF8foZ4KoAq41t03hV2tXweOqSO2jHVwr048dPZBrFq/kR/e/S6L1mjuUhFpHFETzNsE0/Xv7O7fdvdvE1wpvAT8D+gJjAX+Xkcdc4AcM4td13cgUJKgbEm4L7bccndfHaGeaRGPSUL79+zIw+cMpnRDFafe/S7zV5WnOyQRaQaiJpjfA1fHTs0fPr8OON/dNwF/pY7eWu5eDjwDXG1m+WY2BDgBeChB8QeBs82sn5l1IJjJ+YGI9bwJLAQuMbOccP9QgmUHpBYDdyngvyMPpqKqmlNHv6up/kXkG4uaYNoA3RJs7xruA1hH/W065wJ5BPOa/Rf4tbuXmFkPMyszsx4A7j4OuIng1taC8HFlffWE760iSDjfJWib+Q/wU3efFfFYM1b/7u15bOQhVG9xRtwxkYOuG89Z48oZcuMETZApIg0WtRfZGOBeM7uQYLJLJ1h87CaCqwnC13MSvz3g7msI2m7ity/kq0RVs+0W4JaG1BOzv4SgcV8aqG/XtvzisF7c8PIsyjZWA1/Nwgxoqn8RiSzqFcyvCG4xPQx8BnwePh9HcDUBMBP4RWMHKKn34LsLttmmWZhFpKEiXcGE7S2/MrM/AbsTDKP4NGwPqSnzcVIilJTTLMwi0hgaNNAyTCjqpdXM1TYLc5vcHNwdM0tDVCKyo4l0i8zMcs3sIjP7n5l9bGbTYh/JDlJSK9EszNlmrK/czJ+fnMamzVvSFJmI7EiiXsHcCZwIPAm8Q9DIL81UTUP+za/MZvHaCooK8vjzMXuwcE0Ft46fw5K1Ffz7jP1pn9cizZGKSFMWNcGMAH7g7uOTGIs0ITWzMMdPc7FLxzwuenoap9z1DveddSC7dGydviBFpEmL2otsA7AomYHIjuGk/XbmwZ8PZvm6Sk688x2mfbE23SGJSBMVNcHcBPzRzLQCpnDI7p145txDyW2Rxal3v8erM5anOyQRaYKiJoyjgVOBeWb2spk9H/tIYnzSRPXeqS1jzh3CHoVtGPnQZO5/e166QxKRJiZqgllFMJp/ArAMWB33kAzUpW0rHht5CEfvVcioF2Yw6oUSqreo/4eIBKIOtPxZsgORHVNey2zu+sn+XDd2Jve9PY9J89awpnwTS0sr6V6QxwXD+2p6GZEMlcoVLaWZys4yrvh+P9Zu2MgzU75ac05zmIlktloTTDiA8nB3/9LMPqGOsS/uvk8ygpMdy/vzvtxmW80cZkowIpmnriuYp4GN4fOnUhCL7OA0h5mIxKo1wbj7qETPRWpT2xxmuS2yqdhUTV7L7ATvEpHmSuNapNEkmsMsJ8uoqKrmxDvfZp6WYhbJKFEnu+xoZneZ2RwzW2tm62IfyQ5SdgwjBhVxw0l7U1SQhwFFBXn87QcD+b+fH8TydZUc/6+JjJu+NN1hikiKRO1Fdi8wCBgNLEGTXUotauYwi/fi7w7j3Ec+4lcPf8TIb/fiwuF9ycnWBbRIcxY1wQwDjnb395MZjDRfRQV5PPHLg7n2xZmMfvNzPl60ltt/PIid2uWmOzQRSZKof0KuAMqSGYg0f61ysrlmxABuO3VfPvmilO/9ayLvf66JIESaq6gJ5i/A1WbWJpnBSGYYMaiIZ38zhLatcjjtnvcZ/eZnuOuuq0hzE/UW2WXArsAKM1sAVMXu1EBLaai+Xdvy3HlDuPCpaVz/0ixenLqElWWbWKYpZkSajagJRgMtpdG1zW3Bnafvx/mPf8yzH2uKGZHmpt4EY2YtgHzgDndfkPyQJJOYGZPma4oZkeao3jYYd68Cfg1Y8sORTKQpZkSap6iN/P8DjkxmIJK5uhfk1brvicmL1AFAZAcVNcG8BlxvZreZ2RlmdlLsI5kBSvOXaIqZVjlZ9Oqcz4VPTeMXD37IyvUba3m3iDRVURv5bw9//i7BPgc0i6Fst5p2lptfmc2StRVbe5EdP7A79709j5temc2xt73JdSfuzbEDuqY5WhGJKuqKlprTQ5KqtilmzjmsF4fv0YXzn/iYXz38ISfvtzNXHt+Pdrkt0hCliDSEEoc0eX0K2zLm3CH87sjePPvxYo699U3e+XRVusMSkXpETjDhjMqnmdnFZnZF7KOBdYwxs3IzW2Bmp9VR9nwzW2ZmpWZ2n5m1amg9ZnalmbmZHRU1RmmaWmRn8cdj+vLUrw4ht0U2p93zPqNeKOHJyYsYcuMEdrt4LENunMCzUxanO1QRCUW6RWZmBwNjCVa47AIsBrqFr+cDV0f8vDuATUAhsC8w1symuntJ3OcNBy4m6Lm2BBgDjAq3RarHzHYHTgE0P3wzMqhHB8b+7jBufHkm9789H+Orqb01QFOkaYl6BXMz8AhQBFQS/OLvAUwG/hqlAjPLB04GLnf3MnefCDwPnJGg+JnAve5e4u5fAtcAZzWwntuBiwgSkTQjeS2zGXXCADrlt9xm3YiaAZoikn5Re5HtA5zt7m5m1UArd//czC4CHiVIPvXZA6h29zkx26YChyco2x94Lq5coZl1IkhsddZjZj8ANrn7S2a1jw81s5HASIAuXbpQXFwc4TAyS1lZWZM9L6vLE//tsHhtRdJjbsrnJZ10XraVyeckaoKJ/Z+8HOgJzCSYwr97xDraAKVx20qBthHK1jxvW1894YzP1wPH1BeQu48mWESNvn37+tChQ+t7S8YpLi6mqZ6XovcmsDjBaP+8Flnstd/BFCZxrZmmfF7SSedlW5l8TqLeIvsIODB8Xgxca2ZnAv8EpkWsowxoF7etHbA+Qtma5+sj1DMKeMjd50WMS3ZQiQZo5mQZmzZvYdjf3+C+ifPYXL0lTdGJSEPWg6mZ7vYyYCXwL6AD4S2mCOYAOWbWJ2bbQKAkQdmScF9sueXuvjpCPcOA34U90JYBuwBPhLfzpBkZMaiIG07am6KCPIxg1cy//WAgE/48lP17duDqF2dwwh1vM2XhtpNpikjyRR1oOTnm+UrgOw39IHcvN7NnCBYuO4eg99cJwKEJij8IPGBmjxD0ArsMeCBiPcOA2FF4k4A/Ai83NGZp+moboPnAzw7k5enLGPVCCSfd9Q4/PqgHFw3fk/atNUBTJFUaNNDSzA4ws1PDnlyYWb6ZRW3HATgXyCNYgvm/wK/dvcTMephZmZn1AHD3ccBNwOvAgvBxZX31hO9d7e7Lah5ANfClu2vJ5wxiZnx372689qeh/HzIbjz2wUKO/HsxT3/4hSbPFEmRqONgCgm6Ah9IMOygD/A5cAtBt+XfR6nH3dcAIxJsX0jQeB+77Zaw/sj11FJ21yjlpHlq0yqHy4/rx0n7FXHZs9P505NTefLDRRzRtwsPvrvwa3OfaeyMSOOKevVxK7AM6AQsjNn+JEFbjEiT1r97e57+1aE8NmkR17xYwnufr9m6TwM0RZIj6i2yYcBfwkGPsT4jGJci0uRlZRmnDe5B+9Ytt9mnAZoijS9qgskj8Yj4LgS3yER2GMtLE39lF6+tUPuMSCOKmmDeJJyqJeRmlk0wFctrjR2USDLVtYLmGfd+wPTF8eN4RWR7RE0wFwK/MLNXgVbA34EZwBDgkiTFJpIUiQZo5rbI4qRB3SlZUspx/5rI+Y9/zBdfbkhThCLNQ9RxMDPMbG/g1wQzKOcSNPDf4e6arVh2KLWtoDliUBHrKqv4d/Fn3DtxHmOnLeWsIbvym6G9NX5GZDtEHsMSjimJHYuCmfU0syfc/YeNHplIEtU2QLNdbgsuPHZPzjikJ7f8bw7/eetzHp+0iPOO6M0Zh/Rk3PRl3PzKbBavraDovQnq3ixSh4YMkkykgGDqfJFmpVv7PG7+wUB+/q3d+Ou4WVz30kzueP1Tyjdtpqo66Aig7s0iddOSySJ12KtbOx742UE8cs7gryWXGureLFI7JRiRCIb07szm6sRdmJckWDJARJRgRCKrrXtzdpbx0idL2bJFY2hEYtXZBmNmz9fz/vh1WUSarQuG9+WSZz6hoqp667YW2UZBXgvOfeQjeu/UhvOO6M1x+3QjJ1t/u4nU18i/OsJ+LewlGSG2e/PitRUUhd2bvz+wO2M/WcrtE+byh8c/5rbxczj3iN6cOKiIFko0ksHqTDDu/rNUBSKyI6jp3hy/DO7xA7tz3N7d+N+M5fxrwlwufGoa/xg/l18P3Z0fHLAzL3+yLOG4G5Hm7Jt2UxaRUFaWceyArgzvX8jrs1fwz9c+5bJnp3PTuFlUVFWre7NkHF2/izQyM+PIPQsZc+6hPHz2YCqrtqh7s2QkJRiRJDEzvtWnM1XVWxLuV/dmae6UYESSrLbuzQ6cfs97jJ+xXF2cpVlSghFJstpmbz5un658vrKccx6czBF/L+a+ifNYX1mVpihFGp8a+UWSrK7Zm6uqt/BKyTLumziPq1+cwS2vzuEHB+zMWYfuSs9O+Tw7ZbF6n8kOSwlGJAVqm725RXYWx+3TneP26c7URWu5/+15PPzeAh54Zz79urVl7vJyNoVtOOp9Jjsa3SITaSIG7lLAbT8axNsXHclvj+jNzKXrtyaXGup9JjsSJRiRJmandrn88Zi+eC3t/kvWVuC17RRpQpRgRJqounqfHfG3Yu54/VOWr6tMbVAiDaAEI9JE1db77PTBu1DYLpebX5nNoTdO4OwHJvFKybJax9uIpIsa+UWaqLp6nwHMX1XOE5MX8dSHX/DarBV0btOKk/cr4ocH7sInX5Sq95mknRKMSBNWW+8zgF0753PhsXvyx6P3oHj2Sh6fvIh7Js7j7jc/J8ugZuymep9JuugWmcgOLic7i6P6FfKfnx7Au5ccSbvcHOInBqioquamcbPSE6BkLCUYkWZkp7a5rK/cnHDfktJK/vTEVIpnr1B7jaREShOMmXU0szFmVm5mC8zstDrKnm9my8ys1MzuM7NWUeoxs4PN7FUzW2NmK83sSTPrluxjE2kqaut91rplNv+bsYyz7p/EQdeN59Ixn/De56upjrnceXbKYobcOIHdLh7LkBsn8OyUxakKW5qhVLfB3AFsAgqBfYGxZjbV3UtiC5nZcOBi4EhgCTAGGBVuq6+eDsBo4BVgM3A7cD9wbDIPTKSpSLS0c16LbK4/cW++s3dX3pyzihemLmHMR4t59P2FFLZrxff27k67vBzufuMzKqo0c4A0jpQlGDPLB04GBrh7GTDRzJ4HzuCrxFHjTODemsRjZtcAjwAX11ePu78c97m3A28k8dBEmpT6ep8d3a+Qo/sVsmHTZl6buYIXpi7h4fcWbDNrAHw1c4ASjGwPS9WIYDMbBLzj7nkx2/4MHO7u348rOxW43t0fD193BlYCnYEeUesJ9/0B+JG7H5xg30hgJECXLl32f+KJJ77xcTY3ZWVltGnTJt1hNDnN7byUVzm/eW1DrfvvOaY1OVlWbz3N7bw0huZ+To444ogP3f2ARPtSeYusDVAat60UaBuhbM3ztg2px8z2Aa4ATkgUkLuPJridRt++fT12jXUJxK89L4HmeF6u/3ACi2tZBO0Pb2zi8L5dOKZfIUP32In2rVskLNccz8s3lcnnJJUJpgxoF7etHbA+Qtma5+uj1mNmvYGXgd+7+1vbGbNIxkjUdpPbIoufHNyT8o2beXXGCsZOW0pOlnHQbh05ul8hR+1VyC4dW29dVmDx2gqK3puggZ0CpDbBzAFyzKyPu88Ntw0EShKULQn3PRFTbrm7rzazyvrqMbOewHjgGnd/KAnHItLs1Nd2c90I5+Mv1jJ+xnJenbGcUS/MYNQLM+jWrhUryzaxOeyNps4BUiNlCcbdy83sGeBqMzuHoPfXCcChCYo/CDxgZo8AS4HLgAei1GNmRcAE4A53/3cyj0mkualr5oCsLGO/Hh3Yr0cHLjx2T+avKmf8zOXcNG721uRSo6KqmhtfnqUEk+FSPdDyXCAPWAH8F/i1u5eYWQ8zKzOzHgDuPg64CXgdWBA+rqyvnnDfOUAv4MqwzjIzK0vBsYlklF0753POYb1qHbS5bF0lx972Jje8NJOJc1dRGXPrrYbG3TRvKR0H4+5rgBEJti8kaLyP3XYLcEtD6gn3jSIYMyMiKdC9IC9h54B2uTl0zG/J/W/P5+43P6dVThaDe3Xi2306c1ifLsxYUsqlY6ZvbfPRrbXmR5Ndisg3UtvAzqtPGMCIQUVs2LSZ9+et4a05q3hz7kquHTsTmPm1CTlraNxN86IEIyLfSGzngMVrKyiK6xzQumUOR/TdiSP67gQEK3JOnLuKC5+elrC+xWsrttYjOzYlGBH5xmo6B0QZ89G9II8fHrgL/3htbq3jbobcOIGigjwG9+rI4N06Mni3TvTs1BqzYLBnTbdorXfTtCnBiEhaJL61lsV5R/Yhv2U2789bwxuzV/LMR0HDf2G7Vhy0Wyda5WTxwtQlbNysOdOaOiUYEUmL+sbdnDVkN9ydz1aW8f68Nbz/+Rren7ea5es2blNXRVU1fx2nbtFNjRKMiKRNXeNuAMyM3ju1pfdObTl9cE/cnV6XvESiGRSXllYy/NY3GdSjgH13KWBQjw703qkN2TFzqOnWWmopwYjIDsPMau0W3TY3h24Fubw8fRmPTVoEQJtWOeyzc3sG9ShgY9UWHn5/AZVajiBllGBEZIdSW7foa8Ju0e7OvFXlfLxoLVMWrmXKoi/59xuff21htRoVVdXcqFtrSaMEIyI7lPrabsyMXl3a0KtLG07ab2cAKjZV0++KcQlvrS0rreTg61+jf/d29C9qz4DwZ/f2ueq19g0pwYjIDqe+tpt4eS2za7211j4vh0N278T0xaW8PnvF1sGfHVq3YEBRe1pkG2/NXUVVtSbzbCglGBHJCLXdWht1/ICtiaJiUzUzl62jZHEp0xevo2Rp8DNeRVU1Vzw3nQ75Lelb2JbCdq22Xu3U0BIGSjAikiHqu7UGwZVOzYzRNXa7eGzCW2vrKjdz5n0fANA+rwV9C9uyR9c29C1sy/J1ldwzcV7GdyhQghGRjNHQW2tQ+2Se3drncuup+zJn+XpmLVvPnGXree7jJayv3Jywnoqqaq4dO4ODe3VKeMVTozm19yjBiIjUobZbaxcduycH9+rEwb06bd3u7ixbV8khN0xIWNeqsk0cfMNr5LfMpleXNuzeJT/82YZeXfKZvriUK54raTYzTCvBiIjUIcqttRpmRrf2eRTVctXTuU1Lfj+sD5+tLOezlWVMmv8lz368pM7Pr6iq5vqXZvKdvbvSKie71nJN8cpHCUZEpB4NvbVW21XPZd/rt009GzZtZt6qcj5fWc5v/zslYX0r1m9kz8vH0b19Hj07taZnp9b06Jgf/mzN9MVrGfXCzCZ35aMEIyLSyOpbwiBW65Y59O/env7d23Pjy7MSXvl0aN2CMw7ZlYWry1mwZgP/K1nO6vJNdcZQUVXNNS/OYO+d21NUkEdui8RXP8m88lGCERFJgoYsYVCjtiufK7/ff5tf+usrq1iwegML12zg3Ec+Sljf6vJNDPv7GwB0btOKog557Nwhj50L8ijqkMeiNRt48N0F2zUzdU1iatm19/61lVGCERFpIhrS3tM2NxgIOqCofZ1tPpd+dy8Wfxks4vbFlxXMWLKOV2csZ1OYVOJVVFVz6ZhPWLB6A93a59K1fe7Wn21zWwBBcolPhIkowYiINCHb05W6IW0+AFu2OKvKNjL4+tcSjvHZsKmaW8fP2WZ7m1Y5dG2fy6I1G7Ze9dRFCUZEZAfXkCsfgKwsY6d2ubWO8SkqyGPCnw9nxbqNLFtXydLSSpaVVoQ/K/l0RVmkuJRgRESagca88rlgeF9a5WSzS8fW7NKx9TbvG3LjhFqXu46V1aBoRESk2RgxqIgbTtqbooI8jODK5YaT9q43UV0wvC95tfRKi6UrGBGRDLY9Vz6xt+SW1lFOVzAiItJgIwYV8fbFR7Jp2acf1lZGCUZERJJCCUZERJJCCUZERJJCCUZERJJCCUZERJIipQnGzDqa2RgzKzezBWZ2Wh1lzzezZWZWamb3mVmrqPWY2TAzm2VmG8zsdTPrmczjEhGRbaX6CuYOYBNQCJwO3GVm/eMLmdlw4GJgGLAr0AsYFaUeM+sMPANcDnQEJgOPJ+dwRESkNilLMGaWD5wMXO7uZe4+EXgeOCNB8TOBe929xN2/BK4BzopYz0lAibs/6e6VwFXAQDPbM3lHJyIi8VI5kn8PoNrdY6fonAocnqBsf+C5uHKFZtYJ6FFPPf3D1wC4e7mZfRZunxX7IWY2EhgZvtxoZtMbfFTNX2dgVbqDaIJ0XhLTedlWcz8ntTZBpDLBtAFK47aVAm0jlK153jZCPW2AlVE+x91HA6MBzGyyux9Q9yFkHp2XxHReEtN52VYmn5NUtsGUAe3itrUD1kcoW/N8fYR6GvI5IiKSJKlMMHOAHDPrE7NtIFCSoGxJuC+23HJ3Xx2hnq+9N2yz2b2WzxERkSRJWYJx93KC3l1Xm1m+mQ0BTgAeSlD8QeBsM+tnZh2Ay4AHItYzBhhgZiebWS5wBTDN3WfFf0ic0d/sCJstnZfEdF4S03nZVsaeE3NPtGBmkj7MrCNwH3A0sBq42N0fNbMewAygn7svDMv+EbgIyAOeBn7l7hvrqifmc44CbidofHofOMvd56fkIEVEBEhxghERkcyhqWJERCQplGBERCQpMj7BNGR+tExiZsVmVmlmZeFjdrpjSgczO8/MJpvZRjN7IG5fRs55V9s5MbNdzcxjvjNlZnZ5GkNNKTNrZWb3hr9H1pvZFDP7Tsz+jPu+ZHyCIeL8aBnqPHdvEz76pjuYNFkCXEvQqWSrDJ/zLuE5iVEQ8725JoVxpVsOsIhgVpH2BN+NJ8LEm5Hfl1SO5G9yYuY1G+DuZcBEM6uZ1+zitAYnTYK7PwNgZgcAO8fs2jrnXbj/KmCVme0ZoUv8Dq2Oc5LRwiEUV8VsetHM5gH7A53IwO9Lpl/B1DY/mq5gAjeY2Soze9vMhqY7mCZmmznvgJo57zLdAjP7wszuD/9yz0hmVkjwO6aEDP2+ZHqCacj8aJnmIoJlEooIBoq9YGa7pzekJkXfnW2tAg4kGH+2P8G5eCStEaWJmbUgOPb/C69QMvL7kukJRvOW1cLd33f39e6+0d3/D3gb+G6642pC9N2JEy6fMdndN7v7cuA84Bgziz9PzZqZZRHMLLKJ4BxAhn5fMj3BNGR+tEzngKU7iCZEc97Vr2YUd8Z8b8zMgHsJOg2d7O5V4a6M/L5kdIJp4PxoGcPMCsxsuJnlmlmOmZ0OfBt4Jd2xpVp4/LlANpBdc07Y/jnvdni1nRMzG2xmfc0sK1y76Z9AsbvH3xpqzu4C9gK+7+4VMdsz8/vi7hn9IOgy+CxQDiwETkt3TOl+AF2ASQSX72uB94Cj0x1Xms7FVQR/icc+rgr3HUWwiF0FUAzsmu5403lOgB8D88L/S0sJJq3tmu54U3heeobnopLglljN4/RM/b5oLjIREUmKjL5FJiIiyaMEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEI9JMhWuznJLuOCRzKcGIJIGZPRD+go9/vJfu2ERSJaPXgxFJsvEEawvF2pSOQETSQVcwIsmz0d2XxT3WwNbbV+eZ2dhwCd0FZvaT2Deb2d5mNt7MKsxsTXhV1D6uzJlm9km4fPHy+GWdgY5m9mS4JPjn8Z8hkkxKMCLpMwp4HtiXYM2dB8NVIjGz1sA4grmsDgJOBA4lZpliM/slcDdwP7APwXIK8bPzXgE8RzCT7+PAfZmwFrw0DZqLTCQJwiuJnxBMfBjrDne/yMwcuMfdfxHznvHAMnf/iZn9AvgbsLO7rw/3DwVeB/q4+6dm9gXwsLsnXN47/Iwb3f2S8HUOsA4Y6e4PN97RiiSmNhiR5HkTGBm3bW3M83fj9r0LfC98vhfBdO6xC1K9A2wB+pnZOoLVRl+rJ4ZpNU/cfbOZrQR2ihS9yDekBCOSPBvc/dPtfK/x1YJd8Rqy+FtV3GtHt8YlRfRFE0mfgxO8nhk+nwEMNLPYNdsPJfg/O9ODJYkXA8OSHqXIdtIVjEjytDKzrnHbqt19Zfj8JDObRLD41CkEyWJwuO8Rgk4AD5rZFUAHggb9Z2Kuiq4DbjWz5cBYoDUwzN3/nqwDEmkIJRiR5DmKYGXHWIuBncPnVwEnEywtvBL4mbtPAnD3DWY2HLgN+ICgs8BzwO9rKnL3u8xsE/An4K/AGuClJB2LSIOpF5lIGoQ9vH7g7k+lOxaRZFEbjIiIJIUSjIiIJIVukYmISFLoCkZERJJCCUZERJJCCUZERJJCCUZERJJCCUZERJLi/wGhnHaPiXVH9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Exponential Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The schedule function can take the current learning rate as a second argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay_fn(epoch, lr):\n",
    "    return lr * 0.1**(1 / 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to update the learning rate at each iteration rather than at each epoch, you must write your own callback class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 1.1153 - accuracy: 0.7390 - val_loss: 0.9588 - val_accuracy: 0.7338\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6929 - accuracy: 0.7934 - val_loss: 0.5328 - val_accuracy: 0.8318\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6317 - accuracy: 0.8097 - val_loss: 0.7656 - val_accuracy: 0.8278\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5827 - accuracy: 0.8258 - val_loss: 0.5585 - val_accuracy: 0.8382\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5041 - accuracy: 0.8407 - val_loss: 0.5367 - val_accuracy: 0.8574\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.4595 - accuracy: 0.8588 - val_loss: 0.6000 - val_accuracy: 0.8516\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4490 - accuracy: 0.8644 - val_loss: 0.4605 - val_accuracy: 0.8648\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3925 - accuracy: 0.8783 - val_loss: 0.5076 - val_accuracy: 0.8616\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4085 - accuracy: 0.8797 - val_loss: 0.4577 - val_accuracy: 0.8650\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3440 - accuracy: 0.8927 - val_loss: 0.5309 - val_accuracy: 0.8762\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3267 - accuracy: 0.8948 - val_loss: 0.4652 - val_accuracy: 0.8792\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3046 - accuracy: 0.9033 - val_loss: 0.4863 - val_accuracy: 0.8692\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2811 - accuracy: 0.9087 - val_loss: 0.4726 - val_accuracy: 0.8770\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2684 - accuracy: 0.9145 - val_loss: 0.4526 - val_accuracy: 0.8760\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2478 - accuracy: 0.9209 - val_loss: 0.4926 - val_accuracy: 0.8838\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2315 - accuracy: 0.9253 - val_loss: 0.4686 - val_accuracy: 0.8840\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2164 - accuracy: 0.9318 - val_loss: 0.4845 - val_accuracy: 0.8858\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2093 - accuracy: 0.9346 - val_loss: 0.4923 - val_accuracy: 0.8834\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1929 - accuracy: 0.9396 - val_loss: 0.4779 - val_accuracy: 0.8880\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1852 - accuracy: 0.9439 - val_loss: 0.4886 - val_accuracy: 0.8868\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1740 - accuracy: 0.9470 - val_loss: 0.5097 - val_accuracy: 0.8852\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1668 - accuracy: 0.9474 - val_loss: 0.5161 - val_accuracy: 0.8898\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1571 - accuracy: 0.9530 - val_loss: 0.5381 - val_accuracy: 0.8886\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1444 - accuracy: 0.9575 - val_loss: 0.5415 - val_accuracy: 0.8910\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1447 - accuracy: 0.9569 - val_loss: 0.5833 - val_accuracy: 0.8880\n"
     ]
    }
   ],
   "source": [
    "K = keras.backend\n",
    "\n",
    "class ExponentialDecay(keras.callbacks.Callback):\n",
    "    def __init__(self, s=40000):\n",
    "        super().__init__()\n",
    "        self.s = s\n",
    "\n",
    "    def on_batch_begin(self, batch, logs=None):\n",
    "        # Note: the `batch` argument is reset at each epoch\n",
    "        lr = K.get_value(self.model.optimizer.lr)\n",
    "        K.set_value(self.model.optimizer.lr, lr * 0.1**(1 / s))\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        logs = logs or {}\n",
    "        logs['lr'] = K.get_value(self.model.optimizer.lr)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "lr0 = 0.01\n",
    "optimizer = keras.optimizers.Nadam(lr=lr0)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "\n",
    "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
    "exp_decay = ExponentialDecay(s)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[exp_decay])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_steps = n_epochs * len(X_train) // 32\n",
    "steps = np.arange(n_steps)\n",
    "lrs = lr0 * 0.1**(steps / s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA/UElEQVR4nO3dd3wUdfrA8c+TQhLSICSE3puAFAEVUYroWU5FxXLqeaIi6qnncWe9s7ffnd07+x2KXVCk2JWToBTpNfTeEggEUkkI8Pz+mAku6ybZhOxuQp7367Wv7M585zvPfLO7z87Md74jqooxxhhT3cJCHYAxxpjjkyUYY4wxAWEJxhhjTEBYgjHGGBMQlmCMMcYEhCUYY4wxAWEJxtR6IjJCRPIruUyaiLwcqJjcdWwSkbsCUO9lIlKp6wu826gqbXYsROQREXkrWOvzsX4VkctCsN4K21lEbheRKcGKKZgswdRiIjLW/eB4P34OdWyBUsYXxTigXQDWNVJEFolIvojkiMhSEXmiutcTIgFpM19EpDHwF6BWt52bJJcHoOr/AH1F5IwA1B1SEaEOwByzqcC1XtMOhCKQUFHV/cD+6qxTRG4A/gWMBv4H1AO6Af2rcz2hEog2K8dIYK6qbgj0ikQkUlVLAr2e6qSqxSLyIfAn4KdQx1OdbA+m9itW1UyvRzaAiAwSkRIRGVxaWERuEZFcEWnnvk4TkddF5CUR2es+nhGRMI9lGorIO+68/SIyVUS6ecwf4f7KHyoiy0WkQESmiUhbz0BF5EIRWSAiRSKyUUSeFJF6HvM3icgDIvKGG+M2Ebnbc7779BN3T2aT5/o9yrUXkckikunGslBELqhku14EfKaqb6jqOlVdoaqfqOpfvLbptyIyx22XPSLyuYhEexSJLmt73OUTReRNEdklInkiMl1E+nqV+YOIbBaRQhH5Akj1mv+rX9YVHZrx0WaPuP+734nIejeWSSKS7FEmQkRe8HifvCAir4lIWgVteTVw1CEgP9939UTkn267FYjIPBE5x2P+YPd9cL6IzBWRA8A5lK2JiHzptuNmEfm9V0z/EJHV7v9yk4g8Xfq/FJERwMNAN/nlSMEId16C2w4Z7nt7pYhc6VV3uZ8Nt30uEpH6FbRl7aKq9qilD2As8EUFZZ4CtgJJQBegALjOY34akAf8251/BZAD/MWjzGRgFTAQOBHnw7AViHHnjwBKcPamTgZ6AIuAbz3qOAfIBa4H2gNDgNXAsx5lNgF7gNuBDsAdgAL93fkp7uuRQBMgxWP9+R719ARucWPtAPwdZ6+ui9d2v1xOu70OrAHalVPmXOAgzqGfru523wXU93N7BJgBfOm2WwfgcbedmrplTgEOu9vQCbjZrVM94ngEWO4Vm3ebVPT6ESAfmOhuR39gM/CGR5n7gL3AcKAz8JL7Xkkrp42S3PhP85qeRsXvuw+An3Hed+3cdjwA9HTnD3bbcxnwG7dMShlxqNtuN7vt+Hc3rr4eZR4EBgBtgPOBLcDj7rwY4Fmcz0ET9xHj/g9nAivc90M74DzgEn8/G265+sAhYGiov1eq9Tsq1AHY4xj+eU6COeh+MXg+/ulRJhKYB3wGLATGedWRhvNFKh7THgC2uc87uh/OgR7zE90vg5Hu6xFumc4eZa5xvwzC3Nc/Ag96rftiN15xX28CPvIqsxZ4wOO1Apd5lRmBx5dlGW31s1c9aZSfYJoCs931rQXeB/4ARHqUmQl8XE4d5W4PcKa7/TFeZRYD97jPPwS+95r/XwKTYIqARI9pfwfWebzOAO7zeC04X7hp5bRBL7cN21byfdceJwG08lpuEvCq+3ywW/dwPz4rCvzHa9pU4P1ylrnFa/t9tfPZbpwnlFHHCCr4bHhMzwZurGhbatPDDpHVfj/ifIg9H8+UzlTnePTVwAVAY5xfcN5+Vvcd7poNNBeRBOAEnA/QbI86c3B+NXb1WKZYVVd7vN6Bk9wauK/7AH93D6Xlu4dnPgRicX4NllrqFdsON26/iUise3hjhXvoJR/oC7Tytw5VzVDV/jh7QS/ifJm+Acz1OIzRG+f8THnK254+OL9cs7zapTvOFyw47T/bqw7v19Vls/u//VWsIpKI83+aWzrTfc/Mq6DOGPdvkY955b3vTsJp8xVebfNbfmmbUvMriMGzfu/XR97D4vTOm+EeWs0HXqDi90xvIENVV5ZTpqLPRqn9/NJexwU7yV/7FarqugrKnIpzvq0BzmGmfZWoX8qZ5/nlcLCMeWEefx8FPvFRT5bHc+8TtErlzxU+i3O44i6cPYZC4F2cE/WVoqrLgeXAKyJyOs5J2Ctw9h79Ud72hAE7AV+9h3Ldv+W1f6nDPspF+hmfJ3/avrLDr+92/zbE2QPyV5i7rn4+4vLunFBQyZh+RUROBT7GeY+OxvmMXITzXip3UT+qr+izUSqJoz8LtZ7twRznRKQN8DJwG/A98IGIeP+wOEVEPD8opwI7VDUX59hyGB69p9xfmCe68/y1EOccyDofD+8PYHlKgPAKypwOvKuqE1R1KbCNX//qrYrS7Y1z/y4Chh5DfQtxTtgf9tEmuzzWearXct6vs4BUr/9hr2OI61fcPZtMnPMIALjr61fBoutxkmVXH/PKe98twvnybuKjbbZXcTN8tWPpnscAYLuqPq6q81R1LdDaq/wBfv3eWwg0FZETqhgT4HRMAaLd+o4btgdT+0WJSBOvaYdUNUtEwnHOHUxX1TdE5FOcQ1sP45zQLNUMeFFEXsVJHHfjXrOgqmtFZDLwhoiMwvll9yTOl8aHlYjzMeALEdkMjMf5VdcdOFlV76lEPZuAoSIyHefQw14fZdYAl7hxl+Bsb7SPcmUSkddwDmX8gJOgmuKcIygEvnOLPQl8LiLrcNpCcE42v6GqhX6sZirOeZzJInIPv5xAPheYqqo/4XSVniUi9wOf4px3uMSrnjScX79/E5GP3TKBuKjwJeAeEVmDk/huxmmXMvdMVPWwiEzFSfqfes0u7323RkQ+AMaKyF9xvniTcLZtg6p+VoX4LxWReTjtdRnOj4NT3HlrcA7PXYNz6Owc4Cqv5TcBrUXkJJwOAHk4h0jnABNEZLRbTwcgVlUnVSK2M3C2a23lN6vmsj2Y2u8snA+452ORO+9vOG/2GwFUdQ9wHXCfe7in1Ac4v8zm4Fz0NQbn+HOp63GOvU9x/9YHzlXnWgq/qOq3OMfPh7h1zMXplbTF/00F4K9uHVv5ZTu9/QXYhXM462ucE/yVvb7ge5wvn/E4XxoT3elnq+oaAFX9CufL/jw3lulubIf9WYF7/uF8nCT2H5xedeNxemjtcMv8jPP/uxXnfM6lOCebPetZ6c4f5ZY5G6f3YHV7FngPeBunTcFpF1/nVzy9CVzp/uDx5M/77m3gaZzk+wVOj7LNVYz/EZwecEtx2ut6VZ0HoKqf45y7fJFf2vAhr+UnAF/hJJUs4CpVPYzz/5+J82NuJU4iruzh2Ktw2uC4Utp7x9RR7jUMy1X19lDHYmofEVkIzFTVOyooNxun99d77us07H0HgIh0x0lanbw6WdR6dojMGOMXEWmNc+hoOs53xyica45G+bH4zTg9rsyvNQP+cLwlF7AEY4zx32Gca4GewTm8vgI4T1Ur7Cbsdrbw7rJtAFX9ruJStZMdIjPGGBMQdpLfGGNMQNghMleDBg20Q4cOoQ7Dp4KCAmJjY0Mdhk8WW9VYbFVjsVVNIGNbsGDBblVN8Tkz1GPV1JRHp06dtKaaNm1aqEMok8VWNRZb1VhsVRPI2ID5amORGWOMCSZLMMYYYwLCEowxxpiAsARjjDEmICzBGGOMCQhLMMYYYwLCEowxxpiAsARjjDEmICzBGGOMCQhLMMYYYwLCEowxxpiAsARjjDEmICzBGGOMCQhLMMYYYwLCEowxxpiACGqCEZEkEZkoIgUisllEri6n7GgRyRSRHBF5S0SiPObdLiLzRaRYRMb6WHaoiKwSkUIRmSYirQO0ScYYY8oQ7D2YV4ADQCpwDfCaiHTzLiQi5wD3AUOBNkA74FGPIjuAJ4C3fCybDHwGPAgkAfOBcRUFdlgrtyHGGGPKF7QEIyKxwHDgQVXNV9UZwBTgWh/FrwPGqGq6qu4FHgdGlM5U1c9UdRKwx8eylwLpqvqJqhYBjwA9RaRLefFtzTvMnA2+qjPGGFMV4tzxMggrEukNzFLVGI9pdwGDVPVCr7JLgKdUdZz7OhnIApJVdY9HuSeAFqo6wmPaS0A9Vb3VY9py4GFVneC1nlHAKIB6TTr06XfbSzxyWgwRYVJdm10t8vPziYuLC3UYPllsVWOxVY3FVjWBjG3IkCELVLWvr3kRAVmjb3FAjte0HCDej7Klz+PxvdfivWyWP+tR1TeBNwFim3XUbfnKpsjWjDyjXQWrCK60tDQGDx4c6jB8stiqxmKrGoutakIVWzDPweQDCV7TEoA8P8qWPvdV9ljWc0RSjLPX8uLUtezMLfJjNcYYY8oTzASzBogQkY4e03oC6T7KprvzPMvt9Dw8Vo6jlnXP/bQvYz1H1I8Qzu6aSn7xQZ78cqUfqzHGGFOeoCUYVS3A6d31mIjEisgAYBjwno/i7wI3ikhXEWkIPACMLZ0pIhEiEg2EA+EiEi0ipYf7JgLdRWS4W+YhYKmqrqooxocu6Ep0ZBhTluxg1vrdx7C1xhhjgt1N+Y9ADLAL+Ai4VVXTRaSViOSLSCsAVf0GeBqYBmx2Hw971PMAsB+nK/Pv3ecPuMtm4fRWexLYC5wC/M6f4Fom1ef2IR0AeGhyOiWHDh/TxhpjTF0WzJP8qGo2cLGP6VtwTs57TnseeL6Meh7B6X5c1nqmAuV2Sy7LTQPb8emCbazblc/bMzcyamD7qlRjjDF1ng0V4yUqIpxHh3UHnBP+GTn7QxyRMcbUTpZgfBjUKYXzujeh8MAhnvjCTvgbY0xVWIIpwwMXdCUmMpwvl2UwbdWuUIdjjDG1jiWYMjRvEMNfzu4EwAOTllN44GCIIzLGmNrFEkw5rh/Qhm7NEti+bz8vfL8m1OEYY0ytYgmmHBHhYfzj0h6ECYyZsZHl271HujHGGFMWSzAVOLFFIjcMaMthhfs+W8pBuzbGGGP8YgnGD6PP7kTzBjEs357L2FmbQh2OMcbUCpZg/BAbFcETFzvXxjz33Rq2ZheGOCJjjKn5LMH4aUiXxlzQoyn7Sw7xwKTlBOs+OsYYU1tZgqmEhy7sSkJ0BNPXZDFlyY5Qh2OMMTWaJZhKaBwfzd/OPwGAR6akszu/OMQRGWNMzWUJppKu7NeS0zsks7ewhIcmLw91OMYYU2NZgqkkEeH/Lj2R2HrhfLUsky+XZoQ6JGOMqZEswVRBy6T63O8eKnto8nL22KEyY4z5FUswVXT1ya3o364RewoO8PCUcu/GbIwxdZIlmCoKCxOevqwH9euF88XSDL5ZbofKjDHGkyWYY9AyqT73nefcOPOBScvZW3AgxBEZY0zNYQnmGP3+lNac0jaJ3fkHeORzO1RmjDGlLMEco9JDZdGRYUxevIOvltmhMmOMAUsw1aJ1o9gjF2D+beIyduUWhTgiY4wJPUsw1eTaU1szsFMK+wpLuPvTpTZWmTGmzrMEU01EhGcu60GD+pFMX5PF+3O2hDokY4wJKUsw1Sg1IZqnLjkRgCe/XMH6rPwQR2SMMaFjCaaanX9iUy7t3ZyiksP8ZdxiSuwOmMaYOsoSTAA8MqwbzRvEsGRbDi//sC7U4RhjTEhYggmAhOhInr28JyLw8rR1LNqyN9QhGWNM0FmCCZD+7Rsx8vS2HDqs/HncYvKKSkIdkjHGBJUlmAD6628607VpApv3FPLQZLvK3xhTt1iCCaDoyHD+dVVvYiLDmbhoOxMWbAt1SMYYEzRBTTAikiQiE0WkQEQ2i8jV5ZQdLSKZIpIjIm+JSJS/9YjIFSKyUkTyRGSFiFwcwM0qV4fGcTx6UTcAHpy8nA3WddkYU0cEew/mFeAAkApcA7wmIt28C4nIOcB9wFCgDdAOeNSfekSkOfA+8BcgAbgb+FBEGgdmkyp2ed8WXNizGYUHDnHHR4soPngoVKEYY0zQBC3BiEgsMBx4UFXzVXUGMAW41kfx64AxqpquqnuBx4ERftbTAtinql+r40ugAGgfwM0rl4jw5CXdaZkUQ/qOXP759epQhWKMMUEjwRozS0R6A7NUNcZj2l3AIFW90KvsEuApVR3nvk4GsoBkoFV59YhIOPAD8BzwJXAh8DLQWVULvNYzChgFkJKS0mf8+PHVvNVH27DvEE/OKeKQwp9PiqJX4wi/lsvPzycuLi6gsVWVxVY1FlvVWGxVE8jYhgwZskBV+/qcqapBeQBnAJle024C0nyUXQ+c6/E6ElCcw2UV1gPcCOQDB4FC4LcVxdepUycNhtfS1mnre7/Q3o99pxn79vu1zLRp0wIb1DGw2KrGYqsai61qAhkbMF/L+F4N5jmYfJxzIp4SgDw/ypY+z6uoHhE5C3gaGAzUAwYB/xWRXlUPvfqMOqMdZ3RMJrvgAHd8tNCGkjHGHLf8TjAicp6IfOH2ymrpThspIkP9rGINECEiHT2m9QR8XSCS7s7zLLdTVff4UU8v4EdVna+qh1V1HjAHOMvPOAMqLEx44cpepCZEMW/TXp751s7HGGOOT34lGBG5BhgPrAXa4hyyAggH7vGnDnXOf3wGPCYisSIyABgGvOej+LvAjSLSVUQaAg8AY/2sZx5wRukei3vu5wxgqT9xBkNyXBQvX30S4WHCmz9u4JvlmaEOyRhjqp2/ezD3ADep6mic8xqlfsbZY/DXH4EYYBfwEXCrqqaLSCsRyReRVgCq+g3OYa5pwGb38XBF9bjLTgceAT4VkTxgAk6Hge8qEWfA9WuTxP3ndQHg7k+WsGl3QQVLGGNM7eJfNyboCMz2Md3X+ZAyqWo2cLGP6VuAOK9pzwPPV6Yej/kv4/Qcq9FuPL0t8zZl8236Tm79YCET/3ga0ZHhoQ7LGGOqhb97MDuATj6mD8Tp8WWqQER45vKetGlUn5UZuTxs45UZY44j/iaYN4F/uec7AFqKyHU4h7FeC0hkdURCdCSvXtOHqIgwxs3fyvj5W0MdkjHGVAu/EoyqPo1zYv17IBbn3MjrwOuq+krgwqsbujZL4PGLuwPw4KTlLN+eE+KIjDHm2PndTVlV/45zJf3JwKlAiqo+GKjA6por+rbkd/1aUnzwMKPenc/u/OJQh2SMMcfE327Kb4lIvKoWuteXzFXVfLeb8FuBDrKueHRYN3q3asCOnCJu+8AuwjTG1G7+7sFch9Mt2FsM8IfqC6dui4oI5/Xf96FxfBRzNmbzxBcrQh2SMcZUWbkJxr3vSiNAgIbu69JHCnABsDMYgdYVqQnRvH5tH+qFh/HO7M2Mn2cn/Y0xtVNFezC7cS5mVGAFzojGpY9M4L/Aq4EMsC46qVVDHr/YuU3OA5OWs26f3T/GGFP7VHSh5RCcvZcfcO7Bku0x7wCwWVV3BCi2Ou3Kfq1I35HLu7M38/KiYi46s4jGCdGhDssYY/xWboJxh11BRNoCW1XVzjoH0YMXdGVVZh5zN2Zz8/sL+OimU+1Kf2NMreHvdTCbVfWwiDQTkVNFZKDnI9BB1lWR4WG8es1JJEULi7bs455Pl5be78YYY2o8v8YiE5FmwIc4Q8MozmEzz286+1kdIMlxUYzuE80/5h1gypIdtEuJ5c9n+Rq1xxhjahZ/uym/CBwCuuLcIfIM4HJgJXBuQCIzR7SMD+PfV/cmTODFqWuZvHh7qEMyxpgK+ZtgBgH3quoqnD2XLFX9DLgXeDxQwZlfnNkllQd+2xWAuz9dyoLN2RUsYYwxoeVvgonB6bIMTk+yxu7zFUCP6g7K+Hb9gDb8/tRWHDh4mFHvLmBrdmGoQzLGmDL5m2BWAV3c54uBW0SkNXAbYMdrgkREePjCbpzRMZk9BQe4Yew8cotKQh2WMcb45G+CeQlo4j5/DPgNsAHnzpJ/C0BcpgyR4WG8fPVJdGgcx9pd+dz2wUIOHLTe48aYmsffbsofqOpY9/lCoA3QD2ilqp8ELDrjU2JMJG9d149GsfX4ae1u7ptg3ZeNMTWP38P1e3JHVV4IFIjIfdUck/FDq0b1efv6ftSvF85ni7bzzLerQx2SMcYcpcIEIyLJIvJbEfmNiIS70yJF5M/AJuCuwIZoytKjRQNeueYkwsOEV9PW8+7sTaEOyRhjjqhoNOXTgLXA58DXwEwR6QIsBW7H6aLcKtBBmrIN6dyYf1x6IgAPT0nnm+WZIY7IGGMcFe3BPA58i9MV+SWcu1l+Afwf0FFVX1ZV6ysbYpf3bcldv+mEKvzp40XM22TXyBhjQq+iBNMTeFxVlwMP4Fxkeb+qvqt2VrlGuW1IB645xblGZuQ781m7My/UIRlj6riKEkwSzr1fcPdUCoFFgQ7KVJ6I8Niw7pzdNZWc/SVcO2auXYhpjAkpf3qRNfS4s6UCCV53tkwKcIzGT+Fhwr9+15uT2ySRmVvEtWPmsCuvKNRhGWPqKH8STOmdLHcBccA8frmr5W73r6khYuqF898RfenePIFNewr5w5i57Cs8EOqwjDF1kD93tDS1TEJ0JO9cfzJXvDGbVZl5jHh7Hh+MPIXYKL/uzmCMMdXCrztamtqnUVwU7488hctem83irfsY9d58xlzXz+6IaYwJmipdyW9qh6aJMXww8hSS46KYuW4Pd3y0iIOHbNwyY0xwBDXBuJ0CJopIgYhsFpGryyk7WkQyRSRHRN4SkSh/6xGR+iLyqojsdpf/MZDbVZO1SY7l/ZEnkxgTyfcrdvLXT5Zw6LD1MDfGBF6w92BeAQ4AqcA1wGsi0s27kIicA9wHDMUZWLMd8Ggl6nkTp4v1Ce7f0dW9IbVJlyYJvH19P2LrhTN58Q7utiRjjAmCoCUYEYkFhgMPqmq+qs4ApgDX+ih+HTBGVdNVdS/OiAIj/KlHRDoDFwGjVDVLVQ+p6oIAb16Nd1Krhoy94eQjg2PeO2Ephy3JGGMCSIJ1Qb6I9AZmqWqMx7S7gEGqeqFX2SXAU6o6zn2djNMdOhln7LMy6xGRPwB3A1Nxkk4G8IiqTvAR0yhgFEBKSkqf8ePHV+cmV5v8/Hzi4uKqpa7V2Yd4bkERBw7BwBYRjOhWjzCRGhFbdbPYqsZiq5q6GtuQIUMWqGpfX/P86rcqIm+VMUuBImAdME5Vd5RTTRyQ4zUtB4j3o2zp83g/6mkBdAcmAM2A/sCXIrJCVVceFbzqmziH0+jcubMOHjy4nPBDJy0tjeqKbTDQo+cerh87lx+3HaR5s2Y8eXF3wsKqlmSqM7bqZrFVjcVWNRbbr/l7iCwFuBS4GOjgPi52p3UG7gFWi0ivcurIBxK8piUAvgbN8i5b+jzPj3r2AyXAE6p6wO1qPQ3nLpwG6N++EWOu60dURBgfzd3CQ1OW2w3LjDHVzt8EMxNnuP4WqjpQVQfi7Cl8BXwHtAa+BJ4rp441QISIdPSY1hNI91E23Z3nWW6nqu7xo56lfm5TnTagQzL/+UNf6kWE8f7PW3hw8nI7J2OMqVb+Jpg7gcc8h+Z3nz8JjFbVA8A/gV5lVaCqBcBnwGMiEisiA4BhwHs+ir8L3CgiXUWkIc5IzmP9rOdHYAtwv4hEuPMH49x2wHgY2CmFN6/tcyTJ3DthqfUuM8ZUG38TTBzQ1Mf0Ju48gFwqPqfzRyAGZ1yzj4BbVTVdRFqJSL6ItAJQ1W+Ap3EObW12Hw9XVI+7bAlOwjkf59zMf4A/qOoqP7e1ThncuTFvXdePmMhwPlmwjT+PW0yJXYxpjKkG/g5ONREYIyL34Ax2qTg3H3saZ28C9/Wa8ipR1Wycczfe07fwS6IqnfY88Hxl6vGYn45zct/44fSOybxzw8ncMHYeny/ZQXHJIf59dW+iImxYGWNM1fm7B3MLziGm94H1wAb3+Tc4exMAK4GbqjtAExwnt03i/ZGnkBAdwXcrdjLq3QUUlRwKdVjGmFrMrwSjqoWqegvOVfG9gZOAJFW91T0ngqouVtXFAYvUBFyvlg34aNSpJMXWY/qaLK5/ex4FxQdDHZYxppaq1JX8qlqgqktVdUlpYjHHl27NEhk36lQax0cxe8MervnvHPYW2P1kjDGV51eCEZFoEblXRL4TkcUistTzEeggTXB1TI1n/M39ad4ghsVb93H5G7PZsW9/qMMyxtQy/u7BvIoz+OQmYBLOVfKeD3OcaZMcy4RbT6NTahzrduUz/LVZrNvl65pYY4zxzd9eZBcDl6vq1ADGYmqYJonRjL+5Pze+M58Fm/dy2euzeXtEP3q3ahjq0IwxtYC/ezCFwNZABmJqpgb16/H+jacwtEtj9hWWcPV/5pC2eleowzLG1AL+Jpingb+IiN0Bsw6KqRfO69f2YfhJLdhfcoiR78xn0qLtoQ7LGFPD+XuI7GzgDOBcEVmBM5jkEap6UXUHZmqWyPAwnr28B8lx9Xjjxw38edxitu/bT1dsaBljjG/+JpjdOFfzmzpMRLj//BNIiY/iya9W8sy3qxnYIoLTBx4mMtx2bo0xR/Mrwajq9YEOxNQeI89oR4uG9fnzuEX8uO0g1789j1d/fxIJ0ZGhDs0YU4PYz05TJed2b8LHo/qTUA9mrNvNZa/NYtvewooXNMbUGWUmGPciyobu82XeF1fahZamV8sGPHhqDB0ax7FmZz6XvDqLZdu8bzZqjKmryjtENgEodp9/GoRYTC2UUj+MCbecxi3vL2D2hj1c8cZsnr+iJ+ed6OvuDsaYuqTMBKOqj/p6boy3xPqRvHPDydz/2TImLNzGrR8s5M6hHblzaEfCwiTU4RljQsTOwZhqUS/C6cb8t/O7ECbw0v/W8scPFtpozMbUYf4OdpkkIq+JyBoR2SciuZ6PQAdpagcRYdTA9owZ0Y/46Ai+Sc9k+Guz2JptJ/+NqYv8vQ5mDM59YN4EdoBdXWfKNqRzYybdNoCb3pnPqsw8hr0yk1evOYlT2zUKdWjGmCDyN8EMBc5W1TmBDMYcP9qnxDHxtgH86aNFTF+Txe//O4eHLuzKtae2RsTOyxhTF/h7DmYXkB/IQMzxJzEmkrdG9GPUwHYcPKw8NDmd0eMWU3jAzssYUxf4m2D+DjwmInGBDMYcf8LDhL+dfwL/uqo39euFM2nxDi5+ZSbrs+z3ijHHO38TzAPAb4BdIrLSLrQ0lXVRz2ZMvm0A7VNiWbMzn2Evz+TrZRmhDssYE0D+noOxCy3NMeuYGs/k20/n3glL+XJpBrd+sJCbzmjLPed2scEyjTkOVZhgRCQSiAVeUdXNgQ/JHM/ioiJ4+are9GnVkKe+Wsl/ftrIkq05vHRVL5omxoQ6PGNMNarwZ6OqlgC3Atb1x1QLEeGG09vy8ahTSU2IYu6mbM5/6SemrtgZ6tCMMdXI3+MS3wFnBjIQU/f0bZPEl386g8GdU9hbWMLId+fzyJR0ig8eCnVoxphq4O85mP8BT4lID2ABUOA5U1U/q+7ATN2QHBfFW9f1Y8yMjTz97SrGztrE3I3Z/Pvq3rRPsU6LxtRm/iaYl92/f/IxT4Hw6gnH1EVhYcJNA9txSrsk7vhoESsycrnw3zN4bFh3hp/U3C7MNKaW8usQmaqGlfOw5GKqRY8WDfjijtMZ1qsZhQcOcdcnS7jz48XkFJaEOjRjTBVY31BTo8RHR/Lilb145rIexESGM2XJDs558UdmrN0d6tCMMZXkd4JxR1S+WkTuE5GHPB+VrGOiiBSIyGYRubqcsqNFJFNEckTkLRGJqmw9IvKwiKiInOVvjCb0RITL+7bkqzvPoHerBmTmFvH7MXN4ZEo6+w9YBwBjagt/h+s/FVgLPAs8DtyAM3zMXcBllVjfK8ABIBW4BnhNRLr5WN85wH04g2y2AdoBnjc9q7AeEWnvxmaXi9dSbZNj+eTm/tx9TmciwoSxszbx23//xJKt+0IdmjHGD/7uwTwDfAA0B4pwuiy3AuYD//SnAhGJBYYDD6pqvqrOAKYA1/oofh0wRlXTVXUvTlIbUcl6XgbuxUlEppaKCA/jtiEdmHTbADo2jmNDVgGXvjaLF75fQ8mhw6EOzxhTDlGt+NYuIpID9FPVNSKyD+ivqitFpB/woap29KOO3sAsVY3xmHYXMEhVL/QquwR4SlXHua+TgSwgGSexlVuPiFwO/F5Vh4nIJmCkqk71EdMoYBRASkpKn/Hjx1fYFqGQn59PXFzN7LIbzNgOHFImrD3Ad5sOokCr+DBuPLEerRN89zOxdqsai61q6mpsQ4YMWaCqfX3N87ebsudewE6gNbASZwj/Zn7WEQfkeE3LAeL9KFv6PL6ietwRn5/CGZyzXKr6Js5N1OjcubMOHjy4okVCIi0tDYvN8ZuhMHv9Hu7+dAlb9u7nsZ+LuXlgO/40tCPRkUcnGmu3qrHYqsZi+zV/D5EtBPq5z9OAJ0TkOuBfgL+jKecDCV7TEoA8P8qWPs/zo55HgfdUdaOfcZlapn/7Rnz754FcP6ANh1V5NW09v/3XTyzYnB3q0IwxHipzP5gd7vMHcA5X/RtoiHuIyQ9rgAgR8Tyc1hNI91E23Z3nWW6nqu7xo56hwJ/cHmiZQEtgvIjc62ecphaIjYrg4Qu78ekt/WmfEsv6rAIue302j0xJp6DYbmhmTE3g74WW81V1mvs8S1XPU9UEVe2rqsv8rKMA+AznxmWxIjIAGAa856P4u8CNItJVRBriJLWxftYzFOgO9HIfO4CbcXqemeNMn9bOeGa3DWlPmDg9zc558Uemrd4V6tCMqfMqdaGliPQVkSvdnly4X/D+nscB+CMQg3ML5o+AW1U1XURaiUi+iLQCUNVvgKeBacBm9/FwRfW4y+5R1czSB3AI2KuqdgvF41R0ZDh3n9OFybcNoGvTBLbt3c/1b8/j5UVFZOTsD3V4xtRZfiUHEUnF6QrcD2fssY7ABuB5nG7Ld/pTj6pmAxf7mL4F5+S957Tn3fr9rqeMsm38KWdqv+7NE5l8+wDGztzEC1PXMH/nIc56bjqjz+7EiNPaEGE3NTMmqPz9xL0AZAKNgEKP6Z/gR28tY4IlMjyMmwa2Y+pfBtEnNZyCA4d44suVXPDvGSzYvDfU4RlTp/ibYIYCf3cvevS0Hue6FGNqlGYNYrijdzRvjehLi4YxrMrMY/hrs7hvwlKyC+zaW2OCwd8EE4PvK+JTcA6RGVMjndklle9HD+K2Ie2JDBc+nreVwc9M4+2ZG20kAGMCzN8E8yPuUC0uFZFwnKFY/lfdQRlTnWLqOZ0Avr7zDM7omExu0UEe/XwF5730E9PXZIU6PGOOW/4mmHuAm0TkeyAKeA5YAQwA7g9QbMZUqw6N43n3hpP5zx/60qZRfdbtyue6t+Zy49h5bMiyTobGVDd/r4NZAZwIzAK+A6JxTvD3VtX1gQvPmOolIpzdNZVvRw/k/vO6EBcVwf9W7eKcF3/kyS9XkFtkNzczprr43W/Tva7kYVW9QFXPV9UHgHoiUjNHiDSmHFER4dw8qD0/3DWIK/q24OBh5T8/bWTQ09MYM2MjxQftvjPGHKtjvTCgAc7Q+cbUSo3jo3n6sp5Mue10Tm6TxN7CEh7/YgVDn5vOpEXbOXy44tHGjTG+2ZVnxgAntkhk3M2nMua6vnRKjWPb3v38edxiLnx5Bj+ttY4AxlSFJRhjXCLC0BNS+frOgTw9vAdNEqJJ35HLtWPmcu2YOSzf7n2XCGNMeSzBGOMlPEy4ol9Lpt01mHvO7Ux8dAQ/rd3NBf+ewS3vLWBVZm6oQzSmVih3LDIRmVLB8t73ZTHmuBFTL5w/Du7AVf1a8cq0dbz382a+Sc/km/RMftujKaPP6kiHxr7ul2eMgYoHu9zjx3y7sZc5rjWMrccDF3TlpoHteC1tPR/O2cKXSzP4alkGw3o2486zOtE2OTbUYRpT45SbYFT1+mAFYkxNl5oQzSMXdWPUwHa8mraOcfO2MmnxDj5fmsElvZtz+5AOtLFEY8wRdg7GmEpq1iCGJy4+kR/+Opjf9WsJwKcLtnHmc2nc8dEiVmbYORpjwBKMMVXWMqk+/xjegx/+Oogr+7YkPEz4fMkOznvpJ24YO48Fm7NDHaIxIWUJxphj1LpRLP+8rAfT7x7C9QPaEB0Zxg+rdjH8tdlc+cZsflyThapdsGnqHkswxlSTZg1iePjCbsy890zuOLMD8dERzNmYzR/emstFL89k8uLtdosAU6dYgjGmmjWKi+Kvv+nMrPvO5N5zu5AcV49l23O48+PFDHx6Gm9MX09Bie3RmONfRd2UjTFVFB8dya2D23P9gDZ8tnA7Y2ZsYH1WAf/39Sqiw2F+UTrXn9aWVo3qhzpUYwLC9mCMCbDoyHCuPqUV348exNsj+jGgQyOKDsHbMzcx+Nlp3Pr+AuZvyrbzNOa4Y3swxgRJWJgwpEtjhnRpzLtT/seSomSmLNnO18sz+Xp5Jl2bJnBt/9YM69WM+vXso2lqP9uDMSYEWiWE89wVPZl575ncPqQDSbH1WJGRy/2fLeOUp/7Ho5+ns97usmlqOUswxoRQ44Ro7jqnM7PvP5MXruzJSa0akFd0kLdnbmLoc9P5/X/n8M3yTA5a7zNTC9l+uDE1QFREOJf0bsElvVuwfHsO7/+8mUmLtzNj3W5mrNtN08RoLu/Tgsv7tqRlknUKMLWD7cEYU8N0b57IP4b3YM7fzuKhC7rSLjmWjJwi/vXDOs54ehrX/PdnJi/eTlGJ3dbZ1Gy2B2NMDZUYE8kNp7dlxGlt+HnjHsbP28rXyzOZuW4PM9ftITEmkot7NeOKfi3p1iwx1OEa8yuWYIyp4cLChNPaJ3Na+2QeLSxhypLtjJu/leXbc3ln9mbemb2Z7s0TuLxPSy7o0ZRGcVGhDtkYwBKMMbVKYv1Iru3fhmv7tyF9Rw7j521l4qLtLN+ey/Lt6Tz+xQoGdkrhkt7NOeuEVGLqhYc6ZFOHBfUcjIgkichEESkQkc0icnU5ZUeLSKaI5IjIWyIS5U89InKqiHwvItkikiUin4hI00BvmzHB1q1ZIo8O687cv5/FS7/rxZDOKSjww6pd3PHRIvo9OZW7PlnCzHW7OXTYLuI0wRfsPZhXgANAKtAL+FJElqhqumchETkHuA84E9gBTAQedadVVE9D4E3gW+Ag8DLwNnBuIDfMmFCJjgxnWK/mDOvVnN35xXyxZAcTF+9gydZ9fLpgG58u2EZqQhTDejXnwh7N6N48AREJddimDghaghGRWGA40F1V84EZIjIFuJZfEkep64AxpYlHRB4HPgDuq6geVf3aa70vA9MDuGnG1BjJcVGMGNCWEQPasiErn0mLdzBp0Xa2ZBfy5o8bePPHDbRMiuH8E5vy2xObcmLzREs2JmAkWOMfiUhvYJaqxnhMuwsYpKoXepVdAjylquPc18lAFpAMtPK3Hnfen4HfqeqpPuaNAkYBpKSk9Bk/fvwxb2cg5OfnExcXF+owfLLYqiaYsakq6/cdZnbGQebvPERO8S+f+ZQYoV+TCPo1CadNQhgiYu1WRXU1tiFDhixQ1b6+5gXzEFkckOM1LQeI96Ns6fP4ytQjIj2Ah4BhvgJS1TdxDqfRuXNnHTx4cLkbECppaWlYbJVnsf1iCDASOHRYmb8pm6+WZfDV8kyy8or5amMJX20scfZsujchpeQQNwwcRFhYzduzsf9p1YQqtmAmmHwgwWtaApDnR9nS53n+1iMiHYCvgTtV9acqxmzMcSU8TDilXSNOadeIhy7sdlSy2Zq9nzd+3ADA6+n/46wTGnN211QGdEgmOtJ6o5nKC2aCWQNEiEhHVV3rTusJpPsom+7OG+9Rbqeq7hGRoorqEZHWwFTgcVV9LwDbYkyt5yvZfL08k88XbmZ3fjEfz9vKx/O2EhMZzhkdkzm7aypDT0glKbZeqEM3tUTQEoyqFojIZ8BjIjISp/fXMOA0H8XfBcaKyAdABvAAMNafekSkOfAD8Iqqvh7IbTLmeOGZbAbF7yK1cx++X7GTqSt3smx7Dt+t2Ml3K3YSJtC3dRJnntCYwZ1T6Jwab50ETJmC3U35j8BbwC5gD3CrqqaLSCtgBdBVVbeo6jci8jQwDYgBJgAPV1SPO28k0A54WESOLKOqNfPsmzE1jIjQtVkCXZslcOdZHcnI2c9UN8H8vGEPczdlM3dTNv/4ehVNE6MZ1CmFQZ1SGNAxmYToyFCHb2qQoCYYVc0GLvYxfQvOyXvPac8Dz1emHnfeozjXzBhjqkHTxJgjowfkFZUwfU0WaauzmL4mi4ycoiOH0iLChJNaN2RQpxQGd06ha1O73qaus6FijDF+i4+O5IIezbigRzMOH1ZWZOQyfU0W01dnsWDLXuZuzGbuxmye+XY1jeOjGNAhmdPaN2JAh2SaNYipeAXmuGIJxhhTJWFhQvfmiXRvnshtQzqQs7+EWet2k7Y6i7Q1u9iZW8zERduZuGg7AG0a1ee0DskMaJ9M//aNrLNAHWAJxhhTLRJjIjnvxKacd2JTVJU1O/OZuW43s9bvZs6GbDbtKWTTni18OGcLACc0TWBA+0ac1qERfdsk2fmb45AlGGNMtRMROjeJp3OTeG44vS0HDx1m2fYcZq3fw8x1u5m/eS8rM3JZmZHLf2dsJEychNOvTRInt02iX5skUuLttgO1nSUYY0zARYSH0btVQ3q3ashtQzpQVHKIhZv3MnP9bmat38OybTmk78glfUcuY2dtAqBtciz92jQ8knRa2a2iax1LMMaYoIuODOe0Dsmc1iEZgP0HDrFo617mbdzLvE3ZLNyyl427C9i4u4Dx87cB0Dg+itaxB1kXvoHerRrQrVmijTBQw1mCMcaEXEy98CN37QQoOXSYFTtymbfJ6ZU2f/NeduUVsysP5n25EoCIMOd6nd4tG9CrVQN6t2xI60b1rWt0DWIJxhhT40SGh9GzZQN6tmzAyDPaOSNCZ+Xz4Xc/s79+Kou27GP1zjyWbsth6bYc3pm9GYCG9SPp1bIBvVs1pGfLBpzYPNF6q4WQJRhjTI0nInRoHM/AFpEMHtwDgPzigyzduo9FW/exaMs+Fm/dy+78A0xbncW01VlHlm3eIIZuzRI4sXki3Vsk0r1ZonUgCBJLMMaYWikuKuKo8ziqyra9+1m0dR8LN+9l2fYcVuzIZfu+/Wzft5/vVuw8smyThGi6N0+ge/NEJ/E0T6RxfJQdXqtmlmCMMccFEaFlUn1aJtXnop7NAOf+Nxuy8lm+I4dl23JZvsNJOpm5RWTmFjF15a4jyyfF1qNLk3i6NElw/jaNp2PjeGLqWUeCqrIEY4w5boWHCR1T4+mYGs8lvZ1phw8rm/YUsGy70zXa6SKdQ3bBAWat38Os9XuOLB8m0KZRLF2aOomnc5N4TmiSQIuGMTXyhmw1jSUYY0ydEhYmtEuJo11KHMN6NQecw2sZOUWsysxlZUYeqzPzWJWZy/qsAjbsdh5fLcs8UkdsvXA6NI6jfeM4OjSOo0NKHNkFhzl46DAR4WGh2rQaxxKMMabOExGaNYihWYMYzuySemR68cFDrNuV7yacPFZm5LIqM4+svGKWbMthybaj797+0KxvaZsceyT5dHQTUNvk2Dp5zY4lGGOMKUNURDjdmiXSrVniUdOzCw6wblc+63bls3ZXHut25ZO+dQ/ZRYdZvTOP1TuPvhN8mECLhvVpkxxL20bO3zaNYmmTHEuLhjFEHqd7PZZgjDGmkpJi63FyW2cIm1JpaWn0638667PyWbszn3VZTgJavyufzdmFbHEfP3rVFR4mtGwYQ+tGsbRNjqVNo/q0To6lbSMn+dTmQ26WYIwxpprERkXQo0UDerRocNT04oOH2LKn0BlRencBG/cUsGl3AZv3FLIjZ7870nQh09dkHbVcRJjQtEE0LRvWdx5JMbRw/7ZsWJ/kuKga3dnAEowxxgRYVET4kd5s3opKDrElu5CNuwvYvKeAjbudJLRpTwEZOUVszd7P1uz9OHeH9643jOYNnWTTomGM003bfd68YQyNYuuF9NoeSzDGGBNC0ZHhdEqNp1MZyWfb3v1s3VvItr372ZZdeOT51uxC9haWsCGrgA1ZBT7rrhceRpPEaGK0iMk7F9M0MZqmDWJolhhN08QYmiZG06B+ZMCSkCUYY4ypoaIjne7QHRrH+ZyfV1TiJB434Xgmn4ycInL2l7AluxCA1Xu3+6wjJjLcTTxO0mmWGE1qYjSp8dE0TogiNSGaRrH1qnQuyBKMMcbUUvHRkZzQNJITmib4nF9QfJCMnCK++XEOjVt3IiOniIyc/ezIKSJj334ycorILz545FqfsoQJNIqLIjUh6kjiaRwfTWpCdLnxWYIxxpjjVGxUBB0ax9E9OZzB/Vr6LJNXVEJGThE73ISTsW8/O3OL2ZVXdOTv7vwDZOUVk5VXzHJy/V6/JRhjjKnD4qMjiY+O9HkOqFTJocPszi9mZ24xO3OLnHvz5BaxM7eIZ8qp2xKMMcaYckWGh7mdAmJ+Na+8BFN7r+AxxhhTo1mCMcYYExCWYIwxxgSEJRhjjDEBYQnGGGNMQFiCMcYYExBBTTAikiQiE0WkQEQ2i8jV5ZQdLSKZIpIjIm+JSJS/9YjIUBFZJSKFIjJNRFoHcruMMcb8WrD3YF4BDgCpwDXAayLSzbuQiJwD3AcMBdoA7YBH/alHRJKBz4AHgSRgPjAuMJtjjDGmLEFLMCISCwwHHlTVfFWdAUwBrvVR/DpgjKqmq+pe4HFghJ/1XAqkq+onqloEPAL0FJEugds6Y4wx3oJ5JX8n4JCqrvGYtgQY5KNsN2CyV7lUEWkEtKqgnm7uawBUtUBE1rvTV3muRERGAaPcl8UisrzSWxUcycDuUAdRBoutaiy2qrHYqiaQsZV5CiKYCSYOyPGalgP4GgDHu2zp83g/6okDssqZf4Sqvgm8CSAi81W1b/mbEBoWW9VYbFVjsVWNxfZrwTwHkw94jymdAOT5Ubb0eZ4f9VRmPcYYYwIkmAlmDRAhIh09pvUE0n2UTXfneZbbqap7/KjnqGXdczbty1iPMcaYAAlaglHVApzeXY+JSKyIDACGAe/5KP4ucKOIdBWRhsADwFg/65kIdBeR4SISDTwELFXVVd4r8fLmsW1hQFlsVWOxVY3FVjUWmxdR1eCtTCQJeAs4G9gD3KeqH4pIK2AF0FVVt7hl/wLcC8QAE4BbVLW4vHo81nMW8DLOyac5wAhV3RSUjTTGGAMEOcEYY4ypO2yoGGOMMQFhCcYYY0xgqGqdfuAMJzMRKAA2A1cHeH1pQBFOd+p8YLXHvKE4F4MWAtOA1h7zBPgnzjmnPcDTuIc43flt3GUK3TrO8iOW23GG0ikGxnrNC1gswNVuWxcAk4Akf2Nz61aP9svHGdUhKLEBUcAYt0wesAg4rya0W3mxhbrd3DLvAxlALk5v0JE1od3Ki60mtJtH2Y443x3v15R2q/A7prILHG8P4COcscrigNNxLsrsFsD1pXl+sDymJ7vrvhyIxrnV9c8e828GVgMtgOY4nSJu8Zg/G3gep1PEcGAfkFJBLJcCFwOvcfSXeMBiwRlRIQ8Y6Lb5h8DHlYitDc4HPqKMbQpobEAszvBDbXCOAFzgLtMm1O1WQWwhbTePclHu8y5AJtAn1O1WQWwhbzePur4DfsJNMDWh3Sr8vgvUF2lteOB8IA8AnTymvQf8I4DrTMN3ghkFzPKKbT/QxX09CxjlMf/G0jcTzjA8xUC8x/yfPN9MFcT0BEd/iQcsFuAp4EOPee3d/0G8n7FV9IEPWmwe5Za6H9Aa024+YqtR7QZ0xtljuKKmtZtXbDWi3YDfAeNxfkCUJpga1W6+HnX9HExZ46P9aoTnavZ/IrJbRGaKyGB32q/GUANKx1D71XyvOLsBG1Q1r4z5lRXIWLzrXo+b5CsZ42YR2SYib7sjaPuMPdCxiUiqOz/dx/IhbTev2EqFtN1E5FURKT0kkwF85WP5kLRbGbGVClm7iUgC8BjwV6+Qa0S7laeuJ5jKjI9WXe7Fuf1Ac5yLnz4XkfZ+xOJrfLY4ERE/lq2sQMZyrLHuBvrhXOPUx13ug3JiD1hsIhLprvsddS7krTHt5iO2GtFuqvpHd/oZOBdMF1eh/mDGVhPa7XGc0eW3ek2vEe1WnrqeYII+bpmqzlHVPFUtVtV3gJnA+X7E4mt8tnx19l+rezsCGcsxxarOLRrmq+pBVd2J0xngN+6vvKDFJiJhOIdTD7gx+LN8yGKrKe3mxnJIndtstABurUL9QYst1O0mIr2As4AXfIRbY9qtLHU9wVRmfLRAUZzeHhWNoeZrfDbPee1EJL6M+ZUVyFi8626H0/vJ8zBlZWhpVcGKzf0FOAbnhnfDVbWkjOWD3m7lxOYt6O3mQwS/tE9Ne7+VxuYt2O02GOc80BYRyQTuAoaLyEIfy9eEdjtaZU7YHI8P4GOcnmSxwAAC2IsMaACcg9PjIwLnbpwFOCcVU9x1D3fn/5Oje4TcAqzEObTWzH0DePYI+Rl41l32EvzrRRbhlv8/nF+8pXEFLBacY7u5OIchYnG6h/rq1VNWbKe47RUGNMLpATgtyLG97tYT5zW9JrRbWbGFtN2AxjgnquOAcJzPQQHOOIIhbbcKYgt1u9UHmng8ngU+ddss5O+3Cr/zAvFFWpseONfBTHLfUFsI4HUw7htiHs5u5j73H3y2x/yzcE4w7sfpbdbGY57g9GPPdh+++rSnucuuxr/rYB7B+UXm+Xgk0LHg9K/f4rb5ZHxfl+AzNuAqYKO7bAbOwKhNghUbzrF45ehrmfKBa0LdbuXFVgPaLQWYjvO+zwWWATcF471/LLGFut3K+Fy8XxPazZ+HjUVmjDEmIOr6ORhjjDEBYgnGGGNMQFiCMcYYExCWYIwxxgSEJRhjjDEBYQnGGGNMQFiCMeY4IyIjRCQ/1HEYYwnGmAARkbEioh6P3SLyhYh0qUQdj4jI8kDGaUygWIIxJrCmAk3dx29wbu40MaQRGRMklmCMCaxiVc10HwtxRsXtIiIxACLyDxFZLSL7RWSTiDwtItHuvBHAw0A3j72gEe68BBF5TUQyRKRIRFaKyJWeKxaRoSKyXEQKRGSaiLQN5oYbExHqAIypK9yRa68ElqnqfndyAXADsB3oijNYZTHwIM7Ait1xbn082C2f446Y/DXQELgeZ4TbzjiDFpaKAu536y4C3nHrPicwW2fMr1mCMSawzvU44R4LbMW5/w8Aqvq4R9lNIvIUzpDsD6rqfnfZg6qaWVpIRM4G+uOM+r3SnbzBa70RwG2qutpd5lngbREJU9XD1bh9xpTJDpEZE1g/Ar3cxynAD8B3ItISQEQuE5EZIpLpJpMXgFYV1NkbyPBILr4UlyYX1w4gEueWEcYEhSUYYwKrUFXXuY+5wI04dwYcJSKn4tyP6FvgQpzE8QBOIiiPVDAf4KDX69Jh0+0zb4LGDpEZE1wKHMa5kdQAYLvnYTIRae1V/gDOTbA8LQSaisgJFezFGBNSlmCMCawoEWniPm+Ic0/3OOBzIB5oLiLXALNxTsBf5bX8JqC1iJyEc/OnPOB/wBxggoiMxjnJ3wGIVdVJAd0aYyrBdpeNCayzcO6EmIGTFPoBl6tqmqp+DjwDvAgsBc4GHvJafgLwFU5SyQKuck/SnwfMxLmV7UrgJaBeoDfGmMqwO1oaY4wJCNuDMcYYExCWYIwxxgSEJRhjjDEBYQnGGGNMQFiCMcYYExCWYIwxxgSEJRhjjDEBYQnGGGNMQPw/48JuhV1aIY8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(steps, lrs, \"-\", linewidth=2)\n",
    "plt.axis([0, n_steps - 1, 0, lr0 * 1.1])\n",
    "plt.xlabel(\"Batch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Exponential Scheduling (per batch)\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Piecewise Constant Scheduling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "def piecewise_constant_fn(epoch):\n",
    "    if epoch < 5:\n",
    "        return 0.01\n",
    "    elif epoch < 15:\n",
    "        return 0.005\n",
    "    else:\n",
    "        return 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "def piecewise_constant(boundaries, values):\n",
    "    boundaries = np.array([0] + boundaries)\n",
    "    values = np.array(values)\n",
    "    def piecewise_constant_fn(epoch):\n",
    "        return values[np.argmax(boundaries > epoch) - 1]\n",
    "    return piecewise_constant_fn\n",
    "\n",
    "piecewise_constant_fn = piecewise_constant([5, 15], [0.01, 0.005, 0.001])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 1.1511 - accuracy: 0.7326 - val_loss: 0.8456 - val_accuracy: 0.7410\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.7371 - accuracy: 0.7786 - val_loss: 0.6796 - val_accuracy: 0.8092\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.8055 - accuracy: 0.7700 - val_loss: 1.7429 - val_accuracy: 0.4514\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 1.0351 - accuracy: 0.6826 - val_loss: 0.9870 - val_accuracy: 0.6928\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.9185 - accuracy: 0.7098 - val_loss: 0.8727 - val_accuracy: 0.6932\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6905 - accuracy: 0.7481 - val_loss: 0.6694 - val_accuracy: 0.7696\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6115 - accuracy: 0.7713 - val_loss: 0.6956 - val_accuracy: 0.7306\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5791 - accuracy: 0.7793 - val_loss: 0.6659 - val_accuracy: 0.7738\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5622 - accuracy: 0.7881 - val_loss: 0.7363 - val_accuracy: 0.7850\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5253 - accuracy: 0.8470 - val_loss: 0.5484 - val_accuracy: 0.8578\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4401 - accuracy: 0.8694 - val_loss: 0.6724 - val_accuracy: 0.8602\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.4334 - accuracy: 0.8732 - val_loss: 0.5551 - val_accuracy: 0.8504\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4179 - accuracy: 0.8771 - val_loss: 0.6685 - val_accuracy: 0.8554\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4300 - accuracy: 0.8775 - val_loss: 0.5340 - val_accuracy: 0.8584\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4069 - accuracy: 0.8777 - val_loss: 0.6519 - val_accuracy: 0.8478\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3349 - accuracy: 0.8953 - val_loss: 0.4801 - val_accuracy: 0.8778\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2695 - accuracy: 0.9109 - val_loss: 0.4880 - val_accuracy: 0.8786\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2568 - accuracy: 0.9136 - val_loss: 0.4726 - val_accuracy: 0.8822\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2436 - accuracy: 0.9203 - val_loss: 0.4792 - val_accuracy: 0.8842\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2421 - accuracy: 0.9212 - val_loss: 0.5088 - val_accuracy: 0.8838\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2288 - accuracy: 0.9246 - val_loss: 0.5083 - val_accuracy: 0.8830\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2215 - accuracy: 0.9270 - val_loss: 0.5217 - val_accuracy: 0.8846\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2106 - accuracy: 0.9297 - val_loss: 0.5297 - val_accuracy: 0.8834\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2002 - accuracy: 0.9334 - val_loss: 0.5597 - val_accuracy: 0.8864\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2005 - accuracy: 0.9350 - val_loss: 0.5533 - val_accuracy: 0.8868\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.LearningRateScheduler(piecewise_constant_fn)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAsqklEQVR4nO3de5xdVX338c8395nJ5DYJGQhmBjBMICCgKCpF0waJrW2lom0FKXhL1fpUsaLwFOTmrWhptSI1LYgo+iAaLoqiRTJSvHJTMJBEVAIkJCQhCZlkcv89f+x9ksOZc2b2TGafk5nzfb9e5zXn7L32OmuvOXN+s/Zaey1FBGZmZoNtRK0LYGZmw5MDjJmZ5cIBxszMcuEAY2ZmuXCAMTOzXDjAmJlZLhxgrCxJ50rqqnU5eiMpJL251uWwbCRdL+m7OeQ7Nf0szO3HMe3pMSeWe22DwwGmTqV/7JE+dkr6vaTPSmpKk9wEHF7LMmZwMPCdPN9AUrOkKyQ9Kqlb0hpJnZLeKqkqfz95fvn1J29Jr5X0I0nrJG2V9DtJN0qaMNjlqoGnSD5Pv6pxOYaVUbUugNXUXcDZwGjgFOC/gSbgvRHRDXTXsGx9iojVeeYvaRJwLzAZuAj4JbAD+CPgYuBnwBN5luFAIelo4E7gP4EPAluAFwOnA2NrVrBBEhG7gVw/T3UpIvyowwdwPfDdkm3/BTyTPj8X6CrZ/xfAA8A24A/AJ4AxRfvHAJ8EVgDbgd8D/1i0/2jgDmAz8CzwDaA13XcUEEWvG0m+zL9fdPy7gd8WvQ7gzUWvP1b03quBG4r2CfgI8DuSwPkI8LY+6uiLJF+kh5bZNw4Ylz6fDHwF2JDmfRcwpyjtuUAXMA/4TZrnYuCwojQvAm4DngO2AkuBvy06z+JHZ7r95cAPgXXA8yTB8FUl5QxgAXBz+r6/Lz7vSnmXOd8PAk9n+FzNBm4HNqXn/DPg2OLPHPABYGVaX18GGvvze0rPu/A5fAh4Q1r2uen+uenrqUXHtKfbTsz4upDHPOAX6e/kfuClJWV5B/Bkuv87wPuAqPXf94Hy8CUyK9ZN0prpQdJ84EbgC8Ackj+sN5MElIKvAH8HfIgkYLwT2JgefzBwD8kX7CuAU4HxwO2SRkTEY8Aakj9sgJNJvqT+SFKhpT0X6KxQvjOAD5P8gc8C/pykxVHw8bQ8/0AS6D4FfEnSGyrkNwL4W+DGiHi6dH9EbIuIbenL64GTgDem57YVuFNSQ9EhY4ELSertVcAkktZAwRdJguofk9TvB0nrLs0T4PUkl3HelL5uBr5K0vp8Bcnlne9JmlpS3I+RBK/jSC59XieprY+8S60Gpkn64wr7kXQISZAL4HXAS4GrgZFFyU4BjiH5/f8N8FckAaeg199Tegn3DpJAeSJwAfDZSmUaBJ9K3+OlwHrgRklKy/Iqklb/1cDxJIH1shzLMvTUOsL5UZsHJS0Yki+adcBN6etzKWrBkASHi0vyOJ3kv1SRfKkH8PoK73c58KOSbZPTY16Rvr4J+FL6/BPANSSXoF6VbnsaOKvo+L0tGJKgtgwYXea9m0iC5ykl2/8d+F6F8h6U5n9eH/VYOO/XFG2bSBIc31VUlwF0FKU5i6SFNiJ9/TBwSYX3aKfov+teyiLgGXq2UD5V9HoUSQB8Wz/zHknS2giSfwS+k9b5tKI0nyBpQY6pkMf1JH0do4q2/RdwV9bfE0lrbCMwvmj/28ivBTO/KI+T022Hpq+/AdxZUtaFuAWz9+EWTH17vaQuSdtILmXcA/yfCmlfBvxzmr4rHWH2dZIvhVbgBGAPyaWfSse/puT4p9J9R6Q/O9nXgpmb5vVjYK6kWcAMKrRgSC4BjQP+IOlaSW+RVOgbODrdd2fJ+7+36L1LqcL2UkeRnPfPChsiYhPJpZ2ji9Jtj4hlRa9XkbQWJ6WvPwdcJOlnkj4u6WV9vbGkgyR9SdJySZtILj0eBMwsSfpwUdl2AWvTdJlFxO6IeDtwKElL8UngfGCppDlpshOAeyNiRy9ZPZqWoWBVUVmy/J6OAh6OiOIRjj8jPw8XPV+V/iyUdzYvbCVDcjnNUu7kr2/3kPxHuBNYFRE7e0k7gqT5f3OZfWvp+wt5BMmljQ+X2bcm/dkJfDENJiemr5uAt5K0rh6PiJXlMo+IpyR1kFwzPxX4V+ASSSexb7TkX5B8MRardM5rSfoIjurjvHo77+KpyndV2DcCICKulfQD4M9Iyv9TSZ+KiEt7yf8rwHTgPJKW3nbgRyR9YcVKzzEY4AjStP6/CnxV0kXAcpJAcy7ZgnJvZcnye8ryHnvKpC176TeD4vK+4HeW5h9YRW7B1LetEfF4RKzoI7gAPAjMTtOXPnal+0eQ9CFUOn4OsKLM8ZsBYl8/zD+TBJNnSVoxJ5Nc0+/srYCR9IvcERHnkXQEz0mPfZTky7etzHuvqJDXHpJLdmdJOrR0v6RxksaleY8g6Vcp7JsAHJvuyywino6IhRHx1yT9JgvSXYUWwciSQ/4I+I/0nJeQtGAO7s979pJ3lvJuILkkNz7d9CBJn1lpgMsqy+/pUeDYouH0AK8syWdt+rO4Lo4fYJl68xj7+rAKSl/XNQcYy+py4ExJl0s6RtJsSW+WdCVARPwW+Cbw35LOkHSYpFMknZ0efzVJ38RNkk6SdLikUyUtlNRc9D4/JrmmvjjN9wmSL4w30UuASW8MfZekYyUdBryd5L/P36YB7LPAZyW9Q9KLJR0v6T2SFlTKE/i/JP9J/0LS2yXNSY89m2QUU2t63reRdESfIulY4Gsko7q+nrFukfQ5Sa9P6+V4kk73QoB6lqRvYr6k6ZImptuXA2+TdLSklwP/j30BI6tKeZeW7+8lXSPpNElHpHXxLySB9NY02RdJgs03Jb08rau3pufTp4y/p6+TtAavS8vwOpJ/SIo9TnL59VJJR0o6jWSY+WD7PHCapPMlzZL0TpJBC1ZQ604gP2rzoMww5ZL959JzmPJpwP+SdBI/TzJs8/1F+8cCV5IMQd1OMtS0eP8s4FvsG867DPgPXjjU+T30HH58fbptRkl5ijv5Tye5Fr+RZDjufcCfF6UVSf9S4b/ktcD/AK/ro54mknReLyUZFvssSaD7W/Z10GcaplyS71yKOqLTevht+h5rSYLFjKL07yIJdrvZN0z5OJJr/t1pXZ9NMkrv0nJ1VLTtCeDDveVdph5OSM+xMHx4PfBz4OySdHOA75EM/tgM/BQ4ptJnDrgU+E1/fk8kI/YeTPf/muSS2t5O/jTNq0lG1XWnn4vCUOb+dvJXHCiQbnsHSTDrJhn48E9Ad63/vg+Uh9JKMjOz/STp34BTI+LYWpflQOBOfjOzAZJ0PkkLq4tkcMZ7SC6tGrgFY2Y2UJJuIrmcNpFkdosvAZ8Lf7ECDjBmZpYTjyIzM7NcuA8mNWnSpHjxi19c62IccLZs2UJTU1PfCeuM66U810tPw71OHnjggXURMa3cPgeY1PTp07n//vtrXYwDTmdnJ3Pnzq11MQ44rpfyXC89Dfc6kVT2ZmXwJTIzM8uJA4yZmeXCAcbMzHLhAGNmZrlwgDEzs1w4wJiZWS4cYMzMLBcOMGZmlgsHGDMzy4UDjJmZ5cIBxszMcuEAY2ZmuXCAMTOzXDjAmJlZLhxgzMwsF1UNMJKmSLpF0hZJKySd2Uva8yStlrRJ0nWSxhbte7+k+yVtl3R9mWPnSVoqaaukxZLa+irbE8/v4eRP382tD63MdC63PrSSkz99N4ddcMewPM7MbH9VuwVzNbADmA6cBVwjaU5pIknzgQuAeUA7cDhwWVGSVcDHgevKHDsVWARcDEwB7gduylK4lRu7uXDRI31+Cd/60EouXPQIKzd2E8PwODOzwVC1FS0lNQFnAMdERBdwr6TbgbNJgkmxc4BrI2JJeuwVwI2FdBGxKN1+InBoybFvApZExM1pmkuBdZJmR8TSvsrZvXM3//eWR7j38XUV03zvkWfo3rl7yB73mR8s4/QTZlQ8zsxsMFRzyeQjgd0Rsbxo26+B15ZJOwe4rSTddEktEbG+j/eZk6YHICK2SPpduv0FAUbSAmABwJjWF+/dvnXHbhYvqfxf/tYdUWH70Dhu5cZuOjs7Kx5XrKurK3PaeuJ6Kc/10lM910k1A8x4YFPJtk1Ac4a0hefNQF8BZjywNsv7RMRCYCHA2INn7f02njGpgZ9c8CcV3+DkT9/Nyo3dPbYPpeOyrhE+3NcTHyjXS3mul57quU6q2QfTBUwo2TYB2JwhbeF5ubT78z49NIweyfnzO3pNc/78DhpGjxy2x5mZDYZqBpjlwChJs4q2HQcsKZN2SbqvON2aDJfHehyb9v0cUeF9XmDGpAY+9aZj++yfOP2EGXzqTccyY1IDGgLHHTxxHAATxo3KdJyZ2WCo2iWytC9kEXC5pHcBxwNvBF5dJvkNwPWSbgSeAS4Cri/slDSKpOwjgZGSxgG7ImIXcAvwGUlnAHcAHwMe7quDv33CiF4vN5U6/YQZA/qirtVxJ33yLk6ZNc3BxcyqptrDlN8HNADPAt8A3hsRSyTNlNQlaSZARNwJXAksBlakj0uK8rkI6CYZVfa29PlF6bFrSUarfQLYAJwE/G3+p3Zga2tpYsX6LbUuhpnVkWp28hMRzwGnl9n+JEnnfPG2q4CrKuRzKXBpL+9zFzB74CUdfg5raeJHS5+tdTHMrI54qpg60Ta1kXVd2+navqvWRTGzOuEAUyfaW5oAfJnMzKrGAaZOtLU0ArBi/dYal8TM6oUDTJ1oS1swT7gFY2ZV4gBTJ8aPHcXU8WNZsc4tGDOrDgeYOtLe0ugWjJlVjQNMHWlraXKAMbOqcYCpI4dNbWTN89vZusNDlc0sfw4wdaTQ0f/kc+6HMbP8OcDUkcK9ME+4o9/MqsABpo7M3HsvjPthzCx/DjB1ZGLDaKY0jeEJ32xpZlXgAFNn2loa3YIxs6pwgKkz7S1Nni7GzKrCAabOtLc0sWpTN9t27q51UcxsmHOAqTPtUxuJgKc8VNnMcuYAU2f2TXrpAGNm+XKAqTPtHqpsZlXiAFNnJjWOYWLDaM9JZma5c4CpQ+0tjR5JZma5c4CpQ55V2cyqwQGmDrW3NLJyQzc7du2pdVHMbBhzgKlD7VOb2BPw1AZfJjOz/DjA1KHCUGWPJDOzPDnA1KHCUGVP229meXKAqUNTmsbQPHaUWzBmlisHmDokibapjb6b38xy5QBTp9pamtyCMbNcOcDUqfaWRp7e0M3O3R6qbGb5cICpU+0tTezaE6zc0F3ropjZMFXVACNpiqRbJG2RtELSmb2kPU/SakmbJF0naWzWfCT9taTHJG2W9Kik03M8rSGpfWphVmVfJjOzfFS7BXM1sAOYDpwFXCNpTmkiSfOBC4B5QDtwOHBZlnwkzQC+BnwImACcD3xd0kH5nNLQ1LZ3VmV39JtZPqoWYCQ1AWcAF0dEV0TcC9wOnF0m+TnAtRGxJCI2AFcA52bM51BgY0R8PxJ3AFuAI3I8vSFn2vixNI4Z6RaMmeVmVBXf60hgd0QsL9r2a+C1ZdLOAW4rSTddUgsws4987gcek/SXwB3AXwDbgYdL30TSAmABwLRp0+js7BzAaQ1dLWODB5c/RWfn2oppurq66q5esnC9lOd66ame66SaAWY8sKlk2yagOUPawvPmvvKJiN2SbgC+DowjuZT2lojo8a96RCwEFgJ0dHTE3Llz+3E6Q98xTz/AsjWb6e28Ozs7e91fr1wv5bleeqrnOqlmH0wXSZ9IsQnA5gxpC88395WPpFOBK4G5wBiSls1/Szp+4EUfntpamnjqua3s3hO1LoqZDUOZA4ykP5X03XRU1ovSbe+SNC9jFsuBUZJmFW07DlhSJu2SdF9xujURsT5DPscD90TE/RGxJyLuA34BnJqxnHWjvaWRnbuDVRs9VNnMBl+mACPpLOCbwG+Bw4DR6a6RwEey5JFeoloEXC6pSdLJwBuBr5ZJfgPwTklHS5oMXARcnzGf+4BTCi0WSScAp1CmD6beeaiymeUpawvmI8C7I+I8YFfR9p+TtBiyeh/QADwLfAN4b0QskTRTUpekmQARcSfJZa7FwIr0cUlf+aTH/hi4FPiWpM3At4FPRsQP+1HOutDeUggwHqpsZoMvayf/LOBnZbaX6w+pKCKeA04vs/1Jks774m1XAVf1J5+i/V8AvpC1XPXqoOaxjBs9ghXr3IIxs8GXtQWzimSYcanXAL8bvOJYNY0YIdqmNLkFY2a5yBpgFgKfT/s7AF4k6RySy1jX5FIyq4q2lkbPqmxmuch0iSwirpQ0EfgfkntLFpPcvPjZiLg6x/JZztqnNtG5fC179gQjRqjWxTGzYSTzjZYR8c+SPgEcTdLyeTQiunIrmVVFW0sjO3btYfXz2zhkUkOti2Nmw0jWYcrXSWqOiK3p/SW/jIiudJjwdXkX0vJzWGEkmTv6zWyQZe2DOYdkWHCpBuDvBq84Vm1tUz1U2czy0eslMklTAKWPyZKK74EZCbwBWJNf8SxvB08Yx5hRI9zRb2aDrq8+mHVApI9Hy+wPXngDpA0xI0aImVMafTe/mQ26vgLMH5O0Xu4mWYPluaJ9O4AVEbEqp7JZlbS3NHrhMTMbdL0GmHTaFSQdBjwVEXuqUiqrqraWJu59fB0RgeShymY2OLLeB7MCQNIhJAt+jSnZf8/gF82qpb2lkW079/Ds5u1MnzCu1sUxs2EiU4BJA8vXSaaGCZLLZsWLiIwc/KJZtbSlQ5X/sG6LA4yZDZqsw5T/HdhNcpPlVpLp798CPAa8PpeSWdUclg5V9kgyMxtMWe/kfy3whohYKimAtRHxE0nbgStIppCxIergieMYPVK+F8bMBlXWFkwDyZBlSEaSHZQ+fxR4yWAXyqpr1MgRvGiyJ700s8GVNcAsBWanz38FvEdSG/APwMocymVV1tbSyBPr3IIxs8GT9RLZ54DW9PnlwJ3AW0lmVD4nh3JZlbW1NPHLPzznocpmNmiyDlO+sej5g5LaSVo0T0bEuooH2pDR3tLIlh27Wde1g2nNY2tdHDMbBrJeInuBdFblB4Etki4Y5DJZDbR5JJmZDbI+A4ykqZLeIOk0SSPTbaMlfRB4AvhwvkW0ajis6F4YM7PB0Ndsyq8G7gAmktxYeZ+kc4FbgNEkQ5S9HswwMGNyAyNHyHOSmdmg6asFcwXwA5KhyJ8DXgF8F/gUMCsivhAR/kYaBkaPHMGhkxs8q7KZDZq+AsxxwBUR8RvgIpJWzIURcUNERO+H2lDT1tLkFoyZDZq+AswUYC0kHfsk08Q8lHehrDbaW5J1Yfy/g5kNhizDlAsrWRYmuJyQrnS5V0Q8V/ZIG1LaWprYvG0XG7buZErTmL4PMDPrRZYAU7ySpYD7Sl4Hnk15WGhvaQTgifVbHGDMbL9lWdHS6kRh2v4n1m3hpTMn17g0ZjbUZVrR0urDi6Y0MEJ4VmUzGxQDupPfhqexo0ZyyKQG381vZoOiqgFG0hRJt0jaImmFpDN7SXuepNWSNkm6TtLYrPlIapT0RUnr0uO9pHNG7S1NbsGY2aCodgvmamAHMB04C7hG0pzSRJLmAxcA84B24HDgsn7ks5BkiPVR6c/zBvtEhqu2Fq8LY2aDo2oBRlITcAZwcUR0RcS9wO3A2WWSnwNcGxFLImIDyYwC52bJR1IH8JfAgohYGxG7I+KBnE9v2GhvaWLj1p1s3Lqj1kUxsyEu63owg+FIYHdELC/a9muS5ZhLzQFuK0k3XVILMLOPfE4CVgCXSTobeAa4NCK+XfomkhYACwCmTZtGZ2fnQM5rWNm8ZhcAi/7nfzl84ki6urpcL2W4XspzvfRUz3WSKcBIqjShZQDbgMeBmyJiVS/ZjAc2lWzbBDRnSFt43pwhn0OBY4BvA4cArwLukPRoRDz2gsJHLCS5nEZHR0fMnTu3l+LXh0PWbObzD91DS9ts5h4/g87OTlwvPbleynO99FTPdZK1BTMNOAXYA/wm3XYMyY2WDwBvAi6XdEpE/KpCHl3AhJJtE4DNGdIWnm/OkE83sBP4eETsAn4saTFwGvAY1quZUxqR8PLJZrbfsvbB/AT4PnBoRLwmIl5D0lL4HvBDoI1kWv9/7SWP5cAoSbOKth0HLCmTdkm6rzjdmohYnyGfhzOek5UxbvRIDp4wzh39ZrbfsgaYDwCXF0/Nnz7/BHBeROwA/gU4vlIGEbEFWETS0mmSdDLwRuCrZZLfALxT0tGSJpPM5Hx9xnzuAZ4ELpQ0Kt0/l2TZAcugraXJ0/ab2X7LGmDGAweX2d6a7gN4nr4vub0PaACeBb4BvDcilkiaKalL0kyAiLgTuBJYTNJhvwK4pK980mN3kgScPyPpm/kv4O8iYmnGc6177VMbPW2/me23rH0wtwDXSvoIyWSXQbL42JUkrQnS18vLH55IZ10+vcz2J9kXqArbrgKu6k8+RfuXkHTu2wC0tTSxfssOnt+2s9ZFMbMhLGuAeQ/Jl/3Xio7ZRbJc8ofT148B7x7U0llNFGZVftKtGDPbD5kCTNrf8h5J/wQcQTJ67PG0P6SQ5le5lNCqbu+syuu3vLBZaWbWD/260TINKB6lNcy1FdaFWbeFYzwdqpkNUNYbLceRjCSbBxxEyeCAiHjJ4BfNaqVxzCimTxjLE+u3csy0WpfGzIaqrC2YLwJ/BdwM/JSkk9+GsbaWpuReGAcYMxugrAHmdOAtEXFXjmWxA0h7SyOLl62lutPVmdlwkvUK+1bgqTwLYgeWtpYm1m7ezrZdbqya2cBkDTBXAh+S5C7fOtGejiR7duueGpfEzIaqrNc/Xkcy2eXrJT1KMpnkXhHxl4NdMKutwkiyNVvdgjGzgckaYNaR3M1vdaJ9qlswZrZ/st5o+fa8C2IHlrseXcMIwc3Ld/LTT9/N+fM7OP2EGX0ed+tDK/nMD5axamM3h0xqGLbHrdzYzYyfu17MeuMhQtbDrQ+t5MJFj7AnvTq2cmM3Fy56BKDXL5vCcd07d/u4OjrOrBJFlL/GLulh4LURsUHSI/Ry78twuNGyo6Mjli1bVutiHBBO/vTdrNzY3WP76JHi6EMmVjzu0VWb2Lm758fExw3t42ZMauAnF/xJxeOK1fPqjZUM9zqR9EBEnFhuX28tmG8D29Pn3xr0UtkBa1WZ4AKwc3cwqWF0xePKfTn5uKF/XKXPg1lfKgaYiLis3HMb/g6Z1FC2BTNjUgNfeccrKh5XqeXj44b2cYdMaqh4jFlvfF+L9XD+/A4aRo98wbaG0SM5f36Hj/NxZpllnexyCsnyyJUmu5ww+EWzWil06O4dLZVxNFHxcf0ZhTQUjxvO9XLp7UvY2L2T6RPGcuGfHuUOfhuwip38L0gk3QKcACwEVlHS4R8RX8mldFXkTv7yhnsH5UAN53p56MkN/NUXf8qXzn4Z8+e09uvY4VwvAzXc62SgnfzF5gGvi4hfDF6xzOxAdOT0ZgCWrd7c7wBjVixrH8yzQFeeBTGzA0PT2FHMnNLIstWba10UG+KyBph/Bi6X5BV0zerA7NZmlq5+vtbFsCEu6yWyi4B24FlJK+g52eWQv9HSzPaZ3drMXY+tYdvO3YwrGVlmllXWAOMbLc3qSEfrBPYEPP5sF8fMqHz3v1lv+gwwkkYDTcDVEbEi/yKZWa11tCYd/UtXb3aAsQHrsw8mInYC7wWUf3HM7EDQ3tLImFEjWOZ+GNsPWTv5fwhkm+3OzIa8USNHMOug8Sz1SDLbD1n7YH4EfFLSS4AHgC3FOyNi0WAXzMxqq6O1mXt/u67WxbAhLGuA+UL68x/L7AvAw0zMhpmjWiew6MGVPLdlB1OaxtS6ODYEZbpEFhEjenk4uJgNQ/s6+t0PYwPj2ZTNrKzZrfumjDEbiMwBRtIUSWdKukDSx4of/czjFklbJK2QdGYvac+TtFrSJknXSRrb33wkXSIpJJ2atYxmlpjWPJbJjaMdYGzAsk7X/0rgDpIVLqcBK4GD09dPAJdnfL+rgR3AdOB44A5Jv46IJSXvNx+4gGTk2irgFuCydFumfCQdAbwZeCZj2cysiCQ6Wps9kswGLGsL5jPAjcAMYBvJF/9M4H7gX7JkIKkJOAO4OCK6IuJe4Hbg7DLJzwGujYglEbEBuAI4t5/5fAH4KEkgMrMBmN06geVrNrNnT9/LepiVyjqK7CXAOyMiJO0GxkbE7yV9FPg6SfDpy5HA7ohYXrTt18Bry6SdA9xWkm66pBaSwNZrPpLeAuyIiO9Jle8PlbQAWAAwbdo0Ojs7M5xGfenq6nK9lFEv9aJNO9m6YzffunMxBzX2/f9ovdRLf9RznWQNMMWtgDVAG/AYyRT+h2TMYzywqWTbJqA5Q9rC8+a+8klnfP4kcFpfBYqIhSSLqNHR0RHDeVGggRruiyUNVL3Uy8QnN/DlJT9lYtvRzM2wNky91Et/1HOdZL1E9iDw8vR5J/BxSecAnwcezphHF1C6tPIEoNwF3tK0heebM+RzGfDViPhDxnKZWQXFi4+Z9Vd/1oNZlT6/CFgL/AcwmfQSUwbLgVGSZhVtOw5YUibtknRfcbo1EbE+Qz7zgH9MR6CtBl4EfDO9nGdm/eDFx2x/ZLpEFhH3Fz1fC/xpf98oIrZIWkSycNm7SEZ/vRF4dZnkNwDXS7qRZBTYRcD1GfOZB4wuyus+4EPA9/tbZjMjHUnmmy2t//p1o6WkEyX9TTqSC0lNkrL24wC8D2ggWYL5G8B7I2KJpJmSuiTNBIiIO4ErgcXAivRxSV/5pMeuj4jVhQewG9gQEV7y2WwAZrc288T6rWzbubvWRbEhJut9MNNJhgK/nGTusVnA74GrSIYtfyBLPhHxHHB6me1PknTeF2+7Ks0/cz4V0rZnSWdm5c1uncDuPeHFx6zfsrZg/g1YDbQAW4u230yG0VpmNnR1eMoYG6Csl7fmAfMiYkPJfSW/I7kvxcyGqcLiY+6Hsf7K2oJpoPwd8dNILpGZ2TDlxcdsoLIGmHtIp2pJhaSRJFOx/GiwC2VmB5aO1mZfIrN+y3qJ7CPAjyW9HBgL/CvJdC4TgZNzKpuZHSBmtzaz6MGVbNiyg8lefMwyyrrg2KPAscBPgR8C40g6+E+IiN/lVzwzOxDMbk0mz/BlMuuPzPewpPeUFN+LgqQ2Sd+MiL8e9JKZ2QFj3+Jjz/OqI1pqXBobKvZ3RctJJFPnm9kwVlh8zC0Y6w8vmWxmffLiYzYQDjBmlokXH7P+coAxs0w6WpvZumM3T2/ornVRbIjotZNf0u19HF+6LouZDVOFjv6lq59nZktjjUtjQ0Ffo8jWZ9jvhb3M6kDx4mOnZVjd0qzXABMRb69WQczswFZYfMwd/ZaV+2DMLDMvPmb94QBjZpl58THrDwcYM8uso7V57+JjZn1xgDGzzApzknlmZcvCAcbMMissPrZsjQOM9c0BxswyKyw+9tgz7ui3vjnAmFm/ePExy8oBxsz6ZXZrM89u3s6GLeVWUTfbxwHGzPqlw4uPWUYOMGbWL0cVLT5m1hsHGDPrl8LiYx5JZn1xgDGzfiksPvbYMw4w1jsHGDPrNy8+Zlk4wJhZv3nxMcvCAcbM+q2jaPExs0qqGmAkTZF0i6QtklZIOrOXtOdJWi1pk6TrJI3Nko+kV0r6H0nPSVor6WZJB+d9bmb1pKNo8TGzSqrdgrka2AFMB84CrpE0pzSRpPnABcA8oB04HLgsYz6TgYXpcW3AZuDLg38qZvVr7+JjHklmvahagJHUBJwBXBwRXRFxL3A7cHaZ5OcA10bEkojYAFwBnJsln4j4fkTcHBHPR8RW4AvAyTmfnlnd6WhtZqnnJLNe9Lpk8iA7EtgdEcuLtv0aeG2ZtHOA20rSTZfUAszsRz4ArwGWlNshaQGwAGDatGl0dnZmOI360tXV5Xopw/UC47bv4A/rdvLDHy1mzEgBrpdy6rlOqhlgxgObSrZtApozpC08b+5PPpJeAnwMeGO5AkXEQpLLaXR0dMTcuXN7PYF61NnZieulJ9cLdE1ZxXd+9xCHzH4px8yYCLheyqnnOqlmH0wXMKFk2wSSPpK+0haeb86aj6QXA98HPhAR/zvAMptZBbNb3dFvvatmgFkOjJI0q2jbcZS/fLUk3Vecbk1ErM+Sj6Q24C7gioj46iCV38yKtLc0efEx61XVAkxEbAEWAZdLapJ0Msmlq3IB4AbgnZKOljQZuAi4Pks+kmYAdwNXR8R/5nxaZnWrsPiYZ1W2Sqo9TPl9QAPwLPAN4L0RsUTSTEldkmYCRMSdwJXAYmBF+rikr3zSfe8iGdZ8SZpnl6SuKpybWd3xSDLrTTU7+YmI54DTy2x/kqTzvnjbVcBV/ckn3XcZL7xnxsxyMru1mUUPrmTDlh1MbhpT6+LYAcZTxZjZgHnxMeuNA4yZDdhsLz5mvXCAMbMBO6h5LJO8+JhV4ABjZgMmidmtzb5EZmU5wJjZfpndOoHlq734mPXkAGNm+6WjtZktXnzMynCAMbP94sXHrBIHGDPbL0d68TGrwAHGzPbL+LGjeNGUBi8+Zj04wJjZfpvdOsEtGOuhqlPFmNnwJILHn+3i3Dthxs/v5vz5HZx+wow+j7v1oZV85gfLWLWxm0MmNQyr4wrHrNzYPazrZEzri19WKY0DjJntl1sfWsniZWv3vl65sZsLFz0C0OsX1a0PreTCRY/QvXP3sDtuKJRxMI+rRBEeuw7JipbLli2rdTEOOPW8Gl9vXC/7nPzpu1m5secQ5bGjRnDS4S0Vj/vF79ezfdeeYXncUCjjYB33zFc+yPZnfqty6dyCMbP9sqpMcAHYvmsPz3fvrHhcuS+24XLcUChjHseVcoAxs/1yyKSGsi2YGZMauPUfTq54XKWWz3A4biiUMY/jSnkUmZntl/Pnd9AweuQLtjWMHsn58zvq9rihUMbBPq4ct2DMbL8UOoP3jpjKOBKp+Lj+jGAaCsfVU50800s6d/Kn3Mlfnjuzy3O9lOd66Wm414mkByLixHL7fInMzMxy4QBjZma5cIAxM7NcOMCYmVkuHGDMzCwXDjBmZpYLBxgzM8uFA4yZmeXCAcbMzHLhAGNmZrlwgDEzs1w4wJiZWS6qGmAkTZF0i6QtklZIOrOXtOdJWi1pk6TrJI3Nmo+keZKWStoqabGktjzPy8zMeqp2C+ZqYAcwHTgLuEbSnNJEkuYDFwDzgHbgcOCyLPlImgosAi4GpgD3AzflczpmZlZJ1QKMpCbgDODiiOiKiHuB24GzyyQ/B7g2IpZExAbgCuDcjPm8CVgSETdHxDbgUuA4SbPzOzszMytVzQXHjgR2R8Tyom2/Bl5bJu0c4LaSdNMltQAz+8hnTvoagIjYIul36falxW8iaQGwIH25XdJv+n1Ww99UYF2tC3EAcr2U53rpabjXScUuiGoGmPHAppJtm4DmDGkLz5sz5DMeWJvlfSJiIbAQQNL9lRbNqWeul/JcL+W5Xnqq5zqpZh9MFzChZNsEYHOGtIXnmzPk05/3MTOznFQzwCwHRkmaVbTtOGBJmbRL0n3F6dZExPoM+bzg2LTP5ogK72NmZjmpWoCJiC0ko7sul9Qk6WTgjcBXyyS/AXinpKMlTQYuAq7PmM8twDGSzpA0DvgY8HBELC19kxIL9+8Mhy3XS3mul/JcLz3VbZ0oIqr3ZtIU4DrgdcB64IKI+LqkmcCjwNER8WSa9kPAR4EG4NvAeyJie2/5FL3PqcAXSDqffgGcGxFPVOUkzcwMqHKAMTOz+uGpYszMLBcOMGZmlou6DzD9mR+tnkjqlLRNUlf6WFbrMtWCpPdLul/SdknXl+yryznvKtWJpHZJUfSZ6ZJ0cQ2LWlWSxkq6Nv0e2SzpIUl/WrS/7j4vdR9gyDg/Wp16f0SMTx8dtS5MjawCPk4yqGSvOp/zrmydFJlU9Lm5oorlqrVRwFMks4pMJPlsfDMNvHX5eanmnfwHnKJ5zY6JiC7gXkmFec0uqGnh7IAQEYsAJJ0IHFq0a++cd+n+S4F1kmZnGBI/pPVSJ3UtvYXi0qJN35X0B+BlQAt1+Hmp9xZMpfnR3IJJfErSOkk/kTS31oU5wPSY8w4ozHlX71ZIelrSl9P/3OuSpOkk3zFLqNPPS70HmP7Mj1ZvPkqyTMIMkhvFviPpiNoW6YDiz05P64CXk9x/9jKSurixpiWqEUmjSc79K2kLpS4/L/UeYDxvWQUR8YuI2BwR2yPiK8BPgD+rdbkOIP7slEiXz7g/InZFxBrg/cBpkkrraViTNIJkZpEdJHUAdfp5qfcA05/50epdAKp1IQ4gnvOub4W7uOvmcyNJwLUkg4bOiIid6a66/LzUdYDp5/xodUPSJEnzJY2TNErSWcBrgB/UumzVlp7/OGAkMLJQJwx8zrshr1KdSDpJUoekEenaTZ8HOiOi9NLQcHYNcBTwFxHRXbS9Pj8vEVHXD5Ihg7cCW4AngTNrXaZaP4BpwH0kzfeNwM+B19W6XDWqi0tJ/hMvflya7juVZBG7bqATaK91eWtZJ8BbgT+kf0vPkExa21rr8laxXtrSuthGckms8DirXj8vnovMzMxyUdeXyMzMLD8OMGZmlgsHGDMzy4UDjJmZ5cIBxszMcuEAY2ZmuXCAMRum0rVZ3lzrclj9coAxy4Gk69Mv+NLHz2tdNrNqqev1YMxydhfJ2kLFdtSiIGa14BaMWX62R8TqksdzsPfy1fsl3ZEuobtC0tuKD5Z0rKS7JHVLei5tFU0sSXOOpEfS5YvXlC7rDEyRdHO6JPjvS9/DLE8OMGa1cxlwO3A8yZo7N6SrRCKpEbiTZC6rVwB/BbyaomWKJf098CXgy8BLSJZTKJ2d92PAbSQz+d4EXFcPa8HbgcFzkZnlIG1JvI1k4sNiV0fERyUF8N8R8e6iY+4CVkfE2yS9G/gscGhEbE73zwUWA7Mi4nFJTwNfi4iyy3un7/HpiLgwfT0KeB5YEBFfG7yzNSvPfTBm+bkHWFCybWPR85+V7PsZ8Ib0+VEk07kXL0j1U2APcLSk50lWG/1RH2V4uPAkInZJWgsclKn0ZvvJAcYsP1sj4vEBHiv2LdhVqj+Lv+0seR340rhViT9oZrXzyjKvH0ufPwocJ6l4zfZXk/zNPhbJksQrgXm5l9JsgNyCMcvPWEmtJdt2R8Ta9PmbJN1HsvjUm0mCxUnpvhtJBgHcIOljwGSSDv1FRa2iTwD/JmkNcAfQCMyLiH/N64TM+sMBxiw/p5Ks7FhsJXBo+vxS4AySpYXXAm+PiPsAImKrpPnAvwO/JBkscBvwgUJGEXGNpB3APwH/AjwHfC+nczHrN48iM6uBdITXWyLiW7Uui1le3AdjZma5cIAxM7Nc+BKZmZnlwi0YMzPLhQOMmZnlwgHGzMxy4QBjZma5cIAxM7Nc/H/RZ10XkYhjxQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, [piecewise_constant_fn(epoch) for epoch in history.epoch], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Piecewise Constant Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Scheduling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.7116 - accuracy: 0.7769 - val_loss: 0.4869 - val_accuracy: 0.8478\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 2s 947us/step - loss: 0.4912 - accuracy: 0.8390 - val_loss: 0.5958 - val_accuracy: 0.8270\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 2s 987us/step - loss: 0.5222 - accuracy: 0.8379 - val_loss: 0.4869 - val_accuracy: 0.8584\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5061 - accuracy: 0.8467 - val_loss: 0.4588 - val_accuracy: 0.8548\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5216 - accuracy: 0.8469 - val_loss: 0.6096 - val_accuracy: 0.8300\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4984 - accuracy: 0.8546 - val_loss: 0.5359 - val_accuracy: 0.8498\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5104 - accuracy: 0.8579 - val_loss: 0.5457 - val_accuracy: 0.8522\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5375 - accuracy: 0.8538 - val_loss: 0.6445 - val_accuracy: 0.8218\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5333 - accuracy: 0.8522 - val_loss: 0.5472 - val_accuracy: 0.8560\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3280 - accuracy: 0.8902 - val_loss: 0.3826 - val_accuracy: 0.8876\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2410 - accuracy: 0.9135 - val_loss: 0.4025 - val_accuracy: 0.8876\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2266 - accuracy: 0.9180 - val_loss: 0.4540 - val_accuracy: 0.8694\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2129 - accuracy: 0.9221 - val_loss: 0.4310 - val_accuracy: 0.8866\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1959 - accuracy: 0.9270 - val_loss: 0.4406 - val_accuracy: 0.8814\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1975 - accuracy: 0.9277 - val_loss: 0.4341 - val_accuracy: 0.8840\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1409 - accuracy: 0.9464 - val_loss: 0.4220 - val_accuracy: 0.8932\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1181 - accuracy: 0.9542 - val_loss: 0.4409 - val_accuracy: 0.8948\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1124 - accuracy: 0.9560 - val_loss: 0.4480 - val_accuracy: 0.8898\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1070 - accuracy: 0.9579 - val_loss: 0.4610 - val_accuracy: 0.8932\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1016 - accuracy: 0.9606 - val_loss: 0.4845 - val_accuracy: 0.8918\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0848 - accuracy: 0.9686 - val_loss: 0.4829 - val_accuracy: 0.8934\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0792 - accuracy: 0.9700 - val_loss: 0.4906 - val_accuracy: 0.8952\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0751 - accuracy: 0.9720 - val_loss: 0.4951 - val_accuracy: 0.8950\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0687 - accuracy: 0.9739 - val_loss: 0.5109 - val_accuracy: 0.8948\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0683 - accuracy: 0.9752 - val_loss: 0.5241 - val_accuracy: 0.8936\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(lr=0.02, momentum=0.9)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAAEeCAYAAADRiP/HAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABeyElEQVR4nO2deXxU1fXAvycJhE1URFEgJKKCAlVwrxuobbWL1aptVWzVVqm4VG2rtT+1omhbrbW1dasWV3Cvu9W6Je67ghJUFCVUCbsiYSec3x/nDXmZzPIms2U538/nfTJz373n3fcY5sy59yyiqjiO4ziO03pKij0Bx3Ecx2nvuDJ1HMdxnCxxZeo4juM4WeLK1HEcx3GyxJWp4ziO42SJK1PHcRzHyRJXpk6nQ0QaROT4Ys+joyEis0XkN8Weh+MUA1emTptERG4REQ2OdSIyR0SuE5FNiz23XCAiY4J765vk/ITQ/a8XkbkiMkVEKgo912A+x4fmoyJSLyL3iMjWWcpsyOU8HadYuDJ12jJPA1sBVcCJwCHAtcWcUIH5ELv/gcCPga8B9xRxPiuC+fQHjgFGAg+LSGkR5+Q4bQJXpk5bZrWqzlPVz1T1SeBu4FvhDiJygojMEJFVIjJTRM4SkZLQ+W1FpCY4/6GIfC9ufFVgae0a164icmToff/AMlwsIitEZKqI7B86f4iIvBVc51MRuVREumZ5/+uC+5+rqi8ANwJ7ikjvVINE5HAReU9EVovI/0TkPBGR0PnZInK+iPxTRL4Skc9E5OwI89FgPvWqWg1cBIwAtk0yj1+JyLsislxEPheRf4nIJsG5McDNQM+QtTshONdVRC4L5rVcRN4QkYNCcktFZFLwnFeKyEcick7cv/stIvJo3HwmiMj0CPfpOBlTVuwJOE4URGQwcDCwNtR2EnAxcDrwFvbFfmPQ5+rgy/UB4Avg60AP4CqgPMNr9wSeAxYAPwA+B3YKnT8ImAKcATwPDAKuD66Tkz1EEdkSOBxoDI5k/XYB7gUuCea0G/BP4CvgH6GuZwEXAn8Gvg38XUReVNVXMpjWyuBvlyTn1wNnAp8AlcH1/wH8BHg5OPcHYJugf2zJ9+ag7RjgM+A7wCMispuqTsOMgM+BHwELgd2BG4DFwKQM5u84uUNV/fCjzR3ALcA67At2JaDBcVaozxzgJ3HjzgRmBK+/hSmeQaHz+wRyjg/eVwXvd42To8CRweuTgGVA3yRzfR64IK7tsGDukmTMmOAayWROCObegC2vxu7/qjTPbQrwbAJZn4XezwbujOvzEXB+CrnHAw2h9wOBV4D/AV1Dcn+TQsbBwGqgJJHMoG0bTAkPimt/ELg2hew/AU/HfX4eTfAcphf7s+1HxzzcMnXaMs8D44DumELbBvg7gIhsDlQA/xSR60JjyoDYkuYOwOeqOid0/jXsyzoTRgHvquqiJOd3AXYXkd+G2kqCeW8J1Gd4vRizMKusHDgUOAL4vzRjdgAei2t7EbhQRHqr6ldB27txfeYCW6SR3TNwGBLMyn8bOFxV1yTqLCIHAL8L5rQxUAp0xZ7J3CTX2DmQPyO0Mg32DJ4NyT4Z20evxJ5zF6AuzfwdJ2+4MnXaMitU9ePg9S9FpBq4ALMwYvtjJ2NLhomQJO1hYoo1vKcYv2yZTk4Jtn94b4JzCyPMIRlrQvdfKyLbAddgFl0yBLNgExFuX5vgXDofihWY09F6YL6qLk86CZFKTKnfCPweW4LdGbgTU6jJKAnmsluCOa4MZP8Y+Bu2hP4ytoR9KrYEH2M9Lf/dki1HO07WuDJ12hMXAY+LyA2qOldEPge2UdXbkvSfAQwQkQpV/V/QtjvNlUZM2W0VahsZJ+dt4FgR6ZvEOn0b2D6k+PLFROBDEfmHqr6VpM8MbCk7zD7YMu+yLK+vGdzjrpjSPEtVGwHinb+ANZi1GuYdTAluqebklIh9gNdU9epYg4hsE9dnIS3/HePfO07OcG9ep92gqjVALXB+0DQBOCfw4B0qIiNE5Kci8rvg/NPAB8BtIjJSRL4O/BXbi43JXAm8CvxWRIaLyF7AFXGXvgNzPnpQRPYVka1F5Pshb96LgWNE5OJgDtuLyJEicnmE2xoRzC18JPx/qaqfAA9jSjUZfwFGB56rQ0RkLPBrIMpccslH2PfLmcHzOhrbzw4zG+gmIt8Ukb4i0kNVZ2L7vrcEz3CwiOwqIr8RkcODcTOBnUXk2yKynYhcAIyOk/0sMEpEfibm0X0OsHee7tVx3AHJj7Z5kMCBJGg/BnNiqQzeH41Zhqswr90XgaNC/YdgnrirsS/472NOPceH+uwAvIQtY74H7EvIASnoMxALzfky6PcOMCZ0/lvAC8G5r4A3gdNS3N8YmpyK4o9eJHGWAfYK+uyVQvbhwX2swRyEziPkCEUCRyGgBrg6hczjiXMWStCnmVzgl5jX7UrgGcz7VoGqUJ/rgEVB+4SgrUtw/58E9zAP+xGxS3C+K+a1+0Xw7zEJW0qeHTefCdh+9VIsPvkPiZ6pH37k4hDVZNsrjuM4juNEwZd5HcdxHCdLXJk6juM4Tpa4MnUcx3GcLHFl6jiO4zhZ4nGmESkpKdHu3bsXexptjvXr11NS4r/J4vHn0hJ/Jonp6M9lxYoVqqod9wYDXJlGpGvXrixfnjThS6elpqaGMWPGFHsabQ5/Li3xZ5KYjv5cRGRl+l7tnw7/a8FxHMdx8o0rU8dxHKe4iPRB5AFEliNSh8gxKfoORuRRRJYhsohwpjGRGkRWIdIQHB8WYvrgytRxHMcpPtdg2a76AWOB6xAZ3qKXSFfgKSxd5JZYZrLJcb1OQ7VXcAzN66xDuDJ1HMdxiodIT6y84AWoNqD6IpY+8icJeh8PzEX1SlSXo7oK1fhygkXBlanjOI6TN/pCGSJvho5xcV2GAI1YkYMY04CWlinsCcxG5PFgibcGka/F9fljcO4lRMbk7EbSUFBlKkIfER4QYbkIdSIkXRcX4SwR5omwVISbRCgP2stFmBSMXybCOyJ8O27sgSJ8IMIKEapFqAydExEuE2FxcFwukr7u5erVpVRVwZQp0e51yhSoqoKSEjr0uAMOGF2w6+02sJ7nZDS7VczL+/Ucx8kNi2AdqruGjhviuvTCihGEWQpslEDcQOAo4O9Af6xm7kPB8i/Ab4HBwADgBuARWpbnyw+FzKoPeifo3aC9QPcBXQo6PEG/g0Dngw4H3RS0BvRPwbmeoBNAq0BLQL8Hugy0KjjfN5D7Q9BuoH8GfTUk+xegH4IOBB0AOgP05PRz76Gg2qOH6uTJmpLJk60fNB0+Lvtx1zBe11GiV3NKXq+XC6qrq/N/kXaGP5PEdPTnAizXVN+vMEphRVzbrxUeSdD3IYXq0HtRWKqwUxLZTyicnvL6OToKFmcqQmxdfIQqDcCLIhvWxc+N634cMEmV2mDsRKzG4bmqLMdKK8V4VIRPgV2wElCHA7Wq3BuMnQAsEmF7VT4IZP9Flc+C838BTgKuj3IfK1bAqafChyl8xP7+d+vn43I3rveKen7OvyhlPSdwMxNXXMCpp27Zquuddx6MHZt8nOM4BWUmthS8HaofBW07YbWL43mXzOrSKqRfecwFBSvBJsIo4GVVuofafgOMVuWQuL7TgD+ocnfwvi+wEOiryuK4vv2AOmCkKh+IcBXQVZXxoT7TgQtV+bcIS4FvqfJacG5XoFq15ZKCCOOAYH2/5y4QS9qgSIp/HnukiTr4uNaOu4mfcQK3ALCKrkziRE7j6lZdT0R59tnnkg/MAQ0NDfTq1Suv12hv+DNJTEd/Lvvvv/8KVe2ZspPIXZjiOxEYCfwH2AvV2rh+Q7Fawt8HqrGauadhNYl7AHtg9YvXAT/Glnp3RjX/ITKFMH9NYeu+oPPi2k4CrUnQdxbowaH3XYJluqq4fl1Anwb9Z6htUmxJONT2EujxwetG0O1D57YLZEvq+ffYsFRYWZlkPSOgslKbLS36uOzG7Tpgrq6ia7NBy+muuw6sz8v1ckFHX7prDf5MEtPRnwvplnlVUeij8KDCcoU5CscE7YMUGhQGhfoervCxwlcKNQrDg/bNFd5QWKbwpcKrCt9Me+0cHYV0QGoAese19QaWRegbe72hrwglwO1YbNJpGVwnkewG+7dIT48ecOmlqftceqn183G5GTdl+4mU0tisrYRGJg+dmJfrOY5TYFSXoHoYqj1RHYTqHUH7HCxedE6o7/2obotqb1THELNeVReiuhuqG6G6Cap7ovpUAe+hYJZpT9A1oNuF2m6LtyKD9jtALw29PyBs1YIK6M2g1aDd48aOA30p7rorYtYo6MugJ4XO/4yQg1Ly+ffQysroziuTJ5sFJKIdfNz6/F9v5MjEJubIkZGu1717k0VaCOcj1Y5vbbQGfyaJ6ejPhSiWaQc4Cnsx9C7Mo7cn6N4k9+Y9GHQe6DDMm/fZsNIFvR70VdBeCcZuHsg9AvPmvYzm3rwng76PefL2B60lgjdveXl58k9LJ6ZgXwQ77KDar599ZB95JKOhZ56p2rOn6vr1eZpbAjr6F2Rr8GeSmI7+XDqLMi100oZTgO7AAuBOYLwqtSIMEqFBhEEAqjwBXI5tMNcFx4UAQczoL7BN6nnBuAYRxgZjF2Jew5cCX2Ab0keF5vBP4BHgPWA6Fqf0z3zetJMlK1aYu+8xx4AITJ2a0fCqKli+HBYvTtvVcRynVRS0BJsqS4DDErTPwQJ3w21XAlcm6FtHGldnVZ4Gtk9yToFzgsNpD0yfDuvXw777wqOPwjvvZDS8stL+1tVB3755mJ/jOJ0eTyfotH1ilujIkXa0wjIFmD07ZzNyHMdphitTp+0zdSr07m1acdQo+OQTWBqffSw5rkwdx8k3rkydts/UqWaRitjfWFtENtnEdHFdXe6n5jiOA65MnbbO+vXw7rtNSnTUKPvbiqVet0wdx8kXrkydts2sWeaKG1OmW25pRyuckFyZOo6TL1yZOm2bsPNRjJEjM1amVVW2zKuFSUXtOE4nw5Wp07aZOhXKymDYsKa2UaNgxgxYvTqymKoq+Oor+PLLXE/QcRzHlanT1pk61RRpeXlT26hRsG4d1Caq0JSYcKyp4zhOrnFl6rRtpk6FnXZq3hZb8s1gqdfDYxzHySeuTJ22y4IFMHdu8/1SgG22gV69MvLodWXqOE4+cWXqtF2mTbO/8cq0pMSs1Qws0z59oGdPX+Z1HCc/uDJ12i4xyzN+mRds33TaNItDjYCIx5o6jpM/XJk6bZepU6GiAjbbrOW5UaOgocHiUCNSWemWqeM4+cGVqdN2iaURTEQrnZDcMnUcJx+4MnXaJitXwgcfJFemw4db/GmGTkhffGHxpo7jOLnElanTNqmttf3QZMq0vNwUagaWqceaOo6TL1yZOm2TRGkE48kwraCHxziOky9cmTpQXw+jR8O8ecWeSRPhGqbJGDUK5s+PPO+YZerK1HGcXFNQZSpCHxEeEGG5CHUiHJOi71kizBNhqQg3iVAeOneaCG+KsFqEW+LGjRWhIXSsEEFF2CU4P0GEtXF9BuftptsD//d/8OKLMHFisWfSRCzzUUmKj2isHFtE63SLLaBbN1/mdRwn9xTaMr0GWAP0A8YC14kwPL6TCAcB5wIHAlXAYOCiUJe5wCXATfFjVZmiSq/YAZwCfAK8Hep2d7iPKp/k5O7aI/X1cOuttj95881twzpdv95iSFMt8UJT/GlEZeqxpo7j5IuCKVMRegJHABeo0qDKi8DDwE8SdD8OmKRKrSpfABOB42MnVblflQeBxREufRxwmypefCsRF13UVJessbFtWKeffGIxpImSNYTZeGMYPDgjj16PNXUcJx+UFfBaQ4BGVWaG2qYBoxP0HQ48FNevnwibqUZSoACIUAnsB/ws7tQhIiwB6oGrVbkuyfhxwDiAsjKhpqYm6qXbBV0XL2bPm25q+kW1Zg2Nkybx2oEHsqZPn0gyGhoacv5cNn/uOYYDb65bR0Ma2cMHDKDnyy/zesQ5dO06hI8+6ktNzctZzzMV+Xgu7R1/Jonx59JBUNWCHKD7gs6LazsJtCZB31mgB4fedzHzSavi+l0CekuKa14QLx90GGh/0FLQvUDrQY9ON//y8nLtcIwfr1pWpho8XAXVrl1VTzklsojq6urcz+u881RLS1VXrkzfd+JEm/dXX0US/Yc/WPeGhiznmIa8PJd2jj+TxHT05wIs1wLpmWIehdwzbQB6x7X1BpZF6Bt7nahvKn4K3BpuUGWGKnNVaVTlZeAq4MgM5XYMXnnF6oKGWbMGXs6v1ZaWqVNhhx3MWygdMSekWFL8NMScg32p13GcXFJIZToTKBNhu1DbTkCiCs+1wblwv/kZLvHuDfQH7kvTVQGJKrdD8c47MGGCvf7BD6B/f3P+ySB2My+kSiMYT4ZpBT3W1HHaICJ9EHkAkeWI1CGSNNIDkcGIPIrIMkQWIXJ5q+TkmIIpU1WWA/cDF4vQM1B2hwK3J+h+G/BzEYaJsClwPjSFwIhQJkI3oBQoFaGbSIv93+OAf6s2t2ZFOFSETUUQEXYHfknz/dnORV0dbLUVHHSQ1Q79+OPizmfRIvj88+jKtH9/2HzzyE5IHmvqOG2SFpEeiLSI9ECkK/AU8CywJTAQmJyxnDxQ6NCYU4DuwALgTmC8KrUiDAriPQcBqPIEcDlQDdQFx4UhOecDK7HwmWOD1+fHTgaK9kfELfEGHAV8jC0Z3wZcppqwX+dg9mwz18aMsffFdoRIVsM0GSK21BvRMt1yS+ja1Zd5HafNILIh0gPVBlRTRXocD8xF9UpUl6O6CtV3WyEn5xRUmaqyRJXDVOmpyiBV7gja56jFe84J9b1SlX6q9FblBFVWh85NUEXijgmh86tU2USVZxLM4WhVNguut70qf8/bDbfFzELx1NWZuTZkiGmaYivTVDVMkzFyJEyfbvu9aSgpsdt1y9RxCkNfKEPkzdAxLq7LEKAR1fhIj0QW5Z7AbEQeD5Z4axD5Wivk5BxPJ5hPJk5se5mFwjQ2wv/+Z9pFxKzTmpqmuNNiMHUqDBwIfftGHzNqFKxdC++/H6m7x5o6TuFYBOtQ3TV03BDXpRewNK5tKbBRAnEDsdXFv2M+MY8BDwXLv5nIyTmuTPNFfT38619tK7NQPPX1poRiXjn771/8fdNMnI9iZJhW0LMgOU6bIpNIj5XAi6g+juoa4ApgM2CHDOXkHFem+WLiRFNU0HYyC8UTM89iXjnF3jddtcqsy0yV6bbbQo8ekZVpZaXlx1+5MvMpOo6Tc2ZiS8FRIj3ehaTZ7DKRk3NcmeaD+nqzRmOsWdM2rdN4ZbrddubZW11dnPnU1toPj0z2SwFKS21MRI/emCE+Z07Kbo7jFALVDZEeiPREJFWkx2RgT0S+gUgpcCawCHg/Qzk5x5VpPpg40ZRCmLZoncbWOmPKtNj7plFqmCZj5Egbv3592q4ea+o4bY4WkR6o1iIyCJEGRAYBoPohFsFxPfAFpiy/Hyz5JpdTAFyZ5oNXXmla4o3RFjILxVNXZ44+PXs2tY0ZY5b1Rx8Vfj5Tp0KvXpa8PlNGjYKvvoqkIT3W1HHaGKpLUD0M1Z6oDkL1jqB9Dqq9UJ0T6ns/qtui2hvVMc2UZTI5BcCVaT545x04/3xbfuzWDc480yy9YmcWiicWFhOmmPumUWqYJiMDJ6T+/aGszD16HcfJHa5M80VtrTnGDBtmr9sisYQNYWL7poVWplFrmCZjxAj78RJBmZaWQkWFW6aO4+QOV6b5orbWvuBHjGibylTVPHDiLdNi7ZvOng3LlrVemXbrZsnxM3BCcsvUcZxc4co0H6xaZbGaw4fbMXcufPFFsWfVnIULLTYk3jKF4uybZuN8FCODtIIea+o4Ti5xZZoPPvjAli1jlim0Pes03pM3zP77299CLvVOnWrrr8OzyPw1cqT9cFmwIG3Xykrrunp12q6O4zhpcWWaD6ZPt78xyzTc1laIjzENs+225qVTyHjTqVNh++2he/fWy4g5IUVY6o0Z5P/7X+sv5ziOE8OVaT6orYUuXcyZZ9AgC/doT5ZpMfZNW5NGMJ5YsocIS70eHuM4Ti5xZZoPpk+HoUNNoYqYddoWLdONN4ZNNkl8fswYy9g0c2bi87lk8WIzEbNVpn36mJaMoEw9cYPjOEkR6ZLpEFem+SDmyRujLXr0JooxDVPIeNNMa5imYtSoSMu8AwfaFq179DpOJ0fkl4gcEXo/CViJyIeIDI0qxpVprmlogE8/be5IM3y4ec9GcIwpGIliTMPE9k0LoUxbU8M0GSNHmjXd0JCyW1kZDBjglqnjOPwSWAiAyH7Aj4BjgKnAX6IKcWWaa2I1NeOVKbQd61Q1vWVayH3TqVNNcW++efayRo2y+b73XtquHmvqOA4wAJgdvD4EuBfVe4AJWDHySLgyzTWxvdH4Zd7wuWLz5ZeWICGVMoXC7ZvmwvkoRgZpBT3W1HEc4Csg9kv+m8Azweu1QLeoQiIrUxG+LcKjIswQoSJoO1GEA6PK6BTU1lo2nnCy9q22MkeftmKZxjRIqmVeKMy+6erVrathmoyBA80RKaJH7+eft6xJ4DhOp+JJ4MZgr3Rb4PGgfTjwaVQhkZSpCGOBe4CPgK2BmKdTKXBO1IuJ0EeEB0RYLkKdCMek6HuWCPNEWCrCTSKUh86dJsKbIqwW4Za4cVUiqAgNoeOC0HkR4TIRFgfH5SJI1HtIS22tpbUrLW02qTblhJQqxjTMttvaxmI+401nzIB163KnTEUiOyFVVVlujc8+y82lHcdpl5wKvAT0BY5EdUnQvjNWxi0SUS3Tc4CTVDkLWBdqfxUYGfViwDXAGqAfMBa4ToQWKW9EOAg4FzgQqAIGAxeFuswFLgFuSnGtTVTpFRzhQqLjgMOwCuw7At8DfpHBPaRm+vTEWXxi4THFqBMaT0yZprNMC7Fvmos0gvGMGmV7pmlMTo81dRwH1a9QPR3VQ1F9ItR+Iap/iComqjLdDnglQXsD0DuKABF6AkcAF6jSoMqLwMPATxJ0Pw6YpEqtKl8AE4HjYydVuV+VB4HFEecfL/svqnymyueYt9bxqYdEZOlSM3PC+6UxRoywvcr6+pxcKitmz4YePWCzzdL3HTMG5s+HDz/Mz1ymTrV6qttskzuZI0fa8vEHH6TsFvst4U5IjtOJERnWLARG5JuITEbkd4iUphjZjLKI/eYCQ4D4r539gFkRZQwBGlUJe7NMA0Yn6DsceCiuXz8RNlONrEDrRFDgKeBsVRaFZE+Lk50wIawI4zBLlrIyoSbN3mHv6dPZGXhv/XoWx/XdZO1aRgLTpkzhi912i3gL+WH4W2/RY/PNeeO559L27d69O3sAM2+4gbnf/36L8w0NDWmfSypG1tQgVVW88/zzrZYRT4+1a9kdeP/OO5m/OPnHZe1aQWQ/amrqqKqanbPrQ/bPpSPizyQx/lyKziTgKuBDRAZiuqcGW/7tDfwukhRVTXuAngP6PujeoMtAR4MeB7oQ9NSIMvYFnRfXdhJoTYK+s0APDr3vYuuMWhXX7xLQW+LaeoHuCloG2g/0PtD/hs43gm4fer9dIFtSzb+8vFzTcsMNqqD66actzy1YYOeuvDK9nHyz886q3/52tL7r16sOGKD64x8nPF1dXd36eaxfr9q7t+opp7ReRiLWrlXt1k31rLPSdh0wQPW443J7edUsn0sHxZ9JYjr6cwGWawQdUbQDvlQYErw+S6E6eL2/wuyociJZpqpcLsLGmJXXDagGVgNXqHJNNOWfcEm4N7AsQt/Y60R94+faALwZvJ0vwmlAvQi9VfkqiewGe3ZZMn26LVkOGtTy3Oab29EWwmNmz4bdd4/WN7Zv+vTT9ptDcuerxezZ8NVXud0vBcvIsOOOkcNjfJnXcTo1pZgvD5ifzn+C17Mw/55IRA6NUeU8zNtpdyyQdXPVJi/ZCMwEykTYLtS2E5DIxbU2OBfuNz+DJd4wMSUZ0wKJZOfGzba2FoYNg5Ikj7UtePQ2NMCSJek9ecPka980H85HMWIevWkcpyor3QHJcTo504HxiOyLKdOYE9IA2LA9mJaooTE3ibCRKitUeVOV11VpEKGnSEqP2g2oshy4H7g4GLc3cChwe4LutwE/F2GYCJsC50NTCIwIZSJ0w35RlIrQTcSsbBH2EGGoCCUibAb8HahRZWlI9q9EGCBCf+DXYdlZMX16YuejGMOHmzLNl2dsFKJ68obJV7zp1Kn2wyPVM2stI0eaw1cas7OqynLsr1uXspvjOB2X3wInYfukd6IaS5/2feD1qEKiWqbHAYkKTXYHfhr1YsApwZgFWPzOeFVqRRgUxIMOAlDlCeBybDm5LjguDMk5H1iJhc8cG7w+Pzg3GPtlsQz7xbEaODo09p/AI8B7wfnHgrbsWLzYrLdUxa2HDzfLcM6crC/XalKVXkvGNtvkJ9506lSrrpNNDdNkRKxtWlUFjY1WKNxxnE6I6vNYBqS+qP4sdOafwPioYlLumYrQB1seFWBTkWYxpqXAd4H50efMEizGM759DtArru1K4MokciZgeRMTnbuTFIG2wd7oOWSQbCISseXbVFZW7FxtbWbKLJdETdgQRgT23x+efDK3+6ZTp8Lee+dGVjxf+5pZve+8A4cdlrRbONY00Va34zidANVGRFYiMgLbGpyF6uxMRKSzTBdhVqQCM7DM+rFjHvAv4NoMp90xiTkWpbNMw32LwezZ0LUrbLllZuPGjLGqN2liNyOzZIlZ6PnYLwWLox06NK0TkseaOk4nR6QMkT8DX2Chku8BXyByeSZ1TdN58+6PWaXPYgkXloTOrQHqVPEFMjBrc+ONbTk0GZtuatVRiumEVFdnJlgyJ6lkhPdNd9gh+3m8+679zZcyBVvqfeGFlF1i1qg7ITlOp+VybCvwZODFoG1f4I+YwfmbKEJSKlNVngMQYWvgf6qsb+1sOzyxNILplkBjaQWLRbrSa8kYPNiSyNfUwPjI2wjJyWUN02SMGgV33GH72UmyPXXrZka6K1PH6bQcA/wM1f+E2mYhshBbfY2kTCOZJ6rUqbJehP4i7CnCfuEj87l3MFTN2ozilTpihFVJaWzM/7wSka4oeDJynad36lSrptMvchhX5sSs3ghLvb7M6zidlo1JnMlvFrBJVCFRQ2P6i1ADfIZl16/BPG1jR+dm/nyzflLtl8YYPhxWroRPI1f2yR2rVtlcW+v8lMt901zWME1GTH4aj16PNXWcIiPSB5EHEFmOSB0iiSuKiRyPSCMiDaFjTOh8DSKrQueiBMdPA36ZoP0MYGrUW4i6cfY3oBEYBqzA1pN/CLwPHBz1Yh2WKJ68McIevYUmFpLTGssUchdvumaNlV7LtzLt29eWpiNYpnPmWDk2x3GKQouKYogks05eQbVX6KiJO39a6NzQRALiOAc4DpGZiNyKyC2BEj4WODvqDURVpqOB36ryAebZu1CV+7Fg14kpR3YGonjyxhg2rPmYQtKaGNMwsX3TbONNZ8yw8mj5VqYQqbZpVZVNpy0U9HGcTofIhopiqDagmqqiWO6xONMhwL1YiGbv4PXQYC6RiKpMu9OUVmkJsEXwegZWE7RzU1trDi5bbJG+70YbmTIrhmXamhjTMLnaN81nGsF4Ro2yPep994V58xJ28bqmjlNUhgCNqMZXFEtmnYxCZFFgSV6ASLwj7R+D8y81WwJOhepcVM9D9QhUD0f1fKALIvdEvYmoyvQDYPvg9VTgZBEqsRI1n0e9WIcl5nwUNZlBLK1goamrg9LS1OE76dh/f1i40BRUa3npJQvN6dmz9TKiMnKkKf6XXoKJiRdRPNbUcfJHXyhD5M3QMS6uSy/YkO41xlJgowTingdGYAbdEVhIS3gp9rdYFrwBwA3AI4i0tljyJsE1IhFVmV4FxKL8Lwa+BXyCpQf8vwwm1/FQbQqLicrw4ebEU+iEsLNn2zJtWdQytgnIxb7pY4/ZBuUfIhexbz2xHw6qcPPNCa1TjzV1nPyxCNahumvouCGuS/SKYqqfoPopquuDHLoXA0eGzr+G6jJUV6N6K+Yw+51c3k8yoobGTFG1ZPCqvA1UAbsBg1S5N2+zaw98/rmVEcskWfuIEeaE8/HH+ZtXIlobYxpm662hoqJ1ynTdOrjiiqbNySTKLafcFKrD0NiY0Drt2dOq47kydZyiMBOzXqNUFItHaaoI1przOSPDNDhGUD3mbWC5COfmeE7ti0ycj2IUK61gXV3rPXljtGbfVBX+/W/7EXH22U3L4UmUW86or4dbb216v2ZNUgXusaaOUyRUN1QUQ6QnIskriol8G5F+wevtgQuAh4L3myByECLdghSBY4H9gP8W4jbSKlMR+orwXRG+JUJp0NZFhDOB2UTMDtFhie19ZqJMd9jBFEoh903XrjUrOhcJ9seMib5v+uyzsMcecOSRtrTbtWuTEk6h3HLCxIkt412SKHCPNXWcotKiohiqtYgMCuJFY2UoDgTeRWQ5VsT7fiC2X9QFuATLHb8IOB04DNXEsaYiD6c8bHszMimVqQh7AR9hJcseB14SYXvgXeA0LCymc9famD7d8tElSVeXkB49LMykkJbpZ5+ZYsmVMoXUS71vvQXf+hYceKApy5tvhgMOaNkvn9bpK6+Ywg6zZg28/HKLrjHL1GNNHacIqC5B9TBUe6I6CNU7gvY5QbzonOD9b1DtF/QbjOrvUV0bnFuI6m6oboTqJqjuiepTKa66OM3xKVb/OhLpPFEmYibyJcDPgDOBR7FN39uDcmadm6hpBOMZMaKwlmlrioInI7xvGoubjTFzJlxwAdxzj/3AuPJKy+XbrRtcdVVk5ZYTYskazj4b/vEPyzyVxOO6shJWr7YET5kW1HEcpx2iekIuxaVb5t0JmKjKdKz4tgK/U+U2V6SYGVNbm9kSb4zhw03xrF6d+3klItuEDWFi+6bPPMPIM84wy3PuXDj5ZFOujz1mCnXWLDjrLFOkYMpNteWRJkNR1lRU2HNetChpl9hvDF/qdRynNaRTpn2w9WdUWYGlEszzN187oq4OVqxonTIdMcKWOGfOTN83F9TVmRKsqMiNvDFjYMkSNn7vPfjOd2DbbWHSJFOos2bBxRdbSbq2QOye//e/pF081tRxnGyI4s27qQh9RNgMs0x7B+83HHmeY9sltufZmmXemAIu1FLv7NlWpaW8PDfygvlLzLI8+GD48EO4+ur8VoJpDRGUqWdBchwnG6Io0xmYdboAy1TxRvA+5jG1MOrFAuX7gAjLRagTIXFlAOt7lgjzRFgqwk0ilIfOnSbCmyKsFrH419C5PUV4SoQlIiwU4V4RtgqdnyDCWhEaQsfgqPfQjJgijN83jMLQoZaNqFBOSLmIMQ1zyy1N+49dupiiHty6x5h3Bg60vymU6UYbQZ8+rkwdx2kd6ZTp/sABoSPZ+6i0qAwg0jL/oggHAedibtBVWHqoi0Jd5mJOUTfFjwU2xdJIVQGVWBaNm+P63K1Kr9DxSQb30ERtrVk9rVnOLC+H7bYrnGWaixjTGPX1pkxjIS5r1xYmAUNr2WILU/iffZaym8eaOo7TWlJ686ryXK4uJEKsMsAIVRqAF0U2VAaIT/xwHDBJ1TJgiDARmBLrF1SsQYRdgYFxc3487rpXQ+7uoxmZphGMZ/hwmDYtd/NJRmOj1Rj70Y9yIy9V/OY11+TmGrmkpMSs0xSWKZjhnotSrY7jtDNEegAjsZy/zY1M1fujiMgiSWvGDAEaVYmvDDA6Qd/hxLJaNPXrJ8JmqizO8Lr70TIt1SEiLAHqgatVuS7RQBHGAeMAysqEmnBcZWMj+9XW8tnQoXzSyjy1Vb16UTlrFi/897+sz9VeZgLKFy7k6+vWMXP1auZmW4sU2OWpp9goQYjLsief5K0cyM8HIzfaCKZPZ2qK+ZWWbsMnn/SnuvqFyDULktHQ0ND88+L4M0mCP5ciI/INLFFEomQBCpasKC2qWpADdF/QeXFtJ4HWJOg7C/Tg0PsuQRxFVVy/S0BvSXHNHUGXgO4bahsG2h+0FHQv0HrQo9PNv7y8XJvx4YcW2HHzzdpq7rnHZLz1VutlROHFF+06jz+ec9HV1dU5l5kXjjlGtaoqZZe//c0e04IF2V+u3TyXAuLPJDEd/bkAy7VAeqZVB9Qq3KLQPxs5rcrN20qiVwZo2Tf2OlHfhIiwLZa16QxVXoi1qzJDlbmqNKryMpYy6shkcpIS2+tsjSdvjNjYfO+b5jLGtL1SUWHpFFOkOPJYU8fplFQBE1Gdm42QQirTmUCZCFEqA9QG58L95kdd4g1qrT6NJZxomSy5Oa2rKhDzwt1hh4yHbmDbbc0xJt8evTGvmkGdOPNjRYU5Si1YkLSLx5o6To6pr2codCv2NNLwEjA0WyEFU6aqbKgMIEJPEZJXBrB8iD8XYZgIm2LZl26JnRShTIRu2Fp2qQjdRGz/V4QBwLPANapcHy9YhENF2FQEEWF34Jc035+NRm2thYJkU+C6SxfYfvv8W6Z1dVZjrBDFuNsqHmvqOIVn4kR6FdZoaw3XA1cgciIieyCyc7MjIpEckEQShqCAWXWrgI+xcJN0ZvIpWDjLAiyR8HhVakUYhMWzDlNljipPiHA5UI1VEvg3cGFIzvlx74/FQmcmACdioTQXijT1UaVX8PKoYA7lwGfAZaqE6nRFJFtP3hjDh8Orr2YvJxWzZ3fuJV5oijX97DPYbbeEXTbZxKKcXJk6Tg6oq4Mbbyz2LKJwX/A3vmg5ZOCAFNWbd3NgX2A9EFuTHIEtj74FHI5ZnPuqMjWZEFWWAIclaJ8DG5RdrO1K4MokciZgijPRuYtoHpMaf/7oZOcis3atpQE85JCsRTFiBNx1FzQ0QK9e6fu3hro6+NrX8iO7vRDBMgX7zeHLvI6TJR9+CPvsA+vWFXsmUdg6F0Kimt8vYc48A1XZT5X9sPjO/wBPYskRHgP+kotJtXk++sgUajbORzFi1u2MGdnLSoRq7rMftUf69rWE+2mUaVWVW6aO02pUzRodNSplYYk2hWpdyiMiUZXpGcDFasnug+uzArgUOEuVNcBlWNBrxyfmMJSrZd6wzFyzYAGsWuXKVCRS4oZYFiT1mkiOkxmLF8MRR8C4cZabs2vXYs8oOiI7InIbIm8i8gYityKS0XJeVGXaC5ry24bYkqbl2a8obBKI4lFba1l1tt8+e1mDB5vFlC8npFzWMW3vDByYNqVgZSUsWwZffFGgOTlOR+CZZ2DHHeHRR+GKK2wlKD6xS1tF5PvA20AFtgL7BDAIeBuRyHt5UZXpA8AkEX4oQpUIlSL8EJiEeegC7A4UqJ5YkZk+3cJauuXA47u01MJr8mWZxpRpZ7dMwfZNI1im4Eu9jhOJNWvgnHPgm9+E3r3h9dfh17+GqVM31Ct+i6YVzTbKJcClqO6P6gXBsT/wx+BcJKIq05OB/wKTgVnAJ8HrJzAPXYD3gZOiXrhdU1ubm/3SGCNG5M8y9YQNTcQSNzQ2Ju0Se0zuhOQ4afjgA9hzT/jzn62O8VtvwciRxZ5VaxhC4hDN28kg/jSSMlVlhSonY8XCRwE7A31UGR/Ej6LK1FSevB2GVavMASkX+6Uxhg+3L/kvv8ydzBh1dU0xH52digpTpCmq27hl6jhpUIV//hN23tlWeh56CK69Fnr0KPbMWssCYJcE7bsA86MKySiYVpXlqryryrSYEu10fPihpaTLtWUK+bFO3ZO3iXCsaRL69LEIJVemjhOivh5Gj7bvqB/8wCzRffaBd9+F73+/2LPLlhuBfyJyHiL7IzIGkfOxZA6JYk8TEjVpQzfMo/dAEpSoUWXHqBds9+TSkzdG2KN3771zJxdMK7TVot2FJhxrusceCbuIeKyp47Rg4kR44QXYfXeLHb3ySjjjDHPEbP9cguWD/zUwMWibiyUG+ntUIVG9b68FfgDcC7yMZYXonNTWQlmZFfbOFYMGmTmUa8s0FmN6QCb12zswERM3eKyp44T47DOLHVWFlSvhySfhG98o9qxyh1W2+SvwV0Q2CtoiF1WJEVWZHgb8UJWnM71Ah6O2FoYOzW0MVUkJDBuWe2X6xRcW5+HLvMamm0L37pGU6UsvFWZKjtOmaWiAffdtymTUpQs88EDHUqZhWqFEY0S10VcAqb+BOgu5yskbz4gRuQ+P8bCY5oiYdRoh1vTLL/PjD+Y47YbPP4evf735Ms2aNXDzzSmd+NoFIu8ismnw+r3gfeIjIlGV6eXAr0TafPb//LJ8OXz6aW6dj2IMH27ZihYuzJ1MT9jQkgxiTX3f1Om0vPOO7Y9+8IFta4VpbLQ91PbNv4HVodepjkhEXeb9Jpbo/mARZgBrwydVaffuXJF4/33bN8iHZRqTWVsLY8bkRqbHmLakogKeeipll3Cs6U47pezqOB2PRx6Bo4821/ZttrEIhjBr1sDLLxdnbrlC9aLQ6wm5EBnV0lyEZUF6FpiHlU8LH52D2J5mPizTfITH1NVZDdPNNsudzPbOwIHm5p+imoXHmjqdElX429/g0EMtK9trr5llGmQyana8806xZ5s7RJ5FZJME7b0ReTaqmEiWqSonRJ9ZB2b6dCgvt19ruaZ/f0uskMt901iMqUjuZLZ3KiosTri+vsm7N47NNzc/JVemTqdh3ToLdbn2Wjj8cLj99sImYRDpg6Wn/RZmvP0O1TsS9Ds+6Lcy1Po9VGsyktOcMUAij9Ju2IpsJDpHYvpcUVtrv9hKI9WKzQyR3KcV9KLgLQmHxyRRph5r6nQqvvoKfvxjeOIJy7P7xz8WI370GmAN0A+rPvYYItNQTfSF+Aqq+2QtR2Tn0LsdEVkSel8KHAR8HvUGkipTEd4FRqvyhQjvkSK2tNMkbZg+HfbbL3/yhw+He++1ZZRcWJN1dZY702nCY00dp4m6Ovje92w598Yb4cQTCz8HkZ7AEcAIVBuAFxF5GPgJcG4e5byJ6TXF6nLHsxI4PerlU1mmYW+n+6IK7KiUgH0B58P5KMaIEXDDDeZ2vlWiincZsGwZLFnilmk8sZSCaZRpZSW88UYB5uM4xeL11y0V4KpVZpUeeGBeLtMXyhB5M9R0A6rhNH1DgEZUw1XHpgGjk4gchcgiYAmWjP6PqK5rhZytAcEKt+wOhEMp1gALUE1eFSOOpLa8KhfFioEHr5MeUS8mQh8RHhBhuQh1IhyTou9ZIswTYakIN4lQHjp3mghvirBahFsSjD1QhA9EWCFCtQiVoXMiwmUiLA6Oy0VIawZ2W7/eXuTD+ShG2KM3WzzGNDEbb2zZptLEmn7xhdU6LikxK3XKlGjip0yx/gccMLpV4zK5XmvGFGOc04aI5di98Ub726MHvPJK3hQpwCJYh+quoSM+320vYGlc21JgowTingdGYGltjwCOBs5uhRxQrUN1NqolqL4ZvI8d9Zko0kCeFuwAvRP0btBeoPuALgUdnqDfQaDzQYeDbgpaA/qn0PnDQQ8DvQ70lrixfQO5PwTtBvpn0FdD538B+iHoQNABoDNAT0439y3LysyPbdYszRvz59s1/vrX7GU9+qjJeuWV7GWloLq6Oq/y88IOO6gefnjS05Mnq3bt2tx9sUcPa0/F5MnWrxDjCnmtbMaFaZeflQJQ0Ody8smqIvYPuNdeqgsW5P2SwHJN9f0KoxRWxLX9WuGRlOOs31EKb+VATpnCXoG8nzY7Iuq3qInu+wCXkjzRfe8IMjasZ6vSALwoQrL17OOASarUBmMnAlNi/VStILkIuwID48YeDtSqcm/QZwKwSITtVfkgkP0XVT4Lzv8Fq8N6far5d1O1X3H5TICwxRZWod4t0/ySJnHDeedZKF2YFStg/Hh4++3kYm+80foVYlwhr5Vq3Hnnwdixycc5bYDGRivW/eijto2kak6UU6aY63rxmYktBW+H6kdB205AlC9ChQ0ri62TI7I98AhNy76N2BboWmyr87YoNxHVm3cSVsf0BiybfmsS3Q8BGlWJsp49HHgorl8/ETZTTRvXOjzoD1jZOBFmBe0fxJ8PXifcCBVhHDAOYIgoX1VU8Pbzz6e5fHbsNHAgJS+/zDs1NVnJGfz88wzs0oXn33+/ZdB1DmloaKAmy7kWmqGlpfSZNYtXksx7zpzRkGDlf9ky5brrkq/8rFxZWrBxhbxWqnFz5ig1Nc8lHRemPX5WCkFrnkvXxYsZdvHFzLjwQtb06dPsnKxbx0YzZ7LxtGlsMm0aG0+fTtlyq5gZ0zzrRag/80w+OvPMnNxDVqguR+R+4GJETsS8cA8F9mrRV+TbwNuozg+U4AVYAZbM5DTnb8BbQf95wd+NgeuA8zO4j0jLs1+B7hHV3E0iY1/QeXFtJ4HWJOg7C/Tg0PsuwdJSVVy/SxIs804KLwkHbS+BHh+8bgTdPnRuu0C2pJr/TqB6/PHR1jWy4dRTVTfaSHX9+uzk/OhHqtttl5s5paBdLt1deKEtda1enfB0ZWXz5czYUVmZWmwhx7WHOcbTLj8rBaBVz2X8eNWSEtVTTlFduVL1uedUJ05U/cY3mq/H77CDLe1ec41qeXnzf7zu3VXr63N+P/GQbpnXlln7KDyosFxhjsIxQfsghQaFQcH7KxTmB/0+UbhYoUtaOamvvVhhRPB6qcLQ4PVohXfTjg+OqMFEC7B6b9nQAC2Wg3sDibL0x/eNvY6S0T/ddRLJbrBnl5wyyK8nb4zhw80TN423aVq8KHhyKirs62Tu3ISnL720Zbx6jx7WnopCjmsPc3TyxNy5cNNNlnzk+uvNqW70aPj97y23989/DvfdB/Pnw4wZcN11FtancV9xbSnHruoSVA9DtSeqg4glWlCdg2ovVOcE73+Dar+g32BUf4/q2rRyUiNYMRcwj94BwevPgG2j3kJUZXoecLEIvaIKTsBMoEyEcCHQZOvZtcG5cL/5EZZ4W4wN9mq3CV0nkexom5T9+0fqlhUxb+Hvfje7ygyzZ3uC+2SkiTUdO9a2lmLJoyor7X26vcHm47SV46JdLzdzbN04sLHXX+/7pQVlxQpTojvtBKuDqEVVSyTz0EOwaJHtjf7973DEEeaDEeOVV1o6AnSEHLu5YTpNOuF14LeIjAYuAj6OLCWK+Qr6Hugy0BWg74O+Gz6imsGgd2EevT1B9ya5N+/BoPNAh2HevM/S3Ju3DPPU/SPo7cHrsuDc5oHcI4L2y2juzXtycA8DQPuD1hLBm3cXUP3JT7Jb74jC4sW2BCNiSzitYcUKkzFxYm7nloB2uXRXW2vPZ8qUvF2iXT6XiNx9tz2+11/PbFxHfibZkPa5zJypetZZqpts0vTdUITl2tZClGXeYh5wkMLhwevBCjMU1issUBgTVU5Uy/Q+4ArgMuAuWlmiBjgF6I4tG98JjFelVoRBIjSIMMgUPE9gZd+qgbrguDAk53wsO8W5wLHB6/ODsQsxr+FLgS+APYCjQmP/iXluvYf9InksaEvPffflv45f+Bdna+sGzrEVEV/mTULMMk0Ta+okJpYEzH2JQsTiN3P1/bBunVmbBx0EQ4bAP/5hrw891Ap0h2lLy7XtEdX/onp/8PoTVIcBfYF+xHL+RpOT1prsAno5aGXRf0EU8dgFLPiwtdZiVGKOBaDapUvrrvff/9r455/P/fziaLfWxsYbq552Wt7Et9vnEpGhQ1W/+93MxnToZxJ2CMqEuXP1ix13bLIs581TveQS1YoK+z88cKCtMMXOjxypCT3BRo7M7f3kENq6ZZqjI21ojCprRRgPXNsapd+hiFWZv+AC2HLL3Muvrzf5sWxLa9e27noeY5qeCEXCneSMHg133WVGUT7qPrQr6uvhX/9qcghavtz8FbbYwo5+/Zpeb7JJ87zbEyey8XvvWaBv9+62+rV2rWUkuuoqOOSQ5sW5O1Lps2IiUg0RQzxVD4jSLWqc6ZPAAcBNEft3XGJLKtdck3vZEyc2KdIYa9Zkfr3Zs+0brhAOU+2VgQNdmWbBmDHmkDRtGuy8c9ruHZtx40wBgv3/veuupu2aeLp0aVKsG28ML7yAqMKDD8JGG8Epp8DJJ8P22xds+p2UcK3LUmAsFmP6WtC2O7AVMDmqwKjK9BngDyLsiAW3Lg+f1CAjUacgnx5wiTzuGhsz35yqqzPLqyzqP28npKIidcofJyWjR9vfmppOrkzfeMMyC4UpKbEfamVlFp6yYEHzI9b2+uv2/xvsx+9RR1lxbif/qDZVgxH5K3ArcAa2LB1r/xuJMpUkIeq37dXB318mmham2Ts008vLrbpCPolfwqmrs1CZAQNsZyRqWTaPMU1PRYV9oa1ebQXfnYzo3x+23Raeew5+9atiz6ZIrFoF3/lOy/bGRqsJes01ybdn6uth8ODmYyZPhosvzs8WkpOKnwJfb6ZIjWuBV4EzogiJ5M2rSkmKo8Mr0qJRWQmXXw5PPWV7p1HxouDpcY/erBk9Gl54oeXORKdA1ZZkFy1qeS7K6lWiLR33yi0WAnwtQXuitqQUvJy6kyG/+IV9a/3qV/B5hKLva9dahhRP2JCaiHVNneSMHm2l6t57r9gzKQI33GA/cM8/P5FvbXpHIU+i0Ja4CfgXIuciMiY4zgVuBCJbMZE31YLKMQcDg4Cu4XOqXBxVjpMhJSXmKbjjjuaY8PDDqZd7P/vMfvG6ZZoat0yzJrZv+txzlpSn0/Daa3D66XDwwTBhQutkhJRtTU0NY8aMycnUnFZxDpb74AzgD0FbPfAn4C9RhUSyTEXYE/gIS9wwEfgZlmLwN8CRkafstI5tt7UkqI8+Cnfembrv7Nn215VpatKkFHTSM2gQbL11J0veMH++peobONBKmHX6uKAOgOp6VC9HdQCwCbAJqgOCtsgFwqMu8/4Zqyc6AFiFhckMAt7EsiI5+eaXv4Q997RfxPPnJ+8XizH1Zd7U9OgBffq4Ms2S0aPh+ec7yb7punXmcbt4Mfz73/b5cToWql+h+lVrhkZVpjsCV6uiWOHUclXmA78FJrTmwk6GlJZakuuGBlOoyairs2XgmOXlJGfgQF/mzZLRo023zJhR7JkUgHPPNTP8hhtg1Khiz8bJBpF3Edk0eP1e8D7xEZGoe6bhnfL5QCXwPlbOzDMDFIoddoALL4TzzrNfxkcc0bLP7NkWt9C1a8tzTnM8C1LWxLb6nnuuqeBRh+Tuu+Evf4FTT4Wf/KTYs3Gy599ALLPGfbkQGFWZvg3shpVRqwEuEaEflmQ+suZ2csDZZ1vKsVNPtW+yzTZrft5jTKNTUQGvvlrsWbRrqqps77Smxj6SHZLaWqsRutdecOWVxZ6NkwtUL0r4OgsyqWcaq6R8PlZA9R/ApsC4XEzEiUiXLrbcu3gxnHVWy/OuTKNTUWHPccWK9H2dpMT2TVuEvHcEli6FH/zAUv3de6+v+DhJiWSZqvJm6PVC4Nt5m5GTnpEj4Xe/swDvH//YComDBX3PmWNtTnpisaaffw7bbZe6r5OU0aPh9tvhgw9sJ6LDsH49/PSn8Omn8Oyznuu6IyHyHtET3e8YpVtGyVtF2BXYBnhUleUi9ARWq7IuEzlODjjvPLj/fkvqUFtrSbPr683j0C3TaITDY1yZtprwvmmHUqZ//KPFdV91Fey7b7Fn4+SWnOyThomkTIP90YexfVMFtgM+Aa7EQmUi5S50ckh5uS33fv3rto96ww0eY5opHmuaEwYPtvTRNTWWV6RD8N//WunDY45J7T3vtE9ytE8aJuqe6V+x8jSbAeENpnuBb+V6Uk5Edt8dfv1ruPFGePppjzHNlAED7K8r06wQsaXe557rAPum9fWwxx62VTJihP1IjVpgwunURFWmBwLnqfJFXPssLHmDUywuusiWKE86ydKcgTtJRKV7d+jb12NNc8Do0TBvHnz0UbFnkiW//72VRlu5Eh54AHr2LPaMnEIgcgIiTyLyASKfNDsiElWZdqd5rGmMzbFl3kiI0EeEB0RYLkKdCMek6HuWCPNEWCrCTSKUR5EjwlgRGkLHChFUhF2C8xNEWBvXZ3CiObQLune35d66uqYC4n+JnE7S8VjTnBDO09tueest+78UwxVp50DkbCwH71tAFfAgVjy8D5YEPxJRlenzwPGh9ypCKZYB6ZmoFwOuwZRyP6yy+XUiDI/vJMJBwLmYRVwFDAbCa9xJ5agyRZVesQM4BdvfDVeCvjvcR5XIvz7aJPvsAyec0JTT7eabzUxw0uPKNCcMGWJlONttnt5777U40nBeRC+H1lk4CRiH6u+AtcDVqH4fU7CRHVCiKtNzgJNEeAooDy4yA9gb+F0UAYHn7xHABao0qPIi5tSUKJ3IccAkVWqDpeWJBMo8QzkxWbcFqRA7LuF9Ha+LGJ2BA12Z5oB2u2/61Vdw/PHwox9Z+cIYa9b4j9LOw0Dg9eD1SqB38PpOTNdEImqc6QwRvgaMx1IwdcOcj65RpT7itYYAjarMDLVNA0Yn6DsceCiuXz8RNsP2aCPJEaES2A+rchPmEBGWYGV2rlblukQTFmEcQVKKsjKhpo3+7O66eDF73H57U5X2NWtonDSJ1w48kDV5Tsbd0NDQZp9LFAatXcvgL7/khccfp7F795zJbc1z6bp4McMuvpgZF16Y93+3fLDVVv35/PMh3HHHqwwY0HL3p619VnrX1rLDpZfSbf58lg0ZQq9PPqFkXVOU3/q1a6k/+WQ+OvPMvM6jrT2XTsg8oC8wB6gDvg5MBbYlaiwqgKq2+gCtBL0nYt99QefFtZ0EWpOg7yzQg0PvuwRVd6sylHNBfDvoMND+oKWge4HWgx6dbv7l5eXaZhk/XrVr1+blibt2VT3llLxfurq6Ou/XyCuTJ9vzev/9nIpt1XMZP161pKQg/275oLbWHuWkSYnPt5nPytq1qhdeqFpaqlpVpfrii6ojRyYq8W3teabNPJc8ASzXLPRM3g/4l8KE4PXJCisVqhWWKtwYVU7UZd5kbEJ0M7iBJvM5Rm9gWYS+sdfLMpTzU+DWcIMqM1SZq0qjKi8DV9Hea7K+8ootS4VZswZefrk482lPtJVY0/fftzCM9evb7fLiDjvA5pu38X3TWbMsAcNFF1kM6dSpsPfeVqw7kToNFfF2OhgiBwavxgGXAKB6Pbal+B6WRveUqOKyVaaZMBMoEyGcamYnoDZB39rgXLjffFUWR5Ujwt5YRZt0mS4UaN+BZP5F0HpiKQWLrUxPPNH2uqHd7nmH903bHKpwyy2WivODD+Cuu+C22yxzmFN8RPog8gAiyxGpQyRppEdozLOIKCJlobYaRFYh0hAcH6aQ8FQQ+vI7YIsNrap3o/pLVK9GdW3S0XEUTJmqshy4H7hYhJ6BsjsUuD1B99uAn4swTIRNseT6t2Qo5zjg36rNLVYRDhVhUxFEhN2BX9J8f9bpTMQSNxQz1rS+3lYXYrRj55fRoy09dCwZV5tgyRJzMDrhBNhlF3j3Xc9f3fZoEaGBSItIjw2IjCW5z89pqPYKjqEprjkc0yWnA3WIPIbIYYiUphiTlEJapmAmc3dgAeYpNV6VWhEGBfGegwBUeQK4HKjGNoTrgAvTyYmdFKEb8CPilngDjgI+xpaFbwMuU03Yz+kMlJdDv37FtUzPOqulC+yaNe3SOm0z8ab19TaZe++FHXeEhx6CP/0JnnmmaWnfaRuIbIjQQLUB1dQRGiIbY/rgnKyuq/o+qr/BvHl/jK1S3gt8jshliKRSxC1I6c0rwsNpxsfvXaZElSXAYQna5wC94tquxHL/RpYTOr8K289NdO7oqPN1OgnFjjWtrm7Z1tgITzxR+LlkyfDhVmK3pgaOO66IE7nwQqsL9/zzMHSoJazfeeciTqjz0hfKEHkz1HQDqjeE3g8BGlGNEukB8AfgOswLNxF/RORPwIfAeajWpJyg6jrMQr0fkf7YnukJwG8QeQnV/VKOD0gXGrM4wvlPo1zIcdosAwfCzJnp++WD9esti9W3vw3/+Y+1rVhhTjGffmr5+dpRRZuSEthvvyJaplOnwt//bsvkAKWl9lwHt98kZ+2dRbAO1V1TdOkFLI1rWwps1KKnyK5YfoMzMIsynt9iORDWYKuQjyAyEtVZkSarOheRa7GVywnBtSKRUpmqckJUQY7TbqmosHqVxeCllywV5B/+0NTWo4flhd11VzjsMHj1VStO3U4YPdqm/7//FWhFdckSuOMOSwX4zjum0UtK7IdKaaml14yl2nTaItEiNERKgGuBM1Bdl7AAgeproXe3InI08B3gH2lnIfINLCfBYVia3DuBf0W7hcLvmTpO26OiwjLhfPVV4a89ebLlgD300ObtVVVwzz3w4YdWoDqc5q6NU5B908ZGePJJOOoo2GorK5MmApdeaoUeYs+rHTtzdSJmYkvB6SI9egO7AncjMg94I2j/DJFkBWdTR2uIDELkQkQ+BZ7EIkDGAf1RPRXVyCERrkwdp1ixpqtXm8L8wQ8SJ1U/4AC44gp48EFTErkk5qCTByXzta/BJpvkKN40fp6ffGKVXbbeGg46yBTqL35hFulbb5lXdvwPj3YaatRpUN0QoYFIT0SSRWgsxZTdyOD4TtC+C/AaIpsgchAi3RApCzx+9wP+m/C6Ik9hedt/AdwFDEF1DKqTUY1cwCVGpHSCjtOhCceaDk/ujZ9z/vMf+PJLOPbY5H3OOMOUxIUXWozkIYfk5toTJ8KLL9rfHC+BlpbmcN80Ns9jjzWP52efNQv0W9+yHxrf/z5069bU3xOYtFdOwSq0LMB8ccajWovIIGwPdBiqcwg7HYnE/uHnB8u+G2PJF7YHGoEPgMNQTRZruhI4HHgM1cZsb8CVqePELNNCx5pOnmxhOQcemLyPiGVGmjHDFMrrr5t3ajbU18OkSWbB/etfcO65Od/cHD3aHGjnzoX+/bOY57/+ZfOMhbRMnGhuwsnm64lK2ieqiSM0TIH2atFu52YTXsJVXQjslsE1v5/JFNPhy7yO07+/Ka1CLvN+8QU8+igcfTSUpflN2727efSUl5tDUjZ7u2+/DXvs0WS9rVlj3sLjx1sYSY72ZnOyb3rssU2VXMrK4Hvfg/PP9zhRp03iytRxunQxJ5ZCKtN//9sUWaol3jCDBtn+6kcfwU9+krnSe/99+OEPLQNQ/H2uWwe33moasLISzj7blG58IokMGDkSevfOYt/02mube1ivW2fpAN2RyGmjuDJ1HCh8XdPJk2H77TNLJDBmDPz1r7Z+evHF0cZ8+qnV6xwxAv77Xwu36dq1eZ/SUlPqd9wBo0bB3/5mSneHHSwhfDgGN6LjUmmp5ZNvlWV6771w6qm0CH1wRyKnDePK1HHAlg4LtWc6Z45pmbFjWyqMdJx2mu0ZXnSRpchLxty5ppCGDoW774Zf/co8YdetS+yg88YbtuT88MMwf77t0261lV1n6FBTrn/5C/z2t02OS2kYPdoiezIyJh980Kq59OiROMWiOxI5bRRXpo4DTSkFs1jajMwdd9jfY9IXxmiBCFx/Pey2m1mTzz/f3FJcvBjOOQe22cYU4oknWtmxP/8Z+vaNVmGoTx846SRLc/i//5kSLSmB3/wGbr/dlpgnTUqrJWP7ps8/H/HeHn3UEtLvuqtZwF4JyWlHuDJ1HDBluny5harkE1VTSHvv3foUd926wf33m/V2yCFmKZ5/vlmRW29tyu9HPzKz8Nprs3Cnxarq/OpXTZZraVBQY/Vqu3a8lRti552hV6+I+6ZPPAFHHAE77WSve2eU9ttxio4rU8eBwtU1nTatKcwlGwYONAv1q6+aLMUJEyz+8r33zKEol/lo6+vNo7gxFI735ptWkeWNNxIOKSuDffaJsG/61FPmpTx8uCVh8BqjTjvElanjQOFiTSdPNu/hH/4we1lPPdVkKYrAkUfCfffBsGHZy45n4sSWHsRlZfa89tzTlpZXrmwxbPRo++2wcGESudXVlnhh6FC7n003zf3cHacAuDJ1HChMSsHGRrjzTvjOd6xOWTbU11vO2ZilqAqPPZa/0JFEmYXWrTPr9+c/tz3ZnXayJecQKfdNn3/eYke32Qaefjr7Z+I4RcSVqeOAea6WluZXmdbUmJft2LHZy0pkKeYzdCSZ49K775qj09NPW4KF/fazpPMNDYD5EvXokWDf9OWX7UfFoEGW3WjzzfMzb8cpEK5MHQdMkeY7ccPkyeZY873vZS+rreWgPfBA26s9/XTL9fu1r8HTT9OlC+y1V9y+6WuvwcEHm2PUs89aSkXHaee4MnWcGPmMNV2xwrIeHXmkpQfMlighLoWmVy+46ipbvu3aFb75TTjxRA7a40sWvlfPiFPPNE/dgw6CLbaw/dKttirefB0nh3iie8eJUVGRP2X0yCOwbFn2XrztgX32galTzbv4iis4vc/jDGMnNpvxrtVtjVmkAwYUe6aOkzMKapmK0EeEB0RYLkKdCEmj1kU4S4R5IiwV4SYRyqPIEaFKBBWhIXRcEDovIlwmwuLguFwkRfFYp/MQSymYj8QNkyeb/JhHTkene3e47DJ49VVWdtmI7/A4grJ+zVoeP+Ee2ytNw5QpViO9pMT+TpkS7dLtZZzTsSi0ZXoNsAbohxV3fUyEaarNK6qLcBBwLnAAMBd4ALgoaIsqZxNV1iWYwzis1M9OWBX2WIHY67O/PaddU1EBq1bBkiW59SxdtMiWN3/1K/vG7URMmbkbKxeM5ng+poxG1lLGnItv4Z/9duPII5OPu+8+OOuspmibujpLytTQQJsfN26cvc6Fn5nTjlDVghygPUHXgA4Jtd0O+qcEfe8A/UPo/YGg86LIAa0KNpDKkszjZdBxofc/B3013fzLy8vVaUl1dXWxp5A77rvPdh7feSdrUc2eyzXXmNx3381abntj1wFzdQXdmu3sLqe79qM+wYZvxzkqK6M/ow71fygBwHItkJ4p5lFIy3QI0KhKqAQF04BE617DgYfi+vUTYTNgUEQ5dSIbLM+zVVkUkj0tbuzwRBMWYRxmyVJWJtS0up5Ux6WhoaHDPJeNFixgF+C9//yHxVmmFQw/l1HXXkvp4MG8uXhxFjXJ2icnfH43QvMQnhIauYCL+fD0s5KO+8c/toWEuy/K6ad/3ObHzZmj1NREK5nTkf4PdWoKpbVB941Zl6G2k0BrEvSdBXpw6H2X4BdfVTo5oL1AdwUtA+0Heh/of0N9G0G3D73fLpAtqebvlmliOtSv6s8/N7PimmuyFrXhuXz8scm87LKsZbZHpncZmdB0m95lZMpxlZWts/jay7gwHer/UALoJJZpITdwGoD47NW9gWUR+sZeL0snR5UGVd5UZZ0q84HTgG+JbBiTSHaDKnnwOnHaFf36NaXIyxVTpliqv6OPzp3MdsTUm9+hZw9FaDp69lCm3pzaa/rSSy3ZQ5gePay9rY/r2jX9OKfjUUhlOhMoE2G7UNtO0Nz5KKA2OBfuN1+VxRnKATYoydhaTCLZycY6nYnSUgvXyFXiBlXz4h0zpildYSdj7FhLkFRZCSJKZaW9T+ec03wc7WZcly7mu3bUUanHOR2QQprBoHeB3ok5Ee0NuhR0eIJ+B4POAx0Guinos4QclVLJAd0DdChoCehmoHeDVofGngz6PugA0P6gtaAnp5u7L/MmpsMtUe2zj+ro0VmLqa6uVn3tNVvzmzQpa3kdgQ73WUnAPffYP/mdd0Yf09GfC77MmxdOAboDC4A7gfGq1IowKIgHHWQKnieAy4FqoC44LkwnJzg3GHgCW/adDqwGwmts/wQeAd4Lzj8WtDlOU6xpLpg8GcrLrU6n0yk44ggr2pModbLTsSlonKkqS7AYz/j2OUCvuLYrgSszkROcuxNTsMnmoMA5weE4zamosMLbqrZu10pk3Tq46y4roO31OTsNJSVwwQW2Rf7vf+em0p7TPuhcEeSOk46KCksYn7QAZzQ2festk9EZ0gc6zfjhD2H77d067Wy4MnWcMDmqa9rvqaegTx/49rdzMCmnPVFaCuefb0V0Hnyw2LNxCoUrU8cJM3Cg/c1GmS5bRt8XX4Qf/cjiJJxOx1FHwZAhcPHFtmPgdHxcmTpOmJhlmk2s6S23ULp6tVulnZjSUjjvPJg2DR5+uNizcQqBK1PHCbP55mZNZmOZXn65BTc/8USuZuW0Q445BrbZxq3TzoIrU8cJU1KSXXjMc8/BZ59ZhpBbboF583I4Oac9UVZm1unbb8NjjxV7Nm0ckT6IPIDIckTqEElanjM05llEFJGyUFvmcnKEK1PHiae1yrSuDr773ab3jY3m0ul0Wo49Frbe2q3TCITLao4FrkMkYQESAETGkji0MzM5OcSVqePEU1GR+Z7p/Pmw//6wfHlT25o1cPPNbp12Yrp0gf/7P3jjDV/1T4pIT+AI4AJUG1B9EXgY+EmS/htjSXzOiWvPTE6OcWXqOPFUVMDnn0cPEvzySzj4YJgzx749w7h12un56U9h0CC46KLOaZ32hTJE3gwd4+K6DAEaUY0vq5nMovwDcB0Q/ys1Uzk5xZWp48RTUQFr15q1mY4VKyzLUW2tZTtfu7b5+TVr4OWX8zNPp13QtatZp6+9Bk89VezZFJ5FsA7VXUPHDXFdegFL49qWAhu1ECayK7A38I8El4ouJw+4MnWceKLGmq5ZA0ceCS+9ZHl4Z83aUNKyprq6qbzlO6nLjTkdn+OPt49VZ7VO0xCtPKdICXAtcAaq61otJ0+4MnWceKLEmq5fb9+Qjz8O119vCRocJwnl5fC739kiRXV1sWfT5piJLQWnK6vZG9gVuBuRecAbQftniOybgZy84MrUceJJl1JQFU4/He68E/70JxgXvwXkOC352c+gf3+zTp0QqsuB+4GLEemJyN7AocDtcT2XAv2BkcHxnaB9F+C1DOTkBVemjhPPZptBt27Jlenvfw/XXgvnnAO//W1h5+a0W7p1s4/L889bOLLTjBZlNVGtRWQQIg2IDAoKh87bcECsGsV8VNeklFMAXJk6TjwiyWNNr7wSLrkETjzRrFLHyYCTToItt3TrtAWqS1A9DNWeqA5C9Y6gfQ6qvVCdk2DMbFSl2f5pMjkFwJWp4yQiUazpzTfDr39tTkfXX59VvVOnc9K9uy1oVFfDCy8UezZOLnFl6jiJqKhobpk+8IBZo9/8pnnulpYWb25Ou+YXv4AttrCsSE7HwZWp4ySiogLmzrWkC888YzW1dt8d7r/fXDMdp5X06AFnnw1PP+0hyB2JgipTEfqI8IAIy0WoEyFpEmIRzhJhnghLRbhJhPIockTYU4SnRFgiwkIR7hVhq9D5CSKsFaEhdAzO31077ZKBA02R7rijJWUYMsSylffqVeyZOR2A8ePto3TggXDAAaOpqoIpU6KNnTIFqqqsJkN7GGc/Hzo+hbZMWyQhFmmZ6kmEg4BzgQOBKmAwEN6yTyVnU+CGYFwlFrB7c9wl7lalV+j4JCd353QcYuExM2ZY+Y8nn4Q+fYo7J6fD8OCDsHo1rFoFqkJdnUVYpVNUU6ZYv7o6i9BqD+M6C6IFSschQk/gC2CEKjODttuBz1U5N67vHcBsVf4veH8gMEWVLTORE5zbGXhO1VJKiTAB2FaVYzOZf7du3XTVqlUZ3XNnoKamhjFjxhR7GrnnmWfgG9+w1926waefmhtmRDrsc8kCfyZNVFUlVjRlZbYIkoyZM2Fdgtw/bXtcT1SXd3hvvUQlbPLFEKAxpgADpgGjE/QdDjwU16+fCJsBgzKQA7AfLTNgHCLCEqAeuFqV6xINFGEcMA6grEyoqalJconOS0NDQ4d8LttddRX9S0qQ9etZv24d9SefzEdnnhl5fEd9Ltngz6SJOXNGAy31y7p1yuabL2w5IGDGjM3b9bgOjcXB5v8A3Rd0XlzbSaA1CfrOAj049L5LkOi0KkM5O4IuAd031DYMtD9oKeheoPWgR6ebf3l5uTotqa6uLvYUcs/cuardusUy69rRvbtqfX1kER3yuWSJP5MmKiubf7xiR2VlRxzXQ7VAeqaYRyH3TDNJQhzfN/Z6WVQ5ImwLPA6cocqGiC5VZqgyV5VGVV4GrgKOzPBenI7MxIkty695KTUnh1x6aUu3nB49rL2jjuvoFFKZzgTKRIiShLg2OBfuN1+VxVHkiFAJPA1MVE2bl1HpdOsRTkpeecUqwoTxUmpODhk7Fm64war2iSiVlfZ+7NhMxtEuxnUWCuaABCDCXZjyOhFLVPwfYC/V5gpVhIOBW4ADsH3NfwOva+BglEqOCAOA54HrVflzgjkcGpz/EtgNeAD4P1VuTTV3d0BKjDuVJMafS0v8mSSmoz8XEVmhqj2LPY98U+jQmBZJiAMFOCiI9xwEoMoTwOVANVAXHBemkxOcOxELpbkwHEsaGnsU8DG2LHwbcFk6Reo4juM4qSikNy+qLAEOS9A+B6uSHm67ErgyEznBuYtoHpMaf/7oyBN2HMdxnAh4OkHHcRzHyRJXpo7jOI6TJa5MHcdxHCdLCurN254RkfXAymLPow1SBiRIONbp8efSEn8mienoz6W7qnZ4w62gDkjtnLdVdddiT6KtISJv+nNpiT+XlvgzSYw/l45Bh/+14DiO4zj5xpWp4ziO42SJK9Po3FDsCbRR/Lkkxp9LS/yZJMafSwfAHZAcx3EcJ0vcMnUcx3GcLHFl6jiO4zhZ4srUcRzHcbLElWkaRKSPiDwgIstFpE5Ejin2nNoCIlIjIqtEpCE4Piz2nAqNiJwmIm+KyGoRuSXu3IEi8oGIrBCRahHpNJUdkz0XEakSEQ19ZhpE5IIiTrVgiEi5iEwKvkOWicg7IvLt0PlO+3npKLgyTc81wBqgHzAWuE5Ehhd3Sm2G01S1V3AMLfZkisBc4BLgpnCjiPQF7gcuAPoAbwJ3F3x2xSPhcwmxSehzM7GA8yomZcD/gNHAxthn457gB0Zn/7x0CDwDUgpEpCdwBDBCVRuAF0XkYeAnYIXKnc6Lqt4PICK7AgNDpw4HalX13uD8BGCRiGyvqh8UfKIFJsVz6bSo6nJgQqjpURH5FNgF2IxO/HnpKLhlmpohQKOqzgy1TQPcMjX+KCKLROQlERlT7Mm0IYZjnxNgwxfpLPxzE6NORD4TkZsDq6zTISL9sO+XWvzz0iFwZZqaXsDSuLalwEZFmEtb47fAYGAAFnT+iIhsU9wptRn8c5OYRcBuQCVmkW0ETCnqjIqAiHTB7vvWwPL0z0sHwJVpahqA3nFtvYFlRZhLm0JVX1PVZaq6WlVvBV4CvlPsebUR/HOTAFVtUNU3VXWdqs4HTgO+JSLxz6rDIiIlwO2YH8ZpQbN/XjoArkxTMxMoE5HtQm07YUszTnMUkGJPoo1Qi31OgA1779vgn5t4YunXOsXnRkQEmIQ5Mx6hqmuDU/556QC4Mk1BsHdxP3CxiPQUkb2BQ7Fflp0WEdlERA4SkW4iUiYiY4H9gP8We26FJLj3bkApUBp7HsADwAgROSI4/3vg3c7iTJLsuYjIHiIyVERKRGQz4O9AjarGL3F2VK4DdgAOUdVwbeRO/XnpKLgyTc8pQHdgAXAnMF5VO/svxi5Y6MNCbB/sdOAwVe1ssabnYwXjzwWODV6fr6oLMS/wS4EvgD2Ao4o1ySKQ8Llge+xPYMuX04HVwNFFmmNBCeJGfwGMBOaF4mzH+uelY+CJ7h3HcRwnS9wydRzHcZwscWXqOI7jOFniytRxHMdxssSVqeM4juNkiStTx3Ecx8kSV6aO4ziOkyWuTB2nkxLUFj2y2PNwnI6AK1PHKQIickugzOKPV4s9N8dxMsfrmTpO8Xgaq40bZk0xJuI4Tna4Zeo4xWO1qs6LO5bAhiXY00TkMRFZISJ1InJseLCIfE1EnhaRlSKyJLB2N47rc5yIvCciq0VkvojcEjeHPiJyr4gsF5FP4q/hOE40XJk6TtvlIuBhLJ/rDcBtIrIrgIj0wPLcNgC7Az8A9gJuig0WkV8A/wRuBnbESuTF55X+PfAQVrXkbuCmII+s4zgZ4Ll5HacIBBbiscCquFPXqOpvRUSBf6nqSaExTwPzVPVYETkJuAIYqKrLgvNjgGpgO1X9WEQ+Ayar6rlJ5qDAn1T1d8H7MuArYJyqTs7d3TpOx8f3TB2neDwPjItr+zL0+pW4c68A3w1e74CV6QoXkH4ZWA8ME5GvgAHAM2nm8G7shaquE5GFwBaRZu84zgZcmTpO8Vihqh+3cqzQVFw7nkwKta+Ne6/49o/jZIz/p3GctsueCd6/H7yeAewkIhuFzu+F/Z9+X1XnA58DB+Z9lo7juGXqOEWkXES2jGtrDIpFAxwuIm8ANcCRmGLcIzg3BXNQuk1Efg9sijkb3R+ydi8F/ioi84HHgB7Agar6l3zdkON0VlyZOk7x+AZQH9f2OTAweD0BOAL4O7AQOEFV3wBQ1RUichDwN+B1zJHpIeCMmCBVvU5E1gC/Bi4DlgD/ydO9OE6nxr15HacNEnja/lBV7yv2XBzHSY/vmTqO4zhOlrgydRzHcZws8WVex3Ecx8kSt0wdx3EcJ0tcmTqO4zhOlrgydRzHcZwscWXqOI7jOFniytRxHMdxsuT/Aaqq0VYt2x00AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"bo-\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\", color='b')\n",
    "plt.tick_params('y', colors='b')\n",
    "plt.gca().set_xlim(0, n_epochs - 1)\n",
    "plt.grid(True)\n",
    "\n",
    "ax2 = plt.gca().twinx()\n",
    "ax2.plot(history.epoch, history.history[\"val_loss\"], \"r^-\")\n",
    "ax2.set_ylabel('Validation Loss', color='r')\n",
    "ax2.tick_params('y', colors='r')\n",
    "\n",
    "plt.title(\"Reduce LR on Plateau\", fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tf.keras schedulers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5995 - accuracy: 0.7923 - val_loss: 0.4095 - val_accuracy: 0.8606\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3890 - accuracy: 0.8613 - val_loss: 0.3738 - val_accuracy: 0.8692\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3530 - accuracy: 0.8772 - val_loss: 0.3735 - val_accuracy: 0.8692\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3296 - accuracy: 0.8813 - val_loss: 0.3494 - val_accuracy: 0.8798\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3178 - accuracy: 0.8867 - val_loss: 0.3430 - val_accuracy: 0.8794\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2930 - accuracy: 0.8951 - val_loss: 0.3414 - val_accuracy: 0.8826\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2854 - accuracy: 0.8985 - val_loss: 0.3354 - val_accuracy: 0.8810\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2714 - accuracy: 0.9039 - val_loss: 0.3364 - val_accuracy: 0.8824\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2714 - accuracy: 0.9047 - val_loss: 0.3265 - val_accuracy: 0.8846\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2570 - accuracy: 0.9084 - val_loss: 0.3238 - val_accuracy: 0.8854\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2502 - accuracy: 0.9117 - val_loss: 0.3250 - val_accuracy: 0.8862\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2453 - accuracy: 0.9145 - val_loss: 0.3299 - val_accuracy: 0.8830\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2408 - accuracy: 0.9154 - val_loss: 0.3219 - val_accuracy: 0.8870\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2380 - accuracy: 0.9154 - val_loss: 0.3221 - val_accuracy: 0.8860\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2378 - accuracy: 0.9166 - val_loss: 0.3208 - val_accuracy: 0.8864\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2318 - accuracy: 0.9191 - val_loss: 0.3184 - val_accuracy: 0.8892\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2266 - accuracy: 0.9212 - val_loss: 0.3197 - val_accuracy: 0.8906\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2284 - accuracy: 0.9185 - val_loss: 0.3169 - val_accuracy: 0.8906\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2286 - accuracy: 0.9205 - val_loss: 0.3197 - val_accuracy: 0.8884\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2288 - accuracy: 0.9211 - val_loss: 0.3169 - val_accuracy: 0.8906\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2265 - accuracy: 0.9212 - val_loss: 0.3179 - val_accuracy: 0.8904\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2258 - accuracy: 0.9205 - val_loss: 0.3163 - val_accuracy: 0.8914\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2224 - accuracy: 0.9226 - val_loss: 0.3170 - val_accuracy: 0.8904\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2182 - accuracy: 0.9244 - val_loss: 0.3165 - val_accuracy: 0.8898\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2224 - accuracy: 0.9229 - val_loss: 0.3164 - val_accuracy: 0.8904\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
    "learning_rate = keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)\n",
    "optimizer = keras.optimizers.SGD(learning_rate)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For piecewise constant scheduling, try this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = keras.optimizers.schedules.PiecewiseConstantDecay(\n",
    "    boundaries=[5. * n_steps_per_epoch, 15. * n_steps_per_epoch],\n",
    "    values=[0.01, 0.005, 0.001])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1Cycle scheduling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = keras.backend\n",
    "\n",
    "class ExponentialLearningRate(keras.callbacks.Callback):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "        self.rates = []\n",
    "        self.losses = []\n",
    "    def on_batch_end(self, batch, logs):\n",
    "        self.rates.append(K.get_value(self.model.optimizer.lr))\n",
    "        self.losses.append(logs[\"loss\"])\n",
    "        K.set_value(self.model.optimizer.lr, self.model.optimizer.lr * self.factor)\n",
    "\n",
    "def find_learning_rate(model, X, y, epochs=1, batch_size=32, min_rate=10**-5, max_rate=10):\n",
    "    init_weights = model.get_weights()\n",
    "    iterations = math.ceil(len(X) / batch_size) * epochs\n",
    "    factor = np.exp(np.log(max_rate / min_rate) / iterations)\n",
    "    init_lr = K.get_value(model.optimizer.lr)\n",
    "    K.set_value(model.optimizer.lr, min_rate)\n",
    "    exp_lr = ExponentialLearningRate(factor)\n",
    "    history = model.fit(X, y, epochs=epochs, batch_size=batch_size,\n",
    "                        callbacks=[exp_lr])\n",
    "    K.set_value(model.optimizer.lr, init_lr)\n",
    "    model.set_weights(init_weights)\n",
    "    return exp_lr.rates, exp_lr.losses\n",
    "\n",
    "def plot_lr_vs_loss(rates, losses):\n",
    "    plt.plot(rates, losses)\n",
    "    plt.gca().set_xscale('log')\n",
    "    plt.hlines(min(losses), min(rates), max(rates))\n",
    "    plt.axis([min(rates), max(rates), min(losses), (losses[0] + min(losses)) / 2])\n",
    "    plt.xlabel(\"Learning rate\")\n",
    "    plt.ylabel(\"Loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: In the `on_batch_end()` method, `logs[\"loss\"]` used to contain the batch loss, but in TensorFlow 2.2.0 it was replaced with the mean loss (since the start of the epoch). This explains why the graph below is much smoother than in the book (if you are using TF 2.2 or above). It also means that there is a lag between the moment the batch loss starts exploding and the moment the explosion becomes clear in the graph. So you should choose a slightly smaller learning rate than you would have chosen with the \"noisy\" graph. Alternatively, you can tweak the `ExponentialLearningRate` callback above so it computes the batch loss (based on the current mean loss and the previous mean loss):\n",
    "\n",
    "```python\n",
    "class ExponentialLearningRate(keras.callbacks.Callback):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "        self.rates = []\n",
    "        self.losses = []\n",
    "    def on_epoch_begin(self, epoch, logs=None):\n",
    "        self.prev_loss = 0\n",
    "    def on_batch_end(self, batch, logs=None):\n",
    "        batch_loss = logs[\"loss\"] * (batch + 1) - self.prev_loss * batch\n",
    "        self.prev_loss = logs[\"loss\"]\n",
    "        self.rates.append(K.get_value(self.model.optimizer.lr))\n",
    "        self.losses.append(batch_loss)\n",
    "        K.set_value(self.model.optimizer.lr, self.model.optimizer.lr * self.factor)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "430/430 [==============================] - 1s 2ms/step - loss: nan - accuracy: 0.3120\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAERCAYAAABcuFHLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAl3klEQVR4nO3deXhc9X3v8fdXiyVblndZEvIGBoM3MLYJEGMgCWAwISExhECBQLhA25Dk5rZPQ56QxmlI2iRNem/a3hZyoQYCFCgECDuBgKGAwcYGL9gE432Vd1u2tc33/jEjI4S2kc7M+Wn0eT3PeTxz5szo+/OR5+vfbu6OiIhId+XFHYCIiOQGJRQREYmEEoqIiERCCUVERCKhhCIiIpFQQhERkUgUxB1AlIYNG+ZjxoyJOwzJMSu27KO0uICRg/vFHYq0UN/orNy6j6pBfRlS0ifucHqsRYsW7XD3su5+Tk4llDFjxrBw4cK4w5Ac850HlvDy+9Us+P455OdZ3OFIM9v2HebUn77ArV+azBWnjoo7nB7LzNZF8Tlq8hLpwGdPGM6umjqWbNgddygiQVNCEenAWceXUZhvPLNsa9yhiARNCUWkAwOKC5l5XBlPLd2KlioKi25HWJRQRDrhgkkVbNpziGWb9sUdirTC1LUVBCUUkU74zAnDAXhp1faYIxEJlxKKSCcM61/ESSMG8qISikiblFBEOumzJ5SzZMMetu49HHcokuKoEyUkWUsoZlZkZneY2Toz229mi83sgjaunWRmz5rZDjPTb4wE4cITK3GHp5ZuiTsUaUFdKGHIZg2lANgAnAUMBH4APGhmY1q5th54ELgua9GJdODY4f05oaKUJ5VQRFqVtYTi7jXuPtfd17p7wt2fANYA01q5dpW73wEsz1Z8Ip1x0UlHsWjdbjbvORR3KCLBia0PxczKgXEoaUgPcuHkSkDNXqHQPJSwxJJQzKwQuBe4y91XdvOzbjCzhWa2sLq6OpoARdowZlgJ4ysH8OxyzZoPieahhCHrCcXM8oB7gDrgpu5+nrvf7u7T3X16WVm3F8sU6dB5E8pZuG43Ow7Uxh2KSFCymlDMzIA7gHJgjrvXZ/Pni0Rh1sQK3OEPK7bFHYpIULJdQ/k3YDxwkbu32atpScVAn9TzYjMrylKMIu0aX1nKiMF9eU4JJXbqQglLNuehjAZuBKYAW83sQOr4MzMblXrctKHBaOAQH3XYHwJWZStWkfaYGbMmVvDqn3ZwoLYh7nAEMM1ECUI2hw2vc3dz92J379/suNfd16cer09duzZ1bfNjTLZiFenIeRPKqWtM8PIqDQQRaaKlV0S6YPqYIQwp6aPRXiLNKKGIdEF+nnHO+OH8ceV26hoScYfTa2l/mrAooYh00ayJFeyvbeD1D3fGHYqoCyUISigiXTTj2GH065PPc2r2EgGUUES6rLgwn7OPL+P5FdtIJNT0IqKEItIN502oYPv+WpZs3BN3KL2SulDCooQi0g2fOWE4BXmm0V4xUxdKGJRQRLphYN9CTh87lOeWb9OII+n1lFBEuum8iRWs2VHDB9sPxB2KSKyUUES66dzx5QBa20t6PSUUkW6qGFjMlJGD1I8SI9OGKEFQQhGJwHkTy3l3415tDSy9mhKKSARmTawA4Hk1e0kvpoQiEoGxZf0ZW1aiZq8s08C6sCihiETkvIkVLFiziz0H6+IOpddRD0oYlFBEIjJrYgWNCeeF97bHHYpILJRQRCJyYtVAKgcW8/QyNXtJ76SEIhKRvDzj/EkVzP9TNfsP18cdjkjWKaGIRGj25ErqGhK8uFLNXtngqFc+JEooIhGaNmoww0uLeGrplrhD6RWadg3Iz1O3fAiUUEQilJdnXDCpgpdWVVNT2xB3ODkvkRo3rInyYVBCEYnY7MmV1KrZKyv8SEJRRgmBEopIxKaPGcKw/kU8vUzNXpnWNLFRLV5hUEIRiVh+nnH+pHJeXLmdg3Vq9sqkxJGEoowSAiUUkQyYPbmSw/UJXlpVHXcoOa2pD0U1lDAooYhkwKfGDGFoSR+N9sqwhPpQgqKEIpIBBfl5zJpUwYsrt3O4vjHucHJWUx+K0kkYlFBEMmT2pEoO1jWq2SuDXH0oQVFCEcmQ044ZwuB+hWr2yqAjfSj6JguCboNIhhTk5zFrYgUvvLdNzV4Zoj6UsCihiGTQBZMrqalr5JU/7Yg7lJyUUB9KULKWUMysyMzuMLN1ZrbfzBab2QXtXP8dM9tqZnvN7E4zK8pWrCJR+fTYoQzsq2avzGkaNqyUEoJs1lAKgA3AWcBA4AfAg2Y2puWFZjYLuBn4HDAGOAb4UbYCFYlKYX4e500o5w8rtlHboGavqGliY1iyllDcvcbd57r7WndPuPsTwBpgWiuXfw24w92Xu/tu4MfANdmKVSRKs0+sZH9tA6+q2StyiYQmNoYktj4UMysHxgHLW3l5IvBOs+fvAOVmNjQbsYlEacbYYZQWF/DUUu3kGLWmGoo6UcIQS0Ixs0LgXuAud1/ZyiX9gb3Nnjc9Lm3ls24ws4VmtrC6WuP9JTx9CvI4d0I5z6/YSl1DIu5wcoqrDyUoWU8oZpYH3APUATe1cdkBYECz502P97e80N1vd/fp7j69rKws0lhFonLh5Er2HW7gv1er2StKmtgYlqwmFEsOFr8DKAfmuHtbG28vB05q9vwkYJu778xwiCIZccZxwygtKuBpjfaKlBaHDEu2ayj/BowHLnL3Q+1cdzdwnZlNMLPBwC3AvCzEJ5IRRQX5nDOhnOdWbKO+Uc1eUTkyD0U1lCBkcx7KaOBGYAqw1cwOpI4/M7NRqcejANz9GeDnwB+Bdanjh9mKVSQTLphUwZ6D9by+WhXtqGgL4LAUZOsHufs62h+L0b/F9b8CfpXRoESy6MxxZZT0yefpZVs4c5z6+yKhPpSgaOkVkSwpLsznc+PLeXb5NhrU7BUJ9aGERQlFJItmT65gV00dC9bsijuUnKCZ8mFRQhHJorOPH06/Pvk8qdFekWiqoUgYlFBEsqi4MJ/PnDCcZ5dtpTGhL8Pu0jyUsCihiGTZhZMr2VlTx5tq9uo21wZbQdFtEMmys48vo7gwjyeXbo47lB5PfShhUUIRybJ+fQo4Z3w5T767RZMcu+nIPJSY45AkJRSRGHzp5Cp2H6xn/vta0LQ7jiw2rBpKEJRQRGJw5rgyBvcr5HeLN8UdSo/mmocSFCUUkRgU5udx0UlH8fyKbew/3NYaqdKRjyY2KqOEQAlFJCYXn1xFbUOCZ5Zp462uSqS6oJRQwqCEIhKTk0cOYvTQfjy6RM1eXaXFIcOihCISEzPj4ilVvLZ6J9v2HY47nB7po075WMOQFCUUkRhdfHIV7vD4Es1J6QpXH0pQlFBEYnT0sBJOGjmIx95Rs1dXaGJjWJRQRGJ24eQKlm3ax4ZdB+MOpcdRH0pYlFBEYjZrYgUAz63YFnMkPY8f2QI43jgkSQlFJGajh5ZwQkUpz2r4cNrUhxIWJRSRAJw/qYK31u2ien9t3KH0KOpDCYsSikgALphUiTs8vUwbb6VDWwCHRQlFJADHV5QyvnIAD7+t0V7paKqhmNYbDoISikgg5kyt4p0Ne/hg+4G4Q+kxmvpQTN9kQdBtEAnEF6YcRX6e8cjbG+MOpcfQFsBhUUIRCcTw0mLOPG4Yv1u8SfvNd5L6UMKihCISkC9PHcGWvYd5ffXOuEPpEdSHEhYlFJGAnDuhnNLiAh5Ws1enOJopHxIlFJGAFBfm8/kTj+KZZVs5UNsQdzjBUx9KWJRQRAJzybQqDtU38tRSzUnpSCKhPpSQKKGIBGbqqMEcPayEhxep2asjR/pQVEMJghKKSGDMjDlTq1iwZpdWIO5AUx+KaihhUEIRCdCXpo7ADB7RzPl2qYYSlqwmFDO7ycwWmlmtmc1r57oiM/snM9tsZrvN7P+aWWEWQxWJVdWgvpx+zFAefnvjkdng8knurtpJQLJdQ9kM3Arc2cF1NwPTgUnAOGAqcEtmQxMJy5ypI1i/6yBvrd0ddyjBSrhrhFdAsppQ3P0Rd38U6GjW1kXAr919l7tXA78Gvp7p+ERCcsHkCkr65Ktzvh0J1xyUkITah2Kpo/nzEWY2MKZ4RLKuX58CLphcyZNLt3CorjHucILkrv6TkISaUJ4Gvm1mZWZWAXwrdb5fywvN7IZUv8zC6urqrAYpkmlzpo7gQG0Dzy7Xbo6tUR9KWLqdUDLUWf4TYDGwBHgNeBSoB7a3vNDdb3f36e4+vaysLAOhiMTn1KOHUDWor5ZiaYP6UMKSVkIxs2+Z2Zxmz+8ADpnZKjM7Pqqg3P2Qu9/k7lXufgzJPpdF7q56v/QqeXnGnGkjePWDHWzZeyjucIKTcLQsZEDSraF8C6gGMLMzga8AV5CsSfyyozebWYGZFQP5QL6ZFZtZQSvXVZnZUZZ0GvAD4IdpxiqSE+ZMrcJdc1Ja4651vEKSbkKpAtamHl8EPOTuDwJzgdM68f5bgEMkhwVfmXp8i5mNMrMDZjYqdd1Ykk1dNcBdwM3u/lyasYrkhNFDSzhlzGDNSWlFwl2jvAKSbkLZBzR1VJwLvJB6XA8Ud/Rmd5/r7tbimOvu6929v7uvT103393HuHs/dz/e3e9NM06RnHLJtBF8WF3Dkg174g4lKO5Onnrlg5FuQnkO+E2q7+RYkqOxACYCa6IMTEQ+MntyJcWFeeqcb0F9KGFJN6F8A/hvYBhwibvvSp2fCtwfZWAi8pHS4kJmTazg8SWbOVyvsSlNNMorLJ/oEG+Pu+8DvtnKeXWYi2TYnKkjeGzJZl54bzsXnlgZdzhBcDSxMSTpDhue0Hx4sJmda2a/NbPvmVl+9OGJSJMZxw6jYkCxmr2a0cTGsKTb5HUHcDKAmY0AHgOGkGwKuzXa0ESkufw840tTq3j5/Wq27z8cdzhBSCQ0bDgk6SaU8cDbqceXAgvcfTZwFXB5lIGJyCfNmTqCxoTz2OLNcYcSBA0bDku6CSUfqEs9/hzwVOrxaqA8qqBEpHXHDu/PSSMHaU5KiqMaSkjSTSjLgL8ws5kkE8ozqfNVwI4oAxOR1l0ybQQrt+5n+eZ9cYcSO9VQwpJuQvkucD3wEnC/uy9Nnf8C8GaEcYlIGy46sZI++ZqTAlp6JTRpJRR3n09ypvwwd2++4dVtwF9EGZiItG5Qvz6cM2E4jy3ZTF1DIu5wYqUaSljSXr4+teLvITObZGYTzazY3de6+yeWlheRzLh0+kh21dTx3IrevU+KaihhSXceSoGZ/QLYDbwDLAV2m9nPM7Qvioi04szjyqga1Jf7FqyPO5RYqYYSlnRrKD8nuUrwnwPjgONINnVdBfx9tKGJSFvy84wrTh3Fa6t38mH1gbjDiY1qKGFJN6FcAVzn7ne5++rUMQ/4H8CfRR6diLTp0ukjKMgz7n+z99ZSEpopH5R0E8pAknNOWloNDOp2NCLSacNLizl3Qjn/tWhjr10wMuGOab3hYKSbUN4huWtjS99OvSYiWXTFqaPYfbCeZ5f3zs55d9SHEpC0VhsG/gZ4yszOBV4nOVH1dOAo4IKIYxORDswYO4zRQ/tx7xvr+eKUqrjDybqE+lCC0pV5KOOAh4D+wIDU41m0XnMRkQzKyzMu/9Qo3ly7i/e37Y87nKxL7tgYdxTSpCvzUDa7+/fdfY67f9ndbyG59/uc6MMTkY5cOm0EffLzuPeNdXGHknXqQwmLcrtIDze0fxGzJ1fwyNubqKltiDucrEouDhl3FNJECUUkB1x1+mj21zbw2JLetax9wrVjY0iUUERywNRRgzmhopTfvrGuVy1rrx0bw9KpUV5m9ngHlwyIIBYR6SIz46rTR/P93y3j7fV7mDZ6cNwhZUVy6RVllFB0toays4NjDXB3JgIUkc65eEoVJX3ye9XM+eQWwHFHIU06VUNx92szHYiIdE9JUQEXnXQUjy3ZzNwvTKR/UbrTzHoeRzWUkKgPRSSHfOWUkRyqb+TJd3tH53xyYmPcUUgTJRSRHHLyyEEcO7w/D7y1Ie5QsiLZKa+MEgolFJEcYmZcNn0kb6/fw8qtub/nfEJreQVFCUUkx1wybQRFBXnc9Vruz5xXDSUsSigiOWZwSR8unlLFo4s3sfdgfdzhZJQmNoZFCUUkB1396dEcqm/koUW53ZeiiY1hyWpCMbObzGyhmdWa2bx2rjMzu9XMNpnZXjN7ycwmZjFUkR5t4lEDmT56ML99Yx2JRO7OnE84WhoyINmuoWwGbgXu7OC6S4GvAzOBIST3Xrkns6GJ5JarTh/N2p0HeeWDHXGHkjGO+lBCktWE4u6PuPujJGfXt+do4FV3/9DdG4HfAhMyHZ9ILjl/UgXD+vfhntdzt3M+kVAfSkhC7UP5T+BYMxtnZoXA14BnYo5JpEcpKsjnslNG8uLKbWzcfTDucDIioT6UoISaULYArwCrgEMkm8C+09qFZnZDql9mYXV1dRZDFAnfFaeOBuC+Bbm5vpdrC+CghJpQfgicAowEioEfAS+aWb+WF7r77e4+3d2nl5WVZTlMkbBVDerL58aX88BbG6htaIw7nMglVxuOOwppEmpCOQl4wN03unuDu88DBqN+FJG0XXXaaHbW1PHMsq1xhxK55I6NyiihyPaw4QIzKwbygXwzKzaz1pZEfQu41MzKzSzPzK4CCoEPshmvSC4449hhHD2shHmvrY07lMiphhKWbNdQbiHZJ3IzcGXq8S1mNsrMDpjZqNR1PwPeAZYAe0j2n8xx9z1Zjlekx8vLM752+mgWr9/D4vW74w4nUupDCUu2hw3PdXdrccx19/Xu3t/d16euO+zu33D3Sncf4O5T3V2jvES66NLpIyktLuCOV9fEHUqkVEMJS6h9KCISoZKiAr56ykieXraVTXsOxR1OZFRDCYsSikgvcc2MozHgN/M/jDuUyKiGEhYlFJFeompQX748tYr731zPzgO1cYcTCdVQwqKEItKLXD/zGGobEjy0aGPcoUQi4a7FIQOihCLSixxXXsppxwzh3gXraMyBVYhVQwmLEopIL3PVaWPYsOsQ89/v+UsVJdzJ07dYMHQrRHqZ8yaWU1ZaxD1v9PxViLVjY1iUUER6mcL8PC4/ZSR/XLWdDbt69irErj6UoCihiPRCl586ijwz7u3hqxA3ulOg9euDoYQi0gtVDuzLOeOH8+DCDRyu77mrEDc0OvnqRAmG7oRIL3X16WPYVVPH40s2xx1Kl9U3JijMVw0lFEooIr3Up8cOZULlAG6bv5pEDx1C3JhwCpRQgqGEItJLmRk3nnUMq6treHHl9rjDSZu705BQk1dIdCdEerHZkyupGtSX2+avjjuUtDWkalWF6pQPhhKKSC9WmJ/HdWcczVtrd7NoXc/aK6WhMZlQCvL1NRYK3QmRXu6yU0YysG8ht/ewWkpDIgGgYcMBUUIR6eVKigq46rTRPLdiGx9WH4g7nE77qIaihBIKJRQR4WufHkNhfh6/eaXn7OhY31RDUZNXMHQnRISy0iLmTB3Bw29vpHp/z9grpWm1ZDV5hUMJRUQAuH7m0dQ3JrjrtbVxh9IpR5q8lFCCoYQiIgAcU9af8yaUc88b66ipbYg7nA7VNyabvArV5BUM3QkROeLGs8ay91A9D7y1Ie5QOnSkyUud8sFQQhGRI6aOGswpYwZzx6trjtQAQlWvJq/gKKGIyMfceOZYNu05xFNLt8QdSrs+moeir7FQ6E6IyMd89oThjC0r4baXP8Q93EUjG9TkFRwlFBH5mLw848Yzx7Jiyz5e/WBH3OG06aNRXvoaC4XuhIh8whdPPorhpUXc9vKHcYfSpobGpomNqqGEQglFRD6hqCCfa2aM4dUPdrBy6764w2nVkdWGlVCCoYQiIq264lOjKC7M4z9eXRt3KK1q6pTXfijh0J0QkVYN6teHOVNH8Lslm9h5ILzlWDRsODxKKCLSpmtnjKGuIcF9C9bHHconNHXKa6Z8OLJ6J8zsJjNbaGa1Zjavnev+3cwONDtqzWx/FkMVEeDY4aWcNa6Mu99YR11DWBMdP2ryUg0lFNlO7ZuBW4E727vI3f/c3fs3HcD9wEPZCFBEPu7aGWOo3l/LI29vjDuUj/mohqKEEoqsJhR3f8TdHwV2dvY9ZlYCzAHuylRcItK2s8aVcfKoQfzq+feDWjSyQfuhBKcn3Ik5QDUwP+5ARHojM+OWC8ezfX8tt88PZ15Kg/ZDCU5PSChfA+72NtaAMLMbUv0yC6urq7McmkjvMG30EC6cXMnt8z9k277DcYcDaD+UEAWdUMxsJHAWcHdb17j77e4+3d2nl5WVZS84kV7mu+efQGPC+fkzq+IOBfhoPxQ1eYUj9DtxNfCau4dTzxbppUYN7cd1M4/m4bc3smjdrrjD0RbAAcr2sOECMysG8oF8Mys2s4J23nI1MC8rwYlIh7752WM5amAxP3h0+ZG1tOKi1YbDk+0ayi3AIeBm4MrU41vMbFRqvsmopgvN7HRgBBouLBKMfn0K+MHnJ7Biyz5++8a6WGM5sgWwll4JRraHDc91d2txzHX39ak5J+ubXfu6u5e4uyY0igTk/EkVzDxuGL987n2q98e3JEtjwsmz5HL7EgaldhFJi5nxoy9M5HBDI3//9HuxxVHf6NoLJTC6GyKStmPK+nPdGcfwu8WbWLE5nuXtGxoT6j8JjBKKiHTJX5w9lgHFhfz82ZWx/PyGhGuEV2CUUESkSwb2LeQvzx7LS6uqeeG9bVn/+Q2JhOagBEZ3Q0S67GufHsP4ygF88/7FLN24N6s/u6FRNZTQKKGISJcVF+Yz79pTGNyvD9fOe4vtWVyWpb7RtRdKYHQ3RKRbygcU8x/XnkJNbQNX3/kmm/YcysrPbUwktBdKYJRQRKTbxpWXcvvV09i0+xB/+dtFRyYdZlJ9wjXKKzBKKCISiZnHlfGzS07knY17+fULf8r4z2toTGiWfGB0N0QkMrMnVzJn6gj+5Y8f8OS7WzL6sxoaXU1egVFCEZFI3XrxJKaNGsxfP/QO63bWZOznHKxrpKQoP2OfL+lTQhGRSPXtk88/X3EyBfnGJf/+Os8t35qRn3OwroF+fdpbrFyyTQlFRCJXObAv/3nDaZT1L+KGexbxzLLom78O1DbQv0gJJSRKKCKSEROPGsij35jB5KqBfO+RpeyuqYv08w/WNdKvj5q8QqKEIiIZ06cgj19ceiL7Djfwy+ej3Tr4QG0DJaqhBEUJRUQy6oSKAVx12mjuW7Ce5ZujWZ7F3dUpHyAlFBHJuO+cM45B/fow9/HluHu3P6+2IUFjwlVDCYwSiohk3MB+hfzNrON5a+1u7l2wvuM3dKCmtgGAEo3yCooSiohkxVemj+TMcWX8+IkVrNzavU25amobAVRDCYwSiohkRV6e8ctLT2JA30K+ff8SGrqx3ldNXVMNRX0oIVFCEZGsKSst4sdfnMSqbfu71fTV1OTVTzWUoCihiEhWzZpYzoxjh/Kr59/v8tyUmrpkk1d/jfIKihKKiGSVmfG3n59ITW0D19+9kAOp2kY6jtRQ1CkfFCUUEcm64ytK+fXlJ7Nkwx6uufNN6hrS609pSihaeiUsSigiEovZkyv5p8umsHDdbn7x7Mq03nsw1eSlpVfCooQiIrG56KSjuOLUUfy/V9ewZMOeTr9v76F6QMOGQ6OEIiKxuvmCExheWsT3Hlna6a2DV23dz4jBfSkuVA0lJEooIhKrAcWF/OgLk3hvyz7+4emVJBIdL83y7qY9nDhiYBaik3QooYhI7M6fVMHVp4/mjlfXcN7/ns/zK7a1ee2eg3Vs2HWIyVWDshegdIoSiogEYe5FE/k/X51CnsH1dy/kp0+9x56Dn5yn8trqnQCcpBpKcJRQRCQIeXnGF6dU8ftvnsGVp43i9vkfcvY/vsTaHR/fl37ef69lxOC+nHrM0JgilbYooYhIUIoK8rn14sk8+o0ZAFz+mzdY8GGyVrJh10HeXLuLK08bTX6exRmmtEIJRUSCNGXkIO75+qn0Lczn2nlvsXj9bl5atR2AcyeUxxydtCanBnF/WF3DZbe9HncYIhKhgX0L2bTnEJf8++u4O0UFeXzv4XcxUw0lNBbF7mmhMLP9QHc3rh4IdGaf0vaua+21js61fL3pefPzw4AdnYitPdkqX3vP23qcrfKlW7bWzsdRvkzdu9bOp1u+nvS72dq5XC5fZ75bjnf30k7E1j53z5kDWBjBZ9ze3etae62jcy1fb3re4poeU772nrfzOCvlS7dsoZQvU/cuivL1pN/N3la+bH23uLv6UFrx+wiua+21js61fP33bZzvrmyVr73n7ZW7uzrzeemWrbXzcZQvU/eutfO5VL50f19zrXzZ+m7JuSavhe4+Pe44MkXl69lyuXy5XDZQ+Tor12oot8cdQIapfD1bLpcvl8sGKl+n5FQNRURE4pNrNRQREYmJEoqIiESi1yUUMxtjZtVm9lLqKIs7pqiZ2eVmVh13HFEzs3Ize83MXjazF82sMu6YomRmp5vZ66ny3W9mhXHHFCUzG2hmb5rZATObFHc8UTCzn5jZK2b2X2bWL+54otSV+9XrEkrKy+5+durIqS9eM8sDLgE2xB1LBuwAznD3s4C7getijidq64DPpsr3IfDFmOOJ2kHgQuC/4g4kCqkv2bHuPhP4A/D1mEOKWtr3q7cmlBmp/1X81HJv/YYrSP4CdG7rux7E3RvdvalcpcDyOOOJmrtvdvdDqacN5Ng9dPf6HPsP3Ezg6dTjp4EzYowlcl25X0EnFDO7ycwWmlmtmc1r8doQM/udmdWY2Tozu6KTH7sFOBY4ExgOfDnaqDsnE2Uzs3zgK8ADGQg5LRm6d5jZFDNbANwEvB1x2J2WqfKl3n80cAHwRIQhpyWT5QtNN8o6mI+WLtkLDMlSyGnJ5r0MfXHIzcCtwCygb4vX/hWoA8qBKcCTZvaOuy83swpar6Zd4u5bgVoAM3sEOA14ODPhtyvysqU+60F3TwRQ8crIvXP3JcCpZvYV4HvAn2co/o5kpHxmNgC4C7jK3T+5u1T2ZOrfXoi6VFZgN8n1sEj9uSsr0aavq+VLXxTrt2T6SP1lzGv2vCT1lzCu2bl7gH/oxGcNaPb474Grc6hsPwOeA54h+T+mX+fYvStq9ngW8KscK18B8CTJfpRYy5WJ8jW7fh4wKe6ydbeswGTgvtTjG4Bvxl2GTNzLdO5X0E1e7RgHNLr7+83OvQNM7MR7zzKzRWb2ClAF3JeJALuhy2Vz9++6+3nufj7wJ3f/VqaC7Ibu3LupZjbfzP4I/E/gFxmIr7u6U77LgVOBv02NQLwsEwF2U3fKh5k9BZwH/MbMrok+vEi1W1Z3XwqsS32XzALuzH6I3dLhvUz3foXe5NWW/nxyuea9JDtq2+XuvycDi6JFqMtla87DXXeoO/fudZJ9XyHrTvnuIfk/xJB16/fT3WdHHlHmdFhWd/9eViOKVmfKl9b96qk1lAPAgBbnBgD7Y4glarlcNlD5erpcL19zuV7WyMvXUxPK+0CBmR3X7NxJ5MYw0lwuG6h8PV2ul6+5XC9r5OULOqGYWYGZFQP5QL6ZFZtZgbvXAI8Af2dmJWY2g+QksNCbC47I5bKByofK12PkelmzWr64Rx50MCphLuAtjrmp14YAjwI1wHrgirjjVdlUPpWv5x25XtZslk/L14uISCSCbvISEZGeQwlFREQioYQiIiKRUEIREZFIKKGIiEgklFBERCQSSigiIhIJJRSRCJnZXDNbFnccInHQxEbpcVK7zg1z98/HHUtLZtaf5L4tO+OOpS1m5sCl7p4Te7tLOFRDEekEM+vTmevc/UAcycTM8iy5BbRIbJRQJOeY2QQze9LM9pvZdjO7P7U1bdPrp5jZc2a2w8z2mdmrZnZ6i89wM/uGmT1iZjXAT5uas8zsq2a2OvX5j5rZsGbv+1iTl5nNM7MnzOzbZrbJzHab2X+YWb9m15SY2d1mdsDMtpnZ91LvmddOGa9JXT879fPqgPEdlc3M1qYePpQq49pmr12U2nzusJmtMbOfdDaRioASiuQYM6sE5gPLgE8B55DcSOhxM2v6fS8luaLqzNQ1S4CnmieGlB8CT5Hc6vVfU+fGAJcBXyK5k93JwE86CGsmMCkVS9N7v93s9V8CZ6XOf5bkEuIzO1HcYuAW4EZgArCuE2U7JfXn9UBl03MzmwXcC/wLyR37vg5cAvy0E3GIJMW9EqYOHekeJPe4fqKN1/4OeKHFucEkV1j9VBvvMWALcGWzcw78c4vr5gKHgYHNzn0f+KDFNctaxLoBKGh27jfAH1KP+5OsXXy12eslwG6a7f/dSszXpGKc1sHfVVtlu6TFdfOBH7Q4dzHJTZgs7nuuo2ccqqFIrpkGnJlqDjpgZgdIfqEDjAUws+FmdpuZvW9me0nuUDccGNXisxa28vnr3L35tqmbU+9tzwp3b2jjPWOBQuDNphc9uU9FZ0aKNZCsgRyRRtlamgZ8v8Xf230kk1tF+28VSeqpe8qLtCUPeBL461Ze25b68y6gHPgOsBaoBV4AWvYX1LTyGfUtnjsdNx239x5rdi5dte7e2OJcZ8vWUh7wI+ChVl6r7kJs0gspoUiueRv4CsmaRMsv8iZnAN9y9ycBzKycZH9CHD4gmXA+BaxJxdOPZJ/L6i58XmfKVk9y977m3gZOcPcPuvAzRQAlFOm5BpjZlBbn9pDsPL8eeMDMfkbyf9fHkEwyf+Xu+0nupX2lmS0g2aTzc5L9GFnn7gfM7E7gZ2a2g2R/xy0kawxdqbV0pmxrgc+Z2cskazm7SfY9PWFm64AHSTanTSLZ7/Q3XYhDeiH1oUhPNRNY3OL4R3ffDMwAEsAzwHKSSaY2dUByBFN/YBHwn8CdJL9k4/LXwCvA48AfgXdJ9t8c7sJndaZsfwV8hmTf0mIAd38WuDB1/s3UcTPJbWFFOkUz5UUCY2ZFJIcA/8Ldfxl3PCKdpSYvkZiZ2cnAeJK1glLgu6k/H4gzLpF0KaGIhOF/Acfz0VDgM919Y6wRiaRJTV4iIhIJdcqLiEgklFBERCQSSigiIhIJJRQREYmEEoqIiERCCUVERCLx/wGKUUuRqdIcAQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "rates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1, batch_size=batch_size)\n",
    "plot_lr_vs_loss(rates, losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "class OneCycleScheduler(keras.callbacks.Callback):\n",
    "    def __init__(self, iterations, max_rate, start_rate=None,\n",
    "                 last_iterations=None, last_rate=None):\n",
    "        self.iterations = iterations\n",
    "        self.max_rate = max_rate\n",
    "        self.start_rate = start_rate or max_rate / 10\n",
    "        self.last_iterations = last_iterations or iterations // 10 + 1\n",
    "        self.half_iteration = (iterations - self.last_iterations) // 2\n",
    "        self.last_rate = last_rate or self.start_rate / 1000\n",
    "        self.iteration = 0\n",
    "    def _interpolate(self, iter1, iter2, rate1, rate2):\n",
    "        return ((rate2 - rate1) * (self.iteration - iter1)\n",
    "                / (iter2 - iter1) + rate1)\n",
    "    def on_batch_begin(self, batch, logs):\n",
    "        if self.iteration < self.half_iteration:\n",
    "            rate = self._interpolate(0, self.half_iteration, self.start_rate, self.max_rate)\n",
    "        elif self.iteration < 2 * self.half_iteration:\n",
    "            rate = self._interpolate(self.half_iteration, 2 * self.half_iteration,\n",
    "                                     self.max_rate, self.start_rate)\n",
    "        else:\n",
    "            rate = self._interpolate(2 * self.half_iteration, self.iterations,\n",
    "                                     self.start_rate, self.last_rate)\n",
    "        self.iteration += 1\n",
    "        K.set_value(self.model.optimizer.lr, rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.6572 - accuracy: 0.7740 - val_loss: 0.4872 - val_accuracy: 0.8338\n",
      "Epoch 2/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.4580 - accuracy: 0.8397 - val_loss: 0.4274 - val_accuracy: 0.8520\n",
      "Epoch 3/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.4121 - accuracy: 0.8545 - val_loss: 0.4116 - val_accuracy: 0.8588\n",
      "Epoch 4/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3837 - accuracy: 0.8642 - val_loss: 0.3868 - val_accuracy: 0.8688\n",
      "Epoch 5/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3639 - accuracy: 0.8719 - val_loss: 0.3766 - val_accuracy: 0.8688\n",
      "Epoch 6/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3456 - accuracy: 0.8775 - val_loss: 0.3739 - val_accuracy: 0.8706\n",
      "Epoch 7/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3330 - accuracy: 0.8811 - val_loss: 0.3635 - val_accuracy: 0.8708\n",
      "Epoch 8/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3184 - accuracy: 0.8861 - val_loss: 0.3959 - val_accuracy: 0.8610\n",
      "Epoch 9/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3065 - accuracy: 0.8890 - val_loss: 0.3475 - val_accuracy: 0.8770\n",
      "Epoch 10/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2943 - accuracy: 0.8927 - val_loss: 0.3392 - val_accuracy: 0.8806\n",
      "Epoch 11/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2838 - accuracy: 0.8963 - val_loss: 0.3467 - val_accuracy: 0.8800\n",
      "Epoch 12/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2707 - accuracy: 0.9024 - val_loss: 0.3646 - val_accuracy: 0.8696\n",
      "Epoch 13/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2536 - accuracy: 0.9079 - val_loss: 0.3350 - val_accuracy: 0.8842\n",
      "Epoch 14/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2405 - accuracy: 0.9135 - val_loss: 0.3465 - val_accuracy: 0.8794\n",
      "Epoch 15/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2279 - accuracy: 0.9185 - val_loss: 0.3257 - val_accuracy: 0.8830\n",
      "Epoch 16/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2159 - accuracy: 0.9232 - val_loss: 0.3294 - val_accuracy: 0.8824\n",
      "Epoch 17/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2062 - accuracy: 0.9263 - val_loss: 0.3333 - val_accuracy: 0.8882\n",
      "Epoch 18/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1978 - accuracy: 0.9301 - val_loss: 0.3235 - val_accuracy: 0.8898\n",
      "Epoch 19/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1892 - accuracy: 0.9337 - val_loss: 0.3233 - val_accuracy: 0.8906\n",
      "Epoch 20/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1821 - accuracy: 0.9365 - val_loss: 0.3224 - val_accuracy: 0.8928\n",
      "Epoch 21/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1752 - accuracy: 0.9400 - val_loss: 0.3220 - val_accuracy: 0.8908\n",
      "Epoch 22/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1700 - accuracy: 0.9416 - val_loss: 0.3180 - val_accuracy: 0.8962\n",
      "Epoch 23/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1655 - accuracy: 0.9438 - val_loss: 0.3187 - val_accuracy: 0.8940\n",
      "Epoch 24/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1627 - accuracy: 0.9454 - val_loss: 0.3177 - val_accuracy: 0.8932\n",
      "Epoch 25/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1610 - accuracy: 0.9462 - val_loss: 0.3170 - val_accuracy: 0.8934\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 25\n",
    "onecycle = OneCycleScheduler(math.ceil(len(X_train) / batch_size) * n_epochs, max_rate=0.05)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs, batch_size=batch_size,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[onecycle])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Avoiding Overfitting Through Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $\\ell_1$ and $\\ell_2$ regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(100, activation=\"elu\",\n",
    "                           kernel_initializer=\"he_normal\",\n",
    "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "# or l1(0.1) for ℓ1 regularization with a factor or 0.1\n",
    "# or l1_l2(0.1, 0.01) for both ℓ1 and ℓ2 regularization, with factors 0.1 and 0.01 respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 3.2189 - accuracy: 0.7967 - val_loss: 0.7169 - val_accuracy: 0.8340\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.7280 - accuracy: 0.8247 - val_loss: 0.6850 - val_accuracy: 0.8376\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(100, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(10, activation=\"softmax\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 3.2911 - accuracy: 0.7924 - val_loss: 0.7218 - val_accuracy: 0.8310\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.7282 - accuracy: 0.8245 - val_loss: 0.6826 - val_accuracy: 0.8382\n"
     ]
    }
   ],
   "source": [
    "from functools import partial\n",
    "\n",
    "RegularizedDense = partial(keras.layers.Dense,\n",
    "                           activation=\"elu\",\n",
    "                           kernel_initializer=\"he_normal\",\n",
    "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    RegularizedDense(300),\n",
    "    RegularizedDense(100),\n",
    "    RegularizedDense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.7611 - accuracy: 0.7576 - val_loss: 0.3730 - val_accuracy: 0.8644\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4306 - accuracy: 0.8401 - val_loss: 0.3395 - val_accuracy: 0.8722\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(300, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(100, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alpha Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.8023 - accuracy: 0.7146 - val_loss: 0.5781 - val_accuracy: 0.8442\n",
      "Epoch 2/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.5663 - accuracy: 0.7905 - val_loss: 0.5182 - val_accuracy: 0.8520\n",
      "Epoch 3/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5264 - accuracy: 0.8054 - val_loss: 0.4874 - val_accuracy: 0.8600\n",
      "Epoch 4/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5126 - accuracy: 0.8092 - val_loss: 0.4890 - val_accuracy: 0.8598\n",
      "Epoch 5/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5071 - accuracy: 0.8133 - val_loss: 0.4266 - val_accuracy: 0.8696\n",
      "Epoch 6/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4793 - accuracy: 0.8198 - val_loss: 0.4585 - val_accuracy: 0.8640\n",
      "Epoch 7/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4724 - accuracy: 0.8262 - val_loss: 0.4740 - val_accuracy: 0.8612\n",
      "Epoch 8/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4570 - accuracy: 0.8297 - val_loss: 0.4295 - val_accuracy: 0.8656\n",
      "Epoch 9/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4632 - accuracy: 0.8286 - val_loss: 0.4357 - val_accuracy: 0.8736\n",
      "Epoch 10/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4552 - accuracy: 0.8340 - val_loss: 0.4366 - val_accuracy: 0.8674\n",
      "Epoch 11/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4461 - accuracy: 0.8346 - val_loss: 0.4278 - val_accuracy: 0.8684\n",
      "Epoch 12/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4419 - accuracy: 0.8351 - val_loss: 0.5086 - val_accuracy: 0.8558\n",
      "Epoch 13/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4329 - accuracy: 0.8385 - val_loss: 0.4280 - val_accuracy: 0.8728\n",
      "Epoch 14/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4305 - accuracy: 0.8399 - val_loss: 0.4460 - val_accuracy: 0.8628\n",
      "Epoch 15/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4315 - accuracy: 0.8397 - val_loss: 0.4361 - val_accuracy: 0.8706\n",
      "Epoch 16/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4251 - accuracy: 0.8403 - val_loss: 0.4280 - val_accuracy: 0.8758\n",
      "Epoch 17/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4207 - accuracy: 0.8427 - val_loss: 0.5336 - val_accuracy: 0.8584\n",
      "Epoch 18/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4365 - accuracy: 0.8387 - val_loss: 0.4769 - val_accuracy: 0.8736\n",
      "Epoch 19/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4262 - accuracy: 0.8409 - val_loss: 0.4636 - val_accuracy: 0.8706\n",
      "Epoch 20/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4189 - accuracy: 0.8421 - val_loss: 0.4388 - val_accuracy: 0.8760\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 20\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 0s 834us/step - loss: 0.4723 - accuracy: 0.8639\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.47229740023612976, 0.8639000058174133]"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1719/1719 [==============================] - 1s 782us/step - loss: 0.3501 - accuracy: 0.8840\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.3501231074333191, 0.8840363621711731]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4225 - accuracy: 0.8432\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MC Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_probas = np.stack([model(X_test_scaled, training=True)\n",
    "                     for sample in range(100)])\n",
    "y_proba = y_probas.mean(axis=0)\n",
    "y_std = y_probas.std(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.96]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(model.predict(X_test_scaled[:1]), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.57, 0.  , 0.42]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.93, 0.  , 0.05]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.  , 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.08, 0.  , 0.92]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.46, 0.  , 0.53]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.75, 0.  , 0.24]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.43, 0.  , 0.51]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.13, 0.  , 0.28, 0.  , 0.59]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.06, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.02, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.07, 0.  , 0.92]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.08, 0.  , 0.28, 0.  , 0.64]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.29, 0.  , 0.7 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.26, 0.  , 0.22, 0.  , 0.52]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.25, 0.  , 0.72]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.3 , 0.  , 0.06, 0.  , 0.65]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.  , 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.81, 0.  , 0.17]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.25, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.03, 0.  , 0.93]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.84, 0.04, 0.01, 0.  , 0.1 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.2 , 0.  , 0.1 , 0.  , 0.7 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.28, 0.  , 0.72]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.12, 0.  , 0.76]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.75, 0.  , 0.21]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.11, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.28, 0.  , 0.39, 0.  , 0.34]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.04, 0.  , 0.95]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.18, 0.  , 0.51, 0.01, 0.3 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.59, 0.  , 0.01, 0.  , 0.39]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.8 , 0.  , 0.08]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.27, 0.  , 0.67]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.07, 0.  , 0.77, 0.  , 0.16]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.18, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.32, 0.  , 0.64]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.75, 0.  , 0.04, 0.  , 0.21]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.24, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.44, 0.  , 0.38, 0.  , 0.17]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.02, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.12, 0.  , 0.82]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.97]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.28, 0.  , 0.71]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.17, 0.  , 0.82]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.23, 0.  , 0.27, 0.  , 0.5 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.78, 0.  , 0.16]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.03, 0.  , 0.94]],\n",
       "\n",
       "       [[0.  , 0.  , 0.01, 0.  , 0.02, 0.21, 0.01, 0.01, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.06, 0.  , 0.93]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.4 , 0.  , 0.6 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.18, 0.01, 0.69]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.15, 0.  , 0.83]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.24, 0.  , 0.24, 0.  , 0.52]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.11, 0.  , 0.01, 0.  , 0.88]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.2 , 0.  , 0.8 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.04, 0.  , 0.91]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.21, 0.  , 0.77]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.26, 0.  , 0.67, 0.  , 0.07]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.27, 0.  , 0.55, 0.  , 0.19]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.09, 0.  , 0.47, 0.  , 0.43]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.07, 0.  , 0.71, 0.  , 0.22]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.1 , 0.  , 0.63, 0.01, 0.26]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.16, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.15, 0.  , 0.8 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.24, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.13, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.42, 0.  , 0.01, 0.  , 0.58]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.15, 0.  , 0.83]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.79, 0.  , 0.19]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.21, 0.  , 0.79]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.93]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.12, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.06, 0.  , 0.9 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.01, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.08, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.51, 0.  , 0.46]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.96, 0.  , 0.02, 0.  , 0.02]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.38, 0.  , 0.05, 0.  , 0.57]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.07, 0.  , 0.92]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.06, 0.  , 0.89]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.05, 0.  , 0.94]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.37, 0.  , 0.37, 0.  , 0.26]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.37, 0.  , 0.19, 0.  , 0.44]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.11, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.01, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.41, 0.  , 0.59]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.08, 0.  , 0.21, 0.  , 0.71]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.13, 0.  , 0.75]]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(y_probas[:, :1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.1 , 0.  , 0.22, 0.  , 0.68]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(y_proba[:1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.18, 0.  , 0.24, 0.  , 0.29]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_std = y_probas.std(axis=0)\n",
    "np.round(y_std[:1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.argmax(y_proba, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8661"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy = np.sum(y_pred == y_test) / len(y_test)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MCDropout(keras.layers.Dropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)\n",
    "\n",
    "class MCAlphaDropout(keras.layers.AlphaDropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model = keras.models.Sequential([\n",
    "    MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout) else layer\n",
    "    for layer in model.layers\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_20\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_18 (Flatten)         (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout (MCAlphaDro (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_262 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout_1 (MCAlphaD (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "dense_263 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout_2 (MCAlphaD (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "dense_264 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 266,610\n",
      "Trainable params: 266,610\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "mc_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)\n",
    "mc_model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model.set_weights(model.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use the model with MC Dropout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.14, 0.  , 0.25, 0.01, 0.61]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(np.mean([mc_model.predict(X_test_scaled[:1]) for sample in range(100)], axis=0), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Max norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                           kernel_constraint=keras.constraints.max_norm(1.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.5763 - accuracy: 0.8020 - val_loss: 0.3674 - val_accuracy: 0.8674\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3545 - accuracy: 0.8709 - val_loss: 0.3714 - val_accuracy: 0.8662\n"
     ]
    }
   ],
   "source": [
    "MaxNormDense = partial(keras.layers.Dense,\n",
    "                       activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       kernel_constraint=keras.constraints.max_norm(1.))\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    MaxNormDense(300),\n",
    "    MaxNormDense(100),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See appendix A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Deep Learning on CIFAR10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a.\n",
    "*Exercise: Build a DNN with 20 hidden layers of 100 neurons each (that's too many, but it's the point of this exercise). Use He initialization and the ELU activation function.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 activation=\"elu\",\n",
    "                                 kernel_initializer=\"he_normal\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b.\n",
    "*Exercise: Using Nadam optimization and early stopping, train the network on the CIFAR10 dataset. You can load it with `keras.datasets.cifar10.load_data()`. The dataset is composed of 60,000 32 × 32–pixel color images (50,000 for training, 10,000 for testing) with 10 classes, so you'll need a softmax output layer with 10 neurons. Remember to search for the right learning rate each time you change the model's architecture or hyperparameters.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add the output layer to the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use a Nadam optimizer with a learning rate of 5e-5. I tried learning rates 1e-5, 3e-5, 1e-4, 3e-4, 1e-3, 3e-3 and 1e-2, and I compared their learning curves for 10 epochs each (using the TensorBoard callback, below). The learning rates 3e-5 and 1e-4 were pretty good, so I tried 5e-5, which turned out to be slightly better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Nadam(lr=5e-5)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the CIFAR10 dataset. We also want to use early stopping, so we need a validation set. Let's use the first 5,000 images of the original training set as the validation set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.cifar10.load_data()\n",
    "\n",
    "X_train = X_train_full[5000:]\n",
    "y_train = y_train_full[5000:]\n",
    "X_valid = X_train_full[:5000]\n",
    "y_valid = y_train_full[:5000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create the callbacks we need and train the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ERROR: Failed to launch TensorBoard (exited with 255).\n",
       "Contents of stderr:\n",
       "E0213 19:12:55.493896 4621630912 program.py:311] TensorBoard could not bind to port 6006, it was already in use\n",
       "ERROR: TensorBoard could not bind to port 6006, it was already in use"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%tensorboard --logdir=./my_cifar10_logs --port=6006"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1407/1407 [==============================] - 9s 5ms/step - loss: 9.4191 - accuracy: 0.1388 - val_loss: 2.2328 - val_accuracy: 0.2040\n",
      "Epoch 2/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 2.1097 - accuracy: 0.2317 - val_loss: 2.0485 - val_accuracy: 0.2402\n",
      "Epoch 3/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.9667 - accuracy: 0.2844 - val_loss: 1.9681 - val_accuracy: 0.2964\n",
      "Epoch 4/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.8740 - accuracy: 0.3149 - val_loss: 1.9178 - val_accuracy: 0.3254\n",
      "Epoch 5/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.8064 - accuracy: 0.3423 - val_loss: 1.8256 - val_accuracy: 0.3384\n",
      "Epoch 6/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.7525 - accuracy: 0.3595 - val_loss: 1.7430 - val_accuracy: 0.3692\n",
      "Epoch 7/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.7116 - accuracy: 0.3819 - val_loss: 1.7199 - val_accuracy: 0.3824\n",
      "Epoch 8/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.6782 - accuracy: 0.3935 - val_loss: 1.6746 - val_accuracy: 0.3972\n",
      "Epoch 9/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.6517 - accuracy: 0.4025 - val_loss: 1.6622 - val_accuracy: 0.4004\n",
      "Epoch 10/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.6140 - accuracy: 0.4194 - val_loss: 1.7065 - val_accuracy: 0.3840\n",
      "Epoch 11/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5884 - accuracy: 0.4301 - val_loss: 1.6736 - val_accuracy: 0.3914\n",
      "Epoch 12/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5640 - accuracy: 0.4378 - val_loss: 1.6220 - val_accuracy: 0.4224\n",
      "Epoch 13/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5437 - accuracy: 0.4448 - val_loss: 1.6332 - val_accuracy: 0.4144\n",
      "Epoch 14/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5214 - accuracy: 0.4555 - val_loss: 1.5785 - val_accuracy: 0.4326\n",
      "Epoch 15/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5117 - accuracy: 0.4564 - val_loss: 1.6267 - val_accuracy: 0.4164\n",
      "Epoch 16/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4972 - accuracy: 0.4622 - val_loss: 1.5846 - val_accuracy: 0.4316\n",
      "Epoch 17/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4888 - accuracy: 0.4661 - val_loss: 1.5549 - val_accuracy: 0.4420\n",
      "Epoch 18/100\n",
      "<<24 more lines>>\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.3362 - accuracy: 0.5212 - val_loss: 1.6025 - val_accuracy: 0.4500\n",
      "Epoch 31/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3360 - accuracy: 0.5207 - val_loss: 1.5175 - val_accuracy: 0.4602\n",
      "Epoch 32/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.3031 - accuracy: 0.5302 - val_loss: 1.5397 - val_accuracy: 0.4572\n",
      "Epoch 33/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.3082 - accuracy: 0.5308 - val_loss: 1.4997 - val_accuracy: 0.4776\n",
      "Epoch 34/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.2882 - accuracy: 0.5338 - val_loss: 1.5482 - val_accuracy: 0.4620\n",
      "Epoch 35/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.2889 - accuracy: 0.5355 - val_loss: 1.5474 - val_accuracy: 0.4604\n",
      "Epoch 36/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2761 - accuracy: 0.5410 - val_loss: 1.5434 - val_accuracy: 0.4658\n",
      "Epoch 37/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2658 - accuracy: 0.5481 - val_loss: 1.5502 - val_accuracy: 0.4706\n",
      "Epoch 38/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2554 - accuracy: 0.5489 - val_loss: 1.5527 - val_accuracy: 0.4624\n",
      "Epoch 39/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2504 - accuracy: 0.5471 - val_loss: 1.5482 - val_accuracy: 0.4602\n",
      "Epoch 40/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2516 - accuracy: 0.5545 - val_loss: 1.5881 - val_accuracy: 0.4574\n",
      "Epoch 41/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2401 - accuracy: 0.5566 - val_loss: 1.5403 - val_accuracy: 0.4670\n",
      "Epoch 42/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.2305 - accuracy: 0.5570 - val_loss: 1.5343 - val_accuracy: 0.4790\n",
      "Epoch 43/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2228 - accuracy: 0.5615 - val_loss: 1.5344 - val_accuracy: 0.4708\n",
      "Epoch 44/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2161 - accuracy: 0.5619 - val_loss: 1.5782 - val_accuracy: 0.4526\n",
      "Epoch 45/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2124 - accuracy: 0.5641 - val_loss: 1.5182 - val_accuracy: 0.4794\n",
      "Epoch 46/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1870 - accuracy: 0.5766 - val_loss: 1.5435 - val_accuracy: 0.4650\n",
      "Epoch 47/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1925 - accuracy: 0.5701 - val_loss: 1.5532 - val_accuracy: 0.4686\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb1d8ef8790>"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, epochs=100,\n",
    "          validation_data=(X_valid, y_valid),\n",
    "          callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "157/157 [==============================] - 0s 1ms/step - loss: 1.4960 - accuracy: 0.4762\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.4960416555404663, 0.47620001435279846]"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.load_model(\"my_cifar10_model.h5\")\n",
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model with the lowest validation loss gets about 47.6% accuracy on the validation set. It took 27 epochs to reach the lowest validation loss, with roughly 8 seconds per epoch on my laptop (without a GPU). Let's see if we can improve performance using Batch Normalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### c.\n",
    "*Exercise: Now try adding Batch Normalization and compare the learning curves: Is it converging faster than before? Does it produce a better model? How does it affect training speed?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below is very similar to the code above, with a few changes:\n",
    "\n",
    "* I added a BN layer after every Dense layer (before the activation function), except for the output layer. I also added a BN layer before the first hidden layer.\n",
    "* I changed the learning rate to 5e-4. I experimented with 1e-5, 3e-5, 5e-5, 1e-4, 3e-4, 5e-4, 1e-3 and 3e-3, and I chose the one with the best validation performance after 20 epochs.\n",
    "* I renamed the run directories to run_bn_* and the model file name to my_cifar10_bn_model.h5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1407/1407 [==============================] - 19s 9ms/step - loss: 1.9765 - accuracy: 0.2968 - val_loss: 1.6602 - val_accuracy: 0.4042\n",
      "Epoch 2/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.6787 - accuracy: 0.4056 - val_loss: 1.5887 - val_accuracy: 0.4304\n",
      "Epoch 3/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.6097 - accuracy: 0.4274 - val_loss: 1.5781 - val_accuracy: 0.4326\n",
      "Epoch 4/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.5574 - accuracy: 0.4486 - val_loss: 1.5064 - val_accuracy: 0.4676\n",
      "Epoch 5/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.5075 - accuracy: 0.4642 - val_loss: 1.4412 - val_accuracy: 0.4844\n",
      "Epoch 6/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.4664 - accuracy: 0.4787 - val_loss: 1.4179 - val_accuracy: 0.4984\n",
      "Epoch 7/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.4334 - accuracy: 0.4932 - val_loss: 1.4277 - val_accuracy: 0.4906\n",
      "Epoch 8/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.4054 - accuracy: 0.5038 - val_loss: 1.3843 - val_accuracy: 0.5130\n",
      "Epoch 9/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.3816 - accuracy: 0.5106 - val_loss: 1.3691 - val_accuracy: 0.5108\n",
      "Epoch 10/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.3547 - accuracy: 0.5206 - val_loss: 1.3552 - val_accuracy: 0.5226\n",
      "Epoch 11/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 1.3244 - accuracy: 0.5371 - val_loss: 1.3678 - val_accuracy: 0.5142\n",
      "Epoch 12/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.3078 - accuracy: 0.5393 - val_loss: 1.3844 - val_accuracy: 0.5080\n",
      "Epoch 13/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 1.2889 - accuracy: 0.5431 - val_loss: 1.3566 - val_accuracy: 0.5164\n",
      "Epoch 14/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 1.2607 - accuracy: 0.5559 - val_loss: 1.3626 - val_accuracy: 0.5248\n",
      "Epoch 15/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.2580 - accuracy: 0.5587 - val_loss: 1.3616 - val_accuracy: 0.5276\n",
      "Epoch 16/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.2441 - accuracy: 0.5586 - val_loss: 1.3350 - val_accuracy: 0.5286\n",
      "Epoch 17/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.2241 - accuracy: 0.5676 - val_loss: 1.3370 - val_accuracy: 0.5408\n",
      "Epoch 18/100\n",
      "<<29 more lines>>\n",
      "Epoch 33/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.0336 - accuracy: 0.6369 - val_loss: 1.3682 - val_accuracy: 0.5450\n",
      "Epoch 34/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.0228 - accuracy: 0.6388 - val_loss: 1.3348 - val_accuracy: 0.5458\n",
      "Epoch 35/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.0205 - accuracy: 0.6407 - val_loss: 1.3490 - val_accuracy: 0.5440\n",
      "Epoch 36/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 1.0008 - accuracy: 0.6489 - val_loss: 1.3568 - val_accuracy: 0.5408\n",
      "Epoch 37/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9785 - accuracy: 0.6543 - val_loss: 1.3628 - val_accuracy: 0.5396\n",
      "Epoch 38/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9832 - accuracy: 0.6592 - val_loss: 1.3617 - val_accuracy: 0.5482\n",
      "Epoch 39/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 0.9707 - accuracy: 0.6581 - val_loss: 1.3767 - val_accuracy: 0.5446\n",
      "Epoch 40/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9590 - accuracy: 0.6651 - val_loss: 1.4200 - val_accuracy: 0.5314\n",
      "Epoch 41/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9548 - accuracy: 0.6668 - val_loss: 1.3692 - val_accuracy: 0.5450\n",
      "Epoch 42/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9480 - accuracy: 0.6667 - val_loss: 1.3841 - val_accuracy: 0.5310\n",
      "Epoch 43/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9411 - accuracy: 0.6716 - val_loss: 1.4036 - val_accuracy: 0.5382\n",
      "Epoch 44/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9383 - accuracy: 0.6708 - val_loss: 1.4114 - val_accuracy: 0.5236\n",
      "Epoch 45/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9258 - accuracy: 0.6769 - val_loss: 1.4224 - val_accuracy: 0.5324\n",
      "Epoch 46/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9072 - accuracy: 0.6836 - val_loss: 1.3875 - val_accuracy: 0.5442\n",
      "Epoch 47/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.8996 - accuracy: 0.6850 - val_loss: 1.4449 - val_accuracy: 0.5280\n",
      "Epoch 48/100\n",
      "1407/1407 [==============================] - 13s 9ms/step - loss: 0.9050 - accuracy: 0.6835 - val_loss: 1.4167 - val_accuracy: 0.5338\n",
      "Epoch 49/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.8934 - accuracy: 0.6880 - val_loss: 1.4260 - val_accuracy: 0.5294\n",
      "157/157 [==============================] - 1s 2ms/step - loss: 1.3344 - accuracy: 0.5398\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.3343921899795532, 0.5397999882698059]"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "model.add(keras.layers.BatchNormalization())\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100, kernel_initializer=\"he_normal\"))\n",
    "    model.add(keras.layers.BatchNormalization())\n",
    "    model.add(keras.layers.Activation(\"elu\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.Nadam(lr=5e-4)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_bn_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_bn_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
    "\n",
    "model.fit(X_train, y_train, epochs=100,\n",
    "          validation_data=(X_valid, y_valid),\n",
    "          callbacks=callbacks)\n",
    "\n",
    "model = keras.models.load_model(\"my_cifar10_bn_model.h5\")\n",
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* *Is the model converging faster than before?* Much faster! The previous model took 27 epochs to reach the lowest validation loss, while the new model achieved that same loss in just 5 epochs and continued to make progress until the 16th epoch. The BN layers stabilized training and allowed us to use a much larger learning rate, so convergence was faster.\n",
    "* *Does BN produce a better model?* Yes! The final model is also much better, with 54.0% accuracy instead of 47.6%. It's still not a very good model, but at least it's much better than before (a Convolutional Neural Network would do much better, but that's a different topic, see chapter 14).\n",
    "* *How does BN affect training speed?* Although the model converged much faster, each epoch took about 12s instead of 8s, because of the extra computations required by the BN layers. But overall the training time (wall time) was shortened significantly!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### d.\n",
    "*Exercise: Try replacing Batch Normalization with SELU, and make the necessary adjustements to ensure the network self-normalizes (i.e., standardize the input features, use LeCun normal initialization, make sure the DNN contains only a sequence of dense layers, etc.).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1407/1407 [==============================] - 10s 5ms/step - loss: 2.0622 - accuracy: 0.2631 - val_loss: 1.7878 - val_accuracy: 0.3552\n",
      "Epoch 2/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.7328 - accuracy: 0.3830 - val_loss: 1.7028 - val_accuracy: 0.3828\n",
      "Epoch 3/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.6342 - accuracy: 0.4279 - val_loss: 1.6692 - val_accuracy: 0.4022\n",
      "Epoch 4/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5524 - accuracy: 0.4538 - val_loss: 1.6350 - val_accuracy: 0.4300\n",
      "Epoch 5/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4979 - accuracy: 0.4756 - val_loss: 1.5773 - val_accuracy: 0.4356\n",
      "Epoch 6/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4428 - accuracy: 0.4902 - val_loss: 1.5529 - val_accuracy: 0.4630\n",
      "Epoch 7/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3966 - accuracy: 0.5126 - val_loss: 1.5290 - val_accuracy: 0.4682\n",
      "Epoch 8/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3549 - accuracy: 0.5232 - val_loss: 1.4633 - val_accuracy: 0.4792\n",
      "Epoch 9/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3162 - accuracy: 0.5444 - val_loss: 1.4787 - val_accuracy: 0.4776\n",
      "Epoch 10/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2825 - accuracy: 0.5534 - val_loss: 1.4794 - val_accuracy: 0.4934\n",
      "Epoch 11/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2529 - accuracy: 0.5682 - val_loss: 1.5529 - val_accuracy: 0.4982\n",
      "Epoch 12/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2256 - accuracy: 0.5784 - val_loss: 1.4942 - val_accuracy: 0.4902\n",
      "Epoch 13/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2049 - accuracy: 0.5823 - val_loss: 1.4868 - val_accuracy: 0.5024\n",
      "Epoch 14/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1627 - accuracy: 0.6012 - val_loss: 1.4839 - val_accuracy: 0.5082\n",
      "Epoch 15/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1543 - accuracy: 0.6034 - val_loss: 1.5097 - val_accuracy: 0.4968\n",
      "Epoch 16/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1200 - accuracy: 0.6135 - val_loss: 1.5001 - val_accuracy: 0.5120\n",
      "Epoch 17/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1028 - accuracy: 0.6199 - val_loss: 1.4856 - val_accuracy: 0.5056\n",
      "Epoch 18/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0863 - accuracy: 0.6265 - val_loss: 1.5116 - val_accuracy: 0.4966\n",
      "Epoch 19/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0715 - accuracy: 0.6345 - val_loss: 1.5787 - val_accuracy: 0.5070\n",
      "Epoch 20/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0342 - accuracy: 0.6453 - val_loss: 1.4987 - val_accuracy: 0.5144\n",
      "Epoch 21/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0169 - accuracy: 0.6531 - val_loss: 1.6292 - val_accuracy: 0.4462\n",
      "Epoch 22/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1346 - accuracy: 0.6074 - val_loss: 1.5280 - val_accuracy: 0.5136\n",
      "Epoch 23/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9820 - accuracy: 0.6678 - val_loss: 1.5392 - val_accuracy: 0.5040\n",
      "Epoch 24/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9701 - accuracy: 0.6679 - val_loss: 1.5505 - val_accuracy: 0.5170\n",
      "Epoch 25/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3604 - accuracy: 0.6753 - val_loss: 1.5468 - val_accuracy: 0.4992\n",
      "Epoch 26/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0177 - accuracy: 0.6510 - val_loss: 1.5474 - val_accuracy: 0.5020\n",
      "Epoch 27/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9425 - accuracy: 0.6798 - val_loss: 1.5545 - val_accuracy: 0.5076\n",
      "Epoch 28/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9005 - accuracy: 0.6902 - val_loss: 1.5659 - val_accuracy: 0.5138\n",
      "157/157 [==============================] - 0s 1ms/step - loss: 1.4633 - accuracy: 0.4792\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.4633383750915527, 0.47920000553131104]"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.Nadam(lr=7e-4)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_selu_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_selu_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
    "\n",
    "X_means = X_train.mean(axis=0)\n",
    "X_stds = X_train.std(axis=0)\n",
    "X_train_scaled = (X_train - X_means) / X_stds\n",
    "X_valid_scaled = (X_valid - X_means) / X_stds\n",
    "X_test_scaled = (X_test - X_means) / X_stds\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=100,\n",
    "          validation_data=(X_valid_scaled, y_valid),\n",
    "          callbacks=callbacks)\n",
    "\n",
    "model = keras.models.load_model(\"my_cifar10_selu_model.h5\")\n",
    "model.evaluate(X_valid_scaled, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "157/157 [==============================] - 0s 1ms/step - loss: 1.4633 - accuracy: 0.4792\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.4633383750915527, 0.47920000553131104]"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.load_model(\"my_cifar10_selu_model.h5\")\n",
    "model.evaluate(X_valid_scaled, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get 47.9% accuracy, which is not much better than the original model (47.6%), and not as good as the model using batch normalization (54.0%). However, convergence was almost as fast as with the BN model, plus each epoch took only 7 seconds. So it's by far the fastest model to train so far."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### e.\n",
    "*Exercise: Try regularizing the model with alpha dropout. Then, without retraining your model, see if you can achieve better accuracy using MC Dropout.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1407/1407 [==============================] - 9s 5ms/step - loss: 2.0583 - accuracy: 0.2742 - val_loss: 1.7429 - val_accuracy: 0.3858\n",
      "Epoch 2/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.6852 - accuracy: 0.4008 - val_loss: 1.7055 - val_accuracy: 0.3792\n",
      "Epoch 3/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5963 - accuracy: 0.4413 - val_loss: 1.7401 - val_accuracy: 0.4072\n",
      "Epoch 4/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5231 - accuracy: 0.4634 - val_loss: 1.5728 - val_accuracy: 0.4584\n",
      "Epoch 5/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4619 - accuracy: 0.4887 - val_loss: 1.5448 - val_accuracy: 0.4702\n",
      "Epoch 6/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.4074 - accuracy: 0.5061 - val_loss: 1.5678 - val_accuracy: 0.4664\n",
      "Epoch 7/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.3718 - accuracy: 0.5222 - val_loss: 1.5764 - val_accuracy: 0.4824\n",
      "Epoch 8/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3220 - accuracy: 0.5387 - val_loss: 1.4805 - val_accuracy: 0.4890\n",
      "Epoch 9/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.2908 - accuracy: 0.5487 - val_loss: 1.5521 - val_accuracy: 0.4638\n",
      "Epoch 10/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2537 - accuracy: 0.5607 - val_loss: 1.5281 - val_accuracy: 0.4924\n",
      "Epoch 11/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.2215 - accuracy: 0.5782 - val_loss: 1.5147 - val_accuracy: 0.5046\n",
      "Epoch 12/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1910 - accuracy: 0.5831 - val_loss: 1.5248 - val_accuracy: 0.5002\n",
      "Epoch 13/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.1659 - accuracy: 0.5982 - val_loss: 1.5620 - val_accuracy: 0.5066\n",
      "Epoch 14/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.1282 - accuracy: 0.6120 - val_loss: 1.5440 - val_accuracy: 0.5180\n",
      "Epoch 15/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.1127 - accuracy: 0.6133 - val_loss: 1.5782 - val_accuracy: 0.5146\n",
      "Epoch 16/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0917 - accuracy: 0.6266 - val_loss: 1.6182 - val_accuracy: 0.5182\n",
      "Epoch 17/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.0620 - accuracy: 0.6331 - val_loss: 1.6285 - val_accuracy: 0.5126\n",
      "Epoch 18/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0433 - accuracy: 0.6413 - val_loss: 1.6299 - val_accuracy: 0.5158\n",
      "Epoch 19/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0087 - accuracy: 0.6549 - val_loss: 1.7172 - val_accuracy: 0.5062\n",
      "Epoch 20/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 0.9950 - accuracy: 0.6571 - val_loss: 1.6524 - val_accuracy: 0.5098\n",
      "Epoch 21/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9848 - accuracy: 0.6652 - val_loss: 1.7686 - val_accuracy: 0.5038\n",
      "Epoch 22/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9597 - accuracy: 0.6744 - val_loss: 1.6177 - val_accuracy: 0.5084\n",
      "Epoch 23/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9399 - accuracy: 0.6790 - val_loss: 1.7095 - val_accuracy: 0.5082\n",
      "Epoch 24/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9148 - accuracy: 0.6884 - val_loss: 1.7160 - val_accuracy: 0.5150\n",
      "Epoch 25/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 0.9023 - accuracy: 0.6949 - val_loss: 1.7017 - val_accuracy: 0.5152\n",
      "Epoch 26/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.8732 - accuracy: 0.7031 - val_loss: 1.7274 - val_accuracy: 0.5088\n",
      "Epoch 27/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 0.8542 - accuracy: 0.7091 - val_loss: 1.7648 - val_accuracy: 0.5166\n",
      "Epoch 28/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.8499 - accuracy: 0.7118 - val_loss: 1.7973 - val_accuracy: 0.5000\n",
      "157/157 [==============================] - 0s 1ms/step - loss: 1.4805 - accuracy: 0.4890\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.4804893732070923, 0.48899999260902405]"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "\n",
    "model.add(keras.layers.AlphaDropout(rate=0.1))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.Nadam(lr=5e-4)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_alpha_dropout_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_alpha_dropout_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
    "\n",
    "X_means = X_train.mean(axis=0)\n",
    "X_stds = X_train.std(axis=0)\n",
    "X_train_scaled = (X_train - X_means) / X_stds\n",
    "X_valid_scaled = (X_valid - X_means) / X_stds\n",
    "X_test_scaled = (X_test - X_means) / X_stds\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=100,\n",
    "          validation_data=(X_valid_scaled, y_valid),\n",
    "          callbacks=callbacks)\n",
    "\n",
    "model = keras.models.load_model(\"my_cifar10_alpha_dropout_model.h5\")\n",
    "model.evaluate(X_valid_scaled, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model reaches 48.9% accuracy on the validation set. That's very slightly better than without dropout (47.6%). With an extensive hyperparameter search, it might be possible to do better (I tried dropout rates of 5%, 10%, 20% and 40%, and learning rates 1e-4, 3e-4, 5e-4, and 1e-3), but probably not much better in this case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use MC Dropout now. We will need the `MCAlphaDropout` class we used earlier, so let's just copy it here for convenience:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MCAlphaDropout(keras.layers.AlphaDropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a new model, identical to the one we just trained (with the same weights), but with `MCAlphaDropout` dropout layers instead of `AlphaDropout` layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model = keras.models.Sequential([\n",
    "    MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout) else layer\n",
    "    for layer in model.layers\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then let's add a couple utility functions. The first will run the model many times (10 by default) and it will return the mean predicted class probabilities. The second will use these mean probabilities to predict the most likely class for each instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mc_dropout_predict_probas(mc_model, X, n_samples=10):\n",
    "    Y_probas = [mc_model.predict(X) for sample in range(n_samples)]\n",
    "    return np.mean(Y_probas, axis=0)\n",
    "\n",
    "def mc_dropout_predict_classes(mc_model, X, n_samples=10):\n",
    "    Y_probas = mc_dropout_predict_probas(mc_model, X, n_samples)\n",
    "    return np.argmax(Y_probas, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's make predictions for all the instances in the validation set, and compute the accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4892"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "y_pred = mc_dropout_predict_classes(mc_model, X_valid_scaled)\n",
    "accuracy = np.mean(y_pred == y_valid[:, 0])\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get no accuracy improvement in this case (we're still at 48.9% accuracy).\n",
    "\n",
    "So the best model we got in this exercise is the Batch Normalization model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### f.\n",
    "*Exercise: Retrain your model using 1cycle scheduling and see if it improves training speed and model accuracy.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "\n",
    "model.add(keras.layers.AlphaDropout(rate=0.1))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.SGD(lr=1e-3)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "352/352 [==============================] - 2s 6ms/step - loss: nan - accuracy: 0.1255\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(9.999999747378752e-06,\n",
       " 9.999868392944336,\n",
       " 2.6167492866516113,\n",
       " 3.9354368618556435)"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAERCAYAAACO6FuTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAng0lEQVR4nO3deXxV1bn/8c+TgYRMQEgCyBTCPAlIHJFRLc5jtdSrV+tVW/VqtXq1s8O1tWr1d1tbtVhaqqhVW1vnqcUJQSSoKCCiSJihYZ4DSZ7fH/sEY5oDCUnOPjn5vl+v/crZe69zzrM44TxZa+29lrk7IiIidUkKOwAREYlfShIiIhKVkoSIiESlJCEiIlEpSYiISFRKEiIiElVK2AE0pby8PC8sLAw7DBFpAht27GH15l0M7JJDSpKFHU5Cmzt37np3z6/rXEIlicLCQkpKSsIOQ0SawCPvLuMnf5/Paz86nvzstLDDSWhmtizaOXU3iYhIVEoSIiISlZKEiIhEpSQhIiJRKUmIiEhUShIiIhKVkoSIiESlJCEiIlEpSYiISFRKEiIiEpWShIiIRKUkISIiUSlJiIhIVEoSIiISlZKEiIhEpSQhIiJRKUmIiEhUMU0SZjbNzNaY2VYzW2xml0YpZ2Z2u5mtMrMtZvaGmQ2OZawiIhL7lsQdQKG75wCnA7eb2cg6yp0LXAKMBnKBWcAjMYtSRESAGCcJd1/g7uXVu5Gtdx1FewEz3P0Ld68EpgGDYhSmiIhExHxMwszuN7OdwCJgDfBiHcX+DPQxs35mlgpcBLwc5fUuN7MSMyspKytrtrhFRFqjmCcJd78SyCboSnoaKK+j2BrgbeBTYBdB99N1UV5vsrsXu3txfn5+8wQtItJKhXJ1k7tXuvsMoBtwRR1FbgYOB7oD6cCtwHQzy4hdlCIiEvYlsCnUPSYxDHjC3Ve6e4W7TwU6oHEJEZGYilmSMLMCM5tkZllmlmxmE4FvAtPrKD4HONfMOplZkpldCKQCn8cqXhERCf6SjxUn6Fp6kCA5LQOudfdnzKwHsBAY5O7LgTuBAuBDIJMgOZzj7ptjGK+ISKsXsyTh7mXA2CjnlgNZNfZ3A1dFNhERCUnYYxIiIhLHlCRERCQqJQkREYlKSUJERKJSkhARkaiUJEREJColCRERiUpJQkREolKSEBGRqJQkREQkKiUJERGJSklCRESiUpIQEZGolCRERCQqJQkREYlKSUJERKJSkhARkaiUJEREJColCRERiUpJQkREolKSEBGRqJQkREQkKiUJERGJKqZJwsymmdkaM9tqZovN7NL9lC0ys+fNbJuZrTezu2IZq4iIxL4lcQdQ6O45wOnA7WY2snYhM2sDvAZMBzoD3YBpsQxURERinCTcfYG7l1fvRrbedRS9GFjt7ve6+w533+3uH8UqThERCcR8TMLM7jezncAiYA3wYh3FjgJKzeylSFfTG2Y2NMrrXW5mJWZWUlZW1oyRi4i0PjFPEu5+JZANjAaeBsrrKNYNmAT8GjgEeAF4JtINVfv1Jrt7sbsX5+fnN1/gIiKtUChXN7l7pbvPIEgGV9RRZBcww91fcvc9wC+BjsDAGIYpItLqhX0JbAp1j0l8RDBeISIiIYpZkjCzAjObZGZZZpZsZhOBbxJcwVTbNOAoMzvezJKBa4H1wCexildERGLbknCCrqWVwCaCLqRr3f0ZM+thZtvNrAeAu38KXAA8GCl7BnB6pOtJRERiJCVWb+TuZcDYKOeWA1m1jj1NMLAtIiIhCXtMQkRE4piShIiIRKUkISIiUSlJiIhIVEoSIiISlZKEiIhEpSQhIiJRKUmIiEhUShIiIhKVkoSIiESlJCEiIlEpSYiISFRKEiIiEpWShIiIRKUkISIiUSlJiIhIVEoSIiISlZKEiIhEpSQhIiJRKUmIiEhUShIiIhKVkoSIiEQV0yRhZtPMbI2ZbTWzxWZ2aT2eM93M3MxSYhGjiIh8KdYtiTuAQnfPAU4HbjezkdEKm9l/AEoOIiIhiWmScPcF7l5evRvZetdV1szaATcDN8YoPBERqSXmYxJmdr+Z7QQWAWuAF6MU/TnwALA2VrGJiMhXxTxJuPuVQDYwGngaKK9dxsyKgVHAfQd6PTO73MxKzKykrKysqcMVEWnVQrm6yd0r3X0G0A24ouY5M0sC7ge+6+4V9Xitye5e7O7F+fn5zROwiEgrFfYlsCn8+5hEDlAMPGFma4E5keMrzWx0LIMTEWntYnblkJkVABOA54FdwPHAN4HzaxXdAhxSY7878B4wElB/kohIDMXy8lIn6Fp6kKAFswy41t2fMbMewEJgkLsvp8ZgtZmlRx6uq0/3k4iINJ2YJQl3LwPGRjm3HMiKcq4UsOaLTEREogl7TEJEROKYkoSIiETV6CRhZqlNEYiIiMSfBiUJM7vGzM6psT8F2GVmn5pZ/yaPTkREQtXQlsQ1RC5DNbMxwHkEl7B+CNzTpJGJiEjoGnp1U1egNPL4NOApd3/SzD4G3m7KwEREJHwNbUlsBarnvjgB+Gfk8V4gvc5niIhIi9XQlsSrwENm9gHQB3gpcnwwsLQpAxMRkfA1tCVxFfAOkAd83d03Ro4fBjzelIGJiEj4GtSScPetwNV1HL+5ySISEZG40dBLYAfVvNTVzE6IrFv9AzNLbvrwREQkTA3tbpoCjAAws27AM0AuQTfU7U0bmoiIhK2hSWIg8H7k8bnAbHc/GbiQYNpvERFJIA1NEsnAnsjj4/hyfeolQKemCkpEROJDQ5PEfOCKyApxxwEvR453BdY3ZWAiIhK+hiaJm4DLgDeAx93948jx0wlWjxMRkQTS0Etg3zKzfCDH3TfVOPU7YGeTRiYiIqFr8Mp07l5pZrvMbAjBkqRLIqvHiYhIgmnofRIpZnY3sAmYB3wMbDKzu7SuhIhI4mloS+IugktdvwPMiBwbDdxBkHBuaLrQREQkbA1NEucDl7j7izWOLTGzMuD3KEmIiCSUhl7d1I7gnojalgDtGx2NiIjElYYmiXkEq9PV9t3IORERSSANTRI3AheZ2WIz+5OZTTWzT4ELqEdXU2QywDVmtjXyGpdGKXeRmc2NlFsZGRhv8JVYIiLSOA1KEu7+FtAPeArIAnIijydSdwujtjuAQnfPIbgB73YzG1lHuQzgWoJ1K44kuLtb4x0iIjF2MPdJrAZ+VPOYmQ0DzqnHcxfU3I1svYG5tco9UGN3lZk9CoxvaKwiItI4De1uajQzu9/MdgKLgDV8OUng/owBFhywlIiINKmYJwl3vxLIJri/4mmgfH/lzexbQDHwyyjnLzezEjMrKSsra+pwRURatZgnCQim9nD3GUA34Ipo5czsTOAXwEnuXucss+4+2d2L3b04Pz+/WeIVEWmt6jUmYWbPHqBITiPev3eU9zwReAg4pcZssyIiEkP1HbjeUI/zS/dXwMwKgAnA88Au4HiCKT7Or6PsBOBR4Cx31xTkIiIhqVeScPdvNcF7OUHX0oME3VzLgGvd/Rkz6wEsBAa5+3LgJwR3d79oZtXPf9vdT2qCOEREpJ5idoOau5cBY6OcW05w30X1vi53FWnt3MOOQAhp4FpE5EAqq4IkkZxkBygpzUlJQkTiUmWkIZFsShJhUpIQkbhUFWlJJOlbKlT65xeRuFTp6m6KB0oSIhKXqiJJIkndTaFSkhCRuFSlgeu4oCQhInGpsir4qZZEuJQkRCQuVe7rbgo5kFZOSUJE4lJVlZNkYGpJhEpJQkTiUqW7xiPigJKEiMSloCWhJBE2JQkRiUuVVWpJxAMlCRGJS5XumpIjDihJiEhcqqpyktSSCJ2ShIjEJQ1cxwcliVqq7/IUkXBVVulGunigJFHDo7OXMeJ/X+OtxWVhhyLS6lVVOcn6hgpdwn8Ev/rHZzwxZ/lXjm3dvZen31/J5p172F5ewS9f+ZSfvbCQW59dyI7yCi57uITlG3aGFLGIgAau40XMli8Nw56KKh5483M6ZqZxXnF33OHCP8xm9hcbqahyDi/swOBD2jF1ZilJBqP65PHDkwdy9v0zueGpeXTLbUt5RRV3nnMotz+/kFWbd3HbGUPolZcZdtVEEl6Va+A6HiR0kpi/egu791axavMuFqzeSlpKEu98voFTD+3C0K7tuOOlRcwp3cQ5h3XjznOGkhJp2157fF/ueGkRn6xNYUd5BTM+W8+WXXtpm5rMhVNm89p1Y2nbJjnk2okktirdJxEXEjpJvLd0IwBm8OrCdXRtnw7A907oR1F+Fkf0yuWl+Wu5fEzRvgQB8O2xvfnPowtJS0ni1YXreG7eao7olUv/ztlMmvwud7z0CbeePnjfnDKVVc7spRuY/sm/aNsmmdTkJIZ2a8dn67axeN12ivIzuWRUL9JTlVhE6qvStXRpPEjoJFFSupGivEyy0lOYs3Qj63IzaJ+Ruq+7aESPDozo0aHO51a3FE4c0pkTh3Ted/xbowr54zulLNuwk6Fd25GRlszj7y1nxcZdtElJYm9lFV7jAqn87DT+Mnclr8xfy40nDmBUn7zmq7BIAtF9EvEhoZPE4nXbGda9Pe3bpvK3D1bxr227GdG9faNmlfzpqYPomZvBPa8u5u3Pyqhy6J2fyX3fHMHxAzvRJiWJHXsq+GD5ZgZ0zqZTTjovfbyGm59dwAVTZnP314fRrUNbjijM1X8Akf2ojMwCK+GKaZIws2nAcUAmsBa4y91/H6XsdcBNQFvgr8AV7l5e3/eqqKxi9eZdnD7sEArzMnnk3WVsL6vgvOLuja0DF4/qxaQjemAG67fvIT8rjTYpX3ZX5aSnMrZf/r79k4Z2YUy/fE69bwY3PDUPgEFdcvjdhSPpnpvRqHhEElWla4K/eBDrS2DvAArdPQc4HbjdzEbWLmRmE4HvEySUQqAIuLUhb7Rmy24qqpweuRkM795u3/GzDut68NHXkJ6aTFpKMl3bt/1KgogmMy2FKRcV879nDuGX5w5jxaadjLn7dc57cBYvz19LpW7iE/kKDVzHh5i2JNx9Qc3dyNYbmFur6EXAlOryZva/wKMEiaNeVmwM7nPoltuWorwsAHp2zKAgO/1gw2+0ovwsivKDWI4ozOVvH6ziqbkr+M60ubTPSOXIXrkcVdSRY/vksaRsO30KsuhTkB1avCJh0rQc8SHmYxJmdj9wMUE30gfAi3UUGww8U2N/HtDJzDq6+4b6vM/ySJLokZtBUpIx/fqxdMxKa0zoTapHxwy+e3xfrhrfm9cWrmP6on/x7tINvLJg3VfKnTK0Cz84eQDdOqhbSlqXSq0nERdiniTc/Uozuxo4GhgH1DXOkAVsqbFf/Tgb+EqSMLPLgcsBevTowfbyCm5/fiF/nrOC5CSjS7u2APv+go83KclJnDS0CycN7QIELaA3F5fRIzeDkmWbmPzWEl5ZsJY2KUnkZ6dRlJfJ2Yd1wwlaIx+v2kLJso1s311BZloKh3Zrx7Bu7enWoa2WfZQWrUotibgQytVN7l4JzDCzC4ArgF/XKrIdyKmxX/14Wx2vNRmYDFBcXOx3v7yIP89ZAbTMRUu652ZwwVE9ARjTL59vHN6dae8uo3xvFf/atpuS0k1c/fgHX3lOm+QkstJT2F5ewZ6KKgC6dWjLqN55nDqsC6P75v/b+4jEu8oqTcsRD8K+BDaFYEyitgXAMODJyP4wYF19upqWbtjJsG7tGNa9PdnpYVev8bq2b8tNJw7Yt79rTyXzVm4mJcn4eNUWeuVlMqpPHqnJSeypqOLTtdv4cMUm3vi0jFcWruWJkhUM7dqOY/vmcWyfPEb27KCb+qRFqKqCpISfXS7+xexb1MwKgAnA88Au4Hjgm8D5dRR/GJhqZo8Ca4AfA1Pr8z6bduyhY1YbbjtjSFOEHXfatknmqKKOABQX5n7lXJuU4E7vod3aceHRhZRXVPLIrGW8umAdD731BQ+8sYS0lCTOGH4Il40uoig/q8W1tKT1qHQnVVkidLH8U9sJupYeJLj0dhlwrbs/Y2Y9gIXAIHdf7u4vm9ldwOt8eZ/EzfV5k00799C3ID7HH2ItLSWZS0cXcenoInaUV/De0o289sk6/jp3JU+WrCQlyThhUCf+Z2L/uB2zkdZLA9fxIWZJwt3LgLFRzi0nGKyueexe4N6Gvs+mHXvokNnmoGJMZJlpKYwfUMD4AQVcf0I/nv9oDUvX7+Cv76/kH5+sY8KAAo7s1ZFh3dsz+JAcdUlJ6FwD13Gh5Xfa1+AOO/ZU0iEjNexQ4lrHrDQuOqYQgCvH9+bBN77gpflr9l1+m5JkjOjRnpOGdKFfp2yOKsr9ygSIIrGg9STiQ0IliYrIXctqSdRfQXY6Pz1tED89bRDrtu7mwxWb+XDFZl6Zv5bbnl8IBHNT3X3uMIZ3a6/5piRmKqvQ71scSKgkUVkVXP6Zm6EkcTA65aQzcXBnJg7uzI0T+7N++x5mL93Azc8s4Oz7Z9I9ty1Xj+/LucXddA+GNLsqXQIbFxIqSagl0XTMjPzsNE499BBG9c7jH5+s47H3lnPjXz9i2uxlXHBUT8b2y6cgO00JQ5qFpuWIDwmVJKonyeuglkST6pDZhnOLu3POYd14smQFv5+xlBv/8hEAmW2SKcrPYlz/fE4bdgj9OmmuKWkaWk8iPiRUkviyJaGB6+aQlGRMOqIH3zi8O+8v38TC1VtZUraDT9Zs5bevf8590z/n5KGduXJcH4Z0bXfgFxTZj2Cq8LCjkIRKEpWVThJqSTQ3M2Nkz1xG9vzyZr6ybeU8Nns5D765hBc/Xsvw7u254KienHpoF11OKwdF03LEh4S6rrGiqorstBRSdblmzOVnp/Hd4/vy7g+P4+bTBrF1915ueGoeR93xT+56eRFrt+wOO0RpYdTdFB8SqiWxa28lRbrbOlTt2qbyrVG9uPiYQmYt2cDUmaU88OYSJr/1Bacc2oXLRhepK0rqRfdJxIfEShJ7Kjmmd8ewwxCCLqlj+uRxTJ88lm/YyR9nLuXJOSt45sPVjO+fz4QBBZx3eHfSUtQVJXXTfRLxIaH6ZRw4ukhJIt706JjBzacNZuYPjuM7Y3tTumEnP3lmAcfcMZ0H3liipVulTsF6EmFHIQnVkjCguLBD2GFIFO3apvL9kwZw04n9mbVkAw+9/QV3vryIl+avYdLhPThuYAGdcsJbXlbiiwau40NCJYlD2rclo01CVSkhVXdFHd27I898uJq7X/mUH/7tY9o8m8S5xd24dHQRvfIyww5TQqaB6/iQUN+oubrTukUxM84c0ZUzhh/CkrLt/OGdUp4qWcmjs5dz7shuXDm+j5JFK1algeu4kFBJQlomM6NPQTY/P2so1x7fl9+/vZQpM5by1NyVDO3ajivH9ebEIZ01/Ucro2k54oOGhSSuFGSn88OTBzLjpvH8+JSB7NxTwRWPvs+Z989k5pL1YYcnMVSlq5vigpKExKUu7dpy6egiXr1uLHd9/VDKtu7m/Idmc/Ef32PNll1hhycxoPsk4oOShMS15CTjvOLuTL9hHD86eSBzlm7khHvf4mcvLGTr7r1hhyfNqFID13FBSUJahPTUZC4bU8QL14zmuIEFTJmxlOPueZNn563GXfdZJJqqyL0zakmET0lCWpTCvEx+NWkEz1x1LJ1z0rnm8Q+YNPldpi9at++LRVq+ykji18104dNHIC3S0G7t+PtVo7j19MGUbtjBJVNLOO7eN3n8veXsrawKOzxppOq78NXdFD4lCWmxkpOMi44pZMZNE/jVpOHkpKfwg6c/5oR7g24otSxaripXd1O8UJKQFi81OYkzhnfl71eNYspFxaSnJnPN4x9w6n0zeHn+Ws0N1QLta0koSYQuZknCzNLMbIqZLTOzbWb2gZmdFKWsmdntZrbKzLaY2RtmNjhWsUrLZGYcN7ATL14zmv/7xnB27KngO9PmMuGeN5j6zlK2l1eEHaLUU1Wkx1DdTeGLZUsiBVgBjAXaAT8BnjSzwjrKngtcAowGcoFZwCOxCVNauqSkYLqP6deP4/7/OIyOmW245bmFHP3zf3LbcwspXb8j7BDlAPYNXCtHhC5m03K4+w7glhqHnjezpcBIoLRW8V7ADHf/AsDMpgHXxSBMSSDJScbJQ7tw8tAufLhiM398ZykPzyrlD+8sZVz/fG74Wn8tgBSnqrubNC1H+EIbkzCzTkA/YEEdp/8M9DGzfmaWClwEvBzL+CSxDO/enl9NGsE735/Adcf346OVWzj1vhn8x+/fZfG6bWGHJ7VUD1yruyl8oSSJyBf/o8Cf3H1RHUXWAG8DnwK7CLqf6mxJmNnlZlZiZiVlZWXNFbIkiE456Xz3+L68fv04rj+hH5+s2cYpv36bH//9Y9Zt1Trc8UJXN8WPmCcJM0siGF/YA/x3lGI3A4cD3YF04FZgupll1C7o7pPdvdjdi/Pz85spakk07TJSufq4vrxy7RjOLe7On99bwZi7XudnLyxkw/bysMNr9XSfRPyIaZKwYK7nKUAn4Bx3jzb5zjDgCXdf6e4V7j4V6AAMik2k0lrkZ6fx87OGMv36cZxyaBemzFjKmLte555XP2XLLs0NFZbqq5vUkghfrFsSDwADgdPcfX9Tec4BzjWzTmaWZGYXAqnA57EIUlqfHh0zuPe84bx63RjGDSjgvumfM+oX07ntuYWs2Lgz7PBanS+n5VCSCFvMrm4ys57At4FyYG2NBWS+TTD+sBAY5O7LgTuBAuBDIJMgOZzj7ptjFa+0Tn0Ksvnt+Ydx1bitPPT2Fzw8q5SpM5dy3MBOXDmuNyN6aA31WFB3U/yI5SWwy4D9feJZNcruBq6KbCIxN+iQHP7fN4Zz04kDeHhWKU/MWcHZD8zkrBFduXpCXy2r2sw0cB0/NC2HyH50bpfOjScO4I3/Gcclo3rx8vy1TPy/t/j1Pz+jvKIy7PAS1pf3SYQciChJiNRHdnoqPzl1EG/cMI6vDerEva8t5uRfvc2z81Zr1tlmoLmb4oeShEgDFOSk85vzD2Pqtw7HHa55/ANG/WI6v/7nZ5Rt06WzTaVKA9dxI2ZjEiKJZFz/Asb0zefNxWVMnVnKva8t5jfTP+fUQ7tw9XEas2gsDVzHDyUJkYOUlGSMH1DA+AEFLCnbziOzlvFkyQqenbeaSUd055oJfSnISQ87zBZJA9fxQ0lCpAn0zs/iltMHc9X4Ptw3/TMem72cae8upyg/k7OGd+Xskd3o2r5t2GG2GNXDPBqTCJ/GJESaUH52GredMYR/fG8s/zOxPwXZadzz2mKOvXM6l0ydw0crN4cdYovwZXdTyIGIWhIizaEwL5OrxvfhqvF9WLFxJ0/NXcnDs0o5/TfvMGFAAZePKeLIXrmY/lKuk7qb4oeShEgz656bwfdO6Mdlo3sx9Z1S/jizlEmT36V/p2zOPqwrXx/ZjY5ZaWGHGVd0dVP8UGNOJEay04OZZ2d+fwK/OHsomWnJ3PHSIkbf9Tq3PreAL8q2hx1i3NDVTfFDLQmRGEtPTWbSET2YdEQPPlu3jd++/jnT3l3GH98p5ZShXbhhYv9Wfwmtupvih5KESIj6dsrm/yaN4EenDOKRWaX8fsZSXl6wlpE9O3Dm8K6cNqwL2empYYcZc9VXN6m7KXxKEiJxID87je99rT8XHl3In2aW8sqCtfzwbx9z63MLKC7sQN+CbE4a0pnDC3NbRReMpuWIH0oSInEkPzuNGyb25/qv9ePDFZt55sPVzF22iSfmrGDqzFK6tm/LqYd24ciiXEb2zKVd28RsZWjgOn4oSYjEITNjRI8O+9av2LmnglcXrOPpD1YxZcZSfvfWF5hB/07ZHF6YS3FhB8b2y6d9RpuQI28amgU2fihJiLQAGW1SOHNEV84c0ZVdeyr5cMVm5pRuZE7pRp5+fyWPvLuMjDbJnH1YV84Y3pWRPTq06G6p6paEupvCpyQh0sK0bZPM0b07cnTvjgBUVFbx8aotPDJrGX+Zu5Jp7y6nb0EWl40u4owRh5CWkhxyxA23pGwHoO6meGAeydiJoLi42EtKSsIOQyQ028sreHn+WqbMWMona7aSn53GxccUMnFwZ3rnZ7aIO7zfXFzGRX94j2N6d+SR/zpSiSIGzGyuuxfXeU5JQiTxuDszPl/P5Le+4O3P1gPQMbMNw7q3Z+LgThxdlEf33LZxmTR+9LeP+fsHq3j/pye0yFZQS7S/JKHuJpEEZGaM7pvP6L75LNuwg1lLNjCndBNzSjcyfdG/gOBKqqOKOvKN4u4c07tjXIxhuDtvfVbG0b3zlCDihJKESILr2TGTnh0zmXRED9ydRWu3UbJsE3NLN/Lm4jKem7eaXnmZnD2iKyN6dGBot3ahXVo764sNrNi4i8tHF4Xy/vLvlCREWhEzY2CXHAZ2yeHCo3qye28lL81fw7R3l3PPa4v3leuVl8mwbu0Y2bMDRxV1pE9BVrN2Tc1ftYXH3lvO4+8tp3NOOhOHdG6295KG0ZiEiACwZedePlq1mY9WbmHeis3MW7mZdVuDdbs7ZrYhIy2ZrLRUurRLp2fHDHrnZwVbQSb5WWkNSiJl28qZMmMpyzfuYNWmXcxbuYXkJOPCo3py/df6tcqpSMLUagauc3sO9BN++IewwxBJCO5OeUUVW3dXsG33XtyDm9z2VFaxe28lVTW+OpIsuKchKclom5pMkgWtFgPMiPwMksjOPZVsL6/AgLTUJFKSksjNbENeVhtSdfdcKJ78zjGtI0mY2Tbg00a+TDtgSyPL1XXuQMdqn6/er3k8D1hfj9j2J1b1299+tMexql9D61bX8TDq11yfXV3HG1q/lvS7WdexRK5ffb5berp7fp3v6O4JswElTfAakxtbrq5zBzpW+3z1fq0yLaZ++9vfz+OY1K+hdYuX+jXXZ9cU9WtJv5utrX71+W7Z36a23b97rgnK1XXuQMdqn38uyvHGilX99re/v3o3Vn1er6F1q+t4GPVrrs+uruOJVL+G/r4mWv0a9d2SaN1NJR6lXy0RqH4tWyLXL5HrBolfv/1JtJbE5LADaGaqX8uWyPVL5LpB4tcvqoRqSYiISNNKtJaEiIg0ISUJERGJqtUlCTMrNLMyM3sjstV9bXALZmbfNLOysONoambWycxmmtmbZjbdzLqEHVNTMrOjzWxWpH6Pm1lC3XZsZu3M7D0z225mQ8KOpymY2c/M7G0z+4uZZYQdT3NodUki4k13HxfZEurL1MySgK8DK8KOpRmsB45197HAw8B/hRxPU1sGTIjU7wvgjJDjaWo7gVOAv4QdSFOIJLre7j4a+AdwScghNYvWmiRGRbL/zy0eJ9RvnPMJ/hNWhR1IU3P3Snevrlc2sCDMeJqau692912R3QoS7DN0970J9kfZaOClyOOXgGNDjKXZxHWSMLP/NrMSMys3s6m1zuWa2d/MbIeZLTOz8+v5smuAPsAYoAA4u2mjrp/mqJuZJQPnAU80Q8gN0kyfHWY23MxmA/8NvN/EYddbc9Uv8vxewEnA800YcoM0Z/3iTSPq2oEvp7XYAuTGKOSYivepwlcDtwMTgba1zv0W2AN0AoYDL5jZPHdfYGadqbtJ+3V3XwuUA5jZ08BRwF+bJ/z9avK6RV7rSXevioMGUrN8du7+IXCkmZ0H/AD4TjPFfyDNUj8zywH+BFzo7nuaLfoDa67/e/HooOoKbCKY/4jIz40xiTbWGjsfSSw2gg9wao39TIIPrl+NY48Av6jHa+XUeHwH8J8JVLc7gVeBlwn+svl1gn12aTUeTwTuTbD6pQAvEIxLhFqv5qhfjfJTgSFh162xdQWGAo9FHl8OXB12HZpji+vupv3oB1S6++Iax+YBg+vx3LFmNtfM3ga6Ao81R4CNcNB1c/eb3P1r7n4i8Jm7X9NcQTZCYz67w8zsLTN7HbgWuLsZ4musxtTvm8CRwE8jV959ozkCbKTG1A8zexH4GvCQmV3c9OE1qf3W1d0/BpZFvksmAgm5TkG8dzdFk8W/T427hWAwc7/c/TmaflK5pnTQdavJ43eemcZ8drMIxpLiWWPq9wjBX6rxrFG/n+5+cpNH1HwOWFd3/0FMIwpBS21JbAdyah3LAbaFEEtTS+S6gerX0iV6/WpqTXWNqqUmicVAipn1rXFsGIlxSWQi1w1Uv5Yu0etXU2uqa1RxnSTMLMXM0oFkINnM0s0sxd13AE8Dt5lZppmNIrjxKN6b6vskct1A9UP1azFaU10PStgj5we42uAWwGttt0TO5QJ/B3YAy4Hzw45XdVP9VL+Wt7Wmuh7MpqnCRUQkqrjubhIRkXApSYiISFRKEiIiEpWShIiIRKUkISIiUSlJiIhIVEoSIiISlZKESBMys1vMbH7YcYg0Fd1MJy1OZPWwPHc/NexYajOzLIJ1LzaEHUs0ZubAue6eEGtNS/NSS0KkHsysTX3Kufv2MBKEmSVFlq8VaVJKEpJwzGyQmb1gZtvM7F9m9nhkWc3q84eb2atmtt7MtprZDDM7utZruJldZWZPm9kO4OfVXUlmNsnMlkRe/+9mllfjeV/pbjKzqWb2vJl918xWmdkmM/ujmWXUKJNpZg+b2XYzW2dmP4g8Z+p+6nhxpPzJkffbAww8UN3MrDTy8KlIHUtrnDstsiDXbjNbamY/q29ylMSlJCEJxcy6AG8B84EjgOMJFo951syqf9+zCWbyHB0p8yHwYs0v+4ibgRcJlqn8beRYIfAN4CyCFdZGAD87QFijgSGRWKqf+90a5+8BxkaOTyCYjnp0PaqbDvwY+DYwCFhWj7odHvl5GdClet/MJgKPAr8hWHntEoJ1039ejzgkkYU9w6A2bQ3dCNZIfj7KuduAf9Y61oFgZs8jojzHgDXABTWOOXBfrXK3ALuBdjWO/Qj4vFaZ+bViXQGk1Dj2EPCPyOMsglbApBrnM4FN1FhvuY6YL47EOPIA/1bR6vb1WuXeAn5S69iZBAvvWNifubbwNrUkJNGMBMZEumK2m9l2gi9pgN4AZlZgZr8zs8VmtoVgpbECoEet1yqp4/WXuXvNJS1XR567PwvdvSLKc3oDqcB71Sc9WMegPldIVRC0FPZpQN1qGwn8qNa/22MECavz/p8qiaylrnEtEk0S8AJwQx3n1kV+/gnoBFwHlALlwD+B2v3vO+p4jb219p0Dd9vu7zlW41hDlbt7Za1j9a1bbUnArcBTdZwrO4jYJEEoSUiieR84j+Av/tpfztWOBa5x9xcAzKwTQf98GD4nSCJHAEsj8WQQjGEsOYjXq0/d9hKswlbT+8AAd//8IN5TEpiShLRUOWY2vNaxzQQDzJcBT5jZnQR/BRcRJI7r3X0bwdrFF5jZbILulLsIxgVizt23m9kfgDvNbD3B+MGPCf6yP5jWRX3qVgocZ2ZvErRGNhGM5TxvZsuAJwm6soYQjOPceBBxSILQmIS0VKOBD2ptv3T31cAooAp4mWDR+t8SdLuUR557CcGA8Vzgz8AfCL44w3ID8DbwLPA68BHBeMjug3it+tTtemA8wVjNBwDu/gpwSuT4e5Ht+wRLdkorpjuuReKMmaURXM56t7vfE3Y80rqpu0kkZGY2AhhI8Nd7NnBT5OcTYcYlAkoSIvHie0B/vrysdYy7rww1IhHU3SQiIvuhgWsREYlKSUJERKJSkhARkaiUJEREJColCRERiUpJQkREovr/bDaS6ZZQmN4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "rates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1, batch_size=batch_size)\n",
    "plot_lr_vs_loss(rates, losses)\n",
    "plt.axis([min(rates), max(rates), min(losses), (losses[0] + min(losses)) / 1.4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "\n",
    "model.add(keras.layers.AlphaDropout(rate=0.1))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.SGD(lr=1e-2)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "352/352 [==============================] - 3s 6ms/step - loss: 2.2298 - accuracy: 0.2349 - val_loss: 1.7841 - val_accuracy: 0.3834\n",
      "Epoch 2/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.7928 - accuracy: 0.3689 - val_loss: 1.6806 - val_accuracy: 0.4086\n",
      "Epoch 3/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.6475 - accuracy: 0.4190 - val_loss: 1.6378 - val_accuracy: 0.4350\n",
      "Epoch 4/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.5428 - accuracy: 0.4543 - val_loss: 1.6266 - val_accuracy: 0.4390\n",
      "Epoch 5/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.4865 - accuracy: 0.4769 - val_loss: 1.6158 - val_accuracy: 0.4384\n",
      "Epoch 6/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.4339 - accuracy: 0.4866 - val_loss: 1.5850 - val_accuracy: 0.4412\n",
      "Epoch 7/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.4042 - accuracy: 0.5056 - val_loss: 1.6146 - val_accuracy: 0.4384\n",
      "Epoch 8/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.3437 - accuracy: 0.5229 - val_loss: 1.5299 - val_accuracy: 0.4846\n",
      "Epoch 9/15\n",
      "352/352 [==============================] - 2s 5ms/step - loss: 1.2721 - accuracy: 0.5459 - val_loss: 1.5145 - val_accuracy: 0.4874\n",
      "Epoch 10/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.1942 - accuracy: 0.5698 - val_loss: 1.4958 - val_accuracy: 0.5040\n",
      "Epoch 11/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.1211 - accuracy: 0.6033 - val_loss: 1.5406 - val_accuracy: 0.4984\n",
      "Epoch 12/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.0673 - accuracy: 0.6161 - val_loss: 1.5284 - val_accuracy: 0.5144\n",
      "Epoch 13/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 0.9927 - accuracy: 0.6435 - val_loss: 1.5449 - val_accuracy: 0.5140\n",
      "Epoch 14/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 0.9205 - accuracy: 0.6703 - val_loss: 1.5652 - val_accuracy: 0.5224\n",
      "Epoch 15/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 0.8936 - accuracy: 0.6801 - val_loss: 1.5912 - val_accuracy: 0.5198\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 15\n",
    "onecycle = OneCycleScheduler(math.ceil(len(X_train_scaled) / batch_size) * n_epochs, max_rate=0.05)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs, batch_size=batch_size,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[onecycle])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One cycle allowed us to train the model in just 15 epochs, each taking only 2 seconds (thanks to the larger batch size). This is several times faster than the fastest model we trained so far. Moreover, we improved the model's performance (from 47.6% to 52.0%). The batch normalized model reaches a slightly better performance (54%), but it's much slower to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "nav_menu": {
   "height": "360px",
   "width": "416px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}