14_L1_and_L2_Regularization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d8ae27c8",
   "metadata": {},
   "source": [
    "# Chapter 14: L1 and L2 Regularization\n",
    "\n",
    "Regularization methods are those which reduce generalization error. \n",
    "\n",
    "L1 and L2 regularization are used to calculate a number (called a penalty ) added to the loss value to penalize the model for large weights and biases.\n",
    "\n",
    "Large weights might indicate that a neuron is attempting to memorize a data element.\n",
    "\n",
    "It is believed that it would be better to have many neurons contributing to a model’s output, rather than a select few.\n",
    "\n",
    "\n",
    "## 14.1. Forward Pass\n",
    "\n",
    "L1 regularization’s penalty is the sum of all the absolute values for the weights and biases.\n",
    "\n",
    "L2 regularization’s penalty is the sum of the squared weights and biases. This non-linear approach penalizes larger weights and biases more than smaller ones. \n",
    "\n",
    "L2 regularization is commonly used as it does not affect small parameter values substantially and does not allow the model to grow weights too large by heavily penalizing relatively big values.\n",
    "\n",
    "L1 regularization, because of its linear nature, penalizes small weights more than L2 regularization, causing the model to start being invariant to small inputs and variant only to the bigger ones.\n",
    "\n",
    "That’s why L1 regularization is rarely used alone and usually combined with L2 regularization if it’s even used at all.\n",
    "\n",
    "Regularization functions of this type drive the sum of weights and the sum of parameters towards 0, which can also help in cases of exploding gradients (model instability, which might cause weights to become very large values).\n",
    "\n",
    "We use lambda in this equation — where a higher value means a more significant penalty.\n",
    "\n",
    "L1 weight regularization:\n",
    "<center>\n",
    "$\n",
    "L_{1w} = \\lambda \\sum_m | w_m|\n",
    "$\n",
    "</center>\n",
    "\n",
    "L1 bias regularization:\n",
    "<center>\n",
    "$\n",
    "L_{1b} = \\lambda \\sum_n | b_n |\n",
    "$\n",
    "</center>\n",
    "\n",
    "L2 weight regularization:\n",
    "<center>\n",
    "$\n",
    "L_{2w} = \\lambda \\sum_m w_m^2\n",
    "$\n",
    "</center>\n",
    "\n",
    "L2 bias regularization:\n",
    "<center>\n",
    "$\n",
    "L_{2b} = \\lambda \\sum_n b_n^2\n",
    "$\n",
    "</center>\n",
    "\n",
    "Overall loss:\n",
    "<center>\n",
    "$\n",
    "Loss = DataLoss + L_{1w} + L_{1b} + L_{2w} + L_{2b}\n",
    "$\n",
    "</center>\n",
    "\n",
    "\n",
    "## 14.2. Backward Pass\n",
    "\n",
    "The derivative of L2 regularization is relatively simple:\n",
    "<center>\n",
    "$\n",
    "\\begin{align}\n",
    "L_{2w} = \\lambda \\sum_m w_m^2  \\quad \\to \\quad \\frac{∂ L_{2w} }{ ∂ w_m } & = \\frac{∂ }{ ∂ w_m } \\Big[ \\lambda \\sum_m w_m^2 \\Big]  = λ \\frac{∂ }{ ∂ w_m }  w_m^2 =  λ \\cdot 2 w_m^{2-1} = 2 λ w_m \\\n",
    "\\end{align}\n",
    "$\n",
    "</center>\n",
    "\n",
    "L1 regularization’s derivative, on the other hand, requires more explanation.\n",
    "<center>\n",
    "$\n",
    "abs(x) =\n",
    "\\begin{cases}\n",
    "x & x > 0 \\\\\n",
    "-x & x < 0\n",
    "\\end{cases}\n",
    "\\quad \\to \\quad \n",
    "abs'(x) =\n",
    "\\begin{cases}\n",
    "1 & x > 0 \\\\\n",
    "-1 & x < 0\n",
    "\\end{cases}\n",
    "$\n",
    "</center>\n",
    "\n",
    "And the complete partial derivative of L1 regularization with respect to given weight:\n",
    "<center>\n",
    "$\n",
    "\\begin{align}\n",
    "L_{1w} = λ \\sum_m | w_m |  \\quad \\to \\quad \\frac{∂ L_{1w} }{ ∂ w_m } = \\frac{∂ }{ ∂ w_m } λ \\sum_m | w_m | = λ \\frac{∂ }{ ∂ w_m }  | w_m | \n",
    "=  λ  \\begin{cases}\n",
    "1 & w_m > 0 \\\\\n",
    "-1 & w_m < 0\n",
    "\\end{cases} \\\n",
    "\\end{align}\n",
    "$\n",
    "</center>\n",
    "\n",
    "The partial derivative, in this case, equals 1 or -1 depending on the $w_m$ (weight) value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3467ba48",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0, acc: 0.381, loss: 1.099 (data_loss: 1.099, reg_loss: 0.000), lr: 0.02\n",
      "epoch: 100, acc: 0.746, loss: 0.721 (data_loss: 0.663, reg_loss: 0.059), lr: 0.019999010049002574\n",
      "epoch: 200, acc: 0.829, loss: 0.556 (data_loss: 0.466, reg_loss: 0.090), lr: 0.019998010197985302\n",
      "epoch: 300, acc: 0.869, loss: 0.484 (data_loss: 0.385, reg_loss: 0.098), lr: 0.019997010446938183\n",
      "epoch: 400, acc: 0.885, loss: 0.445 (data_loss: 0.345, reg_loss: 0.100), lr: 0.01999601079584623\n",
      "epoch: 500, acc: 0.894, loss: 0.415 (data_loss: 0.317, reg_loss: 0.098), lr: 0.01999501124469445\n",
      "epoch: 600, acc: 0.892, loss: 0.402 (data_loss: 0.307, reg_loss: 0.095), lr: 0.01999401179346786\n",
      "epoch: 700, acc: 0.897, loss: 0.387 (data_loss: 0.297, reg_loss: 0.091), lr: 0.01999301244215147\n",
      "epoch: 800, acc: 0.879, loss: 0.429 (data_loss: 0.332, reg_loss: 0.097), lr: 0.0199920131907303\n",
      "epoch: 900, acc: 0.904, loss: 0.370 (data_loss: 0.276, reg_loss: 0.094), lr: 0.019991014039189386\n",
      "epoch: 1000, acc: 0.905, loss: 0.362 (data_loss: 0.272, reg_loss: 0.091), lr: 0.019990014987513734\n",
      "epoch: 1100, acc: 0.905, loss: 0.355 (data_loss: 0.267, reg_loss: 0.088), lr: 0.01998901603568839\n",
      "epoch: 1200, acc: 0.907, loss: 0.349 (data_loss: 0.263, reg_loss: 0.086), lr: 0.019988017183698373\n",
      "epoch: 1300, acc: 0.906, loss: 0.342 (data_loss: 0.259, reg_loss: 0.083), lr: 0.01998701843152872\n",
      "epoch: 1400, acc: 0.911, loss: 0.340 (data_loss: 0.259, reg_loss: 0.081), lr: 0.019986019779164473\n",
      "epoch: 1500, acc: 0.905, loss: 0.331 (data_loss: 0.252, reg_loss: 0.079), lr: 0.019985021226590672\n",
      "epoch: 1600, acc: 0.907, loss: 0.335 (data_loss: 0.259, reg_loss: 0.077), lr: 0.01998402277379235\n",
      "epoch: 1700, acc: 0.910, loss: 0.322 (data_loss: 0.248, reg_loss: 0.075), lr: 0.01998302442075457\n",
      "epoch: 1800, acc: 0.907, loss: 0.319 (data_loss: 0.247, reg_loss: 0.073), lr: 0.019982026167462367\n",
      "epoch: 1900, acc: 0.912, loss: 0.314 (data_loss: 0.243, reg_loss: 0.071), lr: 0.019981028013900805\n",
      "epoch: 2000, acc: 0.912, loss: 0.319 (data_loss: 0.250, reg_loss: 0.070), lr: 0.019980029960054924\n",
      "epoch: 2100, acc: 0.910, loss: 0.312 (data_loss: 0.243, reg_loss: 0.068), lr: 0.019979032005909798\n",
      "epoch: 2200, acc: 0.910, loss: 0.305 (data_loss: 0.239, reg_loss: 0.067), lr: 0.01997803415145048\n",
      "epoch: 2300, acc: 0.909, loss: 0.310 (data_loss: 0.245, reg_loss: 0.066), lr: 0.019977036396662037\n",
      "epoch: 2400, acc: 0.916, loss: 0.304 (data_loss: 0.240, reg_loss: 0.064), lr: 0.019976038741529537\n",
      "epoch: 2500, acc: 0.916, loss: 0.301 (data_loss: 0.238, reg_loss: 0.063), lr: 0.01997504118603805\n",
      "epoch: 2600, acc: 0.906, loss: 0.302 (data_loss: 0.240, reg_loss: 0.062), lr: 0.01997404373017264\n",
      "epoch: 2700, acc: 0.910, loss: 0.297 (data_loss: 0.236, reg_loss: 0.061), lr: 0.0199730463739184\n",
      "epoch: 2800, acc: 0.916, loss: 0.291 (data_loss: 0.230, reg_loss: 0.060), lr: 0.019972049117260395\n",
      "epoch: 2900, acc: 0.912, loss: 0.290 (data_loss: 0.231, reg_loss: 0.059), lr: 0.019971051960183714\n",
      "epoch: 3000, acc: 0.914, loss: 0.293 (data_loss: 0.234, reg_loss: 0.059), lr: 0.019970054902673444\n",
      "epoch: 3100, acc: 0.917, loss: 0.286 (data_loss: 0.228, reg_loss: 0.058), lr: 0.019969057944714663\n",
      "epoch: 3200, acc: 0.911, loss: 0.288 (data_loss: 0.231, reg_loss: 0.057), lr: 0.019968061086292475\n",
      "epoch: 3300, acc: 0.909, loss: 0.289 (data_loss: 0.233, reg_loss: 0.056), lr: 0.019967064327391967\n",
      "epoch: 3400, acc: 0.910, loss: 0.286 (data_loss: 0.230, reg_loss: 0.055), lr: 0.019966067667998237\n",
      "epoch: 3500, acc: 0.910, loss: 0.286 (data_loss: 0.231, reg_loss: 0.055), lr: 0.019965071108096383\n",
      "epoch: 3600, acc: 0.911, loss: 0.283 (data_loss: 0.229, reg_loss: 0.054), lr: 0.01996407464767152\n",
      "epoch: 3700, acc: 0.913, loss: 0.281 (data_loss: 0.228, reg_loss: 0.053), lr: 0.019963078286708732\n",
      "epoch: 3800, acc: 0.915, loss: 0.304 (data_loss: 0.232, reg_loss: 0.072), lr: 0.019962082025193145\n",
      "epoch: 3900, acc: 0.916, loss: 0.296 (data_loss: 0.228, reg_loss: 0.067), lr: 0.019961085863109868\n",
      "epoch: 4000, acc: 0.917, loss: 0.291 (data_loss: 0.227, reg_loss: 0.065), lr: 0.019960089800444013\n",
      "epoch: 4100, acc: 0.917, loss: 0.289 (data_loss: 0.226, reg_loss: 0.063), lr: 0.019959093837180697\n",
      "epoch: 4200, acc: 0.917, loss: 0.286 (data_loss: 0.226, reg_loss: 0.061), lr: 0.01995809797330505\n",
      "epoch: 4300, acc: 0.917, loss: 0.284 (data_loss: 0.224, reg_loss: 0.059), lr: 0.01995710220880218\n",
      "epoch: 4400, acc: 0.918, loss: 0.283 (data_loss: 0.225, reg_loss: 0.058), lr: 0.019956106543657228\n",
      "epoch: 4500, acc: 0.916, loss: 0.281 (data_loss: 0.224, reg_loss: 0.057), lr: 0.019955110977855316\n",
      "epoch: 4600, acc: 0.909, loss: 0.285 (data_loss: 0.229, reg_loss: 0.056), lr: 0.01995411551138158\n",
      "epoch: 4700, acc: 0.915, loss: 0.280 (data_loss: 0.225, reg_loss: 0.055), lr: 0.019953120144221154\n",
      "epoch: 4800, acc: 0.914, loss: 0.277 (data_loss: 0.222, reg_loss: 0.054), lr: 0.019952124876359174\n",
      "epoch: 4900, acc: 0.911, loss: 0.281 (data_loss: 0.227, reg_loss: 0.053), lr: 0.01995112970778079\n",
      "epoch: 5000, acc: 0.916, loss: 0.277 (data_loss: 0.224, reg_loss: 0.053), lr: 0.019950134638471142\n",
      "epoch: 5100, acc: 0.914, loss: 0.273 (data_loss: 0.221, reg_loss: 0.052), lr: 0.019949139668415376\n",
      "epoch: 5200, acc: 0.917, loss: 0.271 (data_loss: 0.219, reg_loss: 0.052), lr: 0.01994814479759864\n",
      "epoch: 5300, acc: 0.917, loss: 0.270 (data_loss: 0.219, reg_loss: 0.051), lr: 0.019947150026006097\n",
      "epoch: 5400, acc: 0.916, loss: 0.273 (data_loss: 0.223, reg_loss: 0.050), lr: 0.019946155353622895\n",
      "epoch: 5500, acc: 0.915, loss: 0.275 (data_loss: 0.225, reg_loss: 0.050), lr: 0.019945160780434196\n",
      "epoch: 5600, acc: 0.922, loss: 0.267 (data_loss: 0.217, reg_loss: 0.049), lr: 0.019944166306425162\n",
      "epoch: 5700, acc: 0.914, loss: 0.274 (data_loss: 0.226, reg_loss: 0.049), lr: 0.01994317193158096\n",
      "epoch: 5800, acc: 0.912, loss: 0.273 (data_loss: 0.224, reg_loss: 0.048), lr: 0.019942177655886757\n",
      "epoch: 5900, acc: 0.909, loss: 0.282 (data_loss: 0.234, reg_loss: 0.048), lr: 0.019941183479327725\n",
      "epoch: 6000, acc: 0.921, loss: 0.264 (data_loss: 0.216, reg_loss: 0.048), lr: 0.019940189401889033\n",
      "epoch: 6100, acc: 0.913, loss: 0.271 (data_loss: 0.224, reg_loss: 0.047), lr: 0.01993919542355587\n",
      "epoch: 6200, acc: 0.912, loss: 0.272 (data_loss: 0.225, reg_loss: 0.047), lr: 0.019938201544313403\n",
      "epoch: 6300, acc: 0.912, loss: 0.268 (data_loss: 0.222, reg_loss: 0.046), lr: 0.01993720776414682\n",
      "epoch: 6400, acc: 0.919, loss: 0.260 (data_loss: 0.214, reg_loss: 0.046), lr: 0.019936214083041307\n",
      "epoch: 6500, acc: 0.915, loss: 0.264 (data_loss: 0.219, reg_loss: 0.046), lr: 0.01993522050098206\n",
      "epoch: 6600, acc: 0.911, loss: 0.269 (data_loss: 0.224, reg_loss: 0.045), lr: 0.019934227017954262\n",
      "epoch: 6700, acc: 0.917, loss: 0.262 (data_loss: 0.217, reg_loss: 0.045), lr: 0.01993323363394311\n",
      "epoch: 6800, acc: 0.913, loss: 0.262 (data_loss: 0.218, reg_loss: 0.044), lr: 0.0199322403489338\n",
      "epoch: 6900, acc: 0.919, loss: 0.259 (data_loss: 0.214, reg_loss: 0.044), lr: 0.019931247162911534\n",
      "epoch: 7000, acc: 0.915, loss: 0.262 (data_loss: 0.218, reg_loss: 0.044), lr: 0.019930254075861523\n",
      "epoch: 7100, acc: 0.917, loss: 0.259 (data_loss: 0.215, reg_loss: 0.043), lr: 0.019929261087768962\n",
      "epoch: 7200, acc: 0.734, loss: 1.106 (data_loss: 1.052, reg_loss: 0.054), lr: 0.01992826819861907\n",
      "epoch: 7300, acc: 0.917, loss: 0.289 (data_loss: 0.231, reg_loss: 0.059), lr: 0.019927275408397054\n",
      "epoch: 7400, acc: 0.916, loss: 0.282 (data_loss: 0.226, reg_loss: 0.056), lr: 0.019926282717088132\n",
      "epoch: 7500, acc: 0.917, loss: 0.278 (data_loss: 0.224, reg_loss: 0.054), lr: 0.01992529012467752\n",
      "epoch: 7600, acc: 0.918, loss: 0.274 (data_loss: 0.221, reg_loss: 0.054), lr: 0.019924297631150445\n",
      "epoch: 7700, acc: 0.918, loss: 0.271 (data_loss: 0.217, reg_loss: 0.053), lr: 0.019923305236492123\n",
      "epoch: 7800, acc: 0.919, loss: 0.267 (data_loss: 0.215, reg_loss: 0.053), lr: 0.01992231294068779\n",
      "epoch: 7900, acc: 0.923, loss: 0.266 (data_loss: 0.214, reg_loss: 0.052), lr: 0.019921320743722666\n",
      "epoch: 8000, acc: 0.921, loss: 0.264 (data_loss: 0.213, reg_loss: 0.051), lr: 0.019920328645582\n",
      "epoch: 8100, acc: 0.919, loss: 0.268 (data_loss: 0.218, reg_loss: 0.050), lr: 0.019919336646251007\n",
      "epoch: 8200, acc: 0.919, loss: 0.260 (data_loss: 0.211, reg_loss: 0.049), lr: 0.019918344745714942\n",
      "epoch: 8300, acc: 0.923, loss: 0.261 (data_loss: 0.213, reg_loss: 0.048), lr: 0.019917352943959042\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 8400, acc: 0.923, loss: 0.258 (data_loss: 0.210, reg_loss: 0.048), lr: 0.019916361240968555\n",
      "epoch: 8500, acc: 0.919, loss: 0.256 (data_loss: 0.210, reg_loss: 0.047), lr: 0.01991536963672872\n",
      "epoch: 8600, acc: 0.920, loss: 0.256 (data_loss: 0.210, reg_loss: 0.046), lr: 0.019914378131224802\n",
      "epoch: 8700, acc: 0.917, loss: 0.260 (data_loss: 0.212, reg_loss: 0.048), lr: 0.01991338672444204\n",
      "epoch: 8800, acc: 0.919, loss: 0.254 (data_loss: 0.207, reg_loss: 0.047), lr: 0.0199123954163657\n",
      "epoch: 8900, acc: 0.921, loss: 0.254 (data_loss: 0.208, reg_loss: 0.046), lr: 0.019911404206981037\n",
      "epoch: 9000, acc: 0.921, loss: 0.254 (data_loss: 0.209, reg_loss: 0.045), lr: 0.019910413096273318\n",
      "epoch: 9100, acc: 0.922, loss: 0.253 (data_loss: 0.209, reg_loss: 0.044), lr: 0.019909422084227805\n",
      "epoch: 9200, acc: 0.922, loss: 0.252 (data_loss: 0.208, reg_loss: 0.044), lr: 0.019908431170829768\n",
      "epoch: 9300, acc: 0.842, loss: 0.535 (data_loss: 0.479, reg_loss: 0.057), lr: 0.01990744035606448\n",
      "epoch: 9400, acc: 0.916, loss: 0.277 (data_loss: 0.224, reg_loss: 0.053), lr: 0.01990644963991721\n",
      "epoch: 9500, acc: 0.917, loss: 0.272 (data_loss: 0.219, reg_loss: 0.053), lr: 0.01990545902237324\n",
      "epoch: 9600, acc: 0.919, loss: 0.268 (data_loss: 0.216, reg_loss: 0.052), lr: 0.019904468503417844\n",
      "epoch: 9700, acc: 0.920, loss: 0.265 (data_loss: 0.215, reg_loss: 0.051), lr: 0.019903478083036316\n",
      "epoch: 9800, acc: 0.920, loss: 0.264 (data_loss: 0.214, reg_loss: 0.050), lr: 0.019902487761213932\n",
      "epoch: 9900, acc: 0.920, loss: 0.262 (data_loss: 0.212, reg_loss: 0.049), lr: 0.019901497537935988\n",
      "epoch: 10000, acc: 0.921, loss: 0.263 (data_loss: 0.214, reg_loss: 0.049), lr: 0.019900507413187767\n",
      "validation, acc: 0.913, loss: 0.279\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import nnfs\n",
    "from nnfs.datasets import spiral_data\n",
    "\n",
    "nnfs.init()\n",
    "\n",
    "\n",
    "# Dense layer\n",
    "class Layer_Dense:\n",
    "\n",
    "    # Layer initialization\n",
    "    def __init__(self, n_inputs, n_neurons,\n",
    "                 weight_regularizer_l1=0, weight_regularizer_l2=0,\n",
    "                 bias_regularizer_l1=0, bias_regularizer_l2=0):\n",
    "        # Initialize weights and biases\n",
    "        self.weights = 0.01 * np.random.randn(n_inputs, n_neurons)\n",
    "        self.biases = np.zeros((1, n_neurons))\n",
    "        # Set regularization strength\n",
    "        self.weight_regularizer_l1 = weight_regularizer_l1\n",
    "        self.weight_regularizer_l2 = weight_regularizer_l2\n",
    "        self.bias_regularizer_l1 = bias_regularizer_l1\n",
    "        self.bias_regularizer_l2 = bias_regularizer_l2\n",
    "\n",
    "    # Forward pass\n",
    "    def forward(self, inputs):\n",
    "        # Remember input values\n",
    "        self.inputs = inputs\n",
    "        # Calculate output values from inputs, weights and biases\n",
    "        self.output = np.dot(inputs, self.weights) + self.biases\n",
    "    # Backward pass\n",
    "    def backward(self, dvalues):\n",
    "        # Gradients on parameters\n",
    "        self.dweights = np.dot(self.inputs.T, dvalues)\n",
    "        self.dbiases = np.sum(dvalues, axis=0, keepdims=True)\n",
    "\n",
    "        # Gradients on regularization\n",
    "        # L1 on weights\n",
    "        if self.weight_regularizer_l1 > 0:\n",
    "            dL1 = np.ones_like(self.weights)\n",
    "            dL1[self.weights < 0] = -1\n",
    "            self.dweights += self.weight_regularizer_l1 * dL1\n",
    "        # L2 on weights\n",
    "        if self.weight_regularizer_l2 > 0:\n",
    "            self.dweights += 2 * self.weight_regularizer_l2 * self.weights\n",
    "        # L1 on biases\n",
    "        if self.bias_regularizer_l1 > 0:\n",
    "            dL1 = np.ones_like(self.biases)\n",
    "            dL1[self.biases < 0] = -1\n",
    "            self.dbiases += self.bias_regularizer_l1 * dL1\n",
    "        # L2 on biases\n",
    "        if self.bias_regularizer_l2 > 0:\n",
    "            self.dbiases += 2 * self.bias_regularizer_l2 * self.biases\n",
    "\n",
    "        # Gradient on values\n",
    "        self.dinputs = np.dot(dvalues, self.weights.T)\n",
    "\n",
    "\n",
    "# ReLU activation\n",
    "class Activation_ReLU:\n",
    "\n",
    "    # Forward pass\n",
    "    def forward(self, inputs):\n",
    "        # Remember input values\n",
    "        self.inputs = inputs\n",
    "        # Calculate output values from inputs\n",
    "        self.output = np.maximum(0, inputs)\n",
    "\n",
    "    # Backward pass\n",
    "    def backward(self, dvalues):\n",
    "        # Since we need to modify original variable,\n",
    "        # let's make a copy of values first\n",
    "        self.dinputs = dvalues.copy()\n",
    "\n",
    "        # Zero gradient where input values were negative\n",
    "        self.dinputs[self.inputs <= 0] = 0\n",
    "\n",
    "# Softmax activation\n",
    "class Activation_Softmax:\n",
    "\n",
    "    # Forward pass\n",
    "    def forward(self, inputs):\n",
    "        # Remember input values\n",
    "        self.inputs = inputs\n",
    "\n",
    "        # Get unnormalized probabilities\n",
    "        exp_values = np.exp(inputs - np.max(inputs, axis=1,\n",
    "                                            keepdims=True))\n",
    "        # Normalize them for each sample\n",
    "        probabilities = exp_values / np.sum(exp_values, axis=1,\n",
    "                                            keepdims=True)\n",
    "\n",
    "        self.output = probabilities\n",
    "\n",
    "    # Backward pass\n",
    "    def backward(self, dvalues):\n",
    "\n",
    "        # Create uninitialized array\n",
    "        self.dinputs = np.empty_like(dvalues)\n",
    "\n",
    "        # Enumerate outputs and gradients\n",
    "        for index, (single_output, single_dvalues) in \\\n",
    "                enumerate(zip(self.output, dvalues)):\n",
    "            # Flatten output array\n",
    "            single_output = single_output.reshape(-1, 1)\n",
    "            # Calculate Jacobian matrix of the output\n",
    "            jacobian_matrix = np.diagflat(single_output) - \\\n",
    "                              np.dot(single_output, single_output.T)\n",
    "            # Calculate sample-wise gradient\n",
    "            # and add it to the array of sample gradients\n",
    "            self.dinputs[index] = np.dot(jacobian_matrix,\n",
    "                                         single_dvalues)\n",
    "\n",
    "\n",
    "# SGD optimizer\n",
    "class Optimizer_SGD:\n",
    "\n",
    "    # Initialize optimizer - set settings,\n",
    "    # learning rate of 1. is default for this optimizer\n",
    "    def __init__(self, learning_rate=1., decay=0., momentum=0.):\n",
    "        self.learning_rate = learning_rate\n",
    "        self.current_learning_rate = learning_rate\n",
    "        self.decay = decay\n",
    "        self.iterations = 0\n",
    "        self.momentum = momentum\n",
    "\n",
    "    # Call once before any parameter updates\n",
    "    def pre_update_params(self):\n",
    "        if self.decay:\n",
    "            self.current_learning_rate = self.learning_rate * \\\n",
    "                (1. / (1. + self.decay * self.iterations))\n",
    "\n",
    "    # Update parameters\n",
    "    def update_params(self, layer):\n",
    "\n",
    "        # If we use momentum\n",
    "        if self.momentum:\n",
    "\n",
    "            # If layer does not contain momentum arrays, create them\n",
    "            # filled with zeros\n",
    "            if not hasattr(layer, 'weight_momentums'):\n",
    "                layer.weight_momentums = np.zeros_like(layer.weights)\n",
    "                # If there is no momentum array for weights\n",
    "                # The array doesn't exist for biases yet either.\n",
    "                layer.bias_momentums = np.zeros_like(layer.biases)\n",
    "\n",
    "            # Build weight updates with momentum - take previous\n",
    "            # updates multiplied by retain factor and update with\n",
    "            # current gradients\n",
    "            weight_updates = \\\n",
    "                self.momentum * layer.weight_momentums - \\\n",
    "                self.current_learning_rate * layer.dweights\n",
    "            layer.weight_momentums = weight_updates\n",
    "\n",
    "            # Build bias updates\n",
    "            bias_updates = \\\n",
    "                self.momentum * layer.bias_momentums - \\\n",
    "                self.current_learning_rate * layer.dbiases\n",
    "            layer.bias_momentums = bias_updates\n",
    "\n",
    "        # Vanilla SGD updates (as before momentum update)\n",
    "        else:\n",
    "            weight_updates = -self.current_learning_rate * \\\n",
    "                             layer.dweights\n",
    "            bias_updates = -self.current_learning_rate * \\\n",
    "                           layer.dbiases\n",
    "\n",
    "        # Update weights and biases using either\n",
    "        # vanilla or momentum updates\n",
    "        layer.weights += weight_updates\n",
    "        layer.biases += bias_updates\n",
    "\n",
    "    # Call once after any parameter updates\n",
    "    def post_update_params(self):\n",
    "        self.iterations += 1\n",
    "# Adagrad optimizer\n",
    "class Optimizer_Adagrad:\n",
    "\n",
    "    # Initialize optimizer - set settings\n",
    "    def __init__(self, learning_rate=1., decay=0., epsilon=1e-7):\n",
    "        self.learning_rate = learning_rate\n",
    "        self.current_learning_rate = learning_rate\n",
    "        self.decay = decay\n",
    "        self.iterations = 0\n",
    "        self.epsilon = epsilon\n",
    "\n",
    "    # Call once before any parameter updates\n",
    "    def pre_update_params(self):\n",
    "        if self.decay:\n",
    "            self.current_learning_rate = self.learning_rate * \\\n",
    "                (1. / (1. + self.decay * self.iterations))\n",
    "\n",
    "    # Update parameters\n",
    "    def update_params(self, layer):\n",
    "\n",
    "        # If layer does not contain cache arrays,\n",
    "        # create them filled with zeros\n",
    "        if not hasattr(layer, 'weight_cache'):\n",
    "            layer.weight_cache = np.zeros_like(layer.weights)\n",
    "            layer.bias_cache = np.zeros_like(layer.biases)\n",
    "\n",
    "        # Update cache with squared current gradients\n",
    "        layer.weight_cache += layer.dweights**2\n",
    "        layer.bias_cache += layer.dbiases**2\n",
    "\n",
    "        # Vanilla SGD parameter update + normalization\n",
    "        # with square rooted cache\n",
    "        layer.weights += -self.current_learning_rate * \\\n",
    "                         layer.dweights / \\\n",
    "                         (np.sqrt(layer.weight_cache) + self.epsilon)\n",
    "        layer.biases += -self.current_learning_rate * \\\n",
    "                        layer.dbiases / \\\n",
    "                        (np.sqrt(layer.bias_cache) + self.epsilon)\n",
    "\n",
    "    # Call once after any parameter updates\n",
    "    def post_update_params(self):\n",
    "        self.iterations += 1\n",
    "\n",
    "\n",
    "\n",
    "# RMSprop optimizer\n",
    "class Optimizer_RMSprop:\n",
    "\n",
    "    # Initialize optimizer - set settings\n",
    "    def __init__(self, learning_rate=0.001, decay=0., epsilon=1e-7,\n",
    "                 rho=0.9):\n",
    "        self.learning_rate = learning_rate\n",
    "        self.current_learning_rate = learning_rate\n",
    "        self.decay = decay\n",
    "        self.iterations = 0\n",
    "        self.epsilon = epsilon\n",
    "        self.rho = rho\n",
    "\n",
    "    # Call once before any parameter updates\n",
    "    def pre_update_params(self):\n",
    "        if self.decay:\n",
    "            self.current_learning_rate = self.learning_rate * \\\n",
    "                (1. / (1. + self.decay * self.iterations))\n",
    "\n",
    "    # Update parameters\n",
    "    def update_params(self, layer):\n",
    "\n",
    "        # If layer does not contain cache arrays,\n",
    "        # create them filled with zeros\n",
    "        if not hasattr(layer, 'weight_cache'):\n",
    "            layer.weight_cache = np.zeros_like(layer.weights)\n",
    "            layer.bias_cache = np.zeros_like(layer.biases)\n",
    "\n",
    "        # Update cache with squared current gradients\n",
    "        layer.weight_cache = self.rho * layer.weight_cache + \\\n",
    "            (1 - self.rho) * layer.dweights**2\n",
    "        layer.bias_cache = self.rho * layer.bias_cache + \\\n",
    "            (1 - self.rho) * layer.dbiases**2\n",
    "\n",
    "        # Vanilla SGD parameter update + normalization\n",
    "        # with square rooted cache\n",
    "        layer.weights += -self.current_learning_rate * \\\n",
    "                         layer.dweights / \\\n",
    "                         (np.sqrt(layer.weight_cache) + self.epsilon)\n",
    "        layer.biases += -self.current_learning_rate * \\\n",
    "                        layer.dbiases / \\\n",
    "                        (np.sqrt(layer.bias_cache) + self.epsilon)\n",
    "\n",
    "    # Call once after any parameter updates\n",
    "    def post_update_params(self):\n",
    "        self.iterations += 1\n",
    "\n",
    "\n",
    "\n",
    "# Adam optimizer\n",
    "class Optimizer_Adam:\n",
    "\n",
    "    # Initialize optimizer - set settings\n",
    "    def __init__(self, learning_rate=0.001, decay=0., epsilon=1e-7,\n",
    "                 beta_1=0.9, beta_2=0.999):\n",
    "        self.learning_rate = learning_rate\n",
    "        self.current_learning_rate = learning_rate\n",
    "        self.decay = decay\n",
    "        self.iterations = 0\n",
    "        self.epsilon = epsilon\n",
    "        self.beta_1 = beta_1\n",
    "        self.beta_2 = beta_2\n",
    "\n",
    "    # Call once before any parameter updates\n",
    "    def pre_update_params(self):\n",
    "        if self.decay:\n",
    "            self.current_learning_rate = self.learning_rate * \\\n",
    "                (1. / (1. + self.decay * self.iterations))\n",
    "\n",
    "    # Update parameters\n",
    "    def update_params(self, layer):\n",
    "\n",
    "        # If layer does not contain cache arrays,\n",
    "        # create them filled with zeros\n",
    "        if not hasattr(layer, 'weight_cache'):\n",
    "            layer.weight_momentums = np.zeros_like(layer.weights)\n",
    "            layer.weight_cache = np.zeros_like(layer.weights)\n",
    "            layer.bias_momentums = np.zeros_like(layer.biases)\n",
    "            layer.bias_cache = np.zeros_like(layer.biases)\n",
    "\n",
    "        # Update momentum  with current gradients\n",
    "        layer.weight_momentums = self.beta_1 * \\\n",
    "                                 layer.weight_momentums + \\\n",
    "                                 (1 - self.beta_1) * layer.dweights\n",
    "        layer.bias_momentums = self.beta_1 * \\\n",
    "                               layer.bias_momentums + \\\n",
    "                               (1 - self.beta_1) * layer.dbiases\n",
    "        # Get corrected momentum\n",
    "        # self.iteration is 0 at first pass\n",
    "        # and we need to start with 1 here\n",
    "        weight_momentums_corrected = layer.weight_momentums / \\\n",
    "            (1 - self.beta_1 ** (self.iterations + 1))\n",
    "        bias_momentums_corrected = layer.bias_momentums / \\\n",
    "            (1 - self.beta_1 ** (self.iterations + 1))\n",
    "        # Update cache with squared current gradients\n",
    "        layer.weight_cache = self.beta_2 * layer.weight_cache + \\\n",
    "            (1 - self.beta_2) * layer.dweights**2\n",
    "\n",
    "        layer.bias_cache = self.beta_2 * layer.bias_cache + \\\n",
    "            (1 - self.beta_2) * layer.dbiases**2\n",
    "        # Get corrected cache\n",
    "        weight_cache_corrected = layer.weight_cache / \\\n",
    "            (1 - self.beta_2 ** (self.iterations + 1))\n",
    "        bias_cache_corrected = layer.bias_cache / \\\n",
    "            (1 - self.beta_2 ** (self.iterations + 1))\n",
    "\n",
    "        # Vanilla SGD parameter update + normalization\n",
    "        # with square rooted cache\n",
    "        layer.weights += -self.current_learning_rate * \\\n",
    "                         weight_momentums_corrected / \\\n",
    "                         (np.sqrt(weight_cache_corrected) +\n",
    "                             self.epsilon)\n",
    "        layer.biases += -self.current_learning_rate * \\\n",
    "                         bias_momentums_corrected / \\\n",
    "                         (np.sqrt(bias_cache_corrected) +\n",
    "                             self.epsilon)\n",
    "\n",
    "    # Call once after any parameter updates\n",
    "    def post_update_params(self):\n",
    "        self.iterations += 1\n",
    "\n",
    "\n",
    "# Common loss class\n",
    "class Loss:\n",
    "\n",
    "    # Regularization loss calculation\n",
    "    def regularization_loss(self, layer):\n",
    "\n",
    "        # 0 by default\n",
    "        regularization_loss = 0\n",
    "\n",
    "        # L1 regularization - weights\n",
    "        # calculate only when factor greater than 0\n",
    "        if layer.weight_regularizer_l1 > 0:\n",
    "            regularization_loss += layer.weight_regularizer_l1 * np.sum(np.abs(layer.weights))\n",
    "\n",
    "        # L2 regularization - weights\n",
    "        if layer.weight_regularizer_l2 > 0:\n",
    "            regularization_loss += layer.weight_regularizer_l2 * np.sum(layer.weights * layer.weights)\n",
    "\n",
    "        # L1 regularization - biases\n",
    "        # calculate only when factor greater than 0\n",
    "        if layer.bias_regularizer_l1 > 0:\n",
    "            regularization_loss += layer.bias_regularizer_l1 * np.sum(np.abs(layer.biases))\n",
    "\n",
    "        # L2 regularization - biases\n",
    "        if layer.bias_regularizer_l2 > 0:\n",
    "            regularization_loss += layer.bias_regularizer_l2 * np.sum(layer.biases * layer.biases)\n",
    "\n",
    "        return regularization_loss\n",
    "\n",
    "    # Calculates the data and regularization losses\n",
    "    # given model output and ground truth values\n",
    "    def calculate(self, output, y):\n",
    "\n",
    "        # Calculate sample losses\n",
    "        sample_losses = self.forward(output, y)\n",
    "\n",
    "        # Calculate mean loss\n",
    "        data_loss = np.mean(sample_losses)\n",
    "\n",
    "        # Return loss\n",
    "        return data_loss\n",
    "\n",
    "\n",
    "# Cross-entropy loss\n",
    "class Loss_CategoricalCrossentropy(Loss):\n",
    "\n",
    "    # Forward pass\n",
    "    def forward(self, y_pred, y_true):\n",
    "\n",
    "        # Number of samples in a batch\n",
    "        samples = len(y_pred)\n",
    "\n",
    "        # Clip data to prevent division by 0\n",
    "        # Clip both sides to not drag mean towards any value\n",
    "        y_pred_clipped = np.clip(y_pred, 1e-7, 1 - 1e-7)\n",
    "\n",
    "        # Probabilities for target values -\n",
    "        # only if categorical labels\n",
    "        if len(y_true.shape) == 1:\n",
    "            correct_confidences = y_pred_clipped[\n",
    "                range(samples),\n",
    "                y_true\n",
    "            ]\n",
    "\n",
    "        # Mask values - only for one-hot encoded labels\n",
    "        elif len(y_true.shape) == 2:\n",
    "            correct_confidences = np.sum(\n",
    "                y_pred_clipped * y_true,\n",
    "                axis=1\n",
    "            )\n",
    "\n",
    "        # Losses\n",
    "        negative_log_likelihoods = -np.log(correct_confidences)\n",
    "        return negative_log_likelihoods\n",
    "\n",
    "    # Backward pass\n",
    "    def backward(self, dvalues, y_true):\n",
    "\n",
    "        # Number of samples\n",
    "        samples = len(dvalues)\n",
    "        # Number of labels in every sample\n",
    "        # We'll use the first sample to count them\n",
    "        labels = len(dvalues[0])\n",
    "\n",
    "        # If labels are sparse, turn them into one-hot vector\n",
    "        if len(y_true.shape) == 1:\n",
    "            y_true = np.eye(labels)[y_true]\n",
    "\n",
    "        # Calculate gradient\n",
    "        self.dinputs = -y_true / dvalues\n",
    "        # Normalize gradient\n",
    "        self.dinputs = self.dinputs / samples\n",
    "\n",
    "\n",
    "# Softmax classifier - combined Softmax activation\n",
    "# and cross-entropy loss for faster backward step\n",
    "class Activation_Softmax_Loss_CategoricalCrossentropy():\n",
    "\n",
    "    # Creates activation and loss function objects\n",
    "    def __init__(self):\n",
    "        self.activation = Activation_Softmax()\n",
    "        self.loss = Loss_CategoricalCrossentropy()\n",
    "\n",
    "    # Forward pass\n",
    "    def forward(self, inputs, y_true):\n",
    "        # Output layer's activation function\n",
    "        self.activation.forward(inputs)\n",
    "        # Set the output\n",
    "        self.output = self.activation.output\n",
    "        # Calculate and return loss value\n",
    "        return self.loss.calculate(self.output, y_true)\n",
    "\n",
    "\n",
    "    # Backward pass\n",
    "    def backward(self, dvalues, y_true):\n",
    "\n",
    "        # Number of samples\n",
    "        samples = len(dvalues)\n",
    "\n",
    "        # If labels are one-hot encoded,\n",
    "        # turn them into discrete values\n",
    "        if len(y_true.shape) == 2:\n",
    "            y_true = np.argmax(y_true, axis=1)\n",
    "\n",
    "        # Copy so we can safely modify\n",
    "        self.dinputs = dvalues.copy()\n",
    "        # Calculate gradient\n",
    "        self.dinputs[range(samples), y_true] -= 1\n",
    "        # Normalize gradient\n",
    "        self.dinputs = self.dinputs / samples\n",
    "\n",
    "\n",
    "# Create dataset\n",
    "X, y = spiral_data(samples=1000, classes=3)\n",
    "\n",
    "# Create Dense layer with 2 input features and 64 output values\n",
    "dense1 = Layer_Dense(2, 512, weight_regularizer_l2=5e-4, bias_regularizer_l2=5e-4)\n",
    "\n",
    "# Create ReLU activation (to be used with Dense layer):\n",
    "activation1 = Activation_ReLU()\n",
    "\n",
    "# Create second Dense layer with 64 input features (as we take output\n",
    "# of previous layer here) and 3 output values (output values)\n",
    "dense2 = Layer_Dense(512, 3)\n",
    "\n",
    "# Create Softmax classifier's combined loss and activation\n",
    "loss_activation = Activation_Softmax_Loss_CategoricalCrossentropy()\n",
    "\n",
    "# Create optimizer\n",
    "optimizer = Optimizer_Adam(learning_rate=0.02, decay=5e-7)\n",
    "\n",
    "# Train in loop\n",
    "for epoch in range(10001):\n",
    "\n",
    "    # Perform a forward pass of our training data through this layer\n",
    "    dense1.forward(X)\n",
    "\n",
    "    # Perform a forward pass through activation function\n",
    "    # takes the output of first dense layer here\n",
    "    activation1.forward(dense1.output)\n",
    "\n",
    "    # Perform a forward pass through second Dense layer\n",
    "    # takes outputs of activation function of first layer as inputs\n",
    "    dense2.forward(activation1.output)\n",
    "\n",
    "    # Perform a forward pass through the activation/loss function\n",
    "    # takes the output of second dense layer here and returns loss\n",
    "    data_loss = loss_activation.forward(dense2.output, y)\n",
    "\n",
    "    # Calculate regularization penalty\n",
    "    regularization_loss = \\\n",
    "        loss_activation.loss.regularization_loss(dense1) + \\\n",
    "        loss_activation.loss.regularization_loss(dense2)\n",
    "\n",
    "    # Calculate overall loss\n",
    "    loss = data_loss + regularization_loss\n",
    "\n",
    "    # Calculate accuracy from output of activation2 and targets\n",
    "    # calculate values along first axis\n",
    "    predictions = np.argmax(loss_activation.output, axis=1)\n",
    "    if len(y.shape) == 2:\n",
    "        y = np.argmax(y, axis=1)\n",
    "    accuracy = np.mean(predictions==y)\n",
    "\n",
    "    if not epoch % 100:\n",
    "        print(f'epoch: {epoch}, ' +\n",
    "              f'acc: {accuracy:.3f}, ' +\n",
    "              f'loss: {loss:.3f} (' +\n",
    "              f'data_loss: {data_loss:.3f}, ' +\n",
    "              f'reg_loss: {regularization_loss:.3f}), ' +\n",
    "              f'lr: {optimizer.current_learning_rate}')\n",
    "\n",
    "    # Backward pass\n",
    "    loss_activation.backward(loss_activation.output, y)\n",
    "    dense2.backward(loss_activation.dinputs)\n",
    "    activation1.backward(dense2.dinputs)\n",
    "    dense1.backward(activation1.dinputs)\n",
    "\n",
    "    # Update weights and biases\n",
    "    optimizer.pre_update_params()\n",
    "    optimizer.update_params(dense1)\n",
    "    optimizer.update_params(dense2)\n",
    "    optimizer.post_update_params()\n",
    "\n",
    "\n",
    "# Validate the model\n",
    "\n",
    "# Create test dataset\n",
    "X_test, y_test = spiral_data(samples=100, classes=3)\n",
    "\n",
    "# Perform a forward pass of our testing data through this layer\n",
    "dense1.forward(X_test)\n",
    "\n",
    "# Perform a forward pass through activation function\n",
    "# takes the output of first dense layer here\n",
    "activation1.forward(dense1.output)\n",
    "\n",
    "# Perform a forward pass through second Dense layer\n",
    "# takes outputs of activation function of first layer as inputs\n",
    "dense2.forward(activation1.output)\n",
    "\n",
    "# Perform a forward pass through the activation/loss function\n",
    "# takes the output of second dense layer here and returns loss\n",
    "loss = loss_activation.forward(dense2.output, y_test)\n",
    "\n",
    "# Calculate accuracy from output of activation2 and targets\n",
    "# calculate values along first axis\n",
    "predictions = np.argmax(loss_activation.output, axis=1)\n",
    "if len(y_test.shape) == 2:\n",
    "    y_test = np.argmax(y_test, axis=1)\n",
    "accuracy = np.mean(predictions==y_test)\n",
    "\n",
    "print(f'validation, acc: {accuracy:.3f}, loss: {loss:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e55b39c5",
   "metadata": {},
   "source": [
    "<center><img src='./image/14-1.png' style='width: 70%'/><font color='gray'><i>Training with regularization.</i></font></center>\n",
    "\n",
    "Ｔhe training data in the background (dimmed dots) and the validation data in the foreground.\n",
    "\n",
    "After adding the L2 regularization term to the hidden layer, we achieved a lower validation loss (1.017 before adding regularization in, 0.525 now) and higher accuracy (0.797 before, 0.840 now).\n",
    "\n",
    "<center><img src='./image/14-2.png' style='width: 70%'/><font color='gray'><i>Training with regularization and more training data.</i></font></center>\n",
    "\n",
    "A simple increase in data for training can make a large difference. If we grow from 100 samples to 1,000 samples, acc = 0.880, and loss = 0.339.\n",
    "\n",
    "The regularization allows us to create much larger models without fear of overfitting (or memorization).\n",
    "\n",
    "We can test this by increasing the number of neurons per layer.\n",
    "\n",
    "Going with 128 or 256 neurons per layer helps with the training accuracy but not that much with the validation accuracy (128 neurons: acc = 0.890, loss = 0.291; 256 neurons: acc = 0.893, loss = 0.331)\n",
    "\n",
    "Raising this number again to 512 did improve validation accuracy and loss as well (acc = 0.913, loss = 0.279)\n",
    "\n",
    "<center><img src='./image/14-3.png' style='width: 70%'/><font color='gray'><i>Training with regularization and more training data (tuned).</i></font></center>\n",
    "\n",
    "In this case, we see that the accuracies and losses for in-sample and out-of-sample data are almost identical. From here, we could add either more layers and neurons or both. Feel free to tinker with this to try to improve it.\n",
    "\n",
    "Next, we’re going to cover another regularization method: <code>dropout</code>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11f57bf6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nnfs",
   "language": "python",
   "name": "nnfs"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}