demo02-autodiff.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMEBuT39IoyidztaKE/cWOL"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Backpropagation\n",
        "\n",
        "In this demo, we will implement backpropagation from scratch. We will follow the colab notebook that accompanies Boaz Barak's excellent [Yet another backpropagation tutorial](https://windowsontheory.org/2020/11/03/yet-another-backpropagation-tutorial/).\n",
        "\n",
        "I find the following graphic (from Barak) useful in understanding backpropagation. \n",
        "\n",
        "<img src=\"https://windowsontheory.files.wordpress.com/2021/12/backprop.png?w=1024\" width=\"700\">\n",
        "\n",
        "The key idea is that the multivariate chain rule lets us compute $dz/du$ using only information about $u$'s children in the feedforward network."
      ],
      "metadata": {
        "id": "5NnHTCg2D8cI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Building the `Value` class for backpropagation\n",
        "\n",
        "We'll start by building a class to easily store gradients and recursively calculate them.\n",
        "\n",
        "Once we've implemented it, our value function should behave as follows.\n",
        "\n",
        "```\n",
        "a = Value(4)\n",
        "print(a.grad) # 0\n",
        "def f(x): return (x+2)**3 + x**2\n",
        "f(a).backward()\n",
        "print(a.grad) # 116 = 3*(x+2)**2 + 2x when x = 4\n",
        "```\n",
        "\n",
        "In particular, we want to create a `Value` instance from a scalar which will initially have a gradient of 0. Then, we want to apply a function (the network and then loss) so that, when we call backward, each `Value` instance will compute its gradient with respect to the final value of the function. \n",
        "\n",
        "Let's start by initializing. We will need to store the data, a gradient, a method for calculating the partial derivative, and the children in the feedforward network."
      ],
      "metadata": {
        "id": "z-wp9vmnHaI4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class Value:\n",
        "  def __init__(self, data, _children=()):\n",
        "    self.data = data\n",
        "    self.grad = 0\n",
        "    self._backward = lambda: None\n",
        "    self._prev = set(_children)  "
      ],
      "metadata": {
        "id": "lW9QBYE7EAlb"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We'll also need a method for backpropagating the gradient recursively. Note that we can keep the process efficient by maintaining a set of neurons which have already been visited."
      ],
      "metadata": {
        "id": "TNT6db0rr02V"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def backward(self, visited=None):\n",
        "  if visited is None:\n",
        "    visited = set([self])\n",
        "    self.grad = 1\n",
        "  self._backward()\n",
        "  for child in self._prev:\n",
        "    if not child in visited:\n",
        "      visited.add(child)\n",
        "      child.backward(visited)\n",
        "\n",
        "Value.backward = backward"
      ],
      "metadata": {
        "id": "7jC4rID5E4px"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we'll need to implement the backward methods for several common operations."
      ],
      "metadata": {
        "id": "QOVeE9lnsAGc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def __add__(self, other):\n",
        "    other = other if isinstance(other, Value) else Value(other)\n",
        "    out = Value(self.data + other.data, (self, other))\n",
        "\n",
        "    def _backward():\n",
        "        # out = other + self\n",
        "        # d out / d self = d self / d self = 1\n",
        "        # d L / d self = d L / d out * d out / d self = d L / d out = out.grad\n",
        "        self.grad += out.grad \n",
        "        other.grad += out.grad\n",
        "    out._backward = _backward\n",
        "\n",
        "    return out\n",
        "\n",
        "Value.__add__ = __add__\n",
        "\n",
        "def __mul__(self, other):\n",
        "    other = other if isinstance(other, Value) else Value(other)\n",
        "    out = Value(self.data * other.data, (self, other))\n",
        "\n",
        "    def _backward():\n",
        "        # out = other * self\n",
        "        # d out / d self = other * d self / delf = other\n",
        "        # d L / d self = d L / d out * d out / d self = out.grad * other\n",
        "        self.grad += other.data * out.grad\n",
        "        other.grad += self.data * out.grad\n",
        "    out._backward = _backward\n",
        "\n",
        "    return out\n",
        "\n",
        "Value.__mul__ = __mul__"
      ],
      "metadata": {
        "id": "Eo2K9fxbFyzG"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "With these methods in hand, we can recreate the desired behavior! Notice that we wrote the cube operation in a funny way. This is because we have only implemented multiplication (rather than raising to a power)."
      ],
      "metadata": {
        "id": "v2hWBJ4msIP9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "a = Value(4)\n",
        "print(a.grad)\n",
        "def f(x): return (x+2)*(x+2)*(x+2) + x*x\n",
        "f(a).backward()\n",
        "print(a.grad)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "64_Dn1MPGA8h",
        "outputId": "9b349334-a138-47bb-d6bb-a0a8832af3b1"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0\n",
            "116\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Because we *will* want to raise things to powers and do a few more fancy operations, we'll use the following code to make a more comprehensive implementation of the `Value` class."
      ],
      "metadata": {
        "id": "oB4Ey4KTsUst"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### More comprehensive implementation of `Value`"
      ],
      "metadata": {
        "id": "ZyPQAlSPGoag"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import random\n",
        "\n",
        "class Value:\n",
        "    \"\"\" stores a single scalar value and its gradient \"\"\"\n",
        "\n",
        "    def __init__(self, data = None, _children=(), _op=''):\n",
        "        if data is None:\n",
        "          data = random.uniform(-1,1)\n",
        "        self.data = data\n",
        "        self.grad = 0\n",
        "        # internal variables used for autograd graph construction\n",
        "        self._backward = lambda: None\n",
        "        self._prev = set(_children)\n",
        "        self._op = _op # the op that produced this node, for graphviz / debugging / etc\n",
        "    \n",
        "    def backward(self):\n",
        "        # topological order all of the children in the graph\n",
        "        topo = []\n",
        "        visited = set()\n",
        "        def build_topo(v):\n",
        "            if v not in visited:\n",
        "                visited.add(v)\n",
        "                for child in v._prev:\n",
        "                    build_topo(child)\n",
        "                topo.append(v)\n",
        "        build_topo(self)\n",
        "\n",
        "        # go one variable at a time and apply the chain rule to get its gradient\n",
        "        self.grad = 1\n",
        "        for v in reversed(topo):\n",
        "            v._backward()\n",
        "\n",
        "\n",
        "    # Arithmetic operations \n",
        "\n",
        "    def __add__(self, other):\n",
        "        other = other if isinstance(other, Value) else Value(other)\n",
        "        out = Value(self.data + other.data, (self, other), '+')\n",
        "\n",
        "        def _backward():\n",
        "            self.grad += out.grad\n",
        "            other.grad += out.grad\n",
        "        out._backward = _backward\n",
        "\n",
        "        return out\n",
        "\n",
        "    def __mul__(self, other):\n",
        "        other = other if isinstance(other, Value) else Value(other)\n",
        "        out = Value(self.data * other.data, (self, other), '*')\n",
        "\n",
        "        def _backward():\n",
        "            self.grad += other.data * out.grad\n",
        "            other.grad += self.data * out.grad\n",
        "        out._backward = _backward\n",
        "\n",
        "        return out\n",
        "# In first reading you can ignore all code below here - if you understand everything above then you undestand the main concepts        \n",
        "##############################################################################################################################\n",
        "\n",
        "    def __pow__(self, other):\n",
        "        assert isinstance(other, (int, float)), \"only supporting int/float powers for now\"\n",
        "        out = Value(self.data**other, (self,), f'**{other}')\n",
        "\n",
        "        def _backward():\n",
        "            self.grad += (other * self.data**(other-1)) * out.grad\n",
        "        out._backward = _backward\n",
        "\n",
        "        return out\n",
        "\n",
        "    def relu(self):\n",
        "        out = Value(0 if self.data < 0 else self.data, (self,), 'ReLU')\n",
        "\n",
        "        def _backward():\n",
        "            self.grad += (out.data > 0) * out.grad\n",
        "        out._backward = _backward\n",
        "\n",
        "        return out\n",
        "\n",
        "    # Other operations implemented in terms of prior ones \n",
        "    def __float__(self): return float(self.data)\n",
        "    def __neg__(self): return self * -1\n",
        "    def __radd__(self, other): return self + other\n",
        "    def __sub__(self, other):  return self + (-other)\n",
        "    def __rsub__(self, other): return other + (-self)\n",
        "    def __rmul__(self, other): return self * other\n",
        "    def __truediv__(self, other): return self * other**-1\n",
        "    def __rtruediv__(self, other): return other * self**-1\n",
        "    def __repr__(self): return f\"Value(data={self.data}, grad={self.grad})\""
      ],
      "metadata": {
        "id": "ORwgIbh2FLkp"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Backpropagation for Linear Regression\n",
        "\n",
        "Let's apply our new machinery to our trusty linear regression model. Remember this isn't necessary because we can *exactly* solve the linear regression problem (assuming X^T X is full rank). However, the exact solution can be rather inefficient.\n",
        "\n",
        "We'll implement linear regression for a single dimensional problem. That is, we have labelled data $(x,y)$ where $x,y \\in \\mathbb{R}$. Then linear regression is equivalent to finding the line of best fit."
      ],
      "metadata": {
        "id": "aDNibzsoHWWd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class Linear:\n",
        "  def __init__(self):\n",
        "    self.a,self.b = Value(0),Value(0)\n",
        "  def __call__(self,x):\n",
        "    return self.a*x+self.b\n",
        "  def zero_grad(self):\n",
        "    self.a.grad, self.b.grad = 0,0\n",
        "\n",
        "def loss(y,y_): # We'll use the standard l_2 loss\n",
        "  return (y-y_)**2 # Is it okay that we're not multiplying by 1/2?"
      ],
      "metadata": {
        "id": "N_EDRKDAG7SY"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's make some data to fit."
      ],
      "metadata": {
        "id": "-rn4Z4lpuh9N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "n = 20\n",
        "X = [random.random() for i in range(20)]\n",
        "Y = [5*x + 2 + 0.5*random.random() for x in X]\n",
        "plt.scatter(X, Y)\n",
        "plt.xlim(0,1)\n",
        "plt.ylim(0,10)\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        },
        "id": "D9cWrd1_IPVk",
        "outputId": "bdd2d613-03bd-4799-948f-a60adffa8e82"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAPiUlEQVR4nO3df4zk9V3H8eebu8MuFLsoJykLeJjQQ4Q/DjeKXlIRUBAbINgYSFBaiZfUWGvbnDniH1SN4QzaqEljXQutPyqtRbKeop4NByE2QNxjS/mVs0iR3kJlq100crYHvP1jZtnjens78/1+5zszn30+ks3N7Hxnvu98bvc1n/18P5/PRGYiSSrPCcMuQJI0GAa8JBXKgJekQhnwklQoA16SCmXAS1Kh1gz4iLgzIl6KiCeO+N53RcTnI+LL3X9PHWyZkqR+9dKD/xRw5VHf2wXcl5nnAvd170uSRkj0stApIrYAf5eZF3TvHwAuycwXI+LtwAOZuXWQhUqS+rOx4vNOz8wXu7e/Bpy+2oERsQPYAXDyySf/4HnnnVfxlJK0Pu3fv//rmbm53+dVDfg3ZGZGxKp/BmTmDDADMD09nXNzc3VPKUnrSkT8e5XnVZ1F8x/doRm6/75U8XUkSQNSNeD3ADd1b98E/E0z5UiSmtLLNMm7gIeArRFxMCJuBnYDPxERXwYu796XJI2QNcfgM/OGVR66rOFaJEkNciWrJBXKgJekQhnwklQoA16SCmXAS1KhDHhJKpQBL0mFMuAlqVAGvCQVyoCXpEIZ8JJUKANekgplwEtSoQx4SSqUAS9JhTLgJalQBrwkFcqAl6RCGfCSVCgDXpIKZcBLUqEMeEkqlAEvSYUy4CWpUAa8JBXKgJekQhnwklQoA16SCmXAS1KhDHhJKpQBL0mFMuAlqVAGvCQVyoCXpEIZ8JJUqFoBHxEfjIgnI+KJiLgrIt7SVGGSpHoqB3xETAG/Akxn5gXABuD6pgqTJNVTd4hmIzARERuBk4AX6pckSWpC5YDPzAXgd4HngReBlzPzn44+LiJ2RMRcRMwtLi5Wr1SS1Jc6QzSnAtcA5wBnACdHxI1HH5eZM5k5nZnTmzdvrl6pJKkvdYZoLge+kpmLmXkYuAf40WbKkiTVVSfgnwcujoiTIiKAy4CnmylLklRXnTH4R4C7gUeBx7uvNdNQXZKkmjbWeXJm3grc2lAtkqQGuZJVkgplwEtSoQx4SSqUAS9JhTLgJalQBrwkFarWNElJUjWz8wv8xt8+yTdeOQzA5MQmPnL1D3DttqnGzmHAS1LLZucX2Hn3Yxx+Ld/43tKhw+z83GMAjYW8QzSS1LLb9x54U7gvO/x6cvveA42dx4CXpJa9sHSo0mP9MuAlqWVnTE5UeqxfBrwktWznFVvZtCG+7fubTgh2XrG1sfN4kVWSWrZ8EdVZNJJUoGu3TTUa5sfiEI0kFcqAl6RCGfCSVCgDXpIKZcBLUqEMeEkqlNMkJRVhdn6B2/ce4IWlQ5wxOcHOK7YOfBriqDPgJY292fkFbrnncQ4dfg2AhaVD3HLP40BzOzOOI4doJI292/ceeCPclx06/FqjOzOOI3vwksbeajswVt2ZsZThHgNe0libnV/ghAhey2/fX73KzowlDfc4RCNpbC2H8bHCfWLThko7M5Y03GPASxpbxwpjgA0R3HbdhZV63E0P9wyTAS9pbK0Wuq9nVh5OWW1Yp8kP4miLAS9pbA0ijHdesZWJTRve9L2qwz3DZsBLGluDCONrt01x23UXMjU5QQBTkxOVh3uGzVk0ksbWcug2PaWxjQ/jaIMBL2mslRLGg+AQjSQVyoCXpELVGqKJiEngE8AFQAK/kJkPNVGYpOErZcn+elV3DP4PgH/MzHdHxInASQ3UJGkEDGLJvm8Y7ao8RBMRbwPeCdwBkJnfysylpgqTNFxNL9lffsNYWDpEsvKGMTu/0EC1OpY6Y/DnAIvAJyNiPiI+EREnH31QROyIiLmImFtcXKxxOkltanrJfkl7vIyLOgG/EbgI+KPM3Ab8L7Dr6IMycyYzpzNzevPmzTVOJ6lNTa8SLWmPl3FRJ+APAgcz85Hu/bvpBL6kAjS9SrSkPV7GReWAz8yvAV+NiOX/7cuApxqpStLQNb1kv6Q9XsZF3Vk07wc+3Z1B8yzw3volSRoVTa4SHdS2AlpdrYDPzC8C0w3VIqlBozgl0W0F2uVeNFKBSvrYOVXnVgVSgZySKDDgpSI5JVFgwEtFWm3qYQLbd+9z9eg6YcBLBTrWlMRlbhGwfhjwUoGOnMN+LI7Hrw8GvFSoa7dN8YVdlxKrPO54fPkMeKlwbhGwfhnwUgWz8wts372Pc3bdO/IXLd0iYP1yoZPUp0EuIhrE6lO3CFi/DHipT8dbRFQnNAf5xuEWAeuTQzRSnwa1iMjVp2qaAS/1aVAXLV19qqYZ8FKfBnXR0tkuapoBL/Wp6Q/CWOZsFzXNi6xSBYO4aOlsFzXNgJdGiLNd1CSHaCSpUAa8JBXKgJekQhnwklQoA16SCmXAS1KhDHhJKpQBL0mFMuAlqVAGvCQVyoCXpEK5F43GwiA+yk4qnQGvkTfIj7KTSmbAa6Ca6HkP6jNQpdIZ8BqYpnrefpSdVI0XWdWz2fkFtu/exzm77mX77n3Mzi8c9/imPkTaj7KTqjHg1ZPl3vjC0iGSld748UK+qZ63H2UnVWPAqydVeuNN9bwH9RmoUulqj8FHxAZgDljIzHfVL0mjqEpvfOcVW980Bg/Ve95+lJ3UvyZ68B8Anm7gdTTCqvTG7XlLw1WrBx8RZwI/Dfw28KFGKtJIqtobt+ctDU/dIZrfB34NOGW1AyJiB7AD4Oyzz655Og3Lcki7mlQaH5UDPiLeBbyUmfsj4pLVjsvMGWAGYHp6OqueT8Nnb1waL3XG4LcDV0fEc8BngEsj4i8aqUqSVFvlgM/MWzLzzMzcAlwP7MvMGxurTJJUi/PgJalQjexFk5kPAA808Vo6NrfLldQvNxsbA26XK6kKh2jGQFObdklaXwz4MeB2uZKqMODHgNvlSqrCgB8DbpcrqQovso4BtwmQVIUBPybcJkBSvxyikaRC2YNviQuVJLXNgG+BC5UkDYNDNC1woZKkYTDgW+BCJUnDYMC3wIVKkobBgG+BC5UkDYMXWVvgQiVJw2DAH2GQUxldqCSpbQZ8V79TGZ3XLmnUOQbf1c9UxuU3g4WlQyQrbwaz8wstVStJazPgu/qZyui8dknjwIDv6mcqo/PaJY0DA76rn6mMzmuXNA4M+K5rt01x23UXMjU5QQBTkxPcdt2Fx7xw6rx2SePAWTRH6HUqo/PaJY0DA74i57VLGnWtBvzSK4fZvntfI71e56FL0vG1GvALS4d4tTvTpM6e6O6vLklra/Ui6+uZb7pfde6489AlaW1Dn0VTZe6489AlaW1DD/gqc8edhy5Ja2s14E+IeNP9qnPHnYcuSWtr9SLr1OQEp09O1J754jx0SVpb5FEXPgdpeno65+bmWjufJJUgIvZn5nS/zxv6GLwkaTAMeEkqVOWAj4izIuL+iHgqIp6MiA80WZgkqZ46F1lfBT6cmY9GxCnA/oj4fGY+1VBtkqQaKvfgM/PFzHy0e/t/gKcBp7FI0ohoZAw+IrYA24BHjvHYjoiYi4i5xcXFJk4nSepB7YCPiLcCfw38amb+99GPZ+ZMZk5n5vTmzZvrnk6S1KNaAR8Rm+iE+6cz855mSpIkNaHOLJoA7gCezsyPNleSJKkJdXrw24GfAy6NiC92v65qqC5JUk2Vp0lm5j8DseaBkqShcCWrJBXKgJekQhnwklQoA16SCmXAS1KhDHhJKpQBL0mFMuAlqVAGvCQVyoCXpEIZ8JJUKANekgplwEtSoQx4SSqUAS9JhTLgJalQBrwkFcqAl6RCGfCSVCgDXpIKZcBLUqEMeEkqlAEvSYUy4CWpUAa8JBXKgJekQhnwklQoA16SCmXAS1KhDHhJKpQBL0mFMuAlqVAGvCQVyoCXpEIZ8JJUqFoBHxFXRsSBiHgmInY1VZQkqb7KAR8RG4CPAT8FnA/cEBHnN1WYJKmeOj34HwKeycxnM/NbwGeAa5opS5JU18Yaz50CvnrE/YPADx99UETsAHZ0734zIp6occ6SnAZ8fdhFjAjbYoVtscK2WLG1ypPqBHxPMnMGmAGIiLnMnB70OceBbbHCtlhhW6ywLVZExFyV59UZolkAzjri/pnd70mSRkCdgP8X4NyIOCciTgSuB/Y0U5Ykqa7KQzSZ+WpE/DKwF9gA3JmZT67xtJmq5yuQbbHCtlhhW6ywLVZUaovIzKYLkSSNAFeySlKhDHhJKtRAAn6tLQwi4jsi4rPdxx+JiC2DqGPYemiHD0XEUxHxpYi4LyK+dxh1tqHXbS0i4mciIiOi2OlxvbRFRPxs92fjyYj4y7ZrbEsPvyNnR8T9ETHf/T25ahh1tiEi7oyIl1ZbKxQdf9htqy9FxEVrvmhmNvpF54LrvwHfB5wIPAacf9QxvwR8vHv7euCzTdcx7K8e2+HHgZO6t99XYjv02hbd404BHgQeBqaHXfcQfy7OBeaBU7v3v2fYdQ+xLWaA93Vvnw88N+y6B9ge7wQuAp5Y5fGrgH8AArgYeGSt1xxED76XLQyuAf60e/tu4LKIiAHUMkxrtkNm3p+Zr3TvPkxnLUGJet3W4reA3wH+r83iWtZLW/wi8LHM/AZAZr7Uco1t6aUtEvjO7u23AS+0WF+rMvNB4L+Oc8g1wJ9lx8PAZES8/XivOYiAP9YWBlOrHZOZrwIvA989gFqGqZd2ONLNdN6dS7RmW3T/3DwrM+9ts7Ah6OXn4h3AOyLiCxHxcERc2Vp17eqlLT4C3BgRB4G/B97fTmkjqd9MGfxWBVpbRNwITAM/NuxahiEiTgA+CrxnyKWMio10hmkuofNX3YMRcWFmLg21quG4AfhUZv5eRPwI8OcRcUFmvj7swsbBIHrwvWxh8MYxEbGRzp9e/zmAWoapp60cIuJy4NeBqzPzmy3V1ra12uIU4ALggYh4js744p5CL7T28nNxENiTmYcz8yvAv9IJ/NL00hY3A38FkJkPAW+hswnZetT39jCDCPhetjDYA9zUvf1uYF92ryIUZM12iIhtwB/TCfdSx1lhjbbIzJcz87TM3JKZW+hcj7g6MyttsDTievn9mKXTeyciTqMzZPNsm0W2pJe2eB64DCAivp9OwC+2WuXo2AP8fHc2zcXAy5n54vGe0PgQTa6yhUFE/CYwl5l7gDvo/Kn1DJ2LCtc3Xcew9dgOtwNvBT7Xvcb8fGZePbSiB6THtlgXemyLvcBPRsRTwGvAzsws7S/cXtviw8CfRMQH6VxwfU+BnUEAIuIuOm/sp3WvOdwKbALIzI/TuQZxFfAM8Arw3jVfs9C2kqR1z5WsklQoA16SCmXAS1KhDHhJKpQBL0mFMuAlqVAGvCQV6v8BqEcDBNlH4YAAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We'll also use the below helper function for visualizing how we're doing."
      ],
      "metadata": {
        "id": "5UZ0Z5GzukXO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def plot_line(X, Y, model, pause_time=1):\n",
        "  clear_output()\n",
        "  x_values_plotting = [0,1]\n",
        "  y_values_plotting = [model.a * x + model.b for x in x_values_plotting]\n",
        "  plt.plot(x_values_plotting, y_values_plotting, color='red')\n",
        "  plt.scatter(X, Y)\n",
        "  plt.xlim(0,1)\n",
        "  plt.ylim(0,10)\n",
        "  plt.show()\n",
        "  time.sleep(pause_time)  "
      ],
      "metadata": {
        "id": "Jr14Iz-VLxSB"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, we'll train our linear model using gradient descent and $\\ell_2$ loss."
      ],
      "metadata": {
        "id": "FBGeugVxunpj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import time\n",
        "from IPython.display import clear_output\n",
        "\n",
        "model = Linear()\n",
        "learning_rate = 0.03\n",
        "num_epochs = 3\n",
        "for epoch in range(num_epochs): \n",
        "  for (x,y) in zip(X,Y):\n",
        "    model.zero_grad()\n",
        "    loss = (model(x)-y)**2\n",
        "    loss.backward()\n",
        "    model.a -= learning_rate * model.a.grad\n",
        "    model.b -= learning_rate * model.b.grad\n",
        "    plot_line(X, Y, model)\n",
        "  "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        },
        "id": "R95XV98wJXNt",
        "outputId": "0c5167ae-6707-4934-d5f1-6bf6a8be42b6"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVgklEQVR4nO3dfYxcV3nH8d+Ttd1s7CSb4JfEG78n2cT2xh66aoD0BWLaQBIci7cGKSQGhENSQgrUVaKqgraqSOUWQSUk6gKlLZRS0sh1X4iLSKIURCzWrO3E65gYOzheO/EavInBxl7bT/+4M57Z9a5n5t47986c/X6klWd27sw9ulr/9uw55znX3F0AgPBckHcDAACNQcADQKAIeAAIFAEPAIEi4AEgUAQ8AASqasCb2VfM7JCZPVfxvcvN7Dtm9kLx38sa20wAQL1q6cF/VdLbRn3vIUnfdfdrJH23+BwA0ESslkInM5sv6b/cfWnx+S5Jb3b3g2Z2paSn3L2rkQ0FANRnUsz3zXL3g8XHL0uaNd6BZrZG0hpJmjp16q9fd911MU8JABPTli1bDrv7jHrfFzfgz3J3N7Nx/wxw9/WS1ktST0+P9/b2Jj0lAEwoZvbTOO+Lu4rmleLQjIr/Hor5OQCABokb8Bsl3VN8fI+k/0inOQCAtNSyTPIbkn4gqcvM9pvZhyQ9Iul3zewFSW8tPgcANJGqY/Du/r5xXlqRclsAACmikhUAAkXAA0CgCHgACBQBDwCBIuABIFAEPAAEioAHgEAR8AAQKAIeAAJFwANAoAh4AAgUAQ8AgSLgASBQBDwABIqAB4BAEfAAECgCHgACRcADQKAIeAAIFAEPAIEi4AEgUAQ8AASKgAeAQBHwABAoAh4AAkXAA0CgCHgACBQBDwCBIuABIFAEPAAEioAHgEAR8AAQKAIeAAJFwANAoAh4AAhUooA3s4+b2Q4ze87MvmFmF6bVMABAMrED3sw6JX1MUo+7L5XUJunOtBoGAEgm6RDNJEntZjZJ0kWSDiRvEgAgDbED3t0HJP21pH2SDkp61d3/d/RxZrbGzHrNrHdwcDB+SwEAdUkyRHOZpDskLZA0W9JUM7tr9HHuvt7de9y9Z8aMGfFbCgCoS5IhmrdK2uvug+4+LOkxSW9Kp1kAgKSSBPw+SW8ws4vMzCStkLQznWYBAJJKMga/WdKjkn4k6dniZ61PqV0AgIQmJXmzu39K0qdSagsAIEVUsgJAoAh4AAgUAQ8AgSLgASBQBDwABIqAB4BAJVomCQCIZ0PfgP7sP3foyLFhSVJH+2R9euUSrSp0pnYOAh4AMrahb0BrH92m4dN+9ntDx4e19lvbJCm1kGeIBgAytm7TrhHhXjJ8xrVu067UzkPAA0DGDgwdj/VavQh4AMjY7I72WK/Vi4AHgIytvaVLk9vsnO9PvsC09pau1M7DJCsAZKw0icoqGgAI0KpCZ6phPhaGaAAgUAQ8AASKgAeAQBHwABAoAh4AAkXAA0CgWCYJIAgb+ga0btMuHRg6rtkd7Vp7S1fDlyE2OwIeQMvb0Deghx97VseHT0uSBoaO6+HHnpWU3s6MrYghGgAtb92mXWfDveT48OlUd2ZsRfTgAbS88XZgjLszYyjDPQQ8gJa2oW9AF5jptJ+7v3qcnRlDGu5hiAZAyyqF8Vjh3j65LdbOjCEN9xDwAFrWWGEsSW1m+sw7u2P1uNMe7skTAQ+gZY0XumfcYw+njDesk+aNOLJCwANoWY0I47W3dKl9ctuI78Ud7ollaEh66inpc5+TVq+Wli+P/VFMsgJoWWtv6RoxISolD+NSz7/hq2jcpZdekrZujb76+qJ/X3yxfMwVV0QBv21brFOYjzE50Sg9PT3e29ub2fkAhK8lljQOD0u7dpVDvPT1859Hr5tJ114bhXnl1xVXFF+2Le7eU+9p6cEDaGlZ3BmpLkePStu3j+yVP/ecdOJE9PqFF0rd3dK73iUVClGQd3dL06al3hQCHgDicJdefnlkkG/dKu3eHb0mSa97XRTiDzwQBXmhEPXUJ2UTvQQ8AFRz+rT0wgvnjpcfOlQ+ZuHCKMTf//5yz7yzMxp+yUmigDezDklfkrRUkkv6oLv/II2GAchfS4xvp+3YsWhIpbJXvn179H1JmjxZWrJEuu228lj5smXSpZfm2+4xJO3Bf17S4+7+bjObIumiFNoEoAk0omS/6X5hHD48Msj7+qLJ0DNnotcvvTQK8A9/uNwrv/56acqU/Npch9gBb2aXSvptSaslyd1PSjqZTrMA5O18JftxQjnXPV7OnJH27j13iGVgoHzMnDlRgL/nPeWe+fz5uQ6xJJWkB79A0qCkfzCzZZK2SHrQ3X9ZeZCZrZG0RpLmzp2b4HQAspR2yX7avzDGdeKE1N8/sme+bZv02mvR621tUS/8LW8pT3wuWxZNiAYmScBPkvR6SQ+4+2Yz+7ykhyT9aeVB7r5e0nopWgef4HwAMjS7o10DY4R53CrRhuzxcuRIFN6VPfP+funUqej1qVOj8L7rrvIQy5IlUnvrbTsQR5KA3y9pv7tvLj5/VFHAAwhA2lWiiX5hVFZ9VvbMR1d9FgojJz+vvlq6YOLuyBI74N39ZTN7ycy63H2XpBWS+tNrGoA8pV2yX/MvjOFh6fnnR1Z8jlX1eeON0r33lodYilWfKEu0VYGZLVe0THKKpD2SPuDuR8Y7nq0KgIlt9Cqah3+zU7fb4epVn6XhlQZWfTazXLYqcPetkuo+KYDGa6olicWqz1UHt2nVma3SnmKgP7y7fEzOVZ8h4soBAcp1SWI9VZ/33FPumedc9RkiAh4IUGZLEmup+ly6tCWqPkNEwAMBasiSxMHBkZOegVV9hoiABwI03pJEl3TTI0+cfzy+suqzsmc+uuqzUChXfRYK0rx5DLE0GQIeCNBYSxJLRozHL54u7dhx7pLEo0ejgydQ1WeICHggQJVr2Es9+Ut+9QstPrRHi1/ZqyWHfqIlX3pROrzv3KrPu+8uj5cvXRotVURLIuCB0BSrPlft69OqE1u16bHHtfjQXs159ZWzh7wy7XL1z1yga9b+fnm8fNGiCV31GSICHmhlo6s+S2PmR4r1hma6bvpV2nrltfr68rerf+YC9c9aqMNTL1NnR7u+/9DN+bYfDUXAAzHkUkRUutdn5cTn6KrPG24YOfHZ3a2+Hw+luqcMWgcBD9SpkUVEG/oGtO7x53Vq/4B+65cD+uC0V7X40J4o1HePUfX5sY+Vx8vHqfpcVZgqKb09ZdA6Eu1FUy/2okEIbnrkiTGXIMYa8qio+vzx40/r0P9tVtfLezTj2NDZQ3551TxNvbGnHOSFgjR7NksSJ5Bc9qIBJqLYRURVqj4XtE3Syenz9OSiHu2YtUj9Mxfo+ZkLdMms6YyVIxYCHqhTTfuaV1Z9lgJ9rKrPNWvO9syXfG2vTrZNPudzf5Gk+hQTGgEP1KmyiMj8jOYMvaLlP3tRf9BxVLr9s+dWfc6dW77XZ2lJ4hhVnzO+/bNU76AEEPBArU6ckHbs0KqtW9W9+3v6xTM/1MIDP9HFJ4uhXKr6vPnmkRtr1Vj1mfYdlAACHhhL6V6flePlFff6XDRtWhTet32wPPG5ZEmiqs+076AEEPCY2Er3+qwM8r4+6ac/LR9z5ZVRiN9+e7ln3qCqz1WFTgIdqSHgMXFUVn1WBnpF1ae6uqQ3vlG6775ymM+alW+7gZgIeITp6NFoiKVyh8Txqj5LE5/d3dGGW0AgCHi0Nnfp4MFzlyRWVn1On15z1ScQEn7C0TpKVZ+Vwyuj7/W5aFH5Xp+lnjlVn5igCHg0p2PHpGefHdkz375dOl5ckli612flxOeyZdIll+TbbqCJEPDIX7Wqz46OKMDvvbe8JPG667jXJ1AFAY/slO71OXpJ4oED5WNKVZ/vfW+5Z869PoFYCHg0RrHqc0SQb9s28l6fixdLK1aMvNfn5Zfn224gIAQ8kqtS9alS1efdd5cnPhNWfQKojoBH7dylffvOHS8fXfVZKJQnPwsFaeFC7vUJ5ICAx9jqqfq8//7yKhaqPoGmQcDj3KrPvr6o6vPkyej1UtVn5cRnxlWfudwDFWhxBPxEUln1WdkrH6vq88EHy+Pl11yTa9VnI++BCoSMgA/VWFWffX3RmvOSUtXn6tXlnnnKVZ9p9LzXbdo1Yo90STo+fFrrNu0i4IHzIOBDUK3qc8qUqOrzHe8oT3zecEPDqz7T6nnHvgcqMMER8K2mVPVZ2TMfq+rzIx8p98pTqvqstzeeVs+7pnugAjgHAd+szpyR9uwZuanWWFWfhUJ58rNQiL7XgKrPOL3xtHre3MoOiIeAbwaVVZ+lnvl4VZ+lic+Mqz7j9MbT6nlzKzsgnsQBb2ZtknolDbj77cmbFLgjR0b2yseq+ixtd1saYmmCqs84vfE0e97cyg6oXxo9+Acl7ZTEPq2Vaqn6nD27fK/PUs+8Sas+4/TG6XkD+UoU8GZ2laTbJP2lpE+k0qJWVKr6HH0jitFVn296U7nqc/lyaebMfNtdh7i9cXreQH6S9uA/J+mPJV083gFmtkbSGkmaO3duwtM1gddei5Ygjlf12d4+suqzUIiWKLb4vT7pjQOtx9w93hvNbpd0q7vfb2ZvlvRH1cbge3p6vLe3N9b5MldP1WdpeKV0r8+2tvzaDSA4ZrbF3XvqfV+SHvxNklaa2a2SLpR0iZl9zd3vSvCZ+ail6vPqq0dWfRYK0c6J3IgCQJOKHfDu/rCkhyWpogff/OFeWfVZCvTxqj5LPfMMqj4BIG1hr4MfHDx34rNa1ef110c3dAaAFpdKwLv7U5KeSuOzYhld9VkK9cqqz3nzyvf6LPXMG1T12QhslwugXq3Xgy9VfVb2zJus6jNtbJcLII7mDvjRVZ99fdLOneNXfRYKUbgHdq9PtssFEEdzBHxl1Wdlz3ysqs+VK8vj5U1a9Zk2tssFEEf2AT88HPXCR+/HElDVZ9rYLhdAHNkG/M6d0bDKWFWfpfHyAKo+08Z2uQDiyDbg29qkj36Uqs86sU0AgDhib1UQR0ttVQAATSLuVgXhz1ACwATVHKtoJgAKlQBkjYDPAIVKAPLAEE0GzleoBACNQsBngEIlAHkg4DMwXkEShUoAGomAz8DaW7rUPnnken8KlQA0GpOsGaBQCUAeCPgKjVzKuKrQSaADyBQBX1TvUkbWtQNodozBF9WzlLH0y2Bg6Lhc5V8GG/oGMmotAFRHwBfVs5SRde0AWgEBX1TPUkbWtQNoBQR8UT1LGVnXDqAVEPBFqwqd+sw7u9XZ0S6T1NnRrs+8s3vMiVPWtQNoBayiqVDrUkbWtQNoBQR8TKxrB9DsMg34oWPDuumRJ1Lp9bIOHQDOL9OAHxg6rlPFlSZJ9kRnf3UAqC7TSdYzo+7/GnftOOvQAaC63FfRxFk7zjp0AKgu94CPs3acdegAUF2mAX+B2YjncdeOsw4dAKrLdJK1s6NdszraE698YR06AFRnPmris5F6enq8t7c3s/MBQAjMbIu799T7vtzH4AEAjUHAA0CgYge8mc0xsyfNrN/MdpjZg2k2DACQTJJJ1lOSPunuPzKziyVtMbPvuHt/Sm0DACQQuwfv7gfd/UfFx0cl7ZTEMhYAaBKpjMGb2XxJBUmbx3htjZn1mlnv4OBgGqcDANQgccCb2TRJ/y7pD939tdGvu/t6d+9x954ZM2YkPR0AoEaJAt7MJisK96+7+2PpNAkAkIYkq2hM0pcl7XT3z6bXJABAGpL04G+S9H5JN5vZ1uLXrSm1CwCQUOxlku7+PUlW9UAAQC6oZAWAQBHwABAoAh4AAkXAA0CgCHgACBQBDwCBIuABIFAEPAAEioAHgEAR8AAQKAIeAAJFwANAoAh4AAgUAQ8AgSLgASBQBDwABIqAB4BAEfAAECgCHgACRcADQKAIeAAIFAEPAIEi4AEgUAQ8AASKgAeAQBHwABAoAh4AAkXAA0CgCHgACBQBDwCBIuABIFAEPAAEioAHgEAR8AAQKAIeAAKVKODN7G1mtsvMdpvZQ2k1CgCQXOyAN7M2SV+Q9HZJiyW9z8wWp9UwAEAySXrwvyFpt7vvcfeTkv5V0h3pNAsAkNSkBO/tlPRSxfP9km4cfZCZrZG0pvj0hJk9l+CcIZku6XDejWgSXIsyrkUZ16KsK86bkgR8Tdx9vaT1kmRmve7e0+hztgKuRRnXooxrUca1KDOz3jjvSzJEMyBpTsXzq4rfAwA0gSQB/0NJ15jZAjObIulOSRvTaRYAIKnYQzTufsrMPippk6Q2SV9x9x1V3rY+7vkCxLUo41qUcS3KuBZlsa6FuXvaDQEANAEqWQEgUAQ8AASqIQFfbQsDM/s1M/tm8fXNZja/Ee3IWw3X4RNm1m9m283su2Y2L492ZqHWbS3M7F1m5mYW7PK4Wq6Fmb23+LOxw8z+Jes2ZqWG/yNzzexJM+sr/j+5NY92ZsHMvmJmh8arFbLI3xav1XYze33VD3X3VL8UTbj+RNJCSVMkbZO0eNQx90v6YvHxnZK+mXY78v6q8Tq8RdJFxcf3hXgdar0WxeMulvS0pGck9eTd7hx/Lq6R1CfpsuLzmXm3O8drsV7SfcXHiyW9mHe7G3g9flvS6yU9N87rt0r6tiST9AZJm6t9ZiN68LVsYXCHpH8sPn5U0gozswa0JU9Vr4O7P+nux4pPn1FUSxCiWre1+AtJfyXpV1k2LmO1XIsPS/qCux+RJHc/lHEbs1LLtXBJlxQfXyrpQIbty5S7Py3p5+c55A5J/+SRZyR1mNmV5/vMRgT8WFsYdI53jLufkvSqpNc1oC15quU6VPqQot/OIap6LYp/bs5x9//OsmE5qOXn4lpJ15rZ983sGTN7W2aty1Yt1+LTku4ys/2S/kfSA9k0rSnVmymN36oA1ZnZXZJ6JP1O3m3Jg5ldIOmzklbn3JRmMUnRMM2bFf1V97SZdbv7UK6tysf7JH3V3f/GzN4o6Z/NbKm7n8m7Ya2gET34WrYwOHuMmU1S9KfXzxrQljzVtJWDmb1V0p9IWunuJzJqW9aqXYuLJS2V9JSZvahofHFjoBOttfxc7Je00d2H3X2vpB8rCvzQ1HItPiTp3yTJ3X8g6UJFm5BNRHVvD9OIgK9lC4ONku4pPn63pCe8OIsQkKrXwcwKkv5OUbiHOs4qVbkW7v6qu0939/nuPl/RfMRKd4+1wVKTq+X/xwZFvXeZ2XRFQzZ7smxkRmq5FvskrZAkM7teUcAPZtrK5rFR0t3F1TRvkPSqux883xtSH6LxcbYwMLM/l9Tr7hslfVnRn1q7FU0q3Jl2O/JW43VYJ2mapG8V55j3ufvK3BrdIDVeiwmhxmuxSdLvmVm/pNOS1rp7aH/h1notPinp783s44omXFcH2BmUJJnZNxT9Yp9enHP4lKTJkuTuX1Q0B3GrpN2Sjkn6QNXPDPRaAcCERyUrAASKgAeAQBHwABAoAh4AAkXAA0CgCHgACBQBDwCB+n/fLOnjBx4/YAAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### More general datasets\n",
        "\n",
        "Linear regression works well for data that fits a line. But some data doesn't behave so nicely.\n",
        "\n",
        "Next, we'll load labelled pairs $(x,y)$ where $x \\in \\mathbb{R}^2$ and $y \\in \\{\\pm 1\\}$. Clearly, a line won't work for us any more. Instead, we'll need a more rich model."
      ],
      "metadata": {
        "id": "IMuAseYekWDH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.datasets import make_moons, make_blobs\n",
        "X, Y = make_moons(n_samples=100, noise=0.1)\n",
        "Y = 2*Y - 1 # go from {0,1} to {-1, 1}\n",
        "\n",
        "plt.scatter(X[:,0],X[:,1], c=Y, cmap='jet')\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        },
        "id": "YutZA-VBj34q",
        "outputId": "759a3dbb-0420-47ac-f3a4-99872b0a12ab"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD8CAYAAAB6paOMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUVfrA8e+Zlpl0SoBQQu8dQhNEmiK9WKiCLlVFVH6WXRbLWnF1BZWiKCigNBGRJlUElA4G6YReAiRAIGWSqef3R0JMmAkmZDKTcj7Pkwfm1ncgM++9557zHiGlRFEURSm+NL4OQFEURfEtlQgURVGKOZUIFEVRijmVCBRFUYo5lQgURVGKOZUIFEVRijmPJAIhxBwhRKwQ4lA264cIIf4UQhwUQmwXQjTOtO5s+vIoIcReT8SjKIqi5Jyn7gi+AR6+y/ozwANSyobA28CsO9Z3lFI2kVJGeigeRVEUJYd0njiIlHKrEKLKXdZvz/RyJ1DRE+dVFEVR8s4jiSCXRgA/Z3otgfVCCAl8IaW8827BRenSpWWVKlXyKTxFUZSiad++fdeklGF3LvdqIhBCdCQtEbTLtLidlPKSEKIMsEEIcUxKudXNvqOB0QARERHs3aseJyiKouSGEOKcu+Ve6zUkhGgEfAX0kVJev71cSnkp/c9Y4Eegpbv9pZSzpJSRUsrIsDCXhKYoiqLcI68kAiFEBLAMeEJKeSLT8gAhRNDtvwMPAW57HimKoij5wyNNQ0KIhUAHoLQQ4iLwBqAHkFJ+DrwOlAJmCCEA7Ok9hMoCP6Yv0wELpJRrPRGToiiKkjOe6jU06G/WjwRGull+GmjsuoeiKIriLWpksVLoXbyYwNat57h6NcnXoShKoeSL7qOK4hGpqXaGDPmBNWui8fPTkZpq54knGvP55z3QatU1jqLklPq0KIXWhAnr+Pnnk6SmOrh1y4LF4mDBgoP897/b/35nRVEyqESgFEoOh5NvvokiJcWeZbnZbOPTT3f5KCpFKZxUIlAKJZvNicXicLsuIcHi5WgUpXBTzwgUn4qLS2b58mPYbE569qxFRERIjvYzGnXUrVuaw4fjXNa1bVvJ02EqSpGm7ggUn/n++8NEREzlxRfX8dJL66ldexoffvh7jvefObMH/v56tFoBgF6vISjIwJQpXfMrZEUpkoSU0tcx5FpkZKRUtYYKt+vXzVSqNMWljd9k0rF79ygaNCiTo+McORLHRx9t59ChWFq0KM/LL7elSpXQ/AhZUQo9IcQ+d+X+VdOQ4hMrVhxHoxEuy63WtJ4/773XOUfHqVcvjDlz+ng6PEUpVlQiULzq7NmbXL9uJiXFhrubUYdDYrM5vR+YohRjKhEoXnH1ahJ9+y7mwIEr6PVaHA4nVqvd7bZxcclIKUmvQaUoSj5TD4sVr+jZcyF798aQkmInIcFCcrINjUaDu+/6pUuPsHTpEe8HqSjFlEoESr47duwaR47EYbdnbfKxWh1ur/qTk21MnaoGhSmKt6hEoOS7q1eT0Otz96t240ZKPkVzd1arg4QEC4WxN52i3CuVCAoAKSVRUVdYteoEMTGJvg7H45o2DcdqdR0F7OenxWDQuiw3GLT07VvbG6FlSE628uSTywkOfp9Spf5L7drT+PXXs16NQVF8RSUCH4uLS6Z581m0azeHIUOWUb36J4wbt6ZIXZEGB/vxxhsd8PfXZywzGDTodBpCQ41otSKjK6nJpKNcuUBeeuk+r8b42GPfs3jxISwWB3a7k+joG/TosYCjR11HLme2Z88lhg37kS5d5jF16k6SkqxeilhRPEcNKPOxLl3msWXLuSzt5wEBej755GFGjGjmw8g875tvonj99c3ExiZjszkRIq27KIBWK6hTpzTDhzdm9OjmhIQYvRbXmTPx1Ks3g9TUrL2YtFrB8OGNmT3b/TiFuXOjeOaZNaSm2nE6JSaTjgoVgtm3bzTBwX7eCF1RciW7AWXqjsCH4uKS+e238y4PUZOTi14FzePHr/H882u5ciUJi8WB0ykzkgCkJYTY2GQmTGjj1SQAcObMTfz8XJuoHA7J0aPX3O6TkmJj3LifMZttOJ0yfZmdixcTmDFjT77GqyiephKBDyUlWbOdQOXmzaJVQfPllzeQmGi562CxpCSrT56R1K8f5nI3AGnPKu67z30Bu6ioK25HRqem2vnhB9X1VSlcPJIIhBBzhBCxQohD2awXQohPhRAnhRB/CiGaZVo3XAgRnf4z3BPxFBaVK4cSEuLahKDXa+jdu5YPIso/W7acczuSODOnUxIa6t27AYCyZQN58skmWZ5hCJH2vOLFF1u73Sc01OhyJ3db6dIB+RKnouQXT90RfAM8fJf13YCa6T+jgZkAQoiSwBtAK6Al8IYQooSHYirwNBrBnDl9slTQNJl0hIUFMGlSex9H51klStz9C95o1NK/f12Cgjzbtu50StauPcnEiZuYNm0316+b3W43Y0YP3nqrIxERIQQH+9GnT2127x5FhQrBbrevWzeM6tVLZPy/3RYQoGf8+JYefQ+Kkt889rBYCFEFWCWlbOBm3RfAr1LKhemvjwMdbv9IKce42y47RelhMcDRo3F8+ukuTp2Kp3PnqowZE+mTK+P8NGXKTiZN+gWz2ZZluV6vQasVdOtWk/nz+xEQYPDYOS0WO126zCcq6gpJSVZMJh06nYZ164bSpk3e5yw4f/4WDz00n0uXEtFqBRaLnX//u32RS+JK0eHr6qMVgAuZXl9MX5bdchdCiNGk3U0QERGRP1H6SN26Ycyc2dPjx01KsvL113+wYcNpIiJC6NixKiaTjhYtyhMW5t3mi+efb8XJk9eZMycKPz8tFouDjh2r8O9/t6d69RKUKxfo8XPOmLGHfftiMkpd3/7z8ceXcv78C3muZRQREcLRo8+yf/9l4uLMtGxZgZIlTXmOW1G8rdAUnZNSzgJmQdodgY/DKfBu3EghMnIWV68mZ1yFT5++B39/PU6nZMKE1rzzTievFXbTaATTp/fgzTc7cPToNapUCc3xbGT3au7cAy7zHQDEx6dw5Egc9evnbM6DuxFC0Lx5+TwfR1F8yVuJ4BKQ+V68YvqyS6Q1D2Ve/quXYirS/vvf34mJSXSZ1/d2Uvjkk100axbOI4/U82pcYWEBXrsbya5HlpTZr1OU4shbn4YVwLD03kOtgVtSysvAOuAhIUSJ9IfED6UvU/Loxx+PZTu5O6SNVfjkk3sfq3B7DERBLokxcmTTLD2BbitXLpDatUv5ICJFKZg8ckcghFhI2pV9aSHERdJ6AukBpJSfA2uA7sBJwAw8lb7uhhDibeD2CJy3pJQ3PBFTcRcc/PcPXe+lsJvD4eTZZ9cwd+6BjLb+nj1r8u23/fHzK1gtjaNGNWfNmmh++eUsNpsDPz8der2GZcseV3MdKEomqsREETV//gGefno1yck2t+v9/LS8/PJ9vP12p4xlBw9eZfLk3zh0KJbmzcvzz3+2o1atrFfOkyf/xttvb8Fs/qvt3WTSMWJEUz77rHv+vJk8kFKye/clfvvtPOXKBdKvX123dwmKUhxk12tIJYIiSkrJ+PE/89VX+wGRZeSsyaSjbNkA9u8fQ4kSab1ctmw5S/fuCzLq5mi1ApNJz9atT9K0aXjGvuXL/4/Ll5Nczmcy6UhKmuh2tK2iKAWDqjVUzAgh+Oyz7kRHj+e77/rzzTd96N69BmXKBBAeHsjIkc3Q6f7673/22TVZ6uY4HJKkJCsvvpj1kc3Nm6luz2exONyWmlYUpeBTiaCIq1gxmP7962KxOPj113PExSVz+vRN3n13G82azSIhwYLN5uDIEffllnfuvJjldXa1d+rUKY3RWLCeESiKkjMqERQDyclpV/Zmsy2j3k9Kip1LlxL4/PO96HSabNvN7xzhPGVKV4KCDBl3E1qtwN9fz8yZPfL1PXhSfHwKv/xyJtvk5wu//HKGTp3mUrXqVAYNWsqxY+6rnipKflCJoBjYv/+y26kiU1LsLFt2FCEEY8Y0x2TKekVvMuno0qUa69efwmZLa/Zp2LAsBw6MZeTIpjRrFs6QIY3YvXsk7dtX9sp7yav//GcL5ct/TP/+i2nR4ksiI2cRG5vs05gWLjxEr14L2bz5LGfP3mLJkiO0aPElhw/H+jQupfhQD4uLgcOHY2nZ8iuXOj8APXvWYuXKQVitDkaM+Invvz+C0agjOdmGlDLjTsHPT8fatUMK9Sja5cuPMXTosiw9qXQ6Da1bV2Tbtqd8EpPTKQkP/59LMhICevWqxU8/DfJJXErRpB4WF2P16oVRrVqoS6VMf389L7zQCkirvT9/fn/On3+RKVO6otdrcDgkiYlWEhOtXLtmpmvXbzPuDAqjjz/e4dKd1m53sndvDBcu3PJJTLGxySQkuM49ISVs337RzR6K4nkqERQDQghWrx5CrVqlCAjQExLih9Go4z//6UDnztWybFumTAA7d150OyrZZnPyyy9n+P3389x//9eEhk6mUaOZLF9+zFtvJU+uXXNfglqv1xAf7743VH5zNx/FbeHhni/EpyjuqG4exURERAiHDz9DVNQVrl9PITKyfLalruPjUzO6kWaWNjgrhsmTt2UMKDt4MJYhQ5YxY0Z3hg9vkq/vIa969qzFqVPxLt1ctVpB3bqlfRKTyaRn2LBGzJ//Z5YCef7+eiZOvN8nMSnFj7ojKEaEEDRtGk6XLtXuOt/BI4/UJSDAtReR1epg5crjWUYVQ1ohu1de2eg2eRQkr7zSlrAw/4xurkKkfeFOm9YDvd51zmJv+fTTbjz+eH38/LQEBhoIDDTw5psdGDjQZWoPRckX6mGx4sJud/LQQ/PZvfsSyck2NBqB0ajjnXc68cYbm0lMtLrsYzBoiI192esTz+fWjRspTJ++m3XrThEREcILL7SmZUu3U2B43c2bqVy5kkSVKqFqTIaSL1SJCSVX7HYnP/xwhKVLjxASYmTUqGa0alWRevWmc/Soax/3oCAD8fGvFpvyzlJKTp+Ox+mU1KhRUhWxUwoFX89QphQyOp2GAQMaMGBA1uaJ//ynA08++VOWrqj+/nomTGhTbJLAoUOxPPLIEi5cuIUQgjJlAliy5FFatPDcnYXN5mDmzL3MmrUPm83J0KENmTChjUen8lSU29QdQQFw7txNTp68Qd26YZQvH+TrcP7Wl1/uZ+LETSQmWtDrtbz4YmvefLNDsSg4ZzbbqFRpiksJ76AgA+fOvZBRxC+vevdeyKZNZzISrtGoo27d0uzePSpLjShFyQ11R1AApabaGTz4B37++WR6bX87jz1Wnzlz+hToD/uoUc0YMaIpN2+mEhzsV6Bj9bQffzzqtriew+Fk4cJDPPNMizyfY+/emCxJANJ+V6Kjb7BixXH696+b53MoSmbF5xNcAL388np+/vkkqal2bt2ykJrqYOnSI7z77lZfh/a3NBpByZKmYpUEAC5fTsJicZ0H2WxOq93kCTt2XMDhcLosT0qysm3bOY+cQ1EyK16f4gJESsns2X9kmScA0ur/TJ++J5u9FF+7775KGAyuXU0DAw20axfhkXOULx/k9hwmk46IiBCPnENRMlOJIAcSEy28++5WmjT5nHbt5rBo0SHy+mzF4ZAuSeA2dyUHlIKhTZuKtGsXkaVaq8mko2HDMnTtWiNPx7bZHJw9e5MOHapgMum5syOSVqth6NBGeTqHorjjqTmLHwY+AbTAV1LKyXesnwJ0TH/pD5SRUoamr3MAB9PXnZdS9vZETJ6SkmKjVauvOHPmZsYXd1TUFXbsuMAnn3S75+PqdBqaNCnHH39ccVnnqStLxfOEEKxcOYiZM/fy1Vf7cTgkw4c3Zvz4Vnl6WD5r1j5efXUDVqsTp1PSs2dNjh27xsmT8QgBZcsGsmjRI4SFBXjw3ShKmjz3GhJCaIETwIPARdImoh8kpTySzfbPAU2llP9If50kpcxVURVv9hqaPXs/zz+/1qVYmdGo48SJcVSqdO+36rt2XaRz53lYLHbsdoler8Fo1LF9+wgaNCiT19CVQmLFiuMMGvRDlofDJpOOIUMaMWnS/dhsTqpXL6HGKih5lp/VR1sCJ6WUp6WUVmAR0Ocu2w8CFnrgvF7x888n3U4Ar9dr2L79Qp6O3apVRf74YwwjRjSjTZuKPPNMCw4efFolgWIkKcnK88+vdSkRnpJiZ+7cKHbtukSlSsEqCSj5yhNNQxWAzN+IF4FW7jYUQlQGqgK/ZFpsFELsBezAZCnl8mz2HQ2MBoiI8F7TSaVKIeh0Arvd9c6pXLm8V4esWbMUn3/eM8/HUQqfU6du0KbNbOLi3FdFtdmcjBixgrFjNaxf/wSRkYV3LgilYPP2w+KBwFIpZeaO2JXTb1UGA1OFENXd7SilnCWljJRSRoaFhXkjVgDGjm2OwZA1X97uOnn//YVjVi6lYPrHP37i+vWUu26TlGQlPj6V7t2/w2537VKqKJ7giURwCcg8o3nF9GXuDOSOZiEp5aX0P08DvwJNPRCTx9SuXZpFix6hZEkTQUEG/P311K8fxubNw702kvb06XhGjVpJgwYz6N9/MXv2ZPfPqxQWqal2fv/9Qo4rtqam2vntt/P5HJVSXHmiaWgPUFMIUZW0BDCQtKv7LIQQdYASwI5My0oAZimlRQhRGmgL/NcDMXlUr161uXr1JQ4evEpgoIGaNUt57dzHjl2jZcsvSUmxYbdLjhyJY926Uyxe/Cg9e9byWhyKZwlBert/zhKBEIKUFNdnVYriCXm+I5BS2oFxwDrgKLBESnlYCPGWECJzV9CBwCKZtZtSXWCvEOIAsJm0ZwRuexv5mk6noWnTcK8mAYBXX91IUpI14xmFlGn1bp55ZnWexzIovuPnp6Nz56ou04fqdMLtaG273aGaIpV845FxBFLKNcCaO5a9fsfrN93stx1o6IkYiqpt287h7vv+6tVkrl9PoXRpf+8HpeSKlJLt2y9w5UoSrVtXpEKFYABmz+5N27ZzuHEjhdRUO35+OmrUKEFIiJF9+2JISrKh0wn0ei0zZ/YkMFBVHlXyhyo6V8CVLu3vdj5dIVBfDIXAuXM36dx5HlevJqPRCKxWO2PHRvLxx12pUCGY6OjnWL06mpMnb9CkSTk6daqKlJKVK0+wYsVxSpY08Y9/NKVePe91kFCKH1WGuoD78st9vPjiuixjGYxGHYMHN2T27AI1CFtxo2nTLzh48CoOx1+fs4AAPbNn93aZ60FR8lt+DihT8tHIkc0YP74VRqOO4GA/jEYdPXvWZNq0ey9voXjH6dPxHD9+LUsSAEhOtvHpp7t9FJWiuFJNQwWcEIL33uvMq6+2JTr6BhUrBntkIJuS/xITLdmW6VaFBZWCRCWCQiIkxKhGlhYy9euXQa93LSdtNGp57LF6PohIUdxTTUNKjjmsVg4vWcKWt9/m6LJlOGyqX/vd6HQa5szpjb+/PqObqL+/nkqVQnj+ebdVWBTFJ9QdQT6TUvLTT8eZNWsfly8n0bZtJSZNal/omncSL19mduvWpMTHY01KwhAYSECZMozYsYMAL5b8KGz69KnD3r2jmDFjL+fP3+Lhh6szbFhjNQm9UqCoXkP5bMyYVcybF0Vq6l/llTQawbRp3Xj66bzPb+sti/r148SqVUj7X5PpaPR66j/+OP2//daHkSmKklOq15APHDkS55IEAJxOyQsvrOPQoVgfRZY7Ukqi70gCAE6bjaPLlvkoKkVRPEUlgny0adNpl66Dt9lsDubOjfJyRIqiKK5UIshHJUqYsq1QKiUkJFi9HNG9EUJQs2dPhO6Octx6PXX79/dRVIqieIpKBPmob986brsPQlrvkf7963g5onvXY8YMgitUwBAUBEJgCAwkJCKCrlOm+Do0JRvx8Sm89tovNGw4g/btv+aHH46oQoWKW6rXUD4KDDSwfv1QunSZh9n8V/u60ajj4Yer89BDbufgKZCCwsN57sQJjv30E9ePHyesfn1q9eyJVq/3dWiKGwkJFpo3n0VMTCIWS9ozqv37L7Nv32Xee6+zj6NTChrVa8gL7HYnc+b8wapVJwgO9mPgwAZ0717TaxPbFCZxR49ycu1a/IKCqNu/P6aSJX0dUqH08cc7mDTpF1JSsj7gNxp1nDv3AmXKBPgoMsWXsus1pBKBUiBIKVn34ovsmzUL6XSi0elASh5ftowaXbv6OrxC58EH57Fx4xmX5cHBfixY0J8ePdSkRsWR6j6qFGhnNm1i/1dfYU9JwWGxYEtOxmY28/1jj2FLufu8voqrSpVC3N5xOhzOQjeYUcl/KhHkQUKChaefXkVIyGQCAt5jwIClxMQk+jqsQilq7lxsyckuy4UQnNm0yQcRFW63K9ZmptUKKlcOpVmzcB9FpRRUKhHcIyklnTrN5euvo0hIsGA22/jhhyO0bPklZrOqwZNbzjsGq90mAafD4Xadkr0mTcrxzTd9CA01EhRkwGTS0axZOOvXD02fK1lR/uKRRCCEeFgIcVwIcVII8U83658UQsQJIaLSf0ZmWjdcCBGd/jPcE/F4w5Yt5zh+/HpGjwwAh0Ny82YqixYd8mFkhVOjoUPRB7g+wJR2O9U6q14u9+Kxx+oTG/sSW7c+xeHDz7B796iMaTIV77l14QLrXnyR2W3asGLkSK4dO+brkFzkOREIIbTAdKAbUA8YJIRwV2N3sZSySfrPV+n7lgTeAFoBLYE3hBAl8hqTNxw+HIvd7nRZnpxsY//+yz6IqHCr2b07dfv1S0sGQqA1GNCZTPSdOxdDoGrTvld6vZYmTcpRtWqh+FgVOdeOHWNmgwbsnj6dizt3EvXNN8xq3pxz27b5OrQsPDGOoCVwUkp5GkAIsQjoAxzJwb5dgQ1Syhvp+24AHgYWeiCufFWrVin0eg2pd0wnHBCgp359VY0zt4QQ9J03j8idO4levRq/4GAaDBpESKVKvg5NUe7Z2hdewJKQkPFaOhzYzGZWjx3LM4cP+zCyrDyRCCoAFzK9vkjaFf6dHhFCtAdOAC9KKS9ks28FD8SU7zp3rkaFCsGcOnUDmy3tzkCjEZhMeoYMaeTj6AonIQSV2rShUps2vg5FUTzi9MaNbpdfO3aMnZ98Qvzp04Q3a0b9xx9HbzJ5Obq/eOth8UqgipSyEbABmJvbAwghRgsh9goh9sbFxXk8wNzSaATbtj1Fnz510Ok0aLWCjh2rsHPnCIKD/XwdnqIoPpYYE4PMpqODdDrZNHEiuz/9lDXjxjG9Th2Srl71coR/8UQiuARkvn+vmL4sg5TyupTy9iStXwHNc7pvpmPMklJGSikjwwrIRCilS/vz/fePYbFMwmKZxMaNw6heXY2EBbBbLGx+/XX+Fx7OByVKsGzoUBJjYnwdlqJ4za0LF9D6ZX9RaDebAbAlJZEYE8OGl1/2VmguPJEI9gA1hRBVhRAGYCCwIvMGQojMHZd7A0fT/74OeEgIUSL9IfFD6csKFY1GoNWqnriZLe7bl+0ffUTSlSuk3rzJocWLmdW8eZb2UkUpykrXyXlRSafdzrHly/MxmrvL87eXlNIOjCPtC/wosERKeVgI8ZYQonf6ZuOFEIeFEAeA8cCT6fveAN4mLZnsAd66/eBYKbyuHjzIua1bsWcaESztdiwJCUTNzXWroJJHMTGJvPzyBtq1m8OoUSs5etT3TavFgTEkhNYvvOC2W7Q7Gp3vaoB65MxSyjXAmjuWvZ7p7/8C/pXNvnOAOZ6IQykYrkRFITSu1xg2s5mLO3bQ6rnnfBBV8XTy5A1atEgb5Gi1Oti58yILFx5k1arBdOhQxdfhFXmd33+fkjVqsP3DDzFfu0blBx4g9eZNzm/blmUQpdbPj0ZPPOGzOFUZasXjSlZ3X15bZzQSVs/dEBMlv7z66gYSEiw4nWnFJR0OSXKyjdGjV3L8+Dg1yjifCSFoNnIkzUZmjKEl6epVvm7XjqSrV3HabGh0OkrXrUvnd9/1WZwqEbhhszl4//3f+OKLfaSk2OjRoyaTJ3dRozJzqGKbNpSoVo24o0dx2v4qt6E1GLJ8IJT8t27dqYwkkNnZsze5dctCaKjRB1EVb4Fly/LssWOcWr+eGydPUrZRIyq3b+/TpKzKULvRr98i1q07lVHLXasVlC7tz7Fj49QHJ4fM16+zYuRIolevBikp07AhvWfPJrxpU1+HVmwcPRpH/fozcPcR1+k01KhRkvj4FDp1qso773SiWjU1+rioU/MR5NCxY9do1uwLlwk9/P11vP12JyZMuPtgJ6vVgdXqIDDQkC/xFTb21FScdrsqE+EDQ4cuY8GCg24TgVYLt7u4azSCkBA//vzzaSpWVHe998KWksKN6GgCypQhsFw5X4eTLTUfQQ5FRV1Bp3P9ZzGb7WzffsHNHmmSkqwMG/YjQUHvU6LEB9SvP+Ou2xcXOqNRJQEf2b//stskAH8lAQCnU5KUZOWjj7Z7J7AiZtenn/JRmTLMadeOqVWqsKBnTyyJhascvUoEd6hWrYTbNlU/Py1165bOdr8+fRaxZMlhrFYHdruTI0fieOih+Zw6pXrDKr7RoEGZHE+HarM52br1XD5HVPScWL2aTRMnYk1KwpqYiMNi4fTGjSwbOtTXoeWKSgR3aNGifEZBucz0ei1jx7rcUQFw/Pg1duy4kKUkNaQ1E02duivfYlWUu5k48X6XyWmMRp3b5CAE1KihRsXn1u8ffOAyoZLDYuHUunUkF4BSODmlEsEdhBBs3DiMnj1rYTBo0ek0NGpUll9+GZbRaygpycpnn+2iV68FjBu3hs2bz2IwaF2OZbM5OXw41ttvQVGAtMlp1qwZTP36YQgBgYEGnn++FQ8+WA0/v6y/ryaTnldeaeujSAuvxEtuK+Kg1esxF6JEoLqPulGypIllywaQkpI2CCck5K+eQvHxKTRvPourV5Mxm21otQKDQet2bgI/Py333afKKCu+88ADVTh06BnsdidarUAIQVKSlREjVrB8+TG0WkFoqJHPP+9JZGR5X4db6FTt0oWbc+Yg75hhT2g0lKxRw2PnSbpyhf1ffcW1Y8eo1LYtjZ94wqPP3lSvoVz61782MmXKTpdmID8/LRoNpKSkLb/dE+PIkWfVZKsnBCYAACAASURBVOFKgZSYaCEhwUJ4eFCOnyUoWd06f57PmzTBmpiYMVJY7+/Pw1On0mzUKI+c4/L+/XzToQNOmw17air6gACMoaGM3rs31z2UVK8hD/nxx2MuSQBAr9cwenQkFSoEERRkoE+f2uzZM0olAaXACgryo0KFYJUE8iAkIoKxBw7QdORIStWuTdXOnRmwfDnNRo0iOS6OQ4sXc2LVKuwWy98fLBvLn3wSa2Ii9vRZsGzJySRfvcov//63p96GahrKrZAQ92Vl7XbJ//1fG6ZOfdjLESmK4kshlSrRc+bMLMt2Tp3Kxn/+E63BAEIgNBqGrFnjMumSPTUVjU6XbcG51Js33c5x7LTbObZiBb3d7HMv1B1BLo0f34qAAH2WZVqtoHHjslSqFOKjqBRFKShi9u7ll3//G4fFgjUxEWtCApabN1nQvXvGncGVqCi+bNGC9wIDeS8ggB+GDCH11i2XY2n0epdlt+nuMtdBbqlEkEuDBzdkxIimGI1agoP9CAw0ULNmKX744XFfh1asXP3zTw4vWUJsAZr3VVEA9s+endGMk5l0Ojm9YQOJMTF83b49MXv3Ih0OHFYrR5cu5buHXVsTDAEBVHvwQZeEoDOZaD56tMdiVk1DuSSE4JNPuvHKK23ZvfsS5csH0bJlBVXF0UusSUks6NGDmL17EVot0uGg0n33MXDFCp/O+aoot1kTEpBO116EUkqsycnsmTkTh9WaZZ3DauXqwYNc3r+f8GbNsqzr+/XXzO3YkVvnzyOlBCmp0rEj7f75T4/FrBLBPapQIZh+/VRdFm9bN2ECF3ftwpHp4dv5335j08SJPDxlig8jU5Q09R57jGM//eQy0Mxps1Gtc2cOLVqU5ff3NqHRcOPkSZdEEFCmDE8fOsT5bdu4efYs5Zo0oWyjRh6NWTUNKYWGlJI/5893+RDZU1OJmqPmNlIKhlq9elG5ffuMmcmERoPOZKLL5Mn4ly5NxVat0Lm5e3Xa7ZRp2NDtMYUQVG7fnsbDhnk8CYC6I1AKEyldbqlvs7lpk1UUX9BotQxauZITK1dyZOlS/IKDaTpiBOWbNweg2ahR7Pjf/3BYLBlNSDqjkaqdOhFWt65PYlaJQCk0hEZDpXbtOL9tG1nKagpB1Y4dfReYotxBo9VSp29f6vTt67LOv1QpRu3Zw/qXXuLk2rXo/f1pNmoUD7z+upsjeYdHRhYLIR4GPgG0wFdSysl3rJ8AjATsQBzwDynlufR1DuBg+qbnpZR/2zXWlyOLFd+KPXyYOW3b4rBYsKemojMa0ZlMjNy1i1I1a/o6PEUp0LIbWZznOwIhhBaYDjwIXAT2CCFWSCmPZNrsDyBSSmkWQjwN/BcYkL4uRUrZJK9x5ITZbMPplLmaNCY11Y7TKfH3z74/r+I9ZerXZ9yxY+z94guuREVRPjKS5qNHExAW5uvQFKXQ8kTTUEvgpJTyNIAQYhHQB8hIBFLKzZm23wl4tVj3xYsJPPnkcrZsSau3HhlZnrlz+1KrVqls97l8OZERI1awceNppISWLcszZ04fatfOfk6C4sLpcCDSR0v6QmC5cnR44w2fnFtRiiJPfJIrAJmn4rqYviw7I4CfM702CiH2CiF2CiFcG9TSCSFGp2+3Ny4X5V1tNgdt287h11/PYrc7sdud7Np1kfvum01iovv6Hw6Hk7Zt57BhwylstrR9duy4SNu2c0hIuPeaIYXd9RMnmNuxI+8YDLxjMrFs6FBS4uN9HZaiKHnk1Us6IcRQIBL4MNPiyultVoOBqUKI6u72lVLOklJGSikjw3LRDLBmTTTx8Sk4HH89C5Eyrcln0aJDbvdZu/Yk166Zsdtd9/nuuz9zfO6iJOXGDb5q3ZqzW7YgnU6cVitHvv+eeZ07Uxgr2BZ3UVFXmD//ADt3XlT/f4pHmoYuAZmL7ldMX5aFEKIL8G/gASllxmW1lPJS+p+nhRC/Ak2BUx6IC4BTp+LdVgtNTrYRHe1+GslTp+KxWt3vc/z4dU+FVqj88fXXacPmM31pOKxWbkRHc2H7diLaqklNCoOUFBu9ei1kx46LaDQCKSV16pRm48ZhhIYa//4ASpHkiTuCPUBNIURVIYQBGAisyLyBEKIp8AXQW0oZm2l5CSGEX/rfSwNtyfRswROaNi3ndvawwEADzZuHu92nceOy6PXu9ymuk3fEHjyIPSXFZbmUkuvHj/sgIuVevP76Zn7//QJms42kJCvJyTYOHozlmWdW+zo0xYfynAiklHZgHLAOOAoskVIeFkK8JYS43RX0QyAQ+F4IESWEuJ0o6gJ7hRAHgM3A5Dt6G+VZhw5VqFevdJap+QwGDeHhgfTr537wRvv2lalfPyzLPnq9hrAwfx59tJ4nwys0wps3R+/v73ZdmQYNvByNcq/mzIkiNTXrbFpWq4OlS49w7VpyNnspRV2xmKEsKcnKm2/+yvz5f2Kx2GnePJx33ulEmzbZTyOZnGzltdc2M3/+n9jtTvr3r8vkyZ0JCwvwxFsodFJv3eKzWrVIuXYtYzSk1s+P8pGRPLVtW0bRPafDwZGlS/lz/nyERkPTESOo3bu3KsrnZbt3X2L+/D9xOp0MGNCA+++PQAhBYOB7JCfb3O6j12u4775KfPdd/4z5uZWiJbtxBMUiEQBcu2amU6e5nDlzEyHAbnfStWsNlix51G0zkOLq1vnzrH3hBU6uXYvWYKDxE0/QefJkDOk1VaSULOnfn1MbNmQU3NIHBNBg4EB6f/WVL0MvVl577Rc+/ngnqal2pEwbA/Pkk02YNq07jzyymOXLj+N0uv/ca7WCypVDiY5+Ts1cVgQV+0TQs+cC1q9P6w56m8mkY9Kk9kyceL+nQyyWzm7ZwoIePVyqLt4e+Vs2m4JaiudER1+ncePPSUnJ2vzj769ny5YnKVMmgMjIWSQnWzGb7W6PERRk4IcfHufBB9124FMKsWI9Z3FSktUlCQCkpNj54ot9Poqq6Dm1fr1LEgCQDgenN2zwQUTFz5o10W6v9lNTbaxceZyIiBBOnHiOd97pRNWqoW6P4XBIzp1znS1LKbqKRSKwWh1k10RtNruvZqnknqlECbRups/T6PUYS5TwQUTFj9GoQ6t1/VhrtRqMxrTe4qGhRl58sQ2TJrV3mXb1thYtimfvuOKqWCSCkiVN1KjhWk5Cp9PQq1dtH0RUNDUcPNht2QkhBHX79/dBRMVP//513Q4Q02o1DByYtXfXoEENKFs2EIPhr/8zk0lHx45VaNy4XH6HqhQgxSIRAHz9dR8CAw0ZXUL9/fWUKRPAu+928nFkRUdQ+fI8nl5//faPsUQJBq9ejTEkxNfhFQthYQF8+21//P11BAUZCAw0YDTqmDmzO1WrZr0rMxp1DBr0V3LQagW9etXmxx8H3HlYpYgrNg+LAS5dSuCLL/Zx7Ng12rWL4MknmxAc7NqUoeSN3WLh/G+/odFqqdS2LVq9qtzqbbdupbJmTTQOh6RbtxqUKuU6BuStt7bwwQe/Yzb/1Z3U31/PwoWP0Lu3+zvly5cTefvtraxZE02JEiYmTGjN0KGNVPfgQqLY9xpSFOUvdruTkiU/IDHR9RlZ48ZliYoa67L8+nUz9evP4Pr1FOz2tI4XAQF6nn66BR9++GC+x6zkXbHuNaQoSlYJCRa3NbgAzpy56Xb5tGm7uXUrNSMJQFr9rWnTdhMXp0YlF2YqEeRRVNQVFiw4yL59Mb4ORVFyLDTUmO0ETfXru6/uu2nTGVJTXZOHn5+WqKgrHo1P8a5iO2dxbGwyS5ceITHRQrduNWnUqGyu9jebbfTo8R27d8eg0QicTknTpuX4+echBAWp5w5KwabRCN59tyP/938bsjwjMJl0TJ7cxe0+1aqV4PffL7iMU7DZnFSsqEpSFGbF8o5g1aoTVKkylZdeWs9rr22mdeuvePbZ1bmqy/7qqxvYufNSRhVHs9nG3r0xTJiwLh8jV5TcOXjwKs8+u5pHHlnC11//kaXg3NixLfjyy17UqlUSf389LVqUZ82aIbRvX9ntsV58sXXGWITb9HoNjRqVoW5dNVVoYVbsHhYnJ1spW/Yjl8JbAQF6li8fSJcu1XJ0nODg990+aDMadZjNE1UvCsXnFiw4yMiRK7BaHTgckoAAPTVqlGT79hH3PAf3ihXHGTVqJcnJVux2Jw88UJkFCx5x2ytJKXjybfL6wmbTpjNuR14mJ9uYO/cAXbpUQ0r5t1/kFov7Oi3uJrRR7s3ZLVs4MHcuDouFBoMHU7N7d5Vgcyg11c7Ysauy1By6PRnT7Nn7ee65Vvd03N69a9OjR03Onr1JSIiR0qVVAigKil3T0N3ugK5fN9OmzWx0urcJDn6fl15an+0XfufO1VyqMwoBDzxQWX1ZecCGV19lQffuRH3zDQcXLGDpgAH8OHSomlYxh/bsueT299BstrFo0eE8HVur1VC9ekmVBIqQYpcIOneulqX7220mk47Nm8+yc+dFnE5JYqKVGTP2MGTIMrfH+fTTboSGGjGZ0m6qjEYdISFGZszoka/xFwfXo6PZ/dln2MzmjKkxbcnJHPvpJy78/ruPoyscgoL8cDhcf88BQkJUZwYlq2KXCAIDDXz7bT9MJh1GoxaNRuDvr6dq1VDs9qzNOikpdlavjub8eddKjDVqlOTEiXG8+WYHHnusHq+91p4TJ8ZRp05pb72VIuvUunVZ5ka+zWY2c3zlSh9EVPg0blyW8PBAl2KLAQF6xo1r6ZuglAKr2D0jAOjXry6nTo1nyZLDJCZa6datBs8/vxa73fXLx89Py4kT14mIcK2VU6qUP6+8oiZt9zRDYCBC6zpZkFavxy9YdVPMCSEEq1cPoXPnudy6ZQHSnl89/3wrunev6ePoCj+nw8HpjRu5cfIkZRs1IqJdu0LdJOyRRCCEeBj4BNACX0kpJ9+x3g+YBzQHrgMDpJRn09f9CxgBOIDxUkqv9L8MDw/i+edbZ7xu2jSc3bsvucxZYLE4qF3btXKpkn/q9O3LmnHjXJYLrZaGgwf7IKLCqVatUpw9+wLbtp3n2jUzbdtWIjw8yNdhFXpJV6/y9f33k3TlCk6bDY1OR1j9+gzbuBFDYKCvw7sneW4aEkJogelAN6AeMEgIcecM7yOAeCllDWAK8EH6vvWAgUB94GFgRvrxvG7ChNb4+WXNiyaTjt69a1Opkqqc6U3G0FAG/PgjhsBADOlVTHUmE72//JISVav6OrxCRavV0KFDFR59tJ5KAh6ycvRobp49izUxEXtqKtakJK5ERfHLpEm+Du2e5XkcgRCiDfCmlLJr+ut/AUgp38+0zbr0bXYIIXTAFSAM+GfmbTNvd7dz5lfRuaioK4wf/zPbt18gKMjAmDGRvPVWRwwGNaexL9hSUji9cSNOu51qnTurZiHFZ6zJyVz4/Xc0ej3zH3oIaXftTWgqWZJXrl/3QXQ5l5/jCCoAFzK9vgjc2Uk5YxsppV0IcQsolb585x37VnB3EiHEaGA0QEREhAfCdtWkSTm2bn0qX46t5J7eZKJ2r16+DkMp5v6cP59VY8ei0emQUrpNAgDObJYXBoWm15CUcpaUMlJKGRkWpoazK4qS/+KOHGHlmDHYzGYsCQlYExPdbqfR6ajdp4+Xo/McTySCS0ClTK8rpi9zu01601AIaQ+Nc7KvoiiKT+yfPRuHzeZ23e35ufUBAQSGh/Pghx96MzSP8kTT0B6gphCiKmlf4gOBO7t2rACGAzuAR4FfpJRSCLECWCCE+BgoD9QEdnsgJkVRlDwzx8W5bQoSOh1lGjWiXOPGRLRtS/0BA9CbTD6I0DPyfEcgpbQD44B1wFFgiZTysBDiLSFE7/TNZgOlhBAngQn89ZD4MLAEOAKsBZ6VUqpiPUquJFy8yI/Dh/NhmTJ8Ur06O6ZMwelQv0ZK3tXu3dttl1BptxN36BDHli2jYps2hToJQDGsPqoULebr15lerx4p168j07/89f7+1Hv0UfrOnevj6JTCzmm3M7dTJy7v348t2c0sbEJQqW1b/rFt2z0dP/HyZQ7MnUvCxYtU7dyZ2r16odHl3zhfNVWlUiTt/fxzrAkJGUkA0kpRHFqyhJvnzvkwMqUo0Oh0DNu0ie7TpyPcfUFLycUdO7Cnpub62Ge3bOGzmjXZ8p//sGf6dJYPG8bs++7DlpLigchzRyUCpVA7v3Wr2w+hzmDg6oEDPohIKWq0ej1Nhg/HmM04FqHRIDS5+yqVTic/DByILTk54/fXmpRE7KFD7Jk+Pc8x55ZKBEqhVrpuXTR610lWbGYzP48fzzcdOnBi9WofRKYUNY2HDcvoKXSbRqejRrduaA3u53/OTuzhw1iTklyW21NS+PPbb/MU571QiUAptKTTSbnGjd3OUeB0OLh17hzntmxh6eOPs+Pjj30QoVKUdHznHcKbNkUfEIDOZMIQFESJatXo/eWXuT6W1mBAOt2XCb8z2XhDsaw+qhR+DquVbx9+mJg9e9K69wmRVrr6dgXITMnBZjaz+bXXaD5mDIaAAB9FrBR2hoAA/rF9Oxe2b+fqgQOUrFmTap0757pZCKBUrVoEVajAjZMns/yu6gMCiBwzxpNh54i6I1AKpX1ffsmlXbv+ur1O/zAJrdbtXAYanY64w3mbmUtRhBBEtG1Li2eeoUKLFvw2eTJz2rVj6cCBXNy1K1fHGbh8uUvXVH1AADW6d/d02H9L3REohdKBb75Jm8HsTtl0h3ZYrQSULZvPUSnFRcqNG3zRtCnJsbFpD3uF4MTKlXSfMYMmw4fn6Bh2iyVLbzeA1Ph4FvXuzajd3h1Xq+4IlELJbVc+0np46IzGLMs0ej0V27QhtHJlb4SmFAM7Pv6YpKtX/+qxJmVaB4XnnsNuseToGLumTnXp8ea02Yg7fJhYL9+9qkSgFErNRoxA7+86eXpA2bJ0mzYNQ1AQhqAgdEYjlR94gMeXLvV4DNLpdPugWin6jq9ciSObL/zYQ4dydIz4M2fcPjDW6PUkxsTkKb7cUk1DSqHU5KmniF69mlPr1+Ow2dD5+SG0WgYsW0Z4s2Y0GjqU68eP41+6NEHly3v03NeOH2fV2LGc37oVjU5Hg4ED6fbZZ2q+hGIkoLT7ucmddjumkiVzdIxqDz5IzJ49LncF9tRUwps2zXOMuaFKTCiF2qXduzm3dSsBZctSt3//HPcKsiYl4bBac/yhvc187Rqf1apF6s2bGc8jtAYDZZs0YeTOnYV63lol506sWsXSAQOyPKcSOh3lmzVjZA4fGqfcuMGMBg0wX7uGM73CqT4ggJbPPUeX99//m73vTX5OTKMoPlOhZUsqtGyZ4+3N167x01NPcXLdOhCCktWr0/ebb3J8jP2zZ6ddwWW6gHJYrcQdPsyl3bup2OrOOZmUoqhWz57c/+9/s/Xtt9H6+eG02ShVqxYDf/opx8cwlSzJmD/+4Lf33+fEqlWYSpakzYQJ1B8wIB8jd0/dESjFhpSSL5o2Je7IkYwrMABDYCDPHj1KcMWKf3uMHwYP5tDChS7LDYGBdJ8+ncbDhnk0ZqVgS715k5h9+wgsW5YyDRr4Opy/pYrOKcXepV27uHHyZJYkAGlX9Hu/+CJHxygfGYnOTclh6XQWii8CxbOMoaFU69y50P/fq0SgFBnWpCR2T5vGon79WDdhAtejo7Osjz9zxu0oUIfVyrWjR3N0jiZPPYUhICDLcbR+fpSPjCS8WbO8vQFF8RGVCJQiIeXGDWY0bMjGV1/l+PLl7J42jS+aNOHU+vUZ24Q3bZrtBOOX9uwhes2avz2PqUQJRu7eTc0ePdD6+eEXHEzzUaMYnIN9FaWgUs8IlCJhw8svs+vTT3FYrVmWB4aHM+HixYwr+MWPPMLJn3/G7qbmu9BoKN+iBd2nTaN8pEszqqIUeuoZgVKkHf3xR5ckAGC5dYsbp05lvH500SLav/aa2wqP0unk0q5dzGnfniP5MABNUQqqPCUCIURJIcQGIUR0+p8l3GzTRAixQwhxWAjxpxBiQKZ13wghzgghotJ/muQlHqX4MgQFuV3udDqzFPbS6vXc/69/uZShyMyRksLqp59W8x4XE06Hg9MbN/Lnt99muWgoTvJ6R/BPYJOUsiawKf31nczAMCllfeBhYKoQIjTT+pellE3Sf6LyGI9STLUcN86l5ITQainfrBlB4eEu2wdXqHDX49lSUog/fdqjMSoFT/zp03xSrRqL+/dn9dNPM7NBA1aMGpXtXAFFVV4TQR/g9gzhc4G+d24gpTwhpYxO/3sMEAuE5fG8ipJF06eeouHQoWiNxrQ6Q4GBlKpZk8e+/97t9g+88YbbWkW3Oe12jKGh2a5XioZF/fqRePEi1sRErElJ2FNTObRggU9mCfOlvCaCslLKy+l/vwLctc6vEKIlYAAy33+9m95kNEUI4f2peZQiQWg09PriC8YdO0bv2bMZun49zxw5km2dofqPP06XDz5wmww0ej1VOnQgIExdrxQ0V//8k12ffsqf332HNTk5T8eKP32aG9HRLlf/NrOZPTNm5OnYhc3f9hoSQmwEyrlZ9W9grpQyNNO28VJKl+cE6evCgV+B4VLKnZmWXSEtOcwCTkkp38pm/9HAaICIiIjm586du/s7U5QcsFutbHj5ZfZ98QU6oxGH1Ur55s0Z+NNPua5DpOQf6XSyfPhwji5bhtPhQKvXI7RantiwgQotWtzTMWMPHWJ2mzZu5w4Oq1+fZ3JYRbQwya7XUJ66jwohjgMdpJSXb3/RSylru9kumLQk8J6U0m13DCFEB+AlKWXPvzuv6j6qeFpKfDxXDxwgqEIFStWs6etwski6coXd06ZxcdcuyjZsSKvx4wmtUsXXYXnVoUWLWDFyJLY77gLu7B6cG06Hg/+VL485NjbLcp3RSLuJE3ngtdfyFHNBlF/dR1cAt6fjGQ64VFwSQhiAH4F5dyaB9OSBSCvZ2BcoeilYKRRMJUpQpUOHApcErkdHM71uXbZ/9BFnNm5k97RpzGzYkJhidiG078svXZIApI0mv7x//z0dU6PV0m/ePPT+/mj0eiCt+meJatVo/cILeYq3sMlr9dHJwBIhxAjgHPA4gBAiEhgrpRyZvqw9UEoI8WT6fk+m9xD6TggRBgggChibx3gUpUhZ9+KLpN66lVHt1GmzYbXZWDVmDKP37fNxdN5zZ32oDEJkO1o8J2p07crTBw+yb9Ysbl24QPUHH6TBwIF37V5cFKmRxYriA1JKdk6dyvYPP8R87RrlmjSh65QpRLRtm2W79wIC3M7NLDQaJprN6NwMjMtvUkqSY2MxBAS4TL6eX/Z/9RVrn3/e5d/CVLIkL129iiabqUuVrNTIYqXYubBjB2vGjWP1s89y/rfffB1OFr9MmsTmSZNIunwZp81GzJ49fPvQQy7NHPpsJtrR6PU++fI7uW4dUytXZmrlyvy3dGm+f/xxLAkJ+X7exsOHU/G++zISj9ZoRO/vzyOLFqkk4AHqjkApkja8+ip7pk3Dll5TSO/vT7ORI3l46lQfRwbW5GQ+DAtzrXckBLV69WJQpslNNr/xBts//DDLtlo/PxoOHkyfOXO8FTIAVw8eZHbr1lmuyrV+fkS0a8ewjRvz/fzS6eTUhg2c2bQJ/7AwGg0d6nawYLb7S0liTAxavZ6AMmXyMdKCS81QphQbcUePsvuzz7J8edqSk9k3axZNnnqKco0b+zA6SLh4EY1W67pCSq4eOJBlUftJk7h29CgnVq5E6+eHw2qlUtu2dPvsMy9F+5cdH33kMr+uw2Lhwvbt3Dh1ipLVq+foOA6bjcOLF3Pk++/TqreOGUNEu3Z/u5/QaKjRtSs1unbNdeyX9uxh2dChJJw/j5SSck2a8OiiRcWu91V2VCJQipzo1avdPkB0WK2cWLXK54kguEKFbB9whtWtm+W1Vq/nsSVLiD9zhthDhyhVsyal69TxRpgurp844bb0gtZg4Na5czlKBA6bjXmdO3N5//60XkBCcHTZMh544w3avvJKfoRNclwc8zp3xpqYmLEsZs8evr7/fp4/c0Y1LaGeEShFkM5odPvh1uh06N3MLuZthsBAmo8Z4zKqWe/vzwNvvOF2nxJVq1K7Vy+fJQGAiPbt0RoMLssdFgtlGjbM0TGO/vDDX0kAQEpsZjObX3+d5Lg4T4ab4cC8eS6JVzqdpN66lTZ3taISgVL01H3kEbfLhRDUe+wxj57rXiuUPvS//3HfK6/gFxICQlCqTh0GLF9OxdatPRqfJ7V+4QUMgYFZBm/p/f1pPnp0jstxHF22zO14AK3BwNlff3W7z42TJzm4YAFnf/31norBxZ865Xb+CafdTsKFC7k+XlGkEoFS5ASFh9N37lx0JhOGwEAMgYHojEZ6f/UVIZUqeeQcB+bN4+OKFXlbp+Oj8HD25XDO49s0Wi0d3niDV+Pjec1mY9zRo1R/8EGPxJZfgsLDGbV3L/UHDsQ/LIxStWrx0P/+R9dcPIA3lSqV7ShgY0hIltcOq5V5XbowvW5dVowcycJevfisZk1u5fLLO+L++912cxVCUKFly1wdq6hSvYaUIiv15k2if/4Z6XRSs3t3TCXclsHKtYMLF7Jy5MgsvWf0/v50nTKF5qNHe+QcRdXl/fuZ066dyxW6f+nSTEjv0QOQcOkSMxs1IvXGjSzbCa2W8pGRjNy5M8fntKWk8EnVqiTHxmYMzNMajVTt1Ikhq1fn8R0VLmocgVLsGENDaThoEI2GDPFYEgDYPGmSy8Amm9nMr9m07yt/CW/WjK4ff4zOZMIvOBhDUBABZcowdP36jCQA8NNTT7kkAQDpcHAlKorEmJgcn/PncePSHhRnuujV+fnRb968vL2ZIkQ9LleUHEiMiWHv558Te+gQ8WfOuN0m6coVnA6H+66hSobIsWNpMGgQ57dtwxAYSES7dlke7tvM5myfF0Bas1pOS1DfPHuWgwsW5pK1TwAADYhJREFUuHZ7tdn489tvaf388/f0HooalQgU5W9c/uMPvnngARxWKw6LBYRwu11wxYqFLglIp5Ob585hCAz06vwLxpAQavV0X2hYOp1Zrt5d9i1RgpI1auToPDH79qExGOCORGA3mzn7yy8qEaRTTUOK8jdWjhqFNTExLQmA2y8pvb8/nd97z8uR5c2p9euZUqkSMxs0YEqlSszt2JGkq1d9HRaGwEDCI12asYG0LsD95s1DZJOM7xRcoQK46Wmk0espkcMBcMWBSgSKche2lBSuRGUzlbYQGV8ovWfPptETT3g3uDy4fuIEi/v1IzEmBpvZjMNi4fxvvzH/oYcoCB1I+n79NaaSJdGlj/sQWi1+ISGM3LWLqp065fg4FVq1IqRyZcQd40q0ej0tnnnGozEXZqppSFHuQqPTodFqcbgZL+AfFsbLBeAK+l5smjjR5YG3024n/tQpLu/bR/lsrsi9pXSdOow/fZqD333HjehowiMjqffoo7mutiqEYNjGjfwwaBAXduxAaDT4lypFn2++yXHzUnGgEoGi3IVWr6dO//4cW7YMh9WasVxnMtF81Kh7OqbTbseSkIAxNPSeZtbKq5i9ezn2449u1wmtloSLFz2eCKSUXNy5k2PLl6P396fh4MF/OwmQMSTEI1ftgeXKMXzzZpLj4rAlJ6fdIeSwaam4UIlAUf5Gz5kzuZle60doNDjtdqp26kT7XE5l6HQ42Pzaa+z69FOcNhvG0FC6fPghTYYNy6fI3fv1zTezHaHrsFgIb97co+eTUrJy9GgOLVyIzWxGo9Xy+wcf0O2TT2h2j8n0XgSEhYEXH4gXJioRKMrfMIaGMnLnTi7v38/16GjKNmrkUhwuJzZNnJhWGju9SSY5NpY1Tz+Nf8mS2fagyQ9X//wz23V1+vXz2Ojr287++mtaEkjv8um023Ha7fw8fjx1+vXDv3Rpj55PyT31sFhRcii8WTMaDBhwT0nAbrFkSQK3+WIgWnbxa/R6es2ale1+Ukpi9u3j8Pffc/3EiRyf7/DixW5nWdPodJxcuzbHx1Hyj7ojUBQvSI2Pz7Y55ua5c16N5YE33+Tcb79hv6NERuTTT+MXFOR2n5T4eL7t2pW4I0fSHp5brdTo1o1HFy/OMiLYHa3BgBDCtTdSeq+rzG6dP8/h77/HnppKrZ49fV4yvLjI0x2BEKKkEGKDECI6/U+34/iFEA4hRFT6z4pMy6sKIXYJIU4KIRYLIVxr3CpKEeAfFpbRFfJO5Zo08Wosldq0YeCPP1I6/c7AWKIE7f75Tx7873+z3Wfl6NFcOXAAW3IyloQE7KmpnFy7lt8/+OBvz9foiSfcTgYvHQ5qduuW8frA/PlMq1OHXyZO5Nc33mB2mzasffHFe3iHSm7lqeicEP/f3v3HRl3fcRx/vugPekUE2jpEfg02BgpuwyFxsDjFambjqMuAjEpEo0Gii1vmHyNZXMxMHGqymWU/8EcXcQoKYhCMzdCJGhZhogJFGANMnG34IZDJWkh/3L33x33Ljvau19r2vle/70dyuW/vPr173aeXe/f7+X7u+9EjwCkzWylpBTDKzH6epl2TmXU5/Z+kdcBLZva8pFXAbjP7U7bn9ZPOucHo3VWreO2++7qcrG7p1q2hnQXTEolzM5dONzay59lnOXPiBF+5/nomV1aiIUNob2lh5YUXnjdrqsPwSy7hZ42NWZ/nrQcfZNtDDyX3AgoKSMTjLFy37tyxkTMnT/LbceO6nAqiqLSUJVu2MGHu3H54tW6glqqsBq4JtlcDbwJdCkGGQALmATUpv/8AkLUQODcYXbl8OaXl5bz1wAOcbmjg4pkzqVy5MtRTIXcUgYN1daxfsIBEPE68pYWdq1YxYe5cal55hXhra8ZhraajR6lfs4YZixd3OyXzu/ffz9eXLOFQXR2FsRjTqquJlZWdu/9QXV3axYTazp5l79q1XggGWF8LwWgzOxJsHwVGZ2hXImkn0A6sNLONQDnwHzPrWDqoARib6YkkLQOWAUyYMKGPsZ0Lx/SFC5nez4vj9FW8tZUNixeft6fS1tTEv7dto37NGr5x661UXHopx+vru/yuJRJsXraMUwcPZlxdrcOoSZMyfi8g0/cpJPmc/xzIeoxA0uuS9qa5VKe2s+QYU6ZxponB7kgN8JikXp/kw8yeMLNZZjbrIp8L7Fy/adi+Pe35k9qam9m9ejUA82trk4u7pPnAbmtuZtvDD9OSsiZwb331xhvTrvZWWFLC5bfc8rkf1/VM1kJgZpVmNiPN5WXgmKQxAMH18QyP0Rhcf0Ry+GgmcBIYKaljr2QckH2w0TnXr4YUFWU8v1DHrJ6xV17J3fv2UVpenrZdQVERJ/bv/9wZYqNGcfPTT1MYi1EYizGkuJjCWIzZ996b18t3flH0dWhoE7AUWBlcv9y5QTCT6IyZtUiqAOYCj5iZSdoKLACez/T7zrmBNXb2bIpiseTiLSmKhg0775u/I8aPZ/ycORzYtKnLHkS8tZXhYzOO7PbI9EWLmHj11ezbsIH2s2f52k03UTFtWp8e0/VMX2cNlQPrgAnAx8AiMzslaRaw3MzulDQHeBxIkNwDeczMaoPfn0yyCJQBHwBLzKwl2/P6rCHn+lfDjh08e8MNWCJBor0dJC6vqeH7Tz553hj9J++8w18qK887nlAwdCiTr7uOmogt+zgYZZo15GsWO+cAaG1u5p8bN3L21CkmXXstX5oxI227D9ev59V77qGtuZlEPM7U+fOZX1ub8ctoLn94IXDO9ZtEPM7phgZKRo6kZMSIsOO4Hhqo7xE45yJoSEEBIydODDuG6yd+0jnnnIs4LwTOORdxXgiccy7ivBA451zE+cFi50KWaG/nwObNHN6yhQtGj+abt9/uB2JdTnkhcC5E7S0tPDNvHsf27KG1qYmC4mL+/uijLFq/nilVVWHHcxHhQ0POheiD2lqO7tpFa1MTkDxVQ/uZM7y0ZAnxtraQ07mo8ELgXIj2PPdc2vV8E+3tHHnvvRASuSjyQuBciIrSLOEIgBkFQ4fmNoyLLC8EzoXoW3fdRdGwYV1uj5WV5XwtYxddXgicC9FlCxdyeU0NhbEYRaWlFA8fTqy8nMWbN/vKXC5n/KRzzuWBEwcO8PHbb1NaUcGUqioKfVjIDQA/6Zxzeaxi6lQqpk4NO4aLKB8acs65iPNC4JxzEeeFwDnnIs4LgXPORZwXAueci7hBOX1U0qfAxzl4qgrgRA6eZyB49twbrLnBs4chjNwTzeyizjcOykKQK5J2pptzOxh49twbrLnBs4chn3L70JBzzkWcFwLnnIs4LwTdeyLsAH3g2XNvsOYGzx6GvMntxwiccy7ifI/AOecizgtBCkkLJX0oKSEp49F8Sd+TdEDSIUkrcpkxE0llkl6TdDC4HpWhXVzSruCyKdc5U3J024eShkp6Ibh/h6Qv5z5lej3IfpukT1P6+c4wcnYm6c+Sjkvam+F+Sfpd8Lr2SLoi1xkz6UH2ayR9ltLnv8x1xnQkjZe0VdK+4LPlJ2nahN/vZuaX4AJcCkwF3gRmZWhTABwGJgPFwG7gsjzI/giwItheATycoV1THmTN2ofA3cCqYPtHwAth5+5F9tuA34edNU32q4ErgL0Z7q8C6gABVwE7ws7ci+zXAK+EnTNNrjHAFcH2cOBfad4vofe77xGkMLP9ZnYgS7PZwCEz+8jMWoHngeqBT5dVNbA62F4N3Bxilmx60oepr+dF4Drlx0ot+fr3z8rM3gZOddOkGnjGkrYDIyWNyU267vUge14ysyNm9n6w/V9gPzC2U7PQ+90LQe+NBT5J+bmBrn/YMIw2syPB9lFgdIZ2JZJ2StouKaxi0ZM+PNfGzNqBz4DynKTrXk///j8MdvNflDQ+N9H6LF/f2z31bUm7JdVJmh52mM6C4c2ZwI5Od4Xe75FbmEbS68DFae76hZm9nOs8vdFd9tQfzMwkZZoONtHMGiVNBt6QVG9mh/s7a8RtBtaaWYuku0ju2cwLOdMX3fsk39tNkqqAjcCUkDOdI+kCYAPwUzM7HXaeziJXCMysso8P0Qik/oc3LrhtwHWXXdIxSWPM7EiwW3k8w2M0BtcfSXqT5H8ouS4EPenDjjYNkgqBEcDJ3MTrVtbsZpaa8ymSx28Gg9De232V+uFqZq9K+qOkCjML/RxEkopIFoHnzOylNE1C73cfGuq9d4EpkiZJKiZ5IDO02TcpNgFLg+2lQJe9G0mjJA0NtiuAucC+nCX8v570YerrWQC8YcGRtZBlzd5pfHc+yXHhwWATcGswi+Uq4LOU4ca8JunijmNIkmaT/GwL/R+HIFMtsN/MfpOhWfj9HvZR9Xy6AD8gOT7XAhwD/hrcfgnwakq7KpJH/w+THFLKh+zlwN+Ag8DrQFlw+yzgqWB7DlBPcqZLPXBHiHm79CHwK2B+sF0CrAcOAf8AJofdx73I/mvgw6CftwLTws4c5FoLHAHagvf5HcByYHlwv4A/BK+rngwz5/I0+49T+nw7MCfszEGu7wAG7AF2BZeqfOt3/2axc85FnA8NOedcxHkhcM65iPNC4JxzEeeFwDnnIs4LgXPORZwXAuecizgvBM45F3FeCJxzLuL+Bwde+RDh1qxjAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Luckily, we have neural networks! Let's build a three layer (two hidden layer) network to solve this problem."
      ],
      "metadata": {
        "id": "uAV1u8YbvCRl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def neuron(weights, inputs, relu=True):\n",
        "  v = sum(w*x for w,x in zip(weights, inputs))\n",
        "  return v.relu() if relu else v\n",
        "\n",
        "class Net:\n",
        "  # The list comprehension is fancy but this is exactly the architecture\n",
        "  # we've been drawing in class.\n",
        "  def __init__(self, hidden_dim=16):\n",
        "    self.layer_1 = [[Value(), Value()] for i in range(hidden_dim)]\n",
        "    self.layer_2 = [[Value() for j in range(hidden_dim)] for i in range(hidden_dim)]\n",
        "    self.output = [Value() for i in range(hidden_dim)]\n",
        "    self.parameters = [v for L in [self.layer_1,self.layer_2,[self.output]] for w in L for v in w]\n",
        "  \n",
        "  def __call__(self, x):\n",
        "    layer_1_vals = [neuron(w,x) for w in self.layer_1]\n",
        "    layer_2_vals = [neuron(w, layer_1_vals) for w in self.layer_2]\n",
        "    return neuron(self.output, layer_2_vals, relu=False)\n",
        "  \n",
        "  def zero_grad(self):\n",
        "    for p in self.parameters:\n",
        "      p.grad = 0"
      ],
      "metadata": {
        "id": "JFefvQzAkaah"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We'll use the following helper function to visualize how our model is doing."
      ],
      "metadata": {
        "id": "ptPi1I5fvNPK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "def plot_prediction(X, Y, model, title=None, h=.5):\n",
        "  x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
        "  y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
        "  xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
        "                      np.arange(y_min, y_max, h))\n",
        "  Xmesh = np.c_[xx.ravel(), yy.ravel()]\n",
        "  inputs = [list(map(Value, xrow)) for xrow in Xmesh]\n",
        "  scores = list(map(model, inputs))\n",
        "  Z = np.array([s.data > 0 for s in scores])\n",
        "  Z = Z.reshape(xx.shape)\n",
        "  \n",
        "  fig = plt.figure()\n",
        "  clear_output()\n",
        "  if title != None: plt.title(title)\n",
        "  plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral, alpha=0.8)\n",
        "  plt.scatter(X[:, 0], X[:, 1], c=Y, s=40, cmap=plt.cm.Spectral)\n",
        "  plt.xlim(xx.min(), xx.max())\n",
        "  plt.ylim(yy.min(), yy.max())\n",
        "  plt.show()"
      ],
      "metadata": {
        "id": "cJ3eAVeRn8b7"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's train our architecture using gradient descent. Notice we're using a slighty non-standard loss. In particular, we want our prediction $y'$ to be close to the true value $y$. In math notation, we want $y y' =1$ so our loss is $1-y y'$. The problem is the model may output really large weights to make this really negative so we need to bound the loss from below at 0."
      ],
      "metadata": {
        "id": "neiB2_RyvSOu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = Net(20)\n",
        "\n",
        "num_epochs = 10\n",
        "learning_rate = 1\n",
        "for epoch in range(num_epochs): \n",
        "  loss = 0\n",
        "  for x,y in zip(X,Y):\n",
        "    output = model(x)\n",
        "    # want y*output = 1\n",
        "    # set lowest loss at 0\n",
        "    loss += (1 + -y*output).relu() \n",
        "  loss = loss/len(X) # normalize\n",
        "  model.zero_grad()\n",
        "  loss.backward()\n",
        "  for p in model.parameters:\n",
        "    p.data -= learning_rate * p.grad   \n",
        "  accuracy = sum(float(model(x))*y > 0 for x,y in zip(X,Y))/len(X)  \n",
        "  message = f'Epoch: {epoch}, Loss: {round(loss.data, 2)},  Accuracy: {accuracy*100}'\n",
        "  print(message)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_Tq6_24pmHwy",
        "outputId": "54eefd83-0d2c-4d83-eecc-b9cd564b7da6"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 0, Loss: 0.98,  Accuracy: 79.0\n",
            "Epoch: 1, Loss: 0.82,  Accuracy: 84.0\n",
            "Epoch: 2, Loss: 0.37,  Accuracy: 87.0\n",
            "Epoch: 3, Loss: 0.27,  Accuracy: 87.0\n",
            "Epoch: 4, Loss: 0.26,  Accuracy: 87.0\n",
            "Epoch: 5, Loss: 0.26,  Accuracy: 85.0\n",
            "Epoch: 6, Loss: 0.28,  Accuracy: 87.0\n",
            "Epoch: 7, Loss: 0.25,  Accuracy: 87.0\n",
            "Epoch: 8, Loss: 0.25,  Accuracy: 87.0\n",
            "Epoch: 9, Loss: 0.25,  Accuracy: 87.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "plot_prediction(X, Y, model, title=message, h=.5)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "id": "pwQDYE7hxM_5",
        "outputId": "518a7e8b-73f5-4927-da31-a6919bd1fb0c"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy9d5wj53nn+X2qAHTOOU2eYRiGkZhFKpKyZa0k2msr2V5LTrT3rF2n/exJ6+xde3Vn7936VvbaPNkna1cryZYlmVawZIqySEmWRVJMM8PJ7JnpNJ0j0A2g6rk/Ct2D7i50mMY00I3n+/lgBl1VqPetKuBXbz3vE0RVMQzDMHY/TqE7YBiGYWwPJviGYRglggm+YRhGiWCCbxiGUSKY4BuGYZQIJviGYRglggl+iSIiKiKHCt0PwzC2DxP8IkBEekUkISKzWa+PFLpfuRCRMhH5v0VkQEQmRORPRCS6wc++QUT6rncfN4KI7BORr4tIXEROichDa2z7hyJyVkRmMtv+xIr1KiJzWdfvo1vsW3VmP1/eyn52EiLyb0TkFRGZFpFnROSBrHVfXvH7SIrIS2vs68HMdYpnrvHe7TmK4sYEv3h4u6pWZ70+UOgOrcEHgTuBW4AjwKuBXy9oj66NTwLPAU3ArwGfEZGWHNvOAW8H6oD3AX8kIq9Zsc3tWdfvZ7bYtx8GFoA3i0j7Fve1KUQksp3tZdq8B/gw8CME5/jPgc+JiAugqj+Q/fsAvg38dY59NQOfBX4DaASeAT59/Y9iB6Cq9irwC+gFHsqx7v3At4CPAFPAKeDBrPWdwGPAOHAO+NmsdS7wH4DzwAzwLNCTWafAzwNngUngjwHZYH+fAd6Z9fePApc3+Nk3AH051t0E/GOmPyeAd2SteytwMnMc/cC/yyxvBr6Q+cw48BTgbKAfRwgEtSZr2VPAz2/wOB4DfjXrbwUO5fE78QTwe8D3Fo81a90DBII3CVwG3p9ZXgH8F+Bi5rvyzcyyVec8+zsH/DbwGeB/AtPAzwB3A/+UaWMw8/2LZX3+KPAPmXN+JfM9awfiQFPWdq8GRoDoOsf7buC7WX9XZc5pR8i2+wAP2JdjX48A316xrwRwYz5/tzvxZSP8ncE9BKLdDPwW8FkRacys+xTQRyD8PwL8voi8KbPuV4D3EohlLfBTBD/IRd4G3AXcBrwL+H4AEdkjIpMismeNPsmK990iUnetB5gxCf0d8FWgFfg3wCdE5IbMJn8O/Jyq1hA8WTyRWf6rBMffArQRCI9m9vknIvInOZo8ClxQ1ZmsZS9klq/X1wqC83ZixaonRWRIRD4rIvvW288a+99LINKfyLx+YsW6LwP/jeCYjwHPZ1b/IXAH8BqCke2/B/wNNvswgejXZ9r0gF8m+M7dBzwI/G+ZPtQAjwN/T/C9OwR8TVWHCG7Y78ra778CPqWqqcx36gHC+TLgisg9mVH9T2WOayhk258AnlLV3hz7OkpwLQFQ1TmC38+613bXU+g7jr2WRluzBKOpxdfPZta9Hxgga/QNfJfgh9RD8MPMHqX+Z+BjmfengYdztKnAA1l//xXwwQ329z8RPHW0EIzq/pkco7GQz76BkBE+8FqCH7eTteyTwG9n3l8Cfg6oXfG53wX+lk2OrjPn7zsrlv3e4rlb57N/SSB22dfkdUCMQDA/AhwHItf4ffh14PnM+67MNX5V5u8PAZ8L+YxDMIq9fSPnnNUj/CfX6dMvLbZLMIh4Lsd27wa+lXnvZq7p3Rs4ZiG4WaeANDAK3JVj23NknmpyrP9z4MMrln1rrc+UystG+MXDD6pqfdbr/81a16+Zb22GiwQjq05gXJePUi8SiAQEN4Tza7SZPXqKA9Ub7OvvEdi+nycwLXye4Id6ZYOfD6OTwCyUPSLNPpYfJnhSuSgi3xCR+zLL/4BAAL4qIhdE5IMbbG+W4Kknm1oCk1FOROQPCJ4w3pV9TVT1SVVNquok8IvAfgIT1bXwEwSjbFS1H/gGwbwB5L6mzUB5jnUb4XL2HyJyRES+kHlimQZ+P9PGWn2A4OZ7s4jsB94MTKnqdzfQ/k8DP0kwCo8BPw58QUQ6V/TrAYJBxmfW2Nc1XdtSwAR/Z9AlItkmlD0Eo/4BoDHziJ29rj/z/jJwMN+dUdWEqn5AVbtU9QAwBjy7Qqw3ywDQIyLZ38mlY1HVp1X1YQJzz+cJnkhQ1RlV/dVMP94B/IqIPLiB9k4AB1acu9tZbaZZQkR+B/gB4PtUdXqd/SvLzV4bIjMRfBj4UEZshwhMej+amUzNdU1Hgfkc6+aAyqw2XIKns5X9zea/E8wXHVbVWoLR9+LxXAYOhPVfVecJrs2PEzxF/Y/wI13FMeALqnpGVX1V/XuCuYOVE+PvAz6rqrNr7OsEwbUEQESqCM5LzmtbKpjg7wxagX8rIlEReSfByPFLqnqZYIT9n0WkXERuIxgp/c/M5z4K/EcROSwBt4lI01Y7IyJdItKZ2ee9BN4Qv5W1/mMi8rF19lGe/SIwU8WBf585zjcQeMV8SkRiIvJjIlKnqimCiUU/s5+3icihzA1xisD8se6NR1XPEDyh/FamDz9EMJfxNzn6+yGCyemHVHVsxbqjInJMRFwRqSaYOO0HXs6sf7+I9K7XpwzvI5gMvZlABI8RPFFUENxsPgE8JCLvEpGIiDSJyLHMzfYvgP8rc21cEblPRMqAM0C5iPyLzFzJrwNl6/SjhuA8z4rIjcC/zlr3BaBDRH5JAhfdmoyXzSIfJzBFvoONC/7TwL8QkQOZ79WbCSbWjy9ukJk7eRfwsXX29TngFhH54cx36zeBF1X11Ab7snsptE3JXkv21ATBo+jia9Fe+n6We+mcIRhhLn62m+AHOE7wmP3zWetcgh/3KwSPs08D3Zl1y7xKCH5E/ynzfk+mD3ty9Pd1mT7HCeYJfmzF+q+R5S20Yt0bMm2vfB0ieJz/RuY4TwI/lPlMjMBmPkEgQk+TmX8gmFjsJRjF9gG/kdXWnwJ/usZ530cwyZjIHMdDWet+DDiR9bcSePVkX6P/kFn3pszn54BhgieQw1mf/Q3gExv4HpRnjvHtIev+BPhM5v1rCeZNpglG2+/LLK8A/ivBzWYKeBKoyPoeDWb69+9YbcP/nyHX+FTmOJ8imCv5Ztb6WzLXeYLANPjBFZ8/C3xjxbJZ4LU5jl0ybVwi+K6+DPyrFdu8l8DMt8qbjGD0/mNZfz+U6X8ic433Ffp3XgwvyZwco0gRkfcDP6OqubwbigoRiRF4SNymwWi85BGRrwK/qKovF7ov24WIPAH8L1XdUgCakV+2PcDC2N2oapJrn6zclajq9xW6D9uJiNxF4H//cKH7YizHbPiGYeQNEflLAh/9X9Ll3mNGEWAmHcMwjBLBRviGYRglQlHb8BvKKrSzamX8hGEYhpGLkxPDo6oamgSwqAW/s6qW//Xm9xa6G4ZhGDuGY3/1RxdzrTOTjmEYRolggm8YhlEimOAbhmGUCCb4hmEYJYIJvmEYRolggm8YhlEimOAbhmGUCCb4hmEYJYIJvmEYRolggm8YhlEimOAbhmGUCCb4hmEYJYIJvmEYRolggm8YhlEimOAbhmGUCCb4hmEYJYIJvmEYRomQF8EXkb8QkWEROZ5j/RtEZEpEns+8fjMf7RqGYRgbJ18lDj8GfAT4+BrbPKWqb8tTe4ZhGMYmycsIX1WfBMbzsS/DMAzj+rCdNvz7ROQFEfmyiBzdxnYNwzAM8mfSWY/vAXtVdVZE3gp8HjgctqGIPAI8AtBRWbNN3TMMw9j9bMsIX1WnVXU28/5LQFREmnNs+6iq3qmqdzaUVWxH9wzDMEqCbRF8EWkXEcm8vzvT7th2tG0YhmEE5MWkIyKfBN4ANItIH/BbQBRAVf8U+BHgX4tIGkgA71FVzUfbhmEYxsbIi+Cr6nvXWf8RArdNwzAMo0BYpK1hGEaJYIJvGIZRIpjgG4ZhlAgm+IZhGCWCCb5hGEaJYIJvGIZRIpjgG4ZhlAgm+IZhGCWCCb5hGEaJYIJvGIZRIpjgG4ZhlAgm+EbJob6STKTwPb/QXTGMbWW7CqAYRsFRVYZOjXDl1AiqigBNBxrpvq0DcaTQ3TOM646N8I2S4cqpEa68PIyf9lFP8T1l9MI4l77XX+iuGca2YIJv7HgW5pKM9U4w2T+d00yjfjC6973lZRjUU8YvTpJOprejq4ZRUMykY+xYVJXL3xtgrHcCEQEBETj4wD6qm6uWbZtOplE/vOaOOMLCbJJIo/0cjN2NjfCNHctY7wTjFydQX/E9Hz/t46V8zj3Vi5dePtJ3Yy6Sw0yvvhKrjG5Djw2jsNiQxthRBKaZYYbPjuElvdBtfM9n8vIkTfsbl5Y5jkPzwSZGzo+hWWYdcYS6zhqi5Sb4xu7HRvjGjqL36csMnRrJKfYAKFw5M7pqcdet7TTta0AcwYk4S2K/766e69hjwygebIRvLMP3fOZnFnCjLmVVsUJ3ZxkLc0km+6Zz2uKXbTubZH52gfLqsqVl4gh7Xt1F5y3tJOeSRCujRMvW/wmoKjPDc4H5SJWGnnrqOmqCeQPD2EGY4BtLDJ8dZeD4FSAwnVTUl3Pgvr1FY9+OTyQQRzYk+OIIicn5ZYK/SCTmEolVbKhNVeXSM/1MXJ5c8vCZ6p+hurWKg/fvXSb6kwPTDL08TDKeorKhgs6jbVQ2bKwdw9gOzKRjADDZP8XAS0P46WDyU30lPpHgzD9eQHV9gd0OohWbG5/EKrZ+o5odmWPi8tQyd07f85kdnmOyb2pp2ZXTI/R+5xLx8QTp+TTTgzOc/vp5ZoZnt9wHw8gXeRF8EfkLERkWkeM51ouI/D8ick5EXhSRV+ejXSN/DJ4cXuWjjkJ6IV00olXVWLnhbaMVUSobtz66Hr80Ferb73s+Y70TAHhpj4ETV0J9/C8/N7DlPhhGvsiXSedjwEeAj+dY/wPA4czrHuC/Z/43ioTkXDJ0uaqyMJuEtm3uUFhffMX3189/Ey2PcPh1+zZlY5/sn2LgxBWScynKqmJ03NJGfWctkPvpZvHBJz6RQETQkG3nZxbw0z5OxB6mjcKTF8FX1SdFZN8amzwMfFwD28B3RKReRDpUdTAf7Rtbp6ymjPh4YtVyEaGitnxD+1iYSzJ44grTQzOoBsJb3VJFy6GmDe9jLfy0jxAurEv9dYXuY53EKjc+4Txyfoz+FwaXRuiJqXle+c4lel7VSUNPPROXJleN3h3XoWlvPQBu1M15XxARy9NjFA3bNezoAi5n/d2XWWYUCZ23tCPuCmESKKuKUdW8villYS7JqX84G6QpWPDwkh7z0wuMnh/n1OPnGL80ueU+ujEXN7r2V1YkcLXcKOor/S8NhZpj+l8corq5krrOWpysc+O4QlVTBQ09geBX1JUTKV89dhIH6rtrTfCNoqHovHRE5BHgEYCOyo3/cI2tUdtWzb67urn8/OCSj3ttezV77+rekGlk8MQVvFSOPDZe4OlS31mLE3GWomKD6NcsL5css0qsKkrn0Tbqu+qW1osInbe2c/m5gWXBU4vrJCIcemAfjrvxcczC7ELO0bl6Pqn5NPvu6WFqcIbx3iCqt2FvPQ1ddUtCLiIcfGAvZ79+AT8T9eu4DmVVMXpeZeMao3jYLsHvB7KjW7ozy1ahqo8CjwIcbWwrDveQEqGhp5767jrS82mciBOYKjbIzJV1JnYFpganmewPXgCRsgjdxzpo6K5j9PwYl18YXBLy+akFXvnny/Qc82g+cDVitnl/I44rDLx0hWQ8hRt1qWmrpqG7NjMS39xDqxtzc7p5qgbmGhGhvrM2Y9MPp6K2nFvediNTAzMk40kq6iuoaa0yX32jqNguwX8M+ICIfIpgsnbK7PfFiYgQvQZ3RjfqkppfI+OkKgPHg9H7optnKpGi97uXcVyH/peGVo3aA7PK4FJ07CKNexpo3NMQ5LTfoqBGy6NUNVcyOzK3fKQvUNNaTSS28Zue4zo09NStv6FhFIi8CL6IfBJ4A9AsIn3AbwFRAFX9U+BLwFuBc0Ac+Ml8tGsUD82HGul/cbVoL+J7ysJccpX5RD1l4KUhNIfzjfqwEE+GBlDla/S8/549nP3GBZLx1JLrjRt1iVVFmeibor7z+tnhpwanGXp5hIW5JJX1FXQcbd2U+6lhbIZ8eem8d531CvxCPtoyipOWg03MjswxOTANuTwncxjoElPzOferqkQ2YVq6FqLlEW76vsPMjcWZGpxm+OwYqWSa0fPjjF+cZKA8wg1vOkhkA2kYNsPIuTH6X7zqHTQ9NMPMyCwH799HbVt1XtsyDLBIWyNPiAgH7ttL920dkMfBcHVLVd6FNgwRoaqxktELE8FTSuam5ad9FuaSXH4+vxZI3/PpfzHcO+iyVeAyrhMm+EZeqWqsXHvidJM3g7YjzVvr0CaYHYuHT+AqTPZN5TXFRHwykfNcJOeSeKk1soEaxjVigm/klcrGCiJlq00w4ghN++qpqN9cANbl5wa2LZeP5iiPCGwoYdtmCIK1ckZrme++cV0wwTfyiohw6LX7iZZHcCIOjiuIK1S3VNHz6i5ueugw7iY8X5KJFMm51HXs8VWqmipz3lyqWypzThInpuY5++QrPPfZ47zwtyczgVxrp4AorykjmiMaWFzhwrcvMjU0s7kDMIx1KLrAK6Mw+J7P8NkxRi+Mo55PXVctHTe1XpOLZnlNGbe87UZmrsySTASpgivrryYya7+xhcGQZGO5WCuVQj5xoy7dt3XQ9+LVeAAkcLfsPtYZ+pnE9DynnziPnymp6Hkew2dGmRuLc/j1+3PeJESEg/fv5czXz+N7QbDW4mH6KZ/poVlmR+ZoOdxE160deT9WozQxwTdQVc491cvceHxJ6EYvjDPZN8VN33eEaEjagPUQEWrbwyOlW480szCbDIqPu5n89r6GWjiiZZFtLcTScqiJ8toyrpweIRlPUdVcRfsNLZRVh/dh8MSVJbFfRH0lPh4nPp6gqim3i+XijXFqYIYrp0dW5TLyPWX4zBjNB5qKrhiNsTMxwd8lJBMpBk9cYWpgBnGFpv0NtN/QsqHI05nhOeLjieU+9Apeymf4zAhdt+V3hCki7Lmji46bW4lPJoiWR/G9oPi4+n7gky9BHdp9d/dse7RqTWs1Na0bc4ucHY2HLldVZsfm1hR8AMnM3OZ0TZXAXbPlYNOG+mMYa2GCvwtILaQ59Q9nSSe9JbPAlZdHmBma4cgbD64rmDPDs6E2Z/WVqcGZvAv+ItGKKHVZJqObv/8ww+fGSEwkKK8rp/Vwc9GPbKNlEdIhEcbiOOuWT0zNpzj1tXOkF7ycAWsCm04XYRi5MMHfBQyfGQ0Sl2UP0H0lMbXA9NAsdR1rJ6Fzo27O0oGbmWDdKrHKWODHv4NoPdLM5e/1h85H1HXlTrOwMJvk1BPn8BbWdr9UZd3rV0h62nsL3QVjE5jg7wKmB2dCxdpP+8wMry/4jXvqGTx5ZdVyx3VoPdQcmCdG5pgbjxMtj1LfVbupxGq7mca99cQnE4yeH1+ePfO1e3FzFD1JJz1Ofe3cUlbStei5o3NbAs+uhZ72XioP9Ky/oVE0FOc3ydgUbojfOwS+7xtJ/hWrjLL3zm4uPt2HSMY9XKBxXz017dWcfuI889MLS2l/Lz83wOHX7V/XPl0KiAg9xzppO9LC7OgcbtShpq0ax8lthhl7ZXxdt00I8u5HYsX3E10c1ZvY7zyK79tkbJrWQ83Ex+KrzQoSjEA3QuOeemrbqpnsn8b3fGrbayivKePycwMkJueXniAWPVLOfbOX295+kwUIZYhVRmncE36uVZX4eIKpwWnEdZgdmctps8/G95S50bk10zJvNyb2OxsT/F1AXWcNzQcbGTk3DhJM9Cmw987uTZX6i5RFluWeBxjLFP1YifrKzMicJflaB1Xlle9cYnpwJrghL16cDSDOtaWqvl6EiX30NnvK20mY4O8CRITu2ztpOdTM9NAMjutQ11m7qVzuuVhrJGr5XtZn/NLkVbGHDYs9gAg05Hhq2G7WEnundU8humRcAyb4u4iyqlje/bWXioOsQH2lurkqb+2owqiWA0KzJNgthaJGz4/njCgW52rOHPUDLyvJcsE88Jo967p2bgcm9ruHwn+bjG0nvZBm6PQI4xeDwuK1bdV03d5OtGy1+aD7WAdnvn5hWei/4wqtR5qvKQI3jD6/mi+m9pMkeCKJ4vHWaC97nJ2fS2ZlFG42LYeaqO+qBYXKpgq8lM/s8ByOK8HEbxH435vY7y5M8EuM1HyKk185u8wlcPziJBOXprjhwYNUNlQs276yvoIbHzzI4MvDzI7GiZZHaLuhhYbu/JTym9Yon0sdIsVV81MKl8+nDvK+2EnqJJmXdgpFfXcd8zMLq+ZBHDeok5v9lOSUFVeJRBP73YcJfokx9PJwqP+3qnL+W73c8i9uXBWZW15bzv57rs8P/IV0C35IYngf4fl0C6+P7uxiIK2HmxjrHSeVSC+JfjCCr6GquXgnPDcs9gf2b2e3jC1igl9iTPRN51yXnvdYmFmgvHZzOeu3wjjleCFZun0cxrQi5BM7CzfqctObDzN8dpSJy1M4EYeWA4007mvYco6g+ESCvhcGmBuN40QcGvc10HVr+5ZNQZsVe6lo3VJ7xvZhgl9iOO4aIiPg57nQx3q0yxy91K0SfRePdmf1ZPFazGqUF7xmRvxKmiXB7ZERamR7cumvhRt16bi5jY6b2/K2z8TU/FJqZQgS3Y2eHyc+keDIGw5c883ExH53U/hZIeO6sjCXZHZ0LkisBjQfyO3F40QcKuq2b3QPcKs7RgSf5f6Kigvc7o5seD9DfiX/X/Ioz3jtXNB6nvXb+FjyKAN+/jyJiomBkHoC6iuJiXnmxsIzeK6Hif3ux0b4u5TUQpoL375IfCIRJEbzlJZDjXQcbWPi8gSJyYVl24sj7L2re9tTEVdKmndHT/PV9D6GMyacFknwfZGLVMnqLJS5+HJ637KJXw8HD/hyah8/FTuxa9w8F8kl6r7vMzce37TLbFheHBP73UdeBF9E3gL8EeACH1XVD69Y/37gD4DFGbiPqOpH89G2Ec75p3qDQtl6NXhq5Pw4kfIINz50mLHeCYbPjuIlPSobK+m4uXVZVartpNmZ50djp5jXQLDLZXMBXTMaZVrLQtfNEWOaGHXsbG+flUTLw9MyO65DtHxz0bm5xH6VJ86B/auE/tTk2U21tR3MTChf+6TP2eeCyOYjrxYefI9Ddf0uu+tfA1sWfBFxgT8G3gz0AU+LyGOqenLFpp9W1Q9stT1jfRJT8ySm51dFdaqnXDk1SvsNrTTvb6R5f2P4DgrEZoW+lGm7oYVLz/StMuuIsOHcO1s14QRiX1yeRvNzykd/fYb5rOmfk99Rek/6vPOXKmjqcCmrKF3hz8cI/27gnKpeABCRTwEPAysF39gmFuaSiEhoLVgv6aGq2266yQdTGmNGYzRJgoqsm0ONpKiVBSZCvHqqSVK7y0b3AA09dSQmEwyfHVuK1nUc4eBr9+HkSMuczVbE/uqovrjEHuDJz88vE/tF4tPK//j9OI7ArQ9EefOPleNGdt5vYKvkQ/C7gMtZf/cB94Rs98Mi8jrgDPDLqno5ZBsjD1TUlYcmPIMgq+NOE/uEujyWOsiQVuHi4+FwizPKGyOXWUzW+QORXv46dQQfwcPBwcdFeUu0d9fZ7yHIn9R1WwetR5qZHY3jRl1qWqo2lL10t4o9wMl/yu2V5afBB176VgoR+P6f2Pluv5tluyZt/w74pKouiMjPAX8JvClsQxF5BHgEoKOyeCv9FDNlVTFq22uYHlpeGEVcofPW9gL27Nr4fOoQV7QSH2fJffOE30SVl+LeyBAA7U6cn4ydCNwytZJmisct8/oiTPZPMdk3japS21ZN97FOymvC5zTyLfafubC88HqhSWygO+kkPPdkiqm7FSe2C0cDa5APwe8Hsmd8urk6OQuAqo5l/flR4P/MtTNVfRR4FOBoY9v2OoXvQHzPZ2pwBi/pUd1StfRD339vD30vDDLWOwEalCrsurU9Z872YmXcL2NEK/BXeBCncXnWa+Med2hpBF8tKe6PDBagl4XB93xOPX6WVOLq5O300Cynv3aOm7//yKrUytdL7Mvd4vlOxRrGSY6uPxckDrhzNZRVlJajYj6O9mngsIjsJxD69wA/mr2BiHSo6uIv8R3Ay3lot+SZHZ3j3Dd7A08cDe6NDd117L2rG8d12PPqLnqOdeKlfdyos+NMOQDTlOHkyCm8gIsPlGqxxZFzY8vEfhEv5XPi789Q215N+02B91UpiD1A+4NVXPrr6cB2swbqQ7S29L45WxZ8VU2LyAeArxD89v5CVU+IyO8Cz6jqY8C/FZF3AGlgHHj/Vtstdfy0z7mneldlY5zsm6KqsZKWQ0GAlThCej5F3/MjzI3HKasqo+3GFmpadkZAUpMkQlMvAFSTYq3A4d3O0KncgWl+2meyb5rpwRnue0c5ENl1Yq+qTB1fYOSbcdKzPhXdUdreWEXbmyq58vV4UK4zJJRDotB4R3nJmXMgTzZ8Vf0S8KUVy34z6/2HgA/loy0jYHIgPCeO7ynD50aXBH9uLM7Zb1wIUiYoLMwkmRmZpedVnUXnlhlGjaQ45Exw3q8nnTWWj+DxgLuzE6tthYW55IYK0Pie8sLX53n7b920tGw3iD3A0ONzjP1zAs1M08ycTjJ7PsmBn6yn6e4KEgNpnJgw+eI848/ML1Uba7yznI43l2alttIyYO0i0sl0Tk+c9LzH+W/2MjeRCNwwV2ynntL3/CCNe+qLIuf6erwlcpGn0mle9JvxEcrxeMDt4+bIeKG7VjC8pIfjOmvm218kPqMk4x6xSjfvYv/UV9Zv/3ogKZ/aZxLI8owcaApOf3KW2VvrWJI3pwruqMRJ+fhRhwkRzj+uLAtUUcWdTYOCVx2BXVqr2QR/h1LdXEVIVmEgKD04Nbh+8ZD4RCKvVauuF64ob4j28TrtI4VDDH9XulpuhvLacC+cXDgRuW5i31ax/UXWvdk50o5ASDWxyEx6U33yJ+dJvTQM3tWbl3tDE5GO3eclWPzDOyOUytUGdDoAACAASURBVPoKattqkGs1Yis7YnSfjSNQJib2EFy7rlvb173+4kD7DdVU3BmYMHaD2APIOsFlC0+8Qur5ITQe2HvU8/En5/Fnk0sODgCa9Eg9PwRJL7h5ZF7eqTH86YVcu9+x2Ah/B3Pgvj1cOTPKyPkx/JRHdXPVhkb2AG6ZS0X99mbGLDQTWsa0xmiU+V3hn99yqIlYZZTBl4dZmE1SVhVlYW4ewSGd9ImUOZRVudz/OwdD8+LAzhR7AKkvB9cBL8c8hoI/liD59ABOTy3+xaklG76UR4jc3oZTGcUbnAkvLO8r3sUpnFt3V5I4E/wdjDhC+40ttN/YAgReC89/7sRSsrTQz7gShOC/Zu+OdNO8FubV5bHUAYa0GicTqXvImeAtkYu4srNDPeo6a6nL5M7pae8l1r2Pyy9MMzeRoq69jH3vbSPSsXf5h3ImQVsePfuZC4lVk7PFIPYQRBpHb28j9dwQqIaadgDwfPxXJpct0niK1LODxO7vQRNpyDEXpomdPyhYiQn+LkJEaNzbwHjvxKqJWjfm0rS/gfLqMhp66nCjpeOD/IXUAQa0OhO8FZgCzvv1fCOd5k3RvsJ2Lg+s9LHff1f9llMl5BL7Qgt9Nk5tGbEHevBH4qQvTcFMSM6kXPdzz8cfi+PUluEPza6+YQhI3ebmSXYCJvi7jO7b20lMJJifXsD3fRzXQRzh8Ov3Fyz9cSGZ0Sj9S2J/lTQux/1mXq/9O3qUvx0BVcUyqg9DXAe3vRqdS+HNJnML/Ep8RRNp3K4auDCx2jTkCJE9xVNQPl+Y4Bc5ftonmUgRrYjgRsJH5apKMp7CcYVoeZQbHjzI7Gic+ESCWEWEus7aHTdBmy9mNLaUcG0lipDEoYKdmZa51MU+G7ezGu/SVGDe2QiO4FTHENchdmcn6VOj+ONB/QipLSNyYxNSsbm6AjsBE/wiRVXpf3GIkfNjQapjX2naV0/3qzpxnKviNT00w8Vn+kknAx/iivpy9t3TQ01L1Y6JpgWY0wgnvSZmNUqXM8tBZzIvUbSNMp8zUjeKR/k1iP2i0BYDJvYBUhElcmsr6ePDVxcqOD01+JdnltvpBaQigjQETgtSHiF6rD0wg6oiu3hwZIJfpAy8FIi9erqU137s4iSqsPfObgDikwnOf/viskna+HiCM0+c55a33rihvOjFQK9fy2OpA2gmtfFxv5kaSfKe6OktF0UpF49bnRGO+82rInXvdwc27eIZNqouBqK3VaK+MnA6xsU/eganzOHQz99L64HdL/aLuM2VOK/bi04kUAWnvhyJOPgNlaROj0KmQpjTUhWM4Fdc/CC19O52ZDDBL0J8z2fk3Ngqbxv1lPGLk3Td1kEk5nLl1EioR47vKRN9UzTta9iuLl8zaRW+kDqwTIxTuExqGU+lu3hz9NKW23hjpI9KL82zXhtJXCpJ8Rp3gFsjY+t/OIuwUoDFQPS2SnxPeeJ3+hh+bpR03AOBC1/s48jP3cs9/+XtwO4W+0XEEaRp+QS001RB2Wt60LQPjmyoZsBuxQS/CEkv5C7eLY6QjCeJxCpITM2HbuOn/aDE4Q7gkh8ezejjcNpv5CG9RJ9Wc9GrJSYeN7rj1G7Sh14E7o0McY87hIfgopsa2RfjqN73lIGTM8w6Su3kAimnluHvjZJOZJ6IFNLxNCf/6zeZuzxF02/cTFlPDbtZ7NdjvWCtUsAEvwiJlOW+LH7ax8lM3pbXljMfEg3ouE7OAhjFRi77OkAa4bOpQwxoNSkcHJR/8jp50L3ILdeQR0cEIht24whYa2K0UMRHknz5F0+xMJ3GW/CJVETwkj5+KiSvjcLFv3mJvq+c4pavvpvKG4K+l5rYGwEm+EWI4zo0H2xi+Mxo6Pq+5wY49Np9tN/YwtTg9CqzjrhCQ0/hsxluhG5nBi/Ubqo0ME+/Vi+Ze/zMdl/z9rLfnaZKcj8J5YMNe8FsM9/84JPER5NoZjCfmlvnPCh4syku/vqT3PTXP2hiX8LYM06R0paJng1jZniW9EKayoYKDty3h0h5BMcNbJMV9eXc8KaDuDvk8bVCPO53+4ngsehE7eJThoeHs8y2v4ignPWv7/xEsYr9/OQCw8+PL4n9hlGYerLPxL7EsRF+keKnPBxX8EMmZcUR0gtpImUR6jpqufVtNSzMJnFch1jlzvMdvjMyTKsT53teGzMao0dmuCNyhU8nbwjd3kdI6fW7oRWr2AOkEx6S69AdwYk4+Mnwu4GXcTcsd+sh5RF7/AzTnzzDsbIYMw/cw+x+E/zdjgl+kRKrjAXeBDlyhMSqYkvvRWTH2OxzsceZZY8zu2zZYWeC5/zWVVGyDrDPmbou/ShmsQeoaqugrC5GfHj1pHxZfQU3/sK9vPAfn1i1ToH5+3sCsV9IU/szn0TOjFGTTKEi+C+eZvD193L5bQ9tw1EYhcIEv0gRR+i8pZ3+FweXjfLFgbYbm9eNnFVfmeyfYvzSJOIITXsbqO2o2VEJ0+6KXOFUspF5IkuTu1E8jjgTtDj590LabORqIRDg3o98P9943xfxsurZupVR7vlv7+DUV58L7ogh87fSFKQKKP/r55HTo0RSwedFFTeZouMf/4nRO28n0Z7bnGjsbEzwi5iWQ004UYeB41dIxRddEYWhkyOkEml6jnWG+hSrr5x76hXmxuJLN4vpwRnqOmvZd0/PjhH9SknzE7GTfM9r5ZxfTxkex9wRbnAm8t7WVtIUbDd7f6SVt3R18dzvPs7Ys32AEOmuoG+0n/lX4uFiDzhjcQDSnzhOVWr1RK94Po0vvky/Cf6uxQS/yGna28DsyBzjlyaDqNtMiPhY7wSq0HKgkVhldJkr58TlyWViD4Hf9tTADDPDc9S27Zx6nhXicX9kkPsZ3PBnVKFXa+n1aimXNDe549RLSCbFDDtJ7BdpuXcPVZ21DH8zTXouycLoHL0f+ifc2jIod2B+uepreYRTVd288hWfN+ecBAC0MCULje3BBL/I8VIe4xcnQ+vSjl0YZ/ziBCjUd9ey985uHNdh7OJk6GSv7/lMXJ7cUYK/WdIq/E3qMMNaSQoXB5+nvQ7e6F4KjazdiWIPMPrdy7zy6RdIx68GofnxIJ+SLw7i+EhGu9UV5mNlXL7jKG0VtYzcfTvlX3oCd8UoX12H8dtu3s7DMLaZneG7V8Ik46k1Q8EXR/2TfdNcerYfYJ0o0p1hzrlWnvNauJIRewgidtM4POHtYU6Xj292qtgDXPz8CdIhBTr8RJp0dy3JNx5Goy4ac+k/eoRvfODHaapvBuDKA3eRaGvGiwUeXQp4sWiwvKN4jnEtVBVNpNB4alnJQmNtbIRf5MQqo6tG92Gor0xcnqL7WCeN+wIz0MpRvuM6NO7dfTm+s3nJb1nTd/+YOwLkK7Vw4RhPTwZFfsPcdivKmfnDHwTCfez9WIzjv/yzNH/vOI3Pn8ArL2P4vjuYPly4yejN4E/NkzoxAgsZ99OYS/SWFpy60irZeS3kRfBF5C3AHwEu8FFV/fCK9WXAx4E7gDHg3aram4+2dztuNKhUNdY7sWbpQgg8e1KJFA3ddYz3TjA7Esf3gh+84woNPbVUN++clMnXgqfhTzA+srQuLAla9LbK0Lqv4aUAYWWFqO2m6QdvZvBPjuMnVphlKiIsPHwrsHZAlUYijNx9jJG7j13/zuYJfy6JPxrHuzCxfGJ6Pk3quSFi93Yj5TaGXYstnx0RcYE/Bt4M9AFPi8hjqnoya7OfBiZU9ZCIvAf4P4B3b7XtUqHnWCciwugr4wiE2uchGOXHKqOICAcf2MfU4AwTGbfMxr0N1LRW7RgPnWvlsDPJ835LqO/+a9pO0hqd3ZiP/QbrvhaKqqPNtP/c7Qz92Qt482lEQSuipG7tYOHtR3dV9Kz6SvrEMP5oIpiRz1F0PN03TfRQ47b3byeRj9vh3cA5Vb0AICKfAh4GsgX/YeC3M+8/A3xERETN+LYhxBF6XtVJ163tpObTjF4YY+Tc2HL/fFdo2t+4VKtWRKjvrKW+c+f/4DfD3ZEhziQbSKzw3b+nundjYr+Juq+FZu9v3s/zx7qp/MIZIgsOyQePkHrgAE89HqzfDWIP4F2aCsR+LdOmgobVtDWWkQ/B7wIuZ/3dB9yTaxtVTYvIFNAErMoOJiKPAI8AdFSGp84tVZyIQ1l1jM5b2xHHYfjsKKqKAM0Hm+i6tb3QXSw4i777z3ktnPfrKSfNQ80nuaViYEnsXx6v5e+Gexj8dhktjcLDb1rg1TenN10dqii4pZWF24+wmDN1N43sF/H6ptcWewiqWNXE1txEVYMiKK6DxMLLhe52is7gpaqPAo8CHG1ssyeAEESEzlvaaL+5hfS8R6TMLdmatWGUi8d9kSHuY2jV5Oyzww187PRBkungfF0chD/9qwre/eNVPHiN1aGKhd0o9gCkNxAbIBDpzn3c3vAc6dNjwb5UkboyokdbS87mnw+V6AeyZ8C6M8tCtxGRCFBHMHlrbAHHCZKlmdiHs1LsfYVP9+5bEvtFkinhf308zlPfipFMBuJiYl88SN36eaIiN7XkFG9/cp70iRFIesGXQEGnFkg+M7AhD7jdRD6U4mngsIjsF5EY8B7gsRXbPAa8L/P+R4AnzH5vXE/C3C4n9tQxEw9/lPc8+Pijg3zoAxd49tKZzFIT+2Igcqhx7fARV2CNQU/6lcnVJiEF0j7+aDwvfdwpbFnwVTUNfAD4CvAy8FeqekJEfldE3pHZ7M+BJhE5B/wK8MGttmsYuQgTezlaxZ98cQ9+DrdNgFQSxsdSPP4Jn50k9m0VtbtW7AGcmjKid3UGcQdheIo/NZ9ztK5zOSZzPUXjmyuXudPJiwFLVb8EfGnFst/Mej8PvDMfbRnGWuQKqPrn07WMzJSzXqSx78HpZzWYDBdZs2BIMbCbhT4bp6aMyC0tpI+PhE7g+penSU3OE72jY7XrcWXkapDWSna3l/IqSmvGwtjVrBU9++JgGwvJjf26/Yw2bLQ6lC54pC9PoRPzSLmL21OHU29Rn/nGbamCm5X0y6OrI4x9RWeT+GMJ3Obl7rNuYwXexOrazwD+xDzsvV49Lj5M8I1dwXqpEqorFRFF1zDpLO3rBoePfnqOxEWloqGcxjvSlDVHwsU+kSL53QHwffBBp8EfTeAeaiDSs7vTWBQCt60a7/I0OhUi4J7ij8ZXCf5aw3idLS3ffRN8Y8ezkbw4r7szxT8+W0Zqjd+364Ibhf5LPv550BTMOQnGvptg9kA1NJetMqGkz46vdhv0Fe/cBG57NRItTX/v64lEnNBgWwBCajlLeSSY2A3NO7TzSoJuBfPnM3Y0G02C1vOavbzzPZVEo0IsJkSjguNCdT3UNAiNbcKrH4oS2SN4iUDsgWDUnobKs7O0xlanlfbHcgRhScZcYOQdp6s2fALXEdyO1dfIaanMuX1kX/FNxF9PbIRv7Fg2mwTt+/8l3PX6FE9/a5rHPjfM/CzMTgIo0RicGU+TOB+eq0UUkv/UR+SWVtxs+/xaFqI10lob147TXIHTWY0/MBvk1gEQwT3UgFO1OtpWXIfoHR2kXxxG59NL18w91IjTVLGNPS88JvjGjmNThcZXJEFrbIoyFB9hfg7SWR55qSRMP6+sGR2y4JF+dhB/Tx3Rw0GSLqetCn9wNvQm4TTYxO21oAtpNJ5CyiOhJhcRIXpDM35XLf5YPBD71qo1o2adqhjRe7sCN8y0ItVRpAQDFk3wjR1FPvLYH/+WkE6uVmhNQ6zJYWHUX3Pg7l+ewm+rwqktI3KokeTEfBDF6WkwehQhcrS1JAVlK6gfeOD4w7PB05EfRNlGb2tDQmzzTnUMp3rt/DnZiAgS8gRQSpjgGzuG/BQtqUSc2fAGBKajUWLuApLDbRsABe/KLE5tGRJ1id3bjT88hz+RQMoiuJ01JZejJRt/egFvaBY8H6elCqepYkNpudPnxvGH54Jc9xlfe52cJ/XSMLFXWWLAfFC630pjR5EvsQe47YEoj/cvoMtrh6BAsq2cmn3NpL7TH553fZEsxxxxBLe9Grc9P7WCNe3jDc6g00mkKhrcQHZIdsf0+XG8S1ezW/pX5pC6cqK3t61dqtNX/P6Z0BQIOjmPzqdL+iaaL+wMGkVPPsUe4MJej7J2YWFY0SQgoALzHRW0NGVs8501+AMz4aLvCm7r1nPjq69B5sYs048fT5F6ZiAwD/kKjuD1ThJ9VXvRl/DzZ5PLxB4I0hdMzuMPzeJ2rpHuPJPFMhQJ7Pom+FvHzqBR1ORb7D9zIYETEQ79VBPTp5JMvbzA8BUlXRulvroGTXlI1CVypIm0K/iXppd3SAhMFFuIpNW0T/r0KP6VuUyfI0SONOM0VZA+OQKprMeHjHimjo8Qe0133iqWqSo6nsAbjUPEwW2vDvVw2Qze0Gx43npf8QZm1hb8qBP40KdC0lYoSGVp+ctfL0zwjaLleog9XE2VUHe0jOPnHKpmpikbTZGSOCg43bVEDjUQPdyEv68eb3AGfziOuA5uZw1Oa+U1C6+qkvreYBDhmdFGjadJvXiFyG2t6HR4CgCSHjqXQjYxSZmzD76Sen4oaCsTjORfmsY92EBkz9rRwZpIob4imVKay1eu8cFcic3m00sZK53uWvyLU8u3dQSn0wLY8oUJvlGUXG+xB3jq7z1qj0/jJJePKv2+abzKCJGuWpyoi7OnHvbkJ0BHJxcC18CV+udrkMY3F0Juk8cm8fozqQmyhdVXvPMTuC2Voa6Q/lyS9EvDaCLjx+4IkZuag/w2GdzWSvyw6lSO4ITMb6RfmcDrncpaokhjJTo1H5h4HMHtqcU90LDFIzYWMcE3io5tEfuv+ESmUjhhBeF9xbs4RaRrc5koNZUpsBFzcz4B6OxCbuGOp5CqKDobkrLXkbyM7oEgYClsxK2KNxwnsnf5KF/TPqlnBpenkPCU9PER5I4ITm2mQEllFKkrW34zcQSpjOB2LTfn+FPzgdiv6IeOJ4i8uj0wL7mSNxOWEWCCbxQV2yX2APVaRlpzuGgm1/LLXI7Op0mdHEEn50EEYi7RG5twmkImdssjwTYh9g8pixC5qZnU9waXKjMt+fXf3JI/8ctV5UnD1/lX5nLb5i9OIje3kDoxgo5kiolI8JKqKE5XLW5H9aqYBK8vxCMns09/YAb3ppbNHZOxISwyxCgatlPs2ypq1yx6vdEAHT+eIvndfnRi/qpgzqdJvTiMH2KPd5oqw6szOYK7rx6ntozYPV04XbVIXRlORzXRuzpDMkBeO05bVXhKCEdwmlenGvDjyZw3CX82SfKf+6+KPQTnQUETady2qtAANJ1fo/BI2MStkRdshG8UBbny4sD1EXsIimqsMkFAYJ8+tLbdWOeDidbsyddl+IrXO4lzW9uyxeII0Ts6SL0wBPNexjYP7v563LaqzLFFid7QtGb76+HPJtHZJFIRRWpjy54O3J7awKNmwVtmenHaq3BqVtePdapj+LmyTUYcdDpHClJf8QZnQyeCNZEO+UCmvTze3IzlmOAbBWezSdCyCcR+uUBstHAJQPT2NtJnx4N8OBnvk8iRRpyG3Em1VJXkswOBYK+BnyPXulMZJXZvNzqXgpSH1JSFpg64FjTtk3xhCKYWlsxBUh4h+qoOpCzwdJGoS+zuLry+afyROXAd3O7aIKtkWH9bq+DcRFD4d9kKAWeNfmvgP79qcTy15iheQp4yjPxggm8UjK2YcCC32K+sPbtY9zUMcR2iNzajNzQF/t4byHDpjyU2ZHZwcviOL9Ze3UwemI2SfG4IFk1JGvyjcylSL10hdmfn0nYScYLUwBtMDxw51EC6dwriqUDoYy7Rm5rxBmZye2MKOLWr4xU07efOMiogpVZ3cBsxwTcKwnbb69dDRDZe3zSRWt9FMmOTz0YXvCDgajTw95faGJEbmq96uWwR78rsVbFfgc4kryk9gXdlNigpuIgruIcacbtqlsxE/vBcuFkr5oY+NazpbVTmBkFYxnXBzqyx7RSb2G8WqYplPG1yEHFwb2peVtdWfSX5zAD+SPxqwNV0MgjCiq8xgbkJ0hfW8ONH0SzPI39mgXTvJOlLU0GO+BD8uSTpk5n6sVkv7+z4Up+loRynu3b1zbIqSuyertAnJnGCm8aqegGOELmh2VwxryM2wje2lZ0u9hCInJRHVgdQOYJ7pClwQ1whZv7wHKRCbP6ekr44STQfbog5hBsIniiqoqgq6ZOjmayUQTpn7/wE7pHGVXEHXlgQFQSTsX0z+HUxvPOTQbuuBF5P9WVE2mrWNVdFumuR8gjeK5PBk0dVlMiBBiv+fp3ZkuCLSCPwaWAf0Au8S1UnQrbzgJcyf15S1XdspV1jZ7IbxB4yBThe3UH65Aj+eCIY7UccIjc04raGZ8z0s9IYrCS0IPe19KvcReO5RT99bhypil0Ve8i4UCre6TGcujKc6qvmpbU8afzJeRjI8qX3FJ1O4lRtPEe921yZV3dTY322OsL/IPA1Vf2wiHww8/f/HrJdQlWPbbEtYwezW8R+EYm5RI+1BxOQnr9mdC2AUxHFdyR0xCwV+XnQdvbV450ayxFFS5B+OPM+bH3q6UGid3cuJVFz6svxJuZDUyUQT61e7iv+4Cx6oGHHpHMuNbZqw38Y+MvM+78EfnCL+zN2IbtN7LORiIOURda1OzvtuYOd3L35ydMT6ajBPdgAroT/sjMBUTnxlfRLw0t/ul01wb5W4obfuIAglfFcDr98o+BsVfDbVHUw834IaMuxXbmIPCMi3xGRNW8KIvJIZttnJhYSW+yeUWh2s9hvBom6RF/VDjE3EMzMy72hMa9268ieOmKv3YOEBFBtBE2k0URqqc+xuzqDQKhMugSnuZLoXZ3hNwII5grKbGqwWFn3yojI40BYfbFfy/5DVVVEco0f9qpqv4gcAJ4QkZdU9XzYhqr6KPAowNHGtvykBzQKgon9cpy6cmIP9ASRqb6P1JZdl7q34jrBBG2uuQGHZRW7ln8Y1NOlhxGpiBK9vQ3NuKEuPsk4nTWrK1QJSE3MctcXMesKvqo+lGudiFwRkQ5VHRSRDmA4bDtV7c/8f0FE/hF4FRAq+MbuwMQ+HBFB6vLjd78Wblct/lBI0jNHiNzWGphuwiaRHQkV7JUmq8ihRtJJL3AzzaSHkOoo0dtyPeQbxcBWhxePAe/LvH8f8LcrNxCRBhEpy7xvBu4HTm6xXaOIMbEvPE5tGe6hhmCC1ZVMGgTBPdyI21RJ9Na2HH7wTRuKNhZHiN7SSuy+biK3tBK9q5PYXV02WVvkbNXY9mHgr0Tkp4GLwLsARORO4OdV9WeAm4A/ExGf4AbzYVU1wd+lFCIJmhFOpKcOt60afyxTUaqpckmQnaYKond24L0yiT+bxKmKBtk6N1k3V8ojuFZrdsewpSulqmPAgyHLnwF+JvP+28CtW2nH2BlcjyRoYXlxYHeIvSZSpM+MB4IsgtNSSeRwY14nPSXm4naE15J1aspWZfM0djd2aza2TL5NOLD5JGg7DU16JJ8euJqETRX/yhzJyXli93Vfl8lcwzDBN7aE2euvDa9/OnzSNO0HOeS7d8+xGsWDDSOMLbPSjAOsNuOwOr1x5tOrlqwc2cPuEnsAPyyCFYIUBZPz298hoyQwwTeMApAzTbEQ1L01jOuACb5hFAC3u3a1WyQEBcu7widZDWOrmOAbRgFwastwb2i66iefeUVuaUEqLFLVuD7Ys6NhFIhIZw1ua1VgzxdwGsrNO8e4rpjgG0YBkYiDm6N4uGHkGxtOGIZhlAgm+IZhGCWCCb5hGEaJYIJvGIZRIpjgG4ZhlAgm+IZhGCWCCb5hGEaJYIJvGIZRIpjgG4ZhlAgm+IZhGCWCCb5hGEaJYIJvGIZRIpjgG4ZhlAgm+IZhGCXClgRfRN4pIidExBeRO9fY7i0iclpEzonIB7fSpmEYhnFtbHWEfxz4l8CTuTYQERf4Y+AHgJuB94rIzVts1zAMw9gkWyqAoqovA4iE1Oa8yt3AOVW9kNn2U8DDwMmttG0YhmFsju2w4XcBl7P+7sssC0VEHhGRZ0TkmYmFxHXvnGEYRqmw7ghfRB4H2kNW/Zqq/m2+O6SqjwKPAhxtbNN8798wDKNUWVfwVfWhLbbRD/Rk/d2dWWYYhmFsI9th0nkaOCwi+0UkBrwHeGwb2jUMwzCy2Kpb5g+JSB9wH/BFEflKZnmniHwJQFXTwAeArwAvA3+lqie21m3DMAxjs2zVS+dzwOdClg8Ab836+0vAl7bSlmEYhrE1LNLWMAyjRDDBNwzDKBFM8I1rpqe9d9Wy6G2Vqzc8sH/VolOTZ1ct+8yF1XEXT33Fv6a+GYaxGhN845pYFPvKA1c9bhfF3mndc3XDjNhLRevSoqtif/XmsCj25W790rJFsW+rqM1bvw2jlDHBNzaNib1h7ExM8I1NYWJvGDsXE3xjw5jYG8bOxgTf2BAm9oax8zHBN9bFxN4wdgcm+MaamNgbxu7BBN/IiYm9YewuTPCNUEzsDWP3saXkacbupKe9d5nQQyD2y4QeTOwNY4dhgm8sETaqh9xiny30sCj2y1MrfOZCYpnQQyD2JvSGsf2Y4BvA9phwwMTeMAqJCb5h9nrDKBFs0rbEMbE3jNLBBL+EMbE3jNLCBL9EMbE3jNLDBL8EMbE3jNLEBL/EMLE3jNLFBL+EMLE3jNJmS4IvIu8UkRMi4ovInWts1ysiL4nI8yLyzFbaNK4NE3vDMLbqh38c+JfAn21g2zeq6ugW2zOuARN7wzBgi4Kvqi8DiEh+emPkHRN7wzAW2a5IWwW+KiIK/JmqPpprQxF5BHgEoKOyZpu6tzvZTBK08Lw4YGJvGLuHdQVfRB4H2kNW/Zqq/u0G23lAVftFpBX4BxE5papPhm2YuRk8CnC0sU03uH8jiw2P6sGSoBlGCbGu4KvqQ1ttRFX7M/8PC3wx4QAAAyBJREFUi8jngLuBUME3toYlQTMMIxfX3aQjIlWAo6ozmfffB/zu9W63FDF7vWEYa7FVt8wfEpE+4D7giyLylczyThH5UmazNuCbIvIC8F3gi6r691tp11iNib1hGOuxVS+dzwGfC1k+ALw18/4CcPtW2jHWxsTeMIyNYJG2OxwTe8MwNooJ/g7GxN4wjM1ggr9DMbE3DGOziGrxurqLyAhwcY1NmgFL1xBg5+Iqdi6uYufiKqVyLvaqakvYiqIW/PUQkWdUNWfStlLCzsVV7Fxcxc7FVexcmEnHMAyjZDDBNwzDKBF2uuDnTMJWgti5uIqdi6vYubhKyZ+LHW3DNwzDMDbOTh/hG4ZhGBvEBN8wDKNE2NGCLyJ/ICKnRORFEfmciNSv/6ndyUbrC+9mROQtInJaRM6JyAcL3Z9CIiJ/ISLDInK80H0pJCLSIyJfF5GTmd/HLxa6T4VkRws+8A/ALap6G3AG+FCB+1NIFusLl2SdARFxgT8GfgC4GXiviNxc2F4VlI8Bbyl0J4qANPCrqnozcC/wC6X8vdjRgq+qX1XVdObP7wDdhexPIVHVl1X1dKH7UUDuBs6p6gVVTQKfAh4ucJ8KRqai3Hih+1FoVHVQVb+XeT8DvAx0FbZXhWNHC/4Kfgr4cqE7YRSMLuBy1t99lPAP21iNiOwDXgX8c2F7Uji2q4j5NbORmroi8msEj26f2M6+bTd5qi9sGCWHiFQDfwP8kqpOF7o/haLoBX+9mroi8n7gbcCDusuDCvJRX3gX0w/0ZP3dnVlmlDgiEiUQ+0+o6mcL3Z9CsqNNOiLyFuDfA+9Q1Xih+2MUlKeBwyKyX0RiwHuAxwrcp/+/vTu0QSiIoiA6twDqoAgckh5QWGpBITBoEjogFID9htAAfSwCLAnuhbw5FVw12WSTXRVLEuAIPMYYu+o91f46+MAemAHXJFOSQ/WgKt/+F+7ic3m/BS68L+bOY4x77ao6SU7ADZgneSbZVG8qsgDWwPLTiCnJqnpUFZ9WkKQm/v2EL0n6kcGXpCYMviQ1YfAlqQmDL0lNGHxJasLgS1ITL/AIO6xqoI12AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Look how pretty! The neural network is able to fit the data in a way that isn't possible for linear models."
      ],
      "metadata": {
        "id": "8pRsJTAyv0Ce"
      }
    }
  ]
}