RL_part_3.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Anjali001/Reinforcement-Learning/blob/main/RL_part_3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rNuohp44N00i"
      },
      "source": [
        "# The Assignment\n",
        "\n",
        "### Objectives\n",
        "\n",
        "You will be guided through the implementation of a full deep reinforcement learning agent."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nVBcO5mAV9Ow"
      },
      "source": [
        "## Setup\n",
        "\n",
        "Run all the cells in this section, but do not modify them."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z1p0fpbxQLyn"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ps5OnkPmDbMX"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "np.set_printoptions(precision=3, suppress=1)\n",
        "plt.style.use('seaborn-notebook')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3lpweIqAWBX3"
      },
      "source": [
        "## Helpers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iIRC73HLq6VH"
      },
      "source": [
        "# A) Actor-critics\n",
        "\n",
        "You are going to implement an Actor-critic agent that updates a policy parametrised as a deep neural network.\n",
        "\n",
        "The agent learns online from a single stream of experience, updating the parametes of its policy after each transition in the environment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KV03Q3MpveUM"
      },
      "source": [
        "### Install packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wc-kqp3tveUT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "276afc99-dfcd-4b65-e78f-5453c9188364"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: jaxlib==0.3.0 in /usr/local/lib/python3.7/dist-packages (0.3.0+cuda11.cudnn805)\n",
            "Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from jaxlib==0.3.0) (1.0.0)\n",
            "Requirement already satisfied: flatbuffers<3.0,>=1.12 in /usr/local/lib/python3.7/dist-packages (from jaxlib==0.3.0) (2.0)\n",
            "Requirement already satisfied: numpy>=1.19 in /usr/local/lib/python3.7/dist-packages (from jaxlib==0.3.0) (1.21.5)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from jaxlib==0.3.0) (1.4.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py->jaxlib==0.3.0) (1.15.0)\n",
            "fatal: destination path 'bsuite' already exists and is not an empty directory.\n",
            "Processing ./bsuite\n",
            "\u001b[33m  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.\n",
            "   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.\u001b[0m\n",
            "Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (1.0.0)\n",
            "Requirement already satisfied: dm_env in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (1.5)\n",
            "Requirement already satisfied: immutabledict in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (2.2.1)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (3.2.2)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (1.21.5)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (1.3.5)\n",
            "Requirement already satisfied: plotnine in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (0.6.0)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (1.4.1)\n",
            "Requirement already satisfied: scikit-image in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (0.18.3)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (1.15.0)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.7/dist-packages (from bsuite==0.3.5) (1.1.0)\n",
            "Requirement already satisfied: dm-tree in /usr/local/lib/python3.7/dist-packages (from dm_env->bsuite==0.3.5) (0.1.6)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->bsuite==0.3.5) (1.3.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->bsuite==0.3.5) (0.11.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->bsuite==0.3.5) (3.0.7)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->bsuite==0.3.5) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->bsuite==0.3.5) (2018.9)\n",
            "Requirement already satisfied: patsy>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from plotnine->bsuite==0.3.5) (0.5.2)\n",
            "Requirement already satisfied: statsmodels>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from plotnine->bsuite==0.3.5) (0.10.2)\n",
            "Requirement already satisfied: descartes>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from plotnine->bsuite==0.3.5) (1.1.0)\n",
            "Requirement already satisfied: mizani>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from plotnine->bsuite==0.3.5) (0.6.0)\n",
            "Requirement already satisfied: palettable in /usr/local/lib/python3.7/dist-packages (from mizani>=0.6.0->plotnine->bsuite==0.3.5) (3.3.0)\n",
            "Requirement already satisfied: pillow!=7.1.0,!=7.1.1,>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->bsuite==0.3.5) (7.1.2)\n",
            "Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->bsuite==0.3.5) (2.6.3)\n",
            "Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from scikit-image->bsuite==0.3.5) (1.2.0)\n",
            "Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->bsuite==0.3.5) (2.4.1)\n",
            "Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.7/dist-packages (from scikit-image->bsuite==0.3.5) (2021.11.2)\n",
            "Building wheels for collected packages: bsuite\n",
            "  Building wheel for bsuite (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for bsuite: filename=bsuite-0.3.5-py3-none-any.whl size=250321 sha256=d222c6113087d6aae3e4f55f51b7c3a0d149529e938ad0eb1e9f0306dbe7bb90\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-qg3_0ji2/wheels/9b/17/99/f06e5281ca86b7afe85c0515439b017781ea1dbec73e020501\n",
            "Successfully built bsuite\n",
            "Installing collected packages: bsuite\n",
            "  Attempting uninstall: bsuite\n",
            "    Found existing installation: bsuite 0.3.5\n",
            "    Uninstalling bsuite-0.3.5:\n",
            "      Successfully uninstalled bsuite-0.3.5\n",
            "Successfully installed bsuite-0.3.5\n",
            "Requirement already satisfied: dm-haiku==0.0.6 in /usr/local/lib/python3.7/dist-packages (0.0.6)\n",
            "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.7/dist-packages (from dm-haiku==0.0.6) (1.21.5)\n",
            "Requirement already satisfied: absl-py>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from dm-haiku==0.0.6) (1.0.0)\n",
            "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.7/dist-packages (from dm-haiku==0.0.6) (0.8.9)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from dm-haiku==0.0.6) (3.10.0.2)\n",
            "Requirement already satisfied: jmp>=0.0.2 in /usr/local/lib/python3.7/dist-packages (from dm-haiku==0.0.6) (0.0.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.7.1->dm-haiku==0.0.6) (1.15.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install -U jaxlib==0.3.0\n",
        "!git clone https://github.com/deepmind/bsuite.git\n",
        "!pip install bsuite/\n",
        "!pip install dm-haiku==0.0.6"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q442-kGUvTf4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a096941e-2af4-467b-8979-c86e765af9c1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: optax in /usr/local/lib/python3.7/dist-packages (0.1.1)\n",
            "Requirement already satisfied: jaxlib>=0.1.37 in /usr/local/lib/python3.7/dist-packages (from optax) (0.3.0+cuda11.cudnn805)\n",
            "Requirement already satisfied: chex>=0.0.4 in /usr/local/lib/python3.7/dist-packages (from optax) (0.1.1)\n",
            "Requirement already satisfied: absl-py>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from optax) (1.0.0)\n",
            "Requirement already satisfied: typing-extensions>=3.10.0 in /usr/local/lib/python3.7/dist-packages (from optax) (3.10.0.2)\n",
            "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.7/dist-packages (from optax) (1.21.5)\n",
            "Requirement already satisfied: jax>=0.1.55 in /usr/local/lib/python3.7/dist-packages (from optax) (0.3.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.7.1->optax) (1.15.0)\n",
            "Requirement already satisfied: dm-tree>=0.1.5 in /usr/local/lib/python3.7/dist-packages (from chex>=0.0.4->optax) (0.1.6)\n",
            "Requirement already satisfied: toolz>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from chex>=0.0.4->optax) (0.11.2)\n",
            "Requirement already satisfied: scipy>=1.2.1 in /usr/local/lib/python3.7/dist-packages (from jax>=0.1.55->optax) (1.4.1)\n",
            "Requirement already satisfied: opt-einsum in /usr/local/lib/python3.7/dist-packages (from jax>=0.1.55->optax) (3.3.0)\n",
            "Requirement already satisfied: flatbuffers<3.0,>=1.12 in /usr/local/lib/python3.7/dist-packages (from jaxlib>=0.1.37->optax) (2.0)\n",
            "Requirement already satisfied: rlax in /usr/local/lib/python3.7/dist-packages (0.1.2)\n",
            "Requirement already satisfied: absl-py>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from rlax) (1.0.0)\n",
            "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.7/dist-packages (from rlax) (1.21.5)\n",
            "Requirement already satisfied: chex>=0.0.8 in /usr/local/lib/python3.7/dist-packages (from rlax) (0.1.1)\n",
            "Requirement already satisfied: distrax>=0.0.2 in /usr/local/lib/python3.7/dist-packages (from rlax) (0.1.1)\n",
            "Requirement already satisfied: jax>=0.1.55 in /usr/local/lib/python3.7/dist-packages (from rlax) (0.3.1)\n",
            "Requirement already satisfied: jaxlib>=0.1.37 in /usr/local/lib/python3.7/dist-packages (from rlax) (0.3.0+cuda11.cudnn805)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.9.0->rlax) (1.15.0)\n",
            "Requirement already satisfied: dm-tree>=0.1.5 in /usr/local/lib/python3.7/dist-packages (from chex>=0.0.8->rlax) (0.1.6)\n",
            "Requirement already satisfied: toolz>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from chex>=0.0.8->rlax) (0.11.2)\n",
            "Requirement already satisfied: tensorflow-probability>=0.15.0 in /usr/local/lib/python3.7/dist-packages (from distrax>=0.0.2->rlax) (0.16.0)\n",
            "Requirement already satisfied: scipy>=1.2.1 in /usr/local/lib/python3.7/dist-packages (from jax>=0.1.55->rlax) (1.4.1)\n",
            "Requirement already satisfied: opt-einsum in /usr/local/lib/python3.7/dist-packages (from jax>=0.1.55->rlax) (3.3.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from jax>=0.1.55->rlax) (3.10.0.2)\n",
            "Requirement already satisfied: flatbuffers<3.0,>=1.12 in /usr/local/lib/python3.7/dist-packages (from jaxlib>=0.1.37->rlax) (2.0)\n",
            "Requirement already satisfied: gast>=0.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability>=0.15.0->distrax>=0.0.2->rlax) (0.5.3)\n",
            "Requirement already satisfied: cloudpickle>=1.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability>=0.15.0->distrax>=0.0.2->rlax) (1.3.0)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability>=0.15.0->distrax>=0.0.2->rlax) (4.4.2)\n"
          ]
        }
      ],
      "source": [
        "# solution\n",
        "!pip install optax\n",
        "!pip install rlax"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r_pTfi5dSFX5"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "huyKrYpvSHSu"
      },
      "outputs": [],
      "source": [
        "from bsuite.environments import catch\n",
        "\n",
        "import jax\n",
        "import jax.numpy as jnp\n",
        "import haiku as hk"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QCWBfrXLjM4p"
      },
      "outputs": [],
      "source": [
        "# solution\n",
        "import optax\n",
        "import rlax"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M6kEki4XHbPy"
      },
      "source": [
        "### Plotting"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EYdWwRrbHbcl"
      },
      "outputs": [],
      "source": [
        "def plot_learning_curve(list_of_episode_returns):\n",
        "  \"\"\"Plot the learning curve.\"\"\"\n",
        "  plt.figure(figsize=(7, 5))\n",
        "\n",
        "  def moving_average(x, w):\n",
        "    return np.convolve(x, np.ones(w), 'valid') / w\n",
        "\n",
        "  smoothed_returns = moving_average(list_of_episode_returns, 30)\n",
        "  plt.plot(smoothed_returns)\n",
        "\n",
        "  plt.xlabel('Average episode returns')\n",
        "  plt.xlabel('Number of episodes')\n",
        "\n",
        "  ax = plt.gca()\n",
        "  ax.spines['left'].set_visible(True)\n",
        "  ax.spines['bottom'].set_visible(True)\n",
        "  ax.spines['right'].set_visible(False)\n",
        "  ax.spines['top'].set_visible(False)\n",
        "  ax.xaxis.set_ticks_position('bottom')\n",
        "  ax.yaxis.set_ticks_position('left')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xtlrr5d2p7cS"
      },
      "source": [
        "### Neural networks\n",
        "\n",
        "You will use JAX to define a network parametrising:\n",
        "\n",
        "* The values of each state $v(s)$.\n",
        "* The action preferences in each state $\\{p_i(s)\\}_{i\\in\\{1, ..., |A|\\}}$ (you can think of and implement the preferences $\\mathbf{p}(s)$ as a vector output with $|A|$ elements).\n",
        "\n",
        "You will use `Haiku` (https://github.com/deepmind/dm-haiku) to define the network. You will need to:\n",
        "* define the forward pass of the network as some function `fn`\n",
        "* Use `hk.transform(fn)` to convert this in a pair of functions `init_net` and `apply_net`):\n",
        "\n",
        "\n",
        "        init_net, apply_net = hk.transform(fn)\n",
        "\n",
        "The `init` function has signature `parameters = init_net(key, obs)`, where\n",
        "  * `key` is a Jax random-number-generator key that we will generate with `jax.random.PRNGKey`,\n",
        "  * `observation` is the observation of the agent, which will be flattened into a vector by the network, as described below.\n",
        "\n",
        "The `init_net` returns randomly sampled weights for the neural network:\n",
        "\n",
        "        parameters = init_net(jax.random.PRNGKey(1234), observation)\n",
        "\n",
        "The `apply_net` will have signature `v, p = apply_net(parameters, current_key, obs)` \n",
        "  * `parameters`: the (current) parameters of the ageent,\n",
        "  * `key`: a Jax random keym, e.g., generated with `current_key, key = jax.random.split(key)` (we then keep `key`, which is also a new key, around to split again the next time we need a random key),\n",
        "  * `observation`: the observation of the agent.\n",
        "\n",
        "The `apply_net` function then returns a **scalar** value `v` and a **vector** of preferences `p`, which will define the policy of the agent.\n",
        "\n",
        "A lot of the boilerplate code has been written for you.  You just have to implement some functions as indicated in the questions below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gj1nsUYuOoe1"
      },
      "source": [
        "### Q1 [2 marks]\n",
        "\n",
        "Define the forward pass of the neural network. The network must:\n",
        "\n",
        "* take an `observation` as input\n",
        "* reshape the observation into a flat vector `flat_obs`\n",
        "* compute a hidden representation `h = relu(W.dot(flat_obs) + b)`, where `relu(x)` is a 'rectifier linear unit', which computes $\\max(x, 0)$ elementwise (you can just use `jax.nn.relu(x)` to implement this),\n",
        "* compute a vector of action preferences as a linear function of `h`\n",
        "* compute a scalar state value as a linear function of `h`\n",
        "* return the scalar value and vector preferences.\n",
        "\n",
        "Note:\n",
        "* The hidden representation should be a vector of 50 elements.\n",
        "* The action preferences should be a vector of 3 elements (one per each available action).\n",
        "* The value should be a scalar (not a vector with one element).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vRBP0xjtQvTi"
      },
      "outputs": [],
      "source": [
        "def forward_pass(observation):\n",
        "  # Implement forward pass here\n",
        "  flat_obs = jnp.reshape(observation, (-1,))\n",
        "  linear = hk.Linear(50, with_bias=True)\n",
        "  linear_actions = hk.Linear(3)\n",
        "  linear_value = hk.Linear(1)\n",
        "  h = jax.nn.relu(linear(flat_obs))\n",
        "  p = linear_actions(h) #actionpreferncees\n",
        "  v = linear_value(h)[0] #valkue\n",
        "  return v,p\n",
        "\n",
        "# Our forward pass will be deterministic, so we apply `hk.without_apply_rng`.\n",
        "init_net, apply_net = hk.without_apply_rng(hk.transform(forward_pass))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ImJUgzFosvD"
      },
      "source": [
        "### Choosing actions\n",
        "\n",
        "A critical component of an actor-critic agent is a (stochastic) policy, mapping `observations` to `actions`. \n",
        "\n",
        "In deep RL, this mapping is conventionally parametrised by a deep neural network."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1MVXNWwlYW24"
      },
      "source": [
        "### Q2 [2 marks]\n",
        "\n",
        "Implement a softmax policy parametrised by the neural network above (i.e., using the `apply_net` function).\n",
        "\n",
        "The function has signature `action = softmax_policy(parameters, key, obs)`, taking the current network parameters `parameters`, a JAX random `key` and the current `observation`. It should return an `action` sampled from a softmax distribution, so that the probability of selecting action $a$ with preference $p(s, a)$ is proportional to $\\exp(p(s, a))$.\n",
        "\n",
        "Functions to perform random sampling in JAX (e.g. those in `jax.random`) take a random key as input, and they are deterministic function of such a key. In general, in a JAX program you need to use the `jax.random.split` function to generate new random keys before every new sampling. The run loop that runs the experiment later on splits the key between consecutive calls to the `policy`. Thus you can assume that a new random `key` is provided to you on each call to the `policy`. If, however, you find that you need a random key in multiple places within the `policy`, do remember to split the key before each use, using `key1, key2 = jax.random.split(key)`.\n",
        "\n",
        "Note that we 'jit' the function.  This means the function will be compiled, which will make it run faster. This does also suppress print statements, so if you are debugging and want to print please comment out the `@jax.jit` line, but don't forget to put it back before running the experiment below, and especially before submitting your assignment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x-FSOYNOYnUx"
      },
      "outputs": [],
      "source": [
        "@jax.jit\n",
        "def softmax_policy(parameters, key, obs):\n",
        "  \"\"\"Sample action from a softmax policy.\"\"\"\n",
        "  _, action_preferences = apply_net(parameters, obs)\n",
        "  action_probabilities = jax.nn.softmax(action_preferences)\n",
        "  key1, key2 = jax.random.split(key, 2)\n",
        "  action = jax.random.choice(key2, jnp.arange(3), p = action_probabilities)\n",
        "  return action"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iMYcb7Y9krnT"
      },
      "source": [
        "### Learning values and policies\n",
        "\n",
        "An actor-critic agent requires to update the parameters of the network so as to simultaneously improve the value predictions and the policy.\n",
        "\n",
        "In the next section you will define the gradient updates for each of these two components."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GsIlpmNEk5fv"
      },
      "source": [
        "### Q3 [4 marks]\n",
        "\n",
        "Implement a function to compute a stochastic estimate of the policy gradient from a 1 step transition in the environment.\n",
        "\n",
        "* You will use $R_{t+1} + \\gamma v(S_{t+1})$ as an estimate of $q_{\\pi}(S_t, A_t)$\n",
        "* You will use $v(S_{t})$ as a baseline to reduce the variance of the updates.\n",
        "\n",
        "In the code we actually use names `obs_tm1`, `a_tm1` (where `tm1` stands for '$t$ minus one') for the observation and action at time $t-1$, and `r_t`, `discount_t`, `obs_t` for the reward, discount, and observation at time $t$. So the code is offset a single time step in terms of naming as compared to the typical mathematical formulations.  This is just a naming convention, and should not impact the algorithm.\n",
        "\n",
        "The function below must therefore have signature `grads = policy_gradient(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t)`.\n",
        "* Where the inputs are:\n",
        "  * `parameters`: the parameters of the network,\n",
        "  * an observation `obs_tm1`\n",
        "  * the action `a_tm1` selected after observing `obs_tm1`,\n",
        "  * the resulting reward `r_t` and discount `discount_t` and observation `obs_t`, as obsesrved after taking action `a_tm1`.\n",
        "\n",
        "This function should return a stochastic estimate of the policy gradient, where `grads` has the same structure as `parameters` and contains an estimate of the gradient of the expected episodic return wrt to each parameter.\n",
        "\n",
        "The policy-gradient estimate should use bootstrapping, using the value estimates that can be gotten using the saame `parameters` as used for the policy.  So the output of this function can be used in a one-step actor-critic update."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "02RJAKiX85Wx"
      },
      "source": [
        "\n",
        "### Jax hint:\n",
        "Note that you can use `jax.grad(f)` to get the gradient of any (pure) jax function with a scalar output.  For instance, consider:\n",
        "\n",
        "        def f(w, x, y):\n",
        "          # w, x, and y are all vectors\n",
        "          return jnp.sum(w*x + y)\n",
        "\n",
        "        df = jax.grad(f)\n",
        "    \n",
        "then calling `df(w, x, y)` will give the gradient of the output of `f(w, x, y)` with respect to the first input argument --- here called `w`. You can use this new function `df` as just a normal function. For instance, it can be called from other functions, as usual.\n",
        "\n",
        "Run the cell below to see a concrete example in action.  Note that `df(w, x, y)` in the example below evaluates to the same values as `x`.  This is correct, because the gradient of `f` with respect to `w` is indeed `x`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t1TjveqqGnNA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b7d38a31-7a70-47c7-abfd-930aaf7f8041"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "f(w, x, y):  31.0\n",
            "df(w, x, y): [3. 5.]\n"
          ]
        }
      ],
      "source": [
        "def f(w, x, y):\n",
        "  return jnp.sum(w*x + y)\n",
        "\n",
        "df = jax.grad(f)\n",
        "\n",
        "w = jnp.array([1., 2.])\n",
        "x = jnp.array([3., 5.])\n",
        "y = jnp.array([7., 11.])\n",
        "\n",
        "print(f'f(w, x, y):  {f(w, x, y)}')\n",
        "print(f'df(w, x, y): {df(w, x, y)}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wngNEiwrQeF0"
      },
      "source": [
        "### Jax hint:\n",
        "if you have a Haiku network with parameters `w`, then these parameters will typically be a structured dictionary containing the actual weight vectors and matrices that make up the network's parameters.  Suppose you have a function that computes the output of a network, and then uses this to do some stuff, and you want the gradient of that function:\n",
        "\n",
        "        # Define network\n",
        "        def some_haiku_net(...):\n",
        "          ...\n",
        "\n",
        "        # Transform to get init and apply\n",
        "        init, apply = hk.without_apply_rng(hk.transform(some_haiku_net))\n",
        "\n",
        "        # Define a new function\n",
        "        def some_function(w, x, y):\n",
        "           ...some stuff...\n",
        "           # compute output of the network:\n",
        "           output = apply(w, x)\n",
        "           ...some more stuff that uses output...\n",
        "           return ...some_scalar...\n",
        "        \n",
        "        grad_function = jax.grad(some_function)\n",
        "\n",
        "As before, `grads = grad_function(w, x, y)` should now give the gradients of `some_function` with respect to the first input argument, `w`.  These gradients `grads` will have exactly the same shape as the input argment `w`.  But we cannot do things like `w + grads`, because `w` and `grads` are both dictionaries, not just `jnp.array`s.  Instead, we can use Jax' inbuilt tree utils.  For instance, to add the gradients to the weights, we can use:\n",
        "`new_weights = jax.tree_map(lambda w_i, g_i: w_i + 0.1 * g_i, w, grads)`\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AHLGjy8he6vh"
      },
      "outputs": [],
      "source": [
        "def policy_gradient(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t):\n",
        "  def some_function(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t):\n",
        "    v_t, _ = apply_net(parameters, obs_t)\n",
        "    v_tm1, h_tm1 = apply_net(parameters, obs_tm1)\n",
        "\n",
        "    target_tm1 = r_t + discount_t * v_t\n",
        "    td_error = jax.lax.stop_gradient(target_tm1 - v_tm1)\n",
        "    log_act_pref = h_tm1[a_tm1] - jax.lax.stop_gradient(jax.nn.softmax(h_tm1))@h_tm1\n",
        "\n",
        "    return td_error * log_act_pref\n",
        "  grads = jax.grad(some_function)(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t)\n",
        "  return grads"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5XgjefrBlIGN"
      },
      "source": [
        "### Q4 [4 marks]\n",
        "\n",
        "Implement a function to compute a TD(0) update for the parameters of the value function.\n",
        "\n",
        "It must have signature `td_update = value_update(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t)`.\n",
        "* Where the inputs are:\n",
        "  * the current parameters `parameters` of the network,\n",
        "  * an observation `obs_tm1`\n",
        "  * the action `a_tm1` selected after observing `obs_tm1`,\n",
        "  * the resulting reward `r_t` and environment discount `discount_t` \n",
        "  * and the following observation `obs_t`\n",
        "* Returns a stochastic TD(0) semi-gradient update: `td_update` has the same structure as `parameters`. This contains a stochastic estimate of the negative semi-gradient of the expected value prediction loss: a TD(0) update."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CAONu72iTpzx"
      },
      "outputs": [],
      "source": [
        "def value_update(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t):\n",
        "  def some_function(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t):\n",
        "    v_t, _ = apply_net(parameters, obs_t)\n",
        "    v_tm1, _ = apply_net(parameters, obs_tm1)\n",
        "    \n",
        "    target_tm1 = r_t + discount_t * v_t\n",
        "    td_error = jax.lax.stop_gradient(target_tm1) - v_tm1\n",
        "    return -1*(td_error)**2\n",
        "\n",
        "  td_update =  jax.grad(some_function)(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t)\n",
        "  return td_update"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jAXgLZg8mfMd"
      },
      "source": [
        "### Updating shared parameters\n",
        "\n",
        "The policy gradient identifies the direction of change in the parameters that most steeply improve the policy.\n",
        "The value update identifies the direction of change in the parameters that improves the value predictions (according to TD).\n",
        "\n",
        "However, the value and policy share some of the parameters of the network.  How do we combine the two gradient updates?\n",
        "\n",
        "In this assignment, we simply sum the policy and value components.\n",
        "The function that combines the two gradients is implemented for you in the cell below.  Note the use of `jax.tree_multimap` to facilitate adding the structured parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uhKCLe8jjkdZ"
      },
      "outputs": [],
      "source": [
        "@jax.jit\n",
        "def compute_gradient(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t):\n",
        "  pgrads = policy_gradient(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t)\n",
        "  td_update = value_update(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t)\n",
        "  return jax.tree_multimap(lambda pg, td: pg + td, pgrads, td_update)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tYaVb6GcpCRe"
      },
      "source": [
        "### Optimisation\n",
        "\n",
        "In deep learning, gradient updates are typically rescaled and modifed to avoid taking too large a step on a single update (e.g., due to large variance), and to facilitate the optimisation process (it turns out raw stochatic gradients are often not the most effective for updating neural networks).\n",
        "\n",
        "For instance given a candidate gradient update $\\nabla$ we may update our parameters $\\theta$ by;\n",
        "$$\\Delta \\theta = \\theta + \\alpha * \\nabla\\,,$$\n",
        "where $\\alpha$ is a small number between 0 and 1 (e.g., $\\alpha=0.01$ or $\\alpha=0.001$), referred to as `step_size` or `learning_rate`\n",
        "\n",
        "The gradients with respect to each weight of a neural network may however have very different magnitudes. This can make it hard to set a suitable learning rate $\\alpha$.\n",
        "\n",
        "In deep learning, and deep RL, we typically use adaptive learning rates, for instance by rescaling each component of the gradient using statistics tracking the typical size of the updates to that weight. Then the entire update is rescaled using a global `learning_rate` $\\alpha$.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hbFePWfApaMU"
      },
      "source": [
        "### Q5 [2 marks]\n",
        "\n",
        "A popular approach to adaptive gradient rescaling was introduced by the `Adam` algorithm.\n",
        "This algorithm implements the following procedure before applying each update:\n",
        "* Increase an update counter $k \\gets k+1$ (starting at k=0 before any updates),\n",
        "* Update the first moment of each gradient component $\\mu \\gets (1 - \\beta_1) g + \\beta_1 \\mu$ where $g$ is the latest stochastic gradient, where $\\beta_1$ is a parameter for the moving average.\n",
        "* Update the second moment of each gradient component $\\nu_i = (1 - \\beta_2) g_i ^ 2 + \\beta_2 \\nu_i $ where $g$ is the latest gradient update, where $\\beta_2$ is a parameter for the moving average.\n",
        "* Use the following update to update the weights:\n",
        "$$\\Delta w = \\alpha \\frac{\\mu / (1 - \\beta_1 ^ {k})}{\\epsilon + \\sqrt{\\nu / (1 - \\beta_1 ^ {k})}}$$\n",
        "* $\\alpha$ is a global `learning rate`\n",
        "* $\\beta_1$ and $\\beta_2$ define a soft horizon for the per-weight statistics.\n",
        "* $\\epsilon$ makes the rescaling more robust to numerical issues.\n",
        "\n",
        "(See [Kingma et al, 2014](https://arxiv.org/abs/1412.6980) for details, if you are interested.)\n",
        "\n",
        "In the next cell define a pair of functions (`opt_init`, and `opt_update` --- in each case `opt` is short for 'optimiser'), where:\n",
        "\n",
        "The `opt_init` function has signature `opt_state = opt_init(parameters)`.\n",
        "* Takes the network parameters as inputs\n",
        "* Initialises an `optimiser state` holding the per weight statistics.\n",
        "\n",
        "The `opt_update` function has signature `updates, opt_state = opt_update(grads, opt_state)`.\n",
        "* Takes a `gradient` and an `optimisers state`,\n",
        "* and returns the transformed gradient and the updated `optimiser state`.\n",
        "\n",
        "The optimiser state `opt_state` should contain:\n",
        "* The first-order momentum $\\mu$, as updated with a moving-average-parameter $\\beta_1$ which we call `b1` in the code.\n",
        "* The first-order momentum $\\nu$, as updated with a moving-average-parameter $\\beta_2$ which we call `b2` in the code.\n",
        "\n",
        "We will ignore the initial correction Adam typically uses, and will instead use the simpler transformation:\n",
        "$$\\Delta w = \\alpha \\frac{\\mu}{\\epsilon + \\sqrt{\\nu}}$$\n",
        "\n",
        "Set the algorithm's hyper-parameters to $\\alpha=0.003$, $\\beta_1=.9$ and $\\beta_2=.999$, $\\epsilon=10^{-4}$.  You are allowed to hard-code these in, or make them configurable (e.g., pass them as additional arguments to `opt_update`, which is better if you want to play around, of course).  Set the initial moving averages to zero for $\\mu$ and one for $\\nu$.  E.g.,\n",
        "\n",
        "        mu = jax.tree_multimap(jnp.zeros_like, parameters)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "017XRA_BpbZx"
      },
      "outputs": [],
      "source": [
        "def opt_init(parameters):\n",
        "  k = 0\n",
        "  mu = jax.tree_map(jnp.zeros_like, parameters)\n",
        "  v = jax.tree_map(jnp.zeros_like, parameters)\n",
        "  return mu,v,k\n",
        "\n",
        "def opt_update(grads, opt_state):\n",
        "  mu,v,k = opt_state\n",
        "\n",
        "  alpha = 0.003\n",
        "  b1 = 0.9\n",
        "  b2 = 0.999\n",
        "  epsilon = 1e-4\n",
        "\n",
        "  k += 1\n",
        "  mu = jax.tree_multimap(lambda g, m: (1 - b1) * g + b1 * m, grads, mu)\n",
        "  v = jax.tree_multimap(lambda g, v: (1 - b2) * jnp.square(g) + b2 * v, grads, v)\n",
        "  \n",
        "  mu_hat = jax.tree_map(lambda m: m / (1 - jnp.asarray(b1 ** k, m.dtype)), mu)\n",
        "  v_hat = jax.tree_map(lambda v: v / (1 - jnp.asarray(b2 ** k, v.dtype)), v)\n",
        "\n",
        "  updates = jax.tree_multimap(lambda m, v: alpha * m / (jnp.sqrt(v) + epsilon), mu_hat, v_hat)\n",
        "  return updates, (mu,v,k)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DczWvZfNSnTj"
      },
      "source": [
        "### Run experiments\n",
        "\n",
        "Run the cell below to show the performance of the resulting agent.\n",
        "\n",
        "You may also use this section for debugging your implementations.\n",
        "\n",
        "Note however, that most functions are `jitted` for performance,\n",
        "* either using the `@jax.jit` decorator in the function definition\n",
        "* or calling explicitely `fn = jax.jit(fn)`\n",
        "\n",
        "When jitting, the code is compiled on the first time the function is executed\n",
        "* and execution is much faster on subsequent calls.\n",
        "* a notable side effect is that print statements in a jitted function will only execute on the first execution of the function.\n",
        "* to drop into a debugger or print on each function execution you will have to disable the `@jax.jit` annotations and jax.jit calls."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Fz837XTkLxE8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "outputId": "de4205ec-963e-4d55-e595-86241a79ff5a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training agent for 1500 episodes...\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 504x360 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAb4AAAFCCAYAAACO3biLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd5gc1bG3fzVxkzYr7CquAiggJIEkgshBBmOQcSD42gYM9ufrnC8YDLLBvhgcr41tsE2wwYDBxshEEySyEqCMctYqrlbS5knn+6PDnO7p7ukJO2Gn3ufZZ2e6T3dXhznVVaeqDgkhwDAMwzClgiffAjAMwzBMLmHFxzAMw5QUrPgYhmGYkoIVH8MwDFNSsOJjGIZhSgpWfAzDMExJkRXFR0T3E9EBIlpjs56I6P+IaDMRrSKik6R11xDRJvXvmmzIwzAMwzB2ZMviexDARQ7rLwYwQf37AoDfAwAR1QO4DcApAGYDuI2I6lI5MBG9kIa8DMMwTIniy8ZOhBCvE9EYhybzAPxFKNnyi4moloiaAJwD4CUhxGEAIKKXoCjQR90e++STT/4QAM7CZxiGYWTIbkWuxviGA9glfd+tLrNbzjAMwzD9QlEGtxDRfCISRCRaW1vzLQ7DMAxTRORK8e0BMFL6PkJdZrfcESHEfCEECSGoubk5q4IyDMMwA5tcKb4FAD6rRneeCuCoEGIvgBcBzCWiOjWoZa66jGEYhmH6hawEtxDRo1ACVRqJaDeUSE0/AAgh/gDgOQAfBrAZQDeA69R1h4nodgDL1F39SAt0YRiGYZj+gIp9WqKZM2eK5cuX51sMhmEYprDIe1QnwzAMwxQErPgYhmGYkoIVH8MwDFNSZCW4hWEYpr8QQiASEyAAPi+/qzOZw4qPYZiC5oaHluOV9Qfg9RB+9skTcfmMEfkWiSly+PWJYZiC5v1dRwAA0ZjA6t3H8iwNMxBgxccwTEETjsTg8yiR6eFoLM/SMAMBVnwMwxQ0oWgMlUFlVIYVH5MNWPExDFOwCCEQisZQpSq+ECs+Jguw4mMYpmCJxgSEACoCXgBAKMKKj8kcVnwMwxQs4ahSUrGCXZ1MFmHFxzBMwaK5NquCisWnKUKGyQRWfAzDFCyaa7MiwBYfkz1Y8TEMU7B0hyIAgEp1jO9YbwQ72rrQG47mUyy0dfah2Ge2KWVY8TEMU7B8/i/KlGNlfi/K/B6s3HUEZ9+9CJf+5s28yfTsqr04+Y6X8btFW/ImA5MZrPgYhilYOnoVi++a08fgh5dNwRUzR6ChMoDtbV15k+m1jQcAAA8v3pE3GZjMYMXHMEzBEorEMG5wJSY1VePKWaNw1yemYdyQKoSjIu+uRvZ0Fi+s+BiGKVhC0Rj8phkZAur3fEV4kv3E3kyRwIqPYZiCJRyNIegzKT6fR1+XT4j1X9HCio9hmIIlHBUJFp/fq2icfFVxEVAsTXZ1Fi+s+BiGKUiiMYFozErxFYbFxxQvrPgYhilINMXmN7s6VcWXr4LVPMZX/LDiYximINEUW8Ac3OLLb3BLTPVx8hhf8cKKjykp+iJRvL+zHdsPxfPAth7sRGQAuM06esN4b2c7orGBMfgUVsfwAj6jhtFcnUd7wjmXCYhbonuP9mLPkZ6s7PNQZx929GNuYiQaw7s72tHZF+m3Y1gd8/2d7ViytQ0HOnot2+w63I0Dx6zX9Ses+JiS4roHluHy372Nc362CEe7w3hj00Gc9/PXcOuCtfkWLWOuf2g5Pva7t/HE8l35FiUraBadeYxPi/L8zSubci4TYLQ0L/7V61l50Zj327dw9t2L0NHbP8r8kSU78fHfv40vPfJev+zfinsWbsHlv3sbV963GB/+dWKlnb5IFGfetRBzf/V6zmTSYMXHlBRvb2nTPx/pCWHptsMAgMeW7syXSFlDO5eDHX15liQ7aFGbZsV39Smj8iGOjjy2eKw3kpUgG81y7C8rVrO43tx0sF/2b8UitcINoFi0Zrr7lHqrR7pzb7mz4mNKFrnDGhjOQYWBMku5dh5mxTeyrsKwPtdoCvmM8Y0AgL4splX017iltt9cPufJ0j3y+Zyy4mNKllBkIKm7OANF8WkvJuYE9nzn8WlyVepzBGZT8fXPORXCzPUxk0tYlinX49JZUXxEdBERbSCizUR0o8X6XxLRCvVvIxEdkdZFpXULsiEPw7hBVhADKUAvPEAUup7O4DXeHSKC30t5y+MLR2PwEFDuz77i6y8FlY9rZY56DceMMsgy5Vo+X6Y7ICIvgHsAXAhgN4BlRLRACLFOayOE+KbU/qsAZki76BFCTM9UDoZJlYGaAD1QzstujE9blq90hpBaTUZPpM/wRUO2dvrLWi+EZyIcFQj6jN/jn2MoU18kckE2LL7ZADYLIbYKIUIAHgMwz6H91QAezcJxGSYjwpEBOsZXAG6tbGA3xgcouXz5Os9QJIaA16Mn1oeimU2KK59HuJ/OKR/XyjzGZ5ZB/p5r+bKh+IYDkOOnd6vLEiCi0QBaALwqLS4jouVEtJiIPpoFeRjGFQNlLMxMIbzdZwPNIgj47Cy+/Lk6Az5PvIJMhhZfyODy69/glnxivl+5OG87ch3cchWAJ4UQ8ivSaCHETACfAvArIhqXbCdENJ+IBBGJ1tbW/pKVGeCEo0If2xtIY3wDRaHrCexWFp/Xk7fzDKtTJWVrlohcjHXl41qZx/jMVl0+x/iyofj2ABgpfR+hLrPiKpjcnEKIPer/rQAWwTj+Z4kQYr4QgoQQ1NzcnI7MDIOnV+zBP96ze1QLh7c3HzJUmrFCnpT1g73H+lWeDfs6ksqTKdGYwHOr9wJIDG7Rlu1u70FfJDM3Y6qEIjHsaOuG30e6XAtWtqY8KW7rkR48+e5u/HXxDjzw1jZ9+d0vbsB/1u5LiIBMhUg0hjc2HURvOH5t1rUqz4QQSKlSyr6jvfj3ylZ0pVHxZdP+TsP3dXuPoTccxRubDiIaE4Zn6Ml3d2PJ1jasaz2GJ9/d3e+KMBuKbxmACUTUQkQBKMotITqTiCYCqAPwjrSsjoiC6udGAHMArDNvyzD9wTOr9uqJwxWBjOO8+oXWIz341J+W4JP3vuPYbtXuo/rnLQf7Vyl96Fev45yfLerXY7y95RD++b7yUlJT4U9Y3xtWOkZNOeaKF9bu048fUZXTn9/chg37O1Laz/efWo3vPLESP/jXGtyzcIu+fN3eY/jCX9/F+7va05bxr4t34DN/XoqfPPeBvkwurfajZ9x3sV/523v46qPv46F3tqckgxAioTzaTf9cjR89sw6f+fNSPLJkB+6XFP6vX9mEK+9bjHn3vInvPLGy38f8MlZ8QogIgK8AeBHABwD+LoRYS0Q/IqLLpKZXAXhMGF+NJgFYTkQrASwEcKccDcow2aappsxy+cRhg3IsiTu0Sh7JqrEckSp+WFlIxUa7Ws3jtLEN+PDUpoT115w+BkDuq34c6Q4BAD576mjMGdcoLU9NjnaL9hOGVKW9P5l3dyhK841NhwAk5s+lUh1m5e4jKW8DGCNVH7nhFEUOIfDaBqVyzIqdRyyDlpzGdbNJVl5zhRDPAXjOtOxW0/f5Ftu9DWBqNmRgGDfEhMCYhgrMHFOPJ9/dDUD5kYULtLCzW5ePHA0YjgoIIUBFPH2Adj6XzxiOoC8xzP24oYqSyPXYkGaJHD9sEGolSzRVOayiN685fQxu+deatPYno9137e5r+XNnTmjEG5sOpWRNKVMwCURSDD7RFNi5xw/GnPGNmNJcjR1t3fFxP7I/x9kt9ZZKMZtw5RampAibcrAAIOj19FsYeaa47aTMwQv9FSWX6lhWusTn4rNW3n49ojLHik+aI1B+hlKVI5liy6QMmvkehaQgoXSDglI9P3MOpt+rpJ/Iotk9o1bBTNmGFR9TUoQjakSe5A70+/IXIZgMt3KZO9L+soQiObKM41VbrLsoXfHlOAxeS1YPmF6eUrb4krTP5ouLPMtFuhVvUt1GfkEArKNw7ZRpLlz1rPiYkiKk5WBJYwiBPOaEJcNtB+gUKp5NcnWd+hxSGQBkLZUgVbTjBXweQw3RVBWwVacve6az4eo07yvgU5Lu06k0k+qLoX5M9f5p98tNlZr+Ht8DWPExJYQQQlF8prd1v48K1tXpVi6zguwvCzZXrkXdSrHpBAN6ubD8KD6zuzxVOZIpyv6o/6nJnJ7Fl+oYn7HOql5Y3EXuXn+P7wGs+JgSIhoTEAJqDpak+LyenLvM3CJ3FE65XSE1n63M379jX7I8/TneZ7YYzGhjf7l2UfdF4h267JJL1yKSkS9nJvcvYYxPt/gIAa8nrfHDUIr5klZjfPJyQHlZ0J5XGR7jY5gsIo91mF2dqf6wc4XcQTp1rtq5ValVgHNR+qo/y0y5HePLm6tTqtWZjhzJ2mdTocvXMuDLjcUXktyrAKS6psbo46pgYmJBLiy+wszaZZh+YFd7N4B4dJuG0hnEf9j/eHc3nl7ZigsmDcFVs0Y5jjms2XMU72xpwxWzRqKmXAlvj8UEHl2mzOg+qakaPaEo5oxvtN2HHT2hKO59bav+/b0d7ZjVUo9Hl+7E0e4wygNefOTEZjyyZAd+8+pmAEBl0IdDnSH84qWN+MypozG7pd5y34u3tuHpFXvQXFOOcycOwQnDayzbHersw2NLd2LFrqOYOrwGx3rj+Vw/fWE9Th/XgMaqIN7cfAiXTG3CmMZKy/0s234Yi7e0weslfGzGCAyzyafcuL8D/1m7D0u2KrPJ21177f4t355+oreZ3nAUf1++C2dNGGx5Hj2hKB5ZslOXS36GFq4/gM+eNsZ2v48u3QkhgKtmj0SZz4vuUOKLlmGML836n+FoDM+sUpL6tx7qwk3/XAWvR9lxQA1uOdDR5yrdZcvBTl1RuVWWnX0RPLZ0p14RSbtGQZPF9/7OIwhFY/rzKpOLMT5WfEzJ8De10yICBg8KAgCmDq8xjHvsbu/Gt59YCQB4feNBtDRW4swJg233ed2Dy3Cwow9Bv0fv+FbuPoKbn1pjaLf9zktSlvflD/ZjnVR+7MfPfYCbLp6EW59eqy97fs0+PWEZACY3KflS/17Zin+vbLU97lX3LdY///yljbbtHl2yEz9/aaMuj8yf39yG+9/ahpbGSmw92IXNBzrxyyutZxj73pOrsE0tUXWkO4zvf3iSZbv/fe4DLFSTnAGgoTJg2U6r5rI/hfJbyXhx7T7c+vRaTB1eg39/9YyE9a+uP6B/ri0PGBTfoo0HE9prLFx/AD/8t1KXo6EqgFH1FQlt5k1vxlTp5SNdS3bx1jbD90eXxucPGDwoiMNdyovL5gOdmDDUuWjD+T9/Tf/s1vX63Kq9uOPZeMUY7Xem/dfQnoVxg6uwo63bsM7ctj9gxceUDNrYxlfOnYDJzdUYWV+OcYOr8MWH30UkJhCLJZZZSlajUKuoIm9n3ke6aPu5cuZIPL58F0KRGDr7lI7r+KGDsGF/h6Hjv+niibhuTguev+X5rBxflkHjhjNasOVgp66chLC+BmY6et1dH23dw9efgsGDghhpoSQAoLrMj4qAF74susW0aimr9xy1XK9d+y+fO05XvG/+z7k446cL4XWwnjqk8+3qi+rneN2cMfjodGUim+OGDkJ5wItfXDEN3/r7yrRdnZ3qdT5tbAPeUZXg3MlDcf0ZLThpdB22HuzC48t3pfyMupVHPteZo+v0CjvfvPA4nH38YMRiQFcogsqAD0TAiSNqcKgzhEg0ho6+CEKRGE4aVZeSbOnAio8pGbS36JpyP7wewsmjFTdgPCcsluBichv0Im+XrXEnbT9nHteIhRsOIByN6fIMrSnDhv0dBsU8fWQtAj4Ppo2owcrd1p13qpg7vMnN1Rg/pMpglWnWgNN5h6MxlPu96AlHHSMgQ1GBgM+DMyYkdw2PbqjE7sPdSdtlC+3aHz+sWl82oq4Cs1vqsXTbYVv3oXkWAu370OoyTBtZa2h7vFo6L93gFu1+XXJik674RjdU4JSxDQDi1lS6UZqptJszvlH/bZX5vTh9nPU9HVSWWIu1v+HgFqZkCJsG3DWCUk5YQgUU1+kEUhCKxfhMOtX2DRU31HFITZ6qoFLGq0saK9LOK5tjJOYOzxwYBMST2pMpvko98MahnTrJqxsCWS48kKzCm900SfHnx/oeG8vJxfTnw266Ja1dOljVuvSbxrPT2b/bMUf5XHMxVpcuhSsZw2SZkBSKLhOPEBSJs0Sn8aZr1alo9RJTQc5l0ypfaPJUqrNJyPKaQ8ftSCUNwXw9AqZSXUA8KdnJSglFYrqydrI2tEle3RDwEkLRWNbSKpLVo5TTAmRkj4EV8vnK99AqRzHTxHyrNBArJZhyCTKX8sjtcpGWkC6FKxnDZJmwTYcjh8anW/orWWJuOq4r2eLTAnC0fVdahIHroeNShxO1sDRTcXOZ25qT/w3y2uw3FhOIxIQ+9ZNTHlkoGnNdssrv9UAI63NMh6RlxEy5aXE5yLDejOHZiAjJcrSaZzCzPEyrNBDj5/TyH9OpGVvIs4Sw4mNKBq1jNr+JagojFElUfK5/8BFnxZdOzps5/yoUienHqQwmzlignZcnSemrVKwJcwfp9xpLdRn2a3OtNGu3Urf4nF2dbvO4AklcjKmSvH6mtasz4HM+L2MuZtTW5Q4YvQ/poL8s2dyjYLquTreeD8klGrCYVaNQYMXHlAz2b+xScEuCxedybMMwxpeZsjFvo7gXSbX4FHmsLD4r11mm1qdZmVm5Op2OpSw3yux0LbTgFjckczGmSrJAppDkejbK4WxFGV+KhGNyvv4SluY56W5UydqSxy7TTfxPJ7jFV8AWH0d1MiVDKBqD10N6Qq+G5nIKR2MJbjgnJSGPLRnHcRI70LRcnVInprg6BfrUCjOVFjPG+9Tzko+eqRJOtPgIQlh3aMk6/nK/Fx6Hedg02VwHt2R5aqJk+5Fdz6nIYX4p6rN5AZP3lfYYn0XgjDwEmvYYXxqej0KGLT6mYNhzpAc/fnYddvVDiPr2Q114d0e7rhxk5M7gdwu3GNY5dUDyFD3v7mjHPQs3QwiBf6gT3Jr38/SKPXh1/f6EdXaYozoB4FcvbwIAVATcuZHMVTGO9YbxjcdXJLR7e/Mhw/doTOCmf67Gog3GxGy/qVSXzI62bstAE3Nh52RBMG5dnZpVc8/Cza7aO7H9UBf+8Fr83r+wRql+sutwN3787Dq8vvEg7n9rm3pcuzFia4vx4cU79c9vbj6kJ3hbKXjdekxTgWgJ7Hb3SFv+P/9YjavvW4yfPPcBOnqNs6uvaz2Gm59abVjWFYrisaU78ZPnPsChzj7b4xfqLCdmWPExBcOvX96IP76xDXe9uCHr+358uVLBwsm9tOtwDzbs7zCsS2adaOw83I27X9yA3e09WL/vWELbUDSGrz+2Ap97cLlrmWWFYa720VRTntBeK5k2XcoNW7n7iKHNog0H8fYWY3UPAPj1K5sM319Ysw+PLt2Z0G5YTRmG1yYeW2OnxUuLXLA4kKQgeDiF4JZRDUpZsQff3q4n0afLX97ZYfj+xYffAwD84iXlmfzs/Uv1dUNMlUWcIjGPdofRE46nnGw+0Kl/bqpNLNvm9RAoiVXsRIeaZN9cW44bzmgBAEwbEX8e5Ofona1tuO/1rXjL9NJz/1vb9NJsMjf+czXue30rHl68I2Gdhmz1TxhSldY55AJWfEzBsPeoUoVkd3v2Lb5etfN56HOzE9ZpylCrQ3nu8YPx+BdOBZCkMLTqVjptbAMundasHycmgLGDK7H8lgtwwaQhhrapIOdk3T7vBPz2UzP0dXWVftx26WT9+5Lvn48yv2IFfv38Cfj2hccBSMwf7A1bF+M2n2d7d9xSfODaWXjnpvPw/g8uRGNVEEOry/D+Dy7ENy6YkLCf3rC9azXgI2U+OJtrqkV/uh3j++YFEzBrTJ3jebmlz6ZI+Z4jPYbv//jv01FnKqPmNNbYq+73rOMGY+Wtc/G18+PX7HiLkmFElPTlwAnN4B5eW46bL5mEZTdfYCgGMH1kLZ7/+plGGU33TLuWT37xNLxz03nYeMfFhvVahRsrtHv78rfOxowcVGBJF1Z8TMHRH7PdaFZHTbl9GoBWBWVEXQUaqgKG7Sz3qf7I66sCek1J7e2+qaYMjVVBvR5iOsEKIcni83jI8LYe8HrQWBW3PGor4tUviAgtgystj2undJzOs6bCj6aackOHX1cZMLhbtc9W+5HPw2nSXy36062rk4gwXrUqMg1wsc2IMC03W3tAfIzY8tzVZYOrgup1jFt5dkWiA15P2vMMhiIx1KnPAhFZ1r1sNlnsCbmr6vfxQ6rQVFOOgM+TMC7udHwAhvMsRFjxMSWBUySd5lrr6ovqbdxEv+kJzdIYnFYD0ZxMbmdROGEOpjDnZhm+e2wiVV0G6zidp+2ceNJyLWLTSgHJARd+H9nKYBc84kS2pieKuNze+vmxl8Gc9O5GqfszqEijuIqdj2G+vnYvR1b5f8mPH5/6q5ApbOmYkiRZ6ah0cPpBah1BV0hRWnLIvmOVEUOCuSJ0t6o8tX0GTMtTk1ntgCw6zYDPmE/nMUeq2uS4JUs50JDvge3UQNLyKodUBblSidMM4FbltpIRn4k9MzeBreI0PYtWCsBpjM+c++dGgWipK+kQdpEOYl5vl8JjeN5cKjKrdIpChBUfUxI4Jfb6Ta7OgJdsLSYZWTH5TcpT26d5eSqYO01Z0ZktPjN2IfZ2itzczioE3oy8XHN1OuUN6sEtSazOVKyF+ASnmY3xWVlYQogEV6dT0rmTq9OvvwglP7eAL31XZ5+LWqdmt6VdmT5Zebl9GQmpx08211++YcXHlAQhR1enskybHFTroOXtku0zPk5otPh0xZeOxRcRIIp3VOY3cKe3ajv3m62b0eE87Y4jd7BWtUM14gqcEib9tZItJcWnK53MLD6r7a3ktHx+dOXrkMqRQgFxfwbBLW5cnVbbmL+blZfbfaYSlZtPWPExJYFduSkgbklp43MBn8fRfaUhW5EBXXlqVqOxo+uWLD63MzX0qZ2Y1gHJHYrfR46dqJ38bspqAam7Op3KkRkr0NiPX9kVgXYi3RJcdjImLEtwdVo8P7q71criMyaUu1EgilWcngWbSpFvDbOSVXIprV3ngHNt1HSOnw8KX0KGyQJhC/eNRtwqiwem+KVqLvb7lII2TJadNi4XsLD43M7UYJ6iR+5QnIpFK+dgXUbLVvGZOm25b0stuCVZAjshGhM2xbPTCW5Jfp/c4KamqVXVHyB+r5MpfUVedxZfurU607G4rF6O7Aq5W7U3bisKPrAF4JJlTD/wwpp9eGHNXvz48qmWNSVT4a3Nh/DyB/tx60cm65bPoc4+/PT59fja+RNsZ+hO3I+StG3Zcak/VG3mbb8Uvv3W5jac+pNXcNwwY87VZdOa9ZBtebztL+9sN+xT6xR/+fJGfdtwVMDqsmw+0ImfvrAeoUgMnzplVMLbs9yheD1JLD617QNvbcdXzh2Phqog2rtC+OMb2yzbd4WiuGfhZnz53PEAjFGO9mN88WupuTrDkRh+v2gLjvaE4fMQWo/26OvM0bJ3vbgRteUB/Pc549RtU48IdFuC69cvb8L4IVW45MQmy/VW2z+yZCeWbjusf7eq+iPL8ODb23HV7FFYuP4AHlmyE9+4YAL+9/n1ahv1RciFNRsw5TpGojHc8q81OGl0Ha6YOdJymy0HO3Hr02vSUjxW48B21WkA4LFlu/Cxk0Zg2fbDWLCiFQ1VAVw3pwUHO/qw7VCXY4GDQoEVH5N1vvjwuwCAj588AmdOGOx6Oysr4L/+tAQAcMXMkZjUpMx8fdcL6/HEu7ux83A3Hv9/pyXdr5xKYDXoPm5wJSoDXnT0RhDweTBx2CAQEaqCPnT2RbDvWC/2Hes1bLP/aC9u/PBEAEqnoM2c3dalJH5rimfooMR8plAkBiSmV+HFtfvw0jqlpJk2BZGsXII+D44fOghBv+L+bKopQ0NlAKeOa0jYV5PU+by9pQ2XTmvGkm3Gii0/uXwqvv/UaswaU4dl29vx61c26YpP7njL/dbl0cYNrkJFwAuvx5hP99MX1pvaKTmFdRUB3TV5pDuMe1/bCgC64nOap84ON4WqhRD6i8clJ15i2cbKirnbVEFoumm2dI0JQ5R7rxVguP2Zddh6qAu1FX58sFep4qM9u6PqKzGozIeP2Chg5ZwIkZhALCbg8RC2HurCY8t24bFlu2wV3wtr9ukvd0d77BPMNS6cPNTwrMmELAJkpo+s1c8FAP6+fBee1Erz7VdKvvnUbaaNrEl6/HyTFcVHRBcB+DUAL4A/CSHuNK2/FsDdAPaoi34rhPiTuu4aALeoy+8QQjyUDZmY/JNsYk+79lZjYPK+jvUoLkmnChIymtvo3OOtlfDYwVVYedtcRGICXk88QvOhz83Gx3//NgBg/e0X6e3PvGuhopikMb6TR9ehuaYMrWrnp1ljU5oTOwE7V5FcIFuZqdsYqEBEhqobg8r8WHrzBbAyRKqCPl2xacfT9n/7vCn49KmjQUS4evZIEBHm/fZNrNx9FEIIEJF+zR68blZCqoTGyPoKrLptriKXWtvS6tx61KChGaNq8cyqVgDx8VSr65KKq9PN1ERu3IahqMCgoA+r5s/FjrZunPOzRfq6R244BSePrrOdjun4YYMwcdggvcasVgFIK2ZwwaQh+gvg4EFBrLh1rmNCuKzMyzxeV3U75TaXzxietP19nzkZWw524oJfvJ6o+KIxPT1F438/NhW3XToZbV0hzLnz1QSZjvSEUV3mx7DqMtzzqZOSHj/fZKz4iMgL4B4AFwLYDWAZES0QQqwzNX1cCPEV07b1AG4DMBNK4PC76rbtmcrF5J+UJ7tU22drfjUNTUEFHeYH83k9MK+WO7oyyeoxz4au5eqVS5VMzK5OGVdV/KMCoahAtcnaMishpw60zG8M/NCua9Dv1S1f7X9VmdIVRGMCPi/pStLpmgHQ3/I1ZWVVOqzTMHZqHE8FoCvbZHPJWeFmNgM343+aW5mIEgqAl/k9hvtvRZnfm/Dc9qmlwMpNM2kkq4Iin1OZ3+s4ca8sv4YbVycR6ffWHIkGIHAAACAASURBVNFqN05Y5veiTL03Pab7LISyXV1loOBTGYDsBLfMBrBZCLFVCBEC8BiAeS63/RCAl4QQh1Vl9xKAi5JswxQJ6U59kqyjEmpylTAnWdntNw0XGuAczShPWmuu0mK3TMNN2TBt/5lEyJnHv5wqo5gT9uX6mq6OZUrnkNHSRAI2ii/+wpN68rObMT63FpPdPXMdiRmNGWan0CJ5Uw02MVuxbhS3fI5un3O7uf+cJgM257waZEgjlSJfZEPK4QB2Sd93q8vMfJyIVhHRk0SkOardbssUIelOdmllKcYyKOAZzw9LrQNyCupQXJ3GYAxjIIpyLKuACDtLWL5e8TG+LCg+UwfqlMuov3ykmFMXsFBoGtr0TXIKRldIinJ1IZ8d7qJv3VlMeoUch4hGWznUbeWpquIFEVK7hwn3wqX8GgGXz7n+smMR3GL70udwn5WxwcK39oDcpTP8G8AYIcSJUKy6jMbxiGg+EQkiEq2trVkRkMk+6So+q3woeV+kJleROckqyX7txmjssPvxa+Hm8bwzo7tPPpaV28euvJZZ8VnlU6WCOcfNHFovY377N59bMuIVauzzzwJeT7yEm5TXqN1vTUGn5Op0MWO5G5e7PAGuWVG5tfgAo+XVJRVESAVz8QFXFqvkZnV7/axyPYUQjpab030ulhw+IDuKbw8AOdRoBOJBLAAAIUSbEEKbMOtPAE52u60VQoj5QggSQlBzc3PagjP9S7quTssZzOUfZ7quzpQ7IJuKJaqr01xpxMrV6SSPGW0sh0gZH4rEMsuJMlsOfQ6Wr3msLFXrK17o27o0m1aBRttfp4WrM53KLW5mYXfzHPYZXJ2m5G2XuXeAUYl0mQqWu0VzL5tdwE4YXJ0p3rOQ4YUrcQZ3GS2X0Xyfo0IgJlI/13yRDSmXAZhARC1EFABwFYAFcgMikmN3LwPwgfr5RQBziaiOiOoAzFWXMQOAVMsumV1yMpkEvKSTHwY4J24bglssXGROx0pWpLkq4DMUzE4XsxvQKWrS3NZccSQZVhVqjPtXAkesxojMLr2UEthdVG5x8+zI1goRIdU6lX4Ly1OuBJQK5pcQ+XdkVzUl1eAWAPqMHrLSdDPO6vdSQlRuOm7qfJJxVKcQIkJEX4GisLwA7hdCrCWiHwFYLoRYAOBrRHQZgAiAwwCuVbc9TES3Q1GeAPAjIcThhIMwRUnark4rxSf9OFN1dabqttNINs7RowdteA3Lkx0rWfWUiqAXhzqN+YDpYHYDunF1unGLWh7LokKNjFbWy7KSTZrHlPfnnM7gxtUpjPfP60E4qrkqkz9nQQs54kE96Y0tm8dblf3H4PUkRpgaxvhcXj+Ph/TxavN+khVHMLs6tSH4YnF1ZiWPTwjxHIDnTMtulT7fBOAmm23vB3B/NuRgcosQAl9/bAXGNFbi9Y0HsWFfh+HBv/P59bjz+fX4wlljQQRsO9iFP3z6ZD0kf9fhbtz4z1X44WVT0FRTrifedoeiGHPjs/jCWWP1fb29pQ0nja7DVx99T0/UVdpGcMNDy3HexCG44cx4e5l030Z9Huv2Wkf461c2Gb7LEZCOrk4b19sLa/cBUMp/7T/Wl5bMRjmVbe99bSuun9OC51bvs92ntuxzDy5HXWUAWw50pnR8rd2KXUes15uKNGvXDgA+8+elWPidc/DyBwdSOqbc9r7Xt+L9ne249zMzUS9NmHvHM+vwpzfj1Wre39muzwyuPTvTRtYianIrK2kaWvk5967OT/1xMQ6rRQw06yxlF7t6vN5wDJ//y3KslK7p6j1H8dPn12P5jnZcN2cMbrt0CgCTxWfz3Frh83jw3s4jEEJACGD6j15KKnPA58GxXmvLPpMXtVxSHFIyBcnBjj4sWNmK/3tlE1bsOoKecNSyasR9r2/Fva9txX/W7cfBzj59+R3PrsNbm9vwnSdWYf2+DsvtNHojUfzxja0GpddUW4aN+zvx9pY23PHsBwnba8TD+FN98yacfdxgfGfucYbl5rdabZZ1w+wJkhK8bs4YQ3s7C0Tbb215fDb1VFMwZFoaK/XPK3Yd0XPRtNnZZU4f1wgA2Hm4W+9oTx1bjxpJFic0Oc35XQDQUBnAhZOGAgCqLfa350gPdrR16Yn4stzJGN0QL1m3bHs73tthTAGWlR4AfFqtBAQAm9Rn5/eLtgAAuiXZ504eivrKAOaMb0BVILl9cMaERgwZFERnbwR1FQHDulkt9a7PB4g/R3uOdOOldftxRPpN/fO93ViunuMDb23Xl2vjt6PqK/QqQm7Q7texngi2t3Xpy8c63AMrpdhQGUBjVRBnTmh0fex8wiXLmLSxcy49cO0snHXcYIz7/nMJ64zjCfExvWTuqHAkljBLts9jP6mpYds0LT4iwkOfm52w3LwfzcKwC2657dIpuO3SKXh48Q7c8q81tmOfkWgMs8bUGSzNTN6gK4M+/PCyKbhtwVqEowLhaAz1lQFUlyUqnwsnD8WJI2qwardSr/R/PzYVl05zHzhmlvPB62bhnOOHJLQ7aVS87Nd5E4dgdEMFHnhrO0LSMyArs2RUBpXyX8+sUirHJEt7MaZRGJ+dS6YO0z/f/clprmUAgEunNRuu15gbn9U/nzo2saScE0F9HFSRdd60ZkwbWYtb/rXG1pWsncui75xjW2nHikumNuHZ1XsRisYMhck/aVMaDTA+2x+bMRy/uHK66+MVCqz4mLSx62T8Xo9tdQpjWoL6n5KPw4SjsYTUALlsWLJtgeyNP9gpI/PsCXbbWckcjSlRcQGfB16PteWYlqzS2J0csm/Z1mVUarJtrb5rBE3XSE7W1uYftCsGbXtsaZ8Rl1M+AYnRtYXipjNHyPqlabLMwUNyiTmfh1JSeoDx+ZB/Xk6pP3aF04uJ4pSaKQjs8tGcggHsghCSKT6rFICwFFnpvG16Yy122O3HoDgsZ3q3nioIMFqlsks2U5nlQIlwJGZZQs3qWKnmPJr365QDGd/GY4hg1PLHUi15JV93+TlymjdOaWtcn4lbOZuYq9so+Y/WwUOaok+32IEczSvXyHWe8iq+rgiqk1lSGHeaKUpCUWu3i5NlZRXYIUTyGbRDEWEoB6XtK5UCvtl6o7fv1KUQeIcAEisl3yflsLnNB3SDnKsVSjJljdt0DOvjeBy/Wy33e8mgmK1mBUj12OZC306Yn51CsV7MSeLaBL7KMus0gnSLHRiuvxwg4xjVGT9OBgWV8kph3GmmKLFTVukkbyez3Owm+XSbowUkWiXp4q5Tt3d1WilrOYfN4DLN0AoxWFSRaBJXp2xppnatfB4yvP27uUZy7U5tjC+TzhswF/pO7ZkqFFen2a2pvCAo1yUhf079DYbSrJpivP6yxefOM1CsFP8ZMHkjWXSi222I4uNedkMUVmN8ITVgw62cWr5dptgp0GQKyy+NZ5mRxyGzafHpuXyRmGMNRrPMqXaiStJ38rFJ8zF0V1sGRbnlbQw5b0m8AeZnp1A6dE0Bd/bFS57pytDk6uxTvS7Jxm/tCErPpLHep7vnhF2dTMlh90adag6bNqUJANsZ2622C0WiKbk6M6l7KeNU0SX+2b4smPW5xGU0WkWZySy7V5ONA2WqcN0Ex5jbaB1vKJo4/6D748avUSgFi8883U+hJF+bx/j8hjE+s6tTtfgisbTGKOXnQ34uncZZ5XvErk6m5LB7o3YObrEr16UqPpucKbsyZnLnZjWBLZD+tER22EZ1mtx4ZpzG+Po7uKUvotX+dNehpec2Sy63uY2h4zVVT3F/XNnVGX8G7IKv4m3NFl9hmC/mqM6A16M/u53mMT693Ft6dV3txvjcbFPMFP8ZMHnD7ofi1Hld/9By3LNwM8bc+CxeWa9U6ljbegw/eHotAKAyaO2OtMp923m4G7f8a43huxV3vbAhqVypICtQedLSZGHe2vqn3k+swx6S6olmM1xc25eWpJ2sIkcmx3VjMfoMLwdx6/abj6/Eoc6+9Cw+Se67X1Tu9drWozjr7oWW7T/0y9dxwm0v4uan1hiWZ3OMb5CN58IN2vks3a5Ub/R7SZfNbGF96+8rACgKMK2XBinS+I2Nh1xtI0f8Bv3FqUKKU2qmIJDfrs+fOATnTxyCj05vxuBBQQDADy+bon+W0TonKy45sRmj6o0JzDXlfoQiUXhU94vXQzjFohrGtkNdCcvkSNCpI2qSnJE75M55vloyyry8tiIxSXyiWlHDap08ddLcKcMwY1QtZo+px1nHZVYJY1LTIJx7/GBMaqrGiSNq8PGTRti2/fAJTZg+shYXnzAMI+vKUz6WXCqssSpg2+66OWNw8ug6nD9paIIVftEJw2y2suds9fw0ojGBNXuO2rbfsL8jIUgEyG46w98+fyoA4JZLJqW87fSRtThzQiNOaK7B7JZ6nDFhsG01lu1tysteKJqeq1POLdXycsdaVPaRuWx6M8pUhffV8yakfMxCgBPYmbTRxgR+eNkUXHP6mIT115w+BpdNa8aM219yvc9TWurxrQuPw7zfvomVu4+ipbESPaEowlGhV27515fmYGhNELN//IphW/OYDRBXzmdOaLSsWJIOcoL1FbPiFS5kV5nVGEmZ34u6Cr/1GJ/k6pw1ph5PfWlOVmStCPjwwHWJ1WesOGNCI87IoOTUF84ai2/9fSUA5zGi26SXhb1HexP2kSoTh1Xj+a+fic/evxSvbzyYMF4FAJ88eQSeeHe3436yafFNHVGD7Xdekta2tRUB/PX6UxKW11cGcLgrhKDPgw13XIyLf/0Gdh/u1ufQS2c8WC4goP1+/vjZmY7bfGjKMKy//eKUj1VIsOJj0sZNKbBU30LNc9v5vYSIjwxlzfw+suykko2dZQu7Pt3V9DVe6zJrqc54Xoik56Y0XsxMzj9gyllMlUK/9noxdFXOgJcQisYQjSkFpjNxT8u/r0JJ6+hPBv4ZMv2Gm+l+Uv0RxWc6ULbzejzqFDExg1VkdUxnxdf/gQseF7HdAZ/HMp0hJCn1YiXTHLx096GhWy+RxNqvbsLuMy0P199o5yf/l3NZM8rjk65Zob8AZIOBf4ZMv+F20spUMFt82ue+SMwwOarVj9NK8cWVc3Zy+IDMQrgDXo9NybLUJn4tTDJXfKmWK7Pal1L303iN3dyzQu/wE70hHsQE0BuOGpants+4lZztmraFzMA/Q6bfcFMKLOW6i764ixNQglPib7bxN1KrQsbJ8uP6Gzf60O/1OMpZ3J1Oft2L6YTm95cs/YH2O9O8Apq8nVLaQ6rEE9hjOf2t5JvCvtNMQdMfrhHtxysrTGVcTBjeSIkSx/msxnX6w5LKpFqF30c5G4ssBlIthu2EXH4rHcVX6C8dVhYfEK/fmYmrOSyNi5bCMzjwz5DpNzIZW7BDm84oPmWRouCiMaFPmmkeB4zLk1+F4qbb0cYrzYQGhOLL3NWZCbL1Yk5ed/OyUujXXh/bU+U0z9uX8RjfAAiwcgtHdTJp09ePPxSt2xJC6JGhx9SZqLXj+UxvuFYuxJ5Q+uMftrLZePTcujrDUYFQJIaeUBTkAarL/NI4TTG7mdJxdWbvfOVCzu3dIcM6N2N8hT6+qp2f9txr34+o55pR5ZaoQHt3CB6C7VyaAwlWfEzaxF2P6f9Qaiv8ONId1r9rP+oyvxKMMqjMp7/ZvrfzCID4j9VcsPeJd3fha+fHE2r3HOnBvHveAgBks0+rUqtymN10Ff7kATTa+R13y/OW67Pp+ss15Tbl5pwIurhmbtEsnk/+4Z2EdVZFA8yYX6QKDe25J9WyDqoBW9c/tBxAegpLu2aPL9uJ/cf6Cl75ZwtWfEzahPXgFufO665PnIjFW9rw6dNG42O/e9uw7nefOgnb27rxm1c34cLJQ9FUo1QM+fyZLRBC4LOnjUE4GoOXCDEhMH1Urf4D19yDHgJiAmisMlaJeXPTQf1zc23qlUjsuHRaM1bvOYorpeR1ADh34hB8bk4LLp8x3Hbb2WMa8NbmNgBKRZqjPWHD+mJ2M505vhHXn9GCy6Y1u96muaYMN5zRgj+9uQ2/uGJaRse/dFoz7lm4Rf8+qakaw6qDqAz68JXzJuCkUXX4w+tbMWRQEG9tPoTuUBRXzx6FCyYNwYb9HQV/7c0BXVfNHonHl+/Svw+tLkt5n5qi23+sDwDw4ampV84pRljxMWnjdp67K2aOxBUzFSXh9xLCUYHzJg7B/dfOAgCcPh741CmjDNucOKIWv/3USfr3OeMTK4pMbqrGur3HcN7EoXhnyyHHKYomSyWtMiXg82D+ZVMSlns9hFsvney47fmThuCXL28EAJwwvFpXghqF3vk64fEQfvAR5/M3Q0S45SOTcUuK21kxcZjxHt/3mZMxUip/d/HUJlw8tcly2/MnDc34+P2NOUJ6xqg6XD17JB5dqii/qcNTL8ln/u1+4axx6QtYRBTvr4zJO+kEZGhjLdkY2xHSJ7/POk1AI5t1GDNBDkCwmomimBVfoVHoUZqpIiwGKjOdSsq8TaEn8WeLgfVkMDlFTihPlWx38FrQiB2FMnYhn7c8s4PGQOus80kpvES4mfzXCfPvohSuGcCKj8mAdCo9aN6abHTwUklopSKKg8VXKApFliPo8yaM2xSKgh4IFHeEbCKaq1P2eGY6lZT5d1Eov5P+pjTOkukXMsmRy3YHr1V3saNQ3mQNE7H6KEGuYq7VWWgUyj3PNrLHM9uuzoF6zcyUxlky/UI6JY7iY3yZP3qGMT61Ur0dhfL2HzB1VGa5SqXjyQUDzXq2GuOTpyJKL4G9NJ+/0jhLpl/IpNpIv4zxObk6C+QHbRyT8SQUzy4UOQcCnhJIxDY8T9lwdZbI85eVsySii4hoAxFtJqIbLdZ/i4jWEdEqInqFiEZL66JEtEL9W5ANeZjcsOuwMvtzKj8WbXwiGy49wxifz4OukDGhXZ7ktFDGLmQ5Al5PwuShhSInU3j0xxif38NjfGlBRF4A9wC4GMBkAFcTkTkp530AM4UQJwJ4EsBd0roeIcR09e+yTOVhcsOm/R3Y3qYovlTerEfWKXlVwSxMEzR2cKWyz/p4cvqWg50AgLbOPvzq5U368mwcLxvIwSxlfi/KTJGdZQUiZ7GSSQHxQme4WoRhTEOlvqxcqnxT5k+9O/d4SK8WFPB6SqJcGZCdBPbZADYLIbYCABE9BmAegHVaAyHEQqn9YgCfzsJxmTzSqlpTqSaG//yKaXhh7T5cZap6kg63zzsBU5pr8JnTRuObj68AAOw90otxg6twoKNPbzf/0skot0gdyAdEhF9eOQ3r93XgEyePwJTmanz5kfd0a7XGRWktxp5HP38qfvPqJvzXKaOTNy4yvnreeNRV+HHl7Hixh4tOGIZd7d1oqilHbUUgrf3e9YkT8f7OI5g+sjZbohY82VB8wwHskr7vBnCKQ/vrAciFCsuIaDmACIA7hRD/yoJMTD+jjac5leeyYsaoOswYVZcVGeoqA/jvc5RKEyePrsNL6/brkZ3a/8+f2YJr57Rk5XjZ4vIZI/TPQ6vLcMflJ+Cbj6/Mo0QDh1PHNuDUsQ35FqNfGFJdhm/NPd6wrLYigO9+aGJG+503fTjmTU/td1zs5LRkGRF9GsBMAGdLi0cLIfYQ0VgArxLRaiHEFus96PuZD+A2AGhqsi5BxPQvbmZfzyXyXGxAcc1vVwwyMsxAIhu/uD0AZL/VCHWZASK6AMDNAC4TQuh+KCHEHvX/VgCLAMxIdkAhxHwhBAkhqLnZfUFcJnvoEZ0FMhiuBYloKRb9OWVStvEO5IEphilAstErLAMwgYhaiCgA4CoAhuhMIpoB4F4oSu+AtLyOiILq50YAcyCNDTKFS6jAFIs8k7TyP/uT5DIMMzDI2NUphIgQ0VcAvAjAC+B+IcRaIvoRgOVCiAUA7gZQBeAJNSR3pxrBOQnAvUQUg6KE7xRCsOIrAjTFUijzx2kKTld8+pRJhSEfwzCFQ1bG+IQQzwF4zrTsVunzBTbbvQ1gajZkYHJLoY2hyTNJA4U3BskwTOFQGL0WU3QUqqtTk6vQxiAZhikceCJaJmV2He7GB3uPASgci0qbkuWFNXsxqr4Cz6/eB6BwFDPDMIUDKz4mZc68K16PYFBZYTxCg8qUxO9l29uxbPtyfXmhKGYnhtaUAQCa1f8Mw/Qv/DrMZMSMkdlJRs+Uk0bV4faPnpCwfHZL4ScznzSqDn/49En415fn5FsUhikJWPExKSFPjRL0eQqmAr7XQ/iUVMpJoypYGBZpMi46oQlDqtniY5hcwIqPSYlILK74Ci1HzqrAbqGkWzAMUzhwr8CkREia864YcuQ4uIVhGDPcKzApEZZmOS8GpVIq06wwDOOewu+5mIIiJCu+LEwmyzAMk2tY8TEpUWyuToZhGDPcczEpodXoBIrD1ckwDGOmOGK9mbzTF4nimZV7se1QV75FYRiGyQhWfIwrXvngAL79hHGW8MaqYJ6ksaehMoC2rhAAoLEqkGdpGIYpRFjxMa441hMGAFxz2mhMaqqGz+vBWRMa8yxVIs987Qy8sGYf6isDOGlUYVSVYRimsGDFx7hCS2OY1VKPj5xYuLPeN9WU47o5LfkWg2GYAoajExhXaPPccUALwzDFDvdijCtCPKM5wzADBO7FGFcU2ozrDMMw6cK9GOOKuOLjai0MwxQ3rPgYV2ilygptRgaGYZhU4V6McUU4wsEtDMMMDLgXYyxZ23oUv3hpI/Yd7QUA/OO93QDY4mMYpvjhPD7Gku88sQof7D2GcDSGL5w5FkfVBPaGSq6GwjBMccOv74wluw53AwB6w1F0h6MAgNkt9WgowDJlDMMwqcCKj7FEi92MxQTCag5fS0Nl/gRiGIbJEqz4GEdC0Zge0ckTzzIMMxBgxcc4EooIvWoLR3QyDDMQ4J6McSQcjenJ6xzRyTDMQCArPRkRXUREG4hoMxHdaLE+SESPq+uXENEYad1N6vINRPShbMjDZA9F8Sk5fFynk2GYgUDGPRkReQHcA+BiAJMBXE1Ek03NrgfQLoQYD+CXAH6qbjsZwFUApgC4CMDv1P0xBUIoEmNXJ8MwA4ps9GSzAWwWQmwVQoQAPAZgnqnNPAAPqZ+fBHA+EZG6/DEhRJ8QYhuAzer+mAIhJLk6WfExDDMQyEYC+3AAu6TvuwGcYtdGCBEhoqMAGtTli03bDs+CTIxLFm04gC0Hu3D9GcbJW4X6/41Nh/DGpkMAeIyPYZiBQVH2ZEQ0n4gEEYnW1tZ8i1PUXPvAMtz+zDrEYsKwPBKLJbQN8MwMDMMMALKh+PYAGCl9H6Eus2xDRD4ANQDaXG6bgBBivhCChBDU3NycgeiMhparp6EFtMiwq5NhmIFANnqyZQAmEFELEQWgBKssMLVZAOAa9fMnALwqhBDq8qvUqM8WABMALM2CTEyKhCXFF40JRGOs+BiGGZhkPManjtl9BcCLALwA7hdCrCWiHwFYLoRYAODPAP5KRJsBHIaiHKG2+zuAdQAiAL4shIhmKhOTOrKFF44mujkBwOthVyfDMMVPVmZnEEI8B+A507Jbpc+9AD5ps+2PAfw4G3Iw6aOlLACJbk8NgUQrkGEYpthg3xUDwGjlhSPWio9hGGYgwIqPAWC08qwCWwCAwK5OhmGKH1Z8DACjxRdii49hmAEMKz4GAI/xMQxTOmQluIUpfh54azueen8PZo6uw/Id7fkWh2EYpt9gi6+Ekau1PPW+UjdAVnqnjW0wtD/7uCG5EYxhGKYfYYuvhAlblCU7dWw9Fm89DAC4/9pZKA/wZBkMwwws2OIrYayiNysC8XchP9fmZBhmAMKKr4Sxit4s88cfCR+XKGMYZgDCPVsJY1WajOtxMgwz0OFeroSxsvhY8TEMM9DhXq6EYYuPYZhShHu5EsYqUZ0nYGAYZqDDiq+ECUe4EgvDMKUH5/GVID2hKK65fylmjK5NWOdjk49hmAEOK74S5NX1B7B0+2Es3X44Yd2Vs0YBAGoqArkWi2EYJiew4mN0hlYHMbm5Gj+cd0K+RWEYhuk3eIyPYRiGKSlY8ZUgPL0QwzClDCs+hmEYpqRgxVeCEIyRmxzIyTBMKcGKrwQxuzq1ai2CPaAMw5QArPhKkGjMqOECXKaMYZgSgnu8EsRcnDrgUx4DYpcnwzAlACu+ImPT/g5M++F/8LtFm5O2vWfhZoy58Vnc9vQaAMBN/1yFMTc+i+8+ucrQrqbcDwCoLvNnX2CGYZgCgxVfkbFsezuO9oRx1wsbkrZdrlZmeX3TIQDA0m2JlVoA4L7PzsRl05rx+0+fnD1BGYZhChSu3DKACUeF43eNcYMr8X9Xz8iFSAzDMHmHLb4iI5VxOG3aIaGGa1rNv6fskwf3GIYpHVjxFRnmiEwnNEWnWXpWM64zDMOUGhkpPiKqJ6KXiGiT+r/Oos10InqHiNYS0SoiulJa9yARbSOiFerf9EzkKQVSUV6a4guZ/jMMw5QymVp8NwJ4RQgxAcAr6ncz3QA+K4SYAuAiAL8iInkiuO8KIaarfysylGfAY+eutGwbMbo4U9mWYRhmoJKp4psH4CH180MAPmpuIITYKITYpH5uBXAAwOAMj1uypKK8NAsvHDG6PBmGYUqZTBXfUCHEXvXzPgBDnRoT0WwAAQBbpMU/Vl2gvySioJuDEtF8IhJEJFpbW9MSvFgJpaC8QpLCi8ZESuODDMMwA5Wkio+IXiaiNRZ/8+R2QgkdtO1ZiagJwF8BXCeE0MyWmwBMBDALQD2A/3EjtBBivhCChBDU3NzsZpMBg90YXywm0NEbNiw7pn4PRWP6dj6uSM0wTImTNI9PCHGB3Toi2k9ETUKIvapiO2DTrhrAswBuFkIslvatWYt9RPQAgO+kJH0JIrs627tCqKsMAAC+8fgKLFjZije+dy5G1lfgcFcIHb0Rve39b20DAFSV+XCkO64ggz4O7GUYprTINIF9AYBrAXSFigAAGQVJREFUANyp/n/a3ICIAgCeAvAXIcSTpnWa0iQo44NrMpRnwCMrvv0dvbriW7BScfnuOdKDkfUV2Hu0x7DdjrYuAMC3LjwOh7tCmNJcg1fXH8AFk4bkSHKGYZjCIFPFdyeAvxPR9QB2ALgCAIhoJoAvCiFuUJedBaCBiK5Vt7tWjeB8hIgGAyAAKwB8MUN5Bjyy4rNye2rLzOu6+qIAgNPHNWD8kEEAgAsnOw7JMgzDDEgyUnxCiDYA51ssXw7gBvXzwwAettn+vEyOX4r0SQrNKsLTnLSu0dmnuD39PAURwzAlDveCRYas0EKRxFgiu5y97pCi+AI8pscwTInDvWCREZYsPqtKLH02rs5O1dXJFh/DMKUO94JFhqzswhZjfHpdThuLjxUfwzClDveCRYbswnQe41P+B1RF16WO8XH6AsMwpQ73gkVGKImr06z4KoJeAPGoTrb4GIYpdXgi2jyw/1gvDnb0YfyQKvSGo9h8oBNeD2Hq8Br4kiimZOkMx3qU5PRONXm9MqAkrPeEoyACvFy5hWGYEocVX46JxgRO+ckrAICWxkpUBX1YvecoAODbFx6Hr54/wXF7wxifOp63aX+HvuzZ1fvwlfMm4LnV+wAAFQGvvk5wqU6GYRh2deYa2UrbdqgL+4/16t/3d/RabWIgLKUwaNbf/mN9+rL6Sj8AoKZc+X/6uAZ9nbaMYRimlGHFl2PM43KhaEzPrQtb5OWZsQpuMSwzzcHXUBWf8EJWggzDMKUKK74cYx6XC0diqAoqHmc3c+3JilPL2ZOrufSZZluvDMa92RzYwjAMw4ov55iVWzgq9HG4PjeKz6JkWdgit09rJ4/xseJjGIZhxZdzzIovFI2hMqBafDZz7dltb6n4pGV+Lxny9gI+juhkGIZhxZdjrFIQKtVcOzeuTrlWp16lxcIKDEVj8Hs9BiuPLT6GYRhWfDnHKum8Uh/jSx7cEorGdCtOU3hWuX3hiIDf6zEUpQ6w4mMYhmHFl2uslFu5X7H4rJSijBACISkYJqRbd9KMDdF4VGfA5zEoOz+XK2MYhmHFl0uEEFi04UDCck1BLd12GLGYUTG2d4WwavcRAEBEXaeVIfvbkp042hNG65H4bOvhaAzhaAxbD3XBQ2BXJ8MwjAnuCXPIxv2d+NXLmxKWV5f7dett+Y52w7oLfvEaLvvtWzjQ0au7NOsr47l5Nz+1Gku2tenfw9EYfvvqZgBKYrvfGw9oqS7jQj0MwzDcE+aQw10hAIoCumr2KAwZFERV0IfzJw1FV18ET69oRXt3yLBNm7pNe1cYQa9i6Q0dFFd8L67dhxmj6gAA00fWYs2eo3hr8yF9vTzGd9XsUf1zYgzDMEUEK74colls/+/scfjyueMN62aOrsPTK1ptIzuJgL6oOsOCpMwIpIzneT0o83sQiQnIzlLZvVkV5NvNMAzDrs4coik12f2ooSkop5QGLTDGHJ2p5exp+4hI+whwQAvDMIwB7hVziJZqYBVkoi2zyvMDgEhU6AnusuIUEErqghTBKZcw44AWhmEYI+z7yiGhqIPi03LzbHL5lGhNReHJVpwQicnqPeGovp4tPoZhGCOs+HKI7qq0UEaatWZXtiwcjaEvoig+WXFGYkpuX0BKVtdmW1facpkyhmEYGTYHcojmxrSqoKLV0bRLYg9FYvr4n9UYX8AXt/i6+iLx/bKrk2EYxgD3ijkk7OTqTGLxhaIx3WI0b68Ft2jKU3Z18hgfwzCMEXZ15hDdYnNydaptesNRPPHubn39bQvWYvaYesvtQxFljM/akmTFxzAMI8OKL4eEnNIZTMEtr64/gB/8a42+fkdbN3a0dQMAGqoCOGF4NdbsOQYA6ApFDcEtZX4PesMxNFYF4PMox9L+MwzDlDqs+HKI4xifKZ2hozdsuY+/ff4UzBpTj8tnDMc5dy/CgY4+ZXufR1eejVVB7G7vwZNfPB1EhNe/ey6qy/lWMwzDABmO8RFRPRG9RESb1P91Nu2iRLRC/VsgLW8hoiVEtJmIHieiQCbyFDr6GJ+F+9GcwC6nNZT54+1PH9cIv9eDioAP5x4/RF8ekCy+7pAyxldd7gcAjGqoQG3FgL60DMMwrsl0AOhGAK8IISYAeEX9bkWPEGK6+neZtPynAH4phBgPoB3A9RnKU9DYVV4B4mNx+gzqUpCLXakxefxOnm29U43q5FQGhmGYRDJVfPMAPKR+fgjAR91uSEQE4DwAT6azfTHiXLnFmM4gly6rtFF85imH9H04HIdhGKbUybRnHCqE2Kt+3gdgqE27MiJaTkSLiUhTbg0AjgghtKSz3QCGuzkoEc0nIkFEorW1NW3hc01Ij+pMtMTMY3yy4tMmqjXjl/bjl/L4zPtkGIZh4iSNeCCilwEMs1h1s/xFCCGIyLreFjBaCLGHiMYCeJWIVgM4mrK08WPNBzAfAGbOnGl3zIIj7KJWpz7GF0leaFpWbEGvUfH5PAQPR3IyDMMkkFTxCSEusFtHRPuJqEkIsZeImgAkTi+u7GOP+n8rES0CMAPAPwDUEpFPtfpGANiTxjkUDY55fPoYn6LH5eAWO8stYHJ1Gsf82NpjGIaxItPecQGAa9TP1wB42tyAiOqIKKh+bgQwB8A6IYQAsBDAJ5y2H0g4Fql2cHX6bIJU5OhQv49MipCtPYZhGCsyTe66E8Dfieh6ADsAXAEARDQTwBeFEDcAmATgXiKKQVG0dwoh1qnb/w+Ax4joDgDvA/hzhvJkhWhM4J6Fm3Gwow+dfREMKvNh1+FuTG6uxnfmHg8lLifOq+v3Y9P+Tpw+rhFPvrsLHWpUZWXAh2E1ZfjSOeNARAhFrEuOKcuUfb6pzp4eNsypZzPGlxDcEv9utw3DMEypk5HiE0K0ATjfYvlyADeon98GMNVm+60AZmciQ3+wrvUYfvHSxoTlCzccxNzJwzBtZK1h+eceXA4AhmoqMhdOHorjhg7SlVnQwtUpK9NwNGYY45s7eShe33gQ5xw/2LDNyLpy/fOIugqDqzPAFh/DMIwlXM7DArnIs5lOaeYDM+1d1tVWetX9ORWpBoBZY+qwbHu7ovjUtk9+8TTMHFOPuVOGorbcmIQ+d8owvPG9cxGOxtDSWInXNh7U11klyTMMwzCs+CzRFFRlwIuukFEJCocY0q6QtVLUrLdQJAYPAV6baMs6tbpKOCL0IJeR9RUAgCGDyiy30dYDicEuDMMwTCLcO1qgKaoKm8RxO7psrEE5Kd1JIWlWWl80qqc+pJKL5ze4OvnWMgzDWMG9owUhyeIzQw5DZ+GotTkopyg4TRMU9MZTGpzqetphsPjY1ckwDGMJ944W6K5OC4vPydVpuz8pRcHJEpMno3WawijZ9gAHtzAMw9jBis8CJ8UXjhlnSI/FkmvCkFSNxdnVSfrx9XqbnhQsPrmEGbs6GYZhLOHe0YKwmm9n5eqUZ00A4krNcX/yGJ9FnU4NPYk9GlPHA1MrO2bM4+NbyzAMYwX3jhaEHCw+s6ILu1B8IZeuTk1ZhSIxhKMiZauNS5YxDMMkh9MZLPjrOzsAKJVXzPx+0RaEIjFUBn14YvkudPXZ5/xpPLxkJ472hHGoM4TGqqBtO00p/uS5D7B6z1FUl6V2e4xjfKz4GIZhrGDFZ8Hu9m4AwMSmQQnr1rYewx9e24La8gCWbj/suJ/ZLfVYuu0wdh3uxh3PfgAA2LC/w7b9xGHV8BCwbHs7AOBYr32yvBU15X4015Sh9WgvJjdXp7QtwzBMqcCKz4JwTGBSUzUmN8WVx6YfX4xIVOCsuxciFImhz+Ti/M83z0JTTRmqgj70qa7NMr8X5/1sEQ529untPjrdfsrBS05swtnHfwgn3PYiAGDe9OaU5PZ7PXj9e+eiT7VIGYZhmES4d7QgEo2hKug1zn7g9cDvVepshqMiIcilKujDoDI/AEXh6cvLfNh7tFf/nixURQ6osarpmQyf1wMfuzkZhmFs4R7SRCQaQ0yo89tZKJCAz4M+Kc9Owy6YxO/1uIr81JCLVXOACsMwTPbhntWEVmXFPLGrRsDrQVhNNzAvt8LvJURd5PpZb8u3h2EYJttwz2pCs84CPo/NvHmq4jO5Ou3y5jKZFy8dVyfDMAzjDPesJkJScWircmF+LyFk6eq0Hr3LpHQYW3wMwzDZh3tWE2GpRqaV+9Lv9SASE+gLGxWf3VRDmSgvVnwMwzDZh3tWE/JksVaKR3NpdpsmqyWbaRsyUnwO5c0YhmGY9OB0BgBfffR9LNnahjENlTjUpeTcBXweeCyUmWYFug1YSRj7S0GXcfUVhmGY7MM9K4CdbV040NGHpdsPY097D+oq/Dh1bAMGlfkwY1Qtvn3hcXrbMyc0oqbcj0FlPpwwvBr1lQFcOs0+0fz0cQ2orfDr37987vik8vy/s8diyKAgZoyqy+zEGIZhmARIpDPBXAExc+ZMsXz58oz28ck/vK2XCfv5J6fh4yePyIZoDMMwTP6w9a+xxQfjOBzPXM4wDDOw4V4ePKsBwzBMKcG9PMwTuHIkJcMwzECGFR+MFVI4d45hGGZgw708jFVXWPExDMMMbLiXh9nVyZeEYRhmIMO9PIyRnBzcwjAMM7DJqJcnonoieomINqn/EzKuiehcIloh/fUS0UfVdQ8S0TZp3fRM5EkXWdmxq5NhGGZgk2kvfyOAV4QQEwC8on43IIRYKISYLoSYDuA8AN0A/iM1+a62XgixIkN50mLu5KE4YXg1zj1+MMY0VuRDBIZhGCZHZFqrcx6Ac9TPDwFYBOB/HNp/AsDzQojuDI+bVU4f34hnvnpmvsVgGIZhckCmFt9QIcRe9fM+AEOTtL8KwKOmZT8molVE9EsiCro5KBHNJyJBRKK1tTVFkRmGYZhSJqniI6KXiWiNxd88uZ1Qin7aFv4koiYAUwG8KC2+CcBEALMA1MPZWpSPNV8IQUIIam62LxDNMAzDMGaSujqFEBfYrSOi/UTUJITYqyq2Aw67ugLAU0KIsLRvzVrsI6IHAHzHpdwMwzAMkxaZujoXALhG/XwNgKcd2l4Nk5tTVZYgZRbXjwJYk6E8DMMwDONIporvTgAXEtEmABeo30FEM4noT1ojIhoDYCSA10zbP0JEqwGsBtAI4I4M5WEYhmEYR3g+PoZhGGYgwvPxMQzDMAzAio9hGIYpMVjxMQzDMCUFKz6GYRimpCj64BYiOghgRxZ21QygmMvAFLv8QPGfA8uff4r9HFj+7HFICHGR1YqiV3zZgoiEEMI2CqjQKXb5geI/B5Y//xT7ObD8uYFdnQzDMExJwYqPYRiGKSlY8cX5Yb4FyJBilx8o/nNg+fNPsZ8Dy58DeIyPYRiGKSnY4mMYhmFKClZ8DMMwTEnBio9hGIYpKVjxMQzDMCUFKz6GYRimpGDFB4CILiKiDUS0mYhuzLc8VhDRSCJaSETriGgtEX1dXV5PRC8R0Sb1f526nIjo/9RzWkVEJ+X3DBSIyEtE7xPRM+r3FiJaosr5OBEF1OVB9ftmdf2YfMqtylRLRE8S0Xoi+oCITivC6/9N9flZQ0SPElFZId8DIrqfiA4Q0RppWcrXnIiuUdtvIqJr8iz/3eoztIqIniKiWmndTar8G4joQ9LyvPVRVucgrfs2EQkialS/F9w9sEQIUdJ/ALwAtgAYCyAAYCWAyfmWy0LOJgAnqZ8HAdgIYDKAuwDcqC6/EcBP1c8fBvA8lMkYTwWwJN/noMr1LQB/A/CM+v3vAK5SP/8BwH+rn78E4A/q56sAPF4Asj8E4Ab1cwBAbTFdfwDDAWwDUC5d+2sL+R4AOAvASQDWSMtSuuYA6gFsVf/XqZ/r8ij/XAA+9fNPJfknq/1PEECL2i95891HWZ2DunwkgBeh1EpuLNR7YHlO+TpwofwBOA3Ai9L3mwDclG+5XMj9NIALAWwA0KQuawKwQf18L4CrpfZ6uzzKPALAKwDOA/CM+uM4JHUC+r1Qf1CnqZ99ajvKo+w1qtIg0/Jiuv7DAexSOx+feg8+VOj3AMAYk+JI6ZoDuBrAvdJyQ7tcy29adzmAR9TPhr5Hu/6F0EdZnQOAJwFMA7AdccVXkPfA/MeuznhnoLFbXVawqC6nGQCWABgqhNirrtoHYKj6uRDP61cAvgcgpn5vAHBECBFRv8sy6vKr64+q7fNFC4CDAB5QXbV/IqJKFNH1F0LsAfAzADsB7IVyTd9F8dwDjVSvecHdC4nPQbGQgCKSn4jmAdgjhFhpWlUU58CKr8ggoioA/wDwDSHEMXmdUF6lCrIUDxF9BMABIcS7+ZYlTXxQ3D2/F0LMANAFxc2mU8jXHwDUsbB5UJR4M4BKAJbTthQLhX7NnSCimwFEADySb1lSgYgqAHwfwK35liVdWPEBe6D4qjVGqMsKDiLyQ1F6jwgh/qku3k9ETer6JgAH1OWFdl5zAFxGRNsBPAbF3flrALVE5FPbyDLq8qvrawC05VJgE7sB7BZCLFG/PwlFERbL9QeACwBsE0IcFEKEAfwTyn0plnugkeo1L7h7QUTXAvgIgP9SlTdQPPKPg/LytFL9PY8A8B4RDUORnAMrPmAZgAlqZFsAyiD+gjzLlAAREYA/A/hACPELadUC4P+3d24hWlVRHP/9dfKGRkbZS8EUmYlWEhY+aIwYoUL1oESZWhIRJVIS3ehm5EMR3R6STExCocKCMNDU0opEKbMxpTSHkEnCwiAhKtRx9bDW1xw/R+ebaeqb6awfbDj7ctZe+xy/s2ZfXIvKCanb8L2/SvncOGU1AThcWB76zzGzR8zsfDNrxJ/xJjO7FdgMzIxm1fpXxjUz2tftL3szOwj8IGlUFE0BvqGPPP+gFZggaUj8e6qMoU+8gwJdfebrgeskDY9Z73VRVhckTcWX/G8ws98LVWuAm+M07YXASOBzetk3ysx2mdkIM2uM3/MB/ODdQfrIO6jLxmJvS/hJpO/wk1OP1lufU+g4EV/S+RpojjQd33P5CNgHfAicHe0FvBJj2gWMr/cYCmNpov1U50X4j7sFWA0MjPJBkW+J+ot6gd7jgO3xDt7DT6f1qeePe8/fA+wGVuInCHvtOwDexPcjj+If2Du688zxvbSWSPPqrH8Lvt9V+R2/Wmj/aOi/F5hWKK/bN6qjMVTV76f9cEuvewcdpYzOkCRJkpSKXOpMkiRJSkUaviRJkqRUpOFLkiRJSkUaviRJkqRUpOFLkiRJSkUavqR0SNovj07Qr6psbA/20SjpUE/J60K/K+TRF97uQZnNkgb3kKzbJb3TE7KSpLs0dN4kSf6XDAXm4BEXei2S+ptZW41tzwNmAGeZ2fHO2teKmY3rKVlJ0hvIGV9SVhYBT4YnjBOonv0V83G9WNJWSa2SZkm6T9LnEYPsmipZz0dcsl2SJhXKp0vaIunLkDUhypui/QpJzcC0DvSbG/Iq8dxGSBqGe2AZgruPWtjBfaMkrZP0haSdkuYV6kzSUzG72ytpRlXdUEn9JC2Rx5LbKWnL6XSK8gGSlspjsG0Frq7S6aF4djskvR9ur5B0Y8hrjtl5U8evMUm6QT3/93ymTPVIuKeJsbhXknuLZdXXp6h7Lq6vwp1Vz4/8TcBncd2Ie9qZG/km3OvFQNzX4VbgzKgbA7QW2rUR4YA60H0s8CPtYXmeJuLkRZ+HTnFfAx6J4dLID8O9g1TyBjwR16Nwn5wjCnVD8Ygg3wL9onx4DTotADYAZ+BGeTvwTtTNBl4ryLub9hA9O2kPidS/8qwyZeqJlEudSZl5DNgsaXkX76vsn+3AP+aV/JfAxYV2R4BVAGb2saQ/cKMyETd+n7rLTAAaYqkSYJ+ZbT1F35OBtdbu93MpbiQ64xJgNPBWoc+BUbYn8stD172SduCBRIs+Ib/HDdhySZvweH6d6TQZeMPcKfZRSati/AA3AOPxGSq4cT4cdZuAFyW9C6wzs5OifydJd0nDl5SW+MCvxaPCFznGidsAg6rq/4z72+KD/WeUt1Hbb0rAB2Y296QKaTTwWw0yuorw2WC39+vM7LCkMfis9FrgWUlX/kOdFpvZ6x30tVDSZXgUj9WSXjCzZf+gryT5m9zjS8rOImA+vvRXoQVfxkTSFNoDnXaVAcCskDMJGIzPrjYAU8OIEPVX1ShzMzC9shcG3AlsrOG+vcDvkuYU+rxU0pmFNvOifCS+rLmtKEDSucAQM1uPxyI8jDu4Pp1Om4A5khriZOisgsg1wD3hrR95VIIr4nqUeRSAl/FZc63PJ0k6JWd8SakxswOSVgL3F4ofB96QtAD/cLd2U/wvwDhJD+Kzm1vM7AiwT9JsfMlwMG4gt+DhZzrTd7ekh4GNkgxffryrhvuOSboeeEnSA/i+2U/4vmSFBklf4cu3d5nZz1ViLgCWyWPzNeCRw7eZ2fHT6PQacDm+N3goxnhe6LRS0jnAJzFz7gcswZdJnwkDfAz4FY9qkCQ9QkZnSJKEMFjDzOzfWGZNkl5FLnUmSZIkpSJnfEmSJEmpyBlfkiRJUirS8CVJkiSlIg1fkiRJUirS8CVJkiSlIg1fkiRJUirS8CVJkiSl4i9veOJ/FN55gQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# DO NOT CHANGE THIS CELL\n",
        "\n",
        "# Experiment configs.\n",
        "train_episodes = 1500\n",
        "discount_factor = .99\n",
        "\n",
        "# Create environment.\n",
        "env = catch.Catch(seed=42)\n",
        "\n",
        "# Build and initialize network.\n",
        "rng = jax.random.PRNGKey(44)\n",
        "rng, init_rng = jax.random.split(rng)\n",
        "sample_input = env.observation_spec().generate_value()\n",
        "parameters = init_net(init_rng, sample_input)\n",
        "\n",
        "# Initialize optimizer state.\n",
        "opt_state = opt_init(parameters)\n",
        "\n",
        "# Apply updates\n",
        "def apply_updates(params, updates):\n",
        "  return jax.tree_multimap(lambda p, u: p + u, params, updates)\n",
        "\n",
        "# Jit.\n",
        "opt_update = jax.jit(opt_update)\n",
        "apply_updates = jax.jit(apply_updates)\n",
        "\n",
        "print(f\"Training agent for {train_episodes} episodes...\")\n",
        "all_episode_returns = []\n",
        "\n",
        "for _ in range(train_episodes):\n",
        "  episode_return = 0.\n",
        "  timestep = env.reset()\n",
        "  obs_tm1 = timestep.observation\n",
        "\n",
        "  # Sample initial action.\n",
        "  rng, policy_rng = jax.random.split(rng)\n",
        "  a_tm1 = softmax_policy(parameters, policy_rng, obs_tm1)\n",
        "\n",
        "  while not timestep.last():\n",
        "    # Step environment.\n",
        "    new_timestep = env.step(int(a_tm1))\n",
        "\n",
        "    # Sample action from agent policy.\n",
        "    rng, policy_rng = jax.random.split(rng)\n",
        "    a_t = softmax_policy(parameters, policy_rng, new_timestep.observation)\n",
        "\n",
        "    # Update params.\n",
        "    r_t = new_timestep.reward\n",
        "    discount_t = discount_factor * new_timestep.discount\n",
        "    dJ_dtheta = compute_gradient(\n",
        "        parameters, obs_tm1, a_tm1, r_t, discount_t,\n",
        "        new_timestep.observation)\n",
        "    updates, opt_state = opt_update(dJ_dtheta, opt_state)\n",
        "    parameters = apply_updates(parameters, updates)\n",
        "\n",
        "    # Within episode book-keeping.\n",
        "    episode_return += new_timestep.reward\n",
        "    timestep = new_timestep\n",
        "    obs_tm1 = new_timestep.observation\n",
        "    a_tm1 = a_t\n",
        "\n",
        "  # Experiment results tracking.\n",
        "  all_episode_returns.append(episode_return)\n",
        "\n",
        "# Plot learning curve.\n",
        "plot_learning_curve(all_episode_returns)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dowFJ_l-32A"
      },
      "source": [
        "# B) An alternative update\n",
        "\n",
        "You are going to implement a different kind of agent.\n",
        "\n",
        "Like an actor-critic, it learns online from a single stream of experience, updating the parametes after each transition in the environment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aQD0Qw8-_QJy"
      },
      "source": [
        "### Neural networks\n",
        "\n",
        "\n",
        "The agent will reuse the same neural network we defined for the actor-critic:\n",
        "* the scalar output will be trained via TD to estimate state values\n",
        "* the vector preferences `p` will be updated according to a different rule."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I6j0AIF8GhdR"
      },
      "source": [
        "### Choosing actions\n",
        "\n",
        "As in actor-critics, the (stochastic) mapping from `observations` to `actions` depends on the vector of preferences `p` from the network."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jE00LhDL_nbC"
      },
      "source": [
        "### Q6 [2 marks]\n",
        "\n",
        "The new agent's policy will have the signature `action = epsilon_greedy_policy(parameters, key, observation)`,\n",
        "* Take as inputs the current network parameters `parameters`, a JAX random `key` and the current `observation`\n",
        "* Return with probability `0.9` the greedy `action` with respect to the preferences `p`, ties must be broken at random.\n",
        "* Return an action uniformly at random with probability `0.1`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8IwTDROL_Lx5"
      },
      "outputs": [],
      "source": [
        "@jax.jit\n",
        "def epsilon_greedy_policy(parameters, key, observation):\n",
        "  \"\"\"Sample action from an epsilon-greedy policy.\"\"\"\n",
        "  _, action_preferences = apply_net(parameters, observation)\n",
        "\n",
        "  key1, key2 = jax.random.split(key, 2)\n",
        "  greedy = (action_preferences == action_preferences.max(axis=-1, keepdims=True))\n",
        "  greedy = greedy / greedy.sum(axis=-1, keepdims=True)\n",
        "\n",
        "  uniform = jnp.ones(3) / 3\n",
        "\n",
        "  p = jnp.where(jax.random.uniform(key1) <= 0.9, greedy, uniform)\n",
        "\n",
        "  action = jax.random.choice(key2, jnp.arange(3), p=p)\n",
        "  return action"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l1zQiRX5BfeA"
      },
      "source": [
        "### Q7 [3 marks]\n",
        "\n",
        "The parameters $w_p$ of the preferences $p_{w_p}(s, a)$ will be update according to the following gradient-based update:\n",
        "\n",
        "$$\\Delta w_p = \\alpha (R_{t+1} + \\gamma v(S_{t+1}) - p(S_{t}, A_t)) \\nabla p(S_t, A_t))$$\n",
        "\n",
        "where `v` is the state value trained by TD as in the actor critic.\n",
        "\n",
        "You musy implement this in the function `preference_gradient`, with the same signature as `policy_gradient`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6fQhAWXqLNt-"
      },
      "outputs": [],
      "source": [
        "def preference_gradient(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t):\n",
        "  def some_function(net_params, obs_tm1, a_tm1, r_t, discount_t, obs_t):\n",
        "\n",
        "    _, h_tm1 = apply_net(net_params, obs_tm1)\n",
        "    v_t, _ = apply_net(net_params, obs_t)\n",
        "\n",
        "    target_tm1 = r_t + discount_t * v_t\n",
        "    td_error = jax.lax.stop_gradient(target_tm1 - h_tm1[a_tm1])\n",
        "\n",
        "    return td_error * h_tm1[a_tm1]\n",
        "\n",
        "  grads = jax.grad(some_function)(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t)\n",
        "  return grads"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hz4RFKnVF8E8"
      },
      "source": [
        "### Updating shared parameters\n",
        "\n",
        "Just like in the actor critic the overall update to the parameters is a combination of two quantities:\n",
        "* the new update we defined for the vector of preferences\n",
        "* the same TD update to the scalar output that we used in the actor critic"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QnghCzKhF-gI"
      },
      "outputs": [],
      "source": [
        "@jax.jit\n",
        "def compute_gradient(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t):\n",
        "  pgrads = preference_gradient(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t)\n",
        "  vgrads = value_update(parameters, obs_tm1, a_tm1, r_t, discount_t, obs_t)\n",
        "  return jax.tree_multimap(lambda pg, td: pg + td, pgrads, vgrads)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "44QBKEF4HFK8"
      },
      "source": [
        "### Optimisation\n",
        "\n",
        "The gradient updates are rescaled using the same optimiser used for the actor-critic."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dbdHQo23FJit"
      },
      "source": [
        "### Run experiments\n",
        "\n",
        "Run the cell below to show the performance of the new agent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ept5NG1oFLnu",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 355
        },
        "outputId": "6413bd01-474f-4376-d059-5bf4bfcb5d7b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training agent for 15000 episodes...\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 504x360 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAb4AAAFACAYAAADDFRmAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2dd5wURfr/P89sZFlgA2mXtCBJclxEBAURMWI6xRzvTs/TU+9UPD3lTF/TneFnPs/sGQ69EyPmcCoKKkHJAiI5x4VlQ/3+6O6Z6p7qNNMzszPzvF+vfW1Pd3XV09XV9VR6niIhBBiGYRgmWwilWgCGYRiGSSas+BiGYZisghUfwzAMk1Ww4mMYhmGyClZ8DMMwTFaR9oqPiN5NtQwMwzBM+pCbagHiZejQoUcCYJsMhmEYRobsLqR9j49hGIZh/MCKj2EYhskqWPExDMMwWQUrPoZhGCarYMXHMAzDZBWs+BiGYZisghUfwzAMk1Ww4mMYhmGyikAUHxE9SUQbiegHm+tERA8Q0TIimkdEQ6Rr5xLRUv3v3CDkYRiGYRg7gurxPQ1gosP1owD00P9+A+ARACCiMgA3ARgBoBrATURUGpBMScO6ma/x2+8mv0IICCHQ2Bi5Tz6ORRZVHA2NwhRO/m3IYMhh3NtoucdKgyUNObw1HlUYVdx25wzZDLmtf3bpWPPX+syxYsSpiscqlxACdQ2NJlnke4znMmho1MLbxWn37Nb3K98rp2WN1/jfoHgeazp28TU2CtRbZFal41am3K7L4YywjY3R8srIz2UXv6q8qOKRy5HqPfhB9R0EsUm4XJ7c3r3bezPuc8o7a5xGGGt+ybLJYZJFIC7LhBCfEVGVQ5BJAJ4V2hPOJKISIqoAcBiA94UQWwGAiN6HpkBfDEKuZDBz+RZMfnwmnjp/OMb2aosPF27AFS/PwWNnD8W5T36DM6o746+T+nmK67IXv8enSzZBCODMgzqjS1lz3PLmAjx+zlCM7tHG9f59dQ048MZ3cd7BVbjpuL4AgBe/WYXb31qIF39zEPp1aIUb/jsfz89cBQD4/Jqx2LmvDic9/CUGdizBKxePxPlPz8LyTXuwamsNAKC8eT7+ffFInPDQFzi0V1v8v9MHR6V77/tL8MgnP+Gxs4dibO+2+GDBBlz07GwAwFPnDcctby7Artp6vPH7Q3Ds//scR/WrwF+O7YPxf/8UXVs3xx/G98CFT8/CI2cNxUHdygEAl7/4PT5evBHzbpoAIs3z0J3vLsIjn/wUlX63Ns1RkJuD0qI8LFq/C1v37EdxQS7+e+nBaFNciLF/+wRb9+wPh68qL8LvDuuOa16dh/evHIOb31yA2vpGvPLbkZ7ek0xdQyNG/t+H2Lx7P6q7lmHOL9tx4qAOuPOUAQCA0x6fiS27a0FEOKR7a2zZsx9vzF2L6qoylBfn450f1qOkKA+zrh+P+gaBsfd8gl376vDuFWNABBx9/+fYua8elxx2AK6d2BubdtVi+G0f2Mpz18kDcO8HS9C9bTGeu3AE/vX1Kkyd/iNuP6k/ypvn4/ynZ+GV345Evw4tMfaeTzD+wHa47cT+mLd6O45/8As8fOYQvDF3Ld75YT0A4LBebfD0+dWo2V+PPjfOAABUV5XhlYtH4vU5a3DDf37AsxdWY8POWlz58hz85dg+uHvGIuzcV48nzh2Gsb3amuTbu78BY+/5BOt37guf++iPh6Jbm2JTuDfmrsVlL34PAHjltyNR3bVM+byvz1mDq16Zi4ZGgRaFuWhoFPj7qQMxsV8FfvXoV6hrFHj90lEAgOv/Mx8vfK2V/YGdSjD3l+0AgA+uOhTd22rpf7hwAy58Riu7XcqLUJibg9219SjKz8GMK8YgFCJ8tmQTzn96ForycjDtkoNx5H2fRcn1+NlDMaFve9v3JHPv+0vw4jercHT/CnywcAM+uOpQEAEH/uVdnHVQF9zssf6wUt/QiPF//xRrt+/Dw2cOwUXPzsYV43ugumsZfv3MbNx+Un9MGtQBN7+xAE9+sQJtWxRg465a3HXyAJw6vFNUfJt31+Lwv32KUd3LsWjdLnRr0xxPnDtcmfZ1r83Hi9+sQoiAt/8wGic+9CVKi/IwtKoMHyzYgOcurMYpj34FABjdo7VWDt9aiLt/NQCTBnWI6Xn9kKw5vg4AfpF+r9bP2Z13hIimEpEgIrF27dpABfWLURHf98FSAMA97y3Brn31uOOdRahrEHjmq589x/XmvHXYta8eu2vr8diny/HAh0uxt64Bb89f7+n+n7fUQAjgqS9Whs/dqiudd/WKzFB6gPaRL92wG7X1jfhm5VYAwCeLN4WVHgBs2bMfX6/Yip376vHGXHVeP/LJT9jf0IjZP2txvD1/Xfja9f+Zj+Wb92DTrlrM+WU7Nu/ej+dm/owde+uwamsNPl2yCQ9+tAzbaupw7/tLwvdNn7sWu/bVQ25UqpQeACzftAcL1+3Elz9tCSu43bX1mPvLDqzeXmNSegCwcksNrnl1HgBg2ner8fnSzfhmxVZ1prqwY28dNu/W4v9mxVbsr2/Ey7MjRfqbFVvx06Y9WLZxN57+ciXm/LINADBn9fawctleU4ftNXXYtKsW63fuw579DVi+eQ9Wbq7Bzn31AIB5q7VKeubyLY7yXPPqPKzbsQ+fL90MAHh51irsb2jEm/PW4q4ZiwEAj3+2HJt21WLDztqwInhWL6e3vbUwLBeglQcAWLphd+SZ9LJy5zuLsKu2HtO+XY3nZq7E3roG3PrWAmyrqUNDo8DCdTuj5Nu8u9ak9ACY8svgFenc45+p3zsA3D1jcbinsGtfPWr2N+Cf/1sBAJj987awcgMQflYApvP//X6NlNby8PHPW2qweMMurNm+F0s3at8JACxYtxMNjQK7auvx5jz1N3G3ntdeuP/Dpdi4qxZPf7kSq7ftxbod+7Bu+z40ish7iYVd++qxcksN9jc04sVvtGe/74OleGXWL9izvwEPfKjVWU9+oeXXxl21AID5a3Yo41u1tQY79tbh7fnrsXzzHnywcKNt2kZ6jQKYvXIb9tY1YO2OfXhj7lrsrWvAm/MidcTnSzfj+Zk/Y39DI2b86K2ui5e0XNwihJgqhCAhBFVWVqZWFsvv/fUNAILptjeEa31vcTUqhh7260MXqmsC3uTcX+8y/BEeptFPSK5h9+xvUMonHxtDefsVwywqub0iADQ6i44Q2fqx9YTfoeiwPJbbiOT3rQ/fKYYrYxVXiMgQE5FWIVmvO96vOFcrlQvjfnlYVhVnLO/T6RbrMLCWhvV+5zRV+ayiQTEkZxd1PMVKCBHX/QZyeapXPFddg1r4BpuH8lvWDbxMYzSGy2YAD+6BZCm+NQDkvnNH/Zzd+bQhXJkkIO6QHqlb5R2RJfqcUZDsymwQii9kSUNWJo02H5/qQ1QRT/uhUQjUu2RevO/N63NEwqsbIkIADZKs9TaKLx5FLcLvJ74GhQpDLrkyVVWUquwixVvw+pxewrmVca850dCgUHwJ2hhGlSd+keWU33fke7VRfDYK0W9ZV8lhh1DUHYkkWYpvOoBz9NWdBwHYIYRYB2AGgAlEVKovapmgn0s7jPcVZIvFKPxePy5VOEN52rV67Vp3MqqemAkjDUQ3AuToY1m0E1cFLdzvj/dD89uzN7IyWvEJyNmsTfpHpxOPtJH3Q4EsnJBRZaMqa1TvI6S4V47PSVIv+eFWxr1mharHlygC6fFZFklFItf+2T130D0+1W3WU0a5UJWFRBDI4hYiehHaQpXWRLQa2krNPAAQQjwK4G0ARwNYBqAGwPn6ta1EdAuAWXpUNxsLXZhI4ff6YSp7fHBp3QXQ4wuXVT0q+aO16/Gp0lWV+Xjq50aLMlFhkrVRIOTzy/NbCRr5Yb2tUcDUO20UAQ11SjcI6f0EWXfbyaQcXlecU91vbjzZC2vX0DSvYLS93TV+GeMdyM9lO9QZRxOlMaihTpsRFjfZ7BScl0ayMj5lObD81v8nSe8FtqrzdJfrAsClNteeBPBkEHKkgnBlYnM+HozeiNeoVGmGXJSnlx6VW4/PKqfdUKc8lKf6iFSSxDvH56aY5PfWIARCPj89v5WBnTyNQpgq6IZGYVJORjq+RxTkSlr/HyIKdKhTCHXPWWkqEPBQp10wocg7O7zmhJFn5qHO4GloDGbkyDSnLskc6yhQrD1d9foCoQyTaUOdGY+1oAZZsXiNSz3U6TzHV28zni/j2uMLz0VG90pMlbelYvdCXIpP+JvfieXD9jv846T4GkyNBGuPT/vvp1qwytYotdK8zhvbYa00UzbU6aG3aTdnFY7fYxkzek1eykk89XdDowhkyM80wiI9Y/h7tasT7Hp8QQ51Wkc8jPLNii89sJt/i3UiWCZcBjxGpWxNhwu5fYUbPraRWbVyzpRGVPry8JpNj08x8a4q8vEubvEzvxPLh+33PdulYV3cEu1kQLvmp2JoEEKpQQjxN8xMU0Zk9+4UPT5VUVI+U/QQrQq7YTtVb9kOr6+wUaH4Ap4qDccfxOIW+XuWZXZbO2A71Bloj09NWs3xZSvLNu7GF8s0u6pvf96Gr37agmUbd4evGVRNeQvdWjfH8s17wueevaAaY3q2wUXPzMIHCzcqDXRXb9sLAHjt+zWYclRvtG1ZiPqGRpz/9CxUV5WhX4dWeOyzn/DYWcNw1P2fYe2OfaY0/zShZ7ii/F6yWzJ46ouVJpu9xRt2KZ9TtiWqmvIWHj1rCCb2q8CzX63Ene8sCpssPPnFirBNkIG8yu8vr/8YPr74+W/Dx0YefrcqWsZrp81D84JcXHd0b6VsTny2ZBMK8nIcwzz48bLwcd+bZqCiVSG+uu5wfLdqGy594TvUNwps0u2bZPpWtkR+bgjH9K9Qxls15S2cd3BV1Pm9dQ3RgQGMvutj0+8//nsuRkhlYsmG3Zi3ejt+rTsG8MKabXvD9mrz1+wI2zMSEa7/z3xT2O91+8I12/dGxfObZ2fjq5/M9oMnPfIltujxrdq6F7mKGmvJhl2omvKW6VyLgugq54EPl4ZtygDgv5eOMunCT5dswsT7PsMzF1Rjxo/r8eniTfjHOcMQCpGyovz2522mynbILe9HB5JYo39nG3buw+yft9mGG33Xx/j92O544n+RMv7op2obw0Xrd2HE7R9gw87osjOwUwnOHNEZz8/8GYWK8rlkwy4c9+D/wr+XbtiF616bj7tOGWAy9H/pm1WY8lrkPZ4/qgqXju2Oi5/7Fj9t2o1e7VuEr81bHbHNM+wmN+ysVb7vd35YH35vd5zUH+1bFeKWNxfgp017osI+/MkyHNm3Pa6dNg9XTeiJM/7xdVSYuvpoNbdbt081MORIUoePFV88WCuP0/8x0zasrPQArcCM6dkmbATqZkD9/NercNURPbFxVy0+X7o5bKAMAHe8u8ik9AzueW8JOpQ0w469dWhZqL3qVs3ysGNvHQCYlB4AzFEoRxUXP/8dVt5xDG6UFJlfft5SE3Wuf4dWUefe1Q1aTxjs314zLzeE7TX73QNKrNPz8bwnvwkbj6v4ca1mnP29QlkbPP3lSl9pW/naUiaumTbP1/0fLYoYGPdsV4yZy7X4DmjTPMoZQUmzPNt43luwIeqcbABe2aowrARl/rdsc9S5XbX2eWpw4dOzcESfdqZzi9bvwoMfLcNzM7VG2JY9+9GmRYGyBzygY3Q5ciIvVxv4kpWvHXJDyQ2V0gO0vJvr8K19u8qsfK99dR6+W7UdN03/Ec9dOCJ8XlZ6gNaQLcjNCStv4307cdPrSvfKtmlYuevdxfhsySbM/nmbUukB6hGjbTbfZX5ucgYheagzYEqK7CsQGbd5Myu1DobxTq2kNi0KTL9bF+fbhk3EMu2y5vbpWSl1CBuLaMKyMtLXvTHdlVg271ZXpHbYvU+VkmuWb+55PHb2UAz0qECICHk5qsUtnm63idPbuWJFDzI3ZL94R9VDLC7Qnr2pvHO7OUcv+enXTMXOiN0PbvPFqqFmu/eTiKFjFaz44kA5oZ8oGx89Wq+LAwysMjpJlwjF5+b0VsbNYbFfGhvjWMTRVGpBCb/vp940jwrlsYFqNZ3X+UQh1HNS8WSh6r2pVguryn6DsG8o5eaoqjx9zquJvHPrs0dWTbsL6PcRgvjm3eRSOjKw+S6T5aeaFV8cKD/2BL841WKKHD8D4w7yJULx+YnSqXcWy2Ihqy1cuuPfWF52Hxa9mEjGquQI3udb7OzOYnZxBffFN+G8UCRsdfcmk69QfBHl2TTKirWHFFk17X6v3zx3dU7hAbckVeXWbsFR0I4V7GDFFzAJq2gtJgOmSw41lPWKk3wJ6fH56HI5BfXTcwzHJ5pOKz4IgurxOdl7GhCRZ5sqbfFodNhYDZ4B5xXKgLMnm/pGAWFTXHIVQ7LJWlDhFWtZ9+PByW+Ou63Y9oKbslI1Wu0UNA91pgFebZcCQY9XVZl4qaCM2xyHOhNQ6vwMNTop5boYMjbT5vjiMZa3cxBuoBzq9JhOowje646qMpVljPguVcjj0OPLDUVXedTEhjqtZd2PBye/zxDMUKf/NGzdonGPr+mjVnzeXlyshpoqg3PltIVNOk7ipbrHF1bOCiEbYpisa6pDnaql/17C+80Cux6fqgUe1eODdy8aQe0mIMenHI6Vjp082TQ4vHfVIhy/rgETTZ1l4Zsf+fyW90AWt7hEoezx2dyTrFfAii9gEv3xeG2tWzGGSZI91BnLHJ/qnlg+0EaHRQ6pJM+ppeIQPp4en92O7AZRc3wEz12+RiECdzXlNtTZ6NrjU8ebo7ghWW6yvGJVFH6d1fshiKFOt0pP1Wi1G+pMVkOVkjWZmCiGDRsmZs/2btQbJGc98bXSVikRVFeV4Y8TeuK0x6NtBSf2bR+2d7NjVPdyvHDRQRh1x0dYt2Nv3Aph6W1Hocf178QXiYIHTh+MN+euVdqO+WV0j9Yme0evlDXPj9q8NkhaFOZil4ONYLzhVfTr0BI/rIneGNaOJ88bhsc/W+7JFgzQeqVBeCsKircvH42jH/g86nxVeRFWKmxI0wkv33s68+olIzG0S7RDjxiwbdFwjy8OktlQbNksT6n0AHj6COT2jarV64fubYsdDXANCmIwRr38xe9dlV7v9i08DRfG2oNNpNID4FuJxfu+ALOfVC8M7lTqy21WU1J6LQtzsXSj2gtRsnxBJpJMVnoAcPIjXyU8DVZ8CWL57Uejt+QyKF6aF5gNjNu1LLAJ6YxqWKpFodqBzy2T+oaPWzXLw1fXjQOgKR43pTKud1ulcXG83HhsH7x7xRgsvGUihnQucQybjH3TDI4b6N2zzMS+7cPHHUqauYY/c0TnqHOXjj0gfPyXY/u4xuFnmfu/fj3C1pnAg2cM9hyPG4d0b+14/bBebXDnyf19xys38m49oR/OPqhL+LeXBtO5I7tEnetT0dI2/AWjuuK6o/y71EsWhXnBVfMXjOrqGuZ4H99Cl/KieMSJGVZ8CSIUIt9zOU5Y661Y63TVFjJ2PbOC3IiyDVFkrslu+bpMXg4lpEdsLE7Iywm5zngkcxS/yMUnqIzslslLb061ErGZlJ6Xis3PIiNVepFrwb1Ut70PfdmnSjSIyCa+OSEKpMesWhRjkJvj3fQjFcSaj8q4PFRpzXx8C7GMCgUBK744cKv8g/wWVDt2+yFizhC95YldRSfbPMl2XV42yswJUWKGlRQbq9qRCPMMO/JyvT+r3CDyokhUYeQ48hwUlYGf3q8RtWoxRTKHCmNNS97SyY8hvhNOSpqo6dkCxorbc3h5Jx6KYyRsijKOFV8CCfKVWhVdrD2+RhH9Edu1iGX3TiGKrKBrFML12XJCfrd09YYsaiz2Q4nCT+8+X1KSnnp8irhNytOhN2LgZw7OqTJKZkUVa1KNUo8v5MMQP5JudHinBgohQY28FOCWV14e009eeCm7iYAVXxy4vbIgPwbrSJXfZb9G61011Gk3jJNn+tgjH7eXOjSHElNJmhZcuO61lzzF58ervOw2y9tQZ3QYOT0vStdPI8AYAfDi4SWRxJqUaRNf8i+zm/G8FYohjaaKa53m4a34yYsgh2H9wIovDtyHBYJLy6roYneGrRjqtKk47Xp8QrhbFOWEQgmpDOQ4XX0EJlPx+erx+eutqcKYFV/APT6HR0mPHh8ce3yxxOv0nvwY+zd13J7DyzftJy+CmH+NBVZ8ceDeOgoOa73lt06PeEWJLph2wzjyeZJ6cI3CXfHmhBIzH2TeUNzNcDbw5G3xN9Qp9/jc71NVDvKiAKfFKAb+5vgMg+loklm/x6NM6owd6xFMOXQb/s0QvedaaXl5znRQfLwRbQIJshX4wUKzbZuXDT1lvl6xNbyrsnWPPrvWrHx+w87a8PN8tGijaZNTFTv21iW8MthTq97N3GDhOu8G2/HiR/H5XdyiCuF3js+PbaLT8FOQjRm3oeh4kvp40SYAWu81iLrVbaizKc/x7dnv/J3IuO0T6qVO2+2jbrJTfLX1DaZV5UHDPb44sPOiZQx7BfEtHNWvvWsYL7ZgMtaytm1PnTJcy0LzhqV+5rFm/LgB2xJgCG7sWA8AG3ZG7zqfKvpW2tt5WamRKiJPeaQoSBWtCsPHfoZZvVBerDWM1m7fG3XtgDbNA0tn/IHtHK8TCEM6l8YUt9FQDJHZo8y2mjrXkZihVdFeQ2r221fmOSE/pv7pjRcFP+3b1Z7js1OkKp/EQcKKLw5aKXayBoCXfnsQAPVE8N2nDDBVWjJDu0R/5Dcd1zfqXM92xabf3fTK6Ig+7fDNnw/HW5cfgmkXj7Q1urW26O2Ua1VrcyVn1zr7Yso4TB7eKeq83Y4K3dtG5H/1kpGYdf14ZTgVO2oiStoqz2u/O9jTruF3nNQfr/x2ZPj3nyb0dL1nUKcSPHtBte31ypLIO3387KGmaycP6Yj2LSPXq8oj+dpZMuB9+TcHYe6NE6IaMqps79YmkoctCtXl0A2VvdXn14xFmW68PrpHG9O1jqXN0LG0CO9dOQa/Gtox6t6e7Yrx+TVj8eNfj7R13jDjijFYcPOR+PyasThnZBe8fumo8LVvrj/cFJYI6NGuBW45oZ/rs8gG/VaOlBwGnFEdKadje7VRBceEPtEK2WmO9MTBHQIb3SnK99/LsSvzFx7SFfOnTrC9z0u5l/n2hvGuCv7iQw8wfd8yP/z1yKhvvbRI7Sih0IctYCyw4ksAhl2V6lsoKcq3LdxtW0R7Y1ENY7UuNoczFr4M6NAKbVsWom9lKwyrKsNgG88mVnMGu56c1/H3DiXNfI3Vd5UUamVJs6ihVydMeWqpi4Z0LkXv9u49r7Lm+SavL9b8VNG1dXPHHq/cEi7Iy0HnsohC69W+GMWSdxx5eFMesuxcXoRWRXloZikfqgaUnN1BzpPI78K6aKZ5vvYMPdu1QKeyaI8bISJ0KitC84Jc20Zhr/YtUJSfi05lRSAyO3lo28LcIDTyVG402FHeXP0OrfNvOaFQON52NvGqvlunHNZsVl1F9ER5sVoROGFXLodXlTo2iqz57UaLwjxXBS87Uyi3eP8pLsiNcrZgtzAr0XN/rPjiwN2cQX2PXdtRVahUc0DWYOG9ySxhvaZjNz8VpJcOc3qReP22lOXwqufzYjxrXenn5SMTLjsQyFdyLJVtiMi0KjdkUnzSAiKbEqVK1q/8Ktx2+rDGKsutSlG+N4gekBGDF7MUu+TkRVmaXJFrdnOjqvfgNMSXE6KUmjPYlxtnofy+olwPCl4u695WLKex5xYimkhEi4loGRFNUVy/l4jm6H9LiGi7dK1BujY9CHmShdPHBti01J2Wiiu+Hi/bqNh5o7KrMKxR2rW64q287O6WVyH6TULOP+WebR49S5CHClCmUTjLaqpcQxYFQmTS0vK3LueF3SpVVaUahOJTNhykqKx5KcutKquhkPo4ViIOE+KJw/wVEkXKpV35Vue3fRpaQyelmk+Ju2mCz0anBwUfklr2yk1/PdoQJ5q4V3USUQ6AhwAcAWA1gFlENF0IscAII4S4Ugp/GQDZ0+1eIcSgeOVoSoRtiBQfP4Fsu2KqIuBpqXp4U061HFaslZZdqytRww15JvvAOHp8qj3bPMRBlorKi0mBW91rUqQWrzUEs5KW05Pzwu59qXsgkeOY35NyzzuHno2PijSQHp8eRzyOCAjmvJKfz05CVR44LV9J9eIWu7RdlVQMjRP3XiSFi5WqQWk9E6Q/Yz8EkWo1gGVCiOVCiP0AXgIwySH86QBeDCDdlOM61GlTYdkPQSrOqZSntccX9kvo7fPzaseXqOGbeIY63T48L/F5fX4/mHtg5so2ROZei6xA5LwwgkRJ49Lji1V+9VCnlKwlWtNQpyJJk1IJcqgznjhUSozsrwE25d6pxxdKrZNqu6SD7vE5pSXHGR7qjNEBezIIItUOAH6Rfq/Wz0VBRF0AdAXwkXS6kIhmE9FMIjrBS4JENJWIBBGJtWvXxip33MQyhu50j3L4SBHeGswwJvfc47OEs5scj7fysquw8iweYfxg9tWpGup0j8Pac/E6x+cVzWuNWQnIssrpyXlh5xTArQwEOdQpv3Nrum49OrmBH0SjKeIiz8Mcn20c1mFnKI9V6XqJHwh2cUss+JkbNl+PQfG5NK5DFKl31EOd5t9+nLsHSbLV7WQA04QQskVlFyHEMABnALiPiOzXJesIIaYKIUgIQZWV3vd+CpqPFzsbcasKVoiAX7aqd4BWV3DR5zbvrjX9nrt6h5aeJZxdhWGNM1FDmnbGsNZdH/wgKy2VnvDW47PI40nxucQpxZFDZHKXFiLzPKyc317mF1UhErW4xSldOXy9wi2OLNNeH0bTtumTka6XsHa9N/N5AqFOtxHz0+NxCprqHp8dQbgf83tPSGrkqYc6zee87CySCIJIdQ0A2Yiro35OxWRYhjmFEGv0/8sBfALz/F+TZdbKrdheozb8NpaEq8pIy8I8W5ugbgrjYFVB+2GN2iOJtaCr7AIBRMltva/Sxs5QhWELZTU0dqqMaySPK34/voGdImYIqg1aVR5KWtpstGvg5M7LsE88uHs52jks/5Yfo7gwF8s37Ymk3ywPkwZFGmhy71E2e0y+zC4AACAASURBVDAcBhxhsSObuzp6t/sg5vjsbKgMZFtBAKZneujjn6LCy3n/9YqtUddVxu+tW2gyqExajKfq1trZaL5dywL0srEbJEvcfSRHA3682Vi/meFVpajSbTDzckKOnpSqFQbxVgzbt8N7t8PwKn9G+/06qE14WtqYlADA4M4l+HlLdAP8yL7OTgXW7XB2GkEEjOvVFoC2GbWB3WbXqdqdIQiXZbMA9CCirtAU3mRovTcTRNQbQCmAr6RzpQBqhBC1RNQawCgAdwUgU8JZvU3dawOA9rriUDW4BtgYm/774pEoaZaHu95dbDrvp0dkDXrqsE44oG0xfvXoV6bz5cX5WC95PZHrzTE92+D/TuqPvRZPFXYf70WjuwHQjHi7lBfhic9X4N0f16MoL8dUGbx7xWhMvO9zAOYhSrtW6c2T+uLG13+MOn/MgIrw8TVH9sbyTXvw6ZJN4XMlRdEf++Tqzji6fwVOeOgLAIiyQXPqUfx1Ul+cMLgDhnUpRW5OCPdPHoRlG3fj/320zBROfo5OpWYD9KP7V+D4gZUoLy7ACYMqsUWqcM8dWYV2LQvRLC8HrXTZLz+8Bx7+RFMs/Tu0ivKg8sWUcebFOdJxl/IiPHTGEFwzbR4W2Lhs61JehEsOPQCbd9finveW2D77r4Z1RIfSZrj1zQVYunG3yf50b110j+5AG4cJc2+agNe+W42TFUbvbVsU4s3LDkFHPc+umdgr/A0YedqvQys8cuYQXPLCd1H3nzykI247sR8K83Lw2u8OxpxV23Hzm+F1dSDS7PVe/s1B2F1bjzE9Iru+96loif98b26jfzFlnOn34M4luOGYPrj97YXhc53KmuHp86uxY28dttfUIS8nhDKpEfHoWUMxpHMJduytw67aelSVN8eQW94PX3/q/OGoqW3Apf/Snue4gZW4+5QBmLVyK4Z1KUOjEFiyYRcqWjXDHe8sRFFBLv719aqoZ580qBI3HtsHzfJz8I/PV4TP33DMgWjbstDWsL1lYS6eu3AE7ns/+t3ffmJ/7KltwP+WbVbfKynT1y8dhW5tmmPZxt048eEvAWjv7Ppj+uC4gZUY0LEk/J18dvVYAOY66vVLRyEnRLjvg6WmNC4/vIcy7SCJu8cnhKgH8HsAMwAsBPCKEOJHIrqZiI6Xgk4G8JIwT5YcCGA2Ec0F8DGAO+TVoE0Zu1Ei2XBXVanbKbLhVWWBzxOEQoThCoVlNQiWhx8qWxWiQ0kzdG9rbkEbFVNri4GtMUcVChGGVZWh0sZ9mmxY7mX1n53RslzJN8vPwWEW7xvNC6Lbcnk5hEFST9HaynTqMBXk5uCgbuXhla+TBnVAvw7RFYrdykEtfc1o+sJDuqK8uMDUQwuFCEf3r8BYqXVcmJcTViKakbU5vijPLlJ8d508AP06tLJtYLUszMWnV4/F5OrOrjug5+WEcGjPNuGRA7ehUTtHBC0Lc3H+qK5RLvAM+nVohRJdcci9ajkbj+pfYb0NAPC3UweGvXwM6Vxq28sZ0a0chx/YzvRurI4CTq/uHJW3E/q0x9AupabvelCnUjQvyEVlSbNwD1J+p4M7l6Bty0L0aNcCQzqXhj3hGIzt1dbUgBvdvTUK83IwukcbNMvPQfOCXAzuXIr2rQpx3+TBuP3E/sq64f7Jg1FeXICi/Fxcclhkhmh0jzY4fmClbV0zqHMpigtylXEW5OXg4O7lyvsAs3P0gZ1K0KIwD4Ol0R4ibc3AsKoyU56UNo8eXRjYqUS5A7u1jkkEgTipFkK8DeBty7kbLb+nKu77EkD/IGRoKsgLFGKZOo4Hr71D69CYp2Ez/XQQW7yYV/+pw9gpxKiVhlELMNzTj2Vxi4xqEYpJ8bnc72ceUgjhqqBMK0j1sNY8Vuksr+t1/My1qe/3nr/ylE9MZcvy20lmL48TIvN/VRqAt0UzdngJH7LMG0dfVx87p6tegxAPQcx1JmM3MfbcEiN271duFSfiA3C832O46B5P5LedXY0RJkppehcvjMkQ2qOCi6TnrOhU8Vk/JKsicVMsVlSVqSyX23v0skI0sgWU97CaHNEyxFsVRezp3MLFmRCs7zeGpqPlFj+rcdWrsL3JIl8N2kTHPXX7lavO6arSIc9mUWo55Ljc01TJkIwNpFnxBYxcKfpdsRhva8lr/W1dZiz/dLPpc+otesXOhZQXotKL6oEqFJ/lt/UZ/ea7u/1b/BrAiELAPY9UadtVxLLkXjczlnuficbUe42zbAHxeX2R43Oyb7Sm61dsL8/pVkbNZc6bBOqpGPd9Lr3LERvJ2D6aFV/AmHp8Pu8NqmXuhlPFn+PivzB6WNC/1F7m+OxtrKxx+ZfG2sNz80hiJVZXaQZePuyIDZuHSk/R2/Qy9Oa1gvHa+wzCf4mXYXDn+82/g1LWbrKYFbbfHp+XQM6X/Qx1GpdV4eLVW249z+gyEh0o3saKF1jxxYjdRy5MPT6fccY71Onx/uhemzTUaWNXY5y2KolYZJbT9zqkGQlvHWp1lydqqDOGeUGn+PzG4aUu9tPLUlW6ZmUoBZCi8zzHp/93q5CCGOo0D5XFH2FQPT5TfirCxTXH5+E5vdjPRdJ3DhzxEBQdTvNtGtBQp02P0g0e6mxCCCGwbOMu1DU04r0f12O/woAXgMVw2WfLL97FLR7DWRWfF4/1RuzW3lIsEnsaFrQ5HdXDi+rxqYY6zR9SvBP48fb4vCD3styHueT7EL4vfF0Kaxrq9FjBePWZ6WQP6ZW4HaNb7o9n2A6I5Kd7j8+74om+14sczoHc5ta8Ev90i4vSFdbfyRjYjIYVn0emz12L8X//DD2ufwe/ee5bXDNtnjKc/PGrbJ2ciLvu9BiB1fOL7GHlgDbqTST36c8yspt5qfOOvWbD3qrWmo2cbGhupX2raJMHq3Fzgc0iG+tu48Zeesb+elv21Ebd09lit2dV/CU2htx2hu/76qIbPVYv89Yl7DLFCpOL6DS0/N6xtw4926mNsw3M80va8QuS3ZfsMOGgbhHzlo6latMTK8amskf0iWzoqtpT8ukvV3qKz4l90jfz3+/t/GDY86XF/ky1L+CIrloetG1RYHIaoSr7u3Vb1DWSLWXn8miDevntO32G8ua/xobSdmYeMjUunnBkZWunSowgho1up7Lo908AOjiUC6PM2G3iu6/eWU5ro7NOsdP6ntr4vf64EYg5Qzbwv6Vqg04n9lkU3wWjugIApl08Eo9++hMuOaw7Tn7kS8y4YoyveE8Z2hHTvl0ddd7uezuke2uTQWqtpOia5eWYNmI9doDaXqpO7+FeM7EXnpv5c/j8bovHitOrO6MwNwcT+rbD1dPm4f0FG6Iqn1OGdMQ9MxbjhMERbyb/OGcYxv3t0/BvlT3e/ZMHodyyaezYXm1w+4n9MVo3TN5pUcS5IcJpwzTvK+9dOQarttSgIFerfF7+zUGo2d+AAytaYGiXUjTLy8E5I7vg39+uxvgD20btQG6gqtiK8nPx1uWHYOMuTfHOuGIMht/2AR44PdoRUftWhfjNmG6mnditGLqqIDcHFx/WDT+s3YHSojxMro54q3nhohFoFMK10pUbY/f8amD4+KQhHSEAHNarDa57dT5+N7a7UpaTh3YEkVnx/e/acTj3yW9w58kDcPQDn0fdc9cpA2wbh07ICtXOG8ppwzph3c59uOGYA6Ouzf55m+n3wQe0jgrz99MG4dPFm3B477bo0bYYny3ZhLzcEE4eEm1gb2Sd3Di8aHTXqHBOPk5lZOPsR88aii9+2oJDekTL6Bc5TWtP/rOrx2Lu6u24573F+HlLTbjXdfKQjnji8xUY0a0MOUQobZ6PUIhwbP8KfLp4E47s2w4rNu/B/72zCM9cUA1Ac8ZQs7/B5JVFxuqh5anzhocdegDaTjCXH94DJXqdUK/YUy0Z3lxY8Xkklt6Y1Zh7YCfNqHhYVRme0A3LV95xjK80KlsVhl0lWbH74Lq3LTYpPtlId1hVqamysRumMXarblGYhxcuGoEzn/hamWZeTgin6m6+rMbABoX5ISy8ZaLpnNU9lkqMSYOifZ/n5oRwhuS6zNp+PG14p7Dxec92LUy9pxFS7/XVSw4OH0/oG6ngVdi9pr6VrdBXP27TosD0bq38+ejoSlvGMOJtFAIFuTn4xznDosKM6q5VmCbbUYcyVF1VZurd5oQIp+qNgn+eN9z2vrycEE4bbnYPV9Y8H29cdojtPW5uxuwoyIvuSVrp2b4F7jxlgPKa9fFVNpodSpqFy0y3NsVRZU/GWpFP6NNO2UPzurhEbgS6pe0Hc/pmATqXF6FzeRHunqF5xDF6bbk5Ibx/1aHRcYUIfzs10kD67aER4/i8nBBOr452FWhQXGDOm7EKBXnVET3Dx8lYyKKChzo9EsvYd7Sxt9tcjXsa1r3kzNc8yhUyt06d7NiMOF3s2h2xjuPHsnWQV9wWsgRBAqJUpGHM8XkJGzl23CU+CXJH0ootMS/vyylE0M9oRBfeY9Pu2zMtfrEXIt45Rzv8mFMkctNc/wvFeI6vSRNLYYk2rnZJw0ucIfuP23ZZiuWCvDIzRM7L+cOr2uQ5hIBWyiUC6zBPovYUTDT+VnXKCyuc4kxeZsSa703tfVm3RvJiZuOUzYmq572ZryRhtaTPMpaiDh8rPq/EUmc47WcWaxpOnhViMWcgIkeXXRQO5y1uLwRhsGuH9UNKROs2GfttW3saXnHKt2TuABPr+/PU43MIkqh3EzYBiFfxBSaRGdUCp6i0hXE9cfh97dzja+LEUlisla57j8/LEKB/2zdr2bIuf3dUfBQdt+/CHRVnfJWbn8QS0stJQq/EkNtvK91JtOT2+GJLy1vjL777/WDEZ1TQto1O6bzjsyeoovfSmAwnncBi4LvHx3N8TZvY5vjMv90qfK89IdtwNued7LVce3wxVsDavb5vCRN7jy8zhjrDQ2xqc1HX+1JNrGJ46/El7xnDPe9w2jbh5B5fIgWyIWQaxXEOm8gRCx7qzDCcPKPb4Xeo00t5jGUOx1HxucgVmWtSX/eUKzGU7ljrNqui8OuA2gvJqNhUhuheaDpzfKkZ6kwU4WFC28UtEZyeIVEVfTz7dgaJ38/Nq7/YoGHF5xHVRpBufLZ0k+l3c4XRrwkPZaCypFmU0biBXSF6Z/562/jeW7AhyvjaFKce5V7JgFY2f1DZ24Vl1Q3Vh9jsBO9ErN+m1YRkoc1mrPHQQl/OHuvO517oopus9HAxXrdi52QcgGnD3kQT63yiF+8vTmG8GuV7xTCw7qHvkF7ZSr1PpNceV6KG9uS5MrsdVpIxn+Y3BXlPxHYtNRtdp53jg4Lt+BLIz1siu7RffOgBJrsxFbKHjdtP7B8uCLef2B9//s98nF7dCVeO74mpb0TvTA4AdTbjYvKO3yo6lxXh8nHdlfIZhruyFxp5U9fR3e2Nb88Y0Rmb99TirBFdAAAPnD7YVmkD2s7U5z81C/+9dJRp48pjBlTguAGVtvfJGBtyPvnFCgDAgrXBK74j+rTD78d2x3EDK7Fo/c6EDL1dclh3CAFccEhXT+FvO7Ef1m3fF/ZSc8X4HlE7Wyca2Vg/1h7fHhujdRmV5xxZhj43zgCgfTfxYvS4/3JsH3QqK8JvxnRThpOf1qk8BKV87jzZ/GyyR6P2Nso5CVN8JmcYXujfoRWuHN8TY3u3QYvCPLwy+xecOizakUDQsOJLElOO6u0aRv5eZKPsM0Z0Nv22G6Mvd3CT5Zwu4aoJvZTXOpQ0w5rtey3Ot6WJfIceRvOCXFx3VMRQ+/iBzsprbK+2YaPvrZKyPm1YJ4zpqfaiYqVNiwLceFwfLN+8G58s3hR4DwDQenp/OlLLr17t/fXIvFLWPB83HNvHc/gz9caFwcEHtE664jPcxgGxV65e9KXT8G9RfqRKq+7qf6TBipFUp7Ii/MXhfXjV80GoPZVjhDYt3BVOZLg2ACFs8DsKQkT4w/iIN5trJ7rXk0HAQ53piE3ZSuYcTqIxt6D939/oMieT6aTisb3suuGGJ8WXxHkhr4u6vJazRI02evn2jWdJrDlOqpar+IMVXxpiV2wTOeeUDONXO2JR6MaQUrqu6oyXVDy2330NVXiplGNZaBYrfrduco0vZkkCEgCJbRSlyjzBL6z4mhBeW2J2rctEKL507TAZCyAyqRfsh1T0dBPZ8JJJ5kJAr0l57/ElRngvqaeLUkoGrPjSkFT0+FJJLB+sMQ+UCHOGdCDVQ50x4xCFEX8yhzq9mpOkQzFzs0UMMo2mDiu+NMSu4AYx1GRLCkt0LMOsRt2YDhVSIki1EXWsOHue0f77tW2MB+9Dnamd4/OXdgKnRNJE87Hia0J41Vt2wRJirN0EFEdMPT4e6kx6mglteEF2GJ3QZGLC+6rOVArv7Gg7m2BzBgcaGgUO+PPbSUvPa4G0q8wTu7gluZj9ifp/rvBQZ5Z+5SlZ3GJxfh4L+TbG10DEprRmv7utny5FTDIAQNfWzbFi8x4Uetgf0AuGWVBQ8cXCthp7G9qgSJfPjXt8DjgZWxt0a2O/4eZfj+9re01Fm+ICnF7dSblrt4ydZ4M+FS2V541NTVUM7NjKMS27IZxrJvbC1Ueqbf+CQN6w86BuZb7vT8Z8RlNGVvhPn69tMvvvi0cmJK0/TeiJM0d0NlXqXcub48TBHfDgGc5l2crwrpF3fbfNZrNuIwCjumuOGA5w+Dbd+PupA3FEn3b4lUdjajePM/dNHoQj+rTDCYOjN1P2g+HUIhYMj0tbdzs7tIiFx84eipOHdAx7uGnqBNLjI6KJAO4HkAPgCSHEHZbr5wG4G8Aa/dSDQogn9GvnArhBP3+rEOKZIGQKAi9zCecdXIUbX1d7Ujmkh71XExVEhP87Sf2xq2hRmItd+7TW7+NnD7V1H9a8IBebbQr7EX3aeUrLuhrtd4d19yxnLBCR4w7mrvcHKEs6Iiv8wyTHAIng9+N6RJ0LhQj3njbId1yyuy07d21Wt3RWXrjoIN/pWhncuVS5670dbiYWw6vKMLzKfwPOykEu3p+cMIb/OyTAqcORfdvjyL7tA483UcSt+IgoB8BDAI4AsBrALCKaLoRYYAn6shDi95Z7ywDcBGAYtEb6t/q92+KVKwi8rB5zdvCc4DkP+TjmbWCc78vWHhOTeuyKXlMsk8naVy6e6QyjOsvU1d9+CGKosxrAMiHEciHEfgAvAZjk8d4jAbwvhNiqK7v3AUwMQKZA8GIo66Tcklm8nD68qP34sqg/lC6rzIKmKSoHv9h9W02x3k5WOYunMe22i3w2EYTi6wDgF+n3av2clZOJaB4RTSOiTj7vNUFEU4lIEJFYu3ZtrHK74sVTvMNcfFIXVjhJ6rSSLJk+BpNJtroqM8iExo3fDZdTiZe6IgjieXJDOWfrgi+ZZC1ueQNAlRBiALReXVzzeEKIqUIIEkJQZaU3r/2x4GUTUKcKNtHlS07bT4/PFIfLp5Tun0hql4+njkyo27xs+tpUSJaJRTzP3pjlbvxkglB8awB0kn53RGQRCwBACLFFCFGr/3wCwFCv96YSL0OdTrZLyfQaEutQi1cR023IMNu/7Uxo1TfFnp0dyZrji+e9NmS5iY9MEIpvFoAeRNSViPIBTAYwXQ5ARBXSz+MBLNSPZwCYQESlRFQKYIJ+rknQ4KHL57ThZqL0nuojc2pxOvb4XGTM9iHDdCUTXpvdt9UUy2Q69PjcdpHPJuJe1SmEqCei30NTWDkAnhRC/EhENwOYLYSYDuByIjoeQD2ArQDO0+/dSkS3QFOeAHCzEGJrvDIFRYOXoU6HVmmiWlZGT7JFYW7Y1rDOi7A6TvOSVprptllOO3s3ZdKtpxoU6fm2zNh9W02xKCbPjVrsD2+YP6XrtxwkgczxCSHeFkL0FEIcIIS4TT93o670IIS4TgjRVwgxUAgxVgixSLr3SSFEd/3vqSDkCQovhXn9zn2m378e3TV83Lq4AH0qWuKPR/QMVK4zqjujS3mRyUB+l8PO1Xf/agDatSzA9UcfiKryIkyujmxq6/aIU47qjc5lRThvVFW8YieVG47tgzYtCjxtAJyJZEKj3lo/P3jGYFS0KsRxLhsap4JhVaXoUl6EW0/ol5D4h1dpm+peNk5tP+ulF3f7if3RpbyoSeZfsmGXZQ54acTJH+d/Lx2FQZ0iu1DnhAhv/2F04HL1aNcCn1491nTOqdgffEBrfP3n8QCAX4/pZrrm9ohjerbBZ9eMdQnV9BjUqQSzrh+fajFSSPprvoJcs3uvYwdU4tgBTbPSLsrPjfomg+TfFx8cdxzHDaxkpafDLssc8LsiMJUjCFk6osfYkAk9vkx4BqZpwoovTuR5iHRcLZWtc2CZTjqWRSvZupdiLCRrVWmmwIrPgWwoS9lq55bpZILKyIRnYJomrPgCJB1b2dmg3LORNCyKUaTj98SkB6z4AoS/U6apkE7G33bw98QkClZ8DvjtDaVjC5XnBjKTNCyKUWTCMyQLNkr3Bys+B/bVN/gKn45lL1keJxjGL5nQa2WaJqz4HHh+5s+uYQ7t1SZ8nIpFaIZB6+ju/ja9Ndhe477LPJN+ZMKKyAx4hKTRr7IlAOCMEZ1dQjIAG7A74kUpHNCmWPqV/C/1jxN64bJxPZCf668NU1yQi90O3l6Y9CYTdAYP33mnvLgAS249ync9kK1wLjngZRRQbpWmqoUaS2E3RGVzhswkE3QG9/j8wUrPO5xTDnhZ+CG3StOqhaqLymtbmKYKz/ExiYIVnwP+V3UmRo5EkEaiMjGQCQ0a4tqJSRBctBzwOwzILVSmqZC8bXISB39NTKJgxeeAh31oTaTVSKcuLNvxZSaZ8FrT0S6WSQ9Y8Tngu8eXRt+pIWsG1I+Mgozo8aXR98SkF6z4HFi2cbev8OnUQg2v6kz/+pFRkAmOCdLpe2LSC1Z8Dmzevd9TuG5tmgMAigvTxywyPNTJfb6MpKx5PoBI2WQYJkL61NRNiAdOH4webYvRslkeAOC1Sw7Gzr31aFmYl2LJvMNt6cymVbM8fDFlHMp1BZiOcI+PSRSs+DySl0Ooa9B6R4W5IRxY0TJ8raQoHyVF6VnB8FBn5tKhpFmqRYiLdDIPYtILHuqMgUxoifLiFqapk1YOIZi0ghWfR+SeUWZ8j4Y5Q4rFYBgbuMfHJApWfB7JNP0QUd6Z9mRMpsA9PiZRsOLLUrhKYRgmW2HF55FM9XCSoY/FMAxjCys+Gzbs3Gd7LROUBY8iMQyTrQSi+IhoIhEtJqJlRDRFcf0qIlpARPOI6EMi6iJdayCiOfrf9CDkCYIRt3+YahESygmDOgAAhnQuTbEkDMMwySVuOz4iygHwEIAjAKwGMIuIpgshFkjBvgcwTAhRQ0SXALgLwGn6tb1CiEHxypFoMqCTZ+IP43ugumsZDu3ZJtWiMIyJz64ei6013rwmMUwsBNHjqwawTAixXAixH8BLACbJAYQQHwshavSfMwF0DCBdJg6K8nNx+IHtkJvDo91M06JzeREGdSpJtRhMBhNErdcBwC/S79X6OTsuBPCO9LuQiGYT0UwiOsFLgkQ0lYgEEYm1a9f6lzgGMmFej2EYhkny4hYiOgvAMAB3S6e7CCGGATgDwH1EdIBbPEKIqUIIEkJQZWVlgqRlGIZhMpEgFN8aAJ2k3x31cyaIaDyA6wEcL4SoNc4LIdbo/5cD+ATA4ABkYhiGYRglQSi+WQB6EFFXIsoHMBmAaXUmEQ0G8Bg0pbdROl9KRAX6cWsAowDIi2IYhmEYJlDiXtUphKgnot8DmAEgB8CTQogfiehmALOFENOhDW0WA/i37oZolRDieAAHAniMiBqhKeE7LKtBGYZhGCZQAtmWSAjxNoC3LedulI7H29z3JYD+QcjAMAzDMF7gteweGde7LXJ1d/GtitJnw1mGYRjGDG9E65HbTuyH2rpGfL50E4Z1YW8nDMMw6QorPg8U5oVQ0UrbzbqqdfMUS8MwDMPEAw91eoB4Ex+GYZiMgRWfB3gnaIZhmMyBFZ8HQryHD8MwTMbAis8DrPcYhmEyB1Z8HgjxWCfDMEzGwIrPAzzUyTAMkzmw4vNA2xYFqRaBYRiGCQhWfB548dcHpVoEhmEYJiBY8XmgtHl+qkVgGIZhAoIVH8MwDJNVsOJjGIZhsgpWfAqEEKkWgWEYhkkQrPgUsN5jGIbJXFjxKWC9xzAMk7mw4lPAQ50MwzCZCys+Baz2GIZhMhdWfAoaucfHMAyTsbDiUyDrvUGdSlInCMMwDBM4rPhcYP/UDMMwmQUrPgU80skwDJO5sOJTIHh5C8MwTMbCik8B9/gYhmEyl0AUHxFNJKLFRLSMiKYorhcQ0cv69a+JqEq6dp1+fjERHRmEPPEi6z2e4mMYhsks4lZ8RJQD4CEARwHoA+B0IupjCXYhgG1CiO4A7gVwp35vHwCTAfQFMBHAw3p8KYUN2BmGYTKXIHp81QCWCSGWCyH2A3gJwCRLmEkAntGPpwE4nIhIP/+SEKJWCLECwDI9vpRSfduH4WPiZZ0MwzAZRRCKrwOAX6Tfq/VzyjBCiHoAOwCUe7w3CiKaSkSCiMTatWvjED2a2voG7K1rCDROhmEYpumQlotbhBBThRAkhKDKyspA425o5GFOhmGYTCYIxbcGQCfpd0f9nDIMEeUCaAVgi8d7k0o9Kz6GYZiMJgjFNwtADyLqSkT50BarTLeEmQ7gXP34FAAfCW0FyXQAk/VVn10B9ADwTQAyxUyjRfHxDB/DMExmkRtvBEKIeiL6PYAZAHIAPCmE+JGIbgYwWwgxHcA/ATxHRMsAbIWmHKGHewXAAgD1AC4VQqR0go2HOhmGYTKbuBUfAAgh3gbwtuXcjdLxPgC/srn3NgC3BSFHELDiYxiGyWzScnFLImlgGz6GYZiMhhWfBWuPj834GIZhMgtWfBb21zeafhMvb2EYhskoWPFZlSv30QAAEt1JREFU2Lpnf6pFYBiGYRIIKz4L7KKMYRgms2HFxzAMw2QVrPiisKzq5A4gwzBMRsGKzwJbMzAMw2Q2rPhc4A4fwzBMZsGKzwJ3+BiGYTIbVnwWeKiTYRgms2HFZ6G+sdE9EMMwDJO2sOKz8OI3v5h+Hzsw2I1uGYZhmNTCis/C0g27wselRXk4o7pzCqVhGIZhgoYVnwV5jm9AxxLkhHhdJ8MwTCbBis+C4HWdDMMwGQ0rPgu8Dy3DMExmw4rPgmB7BoZhmIyGFZ8FVnsMwzCZDSs+K0J5yDAMw2QIuakWoKlQ39CIRet3oZGHOhmGYTIaVnw6t729EE99sTLVYjAMwzAJhoc6dd6evw4A0Dw/J8WSMAzDMImEFZ+OMcIZIjZYZxiGyWRY8VngGT6GYZjMJi7FR0RlRPQ+ES3V/5cqwgwioq+I6EcimkdEp0nXniaiFUQ0R/8bFI888cAKj2EYJjuIt8c3BcCHQogeAD7Uf1upAXCOEKIvgIkA7iOiEun61UKIQfrfnDjliRs2YGcYhsls4lV8kwA8ox8/A+AEawAhxBIhxFL9eC2AjQDaxJluwmC1xzAMk9nEq/jaCSHW6cfrAbRzCkxE1QDyAfwknb5NHwK9l4gKvCRKRFOJSBCRWLt2bUyCWzE6etzhYxiGyWxcFR8RfUBEPyj+JsnhhDZGaKs2iKgCwHMAzhdCGNucXwegN4DhAMoAXOtFaCHEVCEECSGosjKojWI10ffWNcjpBBQ3wzAM01RwNWAXQoy3u0ZEG4ioQgixTldsG23CtQTwFoDrhRAzpbiN3mItET0F4E++pA+Qzbv3pypphmEYJonEO9Q5HcC5+vG5AF63BiCifAD/AfCsEGKa5VqF/p+gzQ/+EKc8DMMwDONIvIrvDgBHENFSAOP13yCiYUT0hB7mVABjAJynMFt4gYjmA5gPoDWAW+OUh2EYhmEcictXpxBiC4DDFednA7hIP34ewPM294+LJ/1EQ+zFhWEYJuNgzy0O8OIWhmGYzIMVH8MwDJNVsOJjGIZhsgpWfAzDMExWwYqPYRiGySpY8TEMwzBZBSs+hmEYJqtgxccwDMNkFaz4GIZhmKyCFR/DMAyTVbDiYxiGYbIKVnwMwzBMVsGKj2EYhskqWPExDMMwWQUrPoZhGCarYMXnAO9KxDAMk3mw4mMYhmGyClZ8DvAG7AzDMJkHKz6GYRgmq2DFxzAMw2QVrPgc4MUtDMMwmQcrPoZhGCarYMXHMAzDZBWs+BiGYZisghWfAxce0jXVIjAMwzABE5fiI6IyInqfiJbq/0ttwjUQ0Rz9b7p0visRfU1Ey4joZSLKj0eeeGien2P6/dV14zC2d9sUScMwDMMkinh7fFMAfCiE6AHgQ/23ir1CiEH63/HS+TsB3CuE6A5gG4AL45QnZqwLOHPYep1hGCYjiVfxTQLwjH78DIATvN5IRARgHIBpsdwfNFGmC6z3GIZhMpJ4FV87IcQ6/Xg9gHY24QqJaDYRzSQiQ7mVA9guhKjXf68G0CFOeWJGWPp8Ie7xMQzDZCS5bgGI6AMA7RWXrpd/CCEEEdmZfHcRQqwhom4APiKi+QB2+JY2ItNUADcBQEVFRazRmLD2+FjtMQzDZCauik8IMd7uGhFtIKIKIcQ6IqoAsNEmjjX6/+VE9AmAwQBeBVBCRLl6r68jgDVehBZCTAUwFQCGDRsWiH+VqJFO7vExDMNkJPEOdU4HcK5+fC6A160BiKiUiAr049YARgFYIIQQAD4GcIrT/clCCOtQZ4oEYRiGYRJKvIrvDgBHENFSAOP13yCiYUT0hB7mQACziWguNEV3hxBigX7tWgBXEdEyaHN+/4xTnphYva0GdQ1mxUc82MkwDJORuA51OiGE2ALgcMX52QAu0o+/BNDf5v7lAKrjkSEITnz4y6hzBXls288wDJOJcO0OYNOu2qhz+TmcNQzDMJkI1+428NoWhmGYzIQVnw28qpNhGCYzYcXHMAzDZBWs+BiGYZisghUfwzAMk1Ww4mMYhmGyClZ8DMMwTFbBio9hGIbJKljxAehT0TLVIjAMwzBJghUfgMMPbAuAHVMzDMNkA6z4GIZhmKyCFZ9EIBv7MQzDME0aVnyI7LZu3YWdYRiGyTxY8QHskZphGCaLYMXHMAzDZBWs+BiGYZisghUfgE+XbEq1CAzDMEySYMUHYO4v21MtAsMwDJMkWPExDMMwWQUrPoZhGCarYMXHMAzDZBWs+BiGYZisghUfwzAMk1Ww4mMYhmGyClZ8DMMwTFYRl+IjojIiep+Ilur/SxVhxhLRHOlvHxGdoF97mohWSNcGxSMPwzAMw7gRb49vCoAPhRA9AHyo/zYhhPhYCDFICDEIwDgANQDek4JcbVwXQsyJU56Y+OMRPcPHd5zUH/dPZv3LMAyTqeTGef8kAIfpx88A+ATAtQ7hTwHwjhCiJs50A6W8uCB8PLm6cwolYRiGYRJNvD2+dkKIdfrxegDtXMJPBvCi5dxtRDSPiO4logLVTVaIaCoRCSISa9eu9SlyNIK3oGUYhskaXBUfEX1ARD8o/ibJ4YQQAg6bmBNRBYD+AGZIp68D0BvAcABlcO4tymlNFUKQEIIqKyu93MIwDMMwADwMdQohxttdI6INRFQhhFinK7aNDlGdCuA/Qog6KW6jt1hLRE8B+JNHuRmGYRgmJuId6pwO4Fz9+FwArzuEPR2WYU5dWYKICMAJAH6IU56YEDzSyTAMkzXEq/juAHAEES0FMF7/DSIaRkRPGIGIqApAJwCfWu5/gYjmA5gPoDWAW+OUh2EYhmEciWtVpxBiC4DDFednA7hI+r0SQAdFuHHxpM8wDMMwfmHPLQByQpRqERiGYZgkEa8dX0YwaVAl3l+wAb8e3S3VojAMwzAJhhUfgKL8XDx53vBUi8EwDMMkAR7qZBiGYbIKVnwMwzBMVsGKj2EYhskqWPExDMMwWQUrPoZhGCarYMXHMAzDZBWs+BiGYZisghUfwzAMk1Ww4mMYhmGyClZ8DMMwTFZBIs03oyOiTQB+DiCqSgBrA4gnVaSz/Cx76khn+dNZdiC95U8H2TcLISaqLqS94gsKIhJCiLTdpiGd5WfZU0c6y5/OsgPpLX86yw7wUCfDMAyTZbDiYxiGYbIKVnwR/ppqAeIkneVn2VNHOsufzrID6S1/OsvOc3wMwzBMdsE9PoZhGCarYMXHMAzDZBWs+BiGYZisghUfwzAMk1Ww4mMYhmGyClZ8DMMwTFbBig8AEU0kosVEtIyIpqRaHgAgok5E9DERLSCiH4noD/r5MiJ6n4iW6v9L9fNERA/ozzCPiIZIcZ2rh19KROcm8RlyiOh7InpT/92ViL7WZXyZiPL18wX672X69Sopjuv084uJ6Mgkyl5CRNOIaBERLSSikemS90R0pV5mfiCiF4mosCnnPRE9SUQbiegH6VxgeU1EQ4lovn7PA0QUmKstG9nv1svNPCL6DxGVSNeUeWpXB9m9t0TKL137IxEJImqt/25SeR8XQois/gOQA+AnAN0A5AOYC6BPE5CrAsAQ/bgFgCUA+gC4C8AU/fwUAHfqx0cDeAcAATgIwNf6+TIAy/X/pfpxaZKe4SoA/wLwpv77FQCT9eNHAVyiH/8OwKP68WQAL+vHffT3UQCgq/6ecpIk+zMALtKP8wGUpEPeA+gAYAWAZlKen9eU8x7AGABDAPwgnQssrwF8o4cl/d6jEiz7BAC5+vGdkuzKPIVDHWT33hIpv36+E4AZ0DYAaN0U8z6u5061AKn+AzASwAzp93UArku1XAo5XwdwBIDFACr0cxUAFuvHjwE4XQq/WL9+OoDHpPOmcAmUtyOADwGMA/CmXvA3SxVCON/1D2ykfpyrhyPru5DDJVj2VtCUB1nON/m8h6b4ftEroVw9749s6nkPoApm5RFIXuvXFknnTeESIbvl2okAXtCPlXkKmzrI6ZtJtPwApgEYCGAlIoqvyeV9rH881BmpKAxW6+eaDPrw02AAXwNoJ4RYp19aD6Cdfmz3HKl6vvsAXAOgUf9dDmC7EKJeIUdYRv36Dj18qmTvCmATgKdIG6p9goiaIw3yXgixBsA9AFYBWActL79F+uS9QVB53UE/tp5PFhdA6+kA/mV3+mYSBhFNArBGCDHXcind8t4WVnxNHCIqBvAqgCuEEDvla0JrRjU5n3NEdCyAjUKIb1MtS4zkQhv+eUQIMRjAHmjDbWGacN6XApgETXlXAmgOQLknWbrQVPPaDSK6HkA9gBdSLYtXiKgIwJ8B3JhqWRIJKz5gDbTxbIOO+rmUQ0R50JTeC0KI1/TTG4ioQr9eAWCjft7uOVLxfKMAHE9EKwG8BG24834AJUSUq5AjLKN+vRWALSmSHdBapquFEF/rv6dBU4TpkPfjAawQQmwSQtQBeA3a+0iXvDcIKq/X6MfW8wmFiM4DcCyAM3XFDRcZVee3wP69JYoDoDWa5urfb0cA3xFRewc5m1TeeyLVY62p/oPWul8O7WUbE8t9m4BcBOBZAPdZzt8N86T/XfrxMTBPPH+jny+DNl9Vqv+tAFCWxOc4DJHFLf+GeaL+d/rxpTAvsHhFP+4L82KA5Uje4pbPAfTSj6fq+d7k8x7ACAA/AijS5XkGwGVNPe8RPccXWF4jeoHF0QmWfSKABQDaWMIp8xQOdZDde0uk/JZrKxGZ42tyeR/zM6dagKbwB2210hJoK6uuT7U8ukyHQBvemQdgjv53NLRx/w8BLAXwgVTACMBD+jPMBzBMiusCAMv0v/OT/ByHIaL4uukfwjL9gy7Qzxfqv5fp17tJ91+vP9NiJHFFGIBBAGbr+f9f/YNOi7yHtmXMIgA/AHhOr2ibbN4DeBHafGQdtN72hUHmNYBhel78BOBBWBYtJUD2ZdDmvIzv9lG3PIVNHWT33hIpv+X6SkQUX5PK+3j+eFsihmEYJqvgOT6GYRgmq2DFxzAMw2QVrPgYhmGYrIIVH8MwDJNVsOJjGIZhsgpWfEzWQUQr9Z0LQpZz/QJMo4qINgcVn490n9J3Zng5wDjnEFGzgOI6j4imBREXw8RKrnsQhslIigGcDc3Au8lCRDlCiAaPYdsBOBlAiRCi0S28V4QQg4KKi2GaAtzjY7KVqQBuUu1vZu39yb/141uJ6CsiWkVEZxDRFUT0jb7n2BhLXH/T9y6bT0SjpfNHE9EXRPStHtdB+vnD9PBPEdEcAEcp5DtHj8/Y760tEbUA8DE0jy3fEdGVivt6EdE7RDSLiOYS0fnSNUFEf9V7d4uJ6GTLtWIiChHRw6TtNTeXiL5wkkk/n09Ej+n7tH0FoNoi07V63n1HRG/orrFARJP0+ObovfPD1K+RYWIg1Rb0/Md/yf6D5o2iHzRPGH+Qz1mPba7drR8Ph+bA+lL996kA/qcfV0HzvHOO/vswaJ4xCqD5Q/wKQEv9Wl8Aq6RwDbDZAkiXey0iW/bcgsgeelUANtvclwttl4be+u8W0LyHGL8FgBv1417Q/ES2la4VQ9shZCGAkH6+1INMlwF4D0AeNKU8G8A0/dpZAB6X4rsEkS185iKyXVKOkVf8x39B/PFQJ5PN3ADgYyL6p8/7jPmz76BV5sbvbwF0l8LtB/A8AAghPiGivdCUyiHQlN9n0obUufpQJQAsFUJ8ZZP2WABvi8iWPY9BUxJu9ARwIICXpDQL9HOL9N//1GVdTETfQfOxOF2KYzk0BfZPIvoI2l5/bjKNBfCM0Bxm1xHR8/rzA8Dx0FxafafLlAttWyQA+AjAvUT0KoB3hBBRO4QzTKyw4mOyFr2CfxvaTvEy9TBPAxRaru/T72/QK+x9+vkGePumCMC7Qohzoi4QHQhgt4c4/ELQeoMxz9cJIXYQUV9ovdLxAO4koiFxynSrEOJJRVpXElF/aDt7/JuI/i6E+EccaTFMGJ7jY7KdqdB2KGghnVsGbRgTRHQ4Ipug+iUfwBl6PKMBNIPWu3oPwERdiUC/PtxjnB8DONqYCwPwawDve7hvMYAaIjpbSrM3EbWUwpyvn+8BbVhzphwBEbUBUCSEmAFtx4Qd0JwoO8n0EYCziShXXxl6hhTldAC/I20PQRBRAREN1I97CSHmCyHuh9Zr9po/DOMK9/iYrEYIsZqIngPwR+n0XwA8Q0SXQau4V8UY/RYAg4joGmi9m9OFEPsBLCWis6ANGTaDpiC/ADDLg7w/ENEUAO8TkYA2/PhbD/fVE9FxAO4joquhzZttgDYvaZBLRN9DG779rRBioyWaTgD+Qdr+cLnQtpmZKYRodJDpcQADoM0NbtafsZ0u03NE1BrAp3rPOQTgYWjDpHfoCrgewHZoux4wTCDw7gwMw0BXWC2EEIkYZmWYJgUPdTIMwzBZBff4GIZhmKyCe3wMwzBMVsGKj2EYhskqWPExDMMwWQUrPoZhGCarYMXHMAzDZBX/H0eBbRQYRNILAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# DO NOT CHANGE THIS CELL\n",
        "\n",
        "# Experiment configs.\n",
        "train_episodes = 15000\n",
        "discount_factor = .99\n",
        "\n",
        "# Create environment.\n",
        "env = catch.Catch(seed=42)\n",
        "\n",
        "# Build and initialize network.\n",
        "rng = jax.random.PRNGKey(44)\n",
        "rng, init_rng = jax.random.split(rng)\n",
        "sample_input = env.observation_spec().generate_value()\n",
        "parameters = init_net(init_rng, sample_input)\n",
        "\n",
        "# Initialize optimizer state.\n",
        "opt_state = opt_init(parameters)\n",
        "\n",
        "# Apply updates\n",
        "def apply_updates(params, updates):\n",
        "  return jax.tree_multimap(lambda p, u: p + u, params, updates)\n",
        "\n",
        "# Jit.\n",
        "opt_update = jax.jit(opt_update)\n",
        "apply_updates = jax.jit(apply_updates)\n",
        "\n",
        "print(f\"Training agent for {train_episodes} episodes...\")\n",
        "all_episode_returns = []\n",
        "\n",
        "for _ in range(train_episodes):\n",
        "  episode_return = 0.\n",
        "  timestep = env.reset()\n",
        "  obs_tm1 = timestep.observation\n",
        "\n",
        "  # Sample initial action.\n",
        "  rng, policy_rng = jax.random.split(rng)\n",
        "  a_tm1 = epsilon_greedy_policy(parameters, policy_rng, obs_tm1)\n",
        "\n",
        "  while not timestep.last():\n",
        "    # Step environment.\n",
        "    new_timestep = env.step(int(a_tm1))\n",
        "\n",
        "    # Sample action from agent policy.\n",
        "    rng, policy_rng = jax.random.split(rng)\n",
        "    a_t = epsilon_greedy_policy(parameters, policy_rng, new_timestep.observation)\n",
        "\n",
        "    # Update params.\n",
        "    r_t = new_timestep.reward\n",
        "    discount_t = discount_factor * new_timestep.discount\n",
        "\n",
        "    dJ_dtheta = compute_gradient(\n",
        "        parameters, obs_tm1, a_tm1, r_t, discount_t,\n",
        "        new_timestep.observation)\n",
        "    updates, opt_state = opt_update(dJ_dtheta, opt_state)\n",
        "    parameters = apply_updates(parameters, updates)\n",
        "\n",
        "    # Within episode book-keeping.\n",
        "    episode_return += new_timestep.reward\n",
        "    timestep = new_timestep\n",
        "    obs_tm1 = new_timestep.observation\n",
        "    a_tm1 = a_t\n",
        "\n",
        "  # Experiment results tracking.\n",
        "  all_episode_returns.append(episode_return)\n",
        "\n",
        "# Plot learning curve.\n",
        "plot_learning_curve(all_episode_returns)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g80tBzfsBgya"
      },
      "source": [
        "### Q8 [6 marks]\n",
        "\n",
        "[2pts]: What is the best asymptotic average return that can be achieved by the actor-critic agent described in `Part 2A)`? Can this be futher improved, and if so: how? Explain your answer.\n",
        "\n",
        "> *Best asymptotic average return would be 1 (we can see that it keeps increasing with iterations) because when t tends to infinity, agent acts greedy which means probability of occurence of non optimal solution might go till 0. Increasing iterations and changing hyperparameters of algorithm can improve our current average return.*\n",
        "\n",
        "[2pts] What is the best asymptotic average return that can be achieved by the second agent described in `Part 2B)`? Can this be futher improved, and if so: how? Explain your answer.\n",
        "\n",
        "> *Here, best asymtotic average return would be (1- 0.1*(2/3)) = 0.933, this is because 10% of the times when it takes random action, it can take non-optimal action two-third times.*\n",
        "\n",
        "[2pts] What quantity do the preferences `p` estimate in the second agent described in Part B?\n",
        "\n",
        "> *The preferences 'p' in second agent estimate a sort of ranking for possible actions of agent, where the agent takes highest ranked action in greedy case.*"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "RL_part_3.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}