week01_intro/seminar-es-task.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evolution Strategies\n",
    "\n",
    "![img](https://t4.ftcdn.net/jpg/00/17/46/81/240_F_17468143_wY3hsHyfNYoMdG9BlC56HI4JA7pNu63h.jpg)\n",
    "\n",
    "[The paper about the algorithm](https://arxiv.org/abs/1703.03864)\n",
    "\n",
    "Plan:\n",
    "* Study how to use OpenAI gym\n",
    "* Basic prototype of evolutionary strategies that works in one thread on CartPole\n",
    "* Modify the code to make them work in parallel\n",
    "* Use the algorithm to solve the LunarLander\n",
    "* Analyse the influence of hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os\n",
    "if 'google.colab' in sys.modules and not os.path.exists('.setup_complete'):\n",
    "    !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/master/setup_colab.sh -O- | bash\n",
    "\n",
    "    !touch .setup_complete\n",
    "\n",
    "# This code creates a virtual display to draw game images on.\n",
    "# It will have no effect if your machine has a monitor.\n",
    "if type(os.environ.get(\"DISPLAY\")) is not str or len(os.environ.get(\"DISPLAY\")) == 0:\n",
    "    !bash ../xvfb start\n",
    "    os.environ['DISPLAY'] = ':1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import time\n",
    "from copy import deepcopy\n",
    "from joblib import Parallel, delayed\n",
    "from IPython.display import clear_output\n",
    "from IPython import display\n",
    " \n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OpenAI gym environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observation space: Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32)\n",
      "Action space: Discrete(2)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAATPklEQVR4nO3dbaxd5Zne8f/lFwyJmWLwgRjbDCbjqIEoY6IjD2qqiiaZgTKjmnxI60il/oDkfCBKoo7U4hmpk3ywMm2GpFLURCIDGqthQq0SBitK22FoUho1xZgMEGzjYMCFg21swnsmONjn7oezHDb2sc/2een2c87/J23tte/1rL3vB+GLxeO1105VIUlqx7xBNyBJOjMGtyQ1xuCWpMYY3JLUGINbkhpjcEtSY2YsuJNcn2RPkr1Jbp2pz5GkuSYzcR13kvnAz4DfBUaAh4FPV9Wuaf8wSZpjZuqMey2wt6qeqapfAXcD62bosyRpTlkwQ++7HHi+5/UI8DunGrx06dK6/PLLZ6gVSWrPvn37eOmllzLevpkK7vE+7F1rMkk2AhsBLrvsMnbs2DFDrUhSe4aHh0+5b6aWSkaAlT2vVwD7ewdU1e1VNVxVw0NDQzPUhiTNPjMV3A8Dq5OsSnIOsB7YNkOfJUlzyowslVTV0SSfBf47MB+4s6p2zsRnSdJcM1Nr3FTV94Hvz9T7S9Jc5TcnJakxBrckNcbglqTGGNyS1BiDW5IaY3BLUmMMbklqjMEtSY0xuCWpMQa3JDXG4JakxhjcktQYg1uSGmNwS1JjDG5JaozBLUmNMbglqTEGtyQ1Zko/XZZkH/AGcAw4WlXDSS4E/jNwObAP+GdV9crU2pQkHTcdZ9z/uKrWVNVw9/pW4IGqWg080L2WJE2TmVgqWQds6ba3ADfOwGdI0pw11eAu4K+TPJJkY1e7pKoOAHTPF0/xMyRJPaa0xg18tKr2J7kYuD/Jk/0e2AX9RoDLLrtsim1I0twxpTPuqtrfPR8C7gXWAi8mWQbQPR86xbG3V9VwVQ0PDQ1NpQ1JmlMmHdxJ3pvk/OPbwO8BTwDbgA3dsA3AfVNtUpL0jqkslVwC3Jvk+Pv8ZVX9tyQPA1uT3Aw8B3xq6m1Kko6bdHBX1TPAb49T/znw8ak0JUk6Nb85KUmNMbglqTEGtyQ1xuCWpMYY3JLUGINbkhpjcEtSYwxuSWqMwS1JjTG4JakxBrckNcbglqTGGNyS1BiDW5IaY3BLUmMMbklqjMEtSY0xuCWpMQa3JDVmwuBOcmeSQ0me6KldmOT+JE91z0t69m1KsjfJniTXzVTjkjRX9XPG/RfA9SfUbgUeqKrVwAPda5JcCawHruqO+UaS+dPWrSRp4uCuqgeBl08orwO2dNtbgBt76ndX1ZGqehbYC6ydpl4lSUx+jfuSqjoA0D1f3NWXA8/3jBvpaidJsjHJjiQ7Dh8+PMk2JGnume6/nMw4tRpvYFXdXlXDVTU8NDQ0zW1I0uw12eB+MckygO75UFcfAVb2jFsB7J98e5KkE002uLcBG7rtDcB9PfX1SRYlWQWsBrZPrUVJUq8FEw1I8h3gWmBpkhHgT4A/BbYmuRl4DvgUQFXtTLIV2AUcBW6pqmMz1LskzUkTBndVffoUuz5+ivGbgc1TaUqSdGp+c1KSGmNwS1JjDG5JaozBLUmNMbglqTEGtyQ1xuCWpMYY3JLUGINbkhpjcEtSYwxuSWqMwS1JjTG4JakxBrckNcbglqTGGNyS1BiDW5IaY3BLUmMmDO4kdyY5lOSJntoXk7yQ5NHucUPPvk1J9ibZk+S6mWpckuaqfs64/wK4fpz616pqTff4PkCSK4H1wFXdMd9IMn+6mpUk9RHcVfUg8HKf77cOuLuqjlTVs8BeYO0U+pMknWAqa9yfTfJ4t5SypKstB57vGTPS1U6SZGOSHUl2HD58eAptSNLcMtng/ibwfmANcAC4ratnnLE13htU1e1VNVxVw0NDQ5NsQ5LmnkkFd1W9WFXHqmoU+BbvLIeMACt7hq4A9k+tRUlSr0kFd5JlPS8/CRy/4mQbsD7JoiSrgNXA9qm1KEnqtWCiAUm+A1wLLE0yAvwJcG2SNYwtg+wDPgNQVTuTbAV2AUeBW6rq2My0Lklz04TBXVWfHqd8x2nGbwY2T6UpSdKp+c1JSWqMwS1JjTG4JakxBrckNcbglqTGGNxS561XD3Lk9ZcG3YY0oQkvB5Rmo9Gjb7Pvf27h2JFf/Lr2y1cOcMFvfpiV/+Cfk3ne1FJnL4Nbc1LmzWPBovfwytMPv6v+85/9mGUf+X0WvufvDagzaWIulWhOyrz5nHfRyokHSmchg1uSGmNwS1JjDG5JaozBLUmNMbglqTEGtyQ1xuCWpMYY3JLUGINbkhozYXAnWZnkB0l2J9mZ5PNd/cIk9yd5qnte0nPMpiR7k+xJct1MTkCS5pp+zriPAn9YVR8ErgFuSXIlcCvwQFWtBh7oXtPtWw9cBVwPfCOJd+yRpGkyYXBX1YGq+km3/QawG1gOrAO2dMO2ADd22+uAu6vqSFU9C+wF1k5349JUvffiy1lw3vnvqo0eO8rrI7sG1JHUnzNa405yOXA18BBwSVUdgLFwBy7uhi0Hnu85bKSrnfheG5PsSLLj8OHDZ965NEXnLVnOgkXvfXexRnnz4NODaUjqU9/BnWQxcA/whap6/XRDx6nVSYWq26tquKqGh4aG+m1Dkua8voI7yULGQvuuqvpuV34xybJu/zLgUFcfAXrvl7kC2D897UqS+rmqJMAdwO6q+mrPrm3Ahm57A3BfT319kkVJVgGrge3T17IkzW39/ALOR4GbgJ8mebSr/RHwp8DWJDcDzwGfAqiqnUm2ArsYuyLllqo6Nu2dS9IcNWFwV9WPGH/dGuDjpzhmM7B5Cn1Jkk7Bb05KUmMMbklqjMEtSY0xuCWpMQa3JDXG4JakxhjcmrsCmXfyjSurRqk66S4N0lnD4NYcFi758O+eVH3lmUc48ro3PtPZy+DWnJWE+ee856T66NEjUKMD6Ejqj8EtSY0xuCWpMQa3JDXG4JakxhjcktQYg1uSGmNwS1JjDG5JaozBLUmN6efHglcm+UGS3Ul2Jvl8V/9ikheSPNo9bug5ZlOSvUn2JLluJicgSXNNPz8WfBT4w6r6SZLzgUeS3N/t+1pV/Vnv4CRXAuuBq4BLgb9J8gF/MFiSpseEZ9xVdaCqftJtvwHsBpaf5pB1wN1VdaSqngX2Amuno1lpui047/yT71dS8NarBwfTkNSHM1rjTnI5cDXwUFf6bJLHk9yZZElXWw4833PYCKcPemlgFl9yBecuWXZCtXhpz/8eSD9SP/oO7iSLgXuAL1TV68A3gfcDa4ADwG3Hh45z+Ek3N06yMcmOJDsOH/YWmpLUr76CO8lCxkL7rqr6LkBVvVhVx6pqFPgW7yyHjAArew5fAew/8T2r6vaqGq6q4aGhoanMQZLmlH6uKglwB7C7qr7aU+/9/8tPAk9029uA9UkWJVkFrAa2T1/LkjS39XNVyUeBm4CfJnm0q/0R8OkkaxhbBtkHfAagqnYm2QrsYuyKlFu8okSSps+EwV1VP2L8devvn+aYzcDmKfQlSToFvzkpSY0xuCWpMQa3JDXG4JakxhjcktQYg1uSGmNwS+M4duTvOPartwbdhjQug1tz3kUfuOak2psHn+KXL78wgG6kiRncmvPOW+LNK9UWg1uSGmNwS1JjDG5JaozBLUmN6ee2rlJzXnjhBT73uc8xOjo64dgVFyxkwzVLyAn3wNy0aRMjr7494fHz5s3j61//Opdeeulk25XOiMGtWenNN9/kvvvu49ixiW8F/+ErLuFfXnMjvzp27q9rC+e9xYMPPsjjz7w44fHz58/ny1/+8pT6lc6EwS0Bz/3dB9n1+lrGbj1ffPD8hynuG3Rb0rhc49acd2T0PP7vL/4+x+ocjtVCjtU57Hr9d3jt7aWDbk0al8GtOe8XR3+D196+6F21URYwWv7x0Nmpnx8LPjfJ9iSPJdmZ5Etd/cIk9yd5qnte0nPMpiR7k+xJct1MTkCaqt9Y+DIXLTr4rtrCvMWCeRP/xaQ0CP2cUhwBPlZVvw2sAa5Pcg1wK/BAVa0GHuhek+RKYD1wFXA98I0k82eieWk6LMjb/Nbix1i84BWOvnWQl156liVH/opFdWjQrUnj6ufHggt4s3u5sHsUsA64tqtvAX4I/JuufndVHQGeTbIXWAv8eDobl6bLyOHX+fP/chfFXex57uc8+dxLhGK0atCtSePq66qS7oz5EeC3gP9YVQ8luaSqDgBU1YEkF3fDlwP/p+fwka52SgcPHuQrX/nKGTcvncrhw4f7uoYb4OU3fsm9/2v3u2pnEtmjo6PccccdLF3qX2Zq+hw8ePCU+/oK7qo6BqxJcgFwb5IPnWZ4xqmd9OcgyUZgI8Dy5cu56aab+mlF6svTTz/NbbfdRv1/OGueN28e69at44orrpjxz9Lc8e1vf/uU+87oOu6qejXJDxlbu34xybLubHsZcHxBcARY2XPYCmD/OO91O3A7wPDwcL3vfe87k1ak03rttdfIiV+FnEFLly7Ff4c1nRYuXHjKff1cVTLUnWmT5DzgE8CTwDZgQzdsA/z62wrbgPVJFiVZBawGtk+6e0nSu/Rzxr0M2NKtc88DtlbV95L8GNia5GbgOeBTAFW1M8lWYBdwFLilW2qRJE2Dfq4qeRy4epz6z4GPn+KYzcDmKXcnSTqJXw2TpMYY3JLUGO8OqFlp8eLFrFu3ru9ruadi3rx5LF68eMY/RzrO4NastHz5cu65555BtyHNCJdKJKkxBrckNcbglqTGGNyS1BiDW5IaY3BLUmMMbklqjMEtSY0xuCWpMQa3JDXG4JakxhjcktQYg1uSGmNwS1Jj+vmx4HOTbE/yWJKdSb7U1b+Y5IUkj3aPG3qO2ZRkb5I9Sa6byQlI0lzTz/24jwAfq6o3kywEfpTkv3b7vlZVf9Y7OMmVwHrgKuBS4G+SfMAfDJak6THhGXeNebN7ubB71GkOWQfcXVVHqupZYC+wdsqdSpKAPte4k8xP8ihwCLi/qh7qdn02yeNJ7kyypKstB57vOXykq0mSpkFfwV1Vx6pqDbACWJvkQ8A3gfcDa4ADwG3d8Iz3FicWkmxMsiPJjsOHD0+qeUmai87oqpKqehX4IXB9Vb3YBfoo8C3eWQ4ZAVb2HLYC2D/Oe91eVcNVNTw0NDSp5iVpLurnqpKhJBd02+cBnwCeTLKsZ9gngSe67W3A+iSLkqwCVgPbp7dtSZq7+rmqZBmwJcl8xoJ+a1V9L8l/SrKGsWWQfcBnAKpqZ5KtwC7gKHCLV5RI0vSZMLir6nHg6nHqN53mmM3A5qm1Jkkaj9+clKTGGNyS1BiDW5IaY3BLUmMMbklqjMEtSY0xuCWpMQa3JDXG4JakxhjcktQYg1uSGmNwS1JjDG5JaozBLUmNMbglqTEGtyQ1xuCWpMYY3JLUGINbkhpjcEtSYwxuSWqMwS1JjUlVDboHkhwGfgG8NOheZsBSnFdrZuvcnFdbfrOqhsbbcVYEN0CSHVU1POg+ppvzas9snZvzmj1cKpGkxhjcktSYsym4bx90AzPEebVnts7Nec0SZ80atySpP2fTGbckqQ8DD+4k1yfZk2RvklsH3c+ZSnJnkkNJnuipXZjk/iRPdc9LevZt6ua6J8l1g+l6YklWJvlBkt1Jdib5fFdvem5Jzk2yPclj3by+1NWbntdxSeYn+dsk3+tez5Z57Uvy0ySPJtnR1WbF3Calqgb2AOYDTwNXAOcAjwFXDrKnSczhHwEfAZ7oqf174NZu+1bg33XbV3ZzXASs6uY+f9BzOMW8lgEf6bbPB37W9d/03IAAi7vthcBDwDWtz6tnfv8K+Evge7Pl38Wu333A0hNqs2Juk3kM+ox7LbC3qp6pql8BdwPrBtzTGamqB4GXTyivA7Z021uAG3vqd1fVkap6FtjL2D+Ds05VHaiqn3TbbwC7geU0Prca82b3cmH3KBqfF0CSFcDvA3/eU25+Xqcxm+d2WoMO7uXA8z2vR7pa6y6pqgMwFoDAxV29yfkmuRy4mrGz0+bn1i0nPAocAu6vqlkxL+A/AP8aGO2pzYZ5wdh/XP86ySNJNna12TK3M7ZgwJ+fcWqz+TKX5uabZDFwD/CFqno9GW8KY0PHqZ2Vc6uqY8CaJBcA9yb50GmGNzGvJH8AHKqqR5Jc288h49TOunn1+GhV7U9yMXB/kidPM7a1uZ2xQZ9xjwAre16vAPYPqJfp9GKSZQDd86Gu3tR8kyxkLLTvqqrvduVZMTeAqnoV+CFwPe3P66PAP02yj7Elx48l+TbtzwuAqtrfPR8C7mVs6WNWzG0yBh3cDwOrk6xKcg6wHtg24J6mwzZgQ7e9Abivp74+yaIkq4DVwPYB9DehjJ1a3wHsrqqv9uxqem5JhrozbZKcB3wCeJLG51VVm6pqRVVdztifo/9RVf+CxucFkOS9Sc4/vg38HvAEs2Bukzbovx0FbmDsioWngT8edD+T6P87wAHgbcb+S38zcBHwAPBU93xhz/g/7ua6B/gng+7/NPP6h4z97+XjwKPd44bW5wZ8GPjbbl5PAP+2qzc9rxPmeC3vXFXS/LwYu+rsse6x83hOzIa5TfbhNyclqTGDXiqRJJ0hg1uSGmNwS1JjDG5JaozBLUmNMbglqTEGtyQ1xuCWpMb8P/DjrTvnckq1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "\n",
    "tmp_env = gym.make(\"CartPole-v0\")\n",
    "tmp_env.reset()\n",
    "\n",
    "plt.imshow(tmp_env.render('rgb_array'))\n",
    "print(\"Observation space:\", tmp_env.observation_space)\n",
    "print(\"Action space:\", tmp_env.action_space)\n",
    "state_size = tmp_env.observation_space.shape[0]\n",
    "n_actions = tmp_env.action_space.n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to interact with the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial observation: [-0.0441675   0.0355158  -0.03658269 -0.02903847]\n",
      "At each step it returns:\n",
      "new observation code: [-0.04345718 -0.15906295 -0.03716346  0.25188148]\n",
      "reward: 1.0\n",
      "is game over?: False\n"
     ]
    }
   ],
   "source": [
    "print(\"Initial observation:\", tmp_env.reset())\n",
    "\n",
    "new_s, reward, done, _ = tmp_env.step(0)\n",
    "print(\"At each step it returns:\")\n",
    "\n",
    "print(\"new observation code:\", new_s)\n",
    "print(\"reward:\", reward)\n",
    "print(\"is game over?:\", done)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display an episode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_session(env, agent, t_max=500):\n",
    "    total_reward = 0\n",
    "    plt.figure(figsize=(4, 3))\n",
    "    display.clear_output(wait=True)\n",
    "\n",
    "    s = env.reset()\n",
    "    \n",
    "    for t in range(t_max):\n",
    "        plt.gca().clear()\n",
    "        \n",
    "        a = agent.get_action(torch.tensor(s).float())\n",
    "        new_s, r, done, info = env.step(a)\n",
    "        s = new_s\n",
    "        total_reward += r\n",
    "        # Draw game image on display.\n",
    "        plt.imshow(env.render('rgb_array'))\n",
    "\n",
    "        display.display(plt.gcf())\n",
    "        display.clear_output(wait=True)\n",
    "        \n",
    "        if done:\n",
    "            break\n",
    "            \n",
    "    return total_reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RandomPolicy:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total reward:  16.0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQcAAAC4CAYAAADuQCmzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAOdElEQVR4nO3db2xV933H8ffHfzE2ARyMYzBJSOIsI9WWNARFyrRFTbvQbip5gkSlVTyIhCYlU6tNmsgqbaoipGwPqj1ZHqA1KtLaIqQ2Coo6tYy1a0mbEEhCgBCIU0JwTTDhT8xf/7vfPfCJuYGf7Yvx9blcPi/p6p7f7557zwdsPpxz7j9FBGZmV6rJO4CZVSaXg5kluRzMLMnlYGZJLgczS3I5mFlS2cpB0kpJByV1S1pfru2YWXmoHK9zkFQLHAK+AvQAbwDfiIh3p31jZlYW5dpzWAF0R8TvI2IQ2AysKtO2zKwMylUOi4GjReOebM7MbhB1ZXpcJeY+d/wiaR2wDqC5ufmh++67r0xRzGwiu3fv/iQi2q6cL1c59ABLisadQG/xChGxEdgIsHz58ti1a1eZopjZRCQdSc2X67DiDaBL0lJJDcAaYGuZtmVmZVCWPYeIGJb0DPBzoBZ4MSL2l2NbZlYe5TqsICJ+BvysXI9vZuXlV0iaWZLLwcySXA5mluRyMLMkl4OZJbkczCzJ5WBmSS4HM0tyOZhZksvBzJJcDmaW5HIwsySXg5kluRzMLMnlYGZJLgczS3I5mFmSy8HMklwOZpbkcjCzJJeDmSW5HMwsyeVgZkkuBzNLcjmYWZLLwcySXA5mluRyMLOkSctB0ouS+iTtK5prlbRN0vvZ9fyi256V1C3poKQnyhXczMqrlD2HHwArr5hbD2yPiC5gezZG0jJgDXB/dp8XJNVOW1ozmzGTlkNE/Bo4dcX0KmBTtrwJeLJofnNEDETEYaAbWDFNWc1sBk31nEN7RBwDyK4XZvOLgaNF6/Vkc2Z2g5nuE5JKzEVyRWmdpF2Sdp04cWKaY5jZ9ZpqORyX1AGQXfdl8z3AkqL1OoHe1ANExMaIWB4Ry9va2qYYw8zKZarlsBVYmy2vBV4uml8jqVHSUqAL2Hl9Ec0sD3WTrSDpx8BjwAJJPcC/AM8DWyQ9BXwErAaIiP2StgDvAsPA0xExUqbsZlZGk5ZDRHxjnJseH2f9DcCG6wllZvnzKyTNLMnlYGZJLgczS3I5mFmSy8HMklwOZpbkcjCzJJeDmSW5HMwsyeVgZkkuBzNLcjmYWZLLwcySXA5mluRyMLMkl4OZJbkczCzJ5WBmSS4HM0tyOZhZksvBzJJcDmaW5HKw6xaFAhHJbz20G9ik31thNpEoFOjd9TKqqaPp1sXMXnAHDc3zUE1t3tHsOrkc7LqMDF7gk4O/ZfhiP6qppbn9brq++ncuhyrgwwq7LhdP/YGRwYsARGGE+tlzUW19zqlsOrgcbMoigoune4mRobG5ubd/IcdENp1cDnZdPj3yzthybUMTzW13IinHRDZdXA42ZcMX+7l05uOxcf3sudTPnptjIptOk5aDpCWSfinpgKT9kr6VzbdK2ibp/ex6ftF9npXULemgpCfK+Qew/AyeP8PQhf6xcXP7XdTUz8oxkU2nUvYchoF/iIg/Bh4Bnpa0DFgPbI+ILmB7Nia7bQ1wP7ASeEGST11XmYjg0yN7iMLw2Nwtnct8SFFFJi2HiDgWEW9my2eBA8BiYBWwKVttE/BktrwK2BwRAxFxGOgGVkx3cMtXFEY4U3S+oaaugVlzb8sxkU23azrnIOlO4EHgdaA9Io7BaIEAC7PVFgNHi+7Wk81ZFRk8d4qh86fHxg0trcya73KoJiWXg6QW4CfAtyOif6JVE3NXvbZW0jpJuyTtOnHiRKkxrEIM9J9g+NK5sfGcRff5hU9VpqRykFTPaDH8MCJ+mk0fl9SR3d4B9GXzPcCSort3Ar1XPmZEbIyI5RGxvK2tbar5LSefHtlTNBJNty5G8pNf1aSUZysEfB84EBHfK7ppK7A2W14LvFw0v0ZSo6SlQBewc/oiW95Ghga4cPLykWNtYxO3dC7LMZGVQynvrXgU+CawV9Lb2dw/Ac8DWyQ9BXwErAaIiP2StgDvMvpMx9MRMTLtyS03I4MXGPi0b2zc1NpJQ0trjomsHCYth4jYQfo8AsDj49xnA7DhOnJZBTv3cfcV5xvuZfxfEbtR+SDRrklEcP744csTqqGptdOvb6hCLge7NlHgwicfjQ3rGmfT0n53joGsXFwOdk0Gz53iwsnL5TB7we3UNjblmMjKxeVg1+T8iSMUhi+/RXvW/EXU+PMbqpLLwUoWEVw8eRSiMDbnz2+oXi4HK1kURvj06L6xcf3secya15FjIisnl4OVbPDcSQbPFb2fYk4rdbNackxk5eRysJIN9J9gZOD82Njvp6huLgcrSURw5vBbY2PV1DJn0R/59Q1VzOVgJRkZvEh/z4GxcW1DE423+A1z1czlYCUZ6O9jeODyS6Znzevw+ymqnMvBJhURXDzVS2FoYGzO78Ksfi4HK0nx5zeoppam1kU+31DlXA42qZHBC1d9BH3LbffkmMhmgsvBJjV0oZ+BsyfHxs3td1Pb2JxjIpsJLgebVH/P/s995d2cRff6kOIm4HKwCUWhwPm+D8fGqqmjab4/TPxm4HKwCRVGhkbfbJVpaJlP062dOSaymeJysAldOn2Mgf7LXx3Q3H6336J9k3A52ITO9x0mCpc/H7ipdTGq8a/NzcA/ZRtXRIELnxwZG6umlrlL7s8xkc0kl4ONa2TwEmd7D46NG29Z6JdM30RcDjauS2c+/txH0DfOXUhN/awcE9lMcjlYUkRw6czHFIYHx+Z8SHFzcTnYOIIzh98cG9XUNdK88C6/+OkmUsrX4VkVKhQKDA0NjXv70IUznDv+wdi4prEZGpoZGBi4at26ujpqa/2JUNXG5XCT2rFjB88888y4tz/U1c7ffmUptdmewqtvHeKbz/8FJPYcnnvuOVatWlW2rJYPl8NNqr+/n7179457+z0Lb2P36ZWAWNq8j5+/9n/s3Xcwue7p06eT83Zjm/Scg6RZknZK2iNpv6TvZvOtkrZJej+7nl90n2cldUs6KOmJcv4BbPrV1NTQcdfX6Ru4g76B23nz5Aq6e8/mHctmWCknJAeAL0XEnwIPACslPQKsB7ZHRBewPRsjaRmwBrgfWAm8IMkHpDeQBbfMpvPWWUAAQc/xUxw6enKyu1mVmbQcYtRnT3bXZ5cAVgGbsvlNwJPZ8ipgc0QMRMRhoBtYMa2prazOXxrgNzt+wB8O/w8tI3s42/MjhoevPhFp1a2kcw7Z//y7gXuA/4iI1yW1R8QxgIg4Jmlhtvpi4LWiu/dkc+M6d+4cr7766jWHt6k7cODAuLedvzTEf+94lfrXfkdLUwMtTQ0TPlZ3d7d/flWopHKIiBHgAUnzgJckTfQFiaknwuOqlaR1wDqARYsW0dbmjzmfSXPnzp10naHhAqfPXuL02UsTrjdnzhz//KrQNT1bERFnJP2K0XMJxyV1ZHsNHUBftloPsKTobp1Ab+KxNgIbAZYvXx733nvvFOLbVB06dGjaHqu9vR3//KpPKc9WtGV7DEhqAr4MvAdsBdZmq60FXs6WtwJrJDVKWgp0ATunO7iZlVcpew4dwKbsvEMNsCUiXpH0O2CLpKeAj4DVABGxX9IW4F1gGHg6OywxsxvIpOUQEe8ADybmTwKPj3OfDcCG605nZrnxKyRvUg899BBbtmyZlsd6+OGHp+VxrLK4HG5SHR0drF69Ou8YVsH8lm0zS3I5mFmSy8HMklwOZpbkcjCzJJeDmSW5HMwsyeVgZkkuBzNLcjmYWZLLwcySXA5mluRyMLMkl4OZJbkczCzJ5WBmSS4HM0tyOZhZksvBzJJcDmaW5HIwsySXg5kluRzMLMnlYGZJLgczS3I5mFmSy8HMklwOZpbkcjCzJJeDmSUpIvLOgKQTwHngk7yzXGEBzlQKZypdJea6IyLarpysiHIAkLQrIpbnnaOYM5XGmUpXqblSfFhhZkkuBzNLqqRy2Jh3gARnKo0zla5Sc12lYs45mFllqaQ9BzOrILmXg6SVkg5K6pa0fga3+6KkPkn7iuZaJW2T9H52Pb/otmezjAclPVGmTEsk/VLSAUn7JX0r71ySZknaKWlPlum7eWcq2k6tpLckvVJBmT6UtFfS25J2VUquKYmI3C5ALfABcBfQAOwBls3Qtv8c+CKwr2ju34D12fJ64F+z5WVZtkZgaZa5tgyZOoAvZstzgEPZtnPLBQhoyZbrgdeBR/L+u8q29ffAj4BXKuHnl23rQ2DBFXO555rKJe89hxVAd0T8PiIGgc3AqpnYcET8Gjh1xfQqYFO2vAl4smh+c0QMRMRhoJvR7NOd6VhEvJktnwUOAIvzzBWjzmXD+uwSeWYCkNQJ/BXwn0XTuWaaQKXmmlDe5bAYOFo07snm8tIeEcdg9B8qsDCbn/Gcku4EHmT0f+pcc2W7728DfcC2iMg9E/DvwD8ChaK5vDPBaHH+QtJuSesqKNc1q8t5+0rMVeLTJzOaU1IL8BPg2xHRL6U2P3O5ImIEeEDSPOAlSV+YYPWyZ5L010BfROyW9Fgpdyl3piKPRkSvpIXANknvVUiua5b3nkMPsKRo3An05pQF4LikDoDsui+bn7GckuoZLYYfRsRPKyUXQEScAX4FrMw506PA1yV9yOih6Jck/VfOmQCIiN7sug94idHDhNxzTUXe5fAG0CVpqaQGYA2wNcc8W4G12fJa4OWi+TWSGiUtBbqAndO9cY3uInwfOBAR36uEXJLasj0GJDUBXwbeyzNTRDwbEZ0RcSejvzP/GxF/k2cmAEnNkuZ8tgz8JbAv71xTlvcZUeBrjJ6V/wD4zgxu98fAMWCI0QZ/CrgV2A68n123Fq3/nSzjQeCrZcr0Z4zuVr4DvJ1dvpZnLuBPgLeyTPuAf87mc/27KtrWY1x+tiLvn99djD77sAfY/9nvc965pnrxKyTNLCnvwwozq1AuBzNLcjmYWZLLwcySXA5mluRyMLMkl4OZJbkczCzp/wEFfB0PAfIN5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 288x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "total_reward = display_session(tmp_env, RandomPolicy(n_actions))\n",
    "print('Total reward: ', total_reward)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The challenge of the day"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Algorithm](https://i.ibb.co/zFm6BrB/Screenshot-from-2021-09-17-13-38-00.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_session(env, agent, t_max=500):\n",
    "    total_reward = 0\n",
    "    s = env.reset()\n",
    "    \n",
    "    for t in range(t_max):\n",
    "        a = agent.get_action(torch.tensor(s).float())\n",
    "        new_s, r, done, info = env.step(a)\n",
    "        total_reward += r\n",
    "        s = new_s\n",
    "        \n",
    "        if done:\n",
    "            break\n",
    "            \n",
    "    return total_reward\n",
    "\n",
    "\n",
    "def score(env, agent, n=10, t_max=500):\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modification of the model using a noise "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_noise_to_model(model, noise, copy=False):\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Controller of the evolution\n",
    "\n",
    "#### Task 1: Implement the algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EvolutionManager:\n",
    "    def __init__(self, get_env_function, lr=0.001, std=0.01, n_samples = 64, normalize=True):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.lr = lr\n",
    "        self.std = std\n",
    "        self.normalize = normalize\n",
    "        self.n_samples = n_samples\n",
    "        self.mean_reward_history = []\n",
    "        \n",
    "        self.env = get_env_function()\n",
    "        \n",
    "    def get_noised_model(self, model):\n",
    "        pass\n",
    "\n",
    "    def optimize(self, model, noises, rewards):\n",
    "        pass\n",
    "    \n",
    "    def step(self, model):\n",
    "        pass\n",
    "        \n",
    "    def update_log(self, rewards):\n",
    "        mean_reward = np.mean(rewards)\n",
    "        self.mean_reward_history.append(mean_reward)\n",
    "\n",
    "        clear_output(True)\n",
    "        print(\"last mean reward = %.3f\" % mean_reward)\n",
    "        plt.figure(figsize=[8, 4])\n",
    "        plt.subplot(1, 2, 1)\n",
    "        plt.plot(self.mean_reward_history, label='Mean rewards')\n",
    "        plt.legend()\n",
    "        plt.grid()\n",
    "\n",
    "        plt.subplot(1, 2, 2)\n",
    "        plt.hist(rewards)\n",
    "        plt.grid()\n",
    "\n",
    "        plt.show()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task 2: Implement a neural network-based policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLPPolicy(nn.Module):\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Environment 1: CartPole-v0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_env_function():\n",
    "    env = gym.make('CartPole-v0').env\n",
    "    return env\n",
    "tmp_env = get_env_function()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_states = tmp_env.observation_space.shape[0]\n",
    "n_actions = tmp_env.action_space.n\n",
    "model = <>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last mean reward = 493.230\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAD4CAYAAADW+i6uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3hV9Z3v8fc3d3IhIRciJGCQSxAJCFKkihrrDS+FtqMVntZHR+d4zqlVW7VTbafW9tRn2k5nnul0TmcO0zo61Wq1aqVirY5tFKh4CRe5hEi4BAKBXCA3IOT2O3/sHQyQy06yLyvZn9fz5Mnea6+99jcre+/vWuv3+31/5pxDREREvCMm0gGIiIjI6ZScRUREPEbJWURExGOUnEVERDxGyVlERMRj4iIdAEB2drYrKCgYcL1jx46RkpIS+oA8TPvAJ1r3Q2lpaZ1zLifScfQnIyPDTZs2LdJh9Mqr7xuvxgWKbagCia2/z7MnknNBQQEffvjhgOuVlJRQXFwc+oA8TPvAJ1r3g5lVRjqGgeTm5gb0eY4Er75vvBoXKLahCiS2/j7PuqwtIiLiMUrOIiIiHqPkLCIi4jGeaHPuTXt7O1VVVbS2tp5alp6eTllZWQSjijyv7YOkpCTy8/OJj4+PdCgiIqOGZ5NzVVUVaWlpFBQUYGYANDc3k5aWFuHIIstL+8A5R319PVVVVUyZMiXS4YiIjBoBXdY2s71mtsXMNpnZh/5lmWb2ppnt9P8e12P9R8yswszKzey6oQTW2tpKVlbWqcQs3mNmZGVlnXZ1Q0REhm8wbc5XOucudM4t8N9/GHjLOTcdeMt/HzObBSwHLgCWAD83s9ihBKfE7H36H4mIBN9wLmsvA4r9t58CSoBv+pc/55w7CewxswpgIfDuMF5LxLOcc9S2nKTq6An2HznOgYYTZKckMndSBtPGpxIbowMYERmcQJOzA94wMwf8P+fcSiDXOVcN4JyrNrPx/nXzgPU9nlvlX3YaM7sbuBt8RQtKSkpOezw9PZ3m5ubTlnV2dp61LJTGjh3Lrbfeyn/8x38A0NHRwfTp01mwYAEvvPBC2OLoKRz7YPbs2bz99ttkZWUFtH5ra+tZ/79QOtLaxUfVxznW/mdS4kOf+JxzNLVB3Yku6k446k50UXvCnbpdf8LR3tX7c5NioSA9hvPSY5maEcN56TGMS9IgCRHpX6DJ+VLn3EF/An7TzHb0s25v35burAW+BL8SYMGCBe7MSiplZWVndXwKd2eolJQUysvLiYuLY8yYMfzhD38gPz+fuLi4sMXR0dFBXNwn/6Zg74Mztw++S9WpqakBv05SUhLz5s0LWkz9eXP7Yb73wmYaTxjP7W7ls3Mm8qVF5zI3Pz2ol9j3HznOK5sO8NqWQ+yua6H1jOybmZJA/rgxzJ8whvxxyeSPG+P/SWZixhgON7WyeX8Dm/Y3sHl/A2/ua+K1Pb6PwTljk5g7KZ0LJ41j7qR05uRnkJro2b6ZIlGj4OHVQdvWk0uGV1Y0oG8E59xB/+8aM3sZ32Xqw2Y2wX/WPAGo8a9eBUzq8fR84OCwooyg66+/ntWrV3PzzTfz7LPPsmLFCtasWQP4aqfee++9bNmyhY6ODh577DGWLVvG3r17ue222zh27BgA//qv/8oll1xCSUkJjz32GNnZ2WzdupWLLrqIp59++qykUlxczCWXXMK6detYunQpxcXFPPDAA7S0tJCRkcHTTz9NbGws119/PaWlpWzevJkLL7yQyspKJk+ezNSpU9myZQtvvfUWP/jBD2hrayMrK4tnnnmG3NxcHnvsMQ4ePMjevXvJzs7mZz/7GStWrKC2tpaFCxfinDv1933xi1+kqqqKzs5OvvOd73DrrbeG9x/gd7Kjkx/+YQf/uW4vs/PGcuc5MRyOy+V3Gw/wQmkVs/PG8qWLz2Xp3ImkDDHRNZ5o57Ut1by84QDv7z0CwMKCTG5bdG6PBJxM3rgxAybT1JxUpuak8oX5+QC0tndSVt10Kllvrmrkj9sOA2AG08encuXM8Txy/flDil1ERpcBv8XMLAWIcc41+29fC3wfWAXcDvzQ//sV/1NWAb82s38CJgLTgfeHE+T3fr+N7Qeb6OzsJDZ2SH3LzjJr4li++9kLBlxv+fLlfP/73+emm27io48+4s477zyVnB9//HE+85nP8MQTT9DQ0MDChQu5+uqrGT9+PG+++SZJSUns3LmTFStWnKo1vHHjRrZt28bEiRO59NJLWbduHYsXLz7rdRsaGnj77bdpb2/niiuu4JVXXiEnJ4cnn3ySb3/72zzxxBO0trbS1NTEmjVrWLBgAWvWrGHx4sWMHz+e5ORkFi9ezPr16zEzfvGLX/DjH/+Yf/zHfwSgtLSUtWvXMmbMGO677z4WL17Mo48+yurVq1m5ciUAr7/+OhMnTmT1at/RZGNjY1D2/WDtrTvGV5/dwNYDTdxxSQGP3DCTd9euobi4iEeun8nvNh3kmfWVPPLSFh5fXcbn5+XxpUWTmXnO2AG33dbRRUl5DS9vPMBbZTW0dXYxNSeFb1xXyNK5E5mUmRyUvyEpPpZ5k8cxb/KpQQ00HG9jc1Ujm/Y1sLmqgYZj7UF5LREZ+QI5xcgFXvaf3cUBv3bOvW5mHwDPm9ldwD7gFgDn3DYzex7YDnQA9zjnOkMSfRjMmTOHvXv38uyzz3LDDTec9tgbb7zBqlWr+MlPfgL42l737dvHxIkT+epXv8qmTZuIjY3l448/PvWchQsXkp/vO5u68MIL2bt3b6/JufsMtby8nK1bt3LNNdcAvuIseXm+Jvzus+t33nmHb33rW7z++us457jssssA31jxW2+9lerqatra2k4bi7x06VLGjBkDwDvvvMNLL70EwI033si4cb4EUlRUxEMPPcQ3v/lNbrrpplPbDadXNh3g2y9vJTbGWHnbRVx7wTmnPZ6WFM9ti87lyxdPZsO+ozyzfh+/+XA/v1pfyYJzx/GlRZO5fvYEkuI/OahzzrFhXwMvb6zi1Y+qaTjeTnZqAl9aNJnPz8ujKC+4l8j7kpGcwBUzcrhihqcnmRKRCBgwOTvndgNze1leD1zVx3MeBx4fdnR+3We4kSrAsXTpUh566CFKSkqor68/tdw5x4svvkhhYeFp6z/22GPk5uayefNmurq6SEpKOvVYYmLiqduxsbF0dHT0+prdU40557jgggt4911fZ/ee++Cyyy5jzZo1VFZWsmzZMn70ox9hZtx0000A3HvvvTzwwAMsXbr01CX1M7ffrbdkNGPGDEpLS3nttdd45JFHuPbaa3n00UcH3F/BcKKtk8dWbeM3H+5nwbnj+OmKeeRljOlzfTPjonMzuejcTP7uplm8WFrFM+9V8vXfbOb7v9/OLQsmce2sXNbsrON3mw5QWX+cpPgYrp11Dp+fl8fi6dnEx6qjloh4g76NAnDnnXfy6KOPUlRUdNry6667jp/97Gen2mg3btwI+C7/TpgwgZiYGH71q1/R2Tn0CweFhYXU1taeSs7t7e1s27YNgMsvv5ynn36a6dOnExMTQ2ZmJq+99hqXXnrpqTi6z7KfeuqpPl/j8ssv55lnngHgD3/4A0ePHgXg4MGDJCcn8+Uvf5mHHnqIDRs2DPnvGIyPDzez9F/X8nzpfu65cirP3b2o38R8psyUBP7H5efxpweLefqui7l4Sha/XLuHm//9Xf7lTzvJHzeGf7h5Dh98+2r+ZcU8rpw5XolZRDxFXUQDkJ+fz/3333/W8u985zt87WtfY86cOTjnKCgo4NVXX+UrX/kKf/VXf8ULL7zAlVdeOazJwBMSEvjtb3/LfffdR2NjI21tbTzwwANccMEFFBQUAL7kCrB48WKqqqpOXZZ+7LHHuOWWW8jLy2PRokXs2bOn19f47ne/y4oVK5g/fz5XXHEFkydPBmDLli184xvfICYmhvj4eP7t3/5tyH9HIJxzPPfBfr73+22kJsbzqzsvZvH07CFvLybGWDw9m8XTsznc1Mr63fUsnJLJhPTAE/1oY2ZPADcBNc652f5l/wB8FmgDdgF/7Zxr8D/2CHAX0Anc55z7Y0QCF4ky1n3WF0kLFixwZ07OXlZWxvnnn95z1Ut1pSPFi/ugt//VYDW3tvOtl7fy+80HuWx6Nv/4xbmMT0vqc30vT7IeSmZW2qNK31CefznQAvxXj+R8LfAn51yHmf0IwDn3TX+1v2fxjc6YCPw3MGOgPiSFhYWuvLx8qCGGlFffN16NC6IrtmAPpRootv4+zzpzloj7qKqBe5/dSNXRE3zjukL+9xVTiVFVrZBwzr1jZgVnLHujx931wM3+26r2JxIhSs4SUa9vrebeZzeSk5rIb+5exIKCzEiHFO3uBH7jvx1QtT8RCT5PJ2fnnCZW8LjhNos8+Ze9TBqXzEtfuYSM5IQgRSVDYWbfxjf88ZnuRb2s1us/vGc53pycnLCWcx2MlpYWT8bm1bggumJ7sKj30TNDMdzYPJuck5KSqK+v17SRHtY9n3PPoWKD0d7Zxeb9jdz6qUlKzBFmZrfj6yh2lfvkiCvgan89y/EWFhaeVY7XK7zafurVuCC6YrsjzG3O/fFscs7Pz6eqqora2tpTy1pbW4ecCEYLr+2DpKSkU0VVBmtHdTMn2ju56NxxA68sIWNmS/DNKHeFc+54j4eCXu1PRALj2eQcHx9/WkUr8B0lhWuCBa8aTfugtNJXv1rJOXzM7Fl8U71mm1kV8F3gESAR36Q2AOudc/9rtFX7ExlJPJucZfTbsK+BCelJTBxEgREZHufcil4W/7Kf9YNa7U9EAqOySBIxpZVHmT9ZZ80iImdScpaIONTYyoGGE8zXJW0RkbMoOUtEbNjnq9+t9mYRkbMpOUtElFYeJTEuhlkTBp5zWUQk2ig5S0SUVh5lbn4GCXF6C4qInEnfjBJ2re2dbDvYyLxzMyIdioiIJyk5S9htPdBIe6fjIvXUFhHplZKzhF1ppa8zmHpqi4j0TslZwq608igFWclkpyZGOhQREU9Scpawcs6xYd9RnTWLiPRDyVnCat+R49S1tGl8s4hIP5ScJay6i4+obKeISN+UnCWsSiuPkpoYx4zctEiHIiLiWUrOElallQ3Mm5xBbIxFOhQREc9ScpawaW5tp/xQky5pi4gMQMlZwmbz/ka6nCa7EBEZiJKzhE1p5VHM4MLJKtspItIfJWcJmw37jjJjfBpjk+IjHYqIiKcpOUtYdHWp+IiISKCUnCUsKmpbaG7tUHuziEgAlJwlLLonu1ByFhEZmJKzhEVp5VEyUxIoyEqOdCgiIp6n5CxhsaHyKPMnZ2Cm4iMiIgMJODmbWayZbTSzV/33M83sTTPb6f89rse6j5hZhZmVm9l1oQhcRo4jx9rYXXdMncFERAI0mDPn+4GyHvcfBt5yzk0H3vLfx8xmAcuBC4AlwM/NLDY44cpItNE/2cVFqgwWcWb2hJnVmNnWHst0oC3iMQElZzPLB24EftFj8TLgKf/tp4DP9Vj+nHPupHNuD1ABLAxOuDISlVYeJS7GmJOv4iMe8CS+g+aedKAt4jFxAa73z8DfAj2nEsp1zlUDOOeqzWy8f3kesL7HelX+Zacxs7uBuwFyc3MpKSkZMIiWlpaA1hvNRuI+eGvzCSalGe/9ZU3QtjkS94MXOOfeMbOCMxYvA4r9t58CSoBv0uNAG9hjZt0H2u+GI1aRaDZgcjazm4Aa51ypmRUHsM3eevy4sxY4txJYCbBgwQJXXDzwpktKSghkvdFspO2D9s4uKt/6IysWTqa4+IKgbXek7QePG9aBNpx+sJ2Tk+PZAyevHtR5NS6IrtgeLOoI2raGG1sgZ86XAkvN7AYgCRhrZk8Dh81sgv/DPAGo8a9fBUzq8fx84OCQI5QRray6idb2Ls1ENTIFdKANpx9sFxYWBnSwHQlePajzalwQXbHd8fDqoG3rySUpw4ptwDZn59wjzrl851wBvvanPznnvgysAm73r3Y78Ir/9ipguZklmtkUYDrw/pAjlLArq27ia89tpKS8ZuCVB7BBxUdGgsP+A2x0oC3iDcMZ5/xD4Boz2wlc47+Pc24b8DywHXgduMc51zncQCX0qhtP8NALm7nhX9bwu00HefjFLRxvG95lntJ9DUxIT2JixpggRSkhoANtEY8JtEMYAM65EnydRXDO1QNX9bHe48Djw4xNwqSptZ1/L9nFL9fuwTn4m8VTuHhKFn/zXx/yH+/s4f6rpw952xsqNdmFl5jZs/g6f2WbWRXwXXwH1s+b2V3APuAW8B1om1n3gXYHOtAWCZtBJWcZXdo6uvj1e5X8y58qOHKsjWUXTuShawuZlOkrsXlD0Tn8+9u7WL5wErljkwa9/erGExxoOMFdi6cEO3QZIufcij4e0oG2iIcoOUch5xx/2HqIH7++g731x/n0eVl864bzKcpPP229h5ecz39vr+Ef/ljOT26ZO+jX2VDZAKi9WURksJSco8yHe4/w+GtlbNzXwIzcVP7zjk9RXJjTa83ryVnJ/PWlBaxcs5s7Lilgdl56L1vs24Z9R0mMi+H8CWODFb6ISFTQxBdRYldtC//zVx9y87+/y4GjJ/jRXxXx2n2XceXM8f1ORvGVK6cxLjmBH6zejnO9jqLpU2nlUebmZ5AQp7eZiMhg6Mx5lOvqcvxgdRlPvbuXpLgYHrxmBnddNoXkhMD+9elj4vn61dP5zivbeGP7Ya674JyAntfa3sm2g43ctfi8YUQvIhKddEozyn10oJEn1u1h6dyJlHzjSu69anrAibnbioWTmTY+lb9/rYy2jq6AnrPlQCPtnU7tzSIiQ6DkPMrtqG4C4OtXzyAnLXFI24iLjeHbN57P3vrj/Gp9ZUDPKfUXH5k/WZNdiIgMlpLzKLfjUDMpCbHkjxteEZDiGTlcNj2bn/73xxw91jbg+qWVR5mSnUJW6tAOCEREopmS8yhXVt1E4TlpxMT03ekrEGbG3904i5aTHfz0rZ39ruucY+O+o8zTWbOIyJAoOY9izjnKqpuCNpSp8Jw0li+czNPrK9lV29LnevuOHKeupU3tzSIiQ6TkPIpVN7bS1NrBzCCOM/761TNIio/l718r63OdUk12ISIyLErOo9iOQ77OYOefkxa0beakJXLPldP477Ia1lXU9bpOaeVR0hLjmD4+eK8rIhJNlJwj5L3d9Xzr5S2DLuwxGGXVzYDvcnQw/fWlBeRljOEHq8vo7Do7/tLKo1w4OYPYYbZzi4hEKyXnCHnq3b38+r19VB09EbLXKKtuYlLmGNKS4oO63aT4WB6+fiZl1U38tnT/aY81t7ZTfriZ+ZN1SVtEZKiUnCOgs8vxl131AGw90Biy19lxqJmZ54SmrvVNcyYwf3IGP3njY1pOfjLn8+b9jTin9mYRkeFQco6A7QebaDjeDvgqeIVCa3snu2tbgtre3JOZ8Xc3zaK2+ST/7+1dp5aXVh7FDC7UMCoRkSFTco6Atf6OVHkZY9hSFZrkXFHTQpcjqD21zzR/8jiWzp3Iynd2c6DBd3m+dN9RCnPTGBvkS+kiItFEyTkC1lXUUZibxuUzstlyoDEkncLK/GU7Qz1d498uKQTgH17fQVeXY2PlUebrkraIyLAoOYdZa3sn7+89wqXTsinKy6DxRDv7jwS/U1hZdTNj4mOZnJkc9G33lD8umb+5bAq/23SQ35ZW0Xyyg4vUGUxEZFiUnMOstPIobR1dLJ6exZz8dAA+OtAQ9NfZcaiJGeekhWU40/8unkZ2agJ/98pWAJ05i4gMk5JzmK2tqCMuxlg4JYsZuWkkxMYEvd35VNnOEHUGO1NqYhwPXltIW0cXmSkJFGSF9mxdRGS0U3IOs3UVdcybnEFqYhwJcTHMnJDGliD32K5tPsnR4+3MDFNyBvjigknMzU/nihk5mKn4iIjIcMRFOoBo0nC8jS0HGrn/qumnlhXlpbNq80G6utywZ47qtj1MncF6io0xXvhflxCnqmAiIsOmM+cwendXPc7B4mnZp5YV5aXT3NpB5ZHjQXudHYd8ZTtDVYCkLwlxMUE7wBARiWZKzmG0tqKOlIRY5k76pEBHkb9TWDAvbe+obmJiehLpyRprLCIyEik5h9G6ijoWnZdFfOwnu31GbhoJcTFsqQpej+0dh5pDWnxERicz+7qZbTOzrWb2rJklmVmmmb1pZjv9v9UVXyQMlJzDZP+R4+ytP86lPS5pA8THxjBrwlg+ClKP7baOLipqWjh/gqZrlMCZWR5wH7DAOTcbiAWWAw8DbznnpgNv+e+LSIgpOYfJX3b5SnYunp591mNFeelsO9hEVy/TLw5WRU0LHV0u7O3NMirEAWPMLA5IBg4Cy4Cn/I8/BXwuQrGJRBX11g6TtRX15KQlMn186lmPFeWn86v1leypP8bUnLMfH4wdh7p7auvMWQLnnDtgZj8B9gEngDecc2+YWa5zrtq/TrWZje/t+WZ2N3A3QE5ODiUlJWGKfHBaWlo8GZtX44Loiu3Boo6BVwrQcGNTcg6Dri7HXyrquLyPMcDdlcK2VDUGITk3kxAXQ0FWyrC2I9HF35a8DJgCNAAvmNmXA32+c24lsBKgsLDQFRcXhyLMYSspKcGLsXk1Loiu2O54eHXQtvXkkpRhxabL2mGw41Az9cfazmpv7jYtJ5Wk+JigtDuXVTcxIzeVuFj9a2VQrgb2OOdqnXPtwEvAJcBhM5sA4P9dE8EYRaKGvsHDYJ1/ishLp2X1+nicv1PY1iAMpyqrbuZ8tTfL4O0DFplZsvku71wFlAGrgNv969wOvBKh+ESiii5rh8Haijqm5qQwIX1Mn+vMyc/g+Q/309nlhjxZRW3zSepaTmoYlQyac+49M/stsAHoADbiu0ydCjxvZnfhS+C3RC5Kkegx4Jmzf6zj+2a22T8G8nv+5X2OfzSzR8yswszKzey6UP4BXneyo5P39xw5rSpYb4ry0jne1snu2pYhv1a5vzJYuCa8kNHFOfdd59xM59xs59xtzrmTzrl659xVzrnp/t9HIh2nSDQI5LL2SeAzzrm5wIXAEjNbRB/jH81sFr7xkRcAS4Cfm1lsKIIfCTbua+BEe2ef7c3duiuFDafdubunts6cRURGtgGTs/PpPp2L9/84+h7/uAx4zn/UvQeoABYGNeoRZF1FHTEGi6b23t7cbWpOKmPiY4dVxnN7dRO5YxPJTEkY8jZERCTyAmpz9p/5lgLTgP/rb5/qa/xjHrC+x9Or/MvO3OapcZG5ubkBjQfz8ni7vrxWeoIpY2PYsH7dgOtOSnWs3b6PkrG1fa7T3z74cOcJxifaiNtHQzES3wsiIoEKKDk75zqBC80sA3jZzGb3s3pvvZnOKn3Vc1zkggULAhoX6eXxdr1pam1nzx/f4J4rp1FcXDjg+m83b+PZ9/ex+LLL+xwK1dc+aO/s4tCbf+T6+edSXHz+cEP3vJH2XhARGYxBDaVyzjUAJfjakvsa/1gFTOrxtHx8ZQCjzvpd9XQ5Bmxv7jYnP53W9i521R4b9GvtqTtGW2eXhlGJiIwCgfTWzvGfMWNmY/AVK9hB3+MfVwHLzSzRzKYA04H3gx34SLCuoo4x8bHMm5wx8MpAUZ5vvY+GMENVWXV32U4lZxGRkS6Qy9oTgKf87c4xwPPOuVfN7F16Gf/onNtmZs8D2/GNl7zHf1k86qytqGPhlEwS4wLrrH5edgopCb5OYbcsmDTwE3ooq24mPtY4L0dlO0VERroBk7Nz7iNgXi/L6/FVEertOY8Djw87uhGsuvEEu2qPsfxTkwN+TkyMcUFe+pCGU+041MS08WmnzRUtIiIjk77JQ2RdRT0QeHtztzl56ZRVN9He2TWo5+2oblbxERGRUULJOUTWVdSRlZLAzEEmzKL8dE52dLHzcOCVwo4ea+NQUyszNU2kiMiooOQcAs451lbUccm0bGIGWSe7KM8/feSBwDuFlR1SZzARkdFEyTkEdta0UNt8ksV9zELVn4KsFNIS4wZVKWxHta+m9kwNoxIRGRWUnENg7c7uKSIH194Mvk5hs/PS2TKITmE7DjWRnZpATlrioF9PRES8R8k5BNZV1FGQlUz+uOQhPb8oP52y6mbaOgLrFLbjULPOmkVERhEl5yBr7+xi/e76IZ01dyvKS6ets4uPDzcPuG5nl6P8UDPnqzOYiMiooeQcZJv3N3CsrXPA+Zv7Mye/u1PYwJe299Qd42RHl86cRURGESXnIFtbUYcZfHqAKSL7MzkzmbFJcQEVI/lkDmedOYuIjBZKzkG2rqKOorx0MpKHPqeymVGUnx7QcKod1c3ExhjTxqcO+fVERMRblJyDqOVkBxv3NQyrvblbUV4G5YeaOdnRf1nyHYeamJqTEnD9bhER8T4l5yB6f089HV1uWO3N3ebkp9Pe6evs1Z+y6mYVHxERGWWUnINo7c56EuNiuOjcccPeVnelsP7anRtPtHOg4YQ6g4mIjDJKzkG0rqKOTxVkkhQ//EvM+ePGkJEcz9Z+emx3n1WrM5iIyOii5BwkNc2tlB9uDkp7M/g7hQ0wfWR3T+3zdeYsIjKqKDkHyV/8U0QGo72525z8dD4+3Exre++dwsqqmxmXHE/uWJXtFBEZTZScg2RtRR0ZyfHMmhi8s9iivHQ6uhxl1U29Pl5W3cTMc8ZiNriZr0R6Y2YZZvZbM9thZmVm9mkzyzSzN81sp//38DtUiMiAlJyDwDnHuoo6LpmaRewgp4jsT1F+BkCv7c5d/rKdam+WIPop8LpzbiYwFygDHgbecs5NB97y3xeREFNyDoJdtceobmwNWntzt4npSWSlJPTa7rzvyHFOtHeqvVmCwszGApcDvwRwzrU55xqAZcBT/tWeAj4XmQhFoktcpAMYDdZV+KaIDGZ7M/g6hc3OS++1xrbKdkqQnQfUAv9pZnOBUuB+INc5Vw3gnKs2s/G9PdnM7gbuBsjJyaGkpCQsQQ9WS0uLJ2PzalwQXbE9WNQRtG0NNzYl5yBYs7OO/HFjmJw5tCki+zMnP52fl9Rxoq2TMQmfDNHaXt1MjMGMXCVnCYo4YD5wr3PuPTP7KYO4hO2cWwmsBCgsLHTFxcUhCXK4SkpK8PTeAE4AABEjSURBVGJsXo0Loiu2Ox5eHbRtPbkkZVix6bL2MJ3s6OQvu+q4YkZOSDpmFeWl09nl2H5Gp7Ad1U1MyU4JyphqEaAKqHLOvee//1t8yfqwmU0A8P+uiVB8IlFFyXmYPtx7lONtnRQX9nq1b9jm+DuFbak6fRKMHYeamamynRIkzrlDwH4zK/QvugrYDqwCbvcvux14JQLhiUQdXdYepj/vqCEhNoZLhjFFZH9yxyaSnZrIRz3anVtOdrDvyHG+uCA/JK8pUete4BkzSwB2A3+N7wD+eTO7C9gH3BLB+ESihpLzMJV8XMvCKZmkJIZmV5oZc/LTTxtOVd5dGUxnzhJEzrlNwIJeHroq3LGIRDtd1h6G/UeOU1HTQnFhTkhfpygvnYqaFo6d9PUkLKvurqmt5CwiMhopOQ9Dyce1ACFrb+5WlJdOl+NUp7Adh5pIS4pjYnpSSF9XREQiQ8l5GN4uryF/3Bim5qSE9HWK8n3TR27xFyPZUd3M+SrbKSIyaik5D9HJjk7WVdRTXBiaIVQ95Y5NIndsIlsONOKcY8ehZs5X8RERkVFLyXmI3t9zhBPtnVwZ4kva3XzTRzZQd8LRcrJD7c0iIqOYkvMQlZTXkhAbw6dDNITqTEV5GeyuO8bOhi4AZp6jM2cRkdFKyXmISspruPi8TJITwjMabU5+Os7BugPtmMp2ioiMakrOQ7D/yHF21R4LeS/tnmbn+TqFba/v4tzM5JCNqxYRkcgbMDmb2SQz+7N/8vVtZna/f3mfk7Cb2SNmVmFm5WZ2XSj/gEgoKfeVFw71+OaectISmZCehEPFR0RERrtAzpw7gAedc+cDi4B7zGwWfUzC7n9sOXABsAT4uZmNqtkZ/lxey+TMZM7LDu0QqjMV+c+eZ2oOZxGRUW3A5Oycq3bObfDfbgbKgDz6noR9GfCcc+6kc24PUAEsDHbgkdLa7puFKhxDqM40xz/eWXM4i4iMboNquDSzAmAe8B59T8KeB6zv8bQq/7Izt3Vqcvbc3NyAJqX2wqTfW+s6aG3vIqvtECUldWF97fRjXUxIdpw8UEZJ7Y6wvrbXeOG9ICISKgEnZzNLBV4Evuaca+rnrLG3B9xZC3pMzr5gwYKAJmf3wqTfb/9+Gwlx+7h72ZWMSQj/1fpJaZHfB17ghfeCiEioBNRb28zi8SXmZ5xzL/kX9zUJexUwqcfT84GDwQk38t4ur2XReVkRScwiIhIdAumtbcAvgTLn3D/1eKivSdhXAcvNLNHMpgDTgfeDF3LkVNYfY3fdMa4MYy9tERGJPoFc1r4UuA3YYmab/Mu+BfyQXiZhd85tM7Pnge34enrf45zrDHrkEVBSHp5ZqEREJLoNmJydc2vpvR0Z+piE3Tn3OPD4MOLypJLyGs7NSmZKmIdQiYhIdFGFsAC1tnfy7u76sE10ISIi0UvJOUDrd9fT2t7FFWpvFhGREFNyDlBJeS2JcTF8+rzwzEIlIiLRS8k5QG9/XMunp2aRFK8hVCIiElpKzgHYW3eMPXXHKJ6hS9oiIhJ6Ss4B+GQWKnUGExGR0FNyDkDJx7VMyU6hQEOoREQkDJScB9Da3sm7u+rDOnezSKSYWayZbTSzV/33+5y3XURCR8l5AO/urudkR5cuaUu0uB/ftLDdep23XURCS8l5ACU7akiKj+HiKZmRDkUkpMwsH7gR+EWPxX3N2y4iITSo+ZyjUcnHtVwyNVtDqCQa/DPwt0Baj2V9zdt+mp7zs+fk5Hh2rm2vzgPu1bggumJ7sKgjaNsabmxKzv3YU3eMyvrj3LV4SqRDEQkpM7sJqHHOlZpZ8WCf33N+9sLCwoDmZ48Er84D7tW4ILpiu+Ph1UHb1pNLUoYVm5JzP/68wz+Eaobam2XUuxRYamY3AEnAWDN7Gv+87f6z5p7ztotICKnNuR8lH9dyXk4Kk7OSIx2KSEg55x5xzuU75wqA5cCfnHNfpu9520UkhJSc+3CirZP1u+t11izR7ofANWa2E7jGf19EQkyXtfvw7u462jq6uHKmxjdLdHHOlQAl/tv19DFvu4iEjs6c+1BSXsuY+FgWagiViIiEmZJzL5xzlJTXcsnULBLjNIRKRETCS8m5F7vrjrHvyHGKZ6q9WUREwk/JuRcl5bUAmiJSREQiQsm5FyXlNUwbn8qkTA2hEhGR8FNyPsPxtg7e231EZ80iIhIxSs5neHdXPW2dmoVKREQiR8n5DH8uryE5IZZPTdG0tSIiEhlKzj10dTn+VFbDJVOzNYRKREQiRsm5h/f2HOFgYyufnTsh0qGIiEgUU3Lu4eWNVaQmxnHtrHMiHYqIiEQxJWe/1vZOXttyiCWzz2FMgi5pi4hI5Cg5+72x/TAtJzv4wry8SIciIiJRTsnZ7+UNVUxMT2LReVmRDkVERKKckjNQ23ySd3bWsWxeHjExFulwREQkyik5A7/ffJDOLqdL2iIi4gkDJmcze8LMasxsa49lmWb2ppnt9P8e1+OxR8yswszKzey6UAUeTC9vPMDsvLFMz02LdCgiIiIBnTk/CSw5Y9nDwFvOuenAW/77mNksYDlwgf85PzczT3d93nm4mS0HGvn8vPxIhyIiIgIEkJydc+8AR85YvAx4yn/7KeBzPZY/55w76ZzbA1QAC4MUa0i8tPEAsTHG0rkTIx2KiIgIMPQ251znXDWA/3f3LBF5wP4e61X5l3lSV5fjlY0HuHx6NjlpiZEOR0REBIC4IG+vt67OrtcVze4G7gbIzc2lpKRkwI23tLQEtF6gyuo7OdjYytICF9TthlKw98FIpf0gIqPZUJPzYTOb4JyrNrMJQI1/eRUwqcd6+cDB3jbgnFsJrARYsGCBKy4uHvBFS0pKCGS9QK1+YTOpiYf42s1XkhTv6abxU4K9D0Yq7QcRGc2Gell7FXC7//btwCs9li83s0QzmwJMB94fXoihcaKtk9e2VHP97HNGTGIWEZHoMOCZs5k9CxQD2WZWBXwX+CHwvJndBewDbgFwzm0zs+eB7UAHcI9zrjNEsQ/LG9sPcaytky/MVy9tERHxlgGTs3NuRR8PXdXH+o8Djw8nqHB4eeMB8jLGcPGUzEiHIhJxZjYJ+C/gHKALWOmc+6mZZQK/AQqAvcAXnXNHIxWnSLSIygphNc2trNlZx7ILJ6pcp4hPB/Cgc+58YBFwj79uQa81DUQktKIyOa/a5C/XOd+zo7xEwso5V+2c2+C/3QyU4RsG2VdNAxEJoWAPpRoRXt54gKK8dKaNV7lOkTOZWQEwD3iPM2oamNn4Pp5zamhkTk6OZ4e5eXUInlfjguiK7cGijqBta7ixRV1y/vhwM9sONvHdz86KdCginmNmqcCLwNecc01mgTX79BwaWVhYGNDQyEjw6hA8r8YF0RXbHQ+vDtq2nlySMqzYou6y9ksbfOU6P6tynSKnMbN4fIn5GefcS/7Fh/21DDijpoGIhFBUJefOLscrmw5wxYwcslNVrlOkm/lOkX8JlDnn/qnHQ33VNBCREIqq5Lx+dz3Vja18XvM2i5zpUuA24DNmtsn/cwO+mgbXmNlO4Br/fREJsahqc35pwwHSEuO4ZlZupEMR8RTn3Fp6r40PfdQ0EJHQiZoz5xNtnby+tZobiiaoXKeIiHha1CTn7nKdn9fYZhER8bioSc4vbfCV61xYoHKdIiLibVGRnGuaWlmzs5bPzVO5ThER8b6oSM6rNh+ky8Hn52kGKhER8b6oSM4vbTjA3Px0po1PjXQoIiIiAxr1ybn8UDPbq5s0tllEREaMUZ+cX9pYRZzKdYqIyAgyqpNzZ5fjdxt95TqzVK5TRERGiFGdnN/dVc/hppN8Yb46gomIyMgxqpPzSxurSEuK46rze52CVkRExJNGXW1t5xwfVTXy3Af7eHVzNV+Yn6dynSIiMqKMmuTceKKd3208wHMf7Kesuomk+BiWXjiRB68tjHRoIiIigzKik7Nzjg/2HuW59/exeks1Jzu6uGDiWP7P52az7MKJjE2Kj3SIIiIigzYik3N9y0le3FDFcx/sZ3ftMVIT47j5onxWLJzM7Lz0SIcnIiIyLCMmOXd1ObbWdfLCMxt4Y/sh2jsd8ydn8OOb53DTnAkkJ4yYP0VERKRfIyKjfVTVwD2/3sD+I61kJNdx26ICli+cxIzctEiHJiIiEnQjIjmfm5nClOxUbszv4mu3XKne1yIiMqqNiHHO6cnx/NedC1k0MU6JWURERr0RkZxFRESiiZKziIiIxyg5i4iIeIySs4iIiMcoOYuIiHiMkrOI9MvMlphZuZlVmNnDkY5HJBqMiHHOIhIZZhYL/F/gGqAK+MDMVjnntg9nuwUPrw5GeIP2YFEHd/Tx2nt/eGOYoxHpm86cRaQ/C4EK59xu51wb8BywLMIxiYx65pyLdAyYWS1QGcCq2UBdiMPxOu0Dn2jdD+c653LC9WJmdjOwxDn3N/77twEXO+e+esZ6dwN3++/OBraGK8ZB8ur7xqtxgWIbqkBi6/Pz7InL2oF+2ZjZh865BaGOx8u0D3y0H8LGell21hG9c24lsBK8/b/xamxejQsU21ANNzZd1haR/lQBk3rczwcORigWkaih5Cwi/fkAmG5mU8wsAVgOrIpwTCKjnicuaw/CykgH4AHaBz7aD2HgnOsws68CfwRigSecc9sGeJqX/zdejc2rcYFiG6phxeaJDmEiIiLyCV3WFhER8RglZxEREY8ZEclZ5QN9zGyvmW0xs01m9mGk4wkXM3vCzGrMbGuPZZlm9qaZ7fT/HhfJGMUnkp/Vwb5PzOwRf5zlZnZdiGObZGZ/NrMyM9tmZvd7IT4zSzKz981ssz+u73khrjNijDWzjWb2qpdi6+37OKixOec8/YOvE8ou4DwgAdgMzIp0XBHaF3uB7EjHEYG/+3JgPrC1x7IfAw/7bz8M/CjScUb7T6Q/q4N5nwCz/PElAlP8cceGMLYJwHz/7TTgY38MEY0P3zj2VP/teOA9YFGk4zojxgeAXwOveux/etb3cTBjGwlnziofGOWcc+8AR85YvAx4yn/7KeBzYQ1KehPRz+og3yfLgOeccyedc3uACnzxhyq2aufcBv/tZqAMyIt0fM6nxX833v/jIh1XNzPLB24EftFjsSdi60PQYhsJyTkP2N/jfpV/WTRywBtmVuovlxjNcp1z1eD74gPGRzge8eZnta/3ScRiNbMCYB6+s9SIx+e/bLwJqAHedM55Ii6/fwb+FujqscwrsfX2fRy02EbCOOeAygdGiUudcwfNbDzwppnt8J8tiHjBSPqsRiRWM0sFXgS+5pxrMustDN+qvSwLSXzOuU7gQjPLAF42s9n9rB62uMzsJqDGOVdqZsWBPKWXZaH8n571fdzPuoOObSScOat8oJ9z7qD/dw3wMuG/ZOMlh81sAoD/d02E4xFvflb7ep+EPVYzi8eXmJ9xzr3ktficcw1ACbDEI3FdCiw1s734mkg+Y2ZPeyS2vr6PgxbbSEjOKh8ImFmKmaV13wauxbsz/4TDKuB2/+3bgVciGIv4ePGz2tf7ZBWw3MwSzWwKMB14P1RBmO8U+ZdAmXPun7wSn5nl+M+YMbMxwNXAjkjHBeCce8Q5l++cK8D3XvqTc+7LXoitn+/j4MUWyl52QewVdwO+3o27gG9HOp4I7YPz8PX22wxsi6b9ADwLVAPt+I5A7wKygLeAnf7fmZGOUz+R/awO9n0CfNsfZzlwfYhjW4zvMuZHwCb/zw2Rjg+YA2z0x7UVeNS/3BP7rcdrFvNJb+2Ix9bX93EwY1P5ThEREY8ZCZe1RUREooqSs4iIiMcoOYuIiHiMkrOIiIjHKDmLiIh4jJKziIiIxyg5i4iIeMz/B/RCuPPBFXJTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.485602855682373\n"
     ]
    }
   ],
   "source": [
    "algorithm = <>\n",
    "\n",
    "for i in range(15):\n",
    "    t = time.time()\n",
    "    algorithm.step(model)\n",
    "    print(time.time() - t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQcAAAC4CAYAAADuQCmzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAL/ElEQVR4nO3df6zddX3H8eerpRYCKCCXpr+m16yEgZmgDZpJFoI6OmdW/iEp2Zb+QdJ/WKLZkq3MZcNlJG5/mP0zljTT2GRO0kRNG2OyNVXixhxQFGxLqVyFwU0bWgSlKLbc2/f+uF/0WD7tPZR77vc0eT6Sk/P9fr7f7/m+7u3p63y/33PuvakqJOl0S/oOIGk8WQ6SmiwHSU2Wg6Qmy0FSk+UgqWlk5ZBkQ5JDSaaSbB3VfiSNRkbxOYckS4EfAB8DpoFHgDuq6okF35mkkRjVkcONwFRV/aiqTgL3AxtHtC9JIzCqclgNPDcwP92NSTpPXDCix01j7NfOX5JsAbYAXHzxxR+45pprRhRF0tk8+uijL1TVxOnjoyqHaWDtwPwa4PDgClW1DdgGsH79+tq7d++Iokg6myT/1xof1WnFI8C6JJNJ3gZsAnaNaF+SRmAkRw5VNZPkT4H/AJYCX6iqA6PYl6TRGNVpBVX1DeAbo3p8SaPlJyQlNVkOkposB0lNloOkJstBUpPlIKnJcpDUZDlIarIcJDVZDpKaLAdJTZaDpCbLQVKT5SCpyXKQ1GQ5SGqyHCQ1WQ6SmiwHSU2Wg6Qmy0FSk+UgqclykNRkOUhqshwkNVkOkposB0lNloOkpnnLIckXkhxNsn9g7Ioku5M81d1fPrDs7iRTSQ4luXVUwSWN1jBHDl8ENpw2thXYU1XrgD3dPEmuBTYB13Xb3Jdk6YKllbRo5i2Hqvo28OJpwxuB7d30duC2gfH7q+pEVT0NTAE3LlBWSYvoXK85rKiqIwDd/VXd+GrguYH1prsxSeeZhb4gmcZYNVdMtiTZm2TvsWPHFjiGpLfqXMvh+SQrAbr7o934NLB2YL01wOHWA1TVtqpaX1XrJyYmzjGGpFE513LYBWzupjcDOwfGNyVZnmQSWAc8/NYiSurDBfOtkOTLwM3AlUmmgb8FPgvsSHIn8CxwO0BVHUiyA3gCmAHuqqrZEWWXNELzlkNV3XGGRR85w/r3Ave+lVCS+ucnJCU1WQ6SmiwHSU2Wg6Qmy0FSk+UgqclykNRkOUhqshwkNVkOkposB0lNloOkJstBUpPlIKnJcpDUZDlIarIcJDVZDpKaLAdJTZaDpCbLQVKT5aBFdWp2htmTr3JqdqbvKJrHvL+aXloIr754mBee/C9OvHyMn/94mlUf+ARXXnNT37F0FpaDFsWJ4y9wdP83fzk/+9ovekyjYXhaIanJcpDUZDlIarIcJDVZDpKa5i2HJGuTfCvJwSQHknyyG78iye4kT3X3lw9sc3eSqSSHktw6yi9A0mgMc+QwA/x5Vf0W8CHgriTXAluBPVW1DtjTzdMt2wRcB2wA7kuydBThJY3OvOVQVUeq6rvd9HHgILAa2Ahs71bbDtzWTW8E7q+qE1X1NDAF3LjQwSWN1pu65pDk3cANwEPAiqo6AnMFAlzVrbYaeG5gs+luTNJ5ZOhySHIJ8BXgU1X18tlWbYxV4/G2JNmbZO+xY8eGjSFpkQxVDkmWMVcMX6qqr3bDzydZ2S1fCRztxqeBtQObrwEOn/6YVbWtqtZX1fqJiYlzzS9pRIZ5tyLA54GDVfW5gUW7gM3d9GZg58D4piTLk0wC64CHFy6ypMUwzA9efRj4E2Bfkse6sb8CPgvsSHIn8CxwO0BVHUiyA3iCuXc67qqq2QVPLmmk5i2Hqvpv2tcRAD5yhm3uBe59C7kk9cxPSEpqshwkNVkOkposB0lNloOkJstBiyJLljL4plfNzlD1hg/OaoxYDloUF0+8i2UXX/bL+Z8+uw8sh7FmOWhRZOkFJL96up2aOUnjR240RiwHSU2Wg6Qmy0FSk+UgqclykNRkOUhqshwkNVkOkposB0lNloOkJstBUpPlIKnJcpDUZDlIarIcJDVZDpKaLAdJTcP8OTxpKLOzs8zMzDSXnZo5+Wu/M/JUFSdPnoS0X5+WLVvGkiW+dvXJctCC2blzJ/fcc09z2fJlS/j7O97HOy+9EIAnDx7kj/7ug5w6w2+Ku++++7jppptGlFTDsBy0YF566SX27dvXXHbR8gs4/MrHmHrtdwjF7E93s2//fk6doR2OHz8+yqgawrzHbUkuTPJwkseTHEjymW78iiS7kzzV3V8+sM3dSaaSHEpy6yi/AJ0fTtVSnjp+A8dO/AZHT7yLJ17+IFVn+vvMGgfDnNSdAG6pqvcB1wMbknwI2Arsqap1wJ5uniTXApuA64ANwH1Jlo4ivM4vSzLD3G+cLpbkVN9xNI95Tytq7irSK93ssu5WwEbg5m58O/AA8Jfd+P1VdQJ4OskUcCPwnYUMrvNLmGVF/oeXX32Vx6aOcMHPHwQsiHE21DWH7pX/UeA3gX+uqoeSrKiqIwBVdSTJVd3qq4H/Hdh8uhs7o1deeYUHH3zwTYfXeJmamjrjsl+cnOGv/+WLwBd5+Wcn5n2sgwcP8va3v33hwulNG6ocqmoWuD7JZcDXkrz3LKu3TiTfcNUpyRZgC8CqVauYmJgYJorG2KWXXnrW5cOUwuve8Y53+Jzo2Zt6t6KqfpLkAeauJTyfZGV31LASONqtNg2sHdhsDXC48VjbgG0A69evr6uvvvoc4mucrFixYsEea9WqVfic6Ncw71ZMdEcMJLkI+CjwJLAL2NytthnY2U3vAjYlWZ5kElgHPLzQwSWN1jBHDiuB7d11hyXAjqr6epLvADuS3Ak8C9wOUFUHkuwAngBmgLu60xJJ55Fh3q34PnBDY/zHwEfOsM29wL1vOZ2k3vgJSS2YW265hR07dizIY91wwxtej7TILActmMnJSSYnJ/uOoQXij71JarIcJDVZDpKaLAdJTZaDpCbLQVKT5SCpyXKQ1GQ5SGqyHCQ1WQ6SmiwHSU2Wg6Qmy0FSk+UgqclykNRkOUhqshwkNVkOkposB0lNloOkJstBUpPlIKnJcpDUZDlIarIcJDVZDpKaLAdJTZaDpCbLQVJTqqrvDCQ5BvwMeKHvLKe5EjMNw0zDG8dc76qqidMHx6IcAJLsrar1fecYZKbhmGl445qrxdMKSU2Wg6SmcSqHbX0HaDDTcMw0vHHN9QZjc81B0ngZpyMHSWOk93JIsiHJoSRTSbYu4n6/kORokv0DY1ck2Z3kqe7+8oFld3cZDyW5dUSZ1ib5VpKDSQ4k+WTfuZJcmOThJI93mT7Td6aB/SxN8r0kXx+jTM8k2ZfksSR7xyXXOamq3m7AUuCHwHuAtwGPA9cu0r5/F3g/sH9g7B+Brd30VuAfuulru2zLgcku89IRZFoJvL+bvhT4Qbfv3nIBAS7pppcBDwEf6vt71e3rz4B/B74+Dv9+3b6eAa48baz3XOdy6/vI4UZgqqp+VFUngfuBjYux46r6NvDiacMbge3d9HbgtoHx+6vqRFU9DUwxl32hMx2pqu9208eBg8DqPnPVnFe62WXdrfrMBJBkDfAHwL8ODPea6SzGNddZ9V0Oq4HnBuanu7G+rKiqIzD3HxW4qhtf9JxJ3g3cwNwrda+5usP3x4CjwO6q6j0T8E/AXwCnBsb6zgRzxfmfSR5NsmWMcr1pF/S8/zTGxvHtk0XNmeQS4CvAp6rq5aS1+8XLVVWzwPVJLgO+luS9Z1l95JmSfAI4WlWPJrl5mE1GnWnAh6vqcJKrgN1JnhyTXG9a30cO08Dagfk1wOGesgA8n2QlQHd/tBtftJxJljFXDF+qqq+OSy6AqvoJ8ACwoedMHwb+MMkzzJ2K3pLk33rOBEBVHe7ujwJfY+40ofdc56LvcngEWJdkMsnbgE3Arh7z7AI2d9ObgZ0D45uSLE8yCawDHl7onWfuEOHzwMGq+tw45Eoy0R0xkOQi4KPAk31mqqq7q2pNVb2buefMN6vqj/vMBJDk4iSXvj4N/B6wv+9c56zvK6LAx5m7Kv9D4NOLuN8vA0eA15hr8DuBdwJ7gKe6+ysG1v90l/EQ8PsjynQTc4eV3wce624f7zMX8NvA97pM+4G/6cZ7/V4N7OtmfvVuRd//fu9h7t2Hx4EDrz+f+851rjc/ISmpqe/TCkljynKQ1GQ5SGqyHCQ1WQ6SmiwHSU2Wg6Qmy0FS0/8D5VFBcRRcnEoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_session(tmp_env, model, t_max=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task 3: Implement and compare the parallelized version of the algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = <>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last mean reward = 353.210\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAD4CAYAAADW+i6uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXRV5b3/8fc3EBKmMBMCAQLKDDIkIiogVUAUFLVO1AGXWtpaqy71Kg5V9JZe61XbXq/tr2itVL0oKhYUteKQgjgyQ5iHAIFAGIQkQCDD8/sjJ2mQBBJyztn75Hxea2XlnJ0zfHKSJ9/sfZ7nu805h4iIiPhHjNcBRERE5HgqziIiIj6j4iwiIuIzKs4iIiI+o+IsIiLiM/W9DgDQunVrl5KScsrbHTp0iMaNG4c+UDX5LQ8oU3VFaqbFixfvdc61CVOk01JxPPvxdQblqinlqrlaj2fnnOcfqamprjo+//zzat0uXPyWxzllqq5IzQQscj4Ysyf7qDie/fg6O6dcNaVcNVfb8azD2iIiIj6j4iwiIuIzKs4iIiI+44sJYZUpLCwkKyuLgoKC8m3NmjVjzZo1HqY6nt/yQPgzxcfHk5ycTGxsbNieU0SkrvNtcc7KyqJp06akpKRgZgDk5eXRtGlTj5P9m9/yQHgzOefYt28fWVlZdOnSJSzPKSISDXx7WLugoIBWrVqVF2bxHzOjVatWxx3dEBGR2vNtcQZUmCOAfkYiIsHn6+IsEgprd+WyLKfI6xgiIlVScT4JM+Omm24qv15UVESbNm0YN26ch6lCLyUlhb1793odIyRy8gq48aVv+MOSo3ycscvrOCIilfLthDA/aNy4MatWreLIkSM0bNiQefPm0aFDh7BmKCoqon790P2YQv34flJS4rhv5nLyCopIbmLc8+YyZt1xHj3bJXgdTUR8IGXy3KA91itjatdWVHvOp3DJJZcwd27pD2zGjBlMmDCh/GuHDh3i1ltv5eyzz2bgwIHMnj0bgMzMTIYNG8agQYMYNGgQX375JQDp6emMGDGCq6++mp49e3LDDTdQ2sHteCNGjODhhx/mggsu4I9//COLFy/mggsuIDU1lYsvvpjs7GxycnJITU0FYPny5ZgZ27ZtA+CMM87g8OHDvPfee5xzzjkMHDiQkSNHsnv3bgCmTJnCpEmTGD16NDfffDP79u1j9OjRDBw4kJ/97GflmQ4dOsTYsWPp378/ffv25c033wzRqxwe0xZsZsGGvTx+WR/uS4unSVx9bp++iP2HjnkdTUTkOBGxy/TEexms3plLcXEx9erVC8pj9m6fwOOX9Tnl7a6//nqefPJJxo0bx4oVK7j11ltZsGABAM888wwXXnghL7/8MgcOHGDw4MGMHDmStm3bMm/ePOLj49mwYQMTJkxg0aJFACxdupSMjAzat2/P+eefz8KFCxk6dOgJz3vgwAH+9a9/UVhYyAUXXMDs2bNp06YNb775Jo888ggvv/wyBQUF5ObmsmDBAtLS0liwYAEDBgygbdu2NGrUiKFDh/L1119jZrz00ks8/fTTPPvsswAsXryYL774goYNG3LXXXcxdOhQHnvsMebOncu0adMA+Oijj2jfvn35PycHDx4MymvvhWXbD/DMP9dxSd92TBjckX/9azN/uWkg1037ml+8tpjXbj+H2Hr6X1VE/CEiirOXzjrrLDIzM5kxYwaXXnrpcV/77LPP+Oijj3jmmWeA0uVf27Zto3379tx5550sW7aMevXqsX79+vL7DB48mOTkZAAGDBhAZmZmpcX5uuuuA2DdunWsWrWKUaNGAVBcXExSUhIA5513HgsXLmT+/Pk8/PDDfPTRRxw+fJhhw4YBpWvFr7vuOrKzszl27Nhxa5Evv/xyGjZsCMD8+fOZNWsWAGPHjqVFixYA9OvXj/vvv58HH3yQcePGlT9upMkrKOSuGUtJTIjnqavOKp9hPrBTC566qh/3zlzOlDkZTL2yn8dJRURKRURxLtvD9arpx+WXX879999Peno6+/btK9/unOOdd96hR48ex91+ypQpJCYmsnz5ckpKSoiPjy//WlxcXPnlevXqUVRU+azhslONOefo06cPX3311Qm3GTZsGAsWLGDr1q2MHz+e3/3udxQVFXHllVcC8Ktf/Yp7772Xyy+/nPT0dKZMmXLC45epbElU9+7dWbx4MR988AEPPfQQo0eP5rHHHqvqZfIl5xyPvLuKHQeOMPNnQ2jW6PhOZlcNSmbdrjz+Mn8zPZMSuGlIZ4+Sioj8m47jVcOtt97KY489Rr9+x+9ZXXTRRTz//PPl79EuXboUKD38m5SURExMDK+++irFxcWn/dw9evRgz5495cW5sLCQjIwMAIYPH85rr71Gt27diImJoWXLlnz88cecf/755TnKJrBNnz69yucYPnw4r7/+OgAffvgh33//PQA7d+6kUaNG3Hjjjdx///0sWbLktL8Pr7y9OIs5y3dyz0XdSO3cstLbPDCmJz/q0YYn5mTw5aa6OUtdRCKLinM1JCcnc/fdd5+w/YEHHqCwsJCzzjqLvn378utf/xqAO+64g+nTpzNkyBDWr19fq5OBN2jQgLfffpsHH3yQ/v37M2DAgPIJZmUntB8+fDgAQ4cOpVmzZuWHpadMmcI111zDsGHDaN26dZXP8fjjjzN//nwGDRrExx9/TKdOnQBYuXIlgwcPZsCAAUydOpVHH330tL8PL2zak89jszMY0rUld/zozCpvVy/G+OOEgaS0bswdry9h277DYUwpInIiq2y2cLilpaW5sglTZdasWUOvXr2O2+a3XtZ+ywPeZKrsZ1VR2Sz1cDpaVMyVL3xJ9sEjfHj3cNo1iz/u65Vlytx7iPEvLCQxIY5Zd5xPk7jwvutTndfJzBY759LCk+j0VBzPXvzsq0O5aiZacgV7KVVtxrP2nKVOeurDtazOzuW/r+5/QmGuSkrrxrzwk0Fs2nOIe95YRkmJ9/+4ikh0UnGWOufTNbv528JMbjkvhZG9E2t036HdWvPo2F58smY3z85bF6KEIiIn5+vZ2s45nVjB5/zwtkhFu3MLuP+t5fRKSmDyJT1P6zFuOS+FdbvyeOHzTXRPbMr4AeHtCici4ts95/j4ePbt2+e7P/7yb2Xnc664VMxLxSWOe95YRkFhCf/7k4HEx55ewxoz48nxfTk7pQUPvL2CFVkHgpzUv8ysnpktNbP3A9dbmtk8M9sQ+NzC64wi0cC3e87JyclkZWWxZ8+e8m0FBQW+KQTgvzwQ/kzx8fHlTVW89uf0jXy1eR9PX30WZ7RpUqvHalA/hj/fmMr4/13IpL8vZs6d59M2wV8/6xC5G1gDlDUcnwx86px7yswmB64/6FU4kWjh2+IcGxt7XEcrKJ2ZN3DgQI8SnchvecCfmcJh8db9/P6TDVzWvz3XpAbnn4XWTeKYdnMqV//5Kya9upg3Jg057b3xSGBmycBYYCpwb2DzeGBE4PJ0IB0VZ5GQ821xFqmug0cKuWvGMto3j2fqlX2DOk+hT/tmPHdtf37x+hIefnclz17Tvy7Pg/gD8ABQcS1eonMuG8A5l21mbSu7o5lNAiYBJCYmkp6eDkB+fn75ZT9RrpqJllz39Qveed5rm03FWSKac46HZ61kd24Bb/38XBLiY099pxq6pF8S94zsxh8+2UCvdgn8dHjXoD+H18xsHJDjnFtsZiNqen/n3DRgGpSucy5b3xkt62ODRblqJti5bgnzOueTUXGWiPbGd9uZuzKbB8f0ZGCn0M1VuuvCbqzblcd/fbiGMxOb8KMele5ARrLzgcvN7FIgHkgws9eA3WaWFNhrTgJyPE0pEiV8O1tb5FQ27M7jifcyGHpma34W4r3ZmBjj2Wv706NdAnf931I25uSH9PnCzTn3kHMu2TmXAlwPfOacuxGYA0wM3GwiMNujiCJRRcVZIlJBYTG/mrGUxg3q89y1/YmJCf37wI0a1OfFm1NpUD+Gn/59EQcPF4b8OX3gKWCUmW0ARgWui0iIVbs412T9o5k9ZGYbzWydmV0ciuAS3abOXcPaXXk8c23/sC5xSm7RiP93UypZ3x/mzhlLKCouCdtzh4tzLt05Ny5weZ9z7iLnXLfA5/1e5xOJBjXZcy5b/1imbP1jN+DTwHXMrDelh8X6AGOAP5lZ3V1/ImH30apdvPr1Vm4f2sWT937PTmnJb67oy4INe/ntB2vD/vwiUvdVqzhXWP/4UoXN4yld90jg8xUVtr/hnDvqnNsCbAQGByeuRLudB47w4Dsr6NehGQ+MOb32nMFw3dmduOW8FF5euIWZ3233LIeI1E3Vna1dk/WPHYCvK9wuK7DtOFWtizwZv62181seqNuZikscv/uugKPHSrih6zG+/GK+p5mGNXF81yqGh2at4GDWerq1qN0BIj/+7ETEG6cszqex/rGymTknNMiual3kyfhtrZ3f8kDdzvT7eetZ//0Gnru2P1cNql0XsGBlSh1yjCteWMhfMoqYfed5dGje0PNMIhL5qnNYu2z9YybwBnBhxfWPAD9Y/5gFdKxw/2RgZ9ASS1T6ZvM+nv9sA1cN7FDrwhxMzRs14KWJaRwtLOGn0xdx+FjwOgyJSPQ6ZXE+jfWPc4DrzSzOzLoA3YBvg55cosb3h45xz5vL6NSyEU9e0dfrOCc4s21T/mfCQNbsyuU/3lqhM6mJSK3VZp1zpesfnXMZwExgNfAR8EvnXHFtg0p0cs7x4Dsr2Jt/lOcnDKJJnD+b2v2oZ1smj+nJ3JXZPP/ZRq/jiEiEq9FfOudcOqVnpcE5tw+4qIrbTaX0zDYitfLa11v5ePVuHh3bi37JzbyOc1KThndl3a48npu3nu6JTRnTt53XkUQkQqlDmPjWmuxc/nPuGi7o3oZbz+9y6jt4zMz47VX96N+xOffOXMaa7FyvI4lIhFJxFl86cqy0PWezhrE8G6b2nMEQH1uPaTel0jS+PrdPX8S+/KNeRxKRCKTiLL705PsZbNqTz++vHUDrJnFex6mRxIR4pt2Uxt78o/zitSUcK6p7LT5FJLRUnMV35q7IZsa32/nZ8DMY2q2113FOS/+OzXn66rP4NnM/j89ZpRncIlIj/pz6KlFr+/7DTJ61ggEdm3Pf6O5ex6mV8QM6sHZXHn9O30SvpARuPjfF60giEiG05yy+UVhcwt1vLAUHz08YSGy9yP/1/I/RPRjZqy1PvLeahRv3eh1HRCJE5P/1kzrjD5+sZ8m2A0y9qh8dWzbyOk5QxMQYv79uAF1bN+aO15ewdd8hryOJSARQcRZf+HLjXv6Uvolr05K5vH97r+MEVdP4WF6amIYZ3D59EXkFhV5HEhGfU3EWz+3LP8o9by6jS+vGTLm8j9dxQqJzq8b86SeD2Lz3EPe8sYziEk0QE5GqqTiLp5xz/MfbKzhwuJDnJwykUYO6O0fxvDNb8/hlvfl0bQ7PfLzO6zgi4mN19y+hRIS/Lczks7U5TLmsN33a+7s9ZzDcNKRz+QzuHolNuWLgCac6FxHRnrN4Z9WOgzz14VpG9mrLxPNSvI4TFmbGlMv6MLhLSx54ZwXLtx/wOpKI+JCKs3ji0NEi7pqxlBaNY3n66v6YRUZ7zmBoUD+GP98wiLZN45j06iJ25xZ4HUlEfEbFWTzx+JwMtuw7xB+uG0jLxg28jhN2rZrE8eLNaeQVFDHp1cUUFOqsqiLybyrOEnazl+3g7cVZ3PmjMzn3jFZex/FMr6QEnrt2AMu3H+ChWSvV4lNEyqk4S1ht3XeIR95dRWrnFtx9UTev43huTN923DuqO+8u3cGHmVr/LCKlNFtbwuZYUQl3zVhKjMEfrx9A/TrQnjMYfnXhmazblcfG3TmUlLiIOT2miISO/jpK2Dz78TqWZx3kdz8+i+QWdaM9ZzCYGc9e2587B8apMIsIoOIsYTJ//R7+Mn8zPzmnE5f0S/I6ju/Ex9YjJopmrIvIyak4S8gdPOq4d+Zyuic24ddje3sdR0TE9/Ses4RUSYnjxZVHyStwvH77OTRsUM/rSCIivqc9Zwmp17/Zyqq9xfx6XG96tGvqdRwRkYig4iwhU1hcwp/TN9G9RQw3nNPJ6zgiIhFDxVlC5oOV2ew8WMAlXWKjqj2niEht6T1nCQnnHNPmb+aMNo3p30adr0REakJ7zhISX27aR8bOXH46rKuWCImI1JCKs4TEtPmbad0kTucrFhE5DSrOEnTrduXxr/V7uOW8zsTHaumUiEhNqThL0L24YDMNY+txwzmdvY4iIhKRVJwlqHYdLGD2sh1cm5ZMiyg8T7OISDCoOEtQvfJlJsUljtuGdvU6iohIxFJxlqDJP1rE699s5ZK+SXRqpbNORRIzizezb81suZllmNkTge0tzWyemW0IfG7hdVaRaKDiLEHzxrfbyCso4vZhXbyOIjV3FLjQOdcfGACMMbMhwGTgU+dcN+DTwHURCTEVZwmKwuIS/rYwk8EpLRnYSTtXkcaVyg9cjQ18OGA8MD2wfTpwhQfxRKKOOoRJUHywMpsdB47wxOV9vI4ip8nM6gGLgTOBF5xz35hZonMuG8A5l21mbau47yRgEkBiYiLp6ekA5Ofnl1/2E+WqmWjJdV+/oqA9Vm2zqThLrTnneHHBZrq2acyFPSv92y0RwDlXDAwws+bAu2bWtwb3nQZMA0hLS3MjRowAID09nbLLfqJcNRMtuW6ZPDdoj/XKmMa1yqbD2lJrX23ax6odgVadMWrVGemccweAdGAMsNvMkgACn3M8jCYSNVScpdamLdhM6yYNuFKtOiOWmbUJ7DFjZg2BkcBaYA4wMXCzicBsbxKKRJdTFufTWWJhZg+Z2UYzW2dmF4fyGxBvrduVR/q6PUw8N0WtOiNbEvC5ma0AvgPmOefeB54CRpnZBmBU4LqIhFh13nMuW2KRb2axwBdm9iFwFaVLLJ4ys8mULrF40Mx6A9cDfYD2wCdm1j3wfpbUMS8t2Ex8bAw3DlGrzkjmnFsBDKxk+z7govAnEolup9xzPo0lFuOBN5xzR51zW4CNwOCgphZf2J1bwD+W7eDatI5q1SkiEkTVmq1dwyUWHYCvK9w9K7Dth49Z6dKLk/HbdH6/5YHwZnpr3TGKih19Y3NO+pzR/jpVlx8ziYg3qlWca7jEorLpuq6Sx6x06cXJ+G06v9/yQPgy5R8t4q70T7mkXzuuvTTVF5lqQplExM9qNFu7mksssoCOFe6WDOysdVLxlTe/205uQRE/HaYTXIiIBFt1ZmvXdInFHOB6M4szsy5AN+DbYAcX7xQVl/DyF1s4O6WFWnWKiIRAdQ5rJwHTA+87xwAznXPvm9lXwEwzuw3YBlwD4JzLMLOZwGqgCPilZmrXLR+s2sWOA0eYoladIiIhccrifDpLLJxzU4GptU4nvuOcY9r8TXRt3ZiL1KpTRCQk1CFMauSrzaWtOm9Xq04RkZBRcZYaeXH+Zlo1bsBVg9SqU0QkVFScpdrW787j83V7mHieWnWKiISSirNUm1p1ioiEh4qzVEtObgH/WLqTa1I70lKtOkVEQkrFWarllS8zKSwp4fZhXbyOIiJS56k4yykdOlrEa19vZUyfdnRu1djrOCIidZ6Ks5xSeavO4WrVKSISDirOclJFxSX89YstpHVuwSC16hQRCQsVZzmpDwOtOidpr1lEJGxUnKVKpa06N9OldWNG9kr0Oo6ISNRQcZYqfb15Pyt3HOT2YV3UqlNEJIxUnKVKLy4obdX540HJXkcREYkqKs5SqQ278/hsbQ43n6tWnSIi4abiLJV6acEW4urHcNO5atUpIhJuKs5ygpzcAt5duoNr0pLVqlNExAMqznKC6V8FWnUO1fIpEREvqDjLcUpbdW7j4t7tSGmtVp0iIl5QcZbjzFy0nYNHCtWqU0TEQyrOUq5iq87UzmrVKSLiFRVnKfdRxi6yvj+ivWYREY+pOAtQ2qrzRbXqFBHxBRVnAeCbLftZnnWQ24Z2oZ5adYqIeErFWQB4cf5mWjZuwNWpatUpIuI1FWdhY04en67N4eZzO6tVp4iID6g4y79bdQ5Rq04RET9QcY5yOXkFzFpS2qqzVZM4r+OIiAgqzlHv719upbCkhNvUqlNExDdUnKPY4WNFvPr1Vkb3TqSLWnWKiPiGinMUm/ldaavOSWo6IiLiKyrOUaqouIS/LtxCaucWpHZu6XUcERGpQMU5Sv0zYzfb9x/hp8O01ywi4jcqzlHIOce0+ZtIadWIUb3VqlPAzDqa2edmtsbMMszs7sD2lmY2z8w2BD7rjCgiYaDiHIW+DbTqvH1YV7XqlDJFwH3OuV7AEOCXZtYbmAx86pzrBnwauC4iIabiHIVeXFDaqvPHg9SqU0o557Kdc0sCl/OANUAHYDwwPXCz6cAV3iQUiS71vQ4g4bUxJ59P1uRw90XdaNhArTrlRGaWAgwEvgESnXPZUFrAzaxtFfeZBEwCSExMJD09HYD8/Pzyy36iXDUTLbnu61cUtMeqbTYV5yjz1y82E1c/hpvPVatOOZGZNQHeAe5xzuWaVe9tD+fcNGAaQFpamhsxYgQA6enplF32E+WqmWjJdcvkuUF7rFfGNK5VNh3WjiJ78o7yzpIdXJ2qVp1yIjOLpbQwv+6cmxXYvNvMkgJfTwJyvMonEk1OWZxPZxanmT1kZhvNbJ2ZXRzKb0Cq7+9fZVJYXMJtQ7t4HUV8xkp3kf8KrHHOPVfhS3OAiYHLE4HZ4c4mEo2qs+dco1mcga9dD/QBxgB/MjO9uemxslado3ol0rVNE6/jiP+cD9wEXGhmywIflwJPAaPMbAMwKnBdRELslO85ByaDlE0IyTOzirM4RwRuNh1IBx4MbH/DOXcU2GJmG4HBwFfBDi/V99aiLA4cVqtOqZxz7gugqjeYLwpnFhGp4YSwas7i7AB8XeFuWYFtP3ysSmd3nozfZgz6LQ9UnqnEOf53/hHObB5DfuYK0jO9z+Q1ZRIRP6t2ca7BLM7KvuBO2FDF7M6T8duMQb/lgcozfbAymz1HlvCfPx7IiL5JvsjkNWUSET+r1mztGs7izAI6Vrh7MrAzOHGlppxz/GX+5kCrznZexxERkWqozmztms7inANcb2ZxZtYF6AZ8G7zIUhPfZX7P8u0HuE2tOkVEIkZ1DmuXzeJcaWbLAtsepnTW5kwzuw3YBlwD4JzLMLOZwGpKZ3r/0jlXHPTkUi3T5m+mRaNYrlarThGRiFGd2do1nsXpnJsKTK1FLgmCTXvy+WTNbu5Sq04RkYiiDmF12EsLtqhVp4hIBFJxrqNKW3Vm8ePUZFqrVaeISERRca6jXv1KrTpFRCKVzkpVBx05Vszfv97KyF6JnKFWnSJRL6WWZ1u6r19R+RmbMp8aG4xIcgrac66D3lq8nQOHC/mZWnWKiEQkFec6psQ5XlqwhYGdmpPaucWp7yAiIr6j4lzHLN5dzLb9h5k0rCsnabEqIiI+puJchzjn+HBLIZ1bNWJ0H7XqFBGJVCrOdch3md+z+WAJtw/toladIiIRTMW5jigqLmHKnAyaxxk/TlWrThGRSKbiXEf8bWEmq7NzubFXAxo10Ao5EZFIpuJcB2zff5jn5q1nZK+2pCaqh7aISKRTcY5wzjkem70KM3hifF/N0BYRqQN0/DPCfbByF5+v28OjY3vRoXlDNngdSESkGmrbtQz+3bmsLnYt055zBMstKGTKexn07ZDALeeleB1HRESCRHvOEezpj9ayL/8oL088m/r19H+WiEhdob/oEWrx1u95/Ztt3HJeF/olN/M6joiIBJGKcwQqLC7h4VkraZcQz72ju3sdR0REgkyHtSPQSwu2sG53Hi/enEaTOP0IRUTqGu05R5ht+w7zx0/Xc3GfREb1TvQ6joiIhICKcwRxzvHo7FXUj4nhicv7eh1HRERCRMU5gsxZvpP56/fwHxf3oF2zeK/jiIhIiKg4R4iDhwv5z/dX079jc24c0tnrOCIiEkKaTRQhnvpoDd8fLmT6rX11OkgRkTpOe84R4LvM/cz4dju3De1Cn/Za0ywiUtepOPvcsaLSNc0dmjfknpHdvI4jIiJhoMPaPjdt/iY25OTzt1vO1nmaRUSihP7a+9iWvYf4n882MvasJH7Us63XcUREfCkYZ7jyGx3W9innHI/+YyVx9WN4fFxvr+OIiEgYqTj71LtLd7Bw4z4eHNOTtgla0ywiEk1UnH1o/6Fj/GbuGgZ1as5PBnfyOo6IiISZirMP/dcHa8g9Ushvr+pHjNY0i4hEHU0I85mvNu3jrcVZ/GLEGfRsl+B1HBGpodpMTrqvXxG3VLh/5lNjgxFJIpD2nH3kaFExj7y7kk4tG3HXhVrTLOFjZi+bWY6ZraqwraWZzTOzDYHPLbzMKBJNVJx95E+fb2Lz3kP85oq+NGxQz+s4El1eAcb8YNtk4FPnXDfg08B1EQkDFWef2JiTz5/TNzF+QHuGd2/jdRyJMs65+cD+H2weD0wPXJ4OXBHWUCJRTMXZB5xzPPLuSuJjY3h0rNY0i28kOueyAQKf1QlHJEw0IcwH3lqcxTdb9vPUVf1o0zTO6zgiNWZmk4BJAImJiaSnpwOQn59fftlPQpnrvn5Fp33fxIbH3//512cHIxL39avd/SvmCtbrVpvXqcwPXy8/qe3v2CmLs5m9DIwDcpxzfQPbWgJvAilAJnCtc+77wNceAm4DioG7nHP/PO10UWBf/lF++8Eazk5pwbVpHb2OI1LRbjNLcs5lm1kSkFPVDZ1z04BpAGlpaW7EiBFA6R/ysst+Espct9RytvazK/23z1QxV+YNI4LymLV5ncr49fUCeGVM41r9jlXnsPYrVHOiiJn1Bq4H+gTu8ycz08ymk5g6dw2Hjhbx2yu1pll8Zw4wMXB5IhCc3TgROaVTFucaThQZD7zhnDvqnNsCbAQGBylrnfPFhr3MWrqDn19wBt0Sm3odR6KYmc0AvgJ6mFmWmd0GPAWMMrMNwKjAdREJg9M9HnDcRBEzK5so0gH4usLtsgLbTlDVe1Qn47f3r2qT51ix49cLj5DYyOhXbyfp6dmeZwoVZaoeLzM55yZU8aWLwhpEfK8ungHKj4J9sL6y47KushA7v6kAAAmTSURBVBtW9R7Vyfjt/ava5Hnmn+vYfXgjr99+Duef2doXmUJFmarHj5lExBunu5Rqd2CCCD+YKJIFVJzVlAzsPP14ddP63Xn8Zf4mrhrUIaiFWURE6obTLc5VTRSZA1xvZnFm1gXoBnxbu4h1S0mJ4+FZK2kSV59HLu3ldRwREfGh6iylmgGMAFqbWRbwOKUTQ2YGJo1sA64BcM5lmNlMYDVQBPzSOVccouwR6c1F21m09Xv+++qzaNVEa5pFROREpyzONZ0o4pybCkytTai6KievgP/6YA1Durbk6tRkr+OISAWa6CR+ovadYfSb99dQUFjC1Cv7YaY1zSIiUjkV5zBJX5fDnOU7+eWPzuSMNk28jiMiIj6m4hwGR44V8+vZqzijTWN+PqKr13FERMTn/NmUtI7546cb2L7/CG9OGkJcfXUzFRGRk9Oec4it3ZXLSws2c21aMud0beV1HBERiQAqziFUUuJ4aNZKmjWM5WGtaRYRkWpScQ6h17/dxtJtB3h0XC+aN2rgdRwREYkQKs4hsju3gKc/XMvQM1tzxYBKz/0hIiJSKRXnEHnyvdUcKy7hN1f01ZpmERGpERXnEPhs7W7mrszmrou6kdK6sddxREQkwqg4B9nhY0X8+h8ZdE9swk+HaU2ziIjUnNY5B9nv561nx4EjvP3zc2lQX//7iIhIzal6BNGqHQd5eWEmEwZ3Ii2lpddxREQkQqk4B0lxiePhd1fSolEDJo/p6XUcERGJYCrOQfLqV5msyDrIY5f1plmjWK/jiIhIBFNxDoLsg0f473+u44LubbjsrCSv44iISIRTcQ6CKXMyKHZOa5pFRCQoVJxr6eOMXfwzYzf3jOxOx5aNvI4jIiJ1gJZS1cKRIseTczLo2a4ptw3t4nUcERGpI1Sca2HWhmPsyi3ihRsGEVtPByFERCQ4VJyryTnHrtwCVu/MJWNnLhk7D/LJ1iJuHNKZQZ1aeB1PRETqEBXnShSXOLbsPUTGzoOs3pnL6uzSgrz/0DEAzKBLq8YM7VCfB8b08DitiIjUNVFfnAsKi1m3K698b3h1di5rs/M4UlgMQIN6MXRv14RRvRLp0yGB3kkJ9ExKoElcfdLT02karzXNIiISXFFVnA8cPnbcYenV2bls2nOI4hIHQNP4+vROSmDC4E70aZ9A7/YJnNm2id5PFhGRsKqTxdk5x44DR8oL8ersXFbvzGXHgSPlt2mXEE+f9gmM6dOO3u0T6NO+GcktGmqdsoiIeC7ii3NRcQmb9hxidfZBMnYECnF2LgcOFwKl7w93bd2Y1M4tuPnczvRuX3poulWTOI+Ti4iIVC6iivPRIseSbd+X7g0HJmut3ZXH0aISAOLqx9CzXVMu6ZsU2BtOoGe7pjRqEFHfpoiIRLmIqFqrdhzk7jeWsnnPYdwnXwLQrGEsfdonlO8N92nfjK6tG1Nf7w+LiEiEi4ji3LZpHF3bNKFvwjHGnncWfTo0o32zeL0/LBKhUibPDcrjZD41NiiPI+I3EbGb2TYhnhdvTuPKbg0Y3acdHZpr4paIiNRdEVGcRUREoomKs4iIiM+oOIuIiPhMREwIExGpzOlOLLuvXxG3BGlSmkgoaM9ZRETEZ1ScReSkzGyMma0zs41mNtnrPCLRQMVZRKpkZvWAF4BLgN7ABDPr7W0qkbpPxVlETmYwsNE5t9k5dwx4AxjvcSaROs+cc15nwMz2AFurcdPWwN4Qx6kJv+UBZaquSM3U2TnXJhxhAMzsamCMc+72wPWbgHOcc3f+4HaTgEmBqz2AdYHLfnydQblqSrlqrlbj2Reztav7x8bMFjnn0kKdp7r8lgeUqbqUqdoqa8V3wn/0zrlpwLQT7uzP70m5aki5aq622XRYW0ROJgvoWOF6MrDToywiUUPFWURO5jugm5l1MbMGwPXAHI8zidR5vjisXQMnHDbzmN/ygDJVlzJVg3OuyMzuBP4J1ANeds5l1OAhfPc9BShXzShXzdUqmy8mhImIiMi/6bC2iIiIz6g4i4iI+ExEFGe/tQ80s5fNLMfMVnmdpYyZdTSzz81sjZllmNndPsgUb2bfmtnyQKYnvM4EpV2vzGypmb3vdZYyZpZpZivNbJmZLfI6T215PWYrG6Nm1tLM5pnZhsDnFhW+9lAg6zozuzhEmSodoz7IVek49TpXhec6brz6KNcJYzao2Zxzvv6gdBLKJqAr0ABYDvT2ONNwYBCwyuvXp0KmJGBQ4HJTYL0PXicDmgQuxwLfAEN88FrdC/wf8L7XWSpkygRae50jSN+L52O2sjEKPA1MDlyeDPwucLl3IGMc0CWQvV4IMlU6Rn2Qq9Jx6nWuCvmOG68+ynXCmA1mtkjYc/Zd+0Dn3Hxgv5cZfsg5l+2cWxK4nAesATp4nMk55/IDV2MDH57OQDSzZGAs8JKXOeo4z8dsFWN0PDA9cHk6cEWF7W84544657YAGyn9HoKdqaox6nWuqsapp7mgyvHqea6TCFq2SCjOHYDtFa5n4XHR8TszSwEGUvofsKcCh6SWATnAPOec15n+ADwAlHic44cc8LGZLQ60woxkfh2zic65bCgtlEDbwPaw5/3BGPU8VxXj1PNcVD5e/ZALKh+zQcsWCcW5Wu0DpZSZNQHeAe5xzuV6ncc5V+ycG0BpZ6nBZtbXqyxmNg7Icc4t9irDSZzvnBtE6dmffmlmw70OVAuRNmbDmrcGYzRsuWo4TsOS6zTGa7h/72oyZmucLRKKs9oHVpOZxVI66F93zs3yOk9FzrkDQDowxsMY5wOXm1kmpYdaLzSz1zzMU845tzPwOQd4l/Afjgsmv47Z3WaWBBD4nBPYHra8VYxRz3OV+cE49TpXVePV61xAlWM2aNkioTirfWA1mJkBfwXWOOee8zoPgJm1MbPmgcsNgZHAWq/yOOcecs4lO+dSKP09+sw5d6NXecqYWWMza1p2GRgN+GYlwGnw65idA0wMXJ4IzK6w/XozizOzLkA34NtgP/lJxqjXuaoap57mOsl49TQXnHTMBi9bqGayBXlW3KWUzmzcBDzigzwzgGygkNL/iG7zQaahlB4mWQEsC3xc6nGms4ClgUyrgMe8fp0qZBuBT2ZrUzqreXngI8MPv+NB+J48HbOVjVGgFfApsCHwuWWF2z8SyLoOuCREmSodoz7IVek49TrXDzKWj1c/5KpqzAYzm9p3ioiI+EwkHNYWERGJKirOIiIiPqPiLCIi4jMqziIiIj6j4iwiIuIzKs4iIiI+o+IsIiLiM/8fvr1UjG+IErkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26.267060041427612\n"
     ]
    }
   ],
   "source": [
    "algorithm = <>\n",
    "\n",
    "for i in range(15):\n",
    "    t = time.time()\n",
    "    algorithm.step(model)\n",
    "    print(time.time() - t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Environment 2: LunarLanderContinuous-v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last mean reward = 155.350\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe8AAAD4CAYAAADFLW5aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9b3/8dcnG2EJKFtk0yCisskWcccoVFBo0Xtd8KroD73YVqtV/LVUreVXr7fW625blboUd71aq1WrLBr3jX2RsqgRAhQQEBIg+/f3x0zGhMwkE2Y5s7yfj0ceOdvMvHOSzGfO93zP95hzDhEREUkeGV4HEBERkdZR8RYREUkyKt4iIiJJRsVbREQkyah4i4iIJJksrwOEq2vXrq6goKDZbfbs2UP79u3jEyhMyhQeZQpPOJkWLlz4rXOuW5witVo4/8vRkoi/w/0lekbli0wk+Zr9X3bOJcXXyJEjXUveeeedFreJN2UKjzKFJ5xMwAKXAP+zob7C+V+OlkT8He4v0TMqX2Qiydfc/7KazUVERJKMireIiEiSUfEWERFJMknTYS2Y6upqSktLqaioAKBTp06sWrXK41SNpXum3NxcevfuTXZ2dlxeT0QkHSR18S4tLSUvL4+CggLMjLKyMvLy8ryO1Ug6Z3LOsX37dkpLS+nbt2/MX09EJF0kdbN5RUUFXbp0wcy8jiJBmBldunQJtIyIiEh0JHXxBlS4E5x+PyIi0Zf0xVskHfxt8UZ2V1R7HUNEEoSKd4TMjEsuuSQwX1NTQ7du3Zg4caKHqWKvoKCAb7/91usYaWH1v8r4+fNLOGbmHLbvq/M6jogkgKh0WDOzx4CJwFbn3GD/spnAfwLb/Jvd6Jx7w7/uV8DlQC1wjXPurWjk8EL79u1ZsWIF+/bto23btsydO5devXrFNUNNTQ1ZWbHrexjr55fm7a2qCUxnZeg0hHyvYMbrUXuuktsnRO25JPaideT9F2B8kOX3OOeG+b/qC/dAYDIwyP+YP5lZZpRyeOLMM8/k9dd9/0TPPvssF154YWDdnj17mDp1KsceeyzDhw/nlVdeAaCkpIRTTjmFESNGMGLECD766CMAiouLKSoq4txzz+Xoo4/moosuwjdKXmNFRUXceOONnHrqqdx3330sXLiQU089lZEjRzJu3Dg2b97M1q1bGTlyJABLly7FzFi/fj0A/fr1Y+/evfz973/nuOOOY/jw4YwdO5YtW7YAMHPmTKZNm8YZZ5zBlClT2L59O2eccQbDhw/nyiuvDGTas2cPEyZMYOjQoQwePJjnn38+Rns5fc1ftTUwnZfjYRARSRhROZxyzr1nZgVhbj4JeM45Vwl8bWbrgFHAx5Fk+H9/X8nyDTvJzIze54CBPTvymx8OanG7yZMn89vf/paJEyeybNkypk6dyvvvvw/AnXfeyemnn85jjz3Gd999x6hRoxg7dizdu3dn7ty55ObmsnbtWi688EIWLFgAwOLFi1m5ciU9e/bkpJNO4sMPP+Tkk09u8rrfffcd7777LtXV1Zx66qm88sordOvWjeeff56bbrqJxx57jIqKCnbv3s37779PYWEh77//PsOGDaN79+60a9eOk08+mU8++QQz45FHHuGOO+7grrvuAmDhwoV88MEHtG3blmuuuYaTTz6ZW265hddff51Zs2YB8Oabb9KzZ8/Ah5ddu3ZFZd/L9zIaHG1nqAOgiBD767yvNrMpwAJgunNuJ9AL+KTBNqX+ZU2Y2TRgGkB+fj7FxcWN1nfq1ImysjIAqquqcc5RW1sbtfDVVdWB529O3759+eqrr3j88ccZO3Yse/fupaamhrKyMubPn88bb7zBHXfcAcC+fftYtWoVhxxyCDfccAPLly8nMzOTdevWUVZWxt69exk5ciSdOnViz549DBo0iFWrVjF06NBGr1lbW8sPf/hDysrK+OKLL1ixYgVjxowJrMvPz6esrIxjjz2WuXPn8vbbb3Pdddcxb948ysvLOe644ygrK2P16tXceOONbNmyhaqqKg477DDKysqorKxk3LhxgZ+juLiYp556irKyMkaPHs1BBx1EeXk5ffv2Ze7cuVx33XWMHz+eE088sck+q6ioaPK72195eXmL28Sb15mccyzaWssDiysTJpOIJIZYFu8HgVsB5/9+FzAVCHbo0LRdGHDOzQJmARQWFrqioqJG61etWhUYbOS//n2YZwOi5OXlcfbZZ3PzzTdTXFzM9u3bycrKCmR5+eWXOeqooxo9ZubMmfTu3ZtnnnmGuro6cnNzycvLo127drRr1y7w2NzcXLKzs5v8XJmZmXTr1i3wmEGDBvHxx00bL04//XQWLlzIxo0bmTx5Mvfffz9mxjnnnENeXh4zZszg+uuv50c/+hHFxcXMnDmTvLw82rRpQ4cOHQKvm5GRQV5eXmDezOjQoQMFBQUsWrSIN954g1tvvZUzzjiDW265pVGG3Nxchg8f3uw+rD9dkEi8znTra1/w6OKvA/PPXHEcVaUrEm4/iUj8xay3uXNui3Ou1jlXB/wZX9M4+I60+zTYtDewKVY54mXq1KnccsstDBkypNHyMWPG8MADDwTOES9evBjwNS/36NGDjIwMnnzyyYhaDI466ii2bdsWKN7V1dWsXLkSgNGjR/PUU0/Rv39/MjIy6Ny5M3PmzOGkk04K5KjvYDd79uyQrzF69GiefvppAP7xj3+wc+dOADZt2kS7du24+OKLueGGG1i0aNEB/xzS2KMffN1o/sQjunqUREQSTcyKt5n1aDB7DrDCP/0qMNnM2phZX6A/8FmscsRL7969ufbaa5ss/8UvfkF1dTXHHHMMgwcP5te//jUAP/3pT5k9ezbHH388a9asiehm8jk5Obz44ov88pe/ZOjQoQwbNizQAa6goADwFV+Ak08+mU6dOnHwwQcDvhaA8847j1NOOYWuXUMXh9/85je89957jBgxgjlz5nDooYcCsHz5ckaNGsWwYcO47bbbuPnmmw/45xARkfBE61KxZ4EioKuZlQK/AYrMbBi+JvES4EoA59xKM3sB+AKoAa5yzkXvRHWclZeXN1lWVFQUaNps27YtDz/8cJNt+vfvz7JlywLzv/vd75o8FuAPf/hD0Nfd/7znsGHDeO+994JuW9/DHODGG2/kZz/7WWB+0qRJTJo0qcljZs6c2Wi+S5cuzJkzJzB/zz33ADBu3DjGjRsX9HVFRCQ2otXb/MIgix9tZvvbgNui8doiIiLpRiOsiSQBjc0iIg0lffEONoCJJA79fg5MZU3jM0mf3zTWoyQikoiSunjn5uayfft2FYgEVX8/79zcXK+jJJ0tuyobzXfp0MajJCKSiJJ6wOrevXtTWlrKtm2+4dMrKioSrlCke6bc3Fx69+4dl9dKJTv2VgWmH75kpIdJRCQRJXXxzs7Opm/fvoH54uLiFgcDiTdlkgNx9h8/DEyPG3SIh0lEJBEldbO5SCrSaSARaYmKt0iCqazRPbtFpHkq3iIJZl9V0o5ZJCJxouIt4rHaOsdlj3/Gp19tB2BftYq3iDQvqTusiaSCLbsrKF69jeLV23j00kI2fbfP60gikuBUvEU8UlZRTbucrEZH2pfPXuBhotDMrA/wBHAIUAfMcs7dZ2Yzgf8Etvk3vdE594Y3KUXSh4q3iAeccwyZOYfzC3sz5YQCr+OEowaY7pxbZGZ5wEIzm+tfd49z7k4Ps4mkHZ3zFvFAfY/ylxZtZE9ljcdpWuac2+ycW+SfLgNWAb28TSWSvnTkLeKB9Tv2ApCdaZSHKN4Tj+mRkEflZlYADAc+BU4CrjazKcACfEfnO4M8ZhowDSA/P7/JLW1jpby8PG6vdaAiyTh9SPQ++IXKkOj7MF3zqXiLeOCHD3wAQHZGRsjiPeWEAkb17RzPWC0ysw7AS8DPnXO7zexB4FbA+b/fBUzd/3HOuVnALIDCwkLX8J71sVRcXEy8XutARZLxshmvRy1HyUXBMyT6PkzXfGo2F/FAfbN5WWVNyOJ9bMHB8YzUIjPLxle4n3bO/RXAObfFOVfrnKsD/gyM8jKjSLpQ8RbxwCEdfTeG6dO5LeUVTYv3fZOHYZY4N/E2X5hHgVXOubsbLO/RYLNzgBXxziaSjlS8RTxwYr8uAJx8RLcmR94PXTySScMSri/YScAlwOlmtsT/dRZwh5ktN7NlwGnAdZ6mFEkTOuct4oGqWl+z+bOfrW+yrk1W4n2mds59AARrCtA13SIeSLx3CZEUt2LjLl5btjnk+pwELN4iklii8i5hZo+Z2VYzW9FgWWczm2tma/3fD26w7ldmts7MVpvZuGhkEEkWVz+zqNn1WRmJc65bRBJTtD7i/wUYv9+yGcB851x/YL5/HjMbCEwGBvkf8yczy4xSDpGEl5UZ/N+uc/scgITqqCYiiSkqxds59x6wY7/Fk4DZ/unZwNkNlj/nnKt0zn0NrEOXl0iKW799L5+X+P5F1m0tD7pNv27t4xlJRJJYLE+u5TvnNoNvaEWgu395L2BDg+1K0TCLkuJG/887nPfQxyHX/+cpfeOYRkSSnRe9zYO1CbqgG7ZySMVEHCZPmcKTLplCPd/ebRupLPfdXWzZ0sXs/Sb4maRE3E8iEn+xLN5bzKyHc26zfyCHrf7lpUCfBtv1BjYFe4LWDqmYiMPkKVN4Uj7Tm75hLBdX9wTWNll99FH9+fm/9+DZz9Zz+elHhDzvnYj7SUTiL5bN5q8Cl/qnLwVeabB8spm1MbO+QH/gsxjmEEkY983/vnA3vJ47KzODbnltuGZMf3VYE5EWReXI28yeBYqArmZWCvwGuB14wcwuB9YD5wE451aa2QvAF/juEXyVc642GjlEkklOVkZgjPNMXR4mIq0QleLtnLswxKoxIba/DbgtGq8tkqwaFmxd2y2poiCadzq7fULUnivVaCgnEY/8tKhfYDrUtd8iIsHoHUPEAyf268KUEwoC8z075XoXRkSSjm5MIuKB/zvuKHKzM1l325ks3vAdhQWdvY4kIklER94iHhjYsyPgay4/VoVbRFpJxVskzi4o7EObLA3nLyIHTs3mIjH0z3/t5tuyqkbL8nV+W0QipOItEkPj732/ybJ2OTrqFpHIqNlcJM5UvEUkUireInHWNlvFW0Qio+ItEmcFXXXfbhGJjIq3SJx1zM32OoKIJDkVb5E400ioIhIpvY2IxFmGbvkpIhHSpWIicabbf0oiCnU3sOlDargsincKk+jQkbdInOnIW0QipeItEmc68haRSKl4i8RIybd7gi5X8RaRSKl4i8TI/W+vDbpczeYiEikVb5EYCTWSWjIeeZtZHzN7x8xWmdlKM7vWv7yzmc01s7X+7wd7nVUkHah4i8RIqCPszOQ88q4BpjvnBgDHA1eZ2UBgBjDfOdcfmO+fF5EYU/EWiZFQB9gZSfhf55zb7Jxb5J8uA1YBvYBJwGz/ZrOBs71JKJJeYn6dt5mVAGVALVDjnCs0s87A80ABUAKc75zbGessIvFkoY68k7DZvCEzKwCGA58C+c65zeAr8GbWPcRjpgHTAPLz8ykuLo5L1vLy8ri91oGKJOP0ITXRDRNEftv4vE4w4eyXRP8dxypfvAZpOc05922D+fqmttvNbIZ//pdxyiISF6Fax5O5w5qZdQBeAn7unNsd6gPK/pxzs4BZAIWFha6oqChmGRsqLi4mXq91oCLJGI/BU6YPqeGu5d6M51VyUVGL2yT67zhW+bxqwFNTm6S8kOe8k/TI28yy8RXup51zf/Uv3mJmPfzrewBbvconkk7i8XHKAXPMzAEP+z+Bx6SpLRGbT5QpPKmYqbS0Mujy999794CPvr3aT+Y7xH4UWOWcu7vBqleBS4Hb/d9fiXs4kTQUj+J9knNuk79AzzWzf4b7wNY2tSVi84kyhScVM320dxWUfNVk+emnneZZpgicBFwCLDezJf5lN+Ir2i+Y2eXAeuA8L8KJpJuYF2/n3Cb/961m9jIwCn9Tm/+oW01tknK2lVWyt8qbTj6x4Jz7AAjVXDAmnllEJMbnvM2svZnl1U8DZwAr+L6pDdTUJinmu71VHHvbPJ76ZL3XUUQkRcX6yDsfeNnfIzULeMY596aZfY6a2iRFrdlS7nUEEUlxMS3ezrmvgKFBlm9HTW2Souqc8zqCiKS4JBzrSSSxqXaLSKypeItEmVP1FpEYU/EWibK6Zmr3sD4HxS+IiKQsFW+RKGvunPf//viEOCYRkVSl4i0SReWVNTz6wdch12dn6l9ORCKndxKRKPrNKyt5d802r2OISIpT8RaJog0793odQUTSgIq3SBRVVNd6HUFE0oCKt0grrNi4i3f+GXoo/ura0J3VTunfNRaRRCQNeXOHdZEkNfGBDwAouX1C0PVVNcGPvD+ccTpd2ufELJeIpBcVb5EoCnXk3eugtnFOIiKpTM3mIgfgyicXcP/8tfxrV0Wj5VU1dR4lEpF0ouItEqbaBkOnvbVyC3fPXcMVT3zeaJtRfTvHO5aIpCEVb5Ew1NY5bn3tiybLd+2rDkzX1TleXbopnrFEJE2peIuE4YG31/KXj0qaLDcsMP319j1xTCQi6UzFW6QFdXWOe+etDb7OObaXVwLw2tLN8YwlImlMvc1FQnhoaQW3LXqXtVvLQ25TunMfI/9rHg9fMpJ75q2JYzoRSWcq3iIhfLK5FghduBu68smFsQ0jItKAms1Fgli1ebfXEUREQlLxFgli/qotXkcQEQnJs+JtZuPNbLWZrTOzGV7lEAlGt/UUkUTmSfE2s0zgj8CZwEDgQjMb6EUWSW8frP2WFRt3NVn+eclOD9KIiITHqyPvUcA659xXzrkq4DlgkkdZJI1d/OinTHzgAxavj12x1t3ERCTavOpt3gvY0GC+FDhu/43MbBowDSA/P5/i4uJmn7S8vLzFbeJNmcLjdaZz/vQRPxvehsFdM3liZVVUn/vyfvui9rN5vZ9EJDF4VbwtyLImt2Nyzs0CZgEUFha6oqKiZp+0uLiYlraJN2UKjxeZnHPw5huB+QcWVzZa3yEbyqv3f1R4Ftw8lsL/mgcQ1Z8rEX93IhJ/XjWblwJ9Gsz3BjQotMRVVW3zdwA7kMKdYfD5TWPJy029IRTM7DEz22pmKxosm2lmG81sif/rLC8ziqQLr4r350B/M+trZjnAZOBVj7JImnDOsXTDd1TW1AKh770difML+9Atrw1ZGSl5FeZfgPFBlt/jnBvm/3ojyHoRiTJPDg+cczVmdjXwFpAJPOacW+lFFkkfz3++gRl/XQ7AtWP6U1FdG/XX6Nw+B/Adgaca59x7ZlbgdQ4R8XB4VP8ndH1Kl7j5rGRHYPq++cFvNBKpa8f2B8AsBat3aFeb2RRgATDdOdek635rO59GSzJ08Isk4/QhNdENE0R+2/i8TjDh7Jdw9t/yIJeDHoghvTq1+jGx+htMvRNzIiHs3neAvc9aoU1WZsxfI8E8CNyKr8PprcBdwNT9N2pt59NoSYYOfpFkvGzG69ENE8T0ITXctdybUlFyUVGL24Sz/6K1n8LJs79Y/Q2m5Ik5kf2V7tzLvFVbI3qOPp3bRilN6nDObXHO1Trn6oA/4xvDQURiTMVb0sKMl5ZH/BznjujT8kZpxsx6NJg9B1gRalsRiR4Vb0lJzjnunbeGr7/dA8AH676N+Dk7tm190+HFxx8a8esmCjN7FvgYOMrMSs3scuAOM1tuZsuA04DrPA0pkiZ0zltSytbdFdz48nL+77ijuXfeWu6dt5bRR3aL+HmzM40Lju3D68s2s+Cb4EOptslq/Fm45PYJEb9uInHOXRhk8aNxDyIiKt6SWkb993wA1u/YG1j23gHeISy/nbFlr+9a8EW//gHtcrL4/bnHMOaud4Nu//GvxhzQ64iItJaazSUlrdlSHvFzXDcyNzCdl5sN+IdUBQ7v1r7J9vXXeIuIxJqKt6S1rh3ahFzXJshVX3X+Qdky0us6bhFJMGo2l5SwZksZry1t/fD4mc18fA1Wnmv91TtTxVtEPKQjb0kJ5z30Mfe/va7Rsn8b0Ssw/ft/HxL0cZ3aZod8zmCjpGX7q31+p9wm60RE4kXFW1LCriCjp919/rDAdLAe578cfzT/c+7QRssO79qeHH+v8WDjkx/RvQN3nHsM910wrOlKEZE4UfGWtNAuu+kZop8U9WvSyWxwr06BJvFQDePnF/bhYHVOExEPqXhL0jv592+3uE3bnEyGH3pQk+X731nsqEPyqG8tD/e09kMXj+BvV50U3sYiIlGg4i1JbW9VDaU79wGQmx36zzknK4MHLhzeZHm/bh247MQCwNdMfuXow3GBHuXhZRg36BCG9Wn6wUBEJFbU21yS2vbyqsB0phnzrh/N2LvfC/vxGRnGzB8N4qYJA8g0I6NBxTbgrZ+PprKm+ft+p9ntP0XipiCMu4FNH1ITl7urJRodeUtS213xfUe1OgdHdM9rtP7mCQMY1bezb31d6OfJzsxoVLjBV7yPOiSPY3oHP6qeelLfAwstIhIhHXlLUttb9f1RcU2Q6nzFKYdzxSmHA1Bb3x4eJbf8cCC3/HBgVJ9TRCQcOvKWpLZzz/fN5tW1zRfncM9hz546iknDepITZIQ1EZFEoOItSW3akwsD02cNOaTZbQ/r0p6ZYRwpj+rbmfsmD9e5bBFJWCrekhJOPbIb901u2pt8f1NOKIh9GBGRGItZ8TazmWa20cyW+L/OarDuV2a2zsxWm9m4WGWQ1OYanMO+54JhgaFLm6ODaRFJBbHusHaPc+7OhgvMbCAwGRgE9ATmmdmRzrnmr8cR2c/v31wdmG4X5glqNYWLSCrwotl8EvCcc67SOfc1sA4Y5UEOSWJVNXU89O6Xgfk2WToDJCLpI9bveFeb2TIze8zMDvYv6wVsaLBNqX+ZSNhOu7O40byOqEUknUTUbG5m84BgXXxvAh4EbgWc//tdwFSC3+8h6DU+ZjYNmAaQn59PcXFxs3nKy8tb3CbelCk8rc208bt9jeaDPTbU8114dA4DOmfo70lEklZExds5Nzac7czsz8Br/tlSoE+D1b2BTSGefxYwC6CwsNAVFRU1+zrFxcW0tE28KVN4WpOporoW3nyz0bJGj33z9abLGm4bg0zxkoiZRCT+YtZhzcx6OOc2+2fPAVb4p18FnjGzu/F1WOsPfBarHJJ6/rWrotn1V44+nH3V6v8oIqkrlr3N7zCzYfiaxEuAKwGccyvN7AXgC6AGuEo9zaU1Lnrk00bz900e1mj+V2cNiGccEZG4i1nxds5d0sy624DbYvXaktr2P9996pHdPEoicmAa3i0rXe+KJZHR9TWS1B68aAQHtcvxOoaISFypeEvSyWxwh5Ezh/TwMImIiDdUvCWpVFTXUlsX3Vt7iogkG93PW5LKstJdAHRun8NBbbM9TiMi4g0Vb0kqe6pqAHjk0kJGHHpwC1tLNJnZY8BEYKtzbrB/WWfgeaAA31Ul5zvndnqVUSRdqNlcksq+Kt9Vhe1z9LnTA38Bxu+3bAYw3znXH5jvnxeRGFPxlqSyp9J35B3uXcQkepxz7wE79ls8CZjtn54NnB3XUCJpSocvklTqR05T8U4Y+fUjKTrnNptZ92AbtfY+BdGSqGPBTx9SE5jOb9t4PtEo3/cO5G8pVn+DKt6SVP66aCMA7dRsnlRae5+CaEnUseAv22+QlruWJ+7fs/J9r+SiolY/JlZ/g2o2l6SxavNulmz4DoDcbP3pJogtZtYDfPczALZ6nEckLegdUJLGt+WVgWndvzthvApc6p++FHjFwywiaUPFW5LG8o27vI6Q1szsWeBj4CgzKzWzy4HbgR+Y2VrgB/55EYmxxD2RIbKfLbsqyMvNYsktZ3gdJS055y4MsWpMXIOIiIq3JIe756xm9sffAI3HNhcRSUcq3gJAbZ0jwxLrXHJVTR0VNbVkZRj3v73O6zgiIglD57yTWHVtHbv2VUf8POWVNfS78Q0eeverwLKde6qoqa3jvnlrWbS++dEuK2tqca75m4Vs2LGXNVvKwsqzc08VAJfP/pxjZs7h139bGVj3x/8YEdZziIikMh15J6nzHvqIz0t8RfXr353V4hFzTW0ddQ7umruazvtqKWqwbtE3vuf5/Zv/pKqmjrEDuzPh/g8C6++Zt4ZZl4zkjEGHBJZ99vUOdu+rZuzAfEb8di57qmp57LJCTj86n6qaOnburSK/Y25g+1PueAeAktsnUFfnqKlzVNbUUlcHndr5bjBy/QtL2F5exbtrtpGXm0VZhW/ghZcWlQLwzBXHceIRXQ9sh4mIpBAV7yhyzvH0p+sZOyCfQzrlNrvt1rIKuuc13mZPZQ11zpGX6ytm67aW8cuXljPrkpF06dCm0bb1hRtgw459dMjNYltZJXXOMaBHRwBWbtpFVkYG/bq150d/+JAvNu8OPOZ3n73OOcN7cce5xzDlsc8Cy++Zt4aS7Xua5J325EKemDqKBd/s5OXFpWzYsQ+AnxT1Y49/vPGpf1nAn6cU8p9PLADgipP7ctVpR/Da8s2B57l77hrun7+20XOX3D6BfTUuMAALECjc9Tq3z1HhFhHxS6viXVlTyx/eXsdPivrRLieLnXuqePKTb7jqtCMadYJ6c8W/6JibFXaxqK1z/PqVFby14l9s31PFzX9bwZ8uGkGX9jnU1DmOmfkWVbV1PHX5cby3Zlvg/O0f/mM4f3h7Hf/8VxmPX3YsP316Efuqaym5fQIAt7yykoXf7GTq7AXkZBqfl+zk7vOHMqpv50avP/p/3mk0P+GYHrTNzuTFhb4j1uGHHtSocNd7efFGXl68MejyYBoW+XoPFn/ZaL6+cAM88sHXPPLB143W71+4AQoajDYVyg5/U7qIiKRZ8X720/U88PY6zIzrf3Akk/74Iet37OXuuWsAuG/yMBZ+s5Mn/L2af3b6EYwZkM+wPgcB8NW2cg5ul8PB7XMA2Lq7gv96fRXTRh/OM5+ub/RaP316UZPXP/ehjxvNX/3M4sD0//nL54Hps+57nzEDuvPRl9sBWOofVQzg+heWtvhzvr5sc6P5xeu/C7Fl8rhmTH+vI4iIJIy0Kt4bdvqaeitravndG6tYv2Nvo/XXPvStO74AAA/4SURBVLek0fwDb6/jAf9R8p8uGtGoIP+kqB/rd+zl9WWbeXXppqjm/GLz7qBHyrH0+GXHctrR3amsqeWom99ssj4706iu9XVKO+HwLnz81fbAulW/Hc+OvVU8HuRIOxK52Rm8+OMTycnK4Mj8vKg9r4hIsouoeJvZecBMYAAwyjm3oMG6XwGXA7XANc65t/zLR+K7L3Bb4A3gWtdSV+UIbS2rYNRt8wPzDzfoVR2u/Y+k928uDqV3B6O0/MB/vJ6dctm0qyLk+obnmO+/cDjXPOs7mm/Y4avewe2yufLUfsx6ZzVvXn86e6tqKbqzmFsmDuS0o303g2qTlcn9Fw7nyPwOHH1Ix0a9yK95bgnd89rw64kDqatz3DNvDZccfxhtczLpldOWmycOZPKoQ8nv2AYHPPFRCc8v2MCGHfu4ZeJARhx2MI9/+DWvLNnEnecNZUCPvEYd4/7juEOpqK7l1CO7MWFIDzLMyNA13SIiTUR65L0C+Dfg4YYLzWwgMBkYBPQE5pnZkc65WuBBfLcG/ARf8R4P/CPCHI3srarh+c83cOkJBdQ6x+yPSqL59M2a/oMjucvfDN/roLb89jijqtvRjBmQz18XlfLVt3v46alHMPS3c1p8ruenHU9hQWfKK2qCbv/L8UczdkB3XvzxCRzdoyMd2mQFivfSW87ghv9dyrpt5XRpn8O9FwwP9Oo+2m2gu78neP359YZ+NLRnYLphL/YHLhwemM7IMKafcVSTxx7RvUNg+urT+3P16f0p+XYPh3Vph5lx7wXDmHLCYYw8zHfe/h/XnsKaLWV8s3YVPzt7cEJdZy7SUDh9M0TiJaLi7ZxbBUEH9pgEPOecqwS+NrN1wCgzKwE6Ouc+9j/uCeBsoly8//uNVTz1yXpeWbIpcBeqUJ6bdjyTZ30SmD+mdyeWlYY/hvZB7bI5pGMubbIyuHniQI4t6MzPxvTn2/JKsjMzWPzph4wf3AOAC449NPC4NlkZVNbUAb5LvfZU1XLr37/g+QUbALj7/KEcd3gXwHcpVcntE1j4zU7+vnQTz32+nuenncBQ/7n4woLvO7Ctve1M9lbWkpFh3H3BsLB/jlgq6No+MG1mgcINMKBHRwb06Ejxd2tVuEVEwhSrc9698B1Z1yv1L6v2T++/PCgzm4bvKJ38/PwWb2hef9PzxWt957ZbKtzTjmlDxfrlgfm/jG9PTV01V/gTXj44h0dXNN/LubKqml8MzSbD6thTsozikuCZ9pdldVQCFx2dw7vvvgvAmV3hzPHtWb+7loN3raW4uOmoYkUdoWhMW3Z+uYQwW+6biNXN4SOhTOFJxEwiEn8tFm8zmwccEmTVTc65ULf/C3YI5ZpZHpRzbhYwC6CwsNC1dEPz4uJiDht8LCvfLA65zbKZZ3D1M4t5b802Bg0cQNGwXrx8+E52V9Rw6pHdALj7oFIKD+vMoV3acdY3O/j3Bz9mUM+O3Dd5GDf87zLyO7bhnOG9ueWVFTx0yUhGHHpws5mC5X7xqN28v3Yb00b3a/ZnioVY3Rw+EsoUnkTMJCLx12Lxds6NPYDnLQX6NJjvDWzyL+8dZHnUnHZncZNlU044jBUbd/HN9r10zM0OfILIy/X9+MP3K77/NuL7iPX9tXKzMzmiex5/u+qkwLrxg4N9pglPfXOxiIhIa8VqbPNXgclm1sbM+gL9gc+cc5uBMjM73nwnOKcAoY7eW+390u/H+b55woDA9G8nDebFH5/Ix7/y3bnwrCG+ohtO8RzUsxNDenXi1xMHRiumiIhIRCK9VOwc4AGgG/C6mS1xzo1zzq00sxeAL4Aa4Cp/T3OAn/D9pWL/IIqd1erPT/c6qC05WY0/l2RkGDn+y47OL+zDOcN7N9kmmLY5mfz9ZydHK6KIiEjEIu1t/jLwcoh1twG3BVm+ABgcyeu25JT+XQPN3VNOOKzJejMjJ0s9m0VEJDml5C1BzWjxFpUiIiLJKiWLd05mSv5YIiIiQIoW7z6d23kdQUREJGZSsngf7x+ZDIJfWC4iIpLMUuquYsf3yGTRVsfgXp0Cy04fkO9hIhERkehLqeJd56B357YADO7Via/++yzdlUpERFJOSjWb1znIalCsVbhFRCQVpcyR97qt5SzYUguUex1FREQkplLmyPvFhaUtbyQiIpICUqZ4i4iIpIuUKd4d2/rOAJzSv6vHSURERGIrZc5514+Geu8Fw7wNIpKGzKwEKANqgRrnXKG3iURSW8oU7zVbygDIy832OIlI2jrNOfet1yFE0kHKNJs7B93aWli3+RQREUlmKXPkXeccuh+JiGccMMfMHPCwc25Ww5VmNg2YBpCfn09xcXFcQpWXl0fttaYPqYnK8+wvv23snjsalO97B/K3FM2/wYZSpng7p3HMRTx0knNuk5l1B+aa2T+dc+/Vr/QX81kAhYWFrqioKC6hiouLidZrXTbj9ag8z/6mD6nhruWJ+1asfN8ruaio1Y+J5t9gQylzrOpwmKq3iCecc5v837cCLwOjvE0kktpSpnjX1aXQDyOSRMysvZnl1U8DZwArvE0lktoSty2klerqrxUTkXjLB142X9NXFvCMc+5NbyOJpLaUKd4OMLWbi8Sdc+4rYKjXOUTSSUQtzWZ2npmtNLM6MytssLzAzPaZ2RL/10MN1o00s+Vmts7M7rcoVVznHLqJmIiIpINIj7xXAP8GPBxk3ZfOuWDDnT2I75KRT4A3gPHAPyLMQZ16m4uISAwVHMAVB9OH1AS9UqHk9gkRZYnoyNs5t8o5tzrc7c2sB9DROfexc84BTwBnR5KhXp1zKt4iIpIWYnnOu6+ZLQZ2Azc7594HegEN791Z6l8WVGsGdti+vYK6utq4Df4QrlhdoB8JZQqPMolIomqxeJvZPOCQIKtucs69EuJhm4FDnXPbzWwk8DczG0Twlu2Q3cRbM7DDo19+yt6tO2JyMXwkYnWBfiSUKTzKJCKJqsXi7Zwb29ondc5VApX+6YVm9iVwJL4j7d4NNu0NbGrt8wd/TTRIi4iIpIWYjGtiZt3MLNM/fTjQH/jKObcZKDOz4/29zKcAoY7eW8Whc94iIpIeIr1U7BwzKwVOAF43s7f8q0YDy8xsKfAi8GPn3A7/up8AjwDrgC+JQk9z8I2wpiNvERFJBxF1WHPOvYxvHOP9l78EvBTiMQuAwZG8bjDqbS4iIukixUZY8zqFiETDgVxPG8z0ITUUReWZRBJLytzLw+nIW0RE0kTKFO869TYXEZE0kTLFW0feIiKSLlKmePuOvFW+RUQk9aVM8daRt4iIpIvUKd7onLeIiKSHlLlUrM45Mr0OISIJJ1qXnYkkkpQ58q6r0/28RUQkPaRM8VazuYiIpIvUKd7OkaHiLSIiaSBlinedC3lbcBERkZSSMsV7dP9uHHGQuqyJiEjqS5niffPEgZzZN9vrGCIiIjGXMsVbREQkXah4i4iIJBkVbxERkSSj4i0iIpJkVLxFRESSjIq3iETEzMab2WozW2dmM7zOI5IOVLxF5ICZWSbwR+BMYCBwoZkN9DaVSOpT8RaRSIwC1jnnvnLOVQHPAZM8ziSS8swlybCiZrYN+KaFzboC38YhTmsoU3iUKTzhZDrMOdctHmHM7FxgvHPuCv/8JcBxzrmr99tuGjDNP3sUsDoe+UjM3+H+Ej2j8kUmknwh/5eT5n7e4bwZmdkC51xhPPKES5nCo0zhScBMwW4H1OSIwDk3C5gV+ziNJeD+aiLRMypfZGKVT83mIhKJUqBPg/newCaPsoikDRVvEYnE50B/M+trZjnAZOBVjzOJpLykaTYPU9yb5cKgTOFRpvAkVCbnXI2ZXQ28BWQCjznnVnocq6GE2l8hJHpG5YtMTPIlTYc1ERER8VGzuYiISJJR8RYREUkyKVG8vRqe0cz6mNk7ZrbKzFaa2bX+5Z3NbK6ZrfV/P7jBY37lz7nazMbFMFummS02s9cSIZOZHWRmL5rZP/3764QEyHSd//e2wsyeNbPceGcys8fMbKuZrWiwrNUZzGykmS33r7vfzIJdwpXyzKzEvx+WmNkC/7KQ+zMOeaLy+41zvplmttG/D5eY2Vke5kvY99gW8sV+HzrnkvoLXyeZL4HDgRxgKTAwTq/dAxjhn84D1uAbIvIOYIZ/+Qzg9/7pgf58bYC+/tyZMcp2PfAM8Jp/3tNMwGzgCv90DnCQl5mAXsDXQFv//AvAZfHOBIwGRgArGixrdQbgM+AEfNdd/wM4Mx7/A4n2BZQAXfdbFnR/xilPVH6/cc43E7ghyLZe5EvY99gW8sV8H6bCkbdnwzM65zY75xb5p8uAVfiKwiR8xQr/97P905OA55xzlc65r4F1/vxRZWa9gQnAIw0We5bJzDrie5N4FMA5V+Wc+87LTH5ZQFszywLa4bs+Oa6ZnHPvATv2W9yqDGbWA+jonPvY+d4hnmjwGAm9P2MuGr9fD/KF4kW+hHyPDSNfKFHLlwrFuxewocF8Kc3vvJgwswJgOPApkO+c2wy+Xy7Q3b9ZvLLeC/wCqGuwzMtMhwPbgMf9TfmPmFl7LzM55zYCdwLrgc3ALufcHC8zNdDaDL380/HIlugcMMfMFppvSFYIvT+9kgh/Yy252syW+ZvV65ukPc2XYO+xLeWDGO/DVCjeYQ3PGNMAZh2Al4CfO+d2N7dpkGVRzWpmE4GtzrmF4T4kyLJo778sfE1zDzrnhgN78DV1eZbJ/880CV/TVU+gvZld7GWmMITKkAjZEsVJzrkR+O5ydpWZjfY6UCskyu/xQaAfMAzfB9u7/Ms9y5dI77FBX7Rpvpjvw1Qo3p4Oz2hm2fh+aU875/7qX7zF35SJ//vWOGY9CfiRmZXgO4Vwupk95XGmUqDUOVf/ifRFfMXcy0xjga+dc9ucc9XAX4ETPc5Ur7UZSv3T8ciW0Jxzm/zftwIv42uSDLU/vZIIf2MhOee2OOdqnXN1wJ/5vlnXk3wJ+B7bYr547MNUKN6eDc/o79H7KLDKOXd3g1WvApf6py8FXmmwfLKZtTGzvkB/fB2NosY59yvnXG/nXAG+ffG2c+5ijzP9C9hgZkf5F40BvvAyE77m8uPNrJ3/9zgG3/kqLzPVa1UGf7NhmZkd7/9ZpjR4TNows/Zmllc/DZwBrCD0/vRKIvyNhVRfFP3OwbcPPcmXiO+x4eSLyz6MVS+8eH4BZ+Hr5fclcFMcX/dkfE0ey4Al/q+zgC7AfGCt/3vnBo+5yZ9zNTHuEQwU8X1vc08z4Ws+WuDfV38DDk6ATP8P+Kf/H+tJfD1A45oJeBZfs1o1vk/llx9IBqDQ/3N8CfwB/+iJ6fSFr2/FUv/Xyvr3gub2ZxwyReX3G+d8TwLL/f+rrwI9PMyX6O+xofLFfB9qeFQREZEkkwrN5iIiImlFxVtERCTJqHiLiIgkGRVvERGRJKPiLSIikmRUvEVERJKMireIiEiS+f/gm4Zya0+1LwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_env_function():\n",
    "    env = gym.make('LunarLanderContinuous-v2').env\n",
    "    return env\n",
    "tmp_env = get_env_function()\n",
    "n_states = tmp_env.observation_space.shape[0]\n",
    "n_actions = tmp_env.action_space.shape[0]\n",
    "model = <>\n",
    "algorithm = <>\n",
    "for i in range(1000):\n",
    "    algorithm.step(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "297.1928817226966"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQcAAAC4CAYAAADuQCmzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUn0lEQVR4nO3dfXBU9b3H8fd3d/MEGEggPINgCTWIDxAaldg7FB/rs9diuaM1ndFiraDUe+dWLzNt7x+teqfj3OlMa3G06oy2XLR2jE4vVaFowVYKVRBMQCBQYtKEFJQEDCHs9/6xB++KB1hiNmeDn9fMmT3723PO78uGfPZ3HvbE3B0RkaPFoi5ARHKTwkFEQikcRCSUwkFEQikcRCSUwkFEQmUtHMzsCjPbbGZbzey+bPUjItlh2bjOwcziwBbgUqAR+AvwL+7+bq93JiJZka2RQxWw1d23u3sXsAS4Lkt9iUgWZCscxgC70p43Bm0i0k8ksrRdC2n7xP6Lmc0D5gVPK7NUh4icWJu7lx3dmK1waATGpT0fCzSlL+DujwKPApiZvuAhEp2dYY3Z2q34C1BuZhPNLB+YC9RmqS8RyYKsjBzcvdvM5gO/B+LAL919Uzb6EpHsyMqpzJMuQrsVIlFa5+4zjm7UFZIiEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhFA4iEkrhICKhThgOZvZLM2s1s41pbaVm9oqZvRc8lqS9dr+ZbTWzzWZ2ebYKF5HsymTk8CRwxVFt9wHL3b0cWB48x8ymAHOBs4J1fm5m8V6rVkT6zAnDwd1fB/Yc1Xwd8FQw/xRwfVr7Enc/6O4NwFagqpdqFZE+1NNjDiPcvRkgeBwetI8BdqUt1xi0iUg/k+jl7VlIm4cuaDYPmNfL/YtIL+npyKHFzEYBBI+tQXsjMC5tubFAU9gG3P1Rd5/h7jN6WIOIZFFPw6EWqAnma4AX0trnmlmBmU0EyoE1n61EEYnCCXcrzOzXwCxgmJk1Aj8AHgSWmtltwN+AOQDuvsnMlgLvAt3AXe5+OEu1i0gWmXvoIYG+LcIs+iJEPr/Whe3e6wpJEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQmlcBCRUAoHEQl1wnAws3Fm9gczqzOzTWZ2T9BeamavmNl7wWNJ2jr3m9lWM9tsZpdn8x8gItmRycihG/hXd68ALgDuMrMpwH3AcncvB5YHzwlemwucBVwB/NzM4tkoXkSy54Th4O7N7v7XYL4dqAPGANcBTwWLPQVcH8xfByxx94Pu3gBsBap6u3ARya6TOuZgZhOAacCbwAh3b4ZUgADDg8XGALvSVmsM2kSkH0lkuqCZDQJ+Ayx0931mdsxFQ9o8ZHvzgHmZ9i8ifSujkYOZ5ZEKhmfc/fmgucXMRgWvjwJag/ZGYFza6mOBpqO36e6PuvsMd5/R0+JFJHsyOVthwONAnbs/nPZSLVATzNcAL6S1zzWzAjObCJQDa3qvZBHpC5nsVlQD3wDeMbO3g7b/AB4ElprZbcDfgDkA7r7JzJYC75I603GXux/u9cpFJKvM/VOHA/q+CLPoixD5/FoXtnuvKyRFJJTCQURCKRxEJJTCQURCKRxEJJTCQURCKRxEJJTCQURC5UQ4HOdLXCISkZwIh/Hjx1NcXBx1GSKSJifCYdiwYaxYsYJzzz036lJEJJAT4QAwffp0li5dSlVVFfG47ionErWcCQczY/LkySxbtowFCxYoIHpZfn6cWM78tDOTnx9Dh6Oik/GdoPpKSUkJDzzwAOPHj+eBBx5g9+7dUZfU75kZDz98NW1tL9DcDGvWwN698P770N0NOfDF3E8pLi5m8eJK1q79A3v3wurVsH8/NDVBMhl1dZ8PORcOAIWFhSxcuJCKigpuvfVWBUQvGD26mAsuSAXBN78Jhw/De+9Bezu8/jrs3p16zJWgiMfjTJgwiPLyVE233w4HD8LWrdDWBm++CTt3wltvRV3pqStnB5pmxuWXX86LL77I7Nmzoy7nlJJMpsKhqws6O1NTV1fUVR3fkZoPHkxNnZ1w6FDUVZ3acnLkcISZUVVVxTPPPMMdd9zBsmXL6Mr1/8U5at8+eO45aG6GtWtTuxUtLalfuFzV0gKvvJKq9c9/hgMHUm25Mro55bl75FNlZaWfSGdnpz/00EOen5/vpO5mrSnDycy8puYbbhZ9LZlOJSUlfv3110Rex+dkWhv2e5nTI4d0BQUF3HvvvRQUFPCTn/yExsbGqEvqV5JJeu0Td/ToqYwedXbo3yD4RJ9+mPe2vsa+fS096ieXRzWfB/0mHAASiQR333031dXVfP3rX2f79u1Rl/S5NPH0C7jwi/NJxAqPu1zX4Q7isTzWrP0VqQ8o6U/6VThA6jhEZWUlr776KosWLWLJkiXkwk1yPy8KC4sZNewcBuQNxez4x7MTsUIGnzYGM8uZn1FRURGzZ89m5MiRrFixgt27d9PR0RF1WcdUWlrKyJEj+cpXvkJbWxtvvPEGAG1tbXz00UdZ7bvfhQOkAmLixIk88sgjxONxli5dqgOVfSSRyKcwf/AJgwHALEZRQQl5eYV0dR3og+qOraioiEsvvZSFCxcyc+ZM8vLy6OjooL6+ns2bN1NbW8umTZvYtWtXZGERi8UYNGgQFRUVnHnmmVxzzTVMnTqV0aNHM2jQILq7uzlwIPU+bty4kdbWVurr61mzZg3uztq1a2lvb6ezs7NXfh/6ZTgcMXjwYBYvXszFF1/MggULcvoT4FRRVjaJ0gFfyGjZmMUpGzyZgQNLIwuHwsJCLrvsMhYuXEh1dTX5+fkfv1ZcXExVVRVVVVXccsst7N27l5aWFlasWMHq1avZtGkT9fX1HDp0KCsjn1gsRlFREWeeeSYzZ87kvPPOY/bs2ZSVlTFgwIBPfVs5Ly+PwYMHA1BdXQ3wibpaWlro6upiw4YNbN++nZ07d/LHP/4RgG3btrFv3z6SySTJDK8i69fhADBgwABuvfVWiouL+eEPf8g777wTdUmntHg8j3gsL+PlY5ZHPJ5/4gV72ZGRwne/+11mzpz5iVAIY2aUlpZSWlpKRUUF3/72t+ns7GTLli288cYbrF+/nuXLl9PS0sL+/ft7XFdpaSmjRo3ikksu4cILL+Sss85i8uTJJBIJYj24vj09QEaOHAmkvuUMfCIIGhoaaG9vZ+PGjaxfv55kMsny5ctpb29nx44dodvu9+EAqQS+4YYbOOecc5g7dy7r1q2LuqRTklmMkWUVFCZKMl5nSOHpjBx5Jm1tfXPwOB6Pc9VVV2UcCsfbzsCBA5k2bRrTpk0jmUyyf/9+6urq2LJlCy+99BLr16+noaGBgwcPhm7DzCguLv54N+Haa69lypQpjB07NnRk0NtisdjHgVNeXg6kvuAIqRHHgQMHSCaTx7xdwikRDpD6QUyaNIlnn32WBx98kMcff5zDOhfWKwYOHMppp5XxwQdNDCgqJREryHxlM2IWJz9/AGVlk2hp2UJ3d2ev1xiLxZgyZQrz58+npqaGwsLjn0npyfZPO+20T+2GrFu3jg0bNvDiiy+ybds29uzZw9SpUzn//POZPn06s2bNYujQoQwaNKhX6/mszIyBAwced5lTJhyOmDhxIj/96U8ZMmQIv/jFL9i3b1/UJfVzxpmTL+aLZ1zKG+se7dEWYvEE5557PWefcQOr336EzZtX9F51ZkydOpUFCxZw4403UlJS0md3FispKeGSSy7h4osv5p577qGlpYUPP/yQSZMmkUgk+v0dznL2uxWfRUFBAT/60Y9YuXIlNTU1usvUZzBo0FC+OOFyBiTK6Oxs79E2Dh/upqVlMwWJwUwYcz6JxEmMPI4hHo9z9tln88gjj7By5Upuv/12SktLI/mFNDPi8TijR4+moqKCvLy8fh8MkMHIwcwKgdeBgmD559z9B2ZWCvwPMAHYAdzk7nuDde4HbgMOA3e7+++zUv1xJBIJpk2bxmOPPcaCBQtYtmwZTzzxBNu3b8+Zc+79wfDh5ZQO/AI7d69iz55ddHzUyo49K8n0c+Wwd3Kwcx9tbQ38o30rp4+oZujQ52lp2dyjetJ3H2666SZKSjI//iEnJ5PdioPAbHfvMLM8YJWZ/S/wz8Byd3/QzO4D7gO+Z2ZTgLnAWcBo4FUzm+zukRwASCQSVFZWMn36dL71rW/x7LPPUltby8qVK3VtxAnEYnHGjppGIlbI35rX0tW1n7XrlrDhndqT2s6BAx/g7rzf+lfGTK5kzOhzehQOEyZMYNGiRdx4440MGTLklPh0zmUnDAdPfcweuYAgL5gcuA6YFbQ/BawEvhe0L3H3g0CDmW0FqoA/9WbhJ8vMGD58OHfddRe33XYbK1eu5Omnn+bll1/W/SKOYcCAEsaUncc/OrawY+caALq6DvT4moXG5vWc94UOxo+cwcb839HVldkpwbKyMm6++Wa+853vMGnSJIVCH8nogKSZxYF1wCTgZ+7+ppmNcPdmAHdvNrPhweJjgD+nrd4YtOWMwsJCrrjiCi677DK2bNnC4sWLefXVV6mrqztlz3Ds3buXoqKik1pnyJDhxPIP0bLnXbq72096/aN98ME2du1dRSxhFBUVEY8f+2KcvLw8YrEYCxcu5I477mDy5Mk9ug5Aes5OZv/bzIYAvwUWAKvcfUjaa3vdvcTMfgb8yd2fDtofB37n7r85alvzgHkA48ePr9y5c+dn/sf0lLvz4Ycf8vzzz7N48WI2btz48WWq/d3YsWP58pe/zPz58xk2bNhJrm0MGDCE7u6DvXKFo7vT0QFlZcUZHdwsLCxk3LhxGilkmZmtc/cZn3rhZO+9APwA+DdgMzAqaBsFbA7m7wfuT1v+98CFn/V+Dn0hmUx6Z2enr1q1yr/2ta/50KFDo/6efY+nCRMm+I9//GNvbGz0Q4cORf3WSg6jp/dzMLMy4JC7f2BmRcAlwENALVADPBg8vhCsUgv8ysweJnVAshxYc6J+coGZUVBQQHV1NV/60pdoamriySef5IknnqCpqYnu7u6oSzyuvLw8Kisrueaaa6ipqWH06NH61JWeC0sM/+RI4RzgLWADsBH4ftA+FFgOvBc8lqatswjYRmp08dUT9ZErI4cwyWTSm5ub/bnnnvPq6mpPJBKRjwqOnvLy8nzWrFn+wgsv+P79+z2ZTEb9tkk/wjFGDid1zCFbZsyY4WvXro26jOPy4Fr0t956iyeffJJly5bR1tZ2zOvq+0JpaSlXXnklc+bM4dJLL6WwsFAjBTlpxzrmcMpdPp0tR65Fv+iii7jgggtob2/n7bffpq6ujmXLlrFhwwZaW1uzfgOOWCzG0KFDmTNnDnfeeScVFRX6A0CSFRo5fEZH3r/W1lYaGxt57bXXaGhoYPXq1WzZsoWPPvoo4+/PH4+ZUVFRwfz587n66qsZM2aMTu1Jr9DIIUuODONHjBjBiBEjqKysJJlM0t3dTX19PXV1daxatYoVK1bw97//nT179pzU9ouKiigvL+fOO+9kzpw5kX1/QD5/NHLoA+5OR0cH77//Pps2baK2tpb6+nrq6+vp6OgIHVmk36xkxowZDBw4UKEgWXGskYPCIQLuzv79+2lra/v4JqcvvfQSDQ0NJJNJbr75Zq666qrPdLMSkUwpHHKYu5NMJmlqasLdGTt2rI4nSJ/RMYccduR+AOPGjYu6FJGP6eNJREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBREIpHEQklMJBRELlxJ/DM7PdwH6gLepajjIM1ZQJ1ZS5XKzrdHcvO7oxJ8IBwMzWhv29viippsyopszlal1htFshIqEUDiISKpfC4dGoCwihmjKjmjKXq3V9Ss4ccxCR3JJLIwcRySGRh4OZXWFmm81sq5nd14f9/tLMWs1sY1pbqZm9YmbvBY8laa/dH9S42cwuz1JN48zsD2ZWZ2abzOyeqOsys0IzW2Nm64Oa/jPqmtL6iZvZW2b2Ug7VtMPM3jGzt81sba7U1SPuHtkExIFtwBlAPrAemNJHff8TMB3YmNb2X8B9wfx9wEPB/JSgtgJgYlBzPAs1jQKmB/OnAVuCviOrCzBgUDCfB7wJXBD1exX0dS/wK+ClXPj5BX3tAIYd1RZ5XT2Zoh45VAFb3X27u3cBS4Dr+qJjd38d2HNU83XAU8H8U8D1ae1L3P2guzcAW0nV3ts1Nbv7X4P5dqAOGBNlXZ7SETzNCyaPsiYAMxsLXAU8ltYcaU3Hkat1HVfU4TAG2JX2vDFoi8oId2+G1C8qMDxo7/M6zWwCMI3UJ3WkdQXD97eBVuAVd4+8JuC/gX8HkmltUdcEqeB82czWmdm8HKrrpCUi7t9C2nLx9Emf1mlmg4DfAAvdfZ9ZWPd9V5e7HwbOM7MhwG/NbOpxFs96TWZ2NdDq7uvMbFYmq2S7pjTV7t5kZsOBV8ysPkfqOmlRjxwagXFpz8cCTRHVAtBiZqMAgsfWoL3P6jSzPFLB8Iy7P58rdQG4+wfASuCKiGuqBq41sx2kdkVnm9nTEdcEgLs3BY+twG9J7SZEXldPRB0OfwHKzWyimeUDc4HaCOupBWqC+RrghbT2uWZWYGYTgXJgTW93bqkhwuNAnbs/nAt1mVlZMGLAzIqAS4D6KGty9/vdfay7TyD1f2aFu98SZU0AZjbQzE47Mg9cBmyMuq4ei/qIKHAlqaPy24BFfdjvr4Fm4BCpBL8NGAosB94LHkvTll8U1LgZ+GqWarqI1LByA/B2MF0ZZV3AOcBbQU0bge8H7ZG+V2l9zeL/z1ZE/fM7g9TZh/XApiP/n6Ouq6eTrpAUkVBR71aISI5SOIhIKIWDiIRSOIhIKIWDiIRSOIhIKIWDiIRSOIhIqP8DRHFY85VAfzgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_session(tmp_env, model, t_max=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3: Hyperparameter's analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to do some experimental analysis of the algorithm.\n",
    "\n",
    "Plan:\n",
    "- to learn something about the *std* parameter\n",
    "- to learn something about the *learning rate* parameter\n",
    "- to learn something about the *n_samples* parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}