Deep_Q_Learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "collapsed_sections": [
        "2wOXQqxsfkOh"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## **Playing Atari games using DQN**\n",
        "\n",
        "In this notebook, **we'll train a Deep Q-Learning agent** playing Space Invaders using [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), a training framework based on [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.\n",
        "\n",
        "We're using the [RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) with no extensions such as Double-DQN, Dueling-DQN, and Prioritized Experience Replay.\n",
        "\n",
        "⬇️ Here is an example of what **you will achieve** ⬇️\n"
      ],
      "metadata": {
        "id": "3jn8v7OffkOV"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "outputId": "01159ec7-ccae-444a-fb84-09e42295bd40",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 232
        },
        "id": "rdwAHj3MfkOV"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<video controls autoplay><source src=\"https://huggingface.co/ThomasSimonini/ppo-SpaceInvadersNoFrameskip-v4/resolve/main/replay.mp4\" type=\"video/mp4\"></video>\n"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "%%html\n",
        "<video controls autoplay><source src=\"https://huggingface.co/ThomasSimonini/ppo-SpaceInvadersNoFrameskip-v4/resolve/main/replay.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Install RL-Baselines3 Zoo and its dependencies 📚\n",
        "\n"
      ],
      "metadata": {
        "id": "6PSVXWJ7fkOW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# For now we install this update of RL-Baselines3 Zoo\n",
        "!pip install git+https://github.com/DLR-RM/rl-baselines3-zoo@update/hf"
      ],
      "metadata": {
        "id": "oAg9_vn6fkOW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!apt-get install swig cmake ffmpeg"
      ],
      "metadata": {
        "id": "JSefcIANfkOW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lfs3j90QfkOX"
      },
      "source": [
        "To be able to use Atari games in Gymnasium we need to install atari package. And accept-rom-license to download the rom files (games files)."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install gymnasium[atari]\n",
        "!pip install gymnasium[accept-rom-license]"
      ],
      "metadata": {
        "id": "rUu2gbVefkOX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Create a virtual display\n",
        "\n",
        "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames).\n",
        "\n",
        "Hence the following cell will install the librairies and create and run a virtual screen 🖥"
      ],
      "metadata": {
        "id": "xOojX16BfkOX"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7PFxIn3-fkOX"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!apt install python-opengl\n",
        "!apt install ffmpeg\n",
        "!apt install xvfb\n",
        "!pip3 install pyvirtualdisplay"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Virtual display\n",
        "from pyvirtualdisplay import Display\n",
        "\n",
        "virtual_display = Display(visible=0, size=(1400, 900))\n",
        "virtual_display.start()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8977b1ec-0999-4f55-ff6f-a78dedf9e47d",
        "id": "0poI1sNRfkOX"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyvirtualdisplay.display.Display at 0x7f725ff7fa00>"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Build the DQN**\n",
        "\n",
        "in this step we must:\n",
        "* defining a class for DQN\n",
        "* Building the architecture of NN of DQN\n",
        "* Building the Replay Buffer and Storing the transition\n",
        "* Defining the epsilon-greedy policy\n",
        "* Define the Training phase\n",
        "* Updating the Target network\n",
        "* Training the DQN\n",
        "\n",
        "and that's it 😀\n",
        "\n",
        ">**This is a sample code of implementation of DQN for Atari game from scratch [if you need for more detail]. But in the next section we train a DQN using [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), a training framework based on [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.**\n",
        ">\n",
        ">**We're using the [RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) with no extensions such as Double-DQN, Dueling-DQN, and Prioritized Experience Replay.**"
      ],
      "metadata": {
        "id": "ZmQUZkzLfkOh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### From Scratch"
      ],
      "metadata": {
        "id": "2wOXQqxsfkOh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# ======================================================================\n",
        "# import the necessary libraries:\n",
        "import random\n",
        "import gymnasium\n",
        "import numpy as np\n",
        "from collections import deque\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense, Activation, Flatten, Conv2D,MaxPooling2D\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "# ======================================================================\n",
        "\n",
        "# ======================================================================\n",
        "# create the Ms Pacman game environment using Gym:\n",
        "env = gymnasium.make(\"ALE/MsPacman-v5\")\n",
        "\n",
        "# Set the state size:\n",
        "state_size = (88, 80, 1)\n",
        "\n",
        "# Get the number of actions:\n",
        "action_size = env.action_space.n\n",
        "# ======================================================================"
      ],
      "metadata": {
        "id": "Ha5LmoY7fkOh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# ======================================================================\n",
        "# Preprocess the game screen\n",
        "# let's define a function called preprocess_state, which takes the game state\n",
        "# (image of the game screen) as an input and returns the preprocessed game state:\n",
        "color = np.array([210, 164, 74]).mean()\n",
        "def preprocess_state(state):\n",
        "  print(list(state))\n",
        "  state = list(state)\n",
        "  #Crop and resize the image:\n",
        "  image = state[1:176:2, ::2]\n",
        "  #Convert the image to grayscale:\n",
        "  image = image.mean(axis=2)\n",
        "  #Improve the image contrast:\n",
        "  image[image==color] = 0\n",
        "  #Normalize the image:\n",
        "  image = (image - 128) / 128 - 1\n",
        "  #Reshape and return the image:\n",
        "  image = np.expand_dims(image.reshape(88, 80, 1), axis=0)\n",
        "  return image\n",
        "# ======================================================================"
      ],
      "metadata": {
        "id": "Vw6EXhWufkOh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# ======================================================================\n",
        "# Defining the DQN class\n",
        "class DQN:\n",
        "  # First, let's define the init method\n",
        "  def __init__(self, state_size, action_size):\n",
        "  # Define the state size:\n",
        "    self.state_size = state_size\n",
        "    # Define the action size:\n",
        "    self.action_size = action_size\n",
        "    # Define the replay buffer:\n",
        "    self.replay_buffer = deque(maxlen=5000)\n",
        "    # Define the discount factor:\n",
        "    self.gamma = 0.9\n",
        "    # Define the epsilon value:\n",
        "    self.epsilon = 0.8\n",
        "    # Define the update rate at which we want to update the target network:\n",
        "    self.update_rate = 1000\n",
        "    # Define the main network:\n",
        "    self.main_network = self.build_network()\n",
        "    # Define the target network:\n",
        "    self.target_network = self.build_network()\n",
        "    # Copy the weights of the main network to the target network:\n",
        "    self.target_network.set_weights(self.main_network.get_weights())\n",
        "  # ======================================================================\n",
        "  # Building the DQN (CNN Network)\n",
        "  def build_network(self):\n",
        "    # Define the first convolutional layer:\n",
        "    model = Sequential()\n",
        "    model.add(Conv2D(32, (8, 8), strides=4, padding='same', input_shape=self.state_size))\n",
        "    model.add(Activation('relu'))\n",
        "    # Define the second convolutional layer:\n",
        "    model.add(Conv2D(64, (4, 4), strides=2, padding='same'))\n",
        "    model.add(Activation('relu'))\n",
        "    # Define the third convolutional layer:\n",
        "    model.add(Conv2D(64, (3, 3), strides=1, padding='same'))\n",
        "    model.add(Activation('relu'))\n",
        "    # Flatten the feature maps obtained as a result of the third convolutional layer:\n",
        "    model.add(Flatten())\n",
        "    # Feed the flattened maps to the fully connected layer:\n",
        "    model.add(Dense(512, activation='relu'))\n",
        "    model.add(Dense(self.action_size, activation='linear'))\n",
        "    # Compile the model with loss as MSE:\n",
        "    model.compile(loss='mse', optimizer=Adam())\n",
        "    # Return the model:\n",
        "    return model\n",
        "  # ======================================================================\n",
        "  # ======================================================================\n",
        "  # Storing the transition\n",
        "  # we train the DQN by randomly sampling a minibatch of\n",
        "  # transitions from the replay buffer. So, we define a function called store_transition,\n",
        "  # which stores the transition information in the replay buffer:\n",
        "  def store_transistion(self, state, action,reward, next_state, done):\n",
        "    self.replay_buffer.append((state, action,reward, next_state, done))\n",
        "  # ======================================================================\n",
        "  # ======================================================================\n",
        "  # Defining the epsilon-greedy policy\n",
        "  def epsilon_greedy(self, state):\n",
        "    if random.uniform(0,1) < self.epsilon:\n",
        "      return np.random.randint(self.action_size)\n",
        "    Q_values = self.main_network.predict(state)\n",
        "    return np.argmax(Q_values[0])\n",
        "  # ======================================================================\n",
        "  # ======================================================================\n",
        "  # Define the training phase\n",
        "  # Now let's define a function called train for the training network:\n",
        "  def train(self, batch_size):\n",
        "    # Sample a minibatch of transitions from the replay buffer:\n",
        "    minibatch = random.sample(self.replay_buffer, batch_size)\n",
        "    # Compute the target value using the target network:\n",
        "    for state, action, reward, next_state, done in minibatch:\n",
        "      if not done:\n",
        "        target_Q = (reward + self.gamma * np.amax(self.target_network.predict(next_state)))\n",
        "      else:\n",
        "        target_Q = reward\n",
        "        # Compute the predicted value using the main network and store the predicted value in the Q_values:\n",
        "        Q_values = self.main_network.predict(state)\n",
        "        # Update the target value:\n",
        "        Q_values[0][action] = target_Q\n",
        "        # Train the main network:\n",
        "        self.main_network.fit(state, Q_values, epochs=1,verbose=0)\n",
        "  # ======================================================================\n",
        "  # Updating the target network\n",
        "  def update_target_network(self):\n",
        "    self.target_network.set_weights(self.main_network.get_weights())\n",
        "  # ======================================================================\n",
        "# ======================================================================"
      ],
      "metadata": {
        "id": "tTRf_BwyfkOh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# ======================================================================\n",
        "# Training the DQN\n",
        "# First, let's set the number of episodes we want to train the network for:\n",
        "num_episodes = 500\n",
        "# Define the number of time steps:\n",
        "num_timesteps = 20000\n",
        "# Define the batch size:\n",
        "batch_size = 8\n",
        "# Set the number of past game screens we want to consider:\n",
        "num_screens = 4\n",
        "\n",
        "# Instantiate the DQN class:\n",
        "dqn = DQN(state_size, action_size)\n",
        "# Set done to False:\n",
        "done = False\n",
        "# Initialize the time_step:\n",
        "time_step = 0\n",
        "\n",
        "# For each episode:\n",
        "for i in range(num_episodes):\n",
        "  # Set Return to 0:\n",
        "  Return = 0\n",
        "  # Preprocess the game screen:\n",
        "  state = preprocess_state(env.reset())\n",
        "  # For each step in the episode:\n",
        "  for t in range(num_timesteps):\n",
        "    # Render the environment:\n",
        "    env.render()\n",
        "    # Update the time step:\n",
        "    time_step += 1\n",
        "    # Update the target network:\n",
        "    if time_step % dqn.update_rate == 0:\n",
        "      dqn.update_target_network()\n",
        "    # Select the action:\n",
        "    action = dqn.epsilon_greedy(state)\n",
        "    # Perform the selected action:\n",
        "    next_state, reward, done, _ = env.step(action)\n",
        "    # Preprocess the next state:\n",
        "    next_state = preprocess_state(next_state)\n",
        "    # Store the transition information:\n",
        "    dqn.store_transistion(state, action, reward, next_state, done)\n",
        "    # Update the current state to the next state:\n",
        "    state = next_state\n",
        "    # Update the return value:\n",
        "    Return += reward\n",
        "    # If the episode is done, then print the return:\n",
        "    if done:\n",
        "      print('Episode: ',i, ',' 'Return', Return)\n",
        "      break\n",
        "    # If the number of transitions in the replay buffer is greater than the batch size, then train the network:\n",
        "    if len(dqn.replay_buffer) > batch_size:\n",
        "      dqn.train(batch_size)\n",
        "# ======================================================================\n"
      ],
      "metadata": {
        "id": "xx3VV2EofkOi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "WO51ODJgfkOi"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fl8FbAjNfkOi"
      },
      "source": [
        "#### Using [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo)\n",
        "\n",
        "([Read This](https://rl-baselines3-zoo.readthedocs.io/en/master/index.html))\n",
        "\n",
        "**Train our Deep Q-Learning Agent to Play Space Invaders 👾**\n",
        "\n",
        "To train an agent with RL-Baselines3-Zoo, we just need to do two things:\n",
        "\n",
        "1. Create a hyperparameter config file that will contain our training hyperparameters called `dqn.yml`.\n",
        "\n",
        "This is a template example:\n",
        "\n",
        "```\n",
        "SpaceInvadersNoFrameskip-v4:\n",
        "  env_wrapper:\n",
        "    - stable_baselines3.common.atari_wrappers.AtariWrapper\n",
        "  frame_stack: 4\n",
        "  policy: 'CnnPolicy'\n",
        "  n_timesteps: !!float 1e7\n",
        "  buffer_size: 100000\n",
        "  learning_rate: !!float 1e-4\n",
        "  batch_size: 32\n",
        "  learning_starts: 100000\n",
        "  target_update_interval: 1000\n",
        "  train_freq: 4\n",
        "  gradient_steps: 1\n",
        "  exploration_fraction: 0.1\n",
        "  exploration_final_eps: 0.01\n",
        "  # If True, you need to deactivate handle_timeout_termination\n",
        "  # in the replay_buffer_kwargs\n",
        "  optimize_memory_usage: False\n",
        "```\n",
        "\n",
        "For DQN another Hyperparameter you can check [this](https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams/dqn.yml)\n",
        "\n",
        "and check the Unit_3-5 [DRL_HuggingFace] RL_Baseline_Zoo3 section 1-5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xCLLKkCkfkOi"
      },
      "source": [
        "Here we see that:\n",
        "- We use the `Atari Wrapper` that preprocess the input (Frame reduction ,grayscale, stack 4 frames)\n",
        "- We use `CnnPolicy`, since we use Convolutional layers to process the frames\n",
        "- We train it for 10 million `n_timesteps`\n",
        "- Memory (Experience Replay) size is 100000, aka the amount of experience steps you saved to train again your agent with.\n",
        "\n",
        "💡 My advice is to **reduce the training timesteps to 1M,** which will take about 90 minutes on a P100. `!nvidia-smi` will tell you what GPU you're using. At 10 million steps, this will take about 9 hours, which could likely result in Colab timing out. I recommend running this on your local computer (or somewhere else). Just click on: `File>Download`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tOmQOVfEfkOj"
      },
      "source": [
        "In terms of hyperparameters optimization, my advice is to focus on these 3 hyperparameters:\n",
        "- `learning_rate`\n",
        "- `buffer_size (Experience Memory size)`\n",
        "- `batch_size`\n",
        "\n",
        "As a good practice, you need to **check the documentation to understand what each hyperparameters does**: https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html#parameters\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**For config the `dqn.yml` file we need to Hyperparameter Tuning by Optuna:**"
      ],
      "metadata": {
        "id": "DWeb114PsrJf"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w5Il2J0VHPLC"
      },
      "source": [
        "**Tune Hyperparameters automatically with zoo3 and optuna**\n",
        "\n",
        "We use [Optuna](https://optuna.org/) for optimizing the hyperparameters.\n",
        "\n",
        "Tune the hyperparameters for DQN, using a tpe sampler and median pruner, 2 parallels jobs,\n",
        "with a budget of 100 trials and a maximum of 5000 steps:\n",
        "\n",
        "This can help you: 🌝"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAtgAAAKRCAYAAABuhN6hAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAMskSURBVHhe7P0NzFXXfeeL7/uvIbaxeXFIwNTYMSYYKB78piSGIeM+qMkM7ZCWiSom0pTRGBkaqa6CJaZyZvAYCd0rRsKyK1VGInOvNSMNmjKuhjZUGQ9c+zKQOm1cY1PgUkps4/ISYyDEGAe4038/y+d3sp7Nednnec7z/vlIW+fstfZae6219/Oc7/rt316//+1LX/rS3xUiIiIiItIV/n+1TxERERER6QIK7A74zd/8zeI73/lO+hQRERERacSoFNi/+Iu/WGzevFkhLCIiIiKDjhbsDvgv/+W/FI899lj6FBERERFpxHUvOf7O7/xOcdtttxXPPPNMw/2nn366uPPOO9P3a9euFTt27Chefvnl4ld+5VeKr3/968UNN9yQ8j744IPiueeeK/72b/82WZR/93d/t/j0pz+d8uCjjz4qXnjhheKv/uqvetWZlwOs0F/96leL733ve5WELe29//77a3ufEO08dOhQ8c1vfrN45513ioceeii1NT9f3g6Ic+btz/sM9PvRRx9N36dPn54+q7SVdo4fP76YNm1ar3pPnjxZrF27tti7d2+9DsZgyZIl9fESERERkeFLRxZsxCSicMOGDcmSu2bNmrrQ5JN90tkuXbpULF68OOWtXr067ZO+ZcuW4uLFi8XOnTuTWERonjt3rl7uxIkTxYoVK1K5vvD7v//7qX0IZ4QudebtHDduXDF79uxi48aN6TiIdjKJiHZs3769WLhwYRLXiO/169entjMxKDN16tTiypUrqRznfPjhh1O5dsyfP7/4i7/4i1Tu4MGDxaJFi9KYHD16tJg3b17tqCJ9J01xLSIiIjL86UhgY12dOHFiEqaI7Zxf+qVfSuKWlwDZsARjnW0EluMpU6YkETpz5sxkcY5yfMdiHmDFRYC2swh3AqIW0RzCOepG7Ec7Vq5cmfo6efLklNcKRDfWZzh//nwS8VXKvfvuu/VzHzlyJE1eGJM9e/YUEyZMSGPKxnfSRERERGT405HADosz1l8ENkI0XiTEPeTs2bN1CzDiMTh8+HAS3By/bt26ZI0NYXn16tVkLY5ybOGOMpjQnzlz5tTbwiduG0MB44zFv6enJwlsvmu9FhERERkZNBTYWEyxpCI6FyxYUEv9OWH5feONN5KVmmMpg6sHUG7GjBnpO3m4WuQiGks3UA+uFRzPcY1AwOdCvgrUiyhtZkFvBBZ1wEoPuGuEP3l/oF+saEKfEcvN4HynT59ObYf9+/cn6z7uJnwXERERkZHBdQIbV4SbbropWamXL19e/PCHP6zl/FzsxoYAfOmll5IoxO0iXD0ox4uEQN6BAweSy0VeFks4bNu2LX1yvsjrREw3A1HK5ID6tm7dmkR8K/bt21dcvnw5Wdgpg2WdfQj3F/JwG6EvVepsR1j12SAmHsALmUH+XURERESGNwMeKh0LLitwIMDDLQQBPdZXxWCCka/OUibGjZc+c+EtIiIiIsObjnyw+wIv+2ERz8F1A+vwhQsXailShpVNGDdfbhQREREZWQy4wMZCzZrOrGUd7hC8TPjiiy/W/Y3l5+B2gvvJ0qVL60sZioiIiMjIYcBdRERERERExhIDbsEWERERERlLKLBFRERERLqIAltEREREpIuMSh/sWOLu05/+dNr/3ve+1zbUOi8XLlu2LK3L3ejFQpYW5EVN+OCDD4rnnnvOlzRFRERE5DpGtQUbYU3kyHbiugrUQV3UKSIiIiLSDF1ERERERES6iAK7xNe+9rX6et3dCNkuIiIiImMLBXbGzTffXIwbN67uCvLwww8nf24RERERkaqMOYH9O7/zO3ULddlK/dFHHxU7duxI38+fP5/ENqHeRURERESqMuYE9u///u8nC3Vs3XgBUkREREQk0EVERERERKSLKLBFRERERLqIArvGyy+/XHzrW9+qB5kp74uIiIiIVEGBLSIiIiLSRUa1wCa0ebfWs6YO6opw6SIiIiIijfjfvvSlL/1d7buIiIiIiPQTXURERERERLqIAltEREREpIsosEVEREREusgv3HHHHf+u9n3U8Iu/+IvFv/23/7ZYuXJl8bWvfa248cYbB2y5vfxcv/qrv1r87Gc/K44fP17LHTh+5Vd+pfjX//pfp5cu33333eL999+v5QwNv/RLv1Q89dRTxf/6X//ruv63yhsIGBtC4r/33ntDPi6jmbE6zoN9P8vAMNj/u7lvNmzYUPz6r/96w3M9/fTTxW/91m+l36y77rqr+MEPflDLEZGRyKi2YH/ve98b8HDof/u3f1usX7++2LJlS/HRRx/VUgcWfhgQN7t3704Cp8rkgTKbN29OK6GwUU6kEQiBZ599Nt1jVSnfX1u3bu2ovPSmL9egP7BKEteP6zhW6M//7r5cH/5P83+X/9uUK4/1M888k36vMJiIyMhHF5ERytWrV4vz58/X9tqzYsWK4sSJE+kf+Pbt24s5c+YogKRrhFjh/mI7ePBgsWjRolquiAT83+b/t4iMbkblMn1YBn73d3+3+Iu/+IuuWa95fHfnnXfW9j6xjud1Y9FYvXp1sWvXrhQFMiiXu3btWrFjx450TJ6HBeWFF16obI3+5je/Wbzyyiu9zoVg/vrXv17ccMMNaf+DDz4onnvuuSR+chqNDxasfI3vcv8aQZ/Xrl1b3HzzzWk/71uzPHj00UeLc+fOFfPnz0/7b7zxRvH7v//76Xu5HZDnN6Pc93w8sRrdf//9KR2wEGEtIv3WW28tbr/99tTOqteA63bq1Knivvvuu65cq7wY909/+tOpnmbXp0y5b9F+KPetPFad3mPl+xXycq3GOYd6uMbtrlv5Pimfq9m9Uh7LKn0Lyn2MOltdu77c6ydPnixWrVpVXLhwobjnnntSXpV7uT/XoJzX7nzltgdxj7W6Bq3Gqy/3Om1hvI4dO1Z88YtfTGnRjlZ50Gm/c6i70f/u8v+ivN+Nrs+f/umfprGK6/3aa6+lsYH83owx/YM/+IOG40H9Vf52RGR4Myp9sCdOnFj8/cQh/cDFP7X+8uqrrxY7d+5M2+XLl1P9/+//+/8WP/3pT1P+Zz/72eLBBx8s/vqv/7ruW4f4mTp1avF//B//R/F//9//d/EP/sE/KPbt21f88R//cfonCuvWrUt1zp49u/jyl79cHDp0qF5nM/jnzrn+5//8n718Xznvn/zJn9TbiQXxpptuum4MaPvcuXOTiKY8PyRLly5NgprHnpRtN27xw7x3797if//f//fib/7mb9KPCf3nnM3y4OGHH04+iPSdsfzCF76Q/HgRil/5yleK//pf/2tqB36IV65cSY+uW8EP1vLly4s/+qM/SuWokx+4119/PflXzpw5M12D//Sf/lOqk/ZxPfmhRjTs2bMntZN2TJ8+va3vIz+O9957b/18eblmefyQIjh4ivDtb3+7fn24jrSlFf/iX/yLVDdtpFx+POlxvRln7iHue64r99iECRPqfUcAtPOVpm7q4fpGH6Jcq3GOfN4L+I3f+I0kMv+v/+v/ansvU476ow/8Hdx9992pX9TNvYKIY8zye+Whhx4qJk2aVP/7qdI3KP/dPfDAA6neuHacGzG0bdu2+rW7dOlSn+51+v7II48k0cc1OHPmTL39rdra12vA3w8iExeEuFfoVyti/HlPZcqUKfV7Je6xVteAv51u3uv8D2W8+P9NO/7yL/8ylcPXPcayUR5tzMeEsUMYM3mu8v+/0f9uJgj/9J/+0/T/lDrzsWx2fcaPH5/ayP+2H/3oR+l6YAShrtOnT9fr5p7hfPx+NLoPuA/jnhSRkcuYcxFB9IafKBviMqiah2jjH/3kyZNrudWZNm1a+ofLP+P9+/fXUoviyJEjxbhx49rWiUDgB/3FF1+87seDf/hYPaKdCHHOl8Mx/Bh9//vfr5efN29eeqSfW2/aQT38CDBhKNMqD7D2hDWbCQX7M2bMSPtlEIiMVyuYLJw9e/a69mM9Q1xjqW9mOcMKFpZ6rEa33XZb+t6OfLwOHz7cq1yjvLiuiPmAvCr9o10IhhCHOdyjcb0RjYhOxpJrwDlpRyurYSc0G+eA9DVr1iQXEcQVIqtd38jPfbcRJflYcm/EmFH/t771rXTf8pidfnK/09cqxJjkf3dljh49Wv+7wDoa9ff1XmeiEdeAiQ80u9er0OoakMc52j156pRm1wC6fa/n48U5sAYj/FvlseVjQh775f99ncA5mNzzv57//VWhjXF/NTPwkMb/b/6PN/qbFpHRwZgT2Pxghp8oW/5j1CwPqxE+y/guk84n/0jbgWhG8GzcuDFtWMI4R3/gRx8LG49Ly8IC6xU/LNF+xGMOx4e1rds/wt0AsY1Y4UcNscWY80PULYE4Uon7EpHCuMTLaFzPJUuW1F/mHcwXbdsR4qrdhJF3A4DVFegDj+GrgJhCzPO3wD3tS5XSbfhfyz0J/N11Swzzd8v/b+7dcHERkdGHLzlWICwoYYHi0WT4+rWC43hcyz9ptvhnGtYR8gO+Y5FpZPEow3G8JJNbwhBcWIiwdgJiI89n/4knnmgorhFuiNmyYA/CSppbcrAg8qiTx8Scmx+M8ONslVdm8eLFyXKPuOY7k5AYL85XHo94kpA/XeDRO644tJ8NCz3XB79P6sNCD5TJ/ZXbQdsRs2HFbATH8AidMSyT53HNoKenJ31GHpbemEBwDs7VbDUHrhsiGhCucX3jZVfqjnFm3Dhnfo+VaTSWEPdX3PdBs3FuRPQz+t1sLLF4co0YA9K5DzuBftIP/jbz9jYay+gXll6gXNmXthHdutcb0c1rwISevGYTjWbXAOhHlSdozaDubt7rQD9oT6P/iXkebc/7zSf7jEc3oJ0YVXjqGG1tdn2qwN8tZWOcRGR0osCuAI9/sazyCJ4fQ/45sg/xQ0EebiNYX8OaxqNCfJspE1tYQfDxRBBHOt9J6yv8cOEKgYCkPn5433nnnVruzycFuBrEOePHDeHGo/HoH1v+g88PGBb7/NE9FkTK0F+s8zwqxnreLg8QIHEuLLBhpWac8zFhK4sBBAbkj39pP/VTZ0wi4vrgisKPbZwL38mq0CbEHz+s5cf6Mc70D+GQT1oa5VEX/URARl75iQY/uLQbwRZCh+vDdaIMG33kOiMsGGf6HRZ//F/zcS7fY1XGEuJeinslyrUaZ+537vs4F/cKq4pQFzQbS/5G2KcMdb799tu1nNaEMI0N8mvQaCxpA2O2YMGCVAb3IXxj29HJvc55O3mK0J9rwMZ4xdM02sI1ifuBjXEKml0DoCxtp07KVbXWdvNeB/5HRft5IseLhyGwm+WV+80n+6S3otX/7sijvjgf/tT0DRpdH97vEBEJXEVkAOFHKn8bnH/cCF9emGn3z78V9K/RKiIDRYwnP6D5j2RfYAyWLVuWxF/ZMoWox9IVqw00Oi8/fP1xc0FwIP6qPpqlTQjzfBWA8nXNaZVXBdqH8Kuyukh/6e9YdkqjsRxIqoxlf69XGca00YoUzejvNWj199SIbl6Dbt/rrcau03EdznDNXEVEZPSjBXuAQBxiPcwpu5r0Bx7nxqPugYQfwbBODfQ//LIVD8sWFq6w8vHDg7VpMAQhP+j0d7AEIT+6WM8GS1w7lkPPaL4GrRgp12eg4HeA/98iMroZ1RbsWIO1yprOAwE/JDxaDB9JHuXG2rn9JerGl3uofzA7gXY3s7iVrxt0sp5tFTq1YDeilYVJ69PIotvXa7Atra3+ngaawRy7vowrZXg60Mgnvpv/i6sS7cFFp9G5Gc94J6Db//dEZPAZlQJbRERERGSo0EVERERERKSLKLBFRERERLqIPthjBMdEREREZHBwmb5hSPnlSOClF6Lj8ZIM68fG2/extN37779ffOYzn+n1Vn68VJOvUjDSx0ZERERkuKOLyDCFYBVE7YuohvFGOW+/s3QdUQ9zjh8/ntKJKBcgsBHjBHARERERkcFBgV0RrMp5pDosx5G+adOmtOV5iFtEcRwHHPvss8+mvP7w1ltvFbNnz67tfQICGzGdr41NiHDWrx5r68yKiIiIDCUK7AogiBHHzz//fLImY1nGLSOEMqG4WY+aPHybySMUMCGUEbkB4pf0KuvV5uHE2Vi/OTh16lQKYkObgh//+MdJTBO8ATcQ2kagliNHjtSOEBEREZHBYMwJbIRqiNawNgfN8hCrvBwYgpfPSZMmFTNmzEj5uHMQOADOnz+fonQhbvGZRghTHtE7ffr0Yv/+/em4djRzEYGPP/64OHDgQLFo0aJayicgpsNNhHNevXq1OHToUC1XRERERAaDMSewEaohWtnyF/1a5b377ru98tasWdM2ChiW6kuXLiWxG77R3RK84VeNkA9oz9mzZ5OlXPcQERERkaFBF5EKYJXGWp1bu6uCxRp/aUQvVuduCV7qIUxx7oIChw8fLubMmZPcVnQPERERERl8FNgVwDK8e/fu4qtf/WrdfWTz5s3J7aMdWKyxMuMb3clqHq18sANcUPKl/CD8u3nhUfcQERERkcHHdbDHGI6NiIiIyMCiBVtEREREpIuMaoEdLh198Z0ebWC5xq1l48aN9XDpIiIiItJ9RqWLiIiIiIjIUKGLiIiIiIhIF1Fgi4iIiIh0EQW2iIiIiEgXGdXL9MXLfN/73vf6tSTdr/zKrxTLli0rtm3bVl9nWkYHTz/9dHHnnXem72+88UavkPQiIiIifcF1sDMIab569epi165dvcKg90VgU9fatWtTwJicjz76qHjhhRdSZMivf/3rvQLFIPBeeuml1PabbropHcf58vNfuHCh1+Th2rVrxY4dO1J7yxOLDz74oHjuuedS1MdcSAKh35955pn0vZyXl+PceTvLIpQVWlitJU9v1PeY5BAw5/7776+l/nw8qowr7SR6ZSMRHO0I8klVuQ95/4JWdYuIiIh0gi4iAwSCETH52GOPJfGJoOU7aSEmEZdbtmxJ6Wy5uEM49/T01PZ+DhOAEydO1MusWbOmPhkg79KlS/U8vpMW0A7SOSeh1PPlCyNvw4YNaX/x4sVJKC9fvjxFsYxyhGHPyxEG/rXXXiumT5+eBH5A+7dv357KIXaXLFmS6oMYC7ajR48mMR55fQEBTf1xPj7ZJ522Iq6ZhMQ5169f37WQ9SIiIiJlFNg1sGASmnzixInFypUr0/rZCN4Qfojhf/yP/3HL0OXd5PDhw8Vtt93WS7RimZ4wYUJtrzeISUKyIySD/fv3p7SyeMUKTij1adOm1VJ6c/Xq1eL8+fNJ4J89e7ZuCWZigCCeN29e2qfecePG1UPAz58/P32WoS7Aal8Giz1t6Y/AXrRoUWpXTDT4pN1Yyh9++OHi4MGDvZ5IiIiIiAwkCuwauEtgob148WLdEppbm7H43nLLLXUL6axZs/olCgEXCkR9I9GOCAYsyQGuDQjFBQsWpONzS/KUKVOSMI5ycPLkyfRZFrYIYVxQjhw5UkspkhilTgLRnD59Op0HgY/bRM6ZM2eSyEf403/yGSPKzJ07t3ZUb0hHRB86dKiW8nOwJGNpbyb220E7aA/tyqFdtJ8JAP3kOALt0Md84iQiIiLSbcacwEbEhqAti9RWYMEO6zBCkf1GFtlOaOUiAligFy5cWNx44421lE+ss7iFUA43iK1btybrdRVCROMysXPnzl5W3dxFZObMmW0t9J/97GeTdTiELSI2dxPB3zmeBFBf2ed5sOHcuIYwOcJ9RURERGSgGHMCGxEbgpYt3B+GIwhgrLu33357LeXnhI837g+4SOCGgbUWl5AgJgBhyQ4RzT6+1Y2suAhRfLzDes1nDpZm2oRFf9KkSfVw9Ihp0sJNJHywmQhgLc99wXOaWaCr0swCHu3Hqo91X0RERGSw0EUkA/eKKoIMtw3EbCOXh25z7Nix4r777qvtXU8IYNpC23tqL0YiXLFsHz9+vO7mEmAZHz9+fEOBTTkszohTrNKI5rCQ88lLjpTH7QOhnk9WEPtlN5Hw26aeRucL4R1+3H0Bf3XaFfXzVILzsRoMk4X8BUsRERGRgeYX7rjjjn9X+z5q4EXFL33pS0kAlsVlK376058mcY1V9mtf+1oSlKx4ceuttyZXDYQ16ViGsdr+zd/8Ta1ka774xS8mK+6rr75aSymKe+65p1edbHfddVcSi3nbEf0PPfRQ8alPfap4//33iyeeeCK5eESZn/zkJ8l/nLZThraT/8u//MvpRT/8juHRRx9NftA/+MEPkui+++6708uKCPMHH3ywuPfee1N9lEMQY+nnOPr+67/+6ykP3+89e/YUr7/+evGVr3ylePPNN3uN7+c+97lUF+2kf3/913+d6sDKTB9i+cA4V7T/3/ybf5Pa3w76kJf91V/91eJnP/tZ8cd//MdpNZPf+I3fSOm044/+6I+KP/uzP0v9zfPow4cffpiuRX7OfHxERERE+oPrYIv8Pa6DLSIiIt1CFxERERERkS5iqHQZFuCOk0dbzGG1larRHjslj2LJS6BasEVERKS/jEqBLSIiIiIyVOgiIiIiIiLSRRTYIiIiIiJdRB/sEqyhzFJ3+m0PP+LaAGHjhzo6pIiIiEgjXKavRKcCO39JLofyRBckPHlQFoXRTtbIzl/iI0JjXg54AY81qNeuXZvWa456aC8hy9lnTe0QoAHtoF7K3XzzzbXUzl7oizFpVKbc/xi3cnq8qEggHPoWx+XXinW5m41lfi3yPiuwRUREZLihi0gJhFwnIdQJ8sLxhAW/ePFiCg2elydQDfsbNmxI+ytWrEifQFhxAskQECaPNIiIpQyCNsqHsCUEOYIcMd0IRDznokzejghdThptJPIhQr4KBGp57bXXiunTpydBDHxGEJs4V34+oP2RzrliAoHYJshO1BW0G0sRERGRkYACu0YIRiI0bt26tR4evFtgab106VI9tDkQVpxQ6ERvxHpblbfeeiuJ3r6C0N27d28KiV4WuWUQ/oSFj1DmTAoAgY/Q37FjR9rvBCYUjEXUJSIiIjKaUGDXQACvX78+WU2xsHYbhOrUqVOTmAaELWIbscs2YcKEXlbsVpw6dSod32gSgN/5xo0b00SBDXeKRpw/fz4J58mTJ9dSGkObiHBIG0+fPp0mBYD7C0KZdM4R58ut4riCRDoW+Lx/TCwWLVpU2xMREREZPYw5gY0ADNHH1kyAdgt8ijnPunXrktU43B2wACOScRFBpGLRrSqwP/744+LAgQMNBWozF5G+wCQAX+czZ86k/SNHjvRyEwk4B+fCnSWnmYsIYBEfP3588eCDD9ZSREREREYHY05gh39zbAPt3xs+1IjNpUuX1q3OuITk1maEeCduIuGy0c4C3YwpU6YUV69eTQK/GbhwTJo0Kb3gSBtXrlyZrPCkI7qZILRzMWkGTwyYJOALLiIiIjKa0EVkkGAFkCtXriQXCyzVCON46ZAN15RO3EQQqLhudCLKA0Q+Yp+VO1qtwkFbT548WW8j28GDB1M61mh8sPOXNjuFOu64445Uj4iIiMhoQYFdA2GLdRtXjokTJyZrbTdfdkRM4iKCxfbLX/5ysh4fOnSollvULck9PT11NxZ8mMPFJPdtDhDtN9xwQ23vE5r5YHMcfSLt61//eno5sZX1Hss0L0GGz3iA5Tqsziy7x/c4F20NdxJo5YMNjMnx48d7LR8oIiIiMtJxHWwZcbgOtoiIiAxntGCLiIiIiHQRQ6WPcZpFooRGkRuHEizXhkoXERGR4c6oFNgiIiIiIkOFLiIiIiIiIl1EgS0iIiIi0kUU2CIiIiIiXcSXHKUl+UuQo3kc6SdRKlnbOw/pLiLdxf/PIjIWcB3sihAkZfXq1cWuXbuKl19+uZY6/CAgzW233VY888wztZT+0+3x7OtYDuQ16KbAHohr0Cm0gUA/ZVgZ5siRIynYUAQpunbtWgo8xJiW255f+2nTpqU6QxCV74uycMrrjTyidsYYcz3Xrl17XaChjz76KB0zY8aMXu2EKivbEBxq2bJlxbZt2xpeyzjv5cuXe61EQ7n8fLFSDdFKm41lq7b09RoEUT4XoOUxhsgvtx/yNjbq33e/+92U1uwaMH75JLtZOrz77rvpvqHdBKlqNLa7d+++ri/GKxCR0YguIiJ/TwiD/orr4QKiKkLwX7x4sR6WP8QWQom8CH+PAELwtINyCxcuvO5YROvv/d7vFZcuXaqH1V+zZk1dMM6fPz9FKz179mw6FhhrxpxjEYIINL7n1yFvJ1s7cV0Fzv/222+naKq0K9IQ5YjcONf69euTQGw3ls3ozzXgk4nOa6+9VsybNy+lAe2hXZRhY8xymo0XS1wSyRWBG3nU8+qrr7a8BnxOmDCh2LBhQ0o/evRosWrVqno7KUM652SCynleeumllMfEJFi0aFHqo0JaRMYKCuwKYKnJQ6jnob/5odm0aVPx+OOPp9Dq5G3evDml8+NE2SDf5/tTTz2V6mkUSpwf26iPjeMbpce5Ih2LF1alcjng3OVy/YG6o77yucrtjH63GstWtCrX6hqQH2NcPhdiINLzMkHev3Iby2XZb3cNqJtzRHp+PsrTx/wakUbZ/LioIx/r/kJ4eyzLkydPrqU0B4GMiA5hGjA2WISxGjdi7ty5xbFjx1Lo/VwwDgWzZ88u3nzzzeL06dOpXYC1PLf8Djbla8D4MgHYt29fErj5vdcXGHPEcCcCl3POmjUrTZLCEr1nz55i3Lhx111/Jk9cf55ycCxl5syZk/4m2OgXZUVExgoK7Apg3SxboRA4YWXjB4cf7Y0bNyZLDyxevDh9tuKee+4p9u7dm+pDuPT09KR0fpDicSp5bAg84IcLy2CkI3Y4V6TnVqi8HO09d+5cPf3EiRO9LEx9gbqjPsaHx8IhBLBY5e0Pl4N2Y9mMvl6DsMJFO7DAxTgjNiKdOhE0AeIWC2Lkc524JpG3dOnSejvYqKvVNUAY8ziccY90rh0uLwGiHAFKHo/9CQePEMwFDZ/sd1OsIL64/9pdgwChzPXNQVjRnxBiOfSdsaR+tk4EI64LTKxi0sG17A+cl/E7dOhQctOYPn16ah/XjnuDyVt5MjUYlK9BTEjYZ1z7M170jzFHxHcCkw44efJk+gSENH8nU6ZMqaV8AvclEwTGFGI8H3300fT/7Pvf/37l+0tEZDQw5gQ2Pzjx48OGWApa5bUDP0LERTzCrWIpQoTFcYhfRAjw48qPWqM6+KFFAEQbEWWIm1bwA4v4xbIa5fge5yvXWbbkNoPxiTL8qE+aNKn+o0x/iLoYluuqdPsa0I/capz3uxUIntwKTV8QKdRHHo+7ERFVCctkLowR01En4BOLxRJoezzCP378eN3Syif7/RUruRCDmABVgTaOHz++ePDBB2spRcsxZaJDPxFnnQrGbruI9NQmV9wjiGyIyQt1cw4mU4xN1b+DvtLsGjA2WI7Pnz+f9jux+jcaL+49xO9AEP9TmHzu3Lmz198E9zpPnCDuaxGRscKYE9jxIxpbLmJb5Q0X+CHD0hVtLPtgNgOrU25xZYsfdEQPwjbSEXYIkFYgApYsWZIsrZThR50f9yDGEnHAD3BVsdLtaxBW+vAhxbpclehbbFXGZSDAKojgeuSRR5LFNayE/SGEGPcEkyKufzO4d0LsAWNw4MCB5AIQMKHKJws5iENeyuPpAvcCE5eqgrGb0DYmmjFxoj2MZ0xeAu65Tp5E9ZVm14C/rdwViskd/s1VJyVlcveNTgjLdUyaAbHOE4C4H8IHm2OXL1/eq438X+F/FU9thuLvRkRkKNFFpCLNHo22I0QHj0kXLFhQS20NAoofVMrkUA/1IWaA/PzHD3gMXBY6/LhduXIlHd9IAPWFOG/80Pb09Fy3EgEgVhAREFbcvo5lX8phWQ3XBX78c1HYCsaYCUQjUcOkgXqaCZ5G14C2A+ME5OECUkV8YBWk/Fe+8pV0HTuxnLeDuhBHCE/aRNtzMYfAROzlbgKAeLrjjjvqllGslXwvux1RD9c9n9xxPzA+zcZvoAj3Gs4fbaFd4SaSM5BW3zLla8DkI4RrbAhVxovtt37rt1K5GNt88tMI7i+e7vD/p/w/pRVcY+673B2ICT5p5Xtw//796anGYF9TEZHhigK7IvEjhTUJqxKW1nY/JiE6sJRh3fnhD39Yy2kNP148pg4LFhvWrWhDPJalznfeeadW6hPiUWxYC8MqFi+fRTpbO/cLfuyxPFMGC2T0nXK0kR/9aCMihf28XJyHx+C0mx9s6MtYQl/K8cPPZIDjn3jiibR6RBWom/7EI3y2GEsmDfiX5nn5WDa6BrT9xRdfTMKcNPIQ/pynCoj68NHuNqycwX2KmC73DV9zHv3HtQvYx1UlJlXss3xb9I+NFz5ZmYNJUbhjQHmy0YpGPsVVKJfDVQlLNefO+4KwZQLxr/7Vv6ofy0ZZ/gb7+wSlKnEN+JtGNJefUjDhY0JG+3HFijZyDXKx22y86AfvROT/U6o8VeL/BpOhKMP3Ri+y0gbcpmhjuzpFRMYCroMt8vcgkBEH+dq9wwnah0U91iAWGen4f1pERjNasEX+Hh7LN1sFY6hBiCD+sSwrrkVERIY/hkqXMQtuA7hdQETtG24CO9rIy6ydrPQxmsmvWxl8l6u63fQX3JMaRaKERpEZ5RP8/ywiY4FRKbBFRERERIYKXURERERERLqIAltEREREpIsosEVEREREuogvOcqgw5JzrGUNw/XlQhEREZG+4jrY/YRADgR+CRFf5dzNVh8gdDLrHPf09KQ6c1gdgcA1lGO5NlZKiHMBIpVAISFcg3K7YtIRqxwA0dluuOGG9D0I4bt69epeKzZEG6Hch3wFByLG5fU2EtLDfe1pERERkb6gwO4nCGyi1128eLH4gz/4g5TWybkpTzjvRkuwlfMQ5t/4xjfSes1EBST0M2GMCVHMuRHYzQQrS5sRDa7VEmaNjmlWjrYgvnft2pWWIgtBTbQ41mrO85qhwBYREZHRiD7YNRCImzZtSluEBUYAVoGQ2oheBO9AQshvQMxzrrvvvrs4depUSmsFEw5CHA8khML+yU9+UkybNi21s2wRFxERERkrKLAzpk6dWly5cqV47LHHkmsF1lXEaRWOHTuWrMmDwTvvvFPce++9xa233lq89dZbxbhx44rJkyenPFxANm7c2GuSgHUYS/KCBQs6mjgEuKtEfViysV6XQfDfdNNNxZEjR9K5cGNZuXJl0+NFRERERitjTmDjdhFisSw28S8Ov+Tz58/3Eq7t2LdvX3LVePDBB2spA8frr7+ehPSJEyeSNTsHX+cNGzakSQJbuKkgetesWVNs2bKlWLJkSbF169Zkta8CvtVRH+MX4bqxUiOiGUfcQ3bu3Fl3CUFYc/zevXuLdevWFZs3b648WREREREZyYw5gR3CL7Zu+WhjJT5w4EDyxx4opkyZkizsnAu/7L60HXGMSD548GC/Le68KLl9+/Y0joj3Rv7WtBHBD/iIi4iIiIx2dBHpIojXO+64I7lKDHd4eXKw4CnASBgTERERkW6gwO4iCOzjx49ft/xep4QbC77PLJHHd9KagStLvADZyAcbH2gs95HGBo1WLmlEFR/sHM6ZnwsXEVxFuvW0QERERGQ44zJ9MmQgxF2mT0REREYbWrBFRERERLqIodIHCFw6ytEYg3fffbeye8ZoBMu1odJFRERktDIqBbaIiIiIyFChi4iIiIiISBdRYIuIiIiIdBF9sEcYTz/9dHHu3Lm0XN5gw/J8q1evLnbt2tUwqIwMDETcJFImkTOJNvrCCy/Uo2mKiIjI8MNl+oYpzZawG+kCO17+jElPfq2mTZuWAuDEC6DlPKJkhriMFyVbTZ5o79q1a69blxyR+od/+IfFr/3ar/W6RxCyy5YtK7Zt21b09PT0ekk1fxkzf0kzyNsR5718+XK9TKO2EII+rmN5UkiUTML25+Oct0+BLSIiMnzRRUQGHQTuwoULk6jshLNnzyahCkTMRPS2AhGKoCeUO2KW1Vv4Ttrf/M3f1I5qThwfod5XrFiRPoFzk04+W4hroI1vv/12cfXq1WL+/Pm11N6h5bds2ZImDBFAiInLiRMn6vU1Cz0vIiIiwx8FdgWwHG7atCltEZ0QK2bkPfvss3XhF/tYGjdv3lw89dRT6fjHH388WSvbRUIkj2OwkOZRGbFcB7feems6hnTOEUKVNnEcW7mdkKfn5Wjz1q1bG+ZFe0gnImNugS2Xy9vYCoTypUuXeonPdmDZPnXqVBLWMX7UMRhggeZcVcPLz549u3jzzTeL06dPF3Pnzq2l9gbxT3TLmTNnFg888EAxYcKEWo6IiIiMdBTYFZk6dWpx5cqVZF3EHQD3jXYW2JtuuimFMcd6+tBDDyVBhRUzwpo3IqyunCO3kubrZt91113Fzp0765bVxYsXp08gtPrhw4evayd14loSFlKspWGRxVKKxTTyEJPUiZDFrYF2k47VFetzsGjRomL37t31cp2s7X3s2LFUvhMuXLiQPmnbe++9l/qD60h/YCITE4SVK1cmP+cyjAPXn3ENGoWkB47lmh86dKg4cuRIMX369Kb3yfnz59Ox3FdcgwULFvSqS0REREYmY05gIzRDFJXFTKs8hCU+sRDCaPLkyWm/GYjp/fv3p+8nT57s5TeL6MJSHOdqZ9nOOXjwYBJkYVnNRSaifN++fek7bgvr169P37GU4lMc5+N7WGQ5b1ip2RDp1Ek6fsRRXxkELgK1quU6hzrHjx9fPPjgg7WUarz11ltpgtGsTZ3CJCQmCLhvcM0CxoHxwHLPJCN3A2nmItLT05M+uTaIbKhiqY9JDpOYJUuWpCcDPCEQERGRkceYE9gIyRBFuTCCVnndBgGG+I1zIe4H8sU1/IHD/ze2sDizQgVuG5GO73EVYryw7CJEc9eSdtD/AwcOJD/kVtDu/+//+/+SGGdigxD99re/ncoPNOGDzROIpUuXthW89J2JTAhzLNxYsJu5iUyZMiX1LyzzEE8wmER1auEXERGR4YEuIl0AtwLcPhBYiLBGbgadUtVKXgXEKG4ItK0sgNnH/xdrNHBMuLDQBtxcsMBy3KpVq65bkQOYiGB5hU7ai5jEp5pzwJkzZ5IrRljycQWZOHFi8f7776f9oWLPnj1p/JoJ5YBx4poxFjFZYVKDyKZfOYwzop1VTBpNFqr6e4uIiMjwQ4HdT7CoYv3FfxeLJb7FuFX0F+rFsol7AtbQvrhh5LC0G5T9hhF3iLxwH1m+fHnxzjvvpGNpw9GjR+t9w6pKXwHBnbu40E7q6cQKz7HHjx+vi3aEOueLPiNA8TVvteLHYAhR2omLCNb2EP+NfLAR4FyzfAxwDWKSgPhm4sVYcjxPDXA5os9lFx026MSnXURERIYProMtMkLA6u062CIiIsMfLdgiIiIiIl3EUOkyosGqG2HEy4yWsOJ5H0dLn0REREYzo1Jgi4iIiIgMFbqIiIiIiIh0EQW2iIiIiEgXUWCLiIiIiHQRX3KsQF4fobRZv5g1ogca1r4mQMlIfKmNthPREHzJdGjw5UgREZGhwXWwO4CAIKtXry527do1bAU2YbYJvjIcgpT05TrQZ6JKEnilDMFcvvrVr9b2egv38qSqykSIsSLATg5h0V966aVUFxEmY+xjDeof/vCHxZIlS65bteSDDz4onnvuuRR9Mm8jRDvzSQcQip3rRDsIsU75iOrIvbZ27doU4Iay0Xfal49Nlevt+tkiIiKDiy4iw5gQX4qiT0QiwpbQ4xGCnH3SEaO/93u/V5w4caIeonzNmjVtJ0EIVY5FtCJ2+Z6LV0R6T09Pbe8T/vIv/zLVzbGUoSzf169fXxfHiO0NGzakdLaYBEAcTzh1Jk8I5yNHjtRD0gf0iYig+/btS/uzZ88uXnvttRR2ncmEiIiIDF8U2F0AkZSHuWY/wGqZ523dujWJwlaQz3EcTzjyXFBR35NPPnldfpTBIouVNM6HQA/ytuT10l7y8nzSKNvo/FEnn3E8W36ubrNo0aIURj1EM5+EbSc8ec/fi2DEKJbnbnL48OFkHR4IQUtIddo8bdq04tChQ+k7fQnmzZuXJgyIdsT2uHHj6mI7F+IiIiIy/FBg9xOEKJZULJJYJnPLKoJzwoQJyZrJhmVz9+7dbS2r5GMlxbWgEXPmzEnuD9QJuCVEmdwayxYWWdqC60WkI95WrFiR8gBRjqAkj/M+/PDDxZtvvpmEXQg6hN7kyZOTxRXCAsxG/3Fz4Jhug8BlHM+cOVNL+QT6gwBmCzGai/58otMXEMHA+HYCbiobN25s2Q7GFKs1Y0m7aT/jR1/L48w+feVJxunTp3sJcRERERl+jDmBXba65uKnVV4zsEBiWQ03jrBGTpkyJe2X4XhASGEdjnMhVquK04MHDyZBjTC7dOlSvc5mcC7EG9btOB/fEaYB4j8spLg04PLw6quvFsePH68LOtqH6IwJAuMT9a1bt66YNGlSMWPGjJQ3VDCOMZnpBvv37y8WLlxY3HjjjbWU9rRyEYlrwMuHO3furI9l7ibCOF+9ejXdS1w7JjsxueA43URERESGN2NOYOdW17L4aZXXFxBDiE6smWyIYc4BiGNEbJwLcT+QvtYItvBfjq3Ki5C5oMNtASs3IAKx1GPtpi4s2KxUMRA0m0gwQcCyG5bsgQABzLlvv/32Wkr/CB/skydPFsuXL69PqjhPuLzk7iEIbu4hXnBEmK9cuTL5busmIiIiMnzRRaSfYFnEZSOEEu4EWCIRy/gN4xLSiaDtL7QHd4rcwolQu3LlSnJb6dTyGQIz3CTCyh2W6vPnz6fPnp6e4uabb07fBwKEfT7OWM8Rmnv27EmTANpT5YlDXzh27Fhx33331fa6A5bx8ePH1/sD0Uf6RZ8AwY0Yj3uIjScYuomIiIgMX37hjjvu+He176OGiRMnFl/60peSMOmGVRgRxCP/L3/5y8WnPvWpYsGCBcWv/uqvFj/72c+KP/7jP04rPPzGb/xG8bWvfa343Oc+V/zRH/1R8Wd/9mdJQP3ar/1aPY/tgQceSK4XrUAo4nJBvYjWX/7lXy7+4T/8h8ll4MEHH0wuKD/4wQ/SsY8++mivfVw46Dvt43x33XVXykMk5ulsuD0wPvQPgcoqFT/96U9TPTlYb7Gg0ic2wHXkC1/4QhLe1AVYsN9+++00Lv/23/7bdC7aTz/y87WC/tx77731NlYZZ9rCMTHWjBdj//rrr6e8ZvDUAL91LPRYiamX8ULo5vcPY/rQQw+la8+Sg++//34qXx57YCwR47Qh+hD9zo+nXXfffXeyVnNdY9xxB+GYP/mTP0n38Ve+8pXkC5+PG33nPsDXnjry8crPF9xzzz3F5z//+TQe0XYREREZOFwHewBhxQ3cF8ItBAsybgG5763IQMN95zrYIiIig4cuIgMEIh9XjZx48RHLqIiIiIiMTgyVPoBgOYxQ1TCYYdblExeQcqTGIKIojmby+89Q6SIiIoPHqBTYIiIiIiJDhS4iIiIiIiJdRIEtIiIiItJF9MEeInL/WCL/Pffcc2m96qqwlB9L54H+tYPLSLi/REREZOhwmb4hBqHM2sedCGzWWl69enWxa9euIX1hkmUI77zzztreJ0L/D//wD9N61EQijOUJgX4S+TEmAvECYlmclpc2rEJZ8EJMWlasWNHrRcd8MlN+CRWItBjnzvvXaBIzku4zERERGTx0ERmBRBTF4bDcX4T+ZkM0/z//z/+TxPXMmTOTAA0IqHL06NEkUEkntDmBbUjvFhG2nY0w9DFhYcUQ0ggWBIjuAOFMmPcoF+KavrDMImVIJ4z5qlWrevVJREREpBEK7Ipg7dy6dWvxne98J20RlhsrJ1uAANu8eXPKx9KMYIsyfCetr0QbVq5cmaL8Ee2RehGDwDmjPeV2Qp5OG3OxmOexRZ19gTDfhIufP39+2qfPkydProf/Jv3q1asp7Doitj9j0gkIbsK+I+5bQXsIWY5lOkQ6yyuOGzeu3icRERGRZiiwK4DgQtw+//zzyZqJxRO3DtL379+fxGOIRAQYQgxLbbhChHUUC25PT086ri/gDkJo7+3btxcXL16sW15zdwpcGgj1TToWXdqJkKYduF5EW7AyhyU3JgiRh8W3KrhfhCiPCQShvwn3PXfu3HQMaQhq0oF0QrczPgjeGLv+gk96tCX6lMN5pk6dmsYnIJR7TFTYGKd4QnD+/Pn0CYRLpw8RLEhERESkGWNOYCOgQkyx5RbeZnkIM/x7Q4jxOWnSpCTEEL2IrxCJiMfjx4/XXSGwFEd9iNF21lMol+vE8o2PMZZhwC8YVwnAZSMXw9EW6mWCwEShL5RdROg3Vt/cTQQ3EPZJ53yzZs2qi1fEbrfcRHIXkTyIDJMO+sx127t3by9/6WYuIiIiIiJ9ZcwJbARUiCm2XGy1ygs/3tiwJMcLhiESEZPTp0+vu0KEhTj8eBGjVUCIIozjXCFc+wPWVyzfUSfbQEYy3LNnT7LkL168uJd7CAIb9xbcXBC9WJ2xKledQPSFuHaM/9KlS9PTiFaEb3turaYP9Ce3aouIiIg0QheRCiCqsFbn1u6csBgjJnF5COGNhZj9sNzi1zsUcP4rV64kYckkICdcH8KdAzGfrwzSV5gQUPcjjzzSyz2EiUhu9WbjBcKBFNgBop9xiL42g7bTpnCvAVYboT9DuWqLiIiIjAwU2BVAVO3evbuXj2/+kiACFv9mrKN//ud/ntIAtwuEOcc/8cQTxdtvv13L+eSFxLDg4n6ycePG61487Cbbtm1Ln5wn+kAbaDv9W7BgQUrDreP06dPp2Co08sEOsOxjrc7dQ3JrdsDY5WI2r5OXOttZnKuCcMZFhIlOtLORDzaUx4uXMSNNREREpBWugy3XwQuCna5FPVpBcPMkopE7jfeZiIiINEILtkgLXnrppWS9Llu4RURERJphqHS5juFmwcZFpBxxMRiKMPHeXyIiItKKUSmwRURERESGCl1ERERERES6iAJbRERERKSLKLBFRERERLqILzkOMbxQSCTDTl7Ui/7Bc889l9aYHkjiJcODBw8O6IuP+XW7du1asWPHjhEX2MUlDmUo4f6LQFEEdPI+FBEZGlwHe4AhmAtBVJoJ4eEisFut9zxYAjsgCMzq1auLXbt2DVuB3Wy8+iqwy5PCDz74IF1bAvOsXbs2BcQJQjjRBoLylCGfqJXlcrHiSk9PTwq2E/cc9ygBj5iIErW0vGJLtIVIpRyXU54IRZtiUsu1LLcD8raU+5ALw1wwRhmgzsuXL9fv/3Z/ZwH38rJly1LQoPLfW3m1mrJAzdsC7SbujfperrM8XlA+T8Ax06ZN63XfNftf19f7UEREuoMuIkMMP5T8yHayzBwCYv369WnrhrhuB+JpzZo1/lgPIEwoCKsf4eP5ThogYrdv357S+STqJoKS60Hali1biosXL9aPIZ37ifsqz8vvszw8/R133JFENMS15njKUT7uMwRctI8NscikK8Q1Yg/x99prr6WQ+BDtiOPffffd9D3aEn3I8+I+4xjWIN+wYUNKP3r0aLFq1ao0IWVMbrrppiT6uwFjsXz58hSxlXPRbyYhjDP9IsoqkBdbK3EdNLt20Gi8gP8JcXxcg6rnExGR4YECuyL8KEawEbb4kcTqtWnTpuLJJ5+s5yEM+MFGKOSh0MnDshTlCANOWh4inXLsP/XUUynv8ccfT/WwkZe3I+oKOG/kxZa3M87HFueMdKxoWM0in7qAc0Ra1BWU64wy5T7kef2h2TWAch5bu3OSz3WjrRyfh2WP6xd1xfi3Gy+49dZb62Xza9sM6sRSjSU4IMw+aXfddVct5RMOHTpU/OQnP0mWzL6CqDt16lQS1vQJEPSdQJunT5+eAvEE8+fPL65evVrs27cvCeOouy9QdtasWUm8xyQSq/y4cePq1t233nqrmD17dvreX3p6etKkI0Qs4h9Bj/BFxCPm8+vTF8rXrpvjJSIiwwsFdgUQb0uWLKlbkrAssR9iDIvaLbfcUs9DGACii8e6WAfDChePdsNSSH4ZfswRElj1HnrooWLv3r3JEjZjxoy6FZG8Mog68tiol2NCMMT5Ih9BhXCI9NyCyEZdQHtpe1g4A/qOxQ/RwfG5xQ/oAwKRsjEm/REQra4BG99JIy/6En1oBSKHx+uUwxq7aNGilJ5bXtkQW4iwduMFiOKdO3emvkM7K+uUKVOS0Lpw4UItpShOnjyZPsNlJKC9jO2RI0dqKX0jzkXb3nvvveROUFW0M2F49NFHi1deeaXXE5S5c+cWx44dS2PH/dWf6829DjEOQJsZJ/7WgEkCwjT+DvsDkw7GIOfMmTOpfq4n4pt+5RO5dhO4MuVr19fxyid2TNzL94iIiAw9Y05g86MYP05sIQihWR7CA4HFDyFgicL/E2EE+IaGdYs89kMg9AXENBZMQGDEeauC4MCyl4s+frzZj77xI90fKyh9R3QgOIE2sh910gfyEGAhkmJMWl2DZrS7BmUQRghBtrBQszEGuZBBJMckBOEzfvz4huWwWCPCqhBuE/Qd4dSfcQZ8gleuXJnagY8w4j3GvT9gAUY8YkHthBUrVhSnT5/u1QbGlEkUPtxw+PDhXm4PA8HHH39cHDhwoD4pGgxigst9U4Vm164/45VP7BpNfkVEZOgZcwIbgRU/TmwhrqBV3kiBH24E9n/7b/+tlvIJ/LgjgKNvVQXCQNDtcQ6xHUIGS/qLL76YBC4bPsRxLsR9lQkLIhLiyUOjJwbdApHFEwss/kFMRhBPuR8v1vP+iGsmDkwiOCf1fPvb3+5lhW4H9xcTjdw1BEifOHFi/RrgGsWTHdL7QnlSBowP4/Thhx/WUor65CAfu76A9bo8gWJixATpnXfeqU/YOqXZtev2eImIyPBCF5EK8KgY0RY/fuGT2UiokYcIQPRBI/E0kHzta19LP+J52xAGCIR4BI4AL1vY43F4VRFBvxAE8XieT/b767rQjFbXgO8IoU5FdBksoVhmEZyILerkO+fk3DmdjlcruFdwfejp6Un71Ml4Hj9+PIm74QQTNaytZVGO9ZVJSFwDNiZ0fRWMXD9cQnLrNOcm7fXXX6+lfPLCL/d1f63l3Lfl+5lrzpMk2sK9FpOubtDt8RIRkeGFArsCWFhxT1i3bl2yNi1dujQ96g0RxzJckYcvcFhPAbGLKIj8eDExfDnzlyBxSeBHvhXhXpG/ZBd1knfPPffUrWJspNEW/IwpQxq+02XhFpbAeBmTcgg92hR+nrSVPNpOv/ANj3PxyX5Y6PoC4gLrNmMV1r148bDVNYiXxEiPrewK0ozcnxUoBwgrJiGkP/HEE8Xbb7+d0oNG49VXuD7cMwg66qJOxH20pS80Gkv2uT+awaSCsWbMOZ5ylI8XNbnujFfcB7H99m//dppAlidXCF+Wzms1CWl0P8dYspRefl35TloZXn7MlxVsR/73ysbfT6v7mXuM5QHj+rDRViZZfYFr09fxEhGRkYHrYPcTBEmzdXVlcED45Wsgx/U/ceJES5GKkENUxounIqMFJg0I9v5M0kREpO9owZYRT/klQqyDPNLvq4VRREREpD8YKr2faMEeesrXG/BvbWe904I9+sByi/tGI6rcE52Aq0ejKJXAy415hMvBIu9/t/srIiLVGZUCW0RERERkqNBFRERERESkiyiwRURERES6iAJbRERERKSL+JLjEMHLkQTOYP1eovXFEnMDSYwLdPt8vPC1evXqYteuXYP+Ypd0F5Y9ZK1rGKx7U0REZDThOthDTHkN54FkJAvsRiuFIP6++93vpokKQUHya80KITNnzkz9BMqydB8BQ2K1l762ORegQUziyqtY5AK10QoXUa7cv3wVinwyBrE6RLkd5ZUryuf76KOP6v0vlyV0fnk1lcG8N0VEREYTuoiMIRBJ69evT9tIFUwI0ggtTT9effXVFGI6D5WNWEVcM8Gin/Pnz0/RNLsZihrhvGHDhnpbcnEfIbDJB0Q8IGBJR8zGMVGOYwiME/WtWbMmCWXaS+TN3bt3p/QtW7akiIKIX8jbwTEIcQR5EOdhY9KBuCb/kUceSXVFnksVioiIdA8FdkUQJYSQjlDJIXCwErIFiDtCS5OPOMLSGGX43g2Bx/mizghjHemPP/54/Zz5+WhPlMnbC9HmyM/LlfsdYawh7x+hp/P1gMt9z8vlbSnn9YXDhw+nEPPRZgT1uHHj6pbquXPnFseOHUvH5UJ8oEHcI5oJ8R3XqBHkcUwjenp60sQghDh9ImR8o35wzMmTJ1N/WzFlypSOQouLiIhIZyiwK4BwQ2g+//zzydqH5Y9H56Tv378/RQ5sJO7YEI9hJUQYIZj6A/URAjnqRMCtWLGillsUDz30ULFz586UhzCL8yG+SMOiWWbx4sXFpUuX6nVyjrB0Yj3F7YD03HpKfwmygWtG5OGCAAjGb3zjG/V2YGGdPn16qo+8hQsXFtu3b6+fDyFeFdwaQpjH5CKEdFwDBCYWa9LJJ5hMXA+EbBzXH3Dl2LhxY70tMeHK4dxY0rlGrZ4YkIe1esGCBdfVRdu53jlEqKQft9xySy3l53AsZYL777+/3saYOHEvcG8wKconaCIiItIdxpzARjyG4CiLmWZ5iBIEFYKEdD4nTZpUzJgxIwkjxFwu7o4fP14Xd7llGLGTi59mlMuFMArBloumcp0HDx6s++Bisa1yvvPnz6e+xHkCLJ0IsaiPPrFPaHKOu3z5crFv376Ul8MkA4vyypUrUxsRoghs6kNMXrlyJeUx3p1SdhGhvmgXVt0YI/oOTB4QoyG4mUjkfewrrVxE4vrQb8R1lQkEY4xbCBOVJUuWpKcGTEj6SyMXEQh3FdpHO8tPNURERKTvjDmBjdgJwcGWC6NWefjN5nnhIwvheoC4Q0geOXIkpYdlOYRYI+txIxCNiMc4Vy6Mrl692sv6y9Zf/9kQd7wAh1W6G+IOwZsLULYYzxB3gBDthriLJwkI6tw9hOuSW5t56W+g3URyUVtFXOfQbq43E6VFixZdZ5EGJjhMFD788MNays/h2LLFuxW0D1HP2HVD0IuIiIguIpUIC28jNwAIK264WoTwRuywj2DGaop7RX8I62+4WrSCfNxYwpJbhRB3+PFibabfWKJDePHJPhMI8liVA2s151q1alXdB5vyEydO7OW60gjEHZOF8ePHt+1POw4dOpQmH7y8F9ZqxhzhmE9IEJPdchMZSEJUM9bla8B9xISiDNeOY/fs2VNLqQb3tj7ZIiIi3cNl+iqCuM6XNcuXXwPEDT60+TJpiKFYXo0l1PDBxm8WC265PijX2YjoW75cXbMl4rCkhgWV9uG6kBNLs5Xz8iXbyu2Mc0FejtU87r333uKVV15J/c/7DrFEHGAlDzFeXlquGY363ega0J5oI20vLzMX9eAagRDN2wJVrkGja5dfAyzIjSzX5esDlGMyUG5HlWtQTi+3vXy+uAY9PT29rneza9Bo/ERERKQ9CuxRRCtxJ9IpCmwREZG+oYuIiIiIiEgXMVT6KEILdndo5E4T5K4bo5Xc9aSKy4yIiIj0ZlQKbBERERGRoUIXERERERGRLqLAFhERERHpIvpgjzBy/9ix4A8svZd7jKX2IpCOiIiIDD9cpm+Ewot4BCMZaQK7vFZ11WtVnjQBL+B997vfTeJz7969vcpzHkKm84IeUJbAOLk4JdjM6tWri127drVdhzunvP405OtZE0I+XgyMpe7ef//94jOf+UyvFwY5P8fTdoIV5f1rtDY1QnvZsmXFtm3bFNgiIiLDGF1EZNDBCrtw4cIkmjsFIRtRGQknT4AbwrLn4c+pF3GNaEfMEm2S6I4c160Ijoj7PBR8iHuEMUKeqJ45x48fr0e+DGgLYhxxjdAn+E3Ul4fiFxERkZGFArsCWA43bdqUtu985ztpwzJZBSypUYbl8xBVbHzP6+Aczz77bK/8crn+wjJ+UefmzZvrApf0J598sti6dWuvPNqeHwccSzrQ5ijDRl4VELqEkM/FZn8gHDwhwmOMqHfcuHF1K+/cuXOLY8eOpeNyIT5QvPXWW8Xs2bNre5+AwEZM05aAtiCqgfDtIiIiMjpQYFcEAXflypVkXcSKymP/dhZYBDRuHGGVxBUAdwaEH2HTc7GH8MLKSh4bIjbKcWxPT0/tyL5BfayRHXUi7FasWFHLLYo5c+YklwSssoAFllDiCNUQwgjYyZMnF0eOHEn7ixYtKnbv3l2vsxN3FQQv5TsF14zyJCGEdAjsfCzJ5xrEuCJkuzFZwZVj48aN9bbkk6VTp06l8zABCX784x+nMceyTpvyscTKjrWaUPvlukRERGTkMeYEdm5RLouZVnm4NSBA4fz580l4IpCgWTkE9J133llPRxwivBBYiNcQe+xPnz692L9/fyrHPuIxyuGzjEhsB3Xllu8QoGwIO+ppVufBgweTyEPsYV2eNm1aEqRYXsPqSv0I13BdQLDTp6qW6xzcIsaPH188+OCDtZRqlF1EaC/tDDeR6CvWamCiwDiH4KZv3RDYzVxE4OOPPy4OHDhw3QQCMR1uIrTh6tWrxaFDh1IeY4pbyJYtW4olS5akJwO5QBcREZGRw5gT2AjQEEVlYdQqrxWtyuWCkC0XhSH2wkIcYissyyHg3njjjbTfDurMLd9xLkDMbd++vZ7HVsXijChE/CNcEbAhXCH6TVou6KtAuxChWM67AZMTJjwI6tw9hDbn1mYmPPmTg4GCCQTEJAwQ0UwEmLDQBizacX2CuIZMePpi4RcREZGhRxeRAQQLL9bIZhZTRCG+ugguxGaILSzLiG/2KdtfEUo9uLdgEa0qgANEIW2Jl/ZCOOYwocDyCrmgbAdi8o477khW3f7C5IRJxCOPPFK3VjN2tCefWNDObrmJtIIx5/qXxTyTEa4nLkfhatOIKk8sREREZHiiwB5AsPBisVy3bl3dNQPrZIAoRADi0pALV4T3jBkz0vFPPPFE8fbbb9dyfu6OgotHuJ9UcdFgaTdo5jfcCvylcQVBHMYkAKGeu7HQR1btCMtxFTgWF5Sbb765ltKeRj7YQLuwCE+cOLFuZUdE524YgPiGnppPO2tLr1y5smGdrWjlgx3gBkT9OTE+vPAY7aKd3CtRFxu4xrmIiMjIxHWwRUYIPIFwHWwREZHhjxZsEREREZEuYqh0GRZgnY1w4GUGOzw4bji44DRiKMLT52NjqHQREZHhz6gU2CIiIiIiQ4UuIiIiIiIiXUSBLSIiIiLSRRTYIiIiIiJdxJccRyisu8ya0OCLb4OLL9GKiIhIK1wHewRCYJLVq1cXu3btSpEWh4ryyh8ffPBB8dxzz9WD0cQkgFDvBFKh3WvXri327t3b67qwagfBdv7zf/7PxT//5/+8LlyhXGcjyoIXohxh5/MVQfL6Gq1cEm0FAvgQzAcaTWJG+30mIiIifUMXkREIUR7h5MmT6XMoQCwT9GTHjh31MOTr16/vJYQJA//aa68V06dPT2IUcUpkyzx8OOmIa0Tqj3/845SGRbhZna1oVo6l9UjbsGFD2kd0BwhnwqdHuRDXiH5CqlOGdNq9atWq1F4RERGRViiwK4I1Ng9lnYc8x9KZ53FshL/OQ2hjMX322WdTHkTYc7aw8LaC8lu3bk2hvQkJHiHYoy2ci7bk7cnPn6eXQ4KX+5D3rxGI/EZrVgf0Zdy4cfUQ8PPnz0+fhDGfOnVqva+kc9xguLcguC9dulTcdttttZTG0LY5c+Yk0R8inYkE7Yx+iIiIiDRDgV0BhOjChQuL7du3X2fpBAKPRDrHcOyFCxeKo0eP9rLWzp07N6UjJhG+CL0oh9sE7gqtwB1kzZo16RwXL16sW17ztuDSgIglHYvuww8/nNqPYD537lz9fCdOnKhbchHXEHlYfNtBW+gfYr/R5IB9zkdfT58+nfoOIaTj+HxMAtxKmk0EWpGXiz7lcE7EPeMT3HzzzfWJChvjFE8Izp8/nz6BNl69erWYMmVKLUVERESkMWNOYOdWY7bcwtssDyvmlStXkphsZNnNy4V1efLkycWePXuSmwHCDpGIq8T+/ftTGYQ3YjjKIQ45luPYEJaRV8W6HeBjHFZj/IJxlQDcMPBFjjr5jsCnXtoa7eoE2oUgZ3KASA0xzIawP3PmTDruyJEjDd1E2KddueCFbriI5NEWY5xpY9n/u5mLiIiIiEhfGXMCO0RhbLnYapUXVmpArIWFFLcN3AnCus3ntWvXUh5iEpcERGy4Fhw6dCh9Qi4I2UJMsvE90hHwuYW3L2B9zS3wbN0K+c04hX/z4sWLU18nTZpUtygz6cByHGOAmEfUc+xguIeEDzYvMC5dujRds1aEb3turaa9tDW3aouIiIg0QheRDkGEI1THjx+fLLAhwkKULVq0qJdvMmKSl/1whThw4EDdGov7xJIlSypbpvtDWOARlrQ5J1wfwoUDMR8rZ3QCAvSmm25K36mL8cjF/MGDB+vnYJLBOR955JHr3EMGEp4oMA7RjmaElT3cawD3Hdo6lKu2iIiIyMhAgV0BRDDCOtwrEFuvvPJKEq64Y1y+fLnux4twZD9ATCI+cYUI1w2gPkRc2f93oNi2bVv63LhxY/18uMDQB0TjggULUhrtxGe6HZSNetjC/YI+NnL7wF0ESz9jyTnxAceVpnwc9NUHux0IZ9oY7YBGPthQHi/cdyJNREREpBWugy3XgfsLFnYmAWMdBDe+6o3cabzPREREpBFasEVa8NJLLyXrddnCLSIiItIMQ6XLdQw3Cza+4+WIi8FQhIn3/hIREZFWjEqBLSIiIiIyVOgiIiIiIiLSRRTYIiIiIiJdRB9sSUvusTQeEJSlWwFoGoE/9bJly9KSd33xmy5fW4LHhK94q7y+gC96rAne37pERERk7DCqLdgRKVFx3RrGJyIdDndWrFiRomNGAJtc9LbK6wsRvZNJh4iIiEhVdBGREQVrUrPCSSNa5YmIiIgMFgaaqUjuRgG5y0DuSgBYznF/WLt2bYocGG0ou0ewpvL999+f8qouN0c75s2bl77HOXMXmLwtH3zwQfHcc8+lyImknzp1qrjvvvtS9MJG52sUVCVvI0S/SSdiY9QfY06ExlaWY8bg0UcfTUJ4/vz5KS3qLI9Pvs8yfeUQ7tGHVnmtxrlVW3KG27KFIiIiMrzRgl0BxOPChQuL7du3N3Q/CFcCNo7h2AsXLhRHjx6ti2GYO3duSkfcIZQRs1EOIY5QrAJikhDjlENcP/zww6mNCEmEYNSJ2MVtInjooYeKnTt3pjzCtPf09NRymkM/o74tW7YkUU2Y8T179hTjxo2rC1M+2Se9HVOnTi1uueWWVCfjNWvWrHro8mbk7hqIYL7TX8ayVV67caYtV65cSXlV2yIiIiLSijEnsBFeEZWPDQEWNMvDQosIW7lyZTqmTF6OYyZOnFhMnjw5iU2iACLYEMDTp08v9u/fn8ogvBHKUQ7rOMdyHNvmzZvreYjcXPRhmd63b1/6juV6/fr16TviF0ttlOM74jI4ePBg8fLLL6fvCPQ8rxmMQdS3bt26YtKkScWMGTOSeD1+/HiaNACf7JMOeTm2fNywIu/YsSN9P3ToUNqnzoGg1TgD545JAWPzrW99q94HERERkb4w5gR2bpFlC9cKaJUXVlJAqOE2ALgZzJkzp27d5vPatWspD6HGS3eI47D0IiiDeAkzNoQyYp6N75GOOK0i+q5evVpvR2y5u0en0O4lS5bU24kFG0EaHDlyJFl8H3nkkTR5YD9g7PJ25Bb/wabZOIuIiIgMBLqIdAhCERE7fvz4ZAWdMmVKSj958mT6XLRoUa+Q3lisZ8+enSy8Bw4cqAs7XDkQr91yRwgrO4I/rLPNIB+3EqzYrQir8vnz59NnT09P8t8OsPji8vKVr3wlnTus452wePHi5FoSEw/GjvPSRvrSKDx6J3R7nEVERETaocCuAOIMYR1uBvjwvvLKK0nU4qpx+fLl5D5BHlZk9gOEI+4iuG+EWwdQH37QUY6tkftJJ/AyIGzcuLFeZ+4CE+4j5OOfHRb6cHEhP9wpsNAjmGkjbi+kYaVmPweRHj7hVUGkR78Rvy+++GIay/x8tPHYsWO9xrIvDMQ4i4iIiLTCVUTGCAO1EgYCHpFcZQWUkYqriIiIiEgnaMGWPhOuJqyW4ouBIiIiIp8wqgU2K0bgEpC7SUh3wKqLGwcvcY5Wyy595P7BBUZERESkKqPSRUREREREZKjQRUREREREpIsosEVEREREuogCW0RERESki4zqZfo+/elPp30i+VVdrq9c9o033nB5tj5AkBjWCydQDNEfR/MyfiIiIiI5roNdgiAkt912W0chxlutk0xerEJRFpq5CIUPPvigeO6551qG8WZFFFZHyYkJRHlykNfHuZYtW5aC0eRClyA6a9eu7RWhMZ9U5O2H6AOUyzWayDQ7r4iIiMhoRReREohrxHI3QKxPmDCh2LBhQ/HYY4+l9aJXrVqVhDBCGXG9Y8eOlMe2fv36luI6QDhHnWwhalevXp2WzYt0vpPWjmvXrqXw75TZsmVLMWfOnF7RDhHcUSfpIZTzcohrQ5KLiIiIKLDr5GseR0hxrLh9FYyUmzVrVgr/HaJ5z549xbhx44ovfelLKUDLwYMHU343wFJMSHYEe7B///6U1kkfEM979+5Nod2ZCFTl/Pnz6XPGjBnpU0RERGSsosCugUsIlth33323brHNrbWdEkLz5MmT6RMuXLhQXL16tfjsZz+bhPaRI0eSiN28eXNHgh4XEIK8UGbr1q1JXE+ZMiXVzTmCOHenohexTPsQ5xATjlZtnDt3bnH58uXi0KFDtRQRERGRscmYE9iI5hCLbEMd5RHrNq4huFrgclGF3EVkzZo1XbOCN6OZiwi+4ytXrkzjiMW7nf+4iIiIyFhgzAlsLLAhFtnKL+V1i0bWYyzCWIZ//OMfJ2szVuduUbY6QyMrehUaWcMbET7Y+G3fdNNNlfy9RUREREY7uogMEFh5EaiLFi2qpRTppUbS/ut//a/FiRMnuvpSIK4ZiOKenp60j+sJriPHjx/vyM2FMkuXLk0rsFS1RlM/L3BOnTq1a/0RERERGakosLtE7qccftEsTccqIpHOd9IASzqidN26dSkPV4sqVuNmIIZffPHFtAII9eGjzSoinCdgSb04HxsvdkLu6hErm+SW/So+2C+99FLywe6pCXwRERGRsYrrYMuAwkTDdbBFRERkLKEFW0RERESkixgqfZjBKh24ZDSCJQQ7iTA5lGC5NlS6iIiIjEVGpcAWERERERkqdBEREREREekiCmwRERERkS6iwBYRERER6SK+5DhM4OXG2267bdi9xOjLiiIiIiKd4TrYw4SqArs8eSBcOYFhXn755eI3f/M3i69+9aspPYjJRbM8xPLatWtTEJrgjTfe6BWgBlzPWkRERKQauoiMMFavXp3CrD/22GNpW7NmTRLXwQcffFBs2LChnp9PMJrlIdK3b9+e0rZs2ZKiQSL4RURERKRzFNgVwQIc4cLZcgGKdZfw6JHHsQHHRXo5zHie12zt6xys14RbH0iwTu/du7eYOXNmOp+IiIiIdIYCuwIIzYULF9atvGzhQoFgRmA///zzdQvwww8/nNIR2rh9RBmEK/7MgLhGxIZFGbeMdvzt3/5tslYvWLDgOiEf4DqycePGunDPj2mVl3P+/Pli3LhxxeTJk2spIiIiIlKVMSewc6txWWQ2y0PYXrlypVi5cmU6JgchjXBdt25dKsPnpEmTihkzZhTz5s0r7rzzznp9+EBjgX7ggQeSuMZHnLo7AYGNWwhCfsmSJclyjsAP+uIiIiIiIiLdY8wJbCzPITDLIrNVHi8fkgaI5aeffjp9B0KY5+Vyv2heJMzz1q9fX/z4xz9Oef0BVw7E/sGDB4tFixbVUrvDlClTiqtXrxYXLlyopYiIiIhIVXQR6RBEOK4i48ePT64juFNgrW7kbnHu3LlkZc79rgGr9aVLl5KFGyhbxQe7EbigdBOs4UuXLu2TdV1EREREXKavEgjkfCm7fGk8QCDnS+DhivHcc88lgYqlGzeRIJbAy+vkeCzRd999d8tl+srtAKznUabcDuh0mb5y3wKX6RMRERGphgJbKqHAFhEREamGLiIiIiIiIl1kVAtsXCLy1UCkc7Bcs1IJK6gQLl1EREREWjMqXURERERERIYKXURERERERLqIAltEREREpIuM6lVEiLAIsVSdtAZ/a0K5E7XyhRdeuG61EJYJXL16dbFr167rlvEbiRCoh3XEWy2NOFi0GvvIwwf+o48+anhtREREZPgwqi3YEUVRcd0eJiUIud27dyfhqYAbXCIE/tGjR5OYzok8AhyxTrmIiIgMb3QRkTqERycypQwdZ86cqX0TERGRkYqBZipANEYIV4Jy/fkjfMhdUsp5EcmxHViRI3x67hZAWz788MNizpw5qc6IGjl//vxi+fLlxc6dO3tFmHz44YfrUSVbQZ+++c1vFq+88kov949mUSxPnjzZy10kdx+ZO3duceuttxa33357cfny5SQaaR99xyUj2k8dQKRLxixCx8c4x7n37t3b9jqWI1XGOJfdQPJ9vhPyftq0acmdKI9i2SqPc0VbI0pnXPOye1Ie1RPK7Sy7L0XdjdxWuJcM9iMiIjL80YJdgf379xeTJ09Ogg8Qi+PGjUsihzSEz/PPP5/cUbZs2ZJEbaQjrnG7II+tirhGZCECowwCM3cbQJwi9jZs2JD2Fy9enITf2bNnk7gNEGonTpxoK66B/tGnEL2QC9zoG2K/CnfddVcqd9NNNxW33HJLXVzD5z73uTQmM2bMKM6dO5fCvSNky+PMJwJ93759ab8ZiNqFCxcmF4pOxhm4lkyUKEO4+kWLFtVyWuchrA8fPpzyEMlc8wceeCCJa8Y82nHp0qU08QCu69KlS3u1szxx4AkCwp4+iYiIyMhkzAlsLJMEn4kN0RM0y0O8XrhwoS78ELHHjx+vC2yslevWrUtl+Jw0aVISjxyHYG1kfUVAbd68uX4uBGHUjzBGwEUeFs8JEybURRdijzYhnBFwiFNAoE6fPj0dR12U2bNnT8prBVZxhPSLL77YyzJaVeA2gn5TF5Zf2pWDnzFCErGet688zowDArfdBIF8Xg4kGA7XsBMQ93F9jhw50kvctsrDMh3jwjHr169PbYC8T4jwuHb0J65dM8jjKQKTp077IiIiIsODMSewEbJhPSxbEFvlIZQQSAglRCyCK0CI5eV4Ia2ViAJEIaIsyiCmcnGLVTSvk2PbCc1Dhw6lTyyviFTEd15nM3BHwAVl1apVdXE7VDDOWIMfeeSRJEyrtB/oA+METErCrWekwVOPRx99tNi4cWNlK7yIiIgML3QRqUhYK3HHQLiGgMYSi7U6t4QHiPCpU6cm0dQJuE0sWbKkY7GLAD9w4ECynM+ePfs6y3ErsBzzkiN9CegbLh4IdiYWCPDwxQZ8wKdMmZK+48KS5/UVxpl2/Nqv/VpytSgL7HjK0Gi8AVGKC0ZubQ4LMtdhwYIFKa0RuICcPn264USmVV7AGEJPT0/65JxMFsJNh8kD7j2trivjiSW83WRKREREhi8K7IogeBC++ND++Z//eS31k0f6+BPjxhEuHbh+IK7Iww8Z14XIq/LYH5GIP3W4nVQtBwhSRBz+1O2s6O2gPO4ctB+LKu4NtAs4D3nRb15cjLz+wDgjZJmY5E8JglhlI9xiAMHKmMVYIfZxs6Au3DWYJNB+XgL94Q9/WCv1CbkrDuRW41Z5jeB8uNkw/pThnEzGohxPRBiz/Lo2myiIiIjIyMVVRCTBmDVaRWQoYDIxc+bMhqufIKarrizSH2hDvvrIYIHgxhXJVURERERGLlqwpQ5W73wVkqEAAY0FuNHLjfhVY/0daHE9lOSWeRERERmZjGqBHe4LPoZvD2IWyzU+yrg0dOr/3V+woONag4DGjaKRgI4XGUejuMY6vXXr1voSjDmRh6tOrKcuIiIiw5dR6SIiIiIiIjJU6CIiIiIiItJFFNgiIiIiIl1EgS0iIiIi0kVG9TJ9hDAHoiJ288W4bi6XxsoYrPlMJEWXXvtkmTpeTgXCkTdaqk9ERERkOOM62H1gpAnsWDs6j7T4xhtv1AOg5KI2IKIiS/YRhTAPsMKxRJn80z/90+Kf/JN/0qvOjz76qFc/6BvBWgImOtSbT36gkZDmPERBVGCLiIjISEMXkSGGpecIajLQ1utr166lEOIsc7dly5a0HFweHRKRu2HDhpTP9h/+w39I4pqAL0xYAoKgsIzerl27UnnqunjxYqo7+hFL7hGiPK8zn+wgtiN9/fr1imgREREZNSiwKxJrEbOudnk94jwv1t2O8N18Dzju2WefTXl5mQitnlOuMxfDWIYjvVHZdiCCCdZSFs9lCFVOmPH58+enfdo9efLkhiHMczieoDWEDVc4i4iIyFhDgV0BxO7y5ctTABAsrlhrsQhDiOXnn38+5WHRxbUBsPRi8Q1wubhw4UISuAR1WbNmTbLklqG+r3/968Xu3bvrVt5w00Bonzt3rp6OlXnFihUprxPOnz+fRDCCGXDZ2LhxY124MzE4dOhQcfny5Xp0R/p69erVlN6KKVOmpOPoazMiCFBfJwkiIiIiw5UxJ7ARqCHsQkgGzfIQmGfPnk2iuAyiE3FKBELK8Dlp0qRixowZxZ49e5KbBMcgIKdPn17s37//k4It4HwnT568zn+cOrA633///fU28v22226rHdF3yi4inBvrc+4mwmSB/XZW6SrhvnURERERkdHKmBPYWIJD2IWQDFrlteLdd9/tVQ7LNGIcS/WlS5eSwA43i3bW33ZgGQ5f6tjw4+6UKlZmYJKApXvx4sWV3EPgzJkzybUkrOMiIiIiYwldRCqAYGSlD4QyG+4i4YONqwXW6twSnoPFevbs2ckqfeDAgUqWWkQs58NVJIeyV65cSen9camg/NKlS9MqK+3awyQBEf7II49Ucg8BVgrBtQQ3FxEREZGxhsv0VSSWnMP3Gt9o/Kx5iQ8BirjOl7nLl52LtkC+5Fy5DOTlyvmxrF7Uly9z126dbyYF+TJ99AF/8nB5adSWvM7Iz5f2K9cJ+TJ9zdrpMn0iIiIy2lFgy7BEgS0iIiIjFV1ERERERES6yKgW2Lg15KuByPCHa8U1K7usiIiIiIwURqWLiIiIiIjIUKGLiIiIiIhIF1Fgi4iIiIh0EQW2iIiIiEgXGdXL9MVay+3WiR6usPY2AWdibel2xHrVQHTJvkR4HGvE+uaQr/MtIiIi0ldcB3uA6c96zp0K7OB3fud3ittuu63fAjsEey48adO5c+daClHOf//999f2fg71EKWSCI8RCTMPekOEyTwPykFoou7ypKlVu3IRDY0mXFX6JSIiIlIFXUSGMQhkBGUn4rrbEJ1x1qxZKXJjVRCpjz32WLFly5bi4sWLxfbt29N+iFfqJI+0gwcP1kO/I7LXrFmTjqccx6xfv74urjmGicNrr71WzJs3L6W1g/EDzhXbSHyaISIiIiMHBXYFEICbNm1KG2s0V1lbG0GKoMQCjKvKxo0bUzkspRB1Pvnkk/U6QwySt3Xr1pS2efPmJCyDPK9RfjM4hmOjHG2rIpovX75cvPfeex0J7E44c+ZMcdNNNxWTJ0+upTRn/vz5xdWrV1O49QkTJlRqE4JcREREZDBRYFcEV40rV64kCyguBrh9tBK2WJ0RzByLm8OGDRtS2dxtI68Tq21YisOSS9kykRfW2EuXLhWLFy+u5TaHYzg2ynViGT969GixcOHCSkK+U7BEnz17tlJb5s6dWxw7diwdS1+qCGzcTxhnJhUxgREREREZSMacwEZkhRWXLbdEt8rDrQGxBufPny/GjRtXt7q2KtcK6tyzZ0/6jnD+1re+1VZoIiqxPse58C2eNm1aLbc5tHnGjBmVLdc5r7/+epoIVBHyVbj55puLdevWpfZDFV9x2swEhH7A4cOHK7mJxESHCc7MmTM7uj4iIiIifWHMCWwEZlhx2XJ/3FZ5rehrub7AS4BYfONcrBZShbB888Lk2rVrk5sJ7iZV2b9/fzF79uzaXv8IH2ys9oj+KpZlBPbEiROLlStXJpGM6w2W6aqTBfy48eeu8vRBREREpD/oIjLAlK3d/QFRiO8xq10AAhmB2glh0T158mQxZcqUWmp7Dh06lM6NqO0WiH7agWW5neDFWs0qJPlEholGp9b4KtZ+ERERkf6gwB5gEJEXLlyou0TES46twIUhrLTxgiQvKAJLD7JMHfnLly8v3nnnnZQO4apCPq4j+fnKbizQiaUdC/CBAweSe0c3we2GlxxxP4kXOLFSY61mzOj3P/pH/yhNUFjiL4eJRm6NjnFho55/9s/+Wa8XO9kQ831ZMlFERESkKq6DLfL3uA62iIiIdAst2CIiIiIiXcRQ6dIx+D3zomQjd5E8MuNIAMu1odJFRESkm4xKgS0iIiIiMlToIiIiIiIi0kUU2CIiIiIiXUQfbEnLArIkIBC4pkpkxf4S56x6bfJrOtL8vKX7+DcuIiLDGZfpG6YgQFnjudM1m1nv+rbbbuuTSO60bH/GuVOBHfCC5erVq4tdu3b1S2BH2y9dulTvLy88EkiHaJcE5IljWKc70lirm2iau3fvrrebcZszZ079mEZEuRtuuKGW8gkffPBBusaQC8ZIX7FiRVrfu0z+QmaMZfklzb7cC/lLnzlcJ4Im5X3IJzqN+hd94P5tNJbQ7nqW21O+X0bD37qIiIw+dBGRIQExRDTGoRRFiGtA5EVEyMuXL6dPmD9/fgoSlEeMRAQirpcsWZLSEJaI6507dzYV10A5QtVHePuISkn4dgQoIpP2RJRKvpMWYfgJLX/x4sUUXp79XEgTwv61114rpk+fngRnf0CMUz/n4Xycl/24ThHmnrSDBw+m/nPO6F9eLvoGjcayHUwQgHPFpogWEZGRgAK7AljRnnzyyRQdkO9sRAXEchj5ebRA0hEdRBFkC9GDYEAYtRIY5HFMHsWROjkHRL1xrqgfoUP78iiObCFSIr9crq9QL22jjbSVOqmf8wDtffzxx1NfyIt+5+3Pjw8ajWUVOC4vF/1uBpEhCf1+6tSp1C42vufMnTu3OHbsWHH48OEUqj1A5CEUEcDLli0rjh492tD6WhXGgPZgDQ7279+f0lrdK0A+ofj37duX9hGyg8WZM2eSRZp2tqPZWLYC67uIiMhIRIFdkc997nPJcjljxowU8Q8r5LRp01JeWP3YsOAtXLgwpcejfx7zI6JmzZrV0o0AyEMc8iicR+wbNmxI9XIOxCmPw0+cOFE/X1g6w4KIZZS2RX5YOnMLapQjPHlfoV7aRhtpK3VSfy4077vvvtRf8hCkPT09yaKJZRMLJ9bQMo3Gst1EgHyOC+suW/S7HcePHy/uuOOOtL311lvF1atX0zWmTgQe14MNMZ6LXcQw7hAc/9JLL9VS+8aUKVNSPVh4g5MnT6ZP2tIK2sT9SBtPnz6dhOxggVDmura6n6HdWDaDMcZlp8qESUREZDgx5gQ2P9S5pTO3kLbKw0qJDyqicM+ePbXUT8jLrVy5spg4cWKy6iEmX3zxxeRCgH8qfqYhRhAduSU6LLytCEthfn4sggiWdiKUujlHnA8rd0wQWtGpZTiHMYv+IpyriN5mY9kKxvnKlSvp+E7aB1itEbdsTEwCJh+MK6KXPjAhya8Pwnf8+PGVLbgDAdccP30syXDkyJGuuIm0guBC69atS9cHuK7taDeWzeBYricTuZkzZ6ZzVn2iISIiMpSMOYGNyAsrJ1vu09kqrxlYphHQYT3lk5e/cthHAGKpDMKSG+dCSIQYHQgQ+Fgb43y5mGwFYxBl2KqI5L5SZSybEZZvQIiFS00zEMhhNaZP5X5hnc1ddJiQhGsDApa2MoHAZ5ux7Q9M3HDzyIV6WK7Dkt0I3EEmTZpUd9FhgoHFdyDdRMIHm2tDG6tMaFqNZRXib4UnJUwoBnICISIi0g10EeknIZpDCC1atKi+kgJCYNWqVcX3v//99BLcI488UslyB41EV7gQ9PT0pM+wYOIygggBrJllizbfScOVABCH7VwPqsA5sUZWsYRXodVYVgWhjPjDutxXIYYVmHEPoc+GqGQMuX64/AD5uMQwlv2xrB46dCiJ/fy6co1wX2k16cIdhLGKNrLx4uFguInQb86NZbnVODNercayE7p1n4mIiAw0v3DHHXf8u9r3UQNuBV/60peSAOiGVfjRRx9NlkoswJ///OeL119/PVkJSfvv//2/Fw899FCxdOnS4mtf+1o6BlH43nvvFb/1W7+VBOgf/MEfJLGEdfaf/tN/WvzsZz9L+60gnz7843/8j1O9DzzwQPHHf/zHqU9YLLGa/vIv/3I6H64mASKccr/6q7+ayt11113F//gf/yOJ13/4D/9hSsMXnPZhVf/BD36QrJD4TyMssYjG+V599dVarc1ByDI+v/Ebv5HOGX2LMaP+HEQVj/y//OUvF5/61KeKBQsW1Msxro3G8sCBA8VnP/vZpuVw0yCPMaEcAhN3HPzRm3HPPfckv/o///M/L37605+mNO4bRD1uEJz3T/7kT+p59JNx/Qf/4B+kMSWP+unr3XffnfKqXFcojw3naHdd6f+DDz5Y/PVf/3U6B6L2K1/5SvHmm2/2usfpE8fxhIJ23XvvvWlMYrvxxhsr/U0wPnGvv//++w3TaCOTxr/7u79Lk4x//a//dRofrg/p3G+cD//rRmOJVftHP/pR8YUvfCHdb9FGylE3rihY5SOdCWd52cpu/62LiIh0A9fBFpERi3/rIiIyHNFFRERERESkixgqXaRL8GJlHnUwpxxlcajAtxs3lEa+7bzA2G4ZyeGCf+MiIjKcGZUCW0RERERkqNBFRERERESkiyiwRURERES6iAJbRERERKSL+JLjIMDLb0TY6/QFsr6Wa4QvhYmIiIgMDqPago2IJHLcQAtJgqc8++yzaYWG4UqEmyYgywcffFBLFREREZFuo4vIIPDMM8+kaImdWqH7Wk5EREREhg4jOWb85m/+ZgpXHYQbBZZpwlufO3cuhUiHWNe40drHsZ4w4aNjzWGsxhHmGYv3qlWrUlhzwk+/9tprxX333ZfKtioHCO77778/fQ9yd4+8PeWy0NexEREREZFq/MIdd9zx72rfRw0TJ04s/n7iUJw8ebKy9RdxvWTJkiSat23bVly+fLn4yle+Uly7di3lP/zww8XPfvazYt26dSnvC1/4QvHee+8Vf/iHf1j8zd/8TRLNf/RHf5RcRf70T/+0eP/994vjx48Xf/Inf1LceOONSTQjpH/6058Wn/3sZ4tHHnkk1fejH/0oCeZXXnklid/Tp08XL7/8csNy8IMf/KDYuXNn2sj/8MMPi//z//w/Ux7im7Z9+9vfTvlz584t5s2bl8oEfRkbEREREanOmHMRQYR+5zvfqW8Ia5g2bVpx9OjRuug8dOhQEqtTpkxJ+1ild+zYkb6Txz7it68g3Pfv35++90XsYlWfPXt2PTog4nzmzJlJrEff+H7bbbelfBEREREZHMacwEaQ8uJjbCPRTQJrOQL7v/23/1ZL+YSrV68W27dv79U//LhFREREZPDwJccaZ86cKebMmZPEKyxevLi46aabGlqWyRs3blyyZAO+1IjbsHYPNF/72teSG0neNvysr1y5koQ31mwRERERGRp8yTEjf4EQFw5cQhCyiNZ46RDiJcZc4OYvSEY+Yj1/aRJ48fC73/1u8eu//uvFrl27kp80bhyca/Xq1SkNod6oHC8srlix4rqXHOOFy+h3rHUN5fWu+zo2IiIiIlINBXYFENjLli1LLz926is93FBgi4iIiAwsuoiIiIiIiHSRUS2wcbPIVwoZy2C53rx5c7Fx48ZeLiQiIiIi0l1GpYuIiIiIiMhQoYuIiIiIiEgXUWCLiIiIiHSRUb2KSPgal5eqG81E34Fl/Vgfe6B4+umnizvvvDN9H4wxzpdCjGULB7J/IiIiIn1hVFuwEX0jNVpjK1ivG3E71BAlcsOGDUnsdgLrgz/77LNp+cNO4DpyPbmuIiIiIsMVXURGGVh0169fnzatuyIiIiKDj4FmKoCl9dFHH03fp0+fnj5zl4jcdQGqukvkLhZ5dEgs1OPHjy+mTZuW3FwiqiTkESWDiOSYt+Pdd99NFmbI2z916tTihz/8YfHQQw8VP/nJT1q6WWBpXrVqVXHs2LHii1/8YkrL64VmY12OfhltzPsc5H0vj2WUy+GYhx9+WBcRERERGZZowa4IwvTKlSt1FwUEHuISsbdkyZJiy5YtKW/79u1pv537Q7h4UIbt6NGjScxSJ8yfPz+JVvIOHjxYLFq0KIVtX7NmTRKdCN0oGwI0XCjIL0P7T58+XZw8ebK47777it27dxfjxo0rJk+eXDuiMTfddFMxe/bs5ApCHzm+Xd/IX758eZoU0B7KzZkzJ40V4pz9ixcvprEinwkF4pq+L1y4sJ6e901ERERkpDDmBDZijuAzsSH6glZ5WFnDinz+/Pm6OMXKjDiOEOqHDh0qLl++XEyZMiUJRoK7RH2IRazCpGOh3r9/fyoDR44c6SV4EdBhESaP40N89wXav2fPnvSd9tKHgDbRtmgnbY5zYT1H2GMppo8XLlxIfWsF+WfPnk3lgHLsM1at4BxMYlauXJmuhYiIiMhIZMwJbIRkWEfZcreGVnl9AcGIL3TUF5ba4QZtom3RzqH038bCTRsAsT8cXuYUERER6QRdRPrJmTNnkvsDVmBYvHhxcqtoJaTDUovbR8B3rMONypGHe0eIXs45YcKEflm0+wKuH1jY200SsI7jkhKuJHyyjyUe6OfVq1dbWsKZ7OAq0l/LvYiIiMhgo8DuJ1i5cblYt25dsrguXbq02LlzZ1sRum3btiSSwy2D76QFvAgYeYDgDPbt25c+N27cmPLDnSJcXO6///56+f5agHlJEZcN6uKlxV27dqW+IXpxJaENvIjJi4kcg1sNriF79+6tl+OT/XAZYaKAf3mUoW9MUMquKpzvlVdeGTJruoiIiEhfcBWRYQhC+bbbbuu1WsdQgOBdvXp1EtUhjocDiHhXEREREZHhihZsEREREZEuMqoFdu62ICMfriPXM18nW0RERGS4MSpdREREREREhgpdREREREREuogCW0RERESkiyiwRURERES6yKhepo/1meF73/vekC/Xx3rUBFt54YUX2q6RPVyW6RsKeJExXmL84IMPXIpPRERERhyugz1IDKXAZj3rtWvXFpcvX64L1qprSdNugtYEb7zxRj3oDREaCQZDMBqIvDjfzTffnNIhL5eLaHj33Xev62vV9omIiIgMN3QRGSQQkAjnduJ6oLh27VoK4U4o906g3Y899ljaEMkBInr58uXF7t27U96WLVtSyHiEMXA+Qp2Tx+esWbNSGUT5I488ko6PeseipV5ERERGLwrsCiAKN23alLYI4x1CshUcE8ezEVoc63qAdTjywvIbfPjhh+l48rZu3ZraABGivFm5Vrz11lvF7Nmza3v9o6enpzh79mz9CQETB0LGz5s3L+2X+eijj4oLFy4UU6ZMqVu8RUREREYjCuyK4N5x5cqVZHHFpxv3hVwsNwLxGVZarLhXr16t5XwiriHyEaerVq2q1zl//vzk4kLewYMHi0WLFqV0LNCXLl2ql+vEKn7q1KliwoQJdbHeH3BhOXfuXG3vE86cOZPqZ6wQ0StXrkyTANxIXnnlleTqwZggzNetW3fdhENERERkNDDmBDaCNKy/bLklulUeFtgdO3ak7+fPny/GjRtXTJ48Oe23KtcIROX48eOL/fv311KK4siRI73qxC85rMPkcTzlOPeMGTM6slwHH3/8cXHgwIG6WB9IchcRxg13khD24XZy4sSJYuPGjfXJhoiIiMhoYMwJbIRpWH/Z8pcgW+W1oq/l+sLLL79crFmzJr0syYuEuftIFfbt25c+Q8j3FazXWLFzpk2blqzrWKhzDh06lF6wxD0kh3HDF5u2dMOqLiIiIjIc0EVkCMBVAneT3JLMd3yUG7l7kHf69Oleq2lwHJbzkydPXidcW0EdiONmvtI5WMwff/zx+veZM2cmNxDAqo4rSAhjPnnJMbfKB7i78IIl1vcyWOP1yRYREZHRhAJ7iNi2bVvyVw63Er6TFrA0XuQB1l4ou6NApxbzPXv2VBK1iHFEP+fBlQPrdJwLS/revXvrftZ8sk86lH2wd+7cmfLK7c/zREREREYDroM9SOCX7brO1XG8REREZKSiBXuQwCUDC7BiUURERGR0Y6j0ASSPgjhcw36XIzXm5NEXBwss14ZKFxERkZHMqBTYIiIiIiJDhS4iIiIiIiJdRIEtIiIiItJFFNgiIiIiIl3ElxyHIQRtYX1o1pIezBf94rwEwSFSZKOgN92Gcy5btiytAT4Y5xtKWAOc6JeEih8MON/999+fwu6Xz+nLpCIiIgPHqLZgI6wHOnT5QBDh0Gn/YMGkBLG7e/fuJMxGk9hFTG7evDn1cbRBn+gbfSzDCjAbNmxIQYzK+fxN8LcxmPeYiIjIWEEXEalz9erVhuHMZeSCVZr110VERGTwMJJjRfJH6pCvEV1eSxqrIBbgVatWpVDj99xzT/Haa68V9913X8rH/aKnp6cYP358MW3atOTKcu3atWLHjh29QoZzzkbRDJutr83xBLSByK/qHsOYffOb3yxeeeWVXm3gXKdOnUptv/nmm4uPPvqo7j4SLghBjMkv/dIv9ep7nge5CwzkdZbHOdpPOz788MNizpw5xcmTJ1MefWzVP9qxdu3a1O6ccJmg/a2uQd6/vI2tiHsv3JPK4xUuIjEGP/nJT9L1g7xcXNfJkyensTx27FjxxS9+MeXl7c/HHxrdR4zd4cOHG45Ts3tMRERE+o4W7AogmhYuXFhs3749PVZnywOwIHYinWM4durUqcVNN91UjBs3LonLhx56qNi7d28SQDNmzEjl5s+fnyYBlDt48GCxaNGilN4KRNW5c+fq5ztx4kSxYsWKWu4nohMxRR7iE/FE+9uBkKOtIV5zENeIROo8e/ZsmhwAYxDt2LJlSzFz5swkaoG+UycuCozJrFmzUh7Ccvny5UkEUo48xgQQe0uWLEl1RR77lIHPfe5zyYWF8WMMEJqI42aEqGUcEKy0hXpzf+Rm14C2IIZJZ+PaIYjbsXjx4mQxjnKcvyzK8zFYv359SkNccy2jHHWsXr065TGWs2fPTu1nbBhX6mD8SaNv9JFyuBbl4hoYq1bjJCIiIt1lzAlsBM93vvOd+oaQCprlYdnjxb+VK1emY8rk5Thm4sSJye8V4bh///50DMK1LLQQiGFVPHLkSLKmthLD5CFisVrG+fiOEAwQW/v27UvfqRsB184yiYUTS++LL77Y0EJ79OjRejriNCYXjE+0Y926dcWkSZPqkwf6jtDj3CHayZs7d24S6WURCIjA/FyHDh0qLl++XEyZMiXtk4cLC1bhPXv2pLT+0uwa8CSAyUr0D6s617TdZIX20U/GKCYbOdSJuGbCEmOAYIa8T0ySOB8TtXwsGRueDMSYVCGu19atW+uTFRERERk4xpzAzq2ubCGuoFVeWKkBwYUoBQQLbgth3c4tsgMBftJxrthyi2xfoDyCD1eERqKwERyHdTksp1hWEb6jiehbbFUmKwhhrMiMJ5OWsqhF0DNRYKyrPFnoBjEpbGTdFhERke6ji0iHIMIRuGHpDEtiWGlxMQjf4k6g3OnTp1sKOPKwpCPYui3OsIoi3sMC3Y44Ll6K7Onpuc7XuRFnzpxJVlkEOhvW3Bgv8pishMjH3QL3iEZW9UbEk4T8qQTQRtxfwlLcjPwa4FbBBKLZhIN07oVmq5PQZtrDfVG2NlMudwFh7IExBOrDtQeXEaz9OVx7+hFjQlupq5ULCE84GFsREREZHBTYFQgxFe4C+OLyMiDiBncM3BhwkSAPkcp+FXIXBOAcEK4XuCXw0tvGjRvrQo71ooG0KFsWlIMBllDEHy4xtGH69OnXicFG8FSA4xivJ554Ivk2x3iRh3U3xnLp0qXFzp07KwvsEJFlsUlbEbFRbzx9gGbXgM9oZ+Tn7kHUR7vD1zzI3YWizvxJSMB1xAWE81Ae9xwmF5Th2iKaoy1MQGKcufd27drVa0xwQ1qwYEHK1w1ERERk6HEVkSECITaYQUfawZg1WkVkJMFECLcMRHuV697fa0B5fOIHcgUO+oSlG1Hd1+viKiIiIiKDixZsqYMbBS8hjkQQkVibq4rr/oCFGEvxQIvrbsDECUu5iIiIDB6jWmDjYsFj86FwoRhpIBKxkOJqgGtCM9/j4Uq8hDoYTywYJ14YrPLS41CChT3cTcrjkrshiYiISHcZlS4iIiIiIiJDhS4iIiIiIiJdRIEtIiIiItJFRvUqIixxBwQMGQzf3JEAL+ix1BvraRMMheXeeEGQdZ9jWbiqUI5l7qDTMcYHGP/fkXJtyiuOxD0Go20FjriuhPgv3xNx3YCooa4+IiIicj2j2oIdkfgU15+AKERg7969OwnGqutLNwOxuWHDhiS0ZPTAdSUy56xZs65bU5u/Jf6m+NsSERGRxugiMsYgEE5EXxxKQqiN1MkPVltWERnuK4n0FQLpjLbQ9yIiIoOFgWYqgBXv0UcfTW4U8+fPT2nx+JxzEaDlnXfeKR566KEUdS8ena9YsaKXW0HuZsD3W2+9tbj99ttTiHHETLhsxLH3339/+p7n8Yh+3rx5Kb1T94xoazmYDC4BCO9oS/7oP28HlN0GGo01ZcprRIcbyksvvVR337l27VqxY8eOeltinIHIkJD3LXdPCBq5MQQsNbhq1aokFu+5557itddeK+67776UF+PZbJyh3Pd33303Xbu8HZEWhAtOhH/Px5Ix+PDDD1PExvw+aSXQy32AvM/l80Vep+XK91CzeyVgDAxQIyIi0hgt2BWZOnVqccsttySr6/bt29Pj81grmgAts2fPTmsO4zIBixcvTp+tQPgQGIU6Ccvd09OT0hEvCHHS2TgGMRQgrInMRx7CCKGDIGoHIblp68mTJ2spP+euu+5KYcnL7UeQRTtwG0A4t1sje8+ePek8MRnheM595MiRJMaw+lJXIwsp44x/eLlvCMIlS5aksScPsYi4bSauA0KZ0xaOZwLEWCLsZ8yY0XKcY5LAeMT5AoRoOS2INbKjTtagzu8FxDWTik7ukwjHTpn83mNMli9fnurjXIwp9dOvVuWi7PPPP18vxzjn15XrxHWYMmVKLUVERESqMuYENsKJABuxhRiBVnmIQYQMHDp0KO0j0gIsuIiSEJC5NbAZCMQ4DusuYg+wUCOiox1YS4nGFyIay+e+ffvSd8pXcVPAekoY8RdffLGh7/XBgweTOKQeROG0adNSOmMQ7SBS4qRJk3r1uxHUf/z48XpUSIQbltRGltAy+TjjyoI4RiQ2Ih+TZiCm9+/fn74zscj73mycH3jggSSu45p2An1F9Eed1B9jCc3GuRX0IcrE5IhrgPhlYhbjSt/YjzqblaONPEHgera6rljmGaOwiIuIiEg1xpzAzi2ybLkQbpU32GC9zdvSX19fxBLuD7gNVBVLHIfVONrSzOrcCKzVuHkggBFpWNz7A5Oay5cvFytXrkyiEEstk4X+jAk0Gucf//jHtdzOwQKOyI36mEQNR2hX3m+s7uUJEJMyrhsTz0aTMhEREWmMLiJ9gMf6WFYRfe0IKyuP5AlDXgWs2QjbblsNsSLja93OAh3EcfFSZE9PT/LRrgJiLXePCIt7X6Ee6gtB2Ej0xROI/MlDK5qNc1iXw9ed+nJf7GZwnbne1Atc86pjDbSDSd7mzZvbWuaB64JLDecBPtlnctMKytGuVuPE+cePHz8sXogVEREZaSiwK4KwjEfqiLIq1lN8kfGDxTcbX9kf/vCHtZzWILKwgsb52BCPgw0imXaE1RiLNPuAAEMI0jfcDXCvKIvbY8eOpXSsoDFWISLp28SJE1PdW7durYvEZiDQEa8xHmzUk4vjM2fOpM8qbhfQapxxU0GskoZ/8quvvprSIYQ8ojtcTLD20kfcSkgnjWvOy69VYQKElT58p9vB9cFvPK4Pn+yXLdFlyGepxrhmbFVFvYiIiLTHVUQqgPhbtmxZsW3bthH9qJxxabUyxHAG4Z6vWhHX+MSJE0koA2IbP3NE5lC69/QHxHt5BZahoN29Ur4eIiIi8nO0YI8xcG2Jlw9HEmWrNBZeLL1htcaCjCV6pIprJnFY8oeDuAbGt6o7kIiIiPTGUOkVGC0WbKAvvIiXh0ofCZSvKeTrOkv3YLKC60uj8cVyjWsJVFnHW0REZCwyKgW2iIiIiMhQoYuIiIiIiEgXUWCLiIiIiHQRBbaIiIiISBfxJUcZEuJluarXJr+mhABnneqRttSgdA//xkVEZDjjOthDyECsJcw6yrfddlsKjT7Q9GecOxXYAWtdr169uti1a1e/BHa0naX+YjWVqqvFlMVdCH5ghZYbbrghfQ/y1TZirW4CyuTXPVbuCAhlzjXkejaKIskKHyxRWL5/qIdIkgQ54jz5UntVVl0ptyPgOhHVMe9fPtGJ1Wnyvuf9bjTe0O56lttTvl9Gyt+6iIiMLXQRkSEBMUTI86EWRYjEnp6e2l41EIQEuImw7WvWrEnikI3vpCGQEbR8X79+fV0AIyjffvvtFLJ+/vz5KS2I47ds2ZKiSDIJQRBH2sWLF4vt27en/SrLE9K3OJ5PQvVTZysQ9XE85+O87Md1+uijj+ppBw8eTMIakRt9z8vl/aavRKokciZjUIWIqsm5YlNEi4jISECBXQFExKZNm9IWoaXbCZUAC1yUiZDgCAwEEhZcrKCEGyefYyHyoxzfQ5RwXo7L6yWNeqk/D9/NFiKF4+M75Pt8f/zxx+vnzM/XDMqWw6TnIc+b1YkYIyx3+fgg7xdb1XHmuLxc3tdWEMYdiz/tqgLHEbK9r8yePbt48803i9OnTzcN+BMh06uGfK/CoUOHip/85CddrRMLetWw7vSV0PmM97x582qpreG6iIiIjEQU2BXBokhwFqxoPKbm0Xw7UYZ4HD9+fLFhw4ZULiydPB5HAFIPj9EjP9w6Ij+sdkePHu1lZUVAI1TytiCgqB8rKNbTKIuwrcJDDz1U7Ny5M5XBypifrxHUS7tpP22gXPQvuO+++5I7QF4nFk0sm1g4sYaWCQsqG9bQhQsXth1n8jkurLVsVfuNmIXFixenz3bQfvqINbiTCQAwwSCSJtfqyJEjxfTp0xv2DWsvwpVjusVA1IlQ5rq2cqcB+ohY5jg2JijtJnCA+wl/d51MmERERIYDY05g80OdWzpzgdQqDzEYfrb4oiKUwnLXrNzJkyeLiRMnJiFattS2AkESVl42rNK5NQ9Ru2/fvvSdR+b5o/i+wuP+EMdh1YW+WoaBiUGIL4RzFdGbj+XKlSvT+LWzkNJ3Jj8c30n7gv379yeBfuONN9ZSWhPuEEwSlixZ0tAS34ie2qSF9iKyIXcT4TrTb3yZmezkk5W+gD80Y9LNOvHpJiQ9dUJMClvB5AVRzWSG++HSpUuVBDbHcj35+yGEPOfsZEIjIiIyVIw5gY3ICysnW+7T2SqvFc3KhUDAlQIBVlUgrFixIn2GZRur9FBBX/K+VbUM9wXGaM6cOXVLNJ/4EVchLN/AOONqUhVEJ6Lv9ttvr6VUI64vk5NFixbVUhvDpAmRGO473BNYsHM3kfDBZmK2fPnySiK0EbhuQO6DXX660FfCB5t6Z8yYUWlCg6U7d4ViDKq6iUA89aj65EhERGSo0UVkEAiBgIDKfWDLlvAA6zGCj3KILERnVRBXWAsbiZDcKp2vzJBDOUQMVux20D7a2S2/3ilTpqRPBCYgWssrcrSDCQDiD9ecToQY/sG4tPSF/OlCM7BUc63jBcGYQDRyE8GiTvvbCWzuH9w+wgrOBAWXCtIHGsQ614lJQ6txpg/c3yH02RiDqm4iOd30HxcRERlIFNgDSNm9AjHy0ksv1XI/ESk8No9H7mF1RWBhHSTtiSeeSKtOVCVcR8JaGBbGvM7w2c4J9wTKsUJGVes99YY/chVXCUQVIpg+4/6BC0OUo+283BfjwUob7LcrF3mUYcMd4pVXXkkTgKrEudsJ+vK52KCdqwSW6nCRCMKFqOz/zX2BVbydtZbj9u7dW3cD4ZP9bliqq4DLFAKf9nMduB60gT5xnXBz+uIXv5iuY36/hd97T81lJndlYaPcAw880MtNio2/n24uaSkiIjJQuA62JGHP2skIR5GRhH/rIiIyHNGCLSIiIiLSRQyVLlqwS+BW0yh6IkSExZEKrhzliIsBLzDmURaHM/6Ni4jIcGZUCmwRERERkaFCFxERERERkS6iwBYRERER6SL6YI8wWPrvq1/9avo+WP7A4bfL0nH6aY9+8nuMqKEujSciItIZLtM3QuFFPAKcdEtgt6pvOAjsePEwJktVr3Gjl/oI+ENAHsKc5y/1RZ2sA04/Y5+1nvPjqHPZsmXFtm3ber0QyBrZq1evLnbt2lVfixqxynrWIVI5Zu3atWnN7Uhr9uIh4va73/1uyiNEeZC/jFguS99oe4jk2If8ZdZcREOjyVq57SIiIlINXUSkLYhFQm0PlbgOEJYLFy5MwrcTIrx3RBKkHyGMEbxBRFvcs2dPfZ+gKGfPnu11XH+gHgIHEXwlIjDG+NI2hG6ETCf656uvvpomF7T/4sWLKSIi+7Sfugipvnv37nqERKJ+IoyBfs+aNeu6tiPKH3nkkV5jMpJXRhERERluKLArgCDZtGlT2iKqXIiYdiCGogzCDrHDFlbEgHM8++yzvfLL5dqBhZLzBfk+9RNpL+okSh5CNdKxDhM+PfKjHHVEWrnP5TqjDG2l/qeeeuq6PKCeSC/ntQKhS2j2EKb9AYFKffPmzaulXB9tkX1CqBM2Pj+uP8yePbt48803i9OnT6f6+0NPT0/qQ1jwaffRo0frbcVK/t5771137xCSvtEyfSIiItIdFNgVmTp1anHlypVk7cNNoV0Ya0BI4nYRVkLCWPM4vyyEIBd3bIjOKMexiKn+kFtJ2RCqhLiOdKymWE8jH1EPWDY3bNiQ3BVyENdYTwmXzfFl6yluFZMnT05lsbqGJZUxwwpNWvlcVUDwLlq0qLZXDdwrIvx6LugRzlzXaBehuEkD9rl2cT0mTJhQaZLTCspjISds+JEjR4rp06e3vYdaQftw+cjB9YW23nLLLWmfe6ds9UeQI8wjnHl/2iAiIiLXM+YENuIqt57mVtlWeTxuR0zC+fPnk1BCQEKzcgjo3CqMzyviB0GDG0KINvYRW/v370/l2Ef4RDmsy4ip/sB5cqs47Zo2bVott3OwgiLSwtcYEcp+1Hnt2rWUh+/uyZMnU9qMGTPSPhOVlStX1oVuJ+zbt68YP3588eCDD9ZS2tPIRQRoMzA24R4SaUw+uD4x6WFC0l+B3VObJDEGiGzohjW+Fa+//noab/qTw8SJscDffOPGjelJhYiIiHSHMSewEVchtNjyF+Ra5bWiVTms3XkefrUIrFy0hcgK0bVixYr0ifWXMliX+wuWcwRwtANr9VAR4g4Q+52IO8buwIEDyVreX7gG8SSh7B5CGqvQID5jQpI/cegErjNgIY8JF/UyqeqPmwjW6/LEiwkO5/vwww9rKUWauOGa0gjuXSYfTBZ5KiEiIiL9RxeRAQQBxEoVzSyfIXwQWYhGxCMgmhBJ7FO2EzEZggsrOmIOsIhjjQ13AoQU1uSccC2o6i6AFR/3ihBlfLKP60NVEHe4imCRrnpeQATfcccdyQ2lv9Be2s0Yh3sIY47gzN1YEKHxxKEZCHReXgzRTJ9wJWLcw0KeW9Kpvz9uItH2/BrQj3gSEjBxo+0c2wjuBX2yRUREuocCewBBQIava7hm5G4RCB+EHJZNXB8CBBKih+OfeOKJtOpEEO4o+UuJYQHOyyHswiKOUGc5O8qQh+/0O++8k/KCOH9YbDkPwg9XFdKw5uLiQh7iHfcPfMpx9SCNT/bDZaQZCFTGhTJsWNZfeeWV+uSiCgjs48eP91q6rhXNfLCBMeJlQAjrNW1EKMf4AeIZempuHuU6uQb04cUXX0wilzTGjYkS/S1byAHXmYkTJ17nvpET48W5OJZxZp/0qteAdjGBi/GKeyg2rsHOnTvbXjsRERGphutgi0hDmEi5DraIiEjnaMEWEREREekihkqXEQ1+x3kkwxxWD8kjMEo1sFwbKl1ERKTvjEqBLSIiIiIyVOgiIiIiIiLSRRTYIiIiIiJdRIEtIiIiItJFfMlRRgW+mCciIiLDBdfB7gIE/Vi9enWxa9euERmsozwhuXbtWrFjx44UHIXgNGUI3U6wk/IKHpHOeKxdu7ZXIJjyJIdgJ9RdTm/WFoK+5OnQSEi7drOIiIgMNbqISJocnDhxoh7Ce82aNWmigFiOMOEXL16shw4PEU1EyN27d9ePIYIhAhcQxnE8IjoPGY+IJqT7a6+9VsybNy+lAfm/93u/17AtAXVF3vr16xXRIiIiMuxQYFcE4ZiHl45w24TIzsNYkxcCNPKjTJ6O9XfTpk3Fk08+eV2dHMOxjcohTglfHnnl/Px8HMfxrSB/woQJtb3q9PT0pDDwYX1mremjR4/2EszB+fPn0ydh3GH+/PkpFDnh2Tl3tJ06CVv+0ksvpX0RERGRkYgCuwKI0IULF9YtsmyIWnjmmWeus/AilBGciF2IMgjQVatW1UXv1KlTiytXrqQ8ys6aNSuJTcpSR14O8QlYmy9dupTS47w7d+6slzl37ly9HJbgFStWpHLNwAKMhXjBggVJlIcFuh1YoDlXzpkzZ5Jgpl85uJognHHziP1jx46lNtOXENjUSZtpE32JiULeJvysO5lAiIiIiAw2Y05g58KtLN6a5SH4EMJYqDmmCgi/8ePHF/v376+lFMWRI0eKcePGFZMnT077RBrcs2dP+o7I/da3vpVEJ2VzKzW+yojPRuD/PGXKlFRm5syZ6dgq5XI4N64YCHZcObZu3Zos7P2BdoVFn3aFTzRimolEWLUPHz7c0OrNBGbDhg3JzzpHFxEREREZ7ow5gY1wC4HGlr9g1yoPSzVpgGgM6/RAEFZnBCbn5OXBAEF65513pjbgmoJ1O9qJ20VuZWej3VUJK/jBgweLRYsW1VIbg/W6LN6nTZuWLNK4joQPNqL9pptuSpZ3QGDn7jRYpLF4k96oThEREZGRhi4iHYIIRzhinQ73hAsXLiRxiyU5CKt3LlT5zrEI2VYgMhGqYfHl5UHgfM1cVeJ8WJ6jXX2lisjFGo8wDks3n7Qzt9gDfWUSECIaazUThnwSgCAnjzrx067qpiIiIiIyHFFgVwDxh5DF4srG0nSvvPJK3T2BT5YEDP9gjqXMtm3bkk9ylOM7ae1ApCI0KfPEE08Ub7/9dkrnPAcOHKhbf2MLt5Woe+PGjfW8dmK13Dc2aGf5xq1k79699bbwyX6+4kfAS4v4YP/6r/96co9BSOdguWZpPXy0WZJv6dKlqU76MWnSpLo7CeiDLSIiIsMd18EeQTTqFwIav+kXXnihrWV8LMB4uA62iIiIDCVasEcQWH/xZ87B7xnrMK4nIiIiIjL0GCp9hIGFNkKCAyuRaL3uPS6GShcREZGhZFQKbBERERGRoUIXERERERGRLqLAFhERERHpIgpsEREREZEu4kuOwwTWsibASyeRF2VkMZpfvhUREZGf4zrYXYBgLYQC37VrV8NAK1UYbQKbyI4E5LnhhhvSfqzswTgR6r0MYpMlB/MxyK8jeffff39Kh1g9BdauXZuC3OTXmlD2BPZptppI1M2yh7EKC21etmxZCtjDfqNjuNacj+iUBOiJY4BzLV68uNcqL1AW0oN9f4qIiMjgoouIdB1EKEKVqIwRDn39+vVJ6CKe2Sfc+8WLF4stW7ak/SpC8913363Xh8BF6ALimmA7nBcQyqwZ/uKLL7Zdqu/atWtFT09Pba838+fPT+uLRyj3gHbfeuutSShzDGHtCZUfMJnYsGFDva2KaBERkbGFArsirLMcIbrZIjw5ltJ169YVEydOrIcNj1DpHEN+0Gg/6gvrLOUoz/kCvkedraDuJ598sti6dWuqMw8ljuiM9DyP9E2bNqWN/Mcffzx9Rn60J8pFv1tBmPewXA8UEX6d9iFgEcFYzGHRokXF8ePHK60Nfvjw4WQ1j3HKmTt3bnHs2LF0zLx581IafQNENuL67rvvLk6dOpXSRERERECBXQHE18KFC5PVNaySiE7AIosVFsEV+YjQduKOY2bOnFm3dL7xxhspnXJYZ0PQAd9JqyIY58yZkyzH1Au4LACuK2vWrKm3H6tr5E2dOrU4ffp0cfLkyeK+++4rdu/eXYwbN6645557im984xvFzp07UxnqnD59ehLlreBctJcJR5WJQV/AMk0fcB0B+ozVmgkGriEI8CpEBMwYi4BrjvBmzNmoM+/HO++8U9x7773Jkv3WW2+l8eL8gI/1xo0b65OSfLIkIiIio58xJ7Bzq3FZ/DTLQ8xduXIlCcYqFtx2IN4Q1/jgNnJh2LNnT13QsfGdNGA/tyjnVmo4ePBgErhlAVouhx905OHPHPUjjM+fP5++cwziOyzziEYE9pQpU1J+KzgXohz3DSz85XZ2G0QwVmusys3GtRn79+9PE6gbb7yxlvKJ4GbcEeDUzVjmAvv1119PQvrEiRNpcpWji4iIiMjYZswJ7BB+jcRPq7zwHQbEZu7q0W1C0PX09CRRx3fSIg+RH20M3+Z24D6BG0WUw5+5CpTJxSJbJ4KRY8vW9E7AtzkEfw5iHQF85syZWkpRHDlyJIndGKuqMCFhjG+//fZayidPDXJLNJMN0phcMNlizLknFM8iIiJSRheRDkGE4woyfvz4ukUWKydCsJFlFxHIcbhVLFiwIKWFdTncQLCU5ytkAFZVrNwPP/xw+t4fQoyeO3cu7dOW8CVuBcfjW75ixYpaSt/AdYKVONqBWMZiHpZiBDnnx3WlDKuRwL59+9Jnf8HXGvcYuOuuu1Kbc5cg3IAYw1hiT0RERKQZCuwKlN0rsAa/8sordcsxn7glsDwb+RxLGdwuEJZYQZcvX1788Ic/TMcDPsOISY5HRL/66qu1nE84dOhQ7Vvv730h2oeI53y0BR/idnAM/tdMDKLv0bdWMGGI49lwESkvo9cI8nFR4XjKLV26NJ0/LNJYkaNOqGq9rwJCnZcmeTkTgc2EKR/38NUmrxH4YMekRR9sERGRsY3rYA9Tog/4+CJqZfQwGu5PERERaY4W7GEK7hFYv+PlQxEREREZGRgqfZiBf3Ss54wbSV8jQ8onq8KUfdsDXvKMiJGDxUi+L0VERKQ6o1Jgi4iIiIgMFbqIiIiIiIh0EQW2iIiIiEgX0Qd7iMA/mFDcg+0HPFphKTyWSQQiKT733HNdW8JPREREpBNcpm+IGC4Cm3YQ0CYXpKxzvXbt2rR2NcFwWH86IKz6Cy+8kL5zzM0335y+Q0xkonye98Ybb9SXG8zFMOQToHJevIxYnjRBIyFNedYVV2CLiIjIUKGLyBiH8OIsBzh//vxayicCm6ArESURcRwRDRHkEfjl2rVr9WiHiOQlS5bUg9DkeXzOmjUr5bFKCsfleeyTzvbII4+kqIlxvvIEhPNEXjcDzYiIiIh0CwV2BZ5++um0BVhTN2/eXI/QhzDcunXrdZH7SN+0aVPx5JNP1vMQqMGHH36Y6iGd8hwfcL4ok0dPjHM3yoO8HMdxfCuIVoiYnjt3bi2lSFZrAtx0Il7Pnz+fPpuFYMfyTTTERYsWpWiNsfwgn2fPnk3nJ9Q8kRRFRERERjIK7Ars37+/mDx5cl3IYu0lNDaW3LDKPv/888mqivUVF4U4lnDoV65cSXm5JReoBzcW8g4ePJjEJ4SYD0stgnTVqlVJLBOA5tKlS/W83KLM93PnztXzEMkrVqxIec1ARHMcbiLUT9voK5btIEKsNxL0AQIZoR7hxRHKK1euTGUitDxMmDChOHPmTPoe0GbcZXATQWwTKr3Z5CDC0VedQIiIiIgMNmNOYCNCQ6CxhbUZmuVhZcX6GsISMXn8+PG6wMYvGFFIGT4nTZpUt+RiuY1ojNTzrW99qy6I8S8O32ME7fjx44sHHnggfSLqA/IQ9AhfLMXUXRa6CE1Eci6G+Y5wbUfuJkKdV69erQtlaOYikovosh937iJCwJzly5cXX/rSl1JeK3AJicnBxo0bez05AF1EREREZLgz5gQ2wjQEGlsIXGiVd/jw4eQ6gZCdPn16LwsvQjkvt2bNmgGLwEi91M+LhrxImLuWIIxD1MZW5SXK3E2jE/eQENFY7RHoq1evruX0JtxQfuEXfiFZ36dNm1bL+QQmAVixc7gW1MukInedERERERnu6CJSkXjhL1w0QkCHRTm3hPcF3ENOnz5d/OVf/mVyKQl3EeA7FvSwHAPfsSafPHky+S4jiCmHGO2L2wQTiDlz5iSXlnzyUAXaghsLZRu5j2AZR4AzVnGeOI5xo1xY+XMYV32yRUREZKThMn0dgKBdsGBBcnnILdSIxHxpuVg+DmG5bNmyYtu2bb3EMVAXLhxBLEcH0f5Yki5fjq6TcpAvgdcKBC8WcSzNuasHLhrNlunDYr1r1640FnFurN+I5XyZPizd+Zjlfcjzyn3L8xr1zWX6REREZDiiwJZRhQJbREREhhpdREREREREusioFtixpFt//aNl+MM15lrnrjoiIiIiQ8GodBERERERERkqdBEREREREekiCmwRERERkS6iwBYRERER6SKjepm+WDO56lrQoxnWmCZiYpXIjjL6iXXPWau8vE65jFzydevfeOONFBE1aJUnIiLdxXWwxwgjRWCXJ0eIv//5P/9n8YUvfKHYu3dvr+uJYJgwYULx/vvvJ+FAABwC+kTgn3YTq3JgmwiiE0GBoi1EoczTIc4RRMAf2kTYd8RLlCfyZ4x7s4A+0KrORsF+aA+RO7/+9a/XI142Cr7TCNqRBwrKGcvXoDzW0f6BuAYDST4GZVrliYhId9BFRIYNCJ/f+73fS9EgH3vssbStWbOm+I//8T8mYbdkyZJ0DCBqJk+eXLz44ovFxx9/XJw9e7aed8cddySRUwVEWZyLOlatWpVEGRCJkxD1ed2AKFy6dGmxffv2etkQbzkrVqzoJeyAet5+++3i6tWrqf6AOunfli1bGtaJxTHSORZhR11ECsX6HHnr16/vl7CjzrF4DTjf5s2b0/eojy2fHAzWNRARkZGPArsiWMtYZznf+JHFGsQW8IOLZei3f/u30w/2U089lY59/PHHUzpbLhQaQb1PPvlkKk/ZrVu3JjEDfLIfbeCYECNQbif7ZaKOKNuqznIeW/Q3+hrptDsol8vHqBk9PT3JqvjSSy/VUn4OQgeRhaUQFi1aVBw/fjyJHCzzp06dSqIuxhZR1SmHDx9OllJEI8ydO7c4duxYSp83b15KY1xmzpxZ7N69u6VLBePOcYTJz5k9e3bx5ptvFqdPn071A22eM2dOErC5hbYdM2bMqFtNu8VYvQaLFy9O50Uod8JAXAMRERn5KLArgFjEchfWMixZWN0Ql/v3709iIEQFnwiUQ4cOpR/scePGpeMfeuihJKB43M6PcjuwrOHiwvkOHjyYxAwgKLAohrUMEYM4AARF2aqXW+CAvixfvjwJibC0NauTvnAsQoZ0HpdjlUSwIHK+8Y1vFDt37kx5GzZsKKZPn57qB9ob5dgaWRfLINKwnNImxHqI85gk0GbGmskHbgm5CMTKCbT7vffeS4/Ap02bltKqgoBDQCIY6R/t4Tsb52M8OD/X9Pz587VS13PrrbemcO1YdnNLJuUpy71x5MiRNF6ch/uB+4LzcExMWvKJDm4UMR4xSeO6HT16tFi5cmWliVsVxuo1oJ1xXvoa/c4njYN1DUREZOQz5gR2LhrYQjhAq7wy/Njzw8wPLMIiflgRCAhjfqwRTQhwOHnyZPrxDigbFmq28o8zAj7EMUJg/PjxqQzHcGyUwy80RAznRozTpkZwLIIZ39H8mFZ1lgnrIhOAqVOnJmFBmY0bNyaxMmXKlHQc4gr/2CqW60bQHkR77mbA+GExjclHLpzgrbfeKu66665i3759tZT20NfoN8REAJHINebacl4mHYxTFYvlPffck8apTE9PT/qk3Qg8CBeFgHNxHzKZyWnkngCME2lM3tatW3fdE43+MFavAX93jCl/gzlDcQ1ERGRkMuYEdvwYxhYiFprl8UOMVTrEJI/zc8sYj6+xlj3yyCNJEMQPbysoiwU5zpX/YLeCx/OI9yhXFgGt4FisbbmPKzSrk/aQjlCm31jHsVhHO8lDgEU5thizGEvGhrJVRAeiHItlK5hsXLx4sd4G6mTygTWTScO3v/3t60RfK+hrtD23sjNZ4SU/Jg60HxFIGhMliIlEIxBi5XHmE3eFEJMxIcFFIeqs8mSjGYw71wIQpn1lrF6DM2fO1CfNfaVb10BEREY+uohUgB9LrGchAspiGGsdL0z92q/9Wnq8nud1A9wt8BcFRAAiCHDHyEUZYhbxn1vCyyB86QsrSACColmdfEc4hYjGjSQs34iciRMnppfIWoHo4MU9wPLdCoQb52715GAwYPxoa+5qQx8YJ0C45S/7NaI8zlhJcU3IX2KkfgQeFlo2xrs/Ao82N7LadsJYvQYIbMau3f3cjk6vAf9LEPuNxruveSIiMvQosCuAgOaHnR+02Pjxjh93rHUIYFwmECjdoPzYnPNxHh7Lhy8o7h7vvPNOygfELMKDx9RRttEPMD7U9Ic6EQTN6kRMX7lypW5BZOPFRYQgkwis2QsWLKjnxZggEnP3F9rDOdpNPDgfPr5YyinHeSdNmtTS17YV7SyxzaAPTJjChQAQwNDT05P6WR7nRq4wMc6MBeMbrg5BTFKYwGG5RQzGWPPUgP2wBDfy/+XaRhob7cFNIZ4i9IWxeg1wicJ1iglq1MffIMI7GKxrICIiIx/Xwa4AP6K4gMTatlE/1mp+aAGLEo+fu7H+LXUhTPLH5UMBQpolyBApIUpCxAx126R/IA6brYMtIx/+TnkqFf+fclrliYhId9CCXYHyC3/xGDisW4gVLF+NXvoayeDjmr9MVnYnEREREZHrMVR6Bcr1AS9RsURZpLPfLYvQcLFgA9YuHpUHvJA2UqzXjCOP9RsxkvoxEDApHIxQ6V6DwSX/ey3/T2qVJyIi3WVUCmwRERERkaFCFxERERERkS6iwBYRERER6SL6YA9DyquWyNgg91fWP3l4wYo6BGTipd+PPvooLekXK+u0yhMRkbGJy/QNQ0aDwC6/HBnCo6enJ4nImPTEtfrRj36UAoGU1xBGdLZb/rC8nCDjxzrOb775ZjF79uxeywwCbSMoT35+VoXJhVH55TzChedt4BysVR2EIC5P7jp5gZBzNnq5tdlY8qIibchfWONY8j/zmc/0WkYSaDPBWVoJwGZj+d//+38vvvjFL173N0U+0RWjzTFu+aS23VjmAhXy/HLfh3qyXB6fnFZ5IiIyttBFRAYMhF9EzENkhehAAC5cuDCJ0QBRReh1xFpAPuK6k+UPEZ2ErEfUEoikHYh6go9wbsrmIJppe4S/jih/9AWhmkcEDIHJ2tII20jPo1/2h1ZjOWvWrF5t//jjj1MbGLt8jBlbxqSq+MvH8s/+7M9qqc3hXEwQXnvttV7XEZqNJQIdcc05on/r169P15t+QqSzDaW4FhERqYoCuyJYp4hiGBHbEAaRvmnTprTleYgTrIdxHHDss88+e52Qa0XUw/bbv/3byaKXg6jh3NRdbiPR60Jg0Q7KsuXtBI7JIy+ycT7OXa6zfP6+gJglSiHiNgerMtEwY3witHVVQUhEvlWrVhXf//73k6glSh/RAMuQFpEJ586dWxw7diyduywKA8Qe7UU80jbWPMfSXm4X48g64YPJ5cuXi/fee++6e4qIoljlY4zJZ/32qpFGy2MZY9CIWBedczG2Efm03CbIx5Lx4knNwYMHG05COEZERGQkosCuAEIBofn8888nKxqWS4RBCAhEISHFyeMRNnlYRbEW5qINMVcO1dwKHo2zVjFiDmseQnD8+PFJmCB0EciIJkQoYhKRgsU0rH0IGcJwB9SHkMzbSV1YXTk2+nbx4sUUBp12Llq0qNi9e3e9zk78gnELKAv2gL5Qd06MSxzX6Xj98i//crLc5lZOxmbGjBlp/BizXATzHRFH/WzNRCFpXGPGjrpw++B40ukX/WOCAlyDCB+fT676S6ux5D4rPxEgxDjimzEEji+HHm9Fo7EEgi7lE8U8CFNMVhgb7qe8jQFpMZZx7yL6aXtM8qJ/WLU5lrSwZouIiIwExpzA5oc6hApbLoKa5fFjj0/tunXrUjqfkyZNSmILeEyPGAAso4gGxMOePXvqog0BMX369GL//v3puFxQsJVFE+f7l//yXyahGyIHEU3dCOUPP/ywuOOOO1IbEE4IUcqH4GPjuFwA4YaBdRGoMx7Fl8EXliiOgHUSP9++WK6buTUA7WCy8OCDD9ZSPhHY4SbC+ODigBCrAgFT7rnnntreJzAmXBv4u7/7u3S+3J+XyQfXJ0R8WRRybFzvsm84UIZ+MVkJYpLDRAU3Eqz/CNL+0mosX3/99TTByydTXNfcTYQxZb/R9S7TaCwhLNU33nhjGtd8rPiOq0o8GSg/EWg3lrSL+3H79u1pAgMxvriV0A/Kd3PSIiIiMlCMOYGNAA2hwpb/0LfKCx/S2Kr41uaiLR7VhwUxBEXUVxZNIYaXL19eFzIIQepDbPz0pz9NaXfffXcSV9SHLysCNeqkzVVADOUCCIto9D3GhGPIz91O+gPtPXDgQHK3yGECwuQEschkIh+TViD4/vN//s+pvrIIY7KAuD59+nTx+c9/Po0XkxUEIBOZjRs3pr4xBrkojGuOuOVFP4Qy5SAmV80IcYj7Q9lSPxAwbrzQmcMEjzFkLDtxD2k1lsAYcu3ifGfOnEn3KG4lK1euTGPJpCx392k0ltzPTA5jMteM+FvJn7qIiIgMZ3QRqQBWOQRVX6xnIXx4fI4oQSxUBdcMxC6+sIgKyiJIEIGsusH322+/PVkWyccaG1ZGBEw7EQiUw70Ay2EIc0R1GQQ3VllArHUDRChWeHyFAyYg9IuX68KyXJVTp04liz+WY/rPeCGmGS/8lBGY9957bxqnu+66K/Uj7zf9a+QmglClHq4h7aFd1F9F6A2WHzHjRtsRtUG0lbFkTKu6h0B5LAEhzeQOmPwh3mOfMc6t7GyNXhzNx5Lrg1Wdc5SPa0T+NKYd1Md93GhC2CqPv3EmCEyORERE+ooCuwJYqhG7WOX48e3EkouoQcghRMI9oxMQAliteUyO0EHUIKKwpCIYEdEIH8QKq22Ery6W73feeadWS3Moh/APy2NsCAz6l7uxYN3mHFVFbyu/YaCe48ePJ5eEIEQX1tCq7iE5XCtcEOhPTIgYL87FtWD8AHedsuhEjEJPT0/6DChLnVh06QN+6FyTsHxzX7DPdaaf0Wc26MRvvRntxjKuYz6WwBgyllXdQ3IajSUWfyYr1MVkDlepT33qUw0t5ORjccalJCiPJX1hEsm9Rd84F9eFJw75vcfG39BIXrpSRETGDq6DPcZpNFaIKayKBswYXJjUYPHuhiCXgYFJrutgi4hIO7Rgj3GwPOYuGsCjeFagCIuuiIiIiFTHUOmSLNa4OQS85DacrNdYBvNIfznDra39AQs2riDAS4EDYckeK2PZbfJxK49TqzwRERmbjEqBLSIiIiIyVOgiIiIiIiLSRRTYIiIiIiJdRIEtIiIiItJFfMlRpAn5y59E1nQN5pFBq79//zeIiMhg4DrYowSCdqxevbrYtWtXPYQ7ApFAH++//37xmc98ppdAjJUPCKBDFD7ChAe5mHz66ad75cUqCbB27dpegU1CrNCWch5R/ggqEsSKGbnAKa9wce3atWLHjh29QtI3KkcagUti9YYQxnFMuQ9VRVWMXz5uAzXOuZgPaCf96c84R715+kCNc7kPA7USSlVa/R8Yi/8jRERk8NBFZAzwP/7H/0ifK1asSJ+waNGi4uDBg3VxEWGuiRgJiMggD4GNyIolyBBmEWockZWHvM7z+FywYEESYIC4IaDKa6+9lkRnDgKekOWUo32IQY6HVuXysNyEX0e8Au2FaD/bQAmq/o4zbSa93M6+jjMQpp/xInJljCN0e5wpT0j2qJPNgDkiIjJWUWDXQCBs2rSpePLJJ+uhmUOcAVbQPGxzLhYp9/jjjxdbt25NeRFGHSGC1TDK8J00No556qmnUjplyYt8yM9HvbSvr/z4xz9O1kmsj9TDRoCZPXv21I74OVhTCas9YcKEXoKsHefPn0+fhG4vQzjyn/zkJymADcyfPz+FwyZ0POeJPpchBDxBcGgrNCuHGDx16lQSfJFG6HIgb7AYbuPMWBAaPkL0M36N6MY4T5kypeHa2iIiImMRBXbG1KlTi1tuuaVuDZw1a1ZdSGCNC8sceQsXLqwLI0QMlsKNGzfWLZOLFy9Oll5EepQ7evRo0dPTk/IRNJTDovnQQw8Ve/fuTdZIhBPCbPz48XVr5po1a3o9vm8GAmflypV1YZ4/rqc853/00UdT/d///vfrlugc+jRz5swk/sLNAVeBqDOfBOTMnTs3RX9E5JVBsNHfI0eOpH2OPXbsWDo/Aq1RfYD1FItptLNVuYg6ybi/9957xblz55LQxPWB60rb8wlTfxioccYvmHso6s0t0UEn48z4MA6c//Tp06lsI7oxzljbqWPdunX1CaaIiMhYZcwJbERWCJiyiOGxOYIMEDDsh6UwL4e4mjhxYt3iB/hyIpTY1q9fnwQHIgOxEeUQqmFRRUzv378/fT958mQvEcY+9SOwEWlVyd0F2HAnyMGSSr0QVs0gRDQCD9GHkA6auYjkQhOxmPse53n4++7cuTOJT8QaE5ewxB4+fLiXGwL+xIg0ykG4GbQrB2+99VZx11139eobbaXNjCVtpN5GwrUTBmqcm7mI9GWcuffwC8c6DYju3E2k2+MMMQmlX/SPpzAiIiJjkTEnsBE0IWByEdMKRC6P/UNU8YnIakf44oZoQqhWIUQhIoVzd0MUAvViZcytpkEuonPR14oQmvjdYjnN/YlzEZpb4BFwiM8QhVh/sTCHlTR8gynL5IZxgGblvvzlLydrP4KQc3z729++rm8QEx/EMMJzIC2sw2GcsWZPmjQpjRPjxbgxXuEmMlDjDPSLupmAdjJBFBERGS3oItIEHoHjwoElG/9SwLIMvLiG5bAdWKt5xI4QQbgg0jshRCGiLPxqhyMIStwicqHcDKyhuchkQ4yWyyHiGG8stojhZuVwzemE4TyO7ehknHHzYPzy8eJlxrKbyECNM6K9yt8IcF6e9CDMy/1qlcc+6bqkiIjIcEOBnZE/NmelhhdffDGJXB6F4/caebwAxn47cAFBaFDmiSeeKN5+++1aTmuwVlMmNsTPSy+9VMsdfKr4YNM+xqSn5mPeCMph1Qwf4QA/XqzK4VYR4K6DxXb58uVNy1HnL/zCL9RSeoOYzl10YixzF4vhRBUf7CrjjNikn7h35OAuwiSvPMno7zgzkcQCno9z7q4iIiIy1nAd7Bo8yl62bFmxbdu2ZCkUQeCW18GWkUOr/wN9+R8hIiJSFS3YIiIiIiJdZFQL7HjBqxsvCMrYJNw29PMdOXCduF5ctwiJHrTKExER6Raj0kVERERERGSo0EVERERERKSLKLBFRERERLqIAltEREREpIuM6mX64iUmovcN5VJcLAHIusAE3iDyHusOuz5wY3ghlZdTgdDhLpEnIiIiIw3XwR5g+tqWXJRDiE3IJw+RToCQtWvXpiAkIUrzdZyJTBnCNagy8Xj66aeLO++8s7b3SYjtF154IX3nfATnCSKPdcQJPEIAkmeeeaaW+wnlyQ80EtKuQS0iIiIjFV1EBhiELyHXz58/X0tpD+Jy5cqVxe7du+vhqgmZjtBcvXp1Cr8e6XwnDbCOE5EPMd0IhOyGDRvqZduJ6yAPnY1wRkCHiN6yZUtx8eLFYvv27fW8KiDuo87om4iIiMhoQIGdEWvkRrjnfO1jRG+k52trY2netGlT2vI8QksTVpzw6oQARzCTh0W4HfPmzUuitiyAOReCHReTgHDspN11111p/6233ipmz56dvouIiIjI4KPArhGuC7l1OCyrCOYlS5Ykay3pWGvZR/DC1KlTiytXrqQ8LLO4Nly4cOE6Cy/5ZZeJMrRjwoQJxZkzZ2opP2fKlCnF1atXU93ByZMn02e4XJw6dSqVj7blcAwBNhD6W7dubXhMI+6///765KFbAVciCFA36xQREREZDow5gY3oDWHHFpbo+fPnJ1eO3DocTJs2rTh69Gjd/eHQoUPJ1xnBC/geRzlcQagHq3JfoBxuHn3l448/Lg4cOFAsWrSolvJzcheRNWvWVH7RMncR6ZY7hy4iIiIiMloZcwIbt40QdmxV/ZAHC6zTiHdEfZlG4n3GjBnpE/Ec7Nu3L332VeSLiIiISN/RRaQGVmncL3p6emopPwd3jTlz5iS/auAlQqzMVV/o6wQsuaw4smDBgutcOMptxK2CY44fP1688847KQ2o49y5c8mXW0REREQGFwV2DUTpiy++mIR02TcYKzcuIrywSPrSpUuLnTt3DojABs7HCiLxYmS0BfI24k+NzzhW+TJ79uypL/E3EDR6iZP9mIQAy/tF+9nCHQf0wRYREZHRiutgy7AEMe462CIiIjIS0YItIiIiItJFDJU+xilHasxh9ZBG7icDCZZrQ6WLiIjISGZUCmwRERERkaFCFxERERERkS6iwBYRERER6SL6YHeJ8B1udS6iSN522231cOm5//NQ+DuLiIiISPcZ1RbsCMc9XF9wRGjTvnfffbeWIiIiIiIjHV1EugQifjiLeREREREZHBTYNQg5/uyzz9YjEeb7rfJwRyESIREJt27dmvJycAuJiIX3339/LVVERERERisK7H7CGs3r168vtmzZUnz00Ue11E9AXM+cObPYsGFDsm7jZy0iIiIio5sxJ7BzizIbLycOBFi2EdeEazdQioiIiMjYYcwJbFbqwJocmz7TIiIiItJNdBHJuOGGG4oZM2Yk6zO+1OwHrfIagdX60qVLxbx589I+lnJ9sEVERERGPwrsGi+//HJx9uzZYuXKlcXGjRuLY8eOFZcvX26bx4uOWMXXrVtXTJw4MR0TLzvu2LGjmDp1anJFefjhh4tXX301lRERERGR0cuoDjSD//NIcAEh4My5c+cMNCMiIiIyCtCCLSIiIiLSRUa1wCZ0+UCuFNJfsFzTvgiXLiIiIiIjn1HpIiIiIiIiMlToIiIiIiIi0kUU2CIiIiIiXUSBLSIiIiLSRUb1Mn2f/vSn0/73vve9ERWxkZcyWTf7ueeeuy7M+kjvm4iIiMhoZ1RbsBGf3QqHTkCZZ599NgWQGUoQ3OvXry82bNhQfPDBB7VUERERERku6CIiIiIiItJFjORYAdarLq9V/dFHHxUvvPBCceHCheKb3/xm8c477xQPPfRQccMNNyTLMu4dkydPLtauXVvcfPPNvcr81V/9VdrHGv71r389lYE33ngjRXPMXUSiDsjLjrRolSIiIiJjBS3YFXjmmWeKLVu2FBcvXiy2b9+e3E5+53d+py52x40bV8yePbvYuHFjct2AxYsXp3yO43i2o0ePFj09PSk/xPXu3bvr+eVQ6Yh6xPXevXt7nU9EREREhi9jTmAjVImeGFse5bFVXjuwJOMfHT7SWJWxMm/evLle3/3331/cdttt6fi5c+cWJ0+ebGp95iXGf/kv/2Wxc+dOLdQiIiIiI4gxJ7CxEofFmC0Xr63y+sKKFSvSJ1Zt6sMFpCq4mezbt69Yvnx5esFSREREREYGuohUBF/rq1evFlOmTKmltAdr9aVLl5JVG5E8Z86cWk5RHDlypJg6dWrLVUlwH8GtZNWqVckaLiIiIiLDHwV2RRDJuIF89atfTe4eWLvbWZb3799fzJgxIx3/xBNPFG+//XYtpyhefvnl5Fu9cuXKugsJLiplOA8iHSv4UC8RKCIiIiLtcRWREYqriIiIiIgMT7Rgi4iIiIh0kVEtsMOdo5PVQIY7sTIJSwJGuHQRERERGT6MShcREREREZGhQhcREREREZEuosAWEREREekiCmwRERERkS4yqpfpi5cAv/e97w37pexYA5vANM8880yv78MF1uD++te/Xhw8eDCtzS0yloj7/4Ybbig++uij4oUXXij+6q/+qpYrIiLSG9fBHiaMJYFNgJ7Vq1cXu3btSgF3RhKsSPPwww8Xzz33XAo+BNxv3/zmN4tXXnkl9Yf+rV27trh8+XL9uEZpUdd3v/vdNLY333xzqg8Iqx/BjCjXKK8VrcrlYhGuXbtW7Nixo34taBcr8AQxQS1PXKFVHuH+o6/lc5LXqN/QTsC2GstW4x1pREdlHKLNwDHck3feeWfah0btoB/Lli0rtm3bpsAWEZGm6CIilUB8rVmzpt/ieiyAkCNqJ6H158+fX0v9RMjedNNNxeLFi2spP4e87du3F4899lixZcuWFFY/InvmeXzOmjUrnaMd5XILFiyoL1mJeOQ85DFpQjgiOPlcsmRJr3Lsk45IXb9+fUpne/fdd1NdOQjuyOfYEL9ELCX0f5736quvpj6yj/inPr6T1k68thpLaHYNLl68WNx6662pr6QTJZVjAtoRbazSDhERkUYosNvAD/GmTZuSwMhBNDz99NPpO58R7jxfd5sf6KeeeiqJUtL5zIUR+VHm/vvvr6V+wocffpjWuyZv69at152/EbTj8ccfb3i+aG/e1mgnfYxzsfGdtHJ6XgaoO87FRn9y8nNFW0hbt25dMXHixHqY+PK4NIL+cx3Yos68Lfm5Io86aT/XgLQYm/x8+TWo0o7z58/3EmTBlStXipMnT6bvs2fPLt58883i9OnTxdy5c1Na8NZbb6X8ViDqCKM/c+bMYurUqbXUn4M4vnDhQm2vGocOHSp+8pOfFNOmTaul/JwzZ84ksTp58uRi0aJFycIb1mw+z549e10/OmHevHlJuHb7aVKrsWx0DWbMmJE+EdmI67vvvrs4depUShMREekmCuw2YIFDPE2ZMqUu8hCeCJVz586lY3DlCKsXFr+FCxemY+Cee+5JYok8hEpPT09KR9ghoDZs2JDyECA5CABcXMjDwojwqcJDDz1U7Ny587rzAY+/Dx8+nPKwNPJI/YEHHkiPyU+cOJHS2bDq8bi8lcWS/n3jG9+on4t+TJ8+vT4RQPBOmDCh3j/6i3BkrLCcInLCShp57UBsci0oE+2PcW50DTge4Thu3Lg0vowN1wLrJ2ILEY4rTpQjD5eFduDSgBilj7Sd7/QVEOicD0F75MiRNCbRRkDQcWy7CRNCnno4FreKmIzQPlxRuDadwP3EWNCmMghg7hVEO+dDcOdwnzNOVYjgTmxMbri/GtXZDZqNZbtr8M477xT33ntvsmQj0jmWawhMdDuZcImIiDRizAns3GLJlltBm+UhMBDUN954YxJ4iBUER4iGvBxCCOts/GAjSsNyF0KFH3vENQK6mVDKyyESxo8fn8rxg88Pf5wvrM0BYjysj4jpXBjh97pv3770nboRz/QH9uzZkz6BcgiXvN4yjAECNoQfkSURMkxEaCP9px2dCsFW1wfLLb7CEAI0xrnRNaAPiOn9+/enY7Aw50IeYcmkI8ohDtv1mzqwYFM/TxkYX74HPbUJDf1G4AFjFXz88cfFgQMHKk+YIHf1oP/Lly9vK9ChLMyZDMW9wSSBJwnkQbf8/csuItxfCPuBoNlYtrsGr7/+evIVZ1LJRC9HFxEREekGY05gI07jB5Qtf2zdLC+E9Oc///kkPrF+IXgReQgd/GVDAPGJIBoo+MHnhz/aGH6uQwFWz7BQx5aPZ19odX2a0Z9rkAtCtqrjyQTppz/9aRLbfMfqzz3B9xDtMekou1fEJCcmB41gokLd1JuDYOTFPfLbkQtz/OdDXEP4YJOPNZ97in5zvrIbCZOIeFrTKVjFaW8j15RuUB7Lz372s02vAWOG4KefTCj6e6+KiIg0QxeRCiCk+ZEGBO7tt9+evmPNDKET/rdY02KlhGaEkMGCClhpyz7YOdSJL2knQhorLC4UTAhaEb68YfWLclj3Wp2P/mK5XbFiRS3l5zBG1NvMSkse4rGKSKxCX64BIBp5ga8TN4BoO9fuRz/6UXHs2LHkjoK45uVDrOrx8iAbApZ7J/elZlw5d1z/MkwYli5dmp5wMInJCVcP7slugOhm3BClXHvuFyYrMSbcm7SdJxyk/dZv/VZKj6cU7dpBX+kHL1hWsbp3SnksW12DW265JR0jIiIy0CiwK4LIQFghHrGC8SOO2MKChoUuHrdzDPvt4FE/dVIGQcuKCjm56wJg2a1C+JBiuUMkt7PSIVBefPHFJKqiHOK/3fkYB1wOEE7RTsqEMGMZM9wtGuWF6Ap/3TyvL/T1GnBeBGyUY8OS2w6uPX3Dmsw4MNHgnsASzD1BWhATkdxFARCs+SSg7M7B/RHXrpWrRzfgXIh2VuTgnLzkGGOC0Od89Ilt0qRJKZ3848ePX9eOsg82op06WUEk+pDndYN8LBHYza5Bs/NxPeMFSH2wRUSkG7gO9iiCl+6w5iEMus1YHVORHKzwroMtIiLt0IItlei2a4KIiIjIaMVQ6aOIbluwsdbhkhCP38fKOI4EcF0oR2oMeLkRt49uupEMNrjqNHsvgRV2urXqSVXyv4VGER5FRERyRqXAFhEREREZKnQRERERERHpIgpsEREREZEuog/2MAGfUwJ64Fuafx8OhP8pUSIHYoUSGRj0GxYRERkaXKZvmDDUArvVOZsJ7G6Oc7O6CHRCEJFoF+3k5bd80lROi7oIMU8Ev5hoQf6CXJQLqopQ2sR6zzmcm3K8eMga3M8991xa75tjWef8tddeK77yla/UXxgNCF/PsatXr05rn+dQJxEQG73sR98+85nPpLXO82tCn1jTPO8H18+l5URERAYPXUSkLaxGQajtobZeI5yZBCBWy1EQEcdEVOSYMghVIvoR1p0AMay2EiC4I+IfQWd6ahEt24EwzsPEh9hnBY8I2pLzZ3/2Z2kMOZZzvvHGG+l7hGZH9JfzqJMx5zuRCS9evFgPff7v//2/T9cFMY2ABlYWIdBKBIYRERGRoUGB3QYE26ZNm+oiJsAyGUKNz4j+xkYeYE186qmnkkhqFBmO/ChTtlJ++OGHKdodeVu3br3u/M3I25JHyyP9ySefTHXledRLGufPo0fStigXadEvIJ+oj1iHI3pftJO8/Nx8sh91NgKRSQTJRrD0ILAWN1EaidyIUM7HEnFM+XLExBzOgSglxHezCH1nzpypfes7b731VjF79uza3sBBX4i6yJgzxj1/PzkgiuFIXp5PRERkNKDAbgOijDDYU6ZMSUIGsY2Y4dF9CL+wPrJhYcwtqffcc0+xd+/elJdbSBGbM2fOrFtBsVrmIBRxlyAP14xFixbVcppDnbQp2oL7wIoVK2q5RbJ2sj4y5wSsrGGd5vy5NTes1fSN47HY5pAf6WEhph7qI3Q14adD7PLJPuntYFwZ52effTaJYPYD3D2OHTuWrLOI6bJIJq/dOBE2GyI0djkkfVii28HEgglGo8nHqVOn0gSAfgw0L730UvokDDnWa66viIiIDC1jTmAjQkMUlYVRszxEK0LvxhtvTGIbwYirQlg783IInYkTJyYrKSBaQ7RRD+UQ34hrBDQCvhF5Ofxtx48fn8ohKsMizhaW4qgTS3Tk8Z3zBQh1BHBYi3Px2k0QwMePH0+CGPhkv53bQkxYGGdcPnIBzXcEZESSPHz48HVuIli2GacHH3ywltKefFIB8VSiHc1cRODjjz8uDhw4UGlS1F+4ltxH3JNVxlhEREQGnjEnsBGnIYrKwqhZXgjpz3/+80nY3XvvvUnIIfawUmIZDt9YPvHDHSgQUAj6aGP48ALuE9GO2OKFvsGGSQGC+JFHHimmT5+e9qvCOCNQw82C8UdgM3FhAsPkAbeUqVOn9hLhjAPluB7NCMt1WLJzuLZYnuPpQ39A7ENMtAYS7gn8szsZYxERERk4dBGpAEIakQiImdtvvz19R6ThOhLfAatleaWIMmFBDgsslvJGK0UE1Hn69Om6kG4EeVjXwx+3UxCxnYrL6EcjSziWcvyBWTmDdlXxC6YNWOEBgYpbSewzVvHyX2y43OQCG7g+d9xxR3rRsAzHLl++vKmll3PQn1bjXBXqwCJftrKLiIjI6EeBXRGspViIEWYIRsQfAhIhyLJs69atS5ZVjmG/HfjKUidlWMbt1VdfreV8Qtk3OHyiW8EybNDMN7gVYXGNsljJEdu4oJRfZszr3L9/f7FgwYKUXn4ZE4sw/eCzKpznvffeqwvUSZMmFZ/61KeSJbhsoSWfscOlJOD6IKBvvvnmWkpRbzfXCH/4fCzL41zV4t/KBzvA57zdZCuIl0lpT7j5VL12IiIiMrxwHWwZMBCIS5YsMcDJEMOkx3WwRUREBg8t2DIgMMnBuswycoo6ERERGUsYKl26Du4OuDqwQsdQvWTZH3CPaeYTP5L6hOXaUOkiIiKDz6gU2CIiIiIiQ4UuIiIiIiIiXUSBLSIiIiLSRRTYIiIiIiJdxJccK8KSc6ynDITJfu6557oSkERERERERheug10jX3EhaCTMEdosP6fAFhEREZFG6CKSwVJmW7ZsSWG4EddLly7tFZlQRERERKQdCuwmsF4wIdGnTJlSSxERERERaY8Cuwm/9Eu/lD4NzCEiIiIinTDmBDZR+r7zne/UN3yqg5tvvrlYt25dSl+yZImR70RERESkY8acwP793//95GMdW/4SY+6DjRBXXIuIiIhIp+giIiIiIiLSRRTYIiIiIiJdRIFd4+WXXy6+9a1v6RYiIiIiIv1CgS0iIiIi0kVGtcAmtHl5pZC+Qh3UFeHSRUREREQaMSpDpYuIiIiIDBW6iIiIiIiIdBEFtoiIiIhIFxmVLiK/+Iu/WPzu7/5u8elPfzrtf+973+sVUEZGH4S2X7t2bYrGee3atWLHjh1pZRgZHjz99NPFnXfemb6/8cYbKeBT0CpPRERkJDKqBfZf/MVfdCSsEWmrV68udu3aNWTi7Fd+5VeKZcuWFdu2beu1ZOD/v72zd4mji+LwLUwwCEZDwA+QFJrEBIIWgqjYaJVWLPwP7C1CqoAp7Swt7VKlCNjaSARLP1ARCzHxC/xCUBMt8r7PzZz1ZhxnJ7vKavw9MLhz78ydmbMrPPfsmVkkZGlpyV+PyeTp6akbHR11GxsbiW3cmNnW1uYmJiZcf3+/l08jFJlQcGBvby83xlXwS5etra3R2gXhuByfm0Lj0hSfAIVCbPsY6+vrbnh4OFq7mqT3Lkss7VwePXqU+2n8pDYbi7Hfvn17rbFkbN6fsrKyqOU34b5XxRLix7QJZXzcLOdy03Cu+/v7l64B0vqEEEKIu4RKRO4gyOTa2po7Pz93r1+/jlp/iypS2NXVFbVcQN+nT5/8z8Dzc/AvXrzwkmwgbvR9+PDBryeNEYIE2VhHR0e5sUM5ampqcjMzM662ttZLK3Du79+/d9++ffPbswwODnpxRQg7OjpyP1fPkkWui4H4HR4eut3dXX9uIcSsp6cnWrvgumPJtRMD9mFCYfu/e/cuJ8NJseTvyMiIf832tiDXXAsTAiYu1h6OJ4QQQoibQ4IdQfZsaGjIVVZWuoGBAf9IPmTRpIt+2mwho2jwOuxjMeEK9xsbG/MSmcbm5qYX5zi0HRwc+NfI1tzcnNve3nbNzc2+zZifn/f9aZCRnZqacg0NDTlZCwmPVSjE7cGDB+7r169+3SYCPf8LK1n2z58/+/WQ6urqS1ncYsgSS+K3urrqM9qvXr3ybQZtT548SYyRUcpYIu5MqJDoOPX19dcaSyGEEEJkR4IdQaY0no1Fkq20gH7LBNLf0tLihQph7u7uzu1D9pEsJHJO38OHD30mkz7L1OYDmUKQOD6CznEqKip8n8nW4uKiW15e/iOjCVtbW37bfCKP9DFOVVWVX6fcg0nAx48fvbhnOc80OE++7id+4UQAYSV7TSaV67PJB5MUMq9kkpnokJlNE9uspMWS15wP58hCu02ogMw25MtAlyqWNTU1Pl60h5M8rpVjrqys+MliOFEUQgghxM1z7wQ7lDqWMBOdRrgf0kKm24QqDqKGvJFBZTsEO5/wGkjdycmJf/3r1y8v6GF9bU9UsoCgItkQlon8+PHDzc7Ous7OzqglG2FZA9lYrrdQuHZqv3d2dvx60kQAED+OR22wYRMZJBxBRYoLJV8sEWfeK7ZDUo+Pjy+J6PT0tJ9MlZeXRy35KUUsmZxwTCZ3BvGljQz7dU5ahBBCCJHOvRNskw5bEJN8IMfU2VqWmr/U4QKSS8mDlZWw3fj4uBdgpA25QhQZ42+EnnIJhJCM5fPnz93Z2Zn7+fOnFzYkkbEYF9mKl4lYKcFVEwBgfMoXLEtrcN7ILZndQkH4Hz9+nPslTWLz9OlT304mNsvYvE98o8A1ZJ2cXEVSLJn8UBLCjZbEkfMkrvEyETLBiHddXV3UcplSxRLptslcGnzGkX3Il40XQgghRPGoRCQAQUKUEKYQW0fKgOyw1bciLAiYCTtCjViHIFrcYEZmk6/102BbBBDR+/79u89Yvnz50osUAkgpQngTILKPZCNdBmMgsnFZNBDW3t5e/5QVtg1B1pB49i8UhJ9Y2TmyLCws+Hauh5KNLBONYuuI02L57NkzL+82aWIhrvQ1NjZGI/yGGu03b95Ea39SyljyOaMGu6+vL9r6arhWts2KfWOT9D4V2ieEEELcFyTYAQgSomTZQqtdJSNMlpqv2WlHwlkH+pAy2m2x/ZCMsB3ZSrq5LwmEGYEiQ45UA1lUK2cwrAwlLBOBycnJP+SU15Zl59Ft3BgXZu/DumGyrlxDIZhUcoNgCNlWsvucL8dGSu14ZGipYzY5s4Xz/PLlS9E1zEmxZFLC+2hlNmAZ6Pb2dv/XsPff4nlbYgk8RpDXFjMmYfTHP3t8dikVyfKNjRBCCCGKQ8/BLhJEhhpZe76wHbsYsRJ/DxOaUj/DXOSHmnoy+kn/G2l9QgghxF1CGewiiZd82FfxdlOaEEIIIYS4X+in0oskfiyg1vquZ+HICNtPj8fhBk/KIm5Tpjg839t4fmRnwyeYhPwLn5d8hNcfv960PiGEEOIu8k8KthBCCCGEEKVCJSJCCCGEEEJcIxJsIYQQQgghrg3n/gMtp6dXVv8HdQAAAABJRU5ErkJggg==)"
      ],
      "metadata": {
        "id": "F73f0UA-xU9A"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QGbOpuYOmgWH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9fe5a04c-ad31-4ca6-8ffb-43a14eff62b0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2023-06-12 08:43:58.828302: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2023-06-12 08:43:59.732723: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "========== SpaceInvadersNoFrameskip-v4 ==========\n",
            "Seed: 3570770334\n",
            "Loading hyperparameters from: /usr/local/lib/python3.10/dist-packages/rl_zoo3/hyperparams/dqn.yml\n",
            "Default hyperparameters for environment (ones being tuned will be overridden):\n",
            "OrderedDict([('batch_size', 32),\n",
            "             ('buffer_size', 100000),\n",
            "             ('env_wrapper',\n",
            "              ['stable_baselines3.common.atari_wrappers.AtariWrapper']),\n",
            "             ('exploration_final_eps', 0.01),\n",
            "             ('exploration_fraction', 0.1),\n",
            "             ('frame_stack', 4),\n",
            "             ('gradient_steps', 1),\n",
            "             ('learning_rate', 0.0001),\n",
            "             ('learning_starts', 100000),\n",
            "             ('n_timesteps', 10000000.0),\n",
            "             ('optimize_memory_usage', False),\n",
            "             ('policy', 'CnnPolicy'),\n",
            "             ('target_update_interval', 1000),\n",
            "             ('train_freq', 4)])\n",
            "Using 1 environments\n",
            "Overwriting n_timesteps with n=5000\n",
            "Doing 1 intermediate evaluations for pruning based on the number of timesteps. (1 evaluation every 100k timesteps)\n",
            "A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)\n",
            "[Powered by Stella]\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Optimizing hyperparameters\n",
            "/usr/local/lib/python3.10/dist-packages/optuna/samplers/_tpe/sampler.py:278: ExperimentalWarning: ``multivariate`` option is an experimental feature. The interface can change in the future.\n",
            "  warnings.warn(\n",
            "Sampler: tpe - Pruner: median\n",
            "\u001b[32m[I 2023-06-12 08:44:02,460]\u001b[0m A new study created in memory with name: no-name-81ebb0a5-af86-495c-a323-feead57268e5\u001b[0m\n",
            "/usr/local/lib/python3.10/dist-packages/rl_zoo3/hyperparams_opt.py:398: FutureWarning: suggest_uniform has been deprecated in v3.0.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Use suggest_float instead.\n",
            "  exploration_fraction = trial.suggest_uniform(\"exploration_fraction\", 0, 0.5)\n",
            "/usr/local/lib/python3.10/dist-packages/rl_zoo3/hyperparams_opt.py:397: FutureWarning: suggest_uniform has been deprecated in v3.0.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Use suggest_float instead.\n",
            "  exploration_final_eps = trial.suggest_uniform(\"exploration_final_eps\", 0, 0.2)\n",
            "Stacking 4 frames\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/buffers.py:229: UserWarning: This system does not have apparently enough memory to store the complete replay buffer 56.46GB > 11.99GB\n",
            "  warnings.warn(\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 08:44:32,991]\u001b[0m Trial 1 finished with value: 14.0 and parameters: {'gamma': 0.995, 'learning_rate': 0.00011056071120243769, 'batch_size': 512, 'buffer_size': 50000, 'exploration_final_eps': 0.013435583762012239, 'exploration_fraction': 0.422603905998642, 'target_update_interval': 10000, 'learning_starts': 10000, 'train_freq': 256, 'subsample_steps': 2, 'net_arch': 'small'}. Best is trial 1 with value: 14.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 08:44:58,513]\u001b[0m Trial 0 finished with value: 232.0 and parameters: {'gamma': 0.999, 'learning_rate': 0.006286102457164191, 'batch_size': 512, 'buffer_size': 1000000, 'exploration_final_eps': 0.13591434289142784, 'exploration_fraction': 0.17389060113340954, 'target_update_interval': 1000, 'learning_starts': 5000, 'train_freq': 128, 'subsample_steps': 8, 'net_arch': 'medium'}. Best is trial 0 with value: 232.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 08:45:24,476]\u001b[0m Trial 3 finished with value: 218.0 and parameters: {'gamma': 0.95, 'learning_rate': 0.005914368298130324, 'batch_size': 128, 'buffer_size': 100000, 'exploration_final_eps': 0.09106534822654887, 'exploration_fraction': 0.44373670522488257, 'target_update_interval': 1000, 'learning_starts': 5000, 'train_freq': 4, 'subsample_steps': 4, 'net_arch': 'small'}. Best is trial 0 with value: 232.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 08:50:30,334]\u001b[0m Trial 4 finished with value: 170.0 and parameters: {'gamma': 0.999, 'learning_rate': 0.0004124000099972622, 'batch_size': 16, 'buffer_size': 100000, 'exploration_final_eps': 0.18919251763197226, 'exploration_fraction': 0.1594037472044047, 'target_update_interval': 1000, 'learning_starts': 1000, 'train_freq': 16, 'subsample_steps': 1, 'net_arch': 'medium'}. Best is trial 0 with value: 232.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:02:24,623]\u001b[0m Trial 5 finished with value: 151.0 and parameters: {'gamma': 0.999, 'learning_rate': 0.00039293438524206216, 'batch_size': 128, 'buffer_size': 100000, 'exploration_final_eps': 0.09476566088819499, 'exploration_fraction': 0.3805306582183282, 'target_update_interval': 20000, 'learning_starts': 1000, 'train_freq': 1000, 'subsample_steps': 2, 'net_arch': 'tiny'}. Best is trial 0 with value: 232.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:02:50,824]\u001b[0m Trial 6 finished with value: 61.0 and parameters: {'gamma': 0.995, 'learning_rate': 0.7543715613751267, 'batch_size': 32, 'buffer_size': 50000, 'exploration_final_eps': 0.11838974927899654, 'exploration_fraction': 0.09818752270198183, 'target_update_interval': 10000, 'learning_starts': 20000, 'train_freq': 1000, 'subsample_steps': 1, 'net_arch': 'medium'}. Best is trial 0 with value: 232.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:03:17,618]\u001b[0m Trial 7 finished with value: 105.0 and parameters: {'gamma': 0.99, 'learning_rate': 0.005302581418336491, 'batch_size': 128, 'buffer_size': 50000, 'exploration_final_eps': 0.12082565999032807, 'exploration_fraction': 0.44804130922570623, 'target_update_interval': 1000, 'learning_starts': 5000, 'train_freq': 8, 'subsample_steps': 2, 'net_arch': 'tiny'}. Best is trial 0 with value: 232.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/buffers.py:229: UserWarning: This system does not have apparently enough memory to store the complete replay buffer 56.46GB > 11.19GB\n",
            "  warnings.warn(\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:03:46,164]\u001b[0m Trial 8 finished with value: 260.0 and parameters: {'gamma': 0.9999, 'learning_rate': 0.11331729128679324, 'batch_size': 256, 'buffer_size': 1000000, 'exploration_final_eps': 0.12587037965871037, 'exploration_fraction': 0.29057102643104854, 'target_update_interval': 10000, 'learning_starts': 5000, 'train_freq': 8, 'subsample_steps': 2, 'net_arch': 'medium'}. Best is trial 8 with value: 260.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:04:54,139]\u001b[0m Trial 9 finished with value: 171.0 and parameters: {'gamma': 0.98, 'learning_rate': 0.005905746555580141, 'batch_size': 16, 'buffer_size': 50000, 'exploration_final_eps': 0.18292086348434333, 'exploration_fraction': 0.47159380868037504, 'target_update_interval': 20000, 'learning_starts': 1000, 'train_freq': 16, 'subsample_steps': 8, 'net_arch': 'medium'}. Best is trial 8 with value: 260.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:12:12,739]\u001b[0m Trial 2 finished with value: 127.0 and parameters: {'gamma': 0.9999, 'learning_rate': 0.0003264544180580004, 'batch_size': 128, 'buffer_size': 10000, 'exploration_final_eps': 0.1335852715111834, 'exploration_fraction': 0.3010004727435149, 'target_update_interval': 5000, 'learning_starts': 0, 'train_freq': 1, 'subsample_steps': 1, 'net_arch': 'small'}. Best is trial 8 with value: 260.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/buffers.py:229: UserWarning: This system does not have apparently enough memory to store the complete replay buffer 56.46GB > 11.25GB\n",
            "  warnings.warn(\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:12:36,421]\u001b[0m Trial 11 pruned. \u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/buffers.py:229: UserWarning: This system does not have apparently enough memory to store the complete replay buffer 56.46GB > 10.98GB\n",
            "  warnings.warn(\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:13:05,894]\u001b[0m Trial 12 finished with value: 320.0 and parameters: {'gamma': 0.99, 'learning_rate': 0.0345546407179516, 'batch_size': 256, 'buffer_size': 1000000, 'exploration_final_eps': 0.14065093779522275, 'exploration_fraction': 0.2035250590468127, 'target_update_interval': 10000, 'learning_starts': 10000, 'train_freq': 8, 'subsample_steps': 2, 'net_arch': 'medium'}. Best is trial 12 with value: 320.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/buffers.py:229: UserWarning: This system does not have apparently enough memory to store the complete replay buffer 56.46GB > 11.24GB\n",
            "  warnings.warn(\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:20:02,102]\u001b[0m Trial 10 finished with value: 232.0 and parameters: {'gamma': 0.95, 'learning_rate': 8.746699773820807e-05, 'batch_size': 64, 'buffer_size': 10000, 'exploration_final_eps': 0.05882400766042824, 'exploration_fraction': 0.48119152226186895, 'target_update_interval': 20000, 'learning_starts': 0, 'train_freq': 16, 'subsample_steps': 1, 'net_arch': 'medium'}. Best is trial 12 with value: 320.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/buffers.py:229: UserWarning: This system does not have apparently enough memory to store the complete replay buffer 56.46GB > 11.00GB\n",
            "  warnings.warn(\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:42:45,746]\u001b[0m Trial 13 finished with value: 291.0 and parameters: {'gamma': 0.9999, 'learning_rate': 0.019356260282883304, 'batch_size': 256, 'buffer_size': 1000000, 'exploration_final_eps': 0.1168843495265759, 'exploration_fraction': 0.27678126130174235, 'target_update_interval': 10000, 'learning_starts': 0, 'train_freq': 8, 'subsample_steps': 2, 'net_arch': 'medium'}. Best is trial 12 with value: 320.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/buffers.py:229: UserWarning: This system does not have apparently enough memory to store the complete replay buffer 56.46GB > 10.50GB\n",
            "  warnings.warn(\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:49:08,432]\u001b[0m Trial 14 finished with value: 186.0 and parameters: {'gamma': 0.99, 'learning_rate': 0.02362813545151881, 'batch_size': 256, 'buffer_size': 1000000, 'exploration_final_eps': 0.17990366471277808, 'exploration_fraction': 0.137786734546211, 'target_update_interval': 10000, 'learning_starts': 0, 'train_freq': 8, 'subsample_steps': 2, 'net_arch': 'tiny'}. Best is trial 12 with value: 320.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:49:30,945]\u001b[0m Trial 16 pruned. \u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/buffers.py:229: UserWarning: This system does not have apparently enough memory to store the complete replay buffer 56.46GB > 10.11GB\n",
            "  warnings.warn(\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:50:15,873]\u001b[0m Trial 17 finished with value: 286.0 and parameters: {'gamma': 0.99, 'learning_rate': 0.10456066282433218, 'batch_size': 256, 'buffer_size': 1000000, 'exploration_final_eps': 0.18924355765103434, 'exploration_fraction': 0.1603878520008773, 'target_update_interval': 1, 'learning_starts': 10000, 'train_freq': 8, 'subsample_steps': 2, 'net_arch': 'medium'}. Best is trial 12 with value: 320.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/buffers.py:229: UserWarning: This system does not have apparently enough memory to store the complete replay buffer 56.46GB > 10.28GB\n",
            "  warnings.warn(\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "\u001b[32m[I 2023-06-12 09:50:41,837]\u001b[0m Trial 18 finished with value: 202.0 and parameters: {'gamma': 0.9, 'learning_rate': 0.009071560724270131, 'batch_size': 256, 'buffer_size': 1000000, 'exploration_final_eps': 0.11146457167930282, 'exploration_fraction': 0.08946816293548929, 'target_update_interval': 10000, 'learning_starts': 10000, 'train_freq': 1, 'subsample_steps': 2, 'net_arch': 'medium'}. Best is trial 12 with value: 320.0.\u001b[0m\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Number of finished trials:  20\n",
            "Best trial:\n",
            "Value:  320.0\n",
            "Params: \n",
            "    gamma: 0.99\n",
            "    learning_rate: 0.0345546407179516\n",
            "    batch_size: 256\n",
            "    buffer_size: 1000000\n",
            "    exploration_final_eps: 0.14065093779522275\n",
            "    exploration_fraction: 0.2035250590468127\n",
            "    target_update_interval: 10000\n",
            "    learning_starts: 10000\n",
            "    train_freq: 8\n",
            "    subsample_steps: 2\n",
            "    net_arch: medium\n",
            "Writing report to logs/dqn/report_SpaceInvadersNoFrameskip-v4_100-trials-5000-tpe-median_1686563510\n",
            "\u001b[32m[I 2023-06-12 10:10:21,908]\u001b[0m Trial 15 pruned. \u001b[0m\n",
            "terminate called without an active exception\n"
          ]
        }
      ],
      "source": [
        "!python -m rl_zoo3.train --algo dqn --env SpaceInvadersNoFrameskip-v4 -n 5000 -optimize --n-trials 100 --n-jobs 2 --sampler tpe --pruner median"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "4Hq1DEV9ufjE"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v3Ortrt6fkOj"
      },
      "source": [
        "2. We start the training and save the models on `logs` folder 📁\n",
        "\n",
        "- Define the algorithm after `--algo`, where we save the model after `-f` and where the hyperparameter config is after `-c`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QnFXaQR1fkOj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9383be5f-b4f5-40c8-d71d-0eaa29abd406"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2023-06-12 10:36:57.079046: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2023-06-12 10:36:58.063524: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "========== SpaceInvadersNoFrameskip-v4 ==========\n",
            "Seed: 722079798\n",
            "Loading hyperparameters from: dqn.yml\n",
            "Default hyperparameters for environment (ones being tuned will be overridden):\n",
            "OrderedDict([('batch_size', 256),\n",
            "             ('buffer_size', 100000),\n",
            "             ('env_wrapper',\n",
            "              ['stable_baselines3.common.atari_wrappers.AtariWrapper']),\n",
            "             ('exploration_final_eps', 0.01),\n",
            "             ('exploration_fraction', 0.1),\n",
            "             ('frame_stack', 4),\n",
            "             ('gradient_steps', 1),\n",
            "             ('learning_rate', 0.0345546407179516),\n",
            "             ('learning_starts', 10000),\n",
            "             ('n_timesteps', 1000000),\n",
            "             ('optimize_memory_usage', False),\n",
            "             ('policy', 'CnnPolicy'),\n",
            "             ('target_update_interval', 10000),\n",
            "             ('train_freq', 8)])\n",
            "Using 1 environments\n",
            "Creating test environment\n",
            "A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)\n",
            "[Powered by Stella]\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Using cpu device\n",
            "Log path: logs//dqn/SpaceInvadersNoFrameskip-v4_5\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.18e+03 |\n",
            "|    ep_rew_mean      | 185      |\n",
            "|    exploration_rate | 0.993    |\n",
            "| time/               |          |\n",
            "|    episodes         | 4        |\n",
            "|    fps              | 726      |\n",
            "|    time_elapsed     | 0        |\n",
            "|    total_timesteps  | 696      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.3e+03  |\n",
            "|    ep_rew_mean      | 182      |\n",
            "|    exploration_rate | 0.986    |\n",
            "| time/               |          |\n",
            "|    episodes         | 8        |\n",
            "|    fps              | 768      |\n",
            "|    time_elapsed     | 1        |\n",
            "|    total_timesteps  | 1372     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.87e+03 |\n",
            "|    ep_rew_mean      | 126      |\n",
            "|    exploration_rate | 0.982    |\n",
            "| time/               |          |\n",
            "|    episodes         | 12       |\n",
            "|    fps              | 777      |\n",
            "|    time_elapsed     | 2        |\n",
            "|    total_timesteps  | 1828     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.94e+03 |\n",
            "|    ep_rew_mean      | 123      |\n",
            "|    exploration_rate | 0.975    |\n",
            "| time/               |          |\n",
            "|    episodes         | 16       |\n",
            "|    fps              | 782      |\n",
            "|    time_elapsed     | 3        |\n",
            "|    total_timesteps  | 2559     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.85e+03 |\n",
            "|    ep_rew_mean      | 108      |\n",
            "|    exploration_rate | 0.97     |\n",
            "| time/               |          |\n",
            "|    episodes         | 20       |\n",
            "|    fps              | 790      |\n",
            "|    time_elapsed     | 3        |\n",
            "|    total_timesteps  | 3067     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.77e+03 |\n",
            "|    ep_rew_mean      | 109      |\n",
            "|    exploration_rate | 0.966    |\n",
            "| time/               |          |\n",
            "|    episodes         | 24       |\n",
            "|    fps              | 790      |\n",
            "|    time_elapsed     | 4        |\n",
            "|    total_timesteps  | 3449     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.8e+03  |\n",
            "|    ep_rew_mean      | 110      |\n",
            "|    exploration_rate | 0.959    |\n",
            "| time/               |          |\n",
            "|    episodes         | 28       |\n",
            "|    fps              | 795      |\n",
            "|    time_elapsed     | 5        |\n",
            "|    total_timesteps  | 4095     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.77e+03 |\n",
            "|    ep_rew_mean      | 102      |\n",
            "|    exploration_rate | 0.955    |\n",
            "| time/               |          |\n",
            "|    episodes         | 32       |\n",
            "|    fps              | 794      |\n",
            "|    time_elapsed     | 5        |\n",
            "|    total_timesteps  | 4528     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.77e+03 |\n",
            "|    ep_rew_mean      | 105      |\n",
            "|    exploration_rate | 0.949    |\n",
            "| time/               |          |\n",
            "|    episodes         | 36       |\n",
            "|    fps              | 797      |\n",
            "|    time_elapsed     | 6        |\n",
            "|    total_timesteps  | 5183     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.81e+03 |\n",
            "|    ep_rew_mean      | 107      |\n",
            "|    exploration_rate | 0.941    |\n",
            "| time/               |          |\n",
            "|    episodes         | 40       |\n",
            "|    fps              | 802      |\n",
            "|    time_elapsed     | 7        |\n",
            "|    total_timesteps  | 5975     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.98e+03 |\n",
            "|    ep_rew_mean      | 116      |\n",
            "|    exploration_rate | 0.928    |\n",
            "| time/               |          |\n",
            "|    episodes         | 44       |\n",
            "|    fps              | 807      |\n",
            "|    time_elapsed     | 8        |\n",
            "|    total_timesteps  | 7257     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.99e+03 |\n",
            "|    ep_rew_mean      | 119      |\n",
            "|    exploration_rate | 0.923    |\n",
            "| time/               |          |\n",
            "|    episodes         | 48       |\n",
            "|    fps              | 796      |\n",
            "|    time_elapsed     | 9        |\n",
            "|    total_timesteps  | 7768     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.98e+03 |\n",
            "|    ep_rew_mean      | 119      |\n",
            "|    exploration_rate | 0.917    |\n",
            "| time/               |          |\n",
            "|    episodes         | 52       |\n",
            "|    fps              | 776      |\n",
            "|    time_elapsed     | 10       |\n",
            "|    total_timesteps  | 8368     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.95e+03 |\n",
            "|    ep_rew_mean      | 117      |\n",
            "|    exploration_rate | 0.911    |\n",
            "| time/               |          |\n",
            "|    episodes         | 56       |\n",
            "|    fps              | 749      |\n",
            "|    time_elapsed     | 12       |\n",
            "|    total_timesteps  | 9033     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.03e+03 |\n",
            "|    ep_rew_mean      | 138      |\n",
            "|    exploration_rate | 0.902    |\n",
            "| time/               |          |\n",
            "|    episodes         | 60       |\n",
            "|    fps              | 713      |\n",
            "|    time_elapsed     | 13       |\n",
            "|    total_timesteps  | 9894     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.04e+03 |\n",
            "|    ep_rew_mean      | 140      |\n",
            "|    exploration_rate | 0.895    |\n",
            "| time/               |          |\n",
            "|    episodes         | 64       |\n",
            "|    fps              | 228      |\n",
            "|    time_elapsed     | 46       |\n",
            "|    total_timesteps  | 10624    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00967  |\n",
            "|    n_updates        | 77       |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.05e+03 |\n",
            "|    ep_rew_mean      | 142      |\n",
            "|    exploration_rate | 0.889    |\n",
            "| time/               |          |\n",
            "|    episodes         | 68       |\n",
            "|    fps              | 143      |\n",
            "|    time_elapsed     | 78       |\n",
            "|    total_timesteps  | 11262    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0104   |\n",
            "|    n_updates        | 157      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.03e+03 |\n",
            "|    ep_rew_mean      | 140      |\n",
            "|    exploration_rate | 0.882    |\n",
            "| time/               |          |\n",
            "|    episodes         | 72       |\n",
            "|    fps              | 107      |\n",
            "|    time_elapsed     | 111      |\n",
            "|    total_timesteps  | 11910    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00769  |\n",
            "|    n_updates        | 238      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.04e+03 |\n",
            "|    ep_rew_mean      | 140      |\n",
            "|    exploration_rate | 0.875    |\n",
            "| time/               |          |\n",
            "|    episodes         | 76       |\n",
            "|    fps              | 86       |\n",
            "|    time_elapsed     | 146      |\n",
            "|    total_timesteps  | 12595    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0175   |\n",
            "|    n_updates        | 324      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.05e+03 |\n",
            "|    ep_rew_mean      | 141      |\n",
            "|    exploration_rate | 0.869    |\n",
            "| time/               |          |\n",
            "|    episodes         | 80       |\n",
            "|    fps              | 73       |\n",
            "|    time_elapsed     | 179      |\n",
            "|    total_timesteps  | 13249    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00951  |\n",
            "|    n_updates        | 406      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.02e+03 |\n",
            "|    ep_rew_mean      | 134      |\n",
            "|    exploration_rate | 0.863    |\n",
            "| time/               |          |\n",
            "|    episodes         | 84       |\n",
            "|    fps              | 66       |\n",
            "|    time_elapsed     | 209      |\n",
            "|    total_timesteps  | 13844    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0113   |\n",
            "|    n_updates        | 480      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.01e+03 |\n",
            "|    ep_rew_mean      | 133      |\n",
            "|    exploration_rate | 0.857    |\n",
            "| time/               |          |\n",
            "|    episodes         | 88       |\n",
            "|    fps              | 60       |\n",
            "|    time_elapsed     | 237      |\n",
            "|    total_timesteps  | 14399    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0116   |\n",
            "|    n_updates        | 549      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.98e+03 |\n",
            "|    ep_rew_mean      | 130      |\n",
            "|    exploration_rate | 0.85     |\n",
            "| time/               |          |\n",
            "|    episodes         | 92       |\n",
            "|    fps              | 54       |\n",
            "|    time_elapsed     | 277      |\n",
            "|    total_timesteps  | 15169    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00402  |\n",
            "|    n_updates        | 646      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.98e+03 |\n",
            "|    ep_rew_mean      | 132      |\n",
            "|    exploration_rate | 0.847    |\n",
            "| time/               |          |\n",
            "|    episodes         | 96       |\n",
            "|    fps              | 52       |\n",
            "|    time_elapsed     | 294      |\n",
            "|    total_timesteps  | 15496    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0173   |\n",
            "|    n_updates        | 686      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.98e+03 |\n",
            "|    ep_rew_mean      | 131      |\n",
            "|    exploration_rate | 0.838    |\n",
            "| time/               |          |\n",
            "|    episodes         | 100      |\n",
            "|    fps              | 48       |\n",
            "|    time_elapsed     | 337      |\n",
            "|    total_timesteps  | 16325    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00771  |\n",
            "|    n_updates        | 790      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2e+03    |\n",
            "|    ep_rew_mean      | 134      |\n",
            "|    exploration_rate | 0.833    |\n",
            "| time/               |          |\n",
            "|    episodes         | 104      |\n",
            "|    fps              | 46       |\n",
            "|    time_elapsed     | 364      |\n",
            "|    total_timesteps  | 16848    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0059   |\n",
            "|    n_updates        | 855      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.03e+03 |\n",
            "|    ep_rew_mean      | 138      |\n",
            "|    exploration_rate | 0.824    |\n",
            "| time/               |          |\n",
            "|    episodes         | 108      |\n",
            "|    fps              | 42       |\n",
            "|    time_elapsed     | 414      |\n",
            "|    total_timesteps  | 17813    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00595  |\n",
            "|    n_updates        | 976      |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.03e+03 |\n",
            "|    ep_rew_mean      | 139      |\n",
            "|    exploration_rate | 0.817    |\n",
            "| time/               |          |\n",
            "|    episodes         | 112      |\n",
            "|    fps              | 41       |\n",
            "|    time_elapsed     | 451      |\n",
            "|    total_timesteps  | 18513    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00573  |\n",
            "|    n_updates        | 1064     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.01e+03 |\n",
            "|    ep_rew_mean      | 136      |\n",
            "|    exploration_rate | 0.811    |\n",
            "| time/               |          |\n",
            "|    episodes         | 116      |\n",
            "|    fps              | 39       |\n",
            "|    time_elapsed     | 480      |\n",
            "|    total_timesteps  | 19069    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00782  |\n",
            "|    n_updates        | 1133     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.03e+03 |\n",
            "|    ep_rew_mean      | 142      |\n",
            "|    exploration_rate | 0.804    |\n",
            "| time/               |          |\n",
            "|    episodes         | 120      |\n",
            "|    fps              | 38       |\n",
            "|    time_elapsed     | 520      |\n",
            "|    total_timesteps  | 19844    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00777  |\n",
            "|    n_updates        | 1230     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.02e+03 |\n",
            "|    ep_rew_mean      | 141      |\n",
            "|    exploration_rate | 0.798    |\n",
            "| time/               |          |\n",
            "|    episodes         | 124      |\n",
            "|    fps              | 37       |\n",
            "|    time_elapsed     | 552      |\n",
            "|    total_timesteps  | 20431    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00982  |\n",
            "|    n_updates        | 1303     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.03e+03 |\n",
            "|    ep_rew_mean      | 140      |\n",
            "|    exploration_rate | 0.792    |\n",
            "| time/               |          |\n",
            "|    episodes         | 128      |\n",
            "|    fps              | 36       |\n",
            "|    time_elapsed     | 582      |\n",
            "|    total_timesteps  | 21012    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0117   |\n",
            "|    n_updates        | 1376     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2e+03    |\n",
            "|    ep_rew_mean      | 135      |\n",
            "|    exploration_rate | 0.788    |\n",
            "| time/               |          |\n",
            "|    episodes         | 132      |\n",
            "|    fps              | 35       |\n",
            "|    time_elapsed     | 604      |\n",
            "|    total_timesteps  | 21461    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0021   |\n",
            "|    n_updates        | 1432     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.98e+03 |\n",
            "|    ep_rew_mean      | 134      |\n",
            "|    exploration_rate | 0.783    |\n",
            "| time/               |          |\n",
            "|    episodes         | 136      |\n",
            "|    fps              | 34       |\n",
            "|    time_elapsed     | 628      |\n",
            "|    total_timesteps  | 21927    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00217  |\n",
            "|    n_updates        | 1490     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 1.97e+03 |\n",
            "|    ep_rew_mean      | 131      |\n",
            "|    exploration_rate | 0.778    |\n",
            "| time/               |          |\n",
            "|    episodes         | 140      |\n",
            "|    fps              | 34       |\n",
            "|    time_elapsed     | 653      |\n",
            "|    total_timesteps  | 22404    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00792  |\n",
            "|    n_updates        | 1550     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.01e+03 |\n",
            "|    ep_rew_mean      | 133      |\n",
            "|    exploration_rate | 0.767    |\n",
            "| time/               |          |\n",
            "|    episodes         | 144      |\n",
            "|    fps              | 33       |\n",
            "|    time_elapsed     | 709      |\n",
            "|    total_timesteps  | 23495    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00224  |\n",
            "|    n_updates        | 1686     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.01e+03 |\n",
            "|    ep_rew_mean      | 133      |\n",
            "|    exploration_rate | 0.758    |\n",
            "| time/               |          |\n",
            "|    episodes         | 148      |\n",
            "|    fps              | 32       |\n",
            "|    time_elapsed     | 758      |\n",
            "|    total_timesteps  | 24444    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0116   |\n",
            "|    n_updates        | 1805     |\n",
            "----------------------------------\n",
            "Eval num_timesteps=25000, episode_reward=131.00 +/- 32.31\n",
            "Episode length: 2073.40 +/- 311.32\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 2.07e+03 |\n",
            "|    mean_reward      | 131      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.753    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 25000    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00963  |\n",
            "|    n_updates        | 1874     |\n",
            "----------------------------------\n",
            "New best mean reward!\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.04e+03 |\n",
            "|    ep_rew_mean      | 135      |\n",
            "|    exploration_rate | 0.75     |\n",
            "| time/               |          |\n",
            "|    episodes         | 152      |\n",
            "|    fps              | 31       |\n",
            "|    time_elapsed     | 805      |\n",
            "|    total_timesteps  | 25233    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0151   |\n",
            "|    n_updates        | 1904     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.05e+03 |\n",
            "|    ep_rew_mean      | 134      |\n",
            "|    exploration_rate | 0.743    |\n",
            "| time/               |          |\n",
            "|    episodes         | 156      |\n",
            "|    fps              | 30       |\n",
            "|    time_elapsed     | 843      |\n",
            "|    total_timesteps  | 25986    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.000157 |\n",
            "|    n_updates        | 1998     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.04e+03 |\n",
            "|    ep_rew_mean      | 132      |\n",
            "|    exploration_rate | 0.737    |\n",
            "| time/               |          |\n",
            "|    episodes         | 160      |\n",
            "|    fps              | 30       |\n",
            "|    time_elapsed     | 871      |\n",
            "|    total_timesteps  | 26542    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00598  |\n",
            "|    n_updates        | 2067     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.07e+03 |\n",
            "|    ep_rew_mean      | 135      |\n",
            "|    exploration_rate | 0.726    |\n",
            "| time/               |          |\n",
            "|    episodes         | 164      |\n",
            "|    fps              | 29       |\n",
            "|    time_elapsed     | 933      |\n",
            "|    total_timesteps  | 27704    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0172   |\n",
            "|    n_updates        | 2212     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.06e+03 |\n",
            "|    ep_rew_mean      | 133      |\n",
            "|    exploration_rate | 0.721    |\n",
            "| time/               |          |\n",
            "|    episodes         | 168      |\n",
            "|    fps              | 29       |\n",
            "|    time_elapsed     | 955      |\n",
            "|    total_timesteps  | 28133    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00587  |\n",
            "|    n_updates        | 2266     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.05e+03 |\n",
            "|    ep_rew_mean      | 133      |\n",
            "|    exploration_rate | 0.716    |\n",
            "| time/               |          |\n",
            "|    episodes         | 172      |\n",
            "|    fps              | 29       |\n",
            "|    time_elapsed     | 983      |\n",
            "|    total_timesteps  | 28676    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00223  |\n",
            "|    n_updates        | 2334     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.05e+03 |\n",
            "|    ep_rew_mean      | 131      |\n",
            "|    exploration_rate | 0.705    |\n",
            "| time/               |          |\n",
            "|    episodes         | 176      |\n",
            "|    fps              | 28       |\n",
            "|    time_elapsed     | 1041     |\n",
            "|    total_timesteps  | 29798    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00779  |\n",
            "|    n_updates        | 2474     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.09e+03 |\n",
            "|    ep_rew_mean      | 139      |\n",
            "|    exploration_rate | 0.697    |\n",
            "| time/               |          |\n",
            "|    episodes         | 180      |\n",
            "|    fps              | 28       |\n",
            "|    time_elapsed     | 1083     |\n",
            "|    total_timesteps  | 30574    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00806  |\n",
            "|    n_updates        | 2571     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.08e+03 |\n",
            "|    ep_rew_mean      | 139      |\n",
            "|    exploration_rate | 0.691    |\n",
            "| time/               |          |\n",
            "|    episodes         | 184      |\n",
            "|    fps              | 27       |\n",
            "|    time_elapsed     | 1118     |\n",
            "|    total_timesteps  | 31230    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00967  |\n",
            "|    n_updates        | 2653     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.09e+03 |\n",
            "|    ep_rew_mean      | 140      |\n",
            "|    exploration_rate | 0.68     |\n",
            "| time/               |          |\n",
            "|    episodes         | 188      |\n",
            "|    fps              | 27       |\n",
            "|    time_elapsed     | 1174     |\n",
            "|    total_timesteps  | 32305    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0079   |\n",
            "|    n_updates        | 2788     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.11e+03 |\n",
            "|    ep_rew_mean      | 144      |\n",
            "|    exploration_rate | 0.673    |\n",
            "| time/               |          |\n",
            "|    episodes         | 192      |\n",
            "|    fps              | 27       |\n",
            "|    time_elapsed     | 1214     |\n",
            "|    total_timesteps  | 33065    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0114   |\n",
            "|    n_updates        | 2883     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.12e+03 |\n",
            "|    ep_rew_mean      | 147      |\n",
            "|    exploration_rate | 0.664    |\n",
            "| time/               |          |\n",
            "|    episodes         | 196      |\n",
            "|    fps              | 26       |\n",
            "|    time_elapsed     | 1259     |\n",
            "|    total_timesteps  | 33939    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00202  |\n",
            "|    n_updates        | 2992     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.12e+03 |\n",
            "|    ep_rew_mean      | 147      |\n",
            "|    exploration_rate | 0.655    |\n",
            "| time/               |          |\n",
            "|    episodes         | 200      |\n",
            "|    fps              | 26       |\n",
            "|    time_elapsed     | 1304     |\n",
            "|    total_timesteps  | 34815    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00402  |\n",
            "|    n_updates        | 3101     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.14e+03 |\n",
            "|    ep_rew_mean      | 147      |\n",
            "|    exploration_rate | 0.647    |\n",
            "| time/               |          |\n",
            "|    episodes         | 204      |\n",
            "|    fps              | 26       |\n",
            "|    time_elapsed     | 1347     |\n",
            "|    total_timesteps  | 35608    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00795  |\n",
            "|    n_updates        | 3200     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.16e+03 |\n",
            "|    ep_rew_mean      | 149      |\n",
            "|    exploration_rate | 0.638    |\n",
            "| time/               |          |\n",
            "|    episodes         | 208      |\n",
            "|    fps              | 26       |\n",
            "|    time_elapsed     | 1396     |\n",
            "|    total_timesteps  | 36531    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00586  |\n",
            "|    n_updates        | 3316     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 151      |\n",
            "|    exploration_rate | 0.627    |\n",
            "| time/               |          |\n",
            "|    episodes         | 212      |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 1456     |\n",
            "|    total_timesteps  | 37694    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00777  |\n",
            "|    n_updates        | 3461     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 151      |\n",
            "|    exploration_rate | 0.621    |\n",
            "| time/               |          |\n",
            "|    episodes         | 216      |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 1488     |\n",
            "|    total_timesteps  | 38289    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00967  |\n",
            "|    n_updates        | 3536     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 150      |\n",
            "|    exploration_rate | 0.615    |\n",
            "| time/               |          |\n",
            "|    episodes         | 220      |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 1521     |\n",
            "|    total_timesteps  | 38925    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00591  |\n",
            "|    n_updates        | 3615     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 149      |\n",
            "|    exploration_rate | 0.609    |\n",
            "| time/               |          |\n",
            "|    episodes         | 224      |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 1553     |\n",
            "|    total_timesteps  | 39543    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00619  |\n",
            "|    n_updates        | 3692     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 152      |\n",
            "|    exploration_rate | 0.601    |\n",
            "| time/               |          |\n",
            "|    episodes         | 228      |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 1597     |\n",
            "|    total_timesteps  | 40349    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00588  |\n",
            "|    n_updates        | 3793     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.16e+03 |\n",
            "|    ep_rew_mean      | 151      |\n",
            "|    exploration_rate | 0.595    |\n",
            "| time/               |          |\n",
            "|    episodes         | 232      |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 1625     |\n",
            "|    total_timesteps  | 40879    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00587  |\n",
            "|    n_updates        | 3859     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.16e+03 |\n",
            "|    ep_rew_mean      | 151      |\n",
            "|    exploration_rate | 0.59     |\n",
            "| time/               |          |\n",
            "|    episodes         | 236      |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 1653     |\n",
            "|    total_timesteps  | 41410    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00778  |\n",
            "|    n_updates        | 3926     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 152      |\n",
            "|    exploration_rate | 0.579    |\n",
            "| time/               |          |\n",
            "|    episodes         | 240      |\n",
            "|    fps              | 24       |\n",
            "|    time_elapsed     | 1712     |\n",
            "|    total_timesteps  | 42523    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00981  |\n",
            "|    n_updates        | 4065     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.16e+03 |\n",
            "|    ep_rew_mean      | 150      |\n",
            "|    exploration_rate | 0.573    |\n",
            "| time/               |          |\n",
            "|    episodes         | 244      |\n",
            "|    fps              | 24       |\n",
            "|    time_elapsed     | 1746     |\n",
            "|    total_timesteps  | 43180    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00583  |\n",
            "|    n_updates        | 4147     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 150      |\n",
            "|    exploration_rate | 0.564    |\n",
            "| time/               |          |\n",
            "|    episodes         | 248      |\n",
            "|    fps              | 24       |\n",
            "|    time_elapsed     | 1792     |\n",
            "|    total_timesteps  | 44043    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00605  |\n",
            "|    n_updates        | 4255     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.16e+03 |\n",
            "|    ep_rew_mean      | 149      |\n",
            "|    exploration_rate | 0.557    |\n",
            "| time/               |          |\n",
            "|    episodes         | 252      |\n",
            "|    fps              | 24       |\n",
            "|    time_elapsed     | 1830     |\n",
            "|    total_timesteps  | 44785    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0115   |\n",
            "|    n_updates        | 4348     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 150      |\n",
            "|    exploration_rate | 0.549    |\n",
            "| time/               |          |\n",
            "|    episodes         | 256      |\n",
            "|    fps              | 24       |\n",
            "|    time_elapsed     | 1873     |\n",
            "|    total_timesteps  | 45598    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0042   |\n",
            "|    n_updates        | 4449     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.18e+03 |\n",
            "|    ep_rew_mean      | 154      |\n",
            "|    exploration_rate | 0.538    |\n",
            "| time/               |          |\n",
            "|    episodes         | 260      |\n",
            "|    fps              | 24       |\n",
            "|    time_elapsed     | 1931     |\n",
            "|    total_timesteps  | 46672    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0061   |\n",
            "|    n_updates        | 4583     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.18e+03 |\n",
            "|    ep_rew_mean      | 155      |\n",
            "|    exploration_rate | 0.532    |\n",
            "| time/               |          |\n",
            "|    episodes         | 264      |\n",
            "|    fps              | 24       |\n",
            "|    time_elapsed     | 1962     |\n",
            "|    total_timesteps  | 47252    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00418  |\n",
            "|    n_updates        | 4656     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 153      |\n",
            "|    exploration_rate | 0.527    |\n",
            "| time/               |          |\n",
            "|    episodes         | 268      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 1992     |\n",
            "|    total_timesteps  | 47820    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00773  |\n",
            "|    n_updates        | 4727     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.19e+03 |\n",
            "|    ep_rew_mean      | 157      |\n",
            "|    exploration_rate | 0.515    |\n",
            "| time/               |          |\n",
            "|    episodes         | 272      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2054     |\n",
            "|    total_timesteps  | 48977    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0177   |\n",
            "|    n_updates        | 4872     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.2e+03  |\n",
            "|    ep_rew_mean      | 157      |\n",
            "|    exploration_rate | 0.508    |\n",
            "| time/               |          |\n",
            "|    episodes         | 276      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2093     |\n",
            "|    total_timesteps  | 49711    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0199   |\n",
            "|    n_updates        | 4963     |\n",
            "----------------------------------\n",
            "Eval num_timesteps=50000, episode_reward=73.00 +/- 34.87\n",
            "Episode length: 1651.40 +/- 420.48\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 1.65e+03 |\n",
            "|    mean_reward      | 73       |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.505    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 50000    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00634  |\n",
            "|    n_updates        | 4999     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.18e+03 |\n",
            "|    ep_rew_mean      | 155      |\n",
            "|    exploration_rate | 0.503    |\n",
            "| time/               |          |\n",
            "|    episodes         | 280      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2124     |\n",
            "|    total_timesteps  | 50184    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00831  |\n",
            "|    n_updates        | 5022     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 154      |\n",
            "|    exploration_rate | 0.494    |\n",
            "| time/               |          |\n",
            "|    episodes         | 284      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2173     |\n",
            "|    total_timesteps  | 51111    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00434  |\n",
            "|    n_updates        | 5138     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 155      |\n",
            "|    exploration_rate | 0.489    |\n",
            "| time/               |          |\n",
            "|    episodes         | 288      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2198     |\n",
            "|    total_timesteps  | 51592    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0117   |\n",
            "|    n_updates        | 5198     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 153      |\n",
            "|    exploration_rate | 0.482    |\n",
            "| time/               |          |\n",
            "|    episodes         | 292      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2239     |\n",
            "|    total_timesteps  | 52361    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0156   |\n",
            "|    n_updates        | 5295     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 153      |\n",
            "|    exploration_rate | 0.477    |\n",
            "| time/               |          |\n",
            "|    episodes         | 296      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2266     |\n",
            "|    total_timesteps  | 52860    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0106   |\n",
            "|    n_updates        | 5357     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.18e+03 |\n",
            "|    ep_rew_mean      | 157      |\n",
            "|    exploration_rate | 0.466    |\n",
            "| time/               |          |\n",
            "|    episodes         | 300      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2323     |\n",
            "|    total_timesteps  | 53938    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00798  |\n",
            "|    n_updates        | 5492     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.17e+03 |\n",
            "|    ep_rew_mean      | 156      |\n",
            "|    exploration_rate | 0.46     |\n",
            "| time/               |          |\n",
            "|    episodes         | 304      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2354     |\n",
            "|    total_timesteps  | 54511    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00631  |\n",
            "|    n_updates        | 5563     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.19e+03 |\n",
            "|    ep_rew_mean      | 157      |\n",
            "|    exploration_rate | 0.454    |\n",
            "| time/               |          |\n",
            "|    episodes         | 308      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2387     |\n",
            "|    total_timesteps  | 55114    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00989  |\n",
            "|    n_updates        | 5639     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.18e+03 |\n",
            "|    ep_rew_mean      | 157      |\n",
            "|    exploration_rate | 0.449    |\n",
            "| time/               |          |\n",
            "|    episodes         | 312      |\n",
            "|    fps              | 23       |\n",
            "|    time_elapsed     | 2417     |\n",
            "|    total_timesteps  | 55678    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00433  |\n",
            "|    n_updates        | 5709     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.18e+03 |\n",
            "|    ep_rew_mean      | 157      |\n",
            "|    exploration_rate | 0.444    |\n",
            "| time/               |          |\n",
            "|    episodes         | 316      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2442     |\n",
            "|    total_timesteps  | 56160    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00969  |\n",
            "|    n_updates        | 5769     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.18e+03 |\n",
            "|    ep_rew_mean      | 156      |\n",
            "|    exploration_rate | 0.437    |\n",
            "| time/               |          |\n",
            "|    episodes         | 320      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2477     |\n",
            "|    total_timesteps  | 56824    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00982  |\n",
            "|    n_updates        | 5852     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.2e+03  |\n",
            "|    ep_rew_mean      | 157      |\n",
            "|    exploration_rate | 0.428    |\n",
            "| time/               |          |\n",
            "|    episodes         | 324      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2529     |\n",
            "|    total_timesteps  | 57794    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0156   |\n",
            "|    n_updates        | 5974     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.21e+03 |\n",
            "|    ep_rew_mean      | 157      |\n",
            "|    exploration_rate | 0.417    |\n",
            "| time/               |          |\n",
            "|    episodes         | 328      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2585     |\n",
            "|    total_timesteps  | 58851    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0137   |\n",
            "|    n_updates        | 6106     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.23e+03 |\n",
            "|    ep_rew_mean      | 160      |\n",
            "|    exploration_rate | 0.408    |\n",
            "| time/               |          |\n",
            "|    episodes         | 332      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2634     |\n",
            "|    total_timesteps  | 59774    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0062   |\n",
            "|    n_updates        | 6221     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.24e+03 |\n",
            "|    ep_rew_mean      | 161      |\n",
            "|    exploration_rate | 0.401    |\n",
            "| time/               |          |\n",
            "|    episodes         | 336      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2672     |\n",
            "|    total_timesteps  | 60476    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00822  |\n",
            "|    n_updates        | 6309     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.22e+03 |\n",
            "|    ep_rew_mean      | 160      |\n",
            "|    exploration_rate | 0.39     |\n",
            "| time/               |          |\n",
            "|    episodes         | 340      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2730     |\n",
            "|    total_timesteps  | 61571    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00828  |\n",
            "|    n_updates        | 6446     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.22e+03 |\n",
            "|    ep_rew_mean      | 166      |\n",
            "|    exploration_rate | 0.384    |\n",
            "| time/               |          |\n",
            "|    episodes         | 344      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2765     |\n",
            "|    total_timesteps  | 62227    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00389  |\n",
            "|    n_updates        | 6528     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.23e+03 |\n",
            "|    ep_rew_mean      | 166      |\n",
            "|    exploration_rate | 0.378    |\n",
            "| time/               |          |\n",
            "|    episodes         | 348      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2798     |\n",
            "|    total_timesteps  | 62857    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0194   |\n",
            "|    n_updates        | 6607     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.23e+03 |\n",
            "|    ep_rew_mean      | 166      |\n",
            "|    exploration_rate | 0.368    |\n",
            "| time/               |          |\n",
            "|    episodes         | 352      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2850     |\n",
            "|    total_timesteps  | 63818    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00798  |\n",
            "|    n_updates        | 6727     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.24e+03 |\n",
            "|    ep_rew_mean      | 165      |\n",
            "|    exploration_rate | 0.359    |\n",
            "| time/               |          |\n",
            "|    episodes         | 356      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2900     |\n",
            "|    total_timesteps  | 64764    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00996  |\n",
            "|    n_updates        | 6845     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.24e+03 |\n",
            "|    ep_rew_mean      | 164      |\n",
            "|    exploration_rate | 0.352    |\n",
            "| time/               |          |\n",
            "|    episodes         | 360      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2937     |\n",
            "|    total_timesteps  | 65461    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0077   |\n",
            "|    n_updates        | 6932     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.23e+03 |\n",
            "|    ep_rew_mean      | 163      |\n",
            "|    exploration_rate | 0.342    |\n",
            "| time/               |          |\n",
            "|    episodes         | 364      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 2990     |\n",
            "|    total_timesteps  | 66462    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0118   |\n",
            "|    n_updates        | 7057     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.26e+03 |\n",
            "|    ep_rew_mean      | 166      |\n",
            "|    exploration_rate | 0.333    |\n",
            "| time/               |          |\n",
            "|    episodes         | 368      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 3037     |\n",
            "|    total_timesteps  | 67348    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00454  |\n",
            "|    n_updates        | 7168     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.26e+03 |\n",
            "|    ep_rew_mean      | 166      |\n",
            "|    exploration_rate | 0.328    |\n",
            "| time/               |          |\n",
            "|    episodes         | 372      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 3065     |\n",
            "|    total_timesteps  | 67879    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0137   |\n",
            "|    n_updates        | 7234     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.26e+03 |\n",
            "|    ep_rew_mean      | 166      |\n",
            "|    exploration_rate | 0.315    |\n",
            "| time/               |          |\n",
            "|    episodes         | 376      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 3136     |\n",
            "|    total_timesteps  | 69213    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00994  |\n",
            "|    n_updates        | 7401     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.3e+03  |\n",
            "|    ep_rew_mean      | 170      |\n",
            "|    exploration_rate | 0.305    |\n",
            "| time/               |          |\n",
            "|    episodes         | 380      |\n",
            "|    fps              | 22       |\n",
            "|    time_elapsed     | 3190     |\n",
            "|    total_timesteps  | 70211    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00823  |\n",
            "|    n_updates        | 7526     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.3e+03  |\n",
            "|    ep_rew_mean      | 170      |\n",
            "|    exploration_rate | 0.296    |\n",
            "| time/               |          |\n",
            "|    episodes         | 384      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3238     |\n",
            "|    total_timesteps  | 71120    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00477  |\n",
            "|    n_updates        | 7639     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.33e+03 |\n",
            "|    ep_rew_mean      | 173      |\n",
            "|    exploration_rate | 0.29     |\n",
            "| time/               |          |\n",
            "|    episodes         | 388      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3274     |\n",
            "|    total_timesteps  | 71766    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00626  |\n",
            "|    n_updates        | 7720     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.34e+03 |\n",
            "|    ep_rew_mean      | 173      |\n",
            "|    exploration_rate | 0.278    |\n",
            "| time/               |          |\n",
            "|    episodes         | 392      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3334     |\n",
            "|    total_timesteps  | 72900    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0118   |\n",
            "|    n_updates        | 7862     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.35e+03 |\n",
            "|    ep_rew_mean      | 174      |\n",
            "|    exploration_rate | 0.272    |\n",
            "| time/               |          |\n",
            "|    episodes         | 396      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3370     |\n",
            "|    total_timesteps  | 73582    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0177   |\n",
            "|    n_updates        | 7947     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.34e+03 |\n",
            "|    ep_rew_mean      | 173      |\n",
            "|    exploration_rate | 0.266    |\n",
            "| time/               |          |\n",
            "|    episodes         | 400      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3400     |\n",
            "|    total_timesteps  | 74128    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00411  |\n",
            "|    n_updates        | 8015     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.32e+03 |\n",
            "|    ep_rew_mean      | 171      |\n",
            "|    exploration_rate | 0.262    |\n",
            "| time/               |          |\n",
            "|    episodes         | 404      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3425     |\n",
            "|    total_timesteps  | 74594    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00986  |\n",
            "|    n_updates        | 8074     |\n",
            "----------------------------------\n",
            "Eval num_timesteps=75000, episode_reward=237.00 +/- 76.33\n",
            "Episode length: 2229.00 +/- 653.33\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 2.23e+03 |\n",
            "|    mean_reward      | 237      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.258    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 75000    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00957  |\n",
            "|    n_updates        | 8124     |\n",
            "----------------------------------\n",
            "New best mean reward!\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.32e+03 |\n",
            "|    ep_rew_mean      | 172      |\n",
            "|    exploration_rate | 0.251    |\n",
            "| time/               |          |\n",
            "|    episodes         | 408      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3486     |\n",
            "|    total_timesteps  | 75632    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00811  |\n",
            "|    n_updates        | 8203     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.36e+03 |\n",
            "|    ep_rew_mean      | 175      |\n",
            "|    exploration_rate | 0.241    |\n",
            "| time/               |          |\n",
            "|    episodes         | 412      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3538     |\n",
            "|    total_timesteps  | 76634    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00823  |\n",
            "|    n_updates        | 8329     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.34e+03 |\n",
            "|    ep_rew_mean      | 171      |\n",
            "|    exploration_rate | 0.237    |\n",
            "| time/               |          |\n",
            "|    episodes         | 416      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3563     |\n",
            "|    total_timesteps  | 77120    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0158   |\n",
            "|    n_updates        | 8389     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.37e+03 |\n",
            "|    ep_rew_mean      | 181      |\n",
            "|    exploration_rate | 0.223    |\n",
            "| time/               |          |\n",
            "|    episodes         | 420      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3635     |\n",
            "|    total_timesteps  | 78482    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00961  |\n",
            "|    n_updates        | 8560     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.36e+03 |\n",
            "|    ep_rew_mean      | 181      |\n",
            "|    exploration_rate | 0.217    |\n",
            "| time/               |          |\n",
            "|    episodes         | 424      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3666     |\n",
            "|    total_timesteps  | 79067    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0102   |\n",
            "|    n_updates        | 8633     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.36e+03 |\n",
            "|    ep_rew_mean      | 181      |\n",
            "|    exploration_rate | 0.213    |\n",
            "| time/               |          |\n",
            "|    episodes         | 428      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3687     |\n",
            "|    total_timesteps  | 79455    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.023    |\n",
            "|    n_updates        | 8681     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.38e+03 |\n",
            "|    ep_rew_mean      | 183      |\n",
            "|    exploration_rate | 0.206    |\n",
            "| time/               |          |\n",
            "|    episodes         | 432      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3727     |\n",
            "|    total_timesteps  | 80219    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0103   |\n",
            "|    n_updates        | 8777     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.38e+03 |\n",
            "|    ep_rew_mean      | 183      |\n",
            "|    exploration_rate | 0.199    |\n",
            "| time/               |          |\n",
            "|    episodes         | 436      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3762     |\n",
            "|    total_timesteps  | 80883    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0159   |\n",
            "|    n_updates        | 8860     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.38e+03 |\n",
            "|    ep_rew_mean      | 183      |\n",
            "|    exploration_rate | 0.19     |\n",
            "| time/               |          |\n",
            "|    episodes         | 440      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3813     |\n",
            "|    total_timesteps  | 81841    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0117   |\n",
            "|    n_updates        | 8980     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.38e+03 |\n",
            "|    ep_rew_mean      | 186      |\n",
            "|    exploration_rate | 0.179    |\n",
            "| time/               |          |\n",
            "|    episodes         | 444      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3869     |\n",
            "|    total_timesteps  | 82931    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00619  |\n",
            "|    n_updates        | 9116     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.38e+03 |\n",
            "|    ep_rew_mean      | 187      |\n",
            "|    exploration_rate | 0.173    |\n",
            "| time/               |          |\n",
            "|    episodes         | 448      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3900     |\n",
            "|    total_timesteps  | 83501    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00628  |\n",
            "|    n_updates        | 9187     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.38e+03 |\n",
            "|    ep_rew_mean      | 188      |\n",
            "|    exploration_rate | 0.164    |\n",
            "| time/               |          |\n",
            "|    episodes         | 452      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3948     |\n",
            "|    total_timesteps  | 84426    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00622  |\n",
            "|    n_updates        | 9303     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.39e+03 |\n",
            "|    ep_rew_mean      | 189      |\n",
            "|    exploration_rate | 0.158    |\n",
            "| time/               |          |\n",
            "|    episodes         | 456      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 3982     |\n",
            "|    total_timesteps  | 85076    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00628  |\n",
            "|    n_updates        | 9384     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.38e+03 |\n",
            "|    ep_rew_mean      | 190      |\n",
            "|    exploration_rate | 0.149    |\n",
            "| time/               |          |\n",
            "|    episodes         | 460      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 4029     |\n",
            "|    total_timesteps  | 85974    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0136   |\n",
            "|    n_updates        | 9496     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.39e+03 |\n",
            "|    ep_rew_mean      | 191      |\n",
            "|    exploration_rate | 0.141    |\n",
            "| time/               |          |\n",
            "|    episodes         | 464      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 4069     |\n",
            "|    total_timesteps  | 86751    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00691  |\n",
            "|    n_updates        | 9593     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.4e+03  |\n",
            "|    ep_rew_mean      | 194      |\n",
            "|    exploration_rate | 0.131    |\n",
            "| time/               |          |\n",
            "|    episodes         | 468      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 4123     |\n",
            "|    total_timesteps  | 87768    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0101   |\n",
            "|    n_updates        | 9720     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.41e+03 |\n",
            "|    ep_rew_mean      | 195      |\n",
            "|    exploration_rate | 0.125    |\n",
            "| time/               |          |\n",
            "|    episodes         | 472      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 4155     |\n",
            "|    total_timesteps  | 88374    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00994  |\n",
            "|    n_updates        | 9796     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.38e+03 |\n",
            "|    ep_rew_mean      | 190      |\n",
            "|    exploration_rate | 0.118    |\n",
            "| time/               |          |\n",
            "|    episodes         | 476      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 4191     |\n",
            "|    total_timesteps  | 89056    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.00985  |\n",
            "|    n_updates        | 9881     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 2.42e+03 |\n",
            "|    ep_rew_mean      | 193      |\n",
            "|    exploration_rate | 0.105    |\n",
            "| time/               |          |\n",
            "|    episodes         | 480      |\n",
            "|    fps              | 21       |\n",
            "|    time_elapsed     | 4264     |\n",
            "|    total_timesteps  | 90435    |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0346   |\n",
            "|    loss             | 0.0157   |\n",
            "|    n_updates        | 10054    |\n",
            "----------------------------------\n",
            "Saving to logs//dqn/SpaceInvadersNoFrameskip-v4_5\n"
          ]
        }
      ],
      "source": [
        "!python -m rl_zoo3.train --algo dqn  --env SpaceInvadersNoFrameskip-v4 -f logs/ -c dqn.yml"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MJrya-12fkOj"
      },
      "source": [
        "##### Let's evaluate our agent\n",
        "\n",
        "- RL-Baselines3-Zoo provides `enjoy.py`, a python script to evaluate our agent. In most RL libraries, we call the evaluation script `enjoy.py`.\n",
        "- Let's evaluate it for 5000 timesteps 🔥"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e3RfInO5fkOk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a2ea18db-d3af-4810-c7ba-25e77aac39cb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2023-06-12 11:48:26.130721: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2023-06-12 11:48:27.071368: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "Loading latest experiment, id=5\n",
            "Loading logs/dqn/SpaceInvadersNoFrameskip-v4_5/SpaceInvadersNoFrameskip-v4.zip\n",
            "A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)\n",
            "[Powered by Stella]\n",
            "Stacking 4 frames\n",
            "Atari Episode Score: 65.00\n",
            "Atari Episode Length 2769\n",
            "Atari Episode Score: 90.00\n",
            "Atari Episode Length 1927\n",
            "Atari Episode Score: 70.00\n",
            "Atari Episode Length 2749\n",
            "Atari Episode Score: 140.00\n",
            "Atari Episode Length 3813\n",
            "Atari Episode Score: 100.00\n",
            "Atari Episode Length 2977\n",
            "Atari Episode Score: 65.00\n",
            "Atari Episode Length 2737\n",
            "Atari Episode Score: 115.00\n",
            "Atari Episode Length 2845\n"
          ]
        }
      ],
      "source": [
        "!python -m rl_zoo3.enjoy  --algo dqn  --env SpaceInvadersNoFrameskip-v4  --no-render  --n-timesteps 5000  --folder logs/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hqo1kg6MfkOk"
      },
      "source": [
        "##### Record  a Video"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6UnIN1zNfkOl"
      },
      "outputs": [],
      "source": [
        "# Set up display; otherwise rendering will fail\n",
        "import os\n",
        "os.system(\"Xvfb :1 -screen 0 1024x768x24 &\")\n",
        "os.environ['DISPLAY'] = ':1'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8msnPey0fkOl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9b9dda9e-47b5-48f9-caeb-5c7bcf6e6144"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2023-06-12 11:50:23.026603: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2023-06-12 11:50:24.000578: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "Loading latest experiment, id=5\n",
            "Loading logs/dqn/SpaceInvadersNoFrameskip-v4_5/SpaceInvadersNoFrameskip-v4.zip\n",
            "A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)\n",
            "[Powered by Stella]\n",
            "Stacking 4 frames\n",
            "Loading logs/dqn/SpaceInvadersNoFrameskip-v4_5/SpaceInvadersNoFrameskip-v4.zip\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.10/dist-packages/gymnasium/utils/passive_env_checker.py:364: UserWarning: \u001b[33mWARN: No render fps was declared in the environment (env.metadata['render_fps'] is None or not defined), rendering may occur at inconsistent fps.\u001b[0m\n",
            "  logger.warn(\n",
            "Saving video to /content/logs/dqn/SpaceInvadersNoFrameskip-v4_5/videos/final-model-dqn-SpaceInvadersNoFrameskip-v4-step-0-to-step-10000.mp4\n",
            "Moviepy - Building video /content/logs/dqn/SpaceInvadersNoFrameskip-v4_5/videos/final-model-dqn-SpaceInvadersNoFrameskip-v4-step-0-to-step-10000.mp4.\n",
            "Moviepy - Writing video /content/logs/dqn/SpaceInvadersNoFrameskip-v4_5/videos/final-model-dqn-SpaceInvadersNoFrameskip-v4-step-0-to-step-10000.mp4\n",
            "\n",
            "Moviepy - Done !\n",
            "Moviepy - video ready /content/logs/dqn/SpaceInvadersNoFrameskip-v4_5/videos/final-model-dqn-SpaceInvadersNoFrameskip-v4-step-0-to-step-10000.mp4\n"
          ]
        }
      ],
      "source": [
        "!python -m rl_zoo3.record_video --algo dqn --env SpaceInvadersNoFrameskip-v4 --exp-id 0 -f logs/ -n 10000"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sejFEJ2SfkOl"
      },
      "source": [
        "##### Display the video"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CJs-vHETfkOm"
      },
      "outputs": [],
      "source": [
        "import base64\n",
        "from pathlib import Path\n",
        "\n",
        "from IPython import display as ipythondisplay\n",
        "\n",
        "\n",
        "def show_videos(video_path=\"\", prefix=\"\"):\n",
        "    \"\"\"\n",
        "    Taken from https://github.com/eleurent/highway-env\n",
        "\n",
        "    :param video_path: (str) Path to the folder containing videos\n",
        "    :param prefix: (str) Filter the video, showing only the only starting with this prefix\n",
        "    \"\"\"\n",
        "    html = []\n",
        "    for mp4 in Path(video_path).glob(\"{}*.mp4\".format(prefix)):\n",
        "        video_b64 = base64.b64encode(mp4.read_bytes())\n",
        "        html.append(\n",
        "            \"\"\"<video alt=\"{}\" autoplay\n",
        "                    loop controls style=\"height: 400px;\">\n",
        "                    <source src=\"data:video/mp4;base64,{}\" type=\"video/mp4\" />\n",
        "                </video>\"\"\".format(\n",
        "                mp4, video_b64.decode(\"ascii\")\n",
        "            )\n",
        "        )\n",
        "    ipythondisplay.display(ipythondisplay.HTML(data=\"<br>\".join(html)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "5bebad92-1a17-43aa-eb69-4aa94ca3e7fa",
        "id": "b64oSoTUfkOm"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        }
      ],
      "source": [
        "show_videos(video_path='logs/dqg/SpaceInvadersNoFrameskip-v4/videos/', prefix='')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "8JiXuqRVfkOm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **DQN in PyTorch**\n",
        "\n",
        "In this section, **we'll train a Deep Q-Learning agent** in Gym environment using PyTorch."
      ],
      "metadata": {
        "id": "4_Nu0hojusCb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "* defining a class for DQN\n",
        "* Building the architecture of NN of DQN\n",
        "* Building the Replay Buffer and Storing the transition\n",
        "* Defining the epsilon-greedy policy\n",
        "* Define the Training phase\n",
        "* Updating the Target network\n",
        "* Training the DQN\n",
        "\n",
        "and that's it 😀\n",
        "\n",
        "**For Step-by-Step guide notebook for DQN check this [link](https://github.com/Curt-Park/rainbow-is-all-you-need).**"
      ],
      "metadata": {
        "id": "H6f9JU_iJ36I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Configurations for Colab\n",
        "import sys\n",
        "IN_COLAB = \"google.colab\" in sys.modules\n",
        "\n",
        "if IN_COLAB:\n",
        "    !apt install python-opengl\n",
        "    !apt install ffmpeg\n",
        "    !apt install xvfb\n",
        "    !pip install PyVirtualDisplay==3.0\n",
        "    !pip install gymnasium\n",
        "    !pip install stable-baselines3[extra]\n",
        "    !pip install wandb\n",
        "    !pip install gymnasium[atari,accept-rom-license]\n",
        "    import ale_py\n",
        "    # if using gymnasium\n",
        "    import shimmy\n",
        "    from pyvirtualdisplay import Display\n",
        "\n",
        "    # Start virtual display\n",
        "    dis = Display(visible=0, size=(400, 400))\n",
        "    dis.start()\n",
        "\n",
        "import os\n",
        "from typing import Dict, List, Tuple\n",
        "\n",
        "import gymnasium as gym\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "from IPython.display import clear_output\n",
        "\n",
        "import argparse\n",
        "import random\n",
        "import time\n",
        "from distutils.util import strtobool\n",
        "\n",
        "from stable_baselines3.common.atari_wrappers import (\n",
        "    ClipRewardEnv,\n",
        "    EpisodicLifeEnv,\n",
        "    FireResetEnv,\n",
        "    MaxAndSkipEnv,\n",
        "    NoopResetEnv,\n",
        ")\n",
        "from stable_baselines3.common.buffers import ReplayBuffer\n",
        "from torch.utils.tensorboard import SummaryWriter\n",
        "\n",
        "import stable_baselines3\n",
        "from typing import Callable\n"
      ],
      "metadata": {
        "id": "gmRxPn4G5oG9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4805a9c3-a7da-41de-bf33-0373b71b59c0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "\u001b[1;31mE: \u001b[0mUnable to locate package python-opengl\u001b[0m\n",
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "ffmpeg is already the newest version (7:4.4.2-0ubuntu0.22.04.1).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded.\n",
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "The following additional packages will be installed:\n",
            "  libfontenc1 libxfont2 libxkbfile1 x11-xkb-utils xfonts-base xfonts-encodings\n",
            "  xfonts-utils xserver-common\n",
            "The following NEW packages will be installed:\n",
            "  libfontenc1 libxfont2 libxkbfile1 x11-xkb-utils xfonts-base xfonts-encodings\n",
            "  xfonts-utils xserver-common xvfb\n",
            "0 upgraded, 9 newly installed, 0 to remove and 15 not upgraded.\n",
            "Need to get 7,812 kB of archives.\n",
            "After this operation, 11.9 MB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfontenc1 amd64 1:1.1.4-1build3 [14.7 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxfont2 amd64 1:2.0.5-1build1 [94.5 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxkbfile1 amd64 1:1.1.0-1build3 [71.8 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 x11-xkb-utils amd64 7.7+5build4 [172 kB]\n",
            "Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 xfonts-encodings all 1:1.0.5-0ubuntu2 [578 kB]\n",
            "Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 xfonts-utils amd64 1:7.7+6build2 [94.6 kB]\n",
            "Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 xfonts-base all 1:1.0.5 [5,896 kB]\n",
            "Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 xserver-common all 2:21.1.4-2ubuntu1.7~22.04.1 [28.0 kB]\n",
            "Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 xvfb amd64 2:21.1.4-2ubuntu1.7~22.04.1 [863 kB]\n",
            "Fetched 7,812 kB in 0s (40.8 MB/s)\n",
            "Selecting previously unselected package libfontenc1:amd64.\n",
            "(Reading database ... 120500 files and directories currently installed.)\n",
            "Preparing to unpack .../0-libfontenc1_1%3a1.1.4-1build3_amd64.deb ...\n",
            "Unpacking libfontenc1:amd64 (1:1.1.4-1build3) ...\n",
            "Selecting previously unselected package libxfont2:amd64.\n",
            "Preparing to unpack .../1-libxfont2_1%3a2.0.5-1build1_amd64.deb ...\n",
            "Unpacking libxfont2:amd64 (1:2.0.5-1build1) ...\n",
            "Selecting previously unselected package libxkbfile1:amd64.\n",
            "Preparing to unpack .../2-libxkbfile1_1%3a1.1.0-1build3_amd64.deb ...\n",
            "Unpacking libxkbfile1:amd64 (1:1.1.0-1build3) ...\n",
            "Selecting previously unselected package x11-xkb-utils.\n",
            "Preparing to unpack .../3-x11-xkb-utils_7.7+5build4_amd64.deb ...\n",
            "Unpacking x11-xkb-utils (7.7+5build4) ...\n",
            "Selecting previously unselected package xfonts-encodings.\n",
            "Preparing to unpack .../4-xfonts-encodings_1%3a1.0.5-0ubuntu2_all.deb ...\n",
            "Unpacking xfonts-encodings (1:1.0.5-0ubuntu2) ...\n",
            "Selecting previously unselected package xfonts-utils.\n",
            "Preparing to unpack .../5-xfonts-utils_1%3a7.7+6build2_amd64.deb ...\n",
            "Unpacking xfonts-utils (1:7.7+6build2) ...\n",
            "Selecting previously unselected package xfonts-base.\n",
            "Preparing to unpack .../6-xfonts-base_1%3a1.0.5_all.deb ...\n",
            "Unpacking xfonts-base (1:1.0.5) ...\n",
            "Selecting previously unselected package xserver-common.\n",
            "Preparing to unpack .../7-xserver-common_2%3a21.1.4-2ubuntu1.7~22.04.1_all.deb ...\n",
            "Unpacking xserver-common (2:21.1.4-2ubuntu1.7~22.04.1) ...\n",
            "Selecting previously unselected package xvfb.\n",
            "Preparing to unpack .../8-xvfb_2%3a21.1.4-2ubuntu1.7~22.04.1_amd64.deb ...\n",
            "Unpacking xvfb (2:21.1.4-2ubuntu1.7~22.04.1) ...\n",
            "Setting up libfontenc1:amd64 (1:1.1.4-1build3) ...\n",
            "Setting up xfonts-encodings (1:1.0.5-0ubuntu2) ...\n",
            "Setting up libxkbfile1:amd64 (1:1.1.0-1build3) ...\n",
            "Setting up libxfont2:amd64 (1:2.0.5-1build1) ...\n",
            "Setting up x11-xkb-utils (7.7+5build4) ...\n",
            "Setting up xfonts-utils (1:7.7+6build2) ...\n",
            "Setting up xfonts-base (1:1.0.5) ...\n",
            "Setting up xserver-common (2:21.1.4-2ubuntu1.7~22.04.1) ...\n",
            "Setting up xvfb (2:21.1.4-2ubuntu1.7~22.04.1) ...\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for fontconfig (2.13.1-4.2ubuntu5) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.1) ...\n",
            "Collecting PyVirtualDisplay==3.0\n",
            "  Downloading PyVirtualDisplay-3.0-py3-none-any.whl (15 kB)\n",
            "Installing collected packages: PyVirtualDisplay\n",
            "Successfully installed PyVirtualDisplay-3.0\n",
            "Collecting gymnasium\n",
            "  Downloading gymnasium-0.29.0-py3-none-any.whl (953 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m953.8/953.8 kB\u001b[0m \u001b[31m9.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium) (1.22.4)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium) (2.2.1)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium) (4.7.1)\n",
            "Collecting farama-notifications>=0.0.1 (from gymnasium)\n",
            "  Downloading Farama_Notifications-0.0.4-py3-none-any.whl (2.5 kB)\n",
            "Installing collected packages: farama-notifications, gymnasium\n",
            "Successfully installed farama-notifications-0.0.4 gymnasium-0.29.0\n",
            "Collecting stable-baselines3[extra]\n",
            "  Downloading stable_baselines3-2.0.0-py3-none-any.whl (178 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m178.4/178.4 kB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gymnasium==0.28.1 (from stable-baselines3[extra])\n",
            "  Downloading gymnasium-0.28.1-py3-none-any.whl (925 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m925.5/925.5 kB\u001b[0m \u001b[31m13.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (1.22.4)\n",
            "Requirement already satisfied: torch>=1.11 in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (2.0.1+cu118)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (2.2.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (1.5.3)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (3.7.1)\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (4.7.0.72)\n",
            "Requirement already satisfied: tensorboard>=2.9.1 in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (2.12.3)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (5.9.5)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (4.65.0)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (13.4.2)\n",
            "Collecting shimmy[atari]~=0.2.1 (from stable-baselines3[extra])\n",
            "  Downloading Shimmy-0.2.1-py3-none-any.whl (25 kB)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (9.4.0)\n",
            "Collecting autorom[accept-rom-license]~=0.6.0 (from stable-baselines3[extra])\n",
            "  Downloading AutoROM-0.6.1-py3-none-any.whl (9.4 kB)\n",
            "Requirement already satisfied: pygame in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (2.5.0)\n",
            "Collecting jax-jumpy>=1.0.0 (from gymnasium==0.28.1->stable-baselines3[extra])\n",
            "  Downloading jax_jumpy-1.0.0-py3-none-any.whl (20 kB)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium==0.28.1->stable-baselines3[extra]) (4.7.1)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from gymnasium==0.28.1->stable-baselines3[extra]) (0.0.4)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.6.0->stable-baselines3[extra]) (8.1.6)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.6.0->stable-baselines3[extra]) (2.27.1)\n",
            "Collecting AutoROM.accept-rom-license (from autorom[accept-rom-license]~=0.6.0->stable-baselines3[extra])\n",
            "  Downloading AutoROM.accept-rom-license-0.6.1.tar.gz (434 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m434.7/434.7 kB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting ale-py~=0.8.1 (from shimmy[atari]~=0.2.1->stable-baselines3[extra])\n",
            "  Downloading ale_py-0.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m26.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.56.2)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (2.17.3)\n",
            "Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.0.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (3.4.4)\n",
            "Requirement already satisfied: protobuf>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (3.20.3)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (67.7.2)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (0.7.1)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (2.3.6)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (0.41.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->stable-baselines3[extra]) (3.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->stable-baselines3[extra]) (1.11.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->stable-baselines3[extra]) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->stable-baselines3[extra]) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->stable-baselines3[extra]) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.11->stable-baselines3[extra]) (3.25.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.11->stable-baselines3[extra]) (16.0.6)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (0.11.0)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (4.41.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (1.4.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (23.1)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (3.1.0)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->stable-baselines3[extra]) (2022.7.1)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->stable-baselines3[extra]) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->stable-baselines3[extra]) (2.14.0)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.10/dist-packages (from ale-py~=0.8.1->shimmy[atari]~=0.2.1->stable-baselines3[extra]) (6.0.0)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]) (5.3.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]) (0.3.0)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]) (1.16.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.9.1->stable-baselines3[extra]) (1.3.1)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->stable-baselines3[extra]) (0.1.2)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.6.0->stable-baselines3[extra]) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.6.0->stable-baselines3[extra]) (2023.7.22)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.6.0->stable-baselines3[extra]) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.6.0->stable-baselines3[extra]) (3.4)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard>=2.9.1->stable-baselines3[extra]) (2.1.3)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.11->stable-baselines3[extra]) (1.3.0)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]) (0.5.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.9.1->stable-baselines3[extra]) (3.2.2)\n",
            "Building wheels for collected packages: AutoROM.accept-rom-license\n",
            "  Building wheel for AutoROM.accept-rom-license (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for AutoROM.accept-rom-license: filename=AutoROM.accept_rom_license-0.6.1-py3-none-any.whl size=446660 sha256=92f334b2dcef2d2f17adeb753f9b1bfefcd8d1c9620e3a18de9ed4dbf4972f36\n",
            "  Stored in directory: /root/.cache/pip/wheels/6b/1b/ef/a43ff1a2f1736d5711faa1ba4c1f61be1131b8899e6a057811\n",
            "Successfully built AutoROM.accept-rom-license\n",
            "Installing collected packages: jax-jumpy, ale-py, gymnasium, AutoROM.accept-rom-license, autorom, shimmy, stable-baselines3\n",
            "  Attempting uninstall: gymnasium\n",
            "    Found existing installation: gymnasium 0.29.0\n",
            "    Uninstalling gymnasium-0.29.0:\n",
            "      Successfully uninstalled gymnasium-0.29.0\n",
            "Successfully installed AutoROM.accept-rom-license-0.6.1 ale-py-0.8.1 autorom-0.6.1 gymnasium-0.28.1 jax-jumpy-1.0.0 shimmy-0.2.1 stable-baselines3-2.0.0\n",
            "Collecting wandb\n",
            "  Downloading wandb-0.15.8-py3-none-any.whl (2.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m12.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: Click!=8.0.0,>=7.1 in /usr/local/lib/python3.10/dist-packages (from wandb) (8.1.6)\n",
            "Collecting GitPython!=3.1.29,>=1.0.0 (from wandb)\n",
            "  Downloading GitPython-3.1.32-py3-none-any.whl (188 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m188.5/188.5 kB\u001b[0m \u001b[31m15.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: requests<3,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb) (2.27.1)\n",
            "Requirement already satisfied: psutil>=5.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb) (5.9.5)\n",
            "Collecting sentry-sdk>=1.0.0 (from wandb)\n",
            "  Downloading sentry_sdk-1.29.2-py2.py3-none-any.whl (215 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m215.6/215.6 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting docker-pycreds>=0.4.0 (from wandb)\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Requirement already satisfied: PyYAML in /usr/local/lib/python3.10/dist-packages (from wandb) (6.0.1)\n",
            "Collecting pathtools (from wandb)\n",
            "  Downloading pathtools-0.1.2.tar.gz (11 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting setproctitle (from wandb)\n",
            "  Downloading setproctitle-1.3.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from wandb) (67.7.2)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb) (1.4.4)\n",
            "Requirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /usr/local/lib/python3.10/dist-packages (from wandb) (3.20.3)\n",
            "Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from docker-pycreds>=0.4.0->wandb) (1.16.0)\n",
            "Collecting gitdb<5,>=4.0.1 (from GitPython!=3.1.29,>=1.0.0->wandb)\n",
            "  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.0.0->wandb) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.0.0->wandb) (2023.7.22)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.0.0->wandb) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.0.0->wandb) (3.4)\n",
            "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb)\n",
            "  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n",
            "Building wheels for collected packages: pathtools\n",
            "  Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8791 sha256=6d2db752726de681cadcad4a14e06cd5d0120831a5184591136d6485244a5402\n",
            "  Stored in directory: /root/.cache/pip/wheels/e7/f3/22/152153d6eb222ee7a56ff8617d80ee5207207a8c00a7aab794\n",
            "Successfully built pathtools\n",
            "Installing collected packages: pathtools, smmap, setproctitle, sentry-sdk, docker-pycreds, gitdb, GitPython, wandb\n",
            "Successfully installed GitPython-3.1.32 docker-pycreds-0.4.0 gitdb-4.0.10 pathtools-0.1.2 sentry-sdk-1.29.2 setproctitle-1.3.2 smmap-5.0.0 wandb-0.15.8\n",
            "Requirement already satisfied: gymnasium[accept-rom-license,atari] in /usr/local/lib/python3.10/dist-packages (0.28.1)\n",
            "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (1.22.4)\n",
            "Requirement already satisfied: jax-jumpy>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (1.0.0)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (2.2.1)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (4.7.1)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (0.0.4)\n",
            "Requirement already satisfied: shimmy[atari]<1.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (0.2.1)\n",
            "Collecting autorom[accept-rom-license]~=0.4.2 (from gymnasium[accept-rom-license,atari])\n",
            "  Downloading AutoROM-0.4.2-py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (8.1.6)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (2.27.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (4.65.0)\n",
            "Requirement already satisfied: AutoROM.accept-rom-license in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (0.6.1)\n",
            "Requirement already satisfied: ale-py~=0.8.1 in /usr/local/lib/python3.10/dist-packages (from shimmy[atari]<1.0,>=0.1.0->gymnasium[accept-rom-license,atari]) (0.8.1)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.10/dist-packages (from ale-py~=0.8.1->shimmy[atari]<1.0,>=0.1.0->gymnasium[accept-rom-license,atari]) (6.0.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (2023.7.22)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (3.4)\n",
            "Installing collected packages: autorom\n",
            "  Attempting uninstall: autorom\n",
            "    Found existing installation: AutoROM 0.6.1\n",
            "    Uninstalling AutoROM-0.6.1:\n",
            "      Successfully uninstalled AutoROM-0.6.1\n",
            "Successfully installed autorom-0.4.2\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n",
            "  if not hasattr(tensorboard, \"__version__\") or LooseVersion(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# From StableBaseline3\n",
        "class ReplayBuffer(BaseBuffer):\n",
        "    \"\"\"\n",
        "    Replay buffer used in off-policy algorithms like SAC/TD3.\n",
        "\n",
        "    :param buffer_size: Max number of element in the buffer\n",
        "    :param observation_space: Observation space\n",
        "    :param action_space: Action space\n",
        "    :param device: PyTorch device\n",
        "    :param n_envs: Number of parallel environments\n",
        "    :param optimize_memory_usage: Enable a memory efficient variant\n",
        "        of the replay buffer which reduces by almost a factor two the memory used,\n",
        "        at a cost of more complexity.\n",
        "        See https://github.com/DLR-RM/stable-baselines3/issues/37#issuecomment-637501195\n",
        "        and https://github.com/DLR-RM/stable-baselines3/pull/28#issuecomment-637559274\n",
        "        Cannot be used in combination with handle_timeout_termination.\n",
        "    :param handle_timeout_termination: Handle timeout termination (due to timelimit)\n",
        "        separately and treat the task as infinite horizon task.\n",
        "        https://github.com/DLR-RM/stable-baselines3/issues/284\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(\n",
        "        self,\n",
        "        buffer_size: int,\n",
        "        observation_space: spaces.Space,\n",
        "        action_space: spaces.Space,\n",
        "        device: Union[th.device, str] = \"auto\",\n",
        "        n_envs: int = 1,\n",
        "        optimize_memory_usage: bool = False,\n",
        "        handle_timeout_termination: bool = True,\n",
        "    ):\n",
        "        super().__init__(buffer_size, observation_space, action_space, device, n_envs=n_envs)\n",
        "\n",
        "        # Adjust buffer size\n",
        "        self.buffer_size = max(buffer_size // n_envs, 1)\n",
        "\n",
        "        # Check that the replay buffer can fit into the memory\n",
        "        if psutil is not None:\n",
        "            mem_available = psutil.virtual_memory().available\n",
        "\n",
        "        # there is a bug if both optimize_memory_usage and handle_timeout_termination are true\n",
        "        # see https://github.com/DLR-RM/stable-baselines3/issues/934\n",
        "        if optimize_memory_usage and handle_timeout_termination:\n",
        "            raise ValueError(\n",
        "                \"ReplayBuffer does not support optimize_memory_usage = True \"\n",
        "                \"and handle_timeout_termination = True simultaneously.\"\n",
        "            )\n",
        "        self.optimize_memory_usage = optimize_memory_usage\n",
        "\n",
        "        self.observations = np.zeros((self.buffer_size, self.n_envs) + self.obs_shape, dtype=observation_space.dtype)\n",
        "\n",
        "        if optimize_memory_usage:\n",
        "            # `observations` contains also the next observation\n",
        "            self.next_observations = None\n",
        "        else:\n",
        "            self.next_observations = np.zeros((self.buffer_size, self.n_envs) + self.obs_shape, dtype=observation_space.dtype)\n",
        "\n",
        "        self.actions = np.zeros((self.buffer_size, self.n_envs, self.action_dim), dtype=action_space.dtype)\n",
        "\n",
        "        self.rewards = np.zeros((self.buffer_size, self.n_envs), dtype=np.float32)\n",
        "        self.dones = np.zeros((self.buffer_size, self.n_envs), dtype=np.float32)\n",
        "        # Handle timeouts termination properly if needed\n",
        "        # see https://github.com/DLR-RM/stable-baselines3/issues/284\n",
        "        self.handle_timeout_termination = handle_timeout_termination\n",
        "        self.timeouts = np.zeros((self.buffer_size, self.n_envs), dtype=np.float32)\n",
        "\n",
        "        if psutil is not None:\n",
        "            total_memory_usage = self.observations.nbytes + self.actions.nbytes + self.rewards.nbytes + self.dones.nbytes\n",
        "\n",
        "            if self.next_observations is not None:\n",
        "                total_memory_usage += self.next_observations.nbytes\n",
        "\n",
        "            if total_memory_usage > mem_available:\n",
        "                # Convert to GB\n",
        "                total_memory_usage /= 1e9\n",
        "                mem_available /= 1e9\n",
        "                warnings.warn(\n",
        "                    \"This system does not have apparently enough memory to store the complete \"\n",
        "                    f\"replay buffer {total_memory_usage:.2f}GB > {mem_available:.2f}GB\"\n",
        "                )\n",
        "\n",
        "    def add(\n",
        "        self,\n",
        "        obs: np.ndarray,\n",
        "        next_obs: np.ndarray,\n",
        "        action: np.ndarray,\n",
        "        reward: np.ndarray,\n",
        "        done: np.ndarray,\n",
        "        infos: List[Dict[str, Any]],\n",
        "    ) -> None:\n",
        "        # Reshape needed when using multiple envs with discrete observations\n",
        "        # as numpy cannot broadcast (n_discrete,) to (n_discrete, 1)\n",
        "        if isinstance(self.observation_space, spaces.Discrete):\n",
        "            obs = obs.reshape((self.n_envs,) + self.obs_shape)\n",
        "            next_obs = next_obs.reshape((self.n_envs,) + self.obs_shape)\n",
        "\n",
        "        # Same, for actions\n",
        "        action = action.reshape((self.n_envs, self.action_dim))\n",
        "\n",
        "        # Copy to avoid modification by reference\n",
        "        self.observations[self.pos] = np.array(obs).copy()\n",
        "\n",
        "        if self.optimize_memory_usage:\n",
        "            self.observations[(self.pos + 1) % self.buffer_size] = np.array(next_obs).copy()\n",
        "        else:\n",
        "            self.next_observations[self.pos] = np.array(next_obs).copy()\n",
        "\n",
        "        self.actions[self.pos] = np.array(action).copy()\n",
        "        self.rewards[self.pos] = np.array(reward).copy()\n",
        "        self.dones[self.pos] = np.array(done).copy()\n",
        "\n",
        "        if self.handle_timeout_termination:\n",
        "            self.timeouts[self.pos] = np.array([info.get(\"TimeLimit.truncated\", False) for info in infos])\n",
        "\n",
        "        self.pos += 1\n",
        "        if self.pos == self.buffer_size:\n",
        "            self.full = True\n",
        "            self.pos = 0\n",
        "\n",
        "    def sample(self, batch_size: int, env: Optional[VecNormalize] = None) -> ReplayBufferSamples:\n",
        "        \"\"\"\n",
        "        Sample elements from the replay buffer.\n",
        "        Custom sampling when using memory efficient variant,\n",
        "        as we should not sample the element with index `self.pos`\n",
        "        See https://github.com/DLR-RM/stable-baselines3/pull/28#issuecomment-637559274\n",
        "\n",
        "        :param batch_size: Number of element to sample\n",
        "        :param env: associated gym VecEnv\n",
        "            to normalize the observations/rewards when sampling\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        if not self.optimize_memory_usage:\n",
        "            return super().sample(batch_size=batch_size, env=env)\n",
        "        # Do not sample the element with index `self.pos` as the transitions is invalid\n",
        "        # (we use only one array to store `obs` and `next_obs`)\n",
        "        if self.full:\n",
        "            batch_inds = (np.random.randint(1, self.buffer_size, size=batch_size) + self.pos) % self.buffer_size\n",
        "        else:\n",
        "            batch_inds = np.random.randint(0, self.pos, size=batch_size)\n",
        "        return self._get_samples(batch_inds, env=env)\n",
        "\n",
        "    def _get_samples(self, batch_inds: np.ndarray, env: Optional[VecNormalize] = None) -> ReplayBufferSamples:\n",
        "        # Sample randomly the env idx\n",
        "        env_indices = np.random.randint(0, high=self.n_envs, size=(len(batch_inds),))\n",
        "\n",
        "        if self.optimize_memory_usage:\n",
        "            next_obs = self._normalize_obs(self.observations[(batch_inds + 1) % self.buffer_size, env_indices, :], env)\n",
        "        else:\n",
        "            next_obs = self._normalize_obs(self.next_observations[batch_inds, env_indices, :], env)\n",
        "\n",
        "        data = (\n",
        "            self._normalize_obs(self.observations[batch_inds, env_indices, :], env),\n",
        "            self.actions[batch_inds, env_indices, :],\n",
        "            next_obs,\n",
        "            # Only use dones that are not due to timeouts\n",
        "            # deactivated by default (timeouts is initialized as an array of False)\n",
        "            (self.dones[batch_inds, env_indices] * (1 - self.timeouts[batch_inds, env_indices])).reshape(-1, 1),\n",
        "            self._normalize_reward(self.rewards[batch_inds, env_indices].reshape(-1, 1), env),\n",
        "        )\n",
        "        return ReplayBufferSamples(*tuple(map(self.to_torch, data)))"
      ],
      "metadata": {
        "id": "RD441gxG5qHI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C7n2vOST7V5D"
      },
      "source": [
        "### **DQN Network**\n",
        "\n",
        "I implement the Article 2015: The input to the neural network consists of an 84x84x4 image produced by the preprocessing map w. The first hidden layer convolves 32 filters of 8 x 8 with stride 4 with the input image and applies a rectifier nonlinearity. The second hidden layer convolves 64 filters of 4 x 4 with stride 2, again followed by a rectifier nonlinearity. Thisisfollowed by a third convolutional layerthat convolves 64 filters of 3 x 3 3with stride 1 followed by a rectifier. The final hidden layer is fully-connected and consists of 512 rectifier units. The output layer is a fully-connected linear layer with a single output for each valid action. The number of valid actions varied between 4 and 18 on the games we considered. (in my carRacing = 5) We refer to convolutional networks trained with our approach as Deep Q-Networks (DQN)."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "class QNetwork(nn.Module):\n",
        "    def __init__(self, num_actions):\n",
        "        super().__init__()\n",
        "        self.network = nn.Sequential(\n",
        "            nn.Conv2d(4, 32, 8, stride=4),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(32, 64, 4, stride=2),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(64, 64, 3, stride=1),\n",
        "            nn.ReLU(),\n",
        "            nn.Flatten(),\n",
        "            nn.Linear(3136, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, num_actions),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.network(x / 255.0)"
      ],
      "metadata": {
        "id": "g1d2FAvH5cou"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjtiI08xGTre"
      },
      "source": [
        "### **DQN Agent**\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "exp_name = \"DQN.py\"\n",
        "seed = 1\n",
        "torch_deterministic = True\n",
        "cuda = True\n",
        "track = True                             # experiment will be tracked with Weights and Biases\n",
        "wandb_project_name = \"Sarmad-DQN-cleanRL\"\n",
        "wandb_entity = None\n",
        "capture_video = False\n",
        "save_model = True                        # whether to save model into the `runs/{run_name}` folder\"\n",
        "upload_model = False                     # whether to upload the saved model to huggingface\n",
        "hf_entity = \"\"                           # the user or org name of the model repository from the Hugging Face Hub\n",
        "\n",
        "\n",
        "# Algorithm specific arguments\n",
        "env_id = \"Enduro-v4\"\n",
        "total_timesteps = 200000\n",
        "learning_rate = 1e-4      # the learning rate of the optimizer\n",
        "num_envs = 1              # the number of parallel game environments >> if num_env = 1 then vectorized envs are not supported at the moment\n",
        "buffer_size = 100000      # the replay memory buffer size\n",
        "gamma = 0.99              # the discount factor gamma\n",
        "tau = 1.                  # the target network update rate\n",
        "target_network_frequency = 1000  # the timesteps it takes to update the target network\n",
        "batch_size = 32           # the batch size of sample from the reply memory\n",
        "start_e = 1               # the starting epsilon for exploration\n",
        "end_e = 0.01              # the ending epsilon for exploration\n",
        "exploration_fraction = 0.10  # the fraction of `total-timesteps` it takes from start-e to go end-e\n",
        "learning_starts = 80000   # timestep to start learning\n",
        "train_frequency = 4       # the frequency of training\n",
        "\n",
        "\n",
        "\n",
        "def make_env(env_id, seed, idx, capture_video, run_name):\n",
        "    def thunk():\n",
        "        if capture_video and idx == 0:\n",
        "            env = gym.make(env_id, render_mode=\"rgb_array\")\n",
        "            env = gym.wrappers.RecordVideo(env, f\"videos/{run_name}\")\n",
        "        else:\n",
        "            env = gym.make(env_id)\n",
        "        env = gym.wrappers.RecordEpisodeStatistics(env)\n",
        "        env = NoopResetEnv(env, noop_max=30)\n",
        "        env = MaxAndSkipEnv(env, skip=4)\n",
        "        env = EpisodicLifeEnv(env)\n",
        "        if \"FIRE\" in env.unwrapped.get_action_meanings():\n",
        "            env = FireResetEnv(env)\n",
        "        env = ClipRewardEnv(env)\n",
        "        env = gym.wrappers.ResizeObservation(env, (84, 84))\n",
        "        env = gym.wrappers.GrayScaleObservation(env)\n",
        "        env = gym.wrappers.FrameStack(env, 4)\n",
        "        env.action_space.seed(seed)\n",
        "\n",
        "        return env\n",
        "\n",
        "    return thunk\n",
        "\n",
        "\n",
        "# ALGO LOGIC: initialize agent here:\n",
        "class QNetwork(nn.Module):\n",
        "    def __init__(self, env):\n",
        "        super().__init__()\n",
        "        self.network = nn.Sequential(\n",
        "            nn.Conv2d(4, 32, 8, stride=4),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(32, 64, 4, stride=2),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(64, 64, 3, stride=1),\n",
        "            nn.ReLU(),\n",
        "            nn.Flatten(),\n",
        "            nn.Linear(3136, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, env.single_action_space.n),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.network(x / 255.0)\n",
        "\n",
        "\n",
        "def linear_schedule(start_e: float, end_e: float, duration: int, t: int):\n",
        "    slope = (end_e - start_e) / duration\n",
        "    return max(slope * t + start_e, end_e)\n",
        "\n",
        "def evaluate(\n",
        "    model_path: str,\n",
        "    make_env: Callable,\n",
        "    env_id: str,\n",
        "    eval_episodes: int,\n",
        "    run_name: str,\n",
        "    Model: torch.nn.Module,\n",
        "    device: torch.device = torch.device(\"cpu\"),\n",
        "    epsilon: float = 0.05,\n",
        "    capture_video: bool = True,\n",
        "):\n",
        "    envs = gym.vector.SyncVectorEnv([make_env(env_id, 0, 0, capture_video, run_name)])\n",
        "    model = Model(envs).to(device)\n",
        "    model.load_state_dict(torch.load(model_path, map_location=device))\n",
        "    model.eval()\n",
        "\n",
        "    obs, _ = envs.reset()\n",
        "    episodic_returns = []\n",
        "    while len(episodic_returns) < eval_episodes:\n",
        "        if random.random() < epsilon:\n",
        "            actions = np.array([envs.single_action_space.sample() for _ in range(envs.num_envs)])\n",
        "        else:\n",
        "            q_values = model(torch.Tensor(obs).to(device))\n",
        "            actions = torch.argmax(q_values, dim=1).cpu().numpy()\n",
        "        next_obs, _, _, _, infos = envs.step(actions)\n",
        "        if \"final_info\" in infos:\n",
        "            for info in infos[\"final_info\"]:\n",
        "                if \"episode\" not in info:\n",
        "                    continue\n",
        "                print(f\"eval_episode={len(episodic_returns)}, episodic_return={info['episode']['r']}\")\n",
        "                episodic_returns += [info[\"episode\"][\"r\"]]\n",
        "        obs = next_obs\n",
        "\n",
        "    return episodic_returns\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    import stable_baselines3 as sb3\n",
        "\n",
        "    #args = parse_args()\n",
        "    run_name = f\"{env_id}__{exp_name}__{seed}__{int(time.time())}\"\n",
        "    if track:\n",
        "        import wandb\n",
        "\n",
        "        wandb.init(\n",
        "            project=wandb_project_name,\n",
        "            entity=wandb_entity,\n",
        "            sync_tensorboard=True,\n",
        "            #config=vars(args),\n",
        "            name=run_name,\n",
        "            monitor_gym=True,\n",
        "            save_code=True,\n",
        "        )\n",
        "    writer = SummaryWriter(f\"runs/{run_name}\")\n",
        "    #writer.add_text(\n",
        "    #    \"hyperparameters\",\n",
        "    #    \"|param|value|\\n|-|-|\\n%s\" % (\"\\n\".join([f\"|{key}|{value}|\" for key, value in vars(args).items()])),\n",
        "    #)\n",
        "\n",
        "    # TRY NOT TO MODIFY: seeding\n",
        "    random.seed(seed)\n",
        "    np.random.seed(seed)\n",
        "    torch.manual_seed(seed)\n",
        "    torch.backends.cudnn.deterministic = torch_deterministic\n",
        "\n",
        "    device = torch.device(\"cuda\" if torch.cuda.is_available() and cuda else \"cpu\")\n",
        "\n",
        "    # env setup\n",
        "    envs = gym.vector.SyncVectorEnv(\n",
        "        [make_env(env_id, seed + i, i, capture_video, run_name) for i in range(num_envs)]\n",
        "    )\n",
        "    assert isinstance(envs.single_action_space, gym.spaces.Discrete), \"only discrete action space is supported\"\n",
        "\n",
        "    q_network = QNetwork(envs).to(device)\n",
        "    optimizer = optim.Adam(q_network.parameters(), lr=learning_rate)\n",
        "    target_network = QNetwork(envs).to(device)\n",
        "    target_network.load_state_dict(q_network.state_dict())\n",
        "\n",
        "    rb = ReplayBuffer(\n",
        "        buffer_size,\n",
        "        envs.single_observation_space,\n",
        "        envs.single_action_space,\n",
        "        device,\n",
        "        optimize_memory_usage=True,\n",
        "        handle_timeout_termination=False,\n",
        "    )\n",
        "    start_time = time.time()\n",
        "\n",
        "    # TRY NOT TO MODIFY: start the game\n",
        "    obs, _ = envs.reset(seed=seed)\n",
        "    for global_step in range(total_timesteps):\n",
        "        # ALGO LOGIC: put action logic here\n",
        "        epsilon = linear_schedule(start_e, end_e, exploration_fraction * total_timesteps, global_step)\n",
        "        if random.random() < epsilon:\n",
        "            actions = np.array([envs.single_action_space.sample() for _ in range(envs.num_envs)])\n",
        "        else:\n",
        "            q_values = q_network(torch.Tensor(obs).to(device))\n",
        "            actions = torch.argmax(q_values, dim=1).cpu().numpy()\n",
        "\n",
        "        # TRY NOT TO MODIFY: execute the game and log data.\n",
        "        next_obs, rewards, terminated, truncated, infos = envs.step(actions)\n",
        "\n",
        "        # TRY NOT TO MODIFY: record rewards for plotting purposes\n",
        "        if \"final_info\" in infos:\n",
        "            for info in infos[\"final_info\"]:\n",
        "                # Skip the envs that are not done\n",
        "                if \"episode\" not in info:\n",
        "                    continue\n",
        "                print(f\"global_step={global_step}, episodic_return={info['episode']['r']}\")\n",
        "                writer.add_scalar(\"charts/episodic_return\", info[\"episode\"][\"r\"], global_step)\n",
        "                writer.add_scalar(\"charts/episodic_length\", info[\"episode\"][\"l\"], global_step)\n",
        "                writer.add_scalar(\"charts/epsilon\", epsilon, global_step)\n",
        "\n",
        "        # TRY NOT TO MODIFY: save data to reply buffer; handle `final_observation`\n",
        "        real_next_obs = next_obs.copy()\n",
        "        for idx, d in enumerate(truncated):\n",
        "            if d:\n",
        "                real_next_obs[idx] = infos[\"final_observation\"][idx]\n",
        "        rb.add(obs, real_next_obs, actions, rewards, terminated, infos)\n",
        "\n",
        "        # TRY NOT TO MODIFY: CRUCIAL step easy to overlook\n",
        "        obs = next_obs\n",
        "\n",
        "        # ALGO LOGIC: training.\n",
        "        if global_step > learning_starts:\n",
        "            if global_step % train_frequency == 0:\n",
        "                data = rb.sample(batch_size)\n",
        "                with torch.no_grad():\n",
        "                    target_max, _ = target_network(data.next_observations).max(dim=1)\n",
        "                    td_target = data.rewards.flatten() + gamma * target_max * (1 - data.dones.flatten())\n",
        "                old_val = q_network(data.observations).gather(1, data.actions).squeeze()\n",
        "                loss = F.mse_loss(td_target, old_val)\n",
        "\n",
        "                if global_step % 100 == 0:\n",
        "                    writer.add_scalar(\"losses/td_loss\", loss, global_step)\n",
        "                    writer.add_scalar(\"losses/q_values\", old_val.mean().item(), global_step)\n",
        "                    print(\"SPS:\", int(global_step / (time.time() - start_time)))\n",
        "                    writer.add_scalar(\"charts/SPS\", int(global_step / (time.time() - start_time)), global_step)\n",
        "\n",
        "                # optimize the model\n",
        "                optimizer.zero_grad()\n",
        "                loss.backward()\n",
        "                optimizer.step()\n",
        "\n",
        "            # update target network\n",
        "            if global_step % target_network_frequency == 0:\n",
        "                for target_network_param, q_network_param in zip(target_network.parameters(), q_network.parameters()):\n",
        "                    target_network_param.data.copy_(\n",
        "                        tau * q_network_param.data + (1.0 - tau) * target_network_param.data\n",
        "                    )\n",
        "\n",
        "    if save_model:\n",
        "        model_path = f\"runs/{run_name}/{exp_name}.cleanrl_model\"\n",
        "        torch.save(q_network.state_dict(), model_path)\n",
        "        print(f\"model saved to {model_path}\")\n",
        "\n",
        "        episodic_returns = evaluate(\n",
        "            model_path,\n",
        "            make_env,\n",
        "            env_id,\n",
        "            eval_episodes=10,\n",
        "            run_name=f\"{run_name}-eval\",\n",
        "            Model=QNetwork,\n",
        "            device=device,\n",
        "            epsilon=0.05,\n",
        "        )\n",
        "        for idx, episodic_return in enumerate(episodic_returns):\n",
        "            writer.add_scalar(\"eval/episodic_return\", episodic_return, idx)\n",
        "\n",
        "        if upload_model:\n",
        "            from cleanrl_utils.huggingface import push_to_hub\n",
        "\n",
        "            repo_name = f\"{env_id}-{exp_name}-seed{seed}\"\n",
        "            repo_id = f\"{hf_entity}/{repo_name}\" if hf_entity else repo_name\n",
        "            #push_to_hub(args, episodic_returns, repo_id, \"DQN\", f\"runs/{run_name}\", f\"videos/{run_name}-eval\")\n",
        "\n",
        "    envs.close()\n",
        "    writer.close()"
      ],
      "metadata": {
        "id": "xZ2_oiH5IDVo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "9049bf20-60a1-4f97-abd2-bb97df62070d"
      },
      "execution_count": null,
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/notebook/utils.py:280: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n",
            "  return LooseVersion(v) >= LooseVersion(check)\n"
          ]
        },
        {
          "data": {
            "application/javascript": [
              "\n",
              "        window._wandbApiKey = new Promise((resolve, reject) => {\n",
              "            function loadScript(url) {\n",
              "            return new Promise(function(resolve, reject) {\n",
              "                let newScript = document.createElement(\"script\");\n",
              "                newScript.onerror = reject;\n",
              "                newScript.onload = resolve;\n",
              "                document.body.appendChild(newScript);\n",
              "                newScript.src = url;\n",
              "            });\n",
              "            }\n",
              "            loadScript(\"https://cdn.jsdelivr.net/npm/postmate/build/postmate.min.js\").then(() => {\n",
              "            const iframe = document.createElement('iframe')\n",
              "            iframe.style.cssText = \"width:0;height:0;border:none\"\n",
              "            document.body.appendChild(iframe)\n",
              "            const handshake = new Postmate({\n",
              "                container: iframe,\n",
              "                url: 'https://wandb.ai/authorize'\n",
              "            });\n",
              "            const timeout = setTimeout(() => reject(\"Couldn't auto authenticate\"), 5000)\n",
              "            handshake.then(function(child) {\n",
              "                child.on('authorize', data => {\n",
              "                    clearTimeout(timeout)\n",
              "                    resolve(data)\n",
              "                });\n",
              "            });\n",
              "            })\n",
              "        });\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Logging into wandb.ai. (Learn how to deploy a W&B server locally: https://wandb.me/wandb-server)\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: You can find your API key in your browser here: https://wandb.ai/authorize\n",
            "wandb: Paste an API key from your profile and hit enter, or press ctrl+c to quit:\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: /root/.netrc\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "Tracking run with wandb version 0.15.8"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Run data is saved locally in <code>/content/wandb/run-20230806_185404-50smjej9</code>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Syncing run <strong><a href='https://wandb.ai/sarmadzandi4216/Sarmad-DQN-cleanRL/runs/50smjej9' target=\"_blank\">Enduro-v4__DQN.py__1__1691347987</a></strong> to <a href='https://wandb.ai/sarmadzandi4216/Sarmad-DQN-cleanRL' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View project at <a href='https://wandb.ai/sarmadzandi4216/Sarmad-DQN-cleanRL' target=\"_blank\">https://wandb.ai/sarmadzandi4216/Sarmad-DQN-cleanRL</a>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View run at <a href='https://wandb.ai/sarmadzandi4216/Sarmad-DQN-cleanRL/runs/50smjej9' target=\"_blank\">https://wandb.ai/sarmadzandi4216/Sarmad-DQN-cleanRL/runs/50smjej9</a>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "global_step=1104, episodic_return=[0.]\n",
            "global_step=2212, episodic_return=[0.]\n",
            "global_step=3306, episodic_return=[0.]\n",
            "global_step=4408, episodic_return=[0.]\n",
            "global_step=5515, episodic_return=[0.]\n",
            "global_step=6613, episodic_return=[0.]\n",
            "global_step=7720, episodic_return=[0.]\n",
            "global_step=8826, episodic_return=[0.]\n",
            "global_step=9928, episodic_return=[0.]\n",
            "global_step=11026, episodic_return=[0.]\n",
            "global_step=12122, episodic_return=[0.]\n",
            "global_step=13231, episodic_return=[0.]\n",
            "global_step=14326, episodic_return=[0.]\n",
            "global_step=15435, episodic_return=[0.]\n",
            "global_step=16539, episodic_return=[0.]\n",
            "global_step=17646, episodic_return=[0.]\n",
            "global_step=18751, episodic_return=[0.]\n",
            "global_step=19854, episodic_return=[0.]\n",
            "global_step=20957, episodic_return=[0.]\n",
            "global_step=22054, episodic_return=[0.]\n",
            "global_step=23163, episodic_return=[0.]\n",
            "global_step=24263, episodic_return=[0.]\n",
            "global_step=25370, episodic_return=[0.]\n",
            "global_step=26475, episodic_return=[0.]\n",
            "global_step=27584, episodic_return=[0.]\n",
            "global_step=28688, episodic_return=[0.]\n",
            "global_step=29788, episodic_return=[0.]\n",
            "global_step=30893, episodic_return=[0.]\n",
            "global_step=32005, episodic_return=[0.]\n",
            "global_step=33109, episodic_return=[0.]\n",
            "global_step=34212, episodic_return=[0.]\n",
            "global_step=35314, episodic_return=[0.]\n",
            "global_step=36423, episodic_return=[0.]\n",
            "global_step=37530, episodic_return=[0.]\n",
            "global_step=38629, episodic_return=[0.]\n",
            "global_step=39738, episodic_return=[0.]\n",
            "global_step=40834, episodic_return=[0.]\n",
            "global_step=41937, episodic_return=[0.]\n",
            "global_step=43044, episodic_return=[0.]\n",
            "global_step=44152, episodic_return=[0.]\n",
            "global_step=45254, episodic_return=[0.]\n",
            "global_step=46355, episodic_return=[0.]\n",
            "global_step=47460, episodic_return=[0.]\n",
            "global_step=48570, episodic_return=[0.]\n",
            "global_step=49670, episodic_return=[0.]\n",
            "global_step=50780, episodic_return=[0.]\n",
            "global_step=51886, episodic_return=[0.]\n",
            "global_step=52982, episodic_return=[0.]\n",
            "global_step=54086, episodic_return=[0.]\n",
            "global_step=55184, episodic_return=[0.]\n",
            "global_step=56284, episodic_return=[0.]\n",
            "global_step=57386, episodic_return=[0.]\n",
            "global_step=58490, episodic_return=[0.]\n",
            "global_step=59608, episodic_return=[0.]\n",
            "global_step=60724, episodic_return=[0.]\n",
            "global_step=61827, episodic_return=[0.]\n",
            "global_step=62928, episodic_return=[0.]\n",
            "global_step=64039, episodic_return=[0.]\n",
            "global_step=65147, episodic_return=[0.]\n",
            "global_step=66240, episodic_return=[0.]\n",
            "global_step=67350, episodic_return=[0.]\n",
            "global_step=68464, episodic_return=[0.]\n",
            "global_step=69563, episodic_return=[0.]\n",
            "global_step=70668, episodic_return=[0.]\n",
            "global_step=71769, episodic_return=[0.]\n",
            "global_step=72876, episodic_return=[0.]\n",
            "global_step=73985, episodic_return=[0.]\n",
            "global_step=75088, episodic_return=[0.]\n",
            "global_step=76179, episodic_return=[0.]\n",
            "global_step=77284, episodic_return=[0.]\n",
            "global_step=78383, episodic_return=[0.]\n",
            "global_step=79478, episodic_return=[0.]\n",
            "SPS: 210\n",
            "SPS: 210\n",
            "SPS: 210\n",
            "SPS: 210\n",
            "SPS: 209\n",
            "global_step=80583, episodic_return=[0.]\n",
            "SPS: 209\n",
            "SPS: 209\n",
            "SPS: 209\n",
            "SPS: 209\n",
            "SPS: 209\n",
            "SPS: 209\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "global_step=81688, episodic_return=[0.]\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 208\n",
            "SPS: 207\n",
            "SPS: 207\n",
            "SPS: 207\n",
            "global_step=82783, episodic_return=[0.]\n",
            "SPS: 207\n",
            "SPS: 207\n",
            "SPS: 207\n",
            "SPS: 207\n",
            "SPS: 207\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "global_step=83887, episodic_return=[0.]\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "SPS: 206\n",
            "SPS: 205\n",
            "SPS: 205\n",
            "SPS: 205\n",
            "SPS: 205\n",
            "SPS: 205\n",
            "global_step=84985, episodic_return=[0.]\n",
            "SPS: 205\n",
            "SPS: 205\n",
            "SPS: 205\n",
            "SPS: 205\n",
            "SPS: 205\n",
            "SPS: 205\n",
            "SPS: 204\n",
            "SPS: 204\n",
            "SPS: 204\n",
            "SPS: 204\n",
            "SPS: 204\n",
            "global_step=86096, episodic_return=[0.]\n",
            "SPS: 204\n",
            "SPS: 204\n",
            "SPS: 204\n",
            "SPS: 204\n",
            "SPS: 204\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "global_step=87196, episodic_return=[0.]\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "SPS: 203\n",
            "global_step=88289, episodic_return=[0.]\n",
            "SPS: 202\n",
            "SPS: 202\n",
            "SPS: 202\n",
            "SPS: 202\n",
            "SPS: 202\n",
            "SPS: 202\n",
            "SPS: 202\n",
            "SPS: 202\n",
            "SPS: 202\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "global_step=89391, episodic_return=[0.]\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "SPS: 201\n",
            "global_step=90493, episodic_return=[0.]\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "global_step=91604, episodic_return=[11.]\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 200\n",
            "SPS: 199\n",
            "SPS: 199\n",
            "SPS: 199\n",
            "SPS: 199\n",
            "SPS: 199\n",
            "SPS: 199\n",
            "global_step=92697, episodic_return=[0.]\n",
            "SPS: 199\n",
            "SPS: 199\n",
            "SPS: 199\n",
            "SPS: 199\n",
            "SPS: 199\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "global_step=93798, episodic_return=[22.]\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 198\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "global_step=94903, episodic_return=[9.]\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "global_step=96002, episodic_return=[4.]\n",
            "SPS: 197\n",
            "SPS: 197\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "global_step=97110, episodic_return=[0.]\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 196\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "global_step=98216, episodic_return=[0.]\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "SPS: 195\n",
            "global_step=99321, episodic_return=[0.]\n",
            "SPS: 195\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "global_step=100425, episodic_return=[0.]\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 194\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "global_step=101534, episodic_return=[7.]\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "global_step=102643, episodic_return=[0.]\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 193\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "global_step=103745, episodic_return=[0.]\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "SPS: 192\n",
            "global_step=104843, episodic_return=[0.]\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "global_step=105950, episodic_return=[0.]\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 191\n",
            "SPS: 190\n",
            "global_step=107055, episodic_return=[0.]\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "global_step=108154, episodic_return=[10.]\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 190\n",
            "SPS: 189\n",
            "global_step=109260, episodic_return=[16.]\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "global_step=110367, episodic_return=[11.]\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "global_step=111466, episodic_return=[0.]\n",
            "SPS: 189\n",
            "SPS: 189\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "global_step=112580, episodic_return=[4.]\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "global_step=113686, episodic_return=[0.]\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 188\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "global_step=114791, episodic_return=[2.]\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "global_step=115899, episodic_return=[6.]\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 187\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "global_step=117005, episodic_return=[20.]\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "global_step=118108, episodic_return=[0.]\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 186\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "global_step=119219, episodic_return=[11.]\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "global_step=120318, episodic_return=[58.]\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "global_step=121422, episodic_return=[46.]\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 185\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "global_step=122513, episodic_return=[10.]\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "global_step=123618, episodic_return=[11.]\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "SPS: 184\n",
            "global_step=124720, episodic_return=[29.]\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "global_step=125817, episodic_return=[51.]\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "global_step=126915, episodic_return=[24.]\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 183\n",
            "SPS: 182\n",
            "global_step=128004, episodic_return=[89.]\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "global_step=129114, episodic_return=[32.]\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "global_step=130214, episodic_return=[77.]\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "global_step=131325, episodic_return=[28.]\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 182\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "global_step=132435, episodic_return=[11.]\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "global_step=133539, episodic_return=[4.]\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "global_step=134637, episodic_return=[33.]\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 181\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "global_step=135741, episodic_return=[23.]\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "global_step=136852, episodic_return=[22.]\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "global_step=137955, episodic_return=[34.]\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 180\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "global_step=139057, episodic_return=[44.]\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "global_step=140167, episodic_return=[60.]\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "global_step=141273, episodic_return=[46.]\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "global_step=142374, episodic_return=[24.]\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 179\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "global_step=143475, episodic_return=[67.]\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "global_step=144575, episodic_return=[45.]\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "global_step=145677, episodic_return=[57.]\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "global_step=146781, episodic_return=[81.]\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 178\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "global_step=147889, episodic_return=[99.]\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "global_step=148995, episodic_return=[40.]\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "global_step=150097, episodic_return=[67.]\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "global_step=151205, episodic_return=[47.]\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 177\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "global_step=152315, episodic_return=[70.]\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "global_step=153428, episodic_return=[86.]\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "global_step=154532, episodic_return=[46.]\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "global_step=155634, episodic_return=[100.]\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "global_step=156738, episodic_return=[54.]\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 176\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "global_step=157840, episodic_return=[45.]\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "global_step=158947, episodic_return=[113.]\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "global_step=160045, episodic_return=[108.]\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "global_step=161143, episodic_return=[123.]\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "global_step=162242, episodic_return=[121.]\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "global_step=163338, episodic_return=[105.]\n",
            "SPS: 175\n",
            "SPS: 175\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "global_step=164442, episodic_return=[79.]\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "global_step=165555, episodic_return=[126.]\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "global_step=166654, episodic_return=[116.]\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "global_step=167763, episodic_return=[96.]\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "global_step=168867, episodic_return=[145.]\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 174\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "global_step=169966, episodic_return=[134.]\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "global_step=171075, episodic_return=[49.]\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "global_step=172174, episodic_return=[111.]\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "global_step=173275, episodic_return=[79.]\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "global_step=174374, episodic_return=[106.]\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "global_step=175476, episodic_return=[73.]\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 173\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "global_step=176577, episodic_return=[130.]\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "global_step=177686, episodic_return=[105.]\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "global_step=178790, episodic_return=[105.]\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "global_step=179887, episodic_return=[110.]\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "global_step=180988, episodic_return=[139.]\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "global_step=182097, episodic_return=[95.]\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "SPS: 172\n",
            "global_step=183201, episodic_return=[109.]\n",
            "SPS: 172\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "global_step=184305, episodic_return=[158.]\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "global_step=185412, episodic_return=[98.]\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "global_step=186524, episodic_return=[132.]\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "global_step=187632, episodic_return=[120.]\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "global_step=188737, episodic_return=[125.]\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "global_step=189843, episodic_return=[147.]\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "global_step=190940, episodic_return=[126.]\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 171\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "global_step=192032, episodic_return=[135.]\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "global_step=193137, episodic_return=[135.]\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "global_step=194246, episodic_return=[119.]\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "global_step=195349, episodic_return=[115.]\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "global_step=196438, episodic_return=[127.]\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "global_step=197545, episodic_return=[116.]\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "global_step=198645, episodic_return=[88.]\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 170\n",
            "SPS: 169\n",
            "SPS: 169\n",
            "SPS: 169\n",
            "SPS: 169\n",
            "SPS: 169\n",
            "model saved to runs/Enduro-v4__DQN.py__1__1691347987/DQN.py.cleanrl_model\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/gymnasium/utils/passive_env_checker.py:364: UserWarning: \u001b[33mWARN: No render fps was declared in the environment (env.metadata['render_fps'] is None or not defined), rendering may occur at inconsistent fps.\u001b[0m\n",
            "  logger.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Moviepy - Building video /content/videos/Enduro-v4__DQN.py__1__1691347987-eval/rl-video-episode-0.mp4.\n",
            "Moviepy - Writing video /content/videos/Enduro-v4__DQN.py__1__1691347987-eval/rl-video-episode-0.mp4\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": []
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Moviepy - Done !\n",
            "Moviepy - video ready /content/videos/Enduro-v4__DQN.py__1__1691347987-eval/rl-video-episode-0.mp4\n",
            "eval_episode=0, episodic_return=[158.]\n",
            "Moviepy - Building video /content/videos/Enduro-v4__DQN.py__1__1691347987-eval/rl-video-episode-1.mp4.\n",
            "Moviepy - Writing video /content/videos/Enduro-v4__DQN.py__1__1691347987-eval/rl-video-episode-1.mp4\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": []
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Moviepy - Done !\n",
            "Moviepy - video ready /content/videos/Enduro-v4__DQN.py__1__1691347987-eval/rl-video-episode-1.mp4\n",
            "eval_episode=1, episodic_return=[118.]\n",
            "eval_episode=2, episodic_return=[130.]\n",
            "eval_episode=3, episodic_return=[129.]\n",
            "eval_episode=4, episodic_return=[143.]\n",
            "eval_episode=5, episodic_return=[129.]\n",
            "eval_episode=6, episodic_return=[120.]\n",
            "eval_episode=7, episodic_return=[102.]\n",
            "Moviepy - Building video /content/videos/Enduro-v4__DQN.py__1__1691347987-eval/rl-video-episode-8.mp4.\n",
            "Moviepy - Writing video /content/videos/Enduro-v4__DQN.py__1__1691347987-eval/rl-video-episode-8.mp4\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": []
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Moviepy - Done !\n",
            "Moviepy - video ready /content/videos/Enduro-v4__DQN.py__1__1691347987-eval/rl-video-episode-8.mp4\n",
            "eval_episode=8, episodic_return=[158.]\n",
            "eval_episode=9, episodic_return=[143.]\n"
          ]
        }
      ]
    }
  ]
}