a3_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "361425b0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fa8559b4",
   "metadata": {
    "id": "fa8559b4"
   },
   "source": [
    "#  Assignment 3 - Natural Language Generation 💬\n",
    "\n",
    "Welcome to the **third and last assignment** for the **CS-552: Modern NLP course**!\n",
    "\n",
    "> - 😀 Name: **Mathieu Desponds**\n",
    "> - ✉️ Email: **mathieu.desponds@epfl.ch**\n",
    "> - 🪪 SCIPER: **283229**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74ef5391",
   "metadata": {
    "id": "74ef5391"
   },
   "source": [
    "<div style=\"padding:15px 20px 20px 20px;border-left:3px solid green;background-color:#e4fae4;border-radius: 20px;\">\n",
    "\n",
    "## **Assignment Description**\n",
    "- In this assignment, you will be looking at natural language generation (NLG), precisely the task of summarization. You will be exploring ways to generate text and how fine-grained decisions of decoding parameters can affect the generations.\n",
    "    \n",
    "- You will not need to train any models in this assignment. A pretrained one is provided for you by Huggingface.\n",
    "    \n",
    "- In Part 1, you will implement two decoding algorithms (greedy and beam search), as well as two sampling algorithms (top-p and top-k) to replicate (to some extent) what one would get when using Huggingface's `generate` function that you've played with during the Week 7's exercise session.\n",
    "    \n",
    "- For Part 2, you will analyze how varying specific parameters of decoding and sampling algorithms can qualitatively affect the generation.\n",
    "\n",
    "- For Part 3, you will answer some questions on interpreting automatic NLG evaluation metrics.\n",
    "\n",
    "### Table of Contents\n",
    "- **[Setup](#setup)**\n",
    "    - [1) Google Setup](#1-google-colab-setup)\n",
    "    - [2) Local Setup](#2-local-setup)\n",
    "    - [3) Rest of the Setup](#3-rest-of-the-setup-colab-and-local)\n",
    "\n",
    "- **[Introduction: T5 Primer](#introduction-t5-primer)**\n",
    "\n",
    "- **[PART 1: Natural Language Generation Decoding and Sampling Algorithms](#part-1-natural-language-generation-decoding-and-sampling-algorithms)**\n",
    "    - [1.1) Implement decoding and sampling algorithms](#11-implement-decoding-and-sampling-algorithms)\n",
    "    - [1.2) Test your implementations](#12-testing-your-implementation)\n",
    "    \n",
    "- **[PART 2: Qualitative Evaluation of Generation Parameters](#part-2-qualitative-evaluation-of-generation-parameters)**\n",
    "    - [2.1) Beam size for beam-search](#21-beam-size-for-beam-search)\n",
    "    - [2.2) Length penalty for beam-search](#22-length-penalty-for-beam-search)\n",
    "    - [2.3) Top-k for top-k](#23-top-k-for-top-k)\n",
    "    - [2.4) Temperature for top-p](#25-temperature-for-top-p)\n",
    "\n",
    "- **[PART 3: Reflection on Automatic NLG Evaluation Metrics](#part-3-reflection-on-automatic-nlg-evaluation-metrics)**\n",
    "    - [3.1) Description](#31-description)\n",
    "    - [3.2) Task](#32-task)\n",
    "\n",
    "- **[PART 4: Checklist](#part-4-checklist)**\n",
    "    \n",
    "### Deliverables\n",
    "\n",
    "To give us the deliverables you will have to commit the following files if your github classroom repository:\n",
    "\n",
    "- ✅ The python files:\n",
    "    - [ ] `a3_decoding.py`\n",
    "    - [ ] `a3_sampling.py`\n",
    "    - [ ] `a3_utils.py`, if you added any helper functions\n",
    "\n",
    "- ✅ This jupyter notebook `a3_notebook.py` with \n",
    "    - [ ] the answers to Part 2 questions written out in their corresponding cells.\n",
    "        - [ ] Answers to (2.1) questions\n",
    "        - [ ] Answers to (2.2) questions\n",
    "        - [ ] Answers to (2.3) questions\n",
    "        - [ ] Answers to (2.4) questions\n",
    "    - [ ] the answers to Part 3 questions written out in its corresponding cell.\n",
    "\n",
    "### Expected Workload\n",
    "\n",
    "We expect the first part of the assignment, notably Beam search, to take the most out of the complete assignment. \n",
    "You can plan your workload according to that. Keep in mind that this is just our expectation, not a guarantee.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a39cf74a",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "952fafd7",
   "metadata": {},
   "source": [
    "This assignment doesn't require you to train a model, therefore you don't need a GPU. However, if you would like to get faster generations, you may choose to use Colab with GPUs.\n",
    "\n",
    "First, if you are using Google Colab, go through part (1) and (3) of the setup section. If not, and you are using a local machine, go through the part (2) and (3) of the setup section."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78f65721",
   "metadata": {
    "id": "78f65721"
   },
   "source": [
    "### 1) Google Colab Setup\n",
    "\n",
    "PYTHON VERSION: Colab uses **Python version 3.9.16** and that's the python version we will use to grade your assignments. If you use a different Python version, we will not deduct points, but you are taking the risk of showing different results at grading time.\n",
    "\n",
    "If you are using Google Colab for this assignment, you will need to run a few commands to set up our environment. If you are running this notebook on a local machine you can skip this section.\n",
    "Run the following cell to mount your Google Drive. Follow the popped window, sign in to your Google account. (The same non-EPFL account you used to store this notebook!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b57cae0",
   "metadata": {
    "id": "5b57cae0"
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "280557bc",
   "metadata": {
    "id": "280557bc"
   },
   "source": [
    "Now first click the 4th left-side button (named Files), then click the 2nd button that pops under in the columns (named Refresh), under \"/drive/MyDrive/\" find the Assignment 3 repository folder that you uploaded to your Google Drive, copy its path, and fill it in the cell below. If everything is working correctly, then running the folowing cell should print the filenames from the assignment:\n",
    "\n",
    "```\n",
    "['part3_flan_T5_automatic_eval.png', 'requirements_local.txt', 'a3_sampling.py', 'a3_hello.py', 'README.md', 'part3_flan_t5_generations.json', 'a3_decoding.py', 'a3_tests.py', 'part1_input_data.json', 'requirements_colab.txt', 'a3_utils.py', 'a3_notebook.ipynb']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7e8ea3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# TODO: Fill in the path where you download the Assignment folder into\n",
    "ROOT_PATH = \"/content/drive/MyDrive/...\" # Replace with your directory to A3 folder\n",
    "print(os.listdir(ROOT_PATH))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10c8ad1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(ROOT_PATH)\n",
    "\n",
    "from a3_hello import hello_A3\n",
    "hello_A3()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "263b2bc3",
   "metadata": {},
   "source": [
    "Before we start, we also need to run some boilerplate code to set up our environment, same as previous assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "863639a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "requirements = ROOT_PATH + \"/requirements_colab.txt\"\n",
    "%pip install -r {requirements}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5fe3a4f",
   "metadata": {},
   "source": [
    "### 2) Local Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6587440",
   "metadata": {},
   "source": [
    "PYTHON VERSION: To be consistent with the default Colab settings, we prefer if you all use **Python version 3.9.16**\n",
    "If you use a different Python version, we will not deduct points, but you are taking the risk of showing different results at grading time.\n",
    "\n",
    "Below please write your relative filepath to your assignment 3 folder. If everything is working correctly, then running the folowing cell should print the filenames from the assignment:\n",
    "\n",
    "```\n",
    "['part3_flan_T5_automatic_eval.png', 'requirements_local.txt', 'a3_sampling.py', 'a3_hello.py', 'README.md', 'part3_flan_t5_generations.json', 'a3_decoding.py', 'a3_tests.py', 'part1_input_data.json', 'requirements_colab.txt', 'a3_utils.py', 'a3_notebook.ipynb']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1f10ed21",
   "metadata": {
    "id": "1f10ed21",
    "outputId": "d84b4d9a-d459-4c8b-f6f9-4e211f33d166"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['.git', '.gitignore', '.ipynb_checkpoints', 'a3_decoding.py', 'a3_hello.py', 'a3_notebook.ipynb', 'a3_sampling.py', 'a3_tests.py', 'a3_utils.py', 'part1_input_data.json', 'part3_flan_T5_automatic_eval.png', 'part3_flan_t5_generations.json', 'README.md', 'requirements_colab.txt', 'requirements_local.txt']\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "# NOTE: DON'T DO THIS STEP AGAIN IF YOU ARE ON COLAB! Only local!\n",
    "\n",
    "# TODO: Fill in the path where you download the Assignment folder into\n",
    "ROOT_PATH = '.' # Replace with A3 directory path\n",
    "print(os.listdir(ROOT_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6356478",
   "metadata": {},
   "source": [
    "Before we start, we also need to run some boilerplate code to set up our environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "06442d05",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting ipykernel==5.5.6\n",
      "  Downloading ipykernel-5.5.6-py3-none-any.whl (121 kB)\n",
      "Collecting ipywidgets==8.0.5\n",
      "  Downloading ipywidgets-8.0.5-py3-none-any.whl (138 kB)Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Requirement already satisfied: setuptools==67.6.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from -r ./requirements_local.txt (line 3)) (67.6.0)\n",
      "Collecting wheel==0.40.0\n",
      "  Downloading wheel-0.40.0-py3-none-any.whl (64 kB)\n",
      "Requirement already satisfied: pandas==1.5.3 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from -r ./requirements_local.txt (line 5)) (1.5.3)\n",
      "Collecting transformers==4.27.3\n",
      "  Downloading transformers-4.27.3-py3-none-any.whl (6.8 MB)\n",
      "Collecting datasets==2.10.1\n",
      "  Downloading datasets-2.10.1-py3-none-any.whl (469 kB)\n",
      "Requirement already satisfied: evaluate==0.4.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from -r ./requirements_local.txt (line 8)) (0.4.0)\n",
      "Collecting torchmetrics==0.11.4\n",
      "  Downloading torchmetrics-0.11.4-py3-none-any.whl (519 kB)\n",
      "Collecting torchvision==0.15.1\n",
      "  Downloading torchvision-0.15.1-cp39-cp39-win_amd64.whl (1.2 MB)\n",
      "Collecting tqdm==4.65.0\n",
      "  Downloading tqdm-4.65.0-py3-none-any.whl (77 kB)\n",
      "Requirement already satisfied: sentencepiece==0.1.97 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from -r ./requirements_local.txt (line 12)) (0.1.97)\n",
      "Requirement already satisfied: jupyter-client in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (6.1.12)\n",
      "Requirement already satisfied: tornado>=4.2 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (6.1)\n",
      "Requirement already satisfied: ipython>=5.0.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (7.29.0)\n",
      "Requirement already satisfied: ipython-genutils in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (0.2.0)\n",
      "Requirement already satisfied: traitlets>=4.1.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (5.1.0)\n",
      "Collecting jupyterlab-widgets~=3.0\n",
      "  Downloading jupyterlab_widgets-3.0.7-py3-none-any.whl (198 kB)\n",
      "Collecting widgetsnbextension~=4.0\n",
      "  Downloading widgetsnbextension-4.0.7-py3-none-any.whl (2.1 MB)\n",
      "Requirement already satisfied: numpy>=1.20.3 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from pandas==1.5.3->-r ./requirements_local.txt (line 5)) (1.22.4)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from pandas==1.5.3->-r ./requirements_local.txt (line 5)) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from pandas==1.5.3->-r ./requirements_local.txt (line 5)) (2021.3)\n",
      "Requirement already satisfied: pyyaml>=5.1 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from transformers==4.27.3->-r ./requirements_local.txt (line 6)) (5.4.1)\n",
      "Requirement already satisfied: filelock in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from transformers==4.27.3->-r ./requirements_local.txt (line 6)) (3.3.1)\n",
      "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from transformers==4.27.3->-r ./requirements_local.txt (line 6)) (0.13.2)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from transformers==4.27.3->-r ./requirements_local.txt (line 6)) (0.11.1)\n",
      "Requirement already satisfied: regex!=2019.12.17 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from transformers==4.27.3->-r ./requirements_local.txt (line 6)) (2021.8.3)\n",
      "Requirement already satisfied: requests in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from transformers==4.27.3->-r ./requirements_local.txt (line 6)) (2.28.2)\n",
      "Requirement already satisfied: packaging>=20.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from transformers==4.27.3->-r ./requirements_local.txt (line 6)) (23.0)\n",
      "Requirement already satisfied: multiprocess in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from datasets==2.10.1->-r ./requirements_local.txt (line 7)) (0.70.14)\n",
      "Requirement already satisfied: responses<0.19 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from datasets==2.10.1->-r ./requirements_local.txt (line 7)) (0.18.0)\n",
      "Requirement already satisfied: fsspec[http]>=2021.11.1 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from datasets==2.10.1->-r ./requirements_local.txt (line 7)) (2023.1.0)\n",
      "Requirement already satisfied: pyarrow>=6.0.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from datasets==2.10.1->-r ./requirements_local.txt (line 7)) (9.0.0)\n",
      "Requirement already satisfied: dill<0.3.7,>=0.3.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from datasets==2.10.1->-r ./requirements_local.txt (line 7)) (0.3.6)\n",
      "Requirement already satisfied: aiohttp in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from datasets==2.10.1->-r ./requirements_local.txt (line 7)) (3.8.4)\n",
      "Requirement already satisfied: xxhash in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from datasets==2.10.1->-r ./requirements_local.txt (line 7)) (3.2.0)\n",
      "Requirement already satisfied: torch>=1.8.1 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from torchmetrics==0.11.4->-r ./requirements_local.txt (line 9)) (2.0.0)\n",
      "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from torchvision==0.15.1->-r ./requirements_local.txt (line 10)) (8.4.0)\n",
      "Requirement already satisfied: colorama in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from tqdm==4.65.0->-r ./requirements_local.txt (line 11)) (0.4.4)\n",
      "Requirement already satisfied: sympy in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from torch>=1.8.1->torchmetrics==0.11.4->-r ./requirements_local.txt (line 9)) (1.9)\n",
      "Requirement already satisfied: jinja2 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from torch>=1.8.1->torchmetrics==0.11.4->-r ./requirements_local.txt (line 9)) (2.11.3)\n",
      "Requirement already satisfied: networkx in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from torch>=1.8.1->torchmetrics==0.11.4->-r ./requirements_local.txt (line 9)) (2.6.3)\n",
      "Requirement already satisfied: typing-extensions in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from torch>=1.8.1->torchmetrics==0.11.4->-r ./requirements_local.txt (line 9)) (4.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from aiohttp->datasets==2.10.1->-r ./requirements_local.txt (line 7)) (1.3.1)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from aiohttp->datasets==2.10.1->-r ./requirements_local.txt (line 7)) (1.3.3)\n",
      "Requirement already satisfied: attrs>=17.3.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from aiohttp->datasets==2.10.1->-r ./requirements_local.txt (line 7)) (21.2.0)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from aiohttp->datasets==2.10.1->-r ./requirements_local.txt (line 7)) (1.8.2)\n",
      "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from aiohttp->datasets==2.10.1->-r ./requirements_local.txt (line 7)) (3.1.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from aiohttp->datasets==2.10.1->-r ./requirements_local.txt (line 7)) (6.0.4)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from aiohttp->datasets==2.10.1->-r ./requirements_local.txt (line 7)) (4.0.2)\n",
      "Requirement already satisfied: backcall in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (0.2.0)\n",
      "Requirement already satisfied: matplotlib-inline in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (0.1.2)\n",
      "Requirement already satisfied: pygments in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (2.10.0)\n",
      "Requirement already satisfied: decorator in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (5.1.0)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (3.0.20)\n",
      "Requirement already satisfied: jedi>=0.16 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (0.18.0)\n",
      "Requirement already satisfied: pickleshare in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (0.7.5)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from jedi>=0.16->ipython>=5.0.0->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (0.8.2)\n",
      "Requirement already satisfied: wcwidth in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0.0->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (0.2.5)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from python-dateutil>=2.8.1->pandas==1.5.3->-r ./requirements_local.txt (line 5)) (1.16.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from requests->transformers==4.27.3->-r ./requirements_local.txt (line 6)) (2022.12.7)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from requests->transformers==4.27.3->-r ./requirements_local.txt (line 6)) (1.26.15)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from requests->transformers==4.27.3->-r ./requirements_local.txt (line 6)) (3.4)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from jinja2->torch>=1.8.1->torchmetrics==0.11.4->-r ./requirements_local.txt (line 9)) (2.0.1)\n",
      "Requirement already satisfied: pyzmq>=13 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from jupyter-client->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (22.2.1)\n",
      "Requirement already satisfied: jupyter-core>=4.6.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from jupyter-client->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (4.8.1)\n",
      "Requirement already satisfied: pywin32>=1.0 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from jupyter-core>=4.6.0->jupyter-client->ipykernel==5.5.6->-r ./requirements_local.txt (line 1)) (228)\n",
      "Requirement already satisfied: mpmath>=0.19 in c:\\users\\mathi\\anaconda3\\lib\\site-packages (from sympy->torch>=1.8.1->torchmetrics==0.11.4->-r ./requirements_local.txt (line 9)) (1.2.1)\n",
      "Installing collected packages: tqdm, widgetsnbextension, jupyterlab-widgets, datasets, wheel, transformers, torchvision, torchmetrics, ipywidgets, ipykernel\n",
      "  Attempting uninstall: tqdm\n",
      "    Found existing installation: tqdm 4.62.3\n",
      "    Uninstalling tqdm-4.62.3:\n",
      "      Successfully uninstalled tqdm-4.62.3\n",
      "  Attempting uninstall: widgetsnbextension\n",
      "    Found existing installation: widgetsnbextension 3.5.1\n",
      "    Uninstalling widgetsnbextension-3.5.1:\n",
      "      Successfully uninstalled widgetsnbextension-3.5.1\n",
      "  Attempting uninstall: jupyterlab-widgets\n",
      "    Found existing installation: jupyterlab-widgets 1.0.0\n",
      "    Uninstalling jupyterlab-widgets-1.0.0:\n",
      "      Successfully uninstalled jupyterlab-widgets-1.0.0\n",
      "  Attempting uninstall: datasets\n",
      "    Found existing installation: datasets 2.10.0\n",
      "    Uninstalling datasets-2.10.0:\n",
      "      Successfully uninstalled datasets-2.10.0\n",
      "  Attempting uninstall: wheel\n",
      "    Found existing installation: wheel 0.38.4\n",
      "    Uninstalling wheel-0.38.4:\n",
      "      Successfully uninstalled wheel-0.38.4\n",
      "  Attempting uninstall: transformers\n",
      "    Found existing installation: transformers 4.25.1\n",
      "    Uninstalling transformers-4.25.1:\n",
      "      Successfully uninstalled transformers-4.25.1\n",
      "  Attempting uninstall: torchvision\n",
      "    Found existing installation: torchvision 0.14.0\n",
      "    Uninstalling torchvision-0.14.0:\n",
      "      Successfully uninstalled torchvision-0.14.0\n",
      "  Attempting uninstall: torchmetrics\n",
      "    Found existing installation: torchmetrics 0.11.1\n",
      "    Uninstalling torchmetrics-0.11.1:\n",
      "      Successfully uninstalled torchmetrics-0.11.1\n",
      "  Attempting uninstall: ipywidgets\n",
      "    Found existing installation: ipywidgets 7.6.5\n",
      "    Uninstalling ipywidgets-7.6.5:\n",
      "      Successfully uninstalled ipywidgets-7.6.5\n",
      "  Attempting uninstall: ipykernel\n",
      "    Found existing installation: ipykernel 6.4.1\n",
      "    Uninstalling ipykernel-6.4.1:\n",
      "      Successfully uninstalled ipykernel-6.4.1\n",
      "Successfully installed datasets-2.10.1 ipykernel-5.5.6 ipywidgets-8.0.5 jupyterlab-widgets-3.0.7 torchmetrics-0.11.4 torchvision-0.15.1 tqdm-4.65.0 transformers-4.27.3 wheel-0.40.0 widgetsnbextension-4.0.7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -orch (c:\\users\\mathi\\anaconda3\\lib\\site-packages)\n"
     ]
    }
   ],
   "source": [
    "requirements = ROOT_PATH + \"/requirements_local.txt\"\n",
    "%pip install -r {requirements}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00b7e7e0",
   "metadata": {
    "id": "00b7e7e0"
   },
   "source": [
    "### 3) Rest of the setup (Colab and Local)\n",
    "\n",
    "Now, run this cell to load the autoreload extension. This allows us to edit `.py` source files, and re-import them into the notebook for a seamless editing and debugging experience! Otherwise, you will have to restart your kernel each time you make a change in your python files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "id": "3daa4c80",
   "metadata": {
    "id": "3daa4c80"
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from a3_utils import *\n",
    "from a3_decoding import *\n",
    "from a3_sampling import *\n",
    "from a3_tests import *\n",
    "\n",
    "import json\n",
    "import torch\n",
    "from datasets import load_dataset\n",
    "from transformers import (\n",
    "    T5Tokenizer,\n",
    "    T5ForConditionalGeneration\n",
    ")\n",
    "SCIPER = 283229\n",
    "seed = SCIPER\n",
    "torch.manual_seed(seed) # example to set the seed for randomness control\n",
    "torch.set_printoptions(precision=16) # set print number precision at 16"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41aa0625",
   "metadata": {},
   "source": [
    "(Optional) If you would like to setup CUDA with your notebook you can check whether it's available or not. If not, either your local machine doesn't have GPUs (which is fine!) or you need to set GPU setting via `Edit -> Notebook Settings` on Colab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7063a4e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "If you are using Colab, and want a GPU (not necessary for this assignment), please set GPU via Edit -> Notebook Settings.\n"
     ]
    }
   ],
   "source": [
    "if torch.cuda.is_available():\n",
    "    print('Good to go!')\n",
    "else:\n",
    "    print('If you are using Colab, and want a GPU (not necessary for this assignment), please set GPU via Edit -> Notebook Settings.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "CJvcVJzGM6Ow",
   "metadata": {
    "id": "CJvcVJzGM6Ow"
   },
   "source": [
    "And that's it! You are ready to start the assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "QyYxnYH2OOJd",
   "metadata": {
    "id": "QyYxnYH2OOJd"
   },
   "source": [
    "<a name=\"1\"></a>\n",
    "## **Introduction: T5 Primer**\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "AixGNwFjqfsQ",
   "metadata": {
    "id": "AixGNwFjqfsQ"
   },
   "source": [
    "![text_to_text.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAB84AAAKsCAIAAAAC5QBoAAAABGdBTUEAALGPC/xhBQAADD1pQ0NQSUNDIFByb2ZpbGUAAEiJlVcHWFPJFp5bkpBAaKFLCb0JIjWAlBBa6B3BRkgChBJjIKjYkUUF14KKBWzoqoiCFRA7olhYFHtfLKgo62LBrrxJAV33le9Nvrnz558z/zlz7tw7dwBQO8ERiXJRdQDyhAXiuJAA+tiUVDrpKcBkPwPgzuHmi5gxMREAlqH27+XddYBI2ysOUq1/9v/XosHj53MBQGIgTuflc/MgPgAAXs0ViQsAIEp586kFIimGFWiJYYAQL5TiTDmuluJ0Od4js0mIY0HcBoCSCocjzgRA9RLk6YXcTKih2g+xk5AnEAKgRofYNy9vMg/iNIhtoI0IYqk+I/0Hncy/aaYPa3I4mcNYPhdZUQoU5ItyOdP/z3T875KXKxnyYQWrSpY4NE46Z5i3mzmTw6VYBeI+YXpUNMSaEH8Q8GT2EKOULElootweNeTms2DOgA7ETjxOYDjEhhAHC3OjIhR8eoYgmA0xXCHoNEEBOwFiPYgX8vOD4hU2m8ST4xS+0MYMMYup4M9yxDK/Ul/3JTmJTIX+6yw+W6GPqRZlJSRDTIHYolCQFAWxKsSO+Tnx4QqbMUVZrKghG7EkThq/BcRxfGFIgFwfK8wQB8cp7Mvy8ofmi23KErCjFHhfQVZCqDw/WBuXI4sfzgW7xBcyE4d0+PljI4bmwuMHBsnnjj3jCxPjFTofRAUBcfKxOEWUG6Owx834uSFS3gxi1/zCeMVYPKkALki5Pp4hKohJkMeJF2VzwmLk8eDLQARggUBABxJY08FkkA0EnX1NffCfvCcYcIAYZAI+cFAwQyOSZT1CeI0HReBPiPggf3hcgKyXDwoh/3WYlV8dQIast1A2Igc8gTgPhINc+F8iGyUc9pYEHkNG8A/vHFi5MN5cWKX9/54fYr8zTMhEKBjJkEe62pAlMYgYSAwlBhNtcQPcF/fGI+DVH1ZnnIF7Ds3juz3hCaGL8JBwjdBNuDVJUCz+KcpI0A31gxW5SP8xF7gV1HTDA3AfqA6VcR3cADjgrtAPE/eDnt0gy1LELc0K/Sftv83gh7uhsCM7kVGyLtmfbPPzSFU7VbdhFWmuf8yPPNb04Xyzhnt+9s/6Ifs82Ib/bIktxPZj7dhJ7Bx2BGsCdOw41ox1YEeleHh1PZatriFvcbJ4cqCO4B/+hu6sNJP5TnVOvU5f5H0F/GnSdzRgTRZNFwsyswroTLgj8OlsIddxJN3ZydkFAOn+In99vYmV7RuITsd3bv4fAPgcHxwcPPydCzsOwF4P+Pgf+s7ZMODWoQzA2UNcibhQzuHSCwG+JdTgk6YPjIE5sIHzcQbuwBv4gyAQBqJBAkgBE2H0WXCdi8FUMBPMA6WgHCwDq8A6sBFsATvAbrAPNIEj4CQ4Ay6AS+AauANXTw94AfrBO/AZQRASQkVoiD5iglgi9ogzwkB8kSAkAolDUpA0JBMRIhJkJjIfKUcqkHXIZqQW2YscQk4i55Au5BbyAOlFXiOfUAxVQbVQI9QKHYUyUCYajiagE9BMdApahJagS9A1aA26C21ET6IX0GtoN/oCHYCbsjKmg5liDhgDY2HRWCqWgYmx2VgZVonVYPVYC7zPV7BurA/7iBNxGk7HHeAKDsUTcS4+BZ+NL8bX4TvwRrwNv4I/wPvxbwQqwZBgT/AisAljCZmEqYRSQiVhG+Eg4TR8lnoI74hEog7RmugBn8UUYjZxBnExcT2xgXiC2EV8RBwgkUj6JHuSDymaxCEVkEpJa0m7SMdJl0k9pA9KykomSs5KwUqpSkKlYqVKpZ1Kx5QuKz1V+kxWJ1uSvcjRZB55OnkpeSu5hXyR3EP+TNGgWFN8KAmUbMo8yhpKPeU05S7ljbKyspmyp3KsskB5rvIa5T3KZ5UfKH9U0VSxU2GpjFeRqCxR2a5yQuWWyhsqlWpF9aemUguoS6i11FPU+9QPqjRVR1W2Kk91jmqVaqPqZdWXamQ1SzWm2kS1IrVKtf1qF9X61MnqVuosdY76bPUq9UPqN9QHNGgaozWiNfI0Fmvs1Din8UyTpGmlGaTJ0yzR3KJ5SvMRDaOZ01g0Lm0+bSvtNK1Hi6hlrcXWytYq19qt1anVr62p7aqdpD1Nu0r7qHa3DqZjpcPWydVZqrNP57rOJ10jXaYuX3eRbr3uZd33eiP0/PX4emV6DXrX9D7p0/WD9HP0l+s36d8zwA3sDGINphpsMDht0DdCa4T3CO6IshH7Rtw2RA3tDOMMZxhuMewwHDAyNgoxEhmtNTpl1GesY+xvnG280viYca8JzcTXRGCy0uS4yXO6Np1Jz6WvobfR+00NTUNNJaabTTtNP5tZmyWaFZs1mN0zp5gzzDPMV5q3mvdbmFhEWsy0qLO4bUm2ZFhmWa62bLd8b2VtlWy1wKrJ6pm1njXbusi6zvquDdXGz2aKTY3NVVuiLcM2x3a97SU71M7NLsuuyu6iPWrvbi+wX2/fNZIw0nOkcGTNyBsOKg5Mh0KHOocHjjqOEY7Fjk2OL0dZjEodtXxU+6hvTm5OuU5bne6M1hwdNrp4dMvo1852zlznKuerLlSXYJc5Ls0ur1ztXfmuG1xvutHcIt0WuLW6fXX3cBe717v3elh4pHlUe9xgaDFiGIsZZz0JngGeczyPeH70cvcq8Nrn9Ze3g3eO907vZ2Osx/DHbB3zyMfMh+Oz2afbl+6b5rvJt9vP1I/jV+P30N/cn+e/zf8p05aZzdzFfBngFCAOOBjwnuXFmsU6EYgFhgSWBXYGaQYlBq0Luh9sFpwZXBfcH+IWMiPkRCghNDx0eegNthGby65l94d5hM0KawtXCY8PXxf+MMIuQhzREolGhkWuiLwbZRkljGqKBtHs6BXR92KsY6bEHI4lxsbEVsU+iRsdNzOuPZ4WPyl+Z/y7hICEpQl3Em0SJYmtSWpJ45Nqk94nByZXJHePHTV21tgLKQYpgpTmVFJqUuq21IFxQeNWjesZ7za+dPz1CdYTpk04N9FgYu7Eo5PUJnEm7U8jpCWn7Uz7wonm1HAG0tnp1en9XBZ3NfcFz5+3ktfL9+FX8J9m+GRUZDzL9Mlckdmb5ZdVmdUnYAnWCV5lh2ZvzH6fE52zPWcwNzm3IU8pLy3vkFBTmCNsm2w8edrkLpG9qFTUPcVryqop/eJw8bZ8JH9CfnOBFvyQ75DYSH6RPCj0Lawq/DA1aer+aRrThNM6pttNXzT9aVFw0W8z8BncGa0zTWfOm/lgFnPW5tnI7PTZrXPM55TM6ZkbMnfHPMq8nHm/FzsVVxS/nZ88v6XEqGRuyaNfQn6pK1UtFZfeWOC9YONCfKFgYecil0VrF30r45WdL3cqryz/spi7+Pyvo39d8+vgkowlnUvdl25YRlwmXHZ9ud/yHRUaFUUVj1ZErmhcSV9ZtvLtqkmrzlW6Vm5cTVktWd29JmJN81qLtcvWflmXte5aVUBVQ7Vh9aLq9+t56y9v8N9Qv9FoY/nGT5sEm25uDtncWGNVU7mFuKVwy5OtSVvbf2P8VrvNYFv5tq/bhdu7d8TtaKv1qK3dabhzaR1aJ6nr3TV+16Xdgbub6x3qNzfoNJTvAXske57vTdt7fV/4vtb9jP31BywPVB+kHSxrRBqnN/Y3ZTV1N6c0dx0KO9Ta4t1y8LDj4e1HTI9UHdU+uvQY5VjJscHjRccHTohO9J3MPPmodVLrnVNjT11ti23rPB1++uyZ4DOn2pntx8/6nD1yzuvcofOM800X3C80drh1HPzd7feDne6djRc9LjZf8rzU0jWm69hlv8snrwReOXOVffXCtahrXdcTr9+8Mf5G903ezWe3cm+9ul14+/OduXcJd8vuqd+rvG94v+YP2z8aut27jz4IfNDxMP7hnUfcRy8e5z/+0lPyhPqk8qnJ09pnzs+O9Ab3Xno+7nnPC9GLz32lf2r8Wf3S5uWBv/z/6ugf29/zSvxq8PXiN/pvtr91fds6EDNw/13eu8/vyz7of9jxkfGx/VPyp6efp34hfVnz1fZry7fwb3cH8wYHRRwxR/YpgMGKZmQA8Ho7ANQUAGjwfEYZJz//yQoiP7PKEPhPWH5GlBV3AOrh93tsH/y6uQHAnq3w+AX11cYDEEMFIMEToC4uw3XorCY7V0oLEZ4DNsV8Tc9LB/+myM+cP8T9cwukqq7g5/Zf4fh8Y+PMlM8AAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAAOpgAABdwnLpRPAAAAAlwSFlzAAAewwAAHsMBvJeX2gAAIABJREFUeJzs3T3P4za+N2A7SJknwDQpT5sqxQHm+3+CAU6RKu0p0wyQk95P4V2voxeaIinx7bqQYtdjWRQl0dLP/5u6Px6PGwAAAAAAkOqH2g0AAAAAAIC+idoBAAAAACCLqB0AAAAAALKI2gEAAAAAIIuoHQAAAAAAsojaAQAAAAAgi6gdAAAAAACy/Ji5/P1+L9IOAAA4z+PxqN0EAABgZPe0uw4JOwAAPZK5AwAAZ0iZQEbODgBAp1zKAgAAZzhW1e7OBACAMShvBwAACjpQ1S5nBwBgGC5uAQCAgmKr2sO3In/8+Xuh9gAAQEm//vJb4F/VtgMAAEVERe2BnF3IDgBA+/YCd1E7AABQRHrULmQHAKAvm4G7tB0AAMj3ea52OTsAAGPYvIg1aTsAAJDvwGNRX+TsAAB0yqUsAABwhg9R+7rGx80JAAAAAAC8S6lqBwCAfq1rR8whAwAAZBK1AwAAAABAlmNRu9ljAAAYgMtaAACgLFXtAAAAAACQRdQOAAAAAABZRO0AAAAAAJBF1A4AAAAAAFlE7QAAAAAAkEXUDgAAAAAAWUTtAAAAAACQRdQOAAAAAABZRO0AAAAAAJBF1A4AAAAAAFlE7QAAAAAAkEXUDgAAAAAAWUTtAAAAAACQRdQOAAAAAABZRO0AAAAAAJBF1A4AAAAAAFl+rN2A2+12+/WX32632x9//v7+f99fIcerP9f08AB63L/OccjnPKKKsw8814QAAEC/mojaoRGB2Hph1Ht+oUbL1sen3dSv+NHmnT3+zngFAABAU0aI2t1sh627JS3iyWEfnaeF/dujvo7JvX26KP+Ei/V1HgEAAMCpRojaoTixEe1Y5OzPg/P9xV9/+c0R253NXSa5BgAAgH55LCpAH/748/dXAvv+v2/+lAEAAACgNlXtwH+opW1NuMz5jz9/F7IzLeMVAAAATVHVDtCxV9oocy/u119+e/5XuyEAAABAB7qsag8/IXBhr+ptXSu6XjxcMZf8/r01RhboHV1vKYfWW2QfJay3opP272Ji7rLbHsgQD60ofx+l9VXOKsKvN3JMFtzvaW0+Ok4u3v/+5r1XCra51Dm4XraX2fAv+E4peB4d1c54lSZnbM9pc8FluzgLAAAAKmoial/cvFW5lztUtxjOGg4FWIsXj95sH11vmlrr3Vv1BevNVHD/Nl5RW+tcqKXiuRCQkFnfktocfzSu37n3SvjpoPGLxHxCs8dVvr6+U2pp6js08mzKaXPxZa85g1q4JgQAAEhzfzweoX++39//b5s3PGmFn3u3oO9zH8dkQJtzJceHR+vFP5bhJyy791EJiVX89iavMX7VxQ/InCLisvt3XQIc37AiW5GWZkZub2ZfFZHQS0XOhUOKVLXnnEdHx8nILgrXttcaYz82I2apTJkn4C1vzLlsbC/lmvEqU87YnrOPyi4b3+buLLYrfFUMAAAQ1kRVewve7xgjg6fX2xLmSl6v7rXsx6qxnGWPKrW9Oau+cntLOdrmveRxL+BoTS/Hc46K50KOsudR5Dj5/obNYH0z9Vs0+FZvjH0tVTdHPqTImNP+8VxE9e/Q9Xrjl43cR8WXjWwzAADA5DwW9XZLilE2F4m8jw0vuymc+FwQA+Vsb4Lq25sjp80dbW+RfdTR9r5cfC7kKHseHXp//hovHmP3Puf539EFL3beOdjU8Zyj+ndKrX3UwnnUl8VfcwIAAByiqv2YZm8yT2pYs9t7toQAgoWRemmYwLF9Ix02LdPPC611SM6YY7zK9ErbTSYDAAAcpaq9uRvsTePVGAbETNrQxV6L1NEkFe+mOiZfYo7JCzrk119+2/xv8Z5Fw9biD7m+Dk5YqDJeFRnbc8achGU7/T46yf1+V+QOAAAcoqo9Xa2Esdad8PXbG7PGk6bzljIcIp1pWcXzKNNUv+JUNFs/G6845H6/K28HAAAiidpTVAkm1k8kuywvmC2IIVLFY3Jyi+4d7wwdb4vaNFU/G69IJm0HAAAiidoPW2cT73fpZycX67DgdnJeUHF7xR9duP6Y5JDudkHdMXYec/az8Yo00nYAACCGqP2Y9/vzivfke4FI8VkgGtle2nfZMcnYjDnXmLyfjVfseebpm1O0S9sBAICPPBY1UTt343/8+fsFjamyvaOWVQ7vmmOSSJ2eRw6ha+hn4xVre5G6p6QCAABhovY+/PrLb8//ajfkCrOlHq/t7Wv/TnVMvgS294w5KHKOjdnOIwioMl4VGdtzxpyEZTv9PjqDtB0AAEggai9ptlvTC7Y3JimAs/UeWw9zHl3c2jl/T7r1dlQMKWfMaWq86vokkrYDAABHDRW1172Xu2bt7dyvprUkcqmPT+d7vthOb5QS2NhmNd68j462P7yPiodcH4tMA+0f7Dy6Pmevteq6Th3bq6vVzpyxPWfMSVg2POYc6sABTiKTswMAAIeM8FjUP/78/XUL9/wf76+cVN71WtHtkhvI9TYuXjxb5vZm7qP1IoPZ7J9b29lE9WMy09FjMvz+a5oa7ufBzqNeToTe1R3bL1NrvMoZ23PGnFLjldPwdrs9Ho91GbtHpAIAAJtGiNrXzrshXNysLlZ06n37+sMjs7ZDc0esp209aXs/Lri+t1+vPW3VMSKbV1Cgq8P9nLN/c5bdbFhC/tuOmJ0eLtI8b2OTV133PEpQcYydSsWxPVmn41Xy2B5e9vWGKsvORtoOAABEGiRq3yu5Oum2eb2ua+KqQGXZqRnfeqUJq0vbR3tJRGv5YBEVD61kVY7JUhKOySuHmoKr7us8auREaLM6u6C6Y3sVtcarnK7OPPHLLnu0vH2kk2gzbQcAAFj4UJKzuK/o/U4JOjVMWgHAi7G9usUvB4Gr4s2oXWE7AADwbqjHogIAQHFSdQAA4CNROzQhbTZ2AFpmbB+bWWUAAIB3onao7xm4bMYui2foXdcmAPIY2wejsB0AAAgzVzs0Yf30ueqPhQQgk7G9cfFztT+ZsR0AAAhQ1Q5NWKQtshiAARjbByNVBwAAAkTt0Io//vx9M3aRxQD0y9g+PDO2AwAATz/WbgDwD8IXgPEY24fxeDxk6wAAwCZV7QAAAAAAkEXUDgAAsdYztqtzBwAAbqJ2AAAAAADIJGoHAIADFLYDAABronYAAAAAAMgiagcAgGPWhe0AAMDkRO0AAJDLHDIAADA5UTsAAAAAAGQRtQMAwGHmkAEAAN6J2gEAoABzyAAAwMxE7QAATOfXX36r3QQAAGAox6J29yQAAPBkDhkAAOBFVTsAAHNZl4+UCs3NIQMAANP6ELWv7zoUtgMAAAAAwLuUqnZpOwAAnSp7KWsOGQAA4Olz1L55/yBtBwCgO5sXsWXjcnPIAADAnO4xtxaBG4Y//vy9aHsAAKC8vUqR/Jx9fams1B0AACYUFbXfPpXnCNwBAGhT+M8xi8Tii0tlUTsAAEwoNmq/+WNYAADGUioTF7UDAAAHHovqngEAgGGcd3GrQgUAACb046F3P29I3DwAANAvFSQAAEBxB6raX9ycAADQqTMuZV0eAwAAB+Zq315ehTsAAM07Ow1fXxXL3wEAYCrHJpBZcwsBAAAAAMDkUiaQAQAA3ilAAQCAyYnaAQAAAAAgi6gdAADK80wjAACYiqgdAAAAAACyiNoBAKAA07UDAMDMRO0AAAAAAJBF1A4AAKcwXTsAAMxD1A4AAAAAAFlE7QAAUIbp2gEAYFqidgAAAAAAyCJqBwCAs5iuHQAAJiFqBwAAAACALKJ2AAAoxnTtAAAwJ1E7AAAAAABkEbUDAMCJTNcOAAAzELUDAAAAAEAWUTsAAAAAAGQRtQMAQEmejAoAABMStQMAAAAAQBZROwAAnMuTUQEAYHiidgAAAAAAyCJqBwCAwkzXDgAAsxG1AwAAAABAFlE7AAAAAABkEbUDAMDpPBkVAADGJmoHAAAAAIAsonYAACjPk1EBAGAqonYAAAAAAMjyY+0GwD+YxhSAm3JgAAAAenN3K0sLJOwAbHKhQtfWVzgOaQAAGJWqdioTsgMQ8PyakE4CAADQOHO1U5OcHYAY9/vdVwY98isRAADMQ9ROHUITAI7yxQEAAECzTCBDBR+zkr+/fbumJQC05qevXwP/er97zAwAAAAtcr/K1cI5u5AdgKdA5u7qhY54MioAm/y5HjTFFRpFiNq5VOBiQsgOwNpe4O4Chl6I2gF4J2GHxrlUI4eonevsXVII2QEIkLbTu8UlkEMXYFpyduiCqzWSeSwqlcnZAQjb+6ZwswoA9OJ+v7t0gV44YUkmaucim4OUnB2AGL4vAIB+yeygR85cEphAhivI2QHItzmTjCsZ2me6doCZhdO6P/78/bKWAJt+/eW3wL+6bOMQUTtXWF9byNkBSLBO213J0D5RO8C0Ajm7kB2aEgjcXbkRT9TOFUTtAJQibadHnowKMKfNqF3IDs3aDNxduRHPXO2cTs4OAADAbOTs0J3NM9Sk7cQTtQMAPfF7LQDQKTk7tM95Sg5RO1cTkQBQljITuuOgBRjeeqiX30Ev1merizciido5l8EIAAAAABieqB0A6Iw/kAIA+qKkHWAGonYuJRwBACb0eDxqNwEAgFh+HiONqB0AAAAAALKI2gEAAAAAIIuoHQAArubR8QAAMBhROwAAAAAAZBG1AwDA6TwZFQAAxiZqBwAAAACALKJ2AAAAAADIImoHAAAAAIAsonYAAKjgfr/XbgIAAFDMj7UbAHP56evX2+3297dv7//3/ZV2vNq21mBroU0tn+MAAABAQaJ2ANgW+MEpQKoO7Hk8HirZAQBgVKJ2YNs6LkyLHYmnAhoAAACgU6J2ANi2+ZuHX0QAAACANY9FBQCAOswnAwAAwxC1AwAAAABAFhPIANAx07kAAAAALRC1Q0/WDyaNjxdzls3R43qPLrtOexefsLf43pNmN1+PbMPR9x9tc+Z6Sy27/oSfvn7tIm2/vp+BpjweD5PGAJDs119+u91uf/z5+/v/fX8F5uF0oEGidrjUIh3LCSXfXz8awsYvm6PT9W4uHt/mvcVPDUartLluP/co+diYsK8AACjiFT6uzRNHSmCBy5irHTrwsaB1L/Rc/9M6mAssm6OR9a5XHV5vuNkf2/z+hphl//727f2/vdcX/1q3zXv/dGU/9yitn9f/OkNfAQBwgV9/+e35X+2GjGm27p1te2GPqnboySKNvb0FbZu1sZvp3t6EIQXVWu+7o331/oaEZXPWm6N6m2v182upjuZqr9tXQLPu9/vj8ajdCgDGt55n4/V/lXsDFKGqHVoXThIjI7bwsudl3xevN6evLujn4mq1uXo/LxYJV/03ooW+AgCA2+32x5+/P/97vaIYGaAIVe0wrIozSzQbAjbbMAAAgIv98efvr5B94Nr2UbcLaJCqdujex6remH8aY8bn5M2JmYSktZi+5TbvHZMtt7k1+goGZroYANohhgYoSFU7NK2jeaj31Arx07ouprWtTY19cZuLHJM99nMt+goAgGuEC9vXk8zEx/Q5yx4VmAzn0EqPtnlvvZuvX7b5zxW9XlyvN/BP4Tc0sr3QGlE7cJYqIfviiZG3IX6uAAAAuK0iyyIJ5vs0MpvCoWpOAN1mAttjm287zR57Fv4zTgfIJGoHTrHO2d+T7rNT+HXgfht3Rg5tHpu+ghnc73ezygDQpnWV9OKVQC38ZmH4+4tnTBAfKNyOkdzmvaLva/LfcDH7GepuLzRL1A6U955xVwwK98J9s20AAAB89J7VviLU1/+IT3Lf49dFCtzs41g7avPmbrpdErgDCx6LCk3r/bGl7STa4SfHvvTYzxe3ucgx2WM/16KvAABadt9Ru12Fheu4w1OXxNSAN6LHNr/rtNkwElXt0L2P86IEirgHm8c8eXP+/vatu0Cz5Tbv7YiW29wafQVjezwe46UwAKOaecQ+NaXtMQLusc3AlVS1w7CaTc8bDxADzWu25TO3+aevX5//lWhUo3rcvwAAnRq+Pv0yCfPMVNdjm02SDk0RtUPrwvN1RAZt4WWvDOWvSQYT1vLxqa3PF5tqf6025xyTZdu8mII/ZpGONHVMAgAMTLAe49dfflv/d3TZU1tYVo9tBqozgQz05D0cX+Rrm3H5+9ue/+PiWSleK71dEgiut/eWuslVuiu8vz7+IlJxF2+uN35yfJOlxNBXMLb7/f54PGq3AmAiUvV4yXHz+rGc7Zdg99hmoB2idujAOlyLzzQXy65Duo/TuMf8U3hK7nVrz8sKP/bVbX+T1z8JJGTHZX3sqFptzjwmb431c7P0FQBAQWfH6+8/mg4T5a9z9vfEOSaFX4fXt+bz6x7bDLRA1A592CsMjwnacpZNtrnSa2LBQBF9ZHclL5svbWdVaXPmcVWkzYeq/vtV95gEAOhdqdR7wj8/es+aM/PlvYD+119+aza57rHNQF2iduhJTrJ2dNkiKV64fvxUTfXVBZ1fq81X9vN5H3LeGlvoZ6BNj8djmIJHgDZlDrPzBOvxlemlbNaMN67HNgPXE7UDAAAA40gL2efJ1veUzdN7nG6lxzb/8efvz2artYcWiNoBAACA7iUk7OL1nDJtJd4U1OPvHLD2Q+0GAAAA4zxAD+B69/s9fhR9vDm1Ve3LmYo9PmfvMZEv2OYrN39zXeEGvPZ7wrLxbTi0VI8HDLyoagcAAAC6dChhP7UlHVlHmR9z9ucih+Yrf01scvtnwXLLQWrBNq8/6v2V4nPfr9d1O55Zr9uZ2QaF6kxI1A4AdC98m+3WGgDGExmyuwx42ctPAzHoInVdfMLHTHb9hoSU/6hAk2LWfl6bT/2NIbCnju6mQ8sGtPybCpxH1A4ADO7jrbibcKp4PB4mjQFIIGQvIjIv3qyPjs+aA+XVzVY6l2rz3uectOE5e2ozUj/azpztVQLPMO6+ezjV4hro72/farUEgJH89PXrBWtxmcTZFldKDjmAsJiQvZGxdNFU6SEVSbHTLH42aGRsoXGq2gEAtolBAaARHYXsAEzrh9oNAADow/2fajeHATmuANZivnYfj4ecnZkdmqEeOI+oHQDo3uOfrlmp2B0AziZkh4+eYfpmpL54xul1bYJZmUAGABhN/C13qZT8/XPc8ANAPk81h0ivZ4q+z8me/CRbIIeqdgBgXo8tmZ+p2p14ciKAtY/foSrZYWGRpMvZoRZV7QAA/7C+e08OzV8LSgQAIMbHkP2ylkBfnnn6ehoZOTtcSdQOAPDB4sY+IXmXuQNAmBljIJ9gHeoStQNAHT99/br3T39/+3beskU8G/Ba16s916y9uvdb/aOxu8ydsPv97tgAJqSYHYABiNqB+voN6RZp48D63UdwtuTY/flm2QEABL5AfVEC0BFRO0CiV/r809evAmgSrA+bQK16wWU5T0LsrsgdgJkpZgdgJKJ2AIDyXunAocxdpjChx+OR/NxdgK4pZgdgMD/UbgBAr15lxUragYDHv8W8+X6/S10BmIGcHYDxqGoH6us3qu635UfNs6VwnmdwEJOkm1UGgIEJ2QEYlap2AIDrKHLnIzsdGJicHYCBqWpnZOuHBMZX5h5d9vX+vbclvCG5DZHvL7XsUYGHNx5a6ZVt3ltjzKrTjo3Fi++r3nsl/LGLxh89liIXXK/36Kr32rDYilL7eu8Dn69vvhhY+/XHJF07WuQugACgd3J2AMYmamdMe0HhZnxWcNmCAlln/Jsj27y5+MXbe0gj++ga643deyU+//24SCl7q044Bw+dEVV0dx5lWmzUkNt4jfinpwrcAeja3jedrzYAhmECGQb0sbA0kNl9LL+9IO/76evXjwXLi/cvXonf3vW/Hlo2zd/fvi3+O7R45vbmyGx55qojXzz0IXt9VWRLA4fxoXPw+gz31Yb4Y+n684jxRM4qY2qRIYmZgLEF5kMzAAIwElXtjOw97VrMPrFZV7uZDAamcznVuvEJi4S39/0NCctW12Ob0zy35e9v39aH6OLF8CfcLj+ei5yD62XLtvDoxwaaPc8xyXliZpVR3g5AR0waA/To119+q90EuqSqndGEJ1NOyKzXL16QUSbMsZ6wvUX66mI9tjlT/pYmT8ue6bxzsCkTHpNcIKbC3RNTAWifnP1FbAcwA1Xt8C+jzvAwW9I32/YmS6jmBi6mwn1m9/vdbgV6J2cHOuW3MZKpamc6H+eejvmnUzPKtILlo00Kl+ImtOQatebz4QIxx+QFa18cWuF52zs9j+hLZIX7NY0BgEiBydknydnXmym8g35NMnCRT1U7Q6kb1VWXtvkxmXWb00xPvrs51dHZ4fs9j+jFxwp35e0AtMNDUPf8+stvf/z5e+1WACF+FSOHqnbo3rpI/6evX5//1WrSqWbb3vP4E4GnghPfw9k+VgKawL1f4idgGHL2l81NluJByzbP0AmHL5KpaodBbBbhDjy7RfL2whkcclwppsLd/QAAVcjZYzyzPOXt0BQ/g1GEqB2G8p73vcfQo85cMdv2luWXibXJC/zpy+PxMJ8MAE2Rs68Fvq8F7tCIcMg+8whGAlE7Q/n727dnUiZpvUVPNj1MXx2dXBv2vM6I55ASc44Mcx7RF+XtY7P7gL7I2feEfx1XSAstM4JxlLnamc7Heb1j/qmdTC15mvJ2NuEQ07IPrPrc8QnDQqfnEYMJT+Bu9nYALiBnD9MP0CNnLglE7fAvM0dmaT880DV79l3k6R9+W9fnkR+xBvDxcamXtQSA2cjZY3x8tjnQDicsyUTtjCZcGBsZJIWX3Yzbiqw3R2Zhe6DZbaZvbbZqU+PHRps/MtXqqwRdn0dPi+ccVGwJmZS3984dHdAjOfshugXa5zwlh7naGdl7OL7Ij/bi8tfbnv9jvWD8etOWPWrd5uRVX9nsZAW3t4pandzFzr3tnIO3a/PfxeBwtNN66WoG9vFxqW4eAChFzp7g1Tl+AoemGLgoQtTOgNYhV0zOvrnsOixLW/bU3O3j9t72m73OMeP7KtmheTbWDcjZ3lpqHRsL8R2VuY9yNNJXhySfRzn9XHEf0b7w41Kl7QAUIWfPpKMAxmMCGcb097dve3XrHyOnzGUjXywr0LbkZgdery5ze6uodWzcdvqk2Y667Te4SmPidXcevby3sP3WEs9kMgOwm4BmydkBYE1ZE+daXIEJcRhMeAb/ulpuW7IhN4o0i+J91zMBgbhWv7VmvbPsI6BBcnYA2KSqHQBgZOHy9itbwkdSKqB9cnYA2CNqB0ikwvoMh6YgByKZTAaAIuTsABAgagegFc8wfTNSXzwf9bo2wSgej4fydgByyNkBIOzH2g0A6MNPX78+E95FECz2Lejvb98Wafvrlff3VGgZjOLxeOwFJfe7R/gAsEvODgAfqWoHiPXT169i37MtulSHQ3Fq2/tipwAtkLMDQAxRO8Bnm1OaiH1P8ve3b5t9q8OhlMBkMqZuB2BBzg4AkfylMOdaXJZJygAoYvEDmOuZNIFUXZdWtNgv9gVQ1+aXhaEJANZUtQMATMpkMgCEydkBIJ6oHQBgXtJ2APbI2QHgEFE7l9qc8BoAqCg8dfvFjWHNXgCqkLMDwFGidgCgM364PYMHpQLwYuQHgASids6l6gEAemEyGQBu+2O+mzsACBO1czWliADkWH+PuPMvSNreAoc0UJGcHQCSidoBAPgPaTsAC3J2AIghaud068syhe0ApPENcg0PSgWYk0ehAkAOUTsA0DH3/+eRtrdDnwMXkLMDQCZRO1dQ2A5APt8d15O2A0xCzg4A+UTtVCMxASDe5reGCOAC0vYqHNvAleTsAFDE3dcnl9m8gPv727frWwJAX+Ts1e0F6/bCeRZ93nJX+92FhZYPV9aM8ABQiqid6+xdw0nbAQjY+yso1zAXC8Sp9sUZGo/axetEau3QZUHODgAFidq5VOCuTOAOwEJgqjEXMFVI26+07u1GOlnIToJGjl7WTB0DAAWJ2rla+PZM4A7A7dPzPFy9VCRtv0yDUbuQnUzVj2EW5OxFGBthcoZN3onaqSDmWkTmDjChmCdmu3RpgQkHrtHOHDKCJAoyUDRCzp7MkAgEGEgnJ2qnDlcnACRw3dIOafsFGonaXbZRnIGiOjl7GuMhEMmIOi1ROzW5UgEgkiuWBknbz1Y9anepxnkMFBUZvRMYD4EExtUJidqpz1ULAGEuV5olrzlV3enaj16hff/+/aSW0JEvX74cer+xogol7Ue5YwWSGV1nI2qnFS5fAFhwldIFaft5Kkbt8RdmEnY2xWfuxoqLydmPcqMKZDLGTkXUTnNcygBMzsVJd6TtJ6kVtUdejAnZ+SgycDdWXEbOflRgPPz727crWwK076evX/f+yUg7D1E7AAC5pO0nqTJd+8eoXcjOIR8DdwPFNeTsR+0NhkJ2IGAvcDfeTuKH2g0AAKB7ezcP/litO3J2ivt4zBgoLiBnL0XODoQZJSanqh0AgDLUthd38Rwy4cRTyE6mcHm7geI8BucEm50mQQMibda2G3VnoKodAIAy1LZ3Tc7O2cJHkYHiYhKfo+TsQDwjxrRE7QAAFCNtH5KcnVIcS9czdUyCdadJzYCj1uOG6+EZiNoBAChJ2l7QZXFYYO/IRikrcEQZJYqTswPAlUTtAAAUJm0HqE7OXoqSdgAiidoBACgvkLYL3HOc0XtK2rmYwvYLyNkBqvND3YRE7QAAnCKQ6UjTuiBn5zyOrlMZYwGgClE7AABneTweJpPJdHYVqh0Bg9k7qZW0A8DZRO0AAJxL2t4jRcecbe8YMzKcQc4OABcQtQMAcDppO8AFTNEOABWJ2gEAuIK0vZQLekxJO/RIzg4AdYnaAQC4iLS9NXqeuvyoU5CcHQCqE7UDAHAdaXsCYRmzMSAcpccAoAWidgAALiVtByhob/D0Kx0AXEzUDgDA1aTtmU7tKHN6QEfk7ADQDlE7AAAVSNuBm592ziFnB4AqRO0AANQhbY8nOAPWPAqF6qaFAAAgAElEQVQVAJoiagcAoBppO0AaOTsAtEbUDgBATYG0XeAeoHNgZnJ2AGiQqB0AgMoC8ZBAGWBBzg4AbRK1AwBQn7T9IzkacDMkAkDDRO0AADTh8XiYuh0gYG8w9FMcALTgx9oNAACA/3g8HptZ0v1+lyUt6BOYipx9GD99/Xq73f7+9u39/76/Au3LPG5fi69dcyI4DTmJqB0AgLZI24f05cuX2k0o7Pv376U+qsHOKbh1FCFnB4D2idoBAGiOtB3gRc5OC5T9tq+jfbRuYaDOHTpirnYAAFoUmLd92qnbxWowITk7APRC1A4AQKMCQdK0afuCfoCxydkBoCMmkOmYOysA6I5w5Ki9mWRuJpMBRidnB4C+iNq7IVgHgAEsvtDFJTGk7e8CvQGMRM4OAN0RtbfOrRQADOz1RS86CXv2jwelApOQs7Owftzl4hmSkY/BXD95MrDg3mMqN18/7zmch9rc6bKL/Ru/cwvuo5ztXS9+2XNZM5sNxbkzaZeQHQBm48Lso8AF0jy9t+6E5G3f7M/v37+nfVrYly9fzvjYigp2VIOdc9JhsGlz8+c5o9fk7NUtdkELyd17FLuXrh4NUj8uG17k0NrTJLS5+rLhxfeWjenq8/ZR2vYmt3nP+vek+EUyV32qRSON5MMTtbdIyA4AM3N5FiZtF7U3olRHtdkzovZa5OwtaDlqf7eO3SNT0c28Pj5BvqZDctq8+c60vjq07HrxyGVjVhpe7+JzktPq+PXuvS25vP1o49O6+nqi9tmYQKYtQnYA4Hk94EJ8j6nbTdcOo5KzFzfkaPkeIC7CzZ++fl3Hi+9J3+tf9yaiaUSpNh/tq8xlN8Pi5PVesI8u6OeTFOlqOIOovSExFwGPvwwTADCC+88f7kAmSY3ThKduv80XS3VxtFQslr+yQBuSydkjDZmex9uMDgOzysQv22YumdDmcHF0uK9qLfv+tsC/nrePco6N64+rIl0NJxG1tyJ8uSBhB4DBvL7cA5l7F/lpRcrbgWHI2d9NHqaX1Wnm2GDoX9F52bF+huJE7U0ITTkqZAeAoT2/6/cCd5Fx2LRpuzlkpqIqf3jT5uzGsSuFq5JbLmwvLmcbP87wHtPP5NDVNE7UXt/udZWQHQCmEQjcx46M84XT9tsEWRXQtXlydsE6aQ5lpjnPbi3y3NeY1rb5k0bxbPrsn3D67WrG9kPtBsxOzg4AvOxdAIgnwh6PRyCTmqT3JtlMGMzYOfv9n2o353b79/fFR7Wbyb/89PWr2uSnUx+Rqp+hIFXtNcnZAYCFx1/f1LanmXYyGaBT4+XsF+fp/XYUMdbh73t5crPRcHc11J32863DrmYSovZq5OwAwKa9yWTkxR9NNZmM6dqhX6GHdXU1TJ09CvXVGxT0nvBKVG+FJrcJfGzxT4ZpmUCmLXJ2AIAcM08mM/bWwTB6z9nPmBPGRC7sOZr/5ky0UmSSlpbLwANOnUv9pBC/065meKL2OjYvSuTsAMDL5oWBLDVSOG3XjbTjy5cvtZvApTrN2QvG6yL1WgK55NmRaGueU5OnBbV7y07SdS3Q1TRO1F6BnB0AiCFtzxHObsboRvkU9CUwOXuDp3OpeF2qXteQuWTj5cwxv2qU+sBTNd7PtxO6GvKJ2psgZwcANrlIyPFxMpkxAvd3420RjCEw4LSWPufH67L1Zm2Gjwkl7XVDzI9rD88DE168yLLhxcMfEr+KsPx9FP8JpY6reEW6Gk7isagAAJ3xfNRDwo8P1Zk07vv377WbQK72J43Jr1sv1RLO8Pe3b6/M8fk/3l8p9Tmnzsf9/PCENr83b7H4xwbnLHvL6Ooq+6hIPycvnimnx6A4txZXW1/EqFYDAMLuPy/vHFzCJQhnSf126cblZfS2bPZJX9luzFznLW/Rx/a33PhSNjuh31NyoeWcPbN0vWBLWrPomRbmYPmYlsa/YVPkNgY+pHgvhdd16vZm9lVCL8VkxCfto+R+zmzzoVg8od9aOG2fFo0ce/DkpqodAIBJfCxvv7n/Acpp9ue95ITdCNm1vYrjQ4lkkQ/JWVf8inKamrmZe+XVyas+dR+dtMZrku6croaTqGq/lJJ2ACCNwvaCPsZM3fXtYotUtS+0vEWq2m+DVrU3WMwuYY/XYFU7Yzt7Hh4qUtU+G1XtAADMJVzefut/Avfe2w9da62YPS1hN4YAQIIfajdgakraAYBILhvKejwe4SDpfr9nPicQmFA7Ofv93w4t9fi3k1oFAGNT1Q4AwKSecdIAE7iv6/QVtsPFGgnZE34jNFYAQCmidgCALslSS4mZT+YmjeIEMRPN04XqObuEHQBaIGq/jr9BBgBo08fy9lvzgbvCdqiibsguYQeAppirHQCgD6ZrP1vMDMXmcAeePo4GJ4Xa9zfxS5mEHQAuoKq9GnfLAAANGqDC/UVhe+++f/9euwls+Jhxn3Hepf3IZwSALvz9TUAEgxC1AwDAUnzgfmsmzPo46TyQKeYUKzsgSNgBoCOi9qHcf/56e6uXf/7f23AV9KNuF1xptvNotu2lEQ68AcQE7re2i9wVtg/s41NVFcUXdHHILmEHgB6J2gFC7j9/bTAje+V3hzS4IZ2Sn8JsDgXut6ppl8L24X2M1/feLHZPdlnInnzyStgBoBGidsoQPLXPPjoqLc4GYFSRgfutvSJ3he0DOJSwhz9B5h4vMvvOPL8k7AAwDFE7wD90kbBv/l7i1xSACxwN3G+Xx2EK2weTH7JvfqDAPezskF28DgBDErUD/EsXITsALYgP3G9tTCyjsL1TxXP290+Wtq/FJ+AJJ1TOb2DOXwDogqgd4HZb5eyPv75J3tnj7waAp1f4dTRzv50cnG0WtkvbW3NejH6oAQL3pzNC9vy/L3HOAkBfRO0A/1AqRTWdC8A8DhW5P51d6i5tJ9Lk5e2HTtuPp0+RuZucpADQL1E7/7Gu4Q1EhHsFv5uvh6PGQ+v9uPg1sWZOm48uu05sIze54j5KbnPmenOWLXvkvK/9/vPXLtL26/dRmo9HV2Qb4rc38PcNH1fUwrmQY7O1H39Gyt9H+e/fc973UZUvI1pztMh9/WYRG1XMlrYfDcQDJ2bB5yI4/QFgAKL2oSxu7HNCyffXz8sLMte7ufjYbd5b/NRMp0qbezw2+pV8XLXQzwnz/FQ5j5LX29T4nDar0qGlam3v3qpj1lvriKJlaZn7+v2Z0ZvCduINn7anZeKLk6X4A4edjAAwGJfa11neOzVzB75ZxHe0Oi9hrozk9e5VHV5QUZjZVx9nA4/c3qPrXXxO5h8NXNDmKv0c+JyEY6nuH1sk/wRSpZ+P2oti3xuwXnWp82jxaWm/RiSfC5uPELhgrNscYwOrTttHm+stsr3vH5Lc1fELrl8MLJ5guUYXcp2oOK3E3qoDk970FbnGzHje5hZVn6t9rVZHbXZF/vhWPBkvwrjdhfXB8/e3Vu7fgY789HV1X+BbYHSi9uu0H7WHw6mjE4act97Ifz01fkroq7O3d2+9MQ1IWOTUNlfp50MfFePUQzF/jRX3UabMcHxzkfOOjfPOlPMOsPy+OnUfHd3eQw2uNT4fImrvXZWq2EDaLmqvqMGo/Vapr4pE7W0G6zcDdZ9E7UARovYJ/VC7AVQWDi8umJogZ73hZdMmOgjIafMF21tcrTZX7+dSHn99e/535UoTdN3PCeuq1eaOxrrI9eZ8SM56k6ejScjZA0vFvK39058qHm+KfOA96LXSvWWLtIGLfQ+q3brThY/2RhQ/07nYeset8zKAMDn7nMzVzge1koIeE4oe2wy06bwAvVmnDqHGZ9r0frt1UlDYWv5IjsgY/f1tCYXz7Uza3tHRKzqZxE9fv6ptByL5fW5aqtpnd0FdZFPrzZHc5ph5D1qLgVpu8161eMttHkk7/TzJ3ozp54IDadl5aeruo5htyTmer58kirE9/ql2c2hLcrl6I6H5YB4rtVvEKTb3rOwMiLE5Vvi+mISqdv6jVmpQfL2vx9ndf/560rYUecTo3nuaSm0ubnORg6HHfu6RfuY8+T8eJP8mGn6P45kLdFTJywXys/LnJxwqb2+nsL0FkhEW1LYDYX6Tm5yonX9lmu8pwzWZe6315uixzQAdKVKhf/azeaEUqfrkXnH2Xg5eMO/+/v17m09hbV/gPJXCD2/v2dHPHE3gDiwEQnZfGfMQtfMv6xD5dkmOXGu9OZLb3ObmhGkze/Qzxa1z9vfDLDKFT8vZHc9cQ7bOps3Z1YvXlR9K2xW2x9g8o4Upg9lL22//zNTE7jCzj2XsvhqmImrnH/ZCjbP/ar7WenP02GaAZhUvRTcU04LGs/VAhEQtpwbcA9S2r9OK1o7hdXskLL2LGSrNFwHs8S0wG49FZdfmkye7WO/1RfGRbe7oGbAvPT4vt8d+7pF+5iRnP6qh7IKwcP+nk9ayfipj5F2cZzkSH+W3GcpvBtkJLm7zNSMD5zFgAmmMHhMStc/u/vPX53+TrDdHcpt7rKxsuc17O6LlNo9EP18sMOac8bNikZ+7akmoi8/puq77irJOStAOpYQfVy1bpzunHrF143jJe48MocBRxo05idrhUjGpWWu0ObkN3f2edFQL/TwqP2ksfDyi8uefcTxzVMGYLCfmC7ehSg0vjetoEva0H5YKrv2aCF7s3gvDKRDJcDEzUTv/UipHOPo5xecJuWD2mMzC9kCzr0lzItdSq83hQtHw6trp58X0/Wev7mLt9PMkmhrrquzWQys92iE5x3POeEWniiRiBZO7jyF78icztsi0vYU5ZOqm7ZtOzd8VvLfvmj99ALpjcODJY1Fn9/jr2ysLeA9ujgYE6895f2UdfJRa72tdt/NDjVJtvm110QWO7qOFKm0O9HP85PgXt3lC+vkk4XP24vXeLgyOE1Za8IeH5K6u0ldcJjPwOumOK5yzn7FGqOLRw/N7T/pJ4P0TnNcNslMAWBC1sxEoFJkF+2PQUHC9l03bndPmdf6SkB2XFbOPbjXa/LGfA+ttsJ+HNFU/H5pXpOyGL86FWmPdYm6Wk3Lk8MZGrjdyWNt8Me14/riP6Fpm6XrBliwI2eF+v7d/tC9amJm8i90BoH2idm63YBVefI6z9yGHUtFD691c/OyAL7/Nmf2cI2Ef3Sq1Oa2p72+r2M+LBowUOi9U7+cZZJ4LZdc75ErfV5R2PAeaLXnvVHIcdk34JWeniO/fv7cwP0ykLgrbPyqYvL+WddYDQFM6qAUYxuJaShQFQI9m+BmpZctM34VcOWmx18W7oHjOvvmBHT028xY3n3hfW3SZdrpusyXrQ3p9uI40Bub/ljBSbwBAp1S1AwAwr4R4q1aetddU+RoM4P1ETovdlboDQHU/1G4AANCWQ9PTQ6fu/xa/yOPfzmtVgJwdYurcx/B4k7B4wvgGABQhagcA/uMZpm9G6ovno17XJiiqr4T9Sc5OLR3N5z6qnNhd5g4AFzOBDADwH6+ng77PyV7rUaVQ0NGwqf0Uu/0W0ri+noz6tH4+6v0+0ePHkieZMbcMAFxDVTsA8A+LJF3OTu8OFXW2UMO+sNn4ploIXC9tsFLnDgCnUtUOACw98/T1NDJydjpydJaY81qSQ84OC+vC9smllbqrcweAM4jaAYBtgnV6NNJEMXJ2iDHVHDJhr36QuQNAFaJ2AABGMEYZ+4u6XSBZcube/tgIAC0TtQMA0LHBEvaw3tsPRUz+cNRDjmbuitwBIIeoHQCA/ow0UcyaqWO4wJcvXz6+5/v37xe0hAvI3AHgAqJ2AAC6MXbC/iRnh488HDVZWuZuCAKAGKJ2AAA6MMlEMXL2sSkkP485ZI46lLkrcgeAGKJ2AADaNUMZO0BFz2FTkTsA5BO1AwDQnDkTdiXtEM8cMmUlFLkbnQBgQdTOaO4/f33+j8df3+q2BIB3xmc+SkjNRgp65Owz+P79e8wcMqQxh0wR8UXuZpUBgAVROwD8yysLfukoFJ4tyJ5te8eWVpc6Q7Izwzay9uXLlwuma5f4E6bIHQASiNppnTAF8jmPPlqH7O+vt99ve+2HZiVP+zBqlGMeDBrU/gNazSFzgaNF7qOO0gAQ44faDQCAyhY59eOvb4tsXZANRdzfHF328W9nNKxNU20sF1PSzlHxg3DyOA8AAxC1AzC19xj9PWRfB+5Agpx4/TZNwi6TmkpMqbgoPJmz6WzxY7LAHYAJidoB4HbbmSXm9WLLhe2LtrXc1IKev4X4OaRB939K+5BJEvYnT0PlYpE5fvuzx1CRwB0ANpmrHQAKMzk+UymVoQiXX3QFt6sejjoA07VXFP/oVNO4s8f52wLnJhQkap/LutQxHAOt06L1jMblWrdcaczrMQ1Ia/PRvipls7UfY7uPeyqwbNr7W1i2liu3N/kcLHge1Ro3runnMzLx97Xff/565SH9+OvbNSXt+WPO5lKBRQLb9XFFLRyTwyh+P+zWTsQwp+/fv8eUlp+Rts8wNc39fje2XCzy0amvN9hB3HwDtsS5CQWJ2mcRzt0SApHXiy0HDclt3lvwdmawElhpkY86+uaY7a21bC2Zbc48rqqcg1XafGU/t3mkpXnflosH54SRqtZ3SpVjskdn3/26kQvQObwrm7bH5+wK6kkQGbi/3mO4m5aQvVnOTchnrvYpfCzf+5iSLB4beGjZo17T7y4m4V28/nGK3uQ2h7vrpIrR9UrTspv7z18DGx6z3vAbwv8U31c5y9aS2ebM4+ro8VzkPLq4zXv/1P75W1fdjTo65ryW2ltkb3MOjf8F17v+p4Tv0NbcIxRf6eOfin9+p2QNM4vPskvVoQ9cz25UaUr8UG8a9wnZ6V2wjyCHv627zmK0uqwILhyChCdPiFx2b/Eijk7vkNnmwOrO2978fs6fDOHosRF+Q5v9nKnK9pY6BxOmSanV5taOq7QZZs6YlyZyXdes+tQxJ/ID47f0vCP2yh292aTGucj8aH0re32nbd5O91XRHJMgN7tFh+LvnK24bEVpNpt36HRY3moZf5oRmdnZZTMQ4HbHiQkJTCAzkc0g4DWxb/jP58PLtimhzeHQ5ILtLbLehMQn4djI6avq/Zyg+vZe3Fe12ly9n0u5/lei1xojR/UzVp25SONjbM536CTcjB3VQs5OX9ImkzlazN7szxKHmK69HaZx5+njMeBSqooGb72hd6L2wRk3CTj1aibnw3u8zOqxzXxUpWb5EIP8qXRvgCgkk8o+niKfj/ryevPHQHzg6WLokWnc2dPsZfYkXv2//UAjP1vCceZqn0VMxfrkmUJMpnbGdUDZLO+yK5Wcw6bHQy65zbWOqxwtt3lvnu4z2tzR8VlRa0fvSWb+Dn3sqN2uAenVaaVVkX/55MrGtMAZ1AXTuE9rc28mP32HM+xOG+tMhINUtcNSTFwy3lwBOSFRzq8F7VcNr2VO3h1+T1P9cHGbixwMpdrc7DMDYpjVhHjCqSrctbJwtLb9vGbUbgKzODSrjK+q3u3l7Ne3hLDFk5+ANKJ2mF3yV+n6mzg+Ks1ZtpYe20yaXnJ2RyA3AcQo7Eeqp+3j5ezmPWifadyn5dq1ZevnGBlO4RBROyxN9cW/ztnfNz/+UYHLL+O4BDBn2VqS29zm5oTN2eZecvZ3ak+gI0ra2VMxbR8vZ6cjpnEHYCSidphXwUhxL6CPmb8iZ9laemwzR9mVwDXERrxUSduHydkfj4efsvolcB/Vep+6xm6fwnbI4bGosDRhfWjZy52c59v0+GycyDb3eFxd3OYij5fMbHPjf1TxLnJLezzwAGb2/fv3y7LvK9cFMY4+N9WPKwC0RlX7LALFth1FS6da/3J78Xr7Koj2KNRItY6rHC23eW9HFGlzj0fmbb+1ze7EHvkOpZSN4j41Ymw5u7xdwk7jFLmPyvVSL1q+JYTGqWofnG+yNIEvlXm+b3rZ0vvPX5//1W7IZz0eV/O0uWxg2tFhSYDvUKCi7/92xseW/cxmKXnuXWSF++3fRe5ntwcAPhK1T2Qz9OmoHO+C0OrjE0Ff5ecnNSC8j64Uv9KKc30sPuGajkpYS/Xjar2uj2q1OTyHTHh1OW0unrOHW1JwFTFP4pX4F9H7dygtUNJOmlLh+FQhOyMxqwwAHTGBzPje//Dn+T/6+lOgcPtPTTcu667NbbxdGJAlrHTd5lt0X+UsW0vBNlc5DTPPoyptDvRz/OT4MW2WlhLQ+3cozGCS+HixmTHTy0zSM+88GXVsZpUBoH2i9ikscoF1RtBXwHRqxrGOmxMyvrT17q30vFgnfGB8XO/Gc8mjD62cZWvJ3N5bjeMq4ONB1ci5cGi9mW2O/KHo43s4w6F5gYo/6nmk71BqUdJOWRPG6PB0NHC/GW8BuJAJZGbx+OvbZhaw93pTAo0/b42HXi+10vXnn7139lYaud7AOz9+SM6y7+/c/N8nyWxzlePqfS0J51E750Lg9fXbDr1e3NmHZXwZvjlkSun6OxRgZorcRxU/q8zNTO4AXOjuB97LLL7d3ZwTzwQXADwt/1bDhVwPWi5p34yfVExzsc35cNJOk5ZPN85zKEl3SFxJDNIv15yQxgQyAAAAjMB07XN6hYAmlgGgLhPIQBMOTUMMAPRCjS3AZeJnlbn9e2IZv80AUJCoHep7humbkfri+ajXtQkAAKBDh2Zyv5nMHYByTCAD9T3++rZI21+vvL+nQssAAOBMmxO1F3S/ez7ZvJ67PjJGN7EMAPlUtUMTFkm6nB0ABtDp7DFnR58AV0orclfnDkACVe3Qimeevp5GRs4OAACRPBmVPYeenrp4Wxc/lAJQnagd2iJYB4AxdFrSDi1wsnCqQxPLPJleBoAYJpABAIBJ7WVG5pDhGo40Kjo6scyT6WUACFDVDgAAp1MICbV4MiphRyeWWb/ZAQbAk6gdAAAKU/AI0J20zP0mdgfg30TtAAAwr71nSH758uX79+/Xt4d57M0ek59UejIqmd4PQrF7y+4/f729PfDs+X9v4z4CbbbthR6J2gEAoCQPRIWPzNJOL5JL3W9i97a9cuqXywLriqsGziZqBwAANihsB3jJydzXS0neK1on3e+vn5p6V1w1cA1ROwAd27tabeQ6da95txItPPsvRv2BKpTSfp4SmG1D2s4ZAiXtJ50vnoxKQTnTy+wt6Pi8zOL6/Hll+/7i/eevl13uPv76VmvVwElE7cxOeNS+2fbRbNubIxBkxy+rn4GyxpshWtpOWaaOYSRFYvfNZYXvZ3i/fXi/C1gH7qeuPbBqaTv0TtQOQJf2LpRv518lx1tfKLfTNoB3HiPJNcI5e8Fs0SHN9UrF7nufIHwvaDPOftWYX593L8rbgX79ULsBAJBlfR2sEgSopetkJNBUZcgUcVnODtU93pT6zPuWUh/O2fxNLUxCVTsA/XGpCnCxZ0hqJhmS+cGGaS3S9rL5+N6n+e1qk3sH4Gyidmbnu7Z9s+2j2ba3Fv0MsPZxzg3ztpPmY84uFmQepybv8R87xkm3uKRv+Qq/yOw0HW0vTEvUDgAABXQ9e8yLtJ2yYorZrzlT7vd7j6ckw7smeV9LWJEzCOAjUTtzCTxp5NAPwuvPueb35MV6Fw8r32vD+g1H25///j2HHhr5sZM/bmnkbkrr5zQtbG+Oy86FvY7afL3scRX+nIrnPsBJYtL2m8lkiNBOzg69WJ8R7UzI3k5LTnL2HJV7he3vl/qu86F3onY4Jpz3nfe9uLnetGeUH1qq1vbm2+uxhB8JungWfNr2Fl/jrYdjI1ng8Lj+3L+gh/2BKmTqOkD8mLbf3lJUmTsL8dOyn3qaxBzG0L7N08SxXdCVRS3vafvmv563auAaonbmcqjWdW2zoPX9xZPyr3CRdc7nhD/k4/v3tjen5j1zH60Xid9Hpfr5kIrbmyP52Ei2+LSjJSf5/Rze5OvP/S5+AYKpjJd6xMeU77mq2H1aCU897frnKKhr7/QZ78voPK2F3XJ2GIOoHVK8fwsuAtniidveX5MlB8HrD4lfb2AKmkiX/XHc0X1UvJ8vVv2YzD82WraZ7Hd37gMclVAUnJC3Mic5O5zh45kliw87+49HA+U7A/9xMEzlh9oNgG6Eq2ivn8whbb0J7QyvN/nPAi7L2Y+usdb+zdHmMTlMFjzGuQ9cbJgYcZgNoSm1jishIzyOqN3Ycz3++vb+3+v18+5i3m8rXiu9bO3ANVS1QzGDJV/XR+H0wu4DWBg7v3umLWNvI5cZPryDkeSfsB19dyymaix+y/NxGlUhO4xBVTvEqlKrW/YZ6HUT0rOf556j5bZRV8yxceos7Y5JoBESUvJdfxQ5boHW7F3ej/fHwTAnVe2QYrYULP/L3uXCqObZszFbevbcjkCb1iV7o6Z7yttJM+oZAQCwIGqHA9ZPIxw+cy8SpF42RTtXmidkB+DdKzaVufORkB1oyvD370B1onY4bB243wb9zl5nqQkPbJGzD6nIsdEXRy8QaZ5s8X1Lxe68zHMKAAAsiNoh0V6wOMwMEsUj8jG6hZufTwDeiJifpKv04n6/O1yBUgqW3A1ZvQcT8lhUKODx17eBvw5zNm3I6mZeBj7s1xzMAAAwhs1r+6MX/JtTy+4JP/XUvQYMQ9QOse4/f33+d+VKu34KeUe1z133M6eqdeg6JqFTqmWhNc5K4GXxt+mvK+3Fnf7ZtwDv2cLFqwbOJmoHUnyM/1wuTGvgaDiwae1vdZUfC2F4Zo8BgL4sbk7XV8jn3b2uP/myVQOXEbXDYbWyqnb+0OzQSru7XGinn3s0ZEd9fOLrqyDlpPXmH5OH/rgVAAAGtjcB7NGJYd/fHLlgqVUDzfJYVIj1+Ovb+9+XrV+8bL3P7+DL8rKElfb4RJfq/dyjOTvqtdVXbvKcXQ3dMU8FAPSiyL1q2od0dJsMHCVqZy6H5n9Yf/+tw7WYpfIt1ruYm+Wk3C2w0vj1xj8cJmaRs3u7Sj9X3N5kRY6Ni+Wf+zTPtmEAACAASURBVLdgbfjerik75jTYsTAzs8dAp+73ux/GAIAziNrhmEBV6akx6OZ6zw5eq6y0rgk3Oc2cHbX3K0Ktcz8+eX9v+fC7CQAAAKrwe/51FqVPwg5KkaBdQz8DLVj+7uJCrp7lpZ19AQ1zwkIVYpB+ueaENB6LCgAAx5g9BgAAWBC1QwcOTfdMMv0MAAAAQBpRezWSOyI9D5XNA2bx3M7r2jQi/QxAMn9VDQAAmKv9Ous/NJbZEWn9RMTZHkd5Df0MNG79c6ALuSo2LursCGib0xaqMFd7v8zVDmlUtV/HwESyxRWJ/Pck+hkAAACANKJ26MPjr2+bUa/8tyz9DHTEr/iNsCOgfc5TAOACP9ZuAHCAwPca+hmAPetpKAAAAG6q2uvyZFQAIJLLBgAAgJaJ2gEAIJFZKQAAgCdR+6XWN2Mq1ACABBLeKsweA8NwOgMAxYnaAQBa57d5AACAxona63PzDAAEuFRolr8tgI44YQGAs4nar+YKDwDI5HKiCtNNAAAAAaL2JqhWAwA2uUgAAMbgqqYX9hQkE7VXsFmJZiADABY2Lw+UtDfCjgAAAN6J2uuQtgMAdMTsMTAe5zWcbR19yD3at95Hygsgnqi9Lb51AIAnJe0AAFxJKgWZ7m7YKtoro3j89e3ilgAA7di7yXHZVtHiss2+gE45l+Fim7mH0KNN6jwgn6i9Mmk7APBOzt4m8RyMwbkM15N7dMElKBRhApnK9sYsf7MDABNyk9MmEzoDQDK5R/tcgkIpqtqbELh/8zMvAMwgcLfpaq06ZbAwDKczVBHzo7X043rhXzuMkJBA1N6K8BePrxwAGJWbnPbJ5mAY69suZzRcw5+I9cXYCGlE7Q3xMy8AzCPmj6Zdp7VAMAcjcUZDXQL3LhgYIZmovS2+dQCAJxdpjRDMwWD8nQrUJfdonFERcvxYuwH8w3NE88UDADNzh9MyewcAcsg9muUiB/KJ2lvkiwcApuUmpymuxwDgDK8LHl+11bn4hIJE7e0SuAPAPNzkAAATcgkEjETU3jq/9ALA2NxhdsTOgvHc7x5gBgCUIWrvxuL6T/IOAD0S6HTE5RYAABBP1N4rN+oAAABHPR4PP6QBAGf4oXYDAACgAwodAACAAFE7AAAsKXoFAAAOEbUDAAAwLz+tAQBFiNoBAOADs8cAAABhonYAAPgHJa4AAMBRonYAAAAm4u9UAIAziNoBAAAAACCLqB0AAEIUwAIAAB+J2gEA4D9M1A4TcuIDAPlE7QAAAAAAkEXUDgAAu8weA0NyagMAxYnaAQDgX0wiAQAApBG1AwAAAABAFlE7AABsM8UEAAAQSdQOAAC3m9ljYG5GAAAgk6gdAAAAAACyiNoBAGCD2WNgbM5xAKAsUTsAAJg7AgAAyCJqBwAAAACALKJ2AABYMrMETMhftwAAOX6s3QAAAACYlHwfAKo4o7ZG1A4AwOxEXTCnx+NR5fQ35gBAde9fx6Vid1E7AAD8g9ljgJMI2QGgQc8v6Py7AFE7AAAAnEvIDgCNyw/cPRYVAICpyb+Al5MGBOMMAPTifr8nf3GragcAAICzxNyu/8///t8FLQEAbrfbf//X//v4nvv9nlDenrIMAAAMY5GCuTyG2Zw6CIRzdgk7ANQVjt2PXhWI2gEAmNc6BXN5DLM5L2rfy9kl7ADQlEDgfujCQNRegHn3AIAGucyLIWoHTora5ewA0Je9wD3+2kDUnki8DgD0xVXfJrPHAGf85CZnB4AeZabtovZjJOwAQO9c/r0oaQduJwwFm7eNQnYA6MJm2h55bfBD6cYM636/y9kBgAG4qgF45zc2AOBl89fxyBsoUftnbkcBgPG4wlkTtwH5lLQDQO+Sv7h/LNuOwRy9/3T9BAC0YG+GwbXn1c6cEbNfGoBruE8EgAHc759nYjdX+7bIWy/XTABA+2KS9wmvCU3UDryUGhDWn+OeEQA6tb6NErWn+Jizu1oCAHr0MXOf6spwcck31bYDC6J2AGBtcQP18fLABDJL4ZzddRIA0K/nlUwgcI/5o8gxmD0GePd4PAwLAEDYx9slUfs/BK6uhOwAwBjCgfs8aTvA2dxFAsBUfqjdgIbs5ez/87//5woJABhM4PJmwtJOvy4ACwkj4YSDJwCM7WgmLGr/l0DOfnFLAACuEagnGDswGnvrAACAKkTtt5ucHQCY2JxpO8CCv24BADKJ2uXsAMDsJk/b5WvAGdxRAsBsZo/a5ewAALeZLn4m+f0AyGe4AAAOmT1q3zTPrSYAwMvmJZCkCQAAIMbUUfvmraOcHQCY1oRpu9ljgBcDAgCQY96oXc4OALA29uXQ2D8bAAAAFc0bta+NfWMJAJBMQg3MyegHAMSbNGp3wQQAsGeeaWRMFgEAAJQyadS+pqQdAOBlyEujIX8tAMryCxwAkGzGqH19lzXkzSQAAAAAANf4sXYDAABo0f/87//993/9v/dX7vf7SPWeI20LcJ4eh77n6P0qKXsN5orMprX4Qn/X7FHhuG1fj8fVqXo5aPd2XOPN7sV0UbuSdgCACZk9BuAyveRN0LLZzqPZtreWwA8kFDFd1A4AQKThC9sB1h6Phx/nGMw6uxS3jerKwNpx1Z33HbTYfeF954eQeDPO1Q4AwOT8YADEk7wDMJJ1Yi5DL2X2qN2RBAAQMMbFkpgMAICZqUy/xlwTyLjLAgAAIMwcMpnkOJBvtvNotu1lVLNXtQMAcEh38dO6wWaPAQAAipurqn3BL2YAAAB8NNVDodcPxwvfOwcephe/4N47Y96TYDGRwmIT4ld0tK9yli3V5hw521t87ZGr7mUfFTmPPr5+9KPO27/J2xte9tBHFTw2LjsRDrU559ioflytp7upO/7EmzpqBwDgo//53/+Lv6Vp3zxhGTCtRfqQEyy+v1481Dj0/XJSpLLZgMjtTV42s59z2pyj1noDqz6vnzvdR8kuPvdbUPb8/Xg05ut0H5U6Fzq6GZnol/nb6s+Hmz0QAQCasri67egC0uwxQLLFAPJx9Oj9fnOzYDCzjjitdDHtcw6J2djwetc9E9lXHxfcWza/zZufllYbnnxsxNsrnj27nzdXHbneuvsoeamcvspfe5FlNz8n8FEFz9/1i4HFkxXZR9fvnZxxYy9bf/+QWn9MEL4emKiqvbt5RTuy+D3KQ42BkRjTAIDbTHPIvF/zLIKkUys3Fx9+WQ3ja6WBaQoWNq8PY/pqMwSMX2+RZRMkb28p1fs5cr2by16zj5KV6qtGHMrZixxX62WL63QflR031h/SLI9Fhbn893/9v+d/tRsCXKrWuW/MoTWTxGQAR4VrC07NNTbj9Zi8rPja1wJTH+wtmxAbrV8MXzulzXySrOKxkb/ehH4+b/+uV9GUnGOyEck5+8elYt5W/Vxoah+VHTcaz9YXJqpqB7hArQpoldfx9FU8fUXv/FEjkOPxeCyGkXkK29fOzrsDcwWct949gfbkODUIO6nNPTq7H5KPyTb30ZDX+YNt1GCbMzxV7QAAADC16nWR7w1o+cf+mLZ9bHbMso1kskW2t5a0fm5tFxAp5ljNOZ5bHpdaU3bc6K7DVbUDADAgD0QF8q0L22cgUYoRk8OeOn35xWbb3qfZzoV+f1042vJ+j+e+9lG//ZxD1A5zGWwIAyLVOveNOQAMZuA5ZNZP9rs4Z1xMr1H3KuLVmI5ioB7b3KbzzoVm91FfAe5CO+PGqbreR1MRtQMdmK2aAIDiRo3GAMpah4z/v707O3IcRxsFqpyY13ahfSj/jSgfyoUyIO+D4s+r4QJiX8hzoh86VAIBYiP5CQm+3I0feVpVPO18X08aC4FddA7/dSp5cfblmm/RNlqunqsQame88JQR+H7SFlqbD/fTcWCCLkkbKF58kn0ZIo8TmHAzypl3qLzzPTtCnx//k8p8VlGHn1esq1H5VtRn7B9mFPg8aTzGJznMbsKxX1hXJXNO+DjxyUf1Z2bzwA0fgEae+XLUsyBO07vxfUxz4EV8xYjqimWeX92xMGEb3WZJ+NKFD7tNGz2EUDsVbIZ6SUDk8/NGM8g+07NPYqJI8WkPv3yZ5My0P1puHJazafsWGtInB+Z7lvWNz7fQWXWVxK9vOfZLZPfJdfsVAMzscG1vH7NtsvFp5rK18LTzPTRwLHSwYvtmt8Wi/Xm5Mi9az4WE2hnmcOFhtxUTm7wCH9ZKu18Gm7dq4/I4m38NJw/IC2kFvtOzffPk9cnw30Bk5BvZN8rzvcy6URvVGgvxqtTV639LHl9Xa439wroqmXMOv59dz0lpuR8vRAXqesLC9uGLbTcFmDaaGfnMGBZzSzPJfUuV8x0lr55nawLOZKz1LunPv2fdZ39CS88b5YTaGe9zktrcVzWawt7H/H301p3LGSEv7eE1ILAdROQp7P+/j5hL2uENSl77flZsn5Pt2Ser942MrB9yvlWk1tXNxn5rVfpk/2sKAFBofw8wNqoVeZMWjuQerpVpd7vb4UY69XxHmTbMN2epwqYtc0acfX+EJfrzpWnb6O029RzvP6MLwEOFI6dNB1tJjuWlDR8hb9lpZ9lx9stUAb///H3/l5ow3sA+GZNv9cvn0863inazxxJjv7WSPjm8PwPwQHd9J0T/27DLHPusPjl0eWtxmPznR4LUfCMXGGWUuUSV8x2lXT1n5Pujz91plT+/qFKSplIrs6Q/hx/fhlTXhDP229LzRiGr2pnUzSIjNzudt1ueVECj81WNN3bL+4ZFParjYfcYoIX9HjI387kQ+DMQ2fN+ZnO93hSp3dX8nUv2ySYl39dzXr6FZS4xMOt4JfVcayyMqqjwuQfG0c+XX9M/yFT8q/fsZhpSXQPbKLtfnSW5MaF2xhj794BLm2RJ++UVOvydCWfYR/XJRdtoXTH1nLQvyv0U9slHjV8A5nGzHdv3l9r9lffwOhu4abw8wuWfpnWOYW1yP/vyPtS1SR5/S5OU70Z82pI2epWd7ygl9Zw9Fva6tVFSGTaHCrdsUsEC/3Q29jPSxhwkkLDp+G2hpI3aiaz8teaNKoTaGa/iD5JLWDd8mfcz7+V3Jmz3UX1yzj/+atdG644F2invk0+7prBhSTvQzuHC9vtF219XOydUlLQMvNFN6eEpx+8QmFdXZ/WctDNh/6BV9vmOUlLPhWNhVBsFcg8UYGxpR6k7fluvNJ+hjfIG1HLzRjmhdoYJ/MB11yG3dGDxs/B3baCBfXLpvpHhaedLBw+8pgDQ3+23kXlLvXRmX2qTIstNtT6F6mlTk9eqw563VZERz/LjNEqbmrx63dYayJd/3pFnYLu0K0PrAZLRRhlfq36QvHzXfYgTameww5/FbhkfCf/0N3nkMS/Ovmjz9e+TA/vGkDZaeizQWpXnmSdcUzhkSTswxM0WtgMA2YTamcJZoG3O3UUy3GZJ+NKFT9KtT96mb0R62vkyxO2vKQAM9IRtZACAPP8ZXQDY+v3n741DISueWvYq49ssT+7TJydZYN7NimOBDqr3yXtfUwAY4jCq/oSNZQCAMKF2xvj17z/v/0YXhAsZa5AXjWo9qk8u2kbrCvQre5u8FZ7+o8YvZ+weAwAAjCXUDuNNGx4q3+sjJsIYeZxnxtE6nHKtNqriZk388Oh5tqn6JAAc8mMeALAn1M5gGXGTn+jVYdrlAjFLFLjk3eWBZoo8981Wy0klyVAri/Lj5B0hMlXdNqriMq/qY3/42S20pL1DXVXpk0vMqLRgSTvQn3kGANjwWlTG+P3n709A5DPYlBoleX8/I+EoPwV+TR8SqhgEXKKZavXJ/XE+PwlUZmHfyM73LElrhedbUuDCuqqY1/x61tVGUnXVGr8AkOQdbbdL+4rmX+iwt2KZn0YbAULtDLMPguxjImcXqk3azT4nE8ZWAgXe/2tFSfswnNV2/Frj/Yfhsw4ftr+SPhkQrsB2fSOy4Tq3UeH5Nhr7TSeN8Cm/2tRzlbGfeuTyfEv6ZKPxyxIsaQfG+v7+Fm0HAF5C7YwVWNB6GRA5DKvNHEY5PNmZC1zFWfQz6cS7raJ9lfXJy4MEfjrafz/jTFPz/fnX8jZKUni+VcZ+Xl1l65xdXf0Ln90nq4xfAMhgeTsA8Hq9vp6z6md/3+PBGwAgxuY3jHluIA8DW/MUD3iUzYzkeRMAbiDpUchrUQEAuA9xdgAAYAihdgAAlmSvBgAAYB5C7QAA3IQl7QAAwCheiwoAwHosaQcAeO02kv7kpRGUe3ewn77009/0rkNC7QAA3IEl7QDQgUAbwBmhdgAAFmNJOwC1CBxzpqRv9OxX+ywC69wZzpxzb/ZqBwBgeZa0AwAAYwm1AwCwEkvaAQCACdlABgCAtVnSDgDd2PUC4IxV7QAALMOSdgAAYE5WtQMAsDBL2gHmsX8ZY58V0En5nr0x8vDzcPnzznf/UsTwcTbf//zy2ScVBV6weZnX5Zl26B7d+sbwflVuxXxT016+kjTwhez+nN034od200mAJELtAACswZJ2gGmFY0ntQj83yDcQy7785tknE8baDk/z17//LNFG3dygP3fLd2z79unPv//8jZ8ffpJULMDhMeccO/MQagcAYAGHcXZL2gFmsF/aufmkUUT1cEHr54eH+Z4tRs5eSJt9vpfHOXT4tYyQXKTA2t54myW3l21ULq+NSvrGPP0q1ST57rMO5DuqzPvc+8w58eVhOKH2rqoPJ1JN2wTTFgxgEuZJ9sTZAWZwuHFBYGuUFj5vDzZ7qlSPuFU/333hL7+8ie4dfjibpduog1FlnqGuUvvGimUuyeh95JjDelCagdeizuvXv/+8/xtdEACAwWwdAzC/wyhPu/hX+Gf4DiGnKuebVM5RZ1pu6TbqbFSZVxy/c9YVD2dVO+NZqAgQZp68N+2bx5J2gElMe/3qs+UFE5q2TwaMKvNydfXAMRhe2O5RYjZWtQMAMDVL2gHYG7s2ORDVii+Y0Bic+f3n7/u/wBcu/+mBcXmGs6odAIB5eRsqwEJGBbas6+TMisHW4eOoW0YGbKrNwnY1OSGh9nkZJwAAe+LsABMauLT8M2uBJ34Iss+fL5Emf+kxn4TaAQCYlK1jAJawjwEdrrtsZB9wf4m5P97YPplnVJlXrCuYllB7K5vJ6PLqHpi8ku4MwlNk4PuHdyfxWSfle3ayh59Hlj/y+5dHiExekm92JReKL3OgV5x9UiXfQBkyjnNYz5E33IXj6Cz3SxltFF+2+GIkmaR9A98/a/T354cfxhwtspzZCufJ+BNpN8eOmieXGEdTXQdnZusYgCW0vuGMdBYcPHyRIPc2SZ9MMqrMK9bVM+1fjuoHxTkJtdd3+Jzc4eoefm7Pi/XHpC3MN1t5vtktVbeu+tz55ZV5n+rsk/i4ZEy+kcdJ+nL8EQrLnNfKqf0548/HqnezimM/9Vxq9avyTDvkO9CQ60LJPLniOCrxwD75EmcHmNs8V5/DlWQ80Dx9Mt6oMvfMdx81htv4z+gC3M1+IV7ki49/3q18+ZLlmHxf50s1LwVWR17mG5n27Ez3lZC9sPTyfAO/3IbThk85qa4i+0a57DIffj/wYd18f75Wsog+dTQVljmvX1Ucv5dHrqVWmVPbd3/k7Hzzvl+Yb7zyebJQrbmuJG3SPJl3PYrRYhxVvw5WPN952DoGgLD3neSoS17MLY0YImS7HOAPHIOfD0d3PccbsKq9prOAUc9f1AP5Xv5a+POvkaGNw/PtFj7eZPcqON/4tIdzWUzagX0ju8yf39ys/Tz8sG6+h8XY//9ZpuF8m5a5W9qkhQDtrr492/eV1UbhjnpZnsuiZpzv/IaMhZLxm309mmQcpZrh+tuarWMAbuMeF6YfGfeWzGbFFpzwJ6VRjMFCYvStWdXeRPYq4DzhcZIR6gpkEZ+23TN/u/PNiPF1yLdEYZmzS1ulrrK/n5fv0v2qs1HtO6quVmyjEiXnO3ye7Hw9Gu6u5yvODnAbfS5GtXJJPc7h94WxJpfXW0r6WHn/vGWcPXzLmv3Hx6/oR5KSfGspzy51nvnMcd2HhclZ1f4I2df48K+Fbh1grPCCXHf53ajheI3qquSwy40j98QATOt9kerwl7tvn4+rn5fsjL9o3Bzn85P9PUD4+0ylsE+m9o0qaTuPo+H5voLj9yxcXmUMjhq/tfoVc7KqvZqBT+M3WL9W1+8G+xfHtO/ZP43qGyVlniffWkuey9OOCrS16M+txZe5bvtGLrjeTJVnn+8P+4Q5trCes+tqwoj2/Fo0xAwsaQdYzuaS9Cv9ZTy1sn4dXftKng4i862S6bR+/d+W2b92e2cH/mmsdn2y3fL2kjKXtNHM4zf+kSRpDAbSjhq584wdylnVfjfDo7rdMpp5BjxcGjnQqDKrq/jsyg+4KVW7kXKbMZi3XmP46XdTXs8rXo+6jSPOiLMDLOrwzqrPBTRwU5e6vicjWnfvIPvSavXJkoZOTTtqHE04fltU7+Y7YwdvlQkkrz97xmlNqP0m9qO01uAJ/2X9qyyoARTy52Mxkmrp7E8UX23m2PsZWFfZY8E4moQ4O8DqwgHEIVm3PkhevqmpAjeoVcqTkXuttK37Rq0+2bQSIr/f+i+wFx2/2Wlb1HPPfpXNA2xrQu23cvizWNMYR/gnuFtGLlaclQbufDIk3xIrlvlpBrZR/zl2oPKfaV+uR6Q4jLMDAAA/7vr4eSdC7Td0FlzI2/UiMIxn2NAK2P/piatvQHnIte4ce2/d6qr8emQcjXUWZ7ekHQAAWIjXot7c7y5vU+y/odVr3BLFFZdGqqt4ncs8vD9nGF7mKvn+nEXeju2fx7lrHLZ6+97vekQt4uwAAHDJYqAlCLVXMzD8NNvLvodrUSFV9g4bFcbtbHi+GfU884Uqpj9/nvsMV9/ZxuCPQJHC//ScObawnrPravhPOK/5xtGlmP48c/nfxNkBACDgUQ+kNyDUTkjJSL7xLJAXrRtrVJnVFS1ktFFkwHH+uGRPtxkLa5X20p16qTg7AAAkudPjwC0JtdcUXpTX4VG/XWQ8dSTnlSQy1ah6vnzD3s8+v4G0C5V5xXxjMj0zvK7q9o2mV9+n9edwpp11mCer1HP1he3ZlV/Sav3vYlNLG66rye/CxdkBACBs81aqye/weXktajvvwVC4C3Ckn5e5vf73ATvvib0k4Ss3vLIp/+cn4XkkcL6tJ6BR1VUiu8yr5HvYl15Z9Tykrkr6c/9mfVtxDG4KdnmEWnNsifJ5MrvAScmrX49eieO34vWop9T2DX9/CeLsAABwSWx9OV/PeaTZP9Q16q+RwYiMVIHkMUfIS3WW8DJ5fBjo8jipSWJSZX8hnHXkDwMB/btloDceLm69/DA738syRApsIdKofcv7VZU/K9n83B2TpERJmQtX3eb1q0CmhR3jMm0VFefJdmPhMmFJpuHkLa5HPe9ru10Hs22yy76BPAuylxwTYBKbKU58BABuIOlRyKr2+g4X03VYwhzIIuYmL1Ds6qkijxM+VEaSWs4WD5ZUV8/l3vvcb5ZvYbecp8zZmfZ5rltxDBZm+iqYY9sVIDxPljRQ3bmuJG3MPFnrelQlbXZ2q1wHs4mzA4/y699/Zp6TAYDqrGp/usK1pbCWe3f4e58d9GEcnSlc1R4IsmccDWBOHjkB4H6SHoW8FhXgDsQHoZxx1Eh4Mbs4O3AbJjQAeDihduBWAltMrPXOQIAb+Pr6smkMAACwqNRQklA7cB/vGfBwHhz1qsM+LMWFcsZRRV//J/AdcXbgCSz1AIBH8VpU4D5+3uL4GTKr+JrE2Xh4g3LGUUXh2PoPQXYAAGB++6fFy2cZq9qBW9lE0m8cZ9+799lBH8ZRpK8jl6nszA7c3n6W85suADyHVe3A3bwjZfunmptF0O69JQ70YRz1JMgOAACsImNJ++v1+nrOY89+sZWHagCAGNmrMp9zqwnww7MnACzt8PEn5tHGBjIAANRnuxiAH7aRAYClRT7aCLUDAFDH94fRZQEY5nAOFG0HgCWUXLLt1Q4AQA7xdIAkv/79x04yADCtsyB7/IOPVe0AAKSxbh0g7GyS/PXvP5a3A8CEyuPsL6vaAQAAoLrv7+/9+1Hf3g/zVrgDwAwCv4KnLjASagcAAID6AtH2l4A7AAx1+XdmGX/IK9QOAAAATYSj7S+vSwWAKeVtmCnUDnX83CLfaVnKLU/q4c6e5TQxHQRCCbfsgaZQAN7ez+rhgDsAMI/sF1N5LSrAU1gzBQAwirdJA8D8vr+/Sy7ZVrXDMasRuZnPOPumV4dD8MYCtey7kJ9/YhiDALdheTsATKvKj+JC7QDPso/Wid8BAHTz8yQv5g4Aw9X9szOhdoD7syoWAGA2tpQBgJuxVzsAAAAAABQRagcAAAAAgCI2kBlj/yK48K4Ol5s/ZHzhsgyRO07EfC3+fANvbsz+ZqSzt/Mdfh6T4yZhZCFT+0YteaU9TBtIvukt+0ZMataSujo839Q+n5FviaR8S/pzxbGQV1cZ81VF5fNzyWjKM6rMS4yFV8TQzhv7SfNG4AiXXwt/blcoAACASQi19xZ+fu72wHxWjG55nZ3v7z9/U8s2YZTh7JQzYsev9n0ju7RnaV8RZd4nPPvk8CAl4yhQyTHmaaMO+RaqVVfd5qvyei4ZTXlGlXlUn5wq36Se2b9vAAAA0NnXc97Esn+9e//n28OFeDFr3Cquag8UYJ+2er4x55u6TL5RO6a+RvJssWH8WtGMuioXXqQZzvqywPuEkUnC7ZuRbyDt/sOk5EPaKGMpcclrUfPSltRV3nxVKLuey8d+tlFlzr6WBQ6VtyQ89XzzrmXZ80atvtGu2yfZFPs5N5AAAACRrGof4/NpefPQ3meN274ALRxGBy7P9ycMEVMVc64HzGjfvLoqdBbRPgskXaYNbPex9/7yYWA98McNJfmWnO9rpjZKquch6tZVh/mqVj2v2DdKyrzi+VbJ9xU9b8SktbYdAADgRgmxPgAAC/tJREFUHrwWtZ/wqrSej9m18opcBjj8fDvLON/hdVWSbzhtYHlydo4l+WbnPnkbTRVzr1tXneeKzv2qCmPh1WUstJsnAQAAuAer2icy4VN30gLzbvlO8qf0Dzeq8p/W6E8731GWq+epflzZaFSZy7URAAAAT2NVez+TrD/tE62IiYaLm7yNqqun/WLRerPyh1Tjpbp1pVYvxdRz3YvOJNeyPp42TwIAAFDCqvYxFnp6P9s1O/IUYmIx4SXzm39dqOpSldfVKKMibv3zHdtGa0U29ecnGDUhayMAAAAmJNTe1f4tagsFjkftIcPMnhNkH+tp5zuKeo408FqmjQAAAJiWUPsA+yDFa6mY+6fL0q51OmMtV1f7mNfh3x/cJt/XoDYaeL4l9Ofb638t00YAAADMTKh9mLMAwYQ7KuwXmPeJaOxfjrroDxK39NkHejbHqHxHedr5jqKes3W7lmkjAAAAJue1qFP4/efvEoGDjAi7ZYbxFq2rUV13kgXm3SwxRXzSnx+oz7VMGwEAADAnofZ+fv37z/u/0QXJcRjaCMQ7aoVCfo7zWXU3i7MMD1Uv2idTlZzvzbpcU+qqs0B/bjRnLn0tS/W0eRIAAIASQu2kyYiwxESCeLtNXd34dalTtdHkvWKquioxYWn9pLExYRsBAADwNELtA5Qsqj1MOyTEkPRC1ECxUwvfP8DUoXob1VV8vnX71fxx9ozzHdVGAdNG9iesqxLzlzPcn9vNmfPUzGVJqsx1M1x/56lzAAAA9rwWtZ/Pl4t+BkHyFolnJCxUmGN2sfuf6T7ffeGbRvyHNPFPpq/EaE52wkIZ+R62aXzyw9y7nfKoen4Vj4VRU1aegfWcKtwuPfPtlvVnpt0Gb8V5I9vA6xEAAADxrGrvav88vH9WP3tm3nz+mXDax+z9K/I257vK+2DfmgZWRtXVPtOMlcv7hO2atTDfQPKYtP3baFQ9h8UsIl5r7M9Zz5fClbz/QqN847P+2ed9v+F74J8OD1g+9kvSDg9zT/47EAAAwDNZ1d5bYDVczNN+t0hKuABJmZ4tHswoeeegxlljNY0j16qrpExfR5HQ/YfxCZsqzLc8eec2GlXPl2W4LMaQ/pxthnrO0H+aCufbNOsWgzcy+fDuMaqhAQAAiPf1/f09ugydfH19bT7xgLqK4esHeQg9DUj1nHljE+h/zg0kAABAJBvIMLvnRDEAAAAAgEUJtQMPEtgSx97HwCHzBgAAADGE2pmaJe1U9O5Oh6GxJd6BCfRn3gAAACCSvdqZkTe/0cj+rYbLvQYT6My88WavdgAAgDCr2lnAQ6IYdLDpS8+MlwFJzBsAAADE+O/oAsCWP8mnqXen8pcTQDzzBgAAAJdsIAMAwAUbyAAAAITZQAYAAAAAAIoItQMAAAAAQBGhdgAAAAAAKCLUDgAAAAAARYTaAQAAAACgiFA7AAAAAAAUEWoHAAAAAIAi/x1dAMb49e8/h5///vO3c0mgm59ur58DPb0nn5+Zx1wEAABwS0LtT3QWZwfu7WkBvhXPV5n7WLHMAAAATE6o/XE+4+ybEIMQPGsRLAMAAABgEvZqf659dFK8EgAAAAAgg1D7s1gFDAAAAABQnQ1kAJ7iab+xrXi+ytzHimUGAABgcla1AwAAAABAEava17N/eWlgdd7Zm04PP2+3yi+pzCVpI3fIiflaXpn3R049TkldxTurgffnZ+/LPStJ//aN+TypveKTHGbXp2+kCrzo+DKjy9JW75abHPcvcA680vmwhJ+SStutPx9+PyPTjHxL0lbsV0mdqkraPcvbAQAAqEWofSXhOOOc8YLCMh8m73C+tfINxHea5ttTdpnH9uezYnfrk6l9Y6DsuqqV3dkn7cL9GdmN6s8rlvks68mvCyU2BZu2nAAAAJT4+v7+Hl2GTr6+vjafrPWsu1/Bl7GMsfNrUQ9LGL8U8fKUwytbL5dgR67RLinz4UFa5JvqsDBnq4YDJc8u85D+fLb4vXWfDOT+iusbtcRnVF5X2WU7y2v/Yd0eUjJfFc51h4e6ZZljMj07TknawNHWuhMYa1Pbz7mBBAAAiGRV+xoOo0uBbShm8xnL2Pzt/9na2MM4SGTaEnXz3R+kT74xBUvtNoE4e2qZZ+jPw/tkfN8YrvMY/MnlMLCe0XXzcv/8//jz7V9X5fkOL3PG2F/xOggAAMDteS3qYg6jHnPGGsJrBjPizpFpS0pVN9/477c733aqlHlUfx7eJ+ds00P9++SQUTDnfBW2YpkjswiP/ZK0AAAA0I5V7Qu4a+CgUTTnZ9Fr0/WYlNAuLKek047q8MuVueRPFjr8uQMAAACEWdW+kkDsY8KF7dlFitlCt0UYqG6+tZa0px4t1edOETGfbz4cUub+RvUNuimZQkdNvyuWGQAAAG7MqnZ6yHsBXUwM6HDp+tnyxshiZOdbaFS+nzY7NceUJ+Y7gTKvFemboY1oreSFmaNetrlimQEAAOBmrGqnod9//m5iN7/+/ef9X58CrBXG7SwprNbozwg0EPMoma9GzXUrljnMUn0AAADWZVU7zR0ulB64A8nlYVfcWHmU7DLvY2Gfh5o2UrZiG5Eke74qTFtixTIDAADA/Qi108lZILXRbhv7PWSmjd7OoHPlfGYnlseESuarznNdlXxHlXnPTjgAAACsywYyDLDfuOBQlfhvxkFGBeWH/xjw0yjxO7YXlnm5iNjwNqK/yPmqetoSK5YZAAAAVifUvpJAmG/C1XzZ2/7WOoXD4/Tfr+bS8CaL6Vcbw8vc2dPO94FKtikftcX5imUGAACAGxNqX8CTw3wZUeDDb6ZGlKrkm6FzvpFdK/y16mWePPw3qm/A7ZWMoNaj7/DHif2HfsMAAAB4MqH2xRw+wE+4pP1T4cL2wCmnHjl7w+LCfJMK1jPfEo3KPG10fsU2IsPM0d4W+c5Z5uyf9y7T5tkE088+PPwaAAAAz/Gg16J+f39/fX2NLkWmz5d8vv9n/9rP2ezL/MotdvYpF9bSqKoeku8mPpWae16Zf1K9siJT4XHR9JenVYYhkUrmq4pzXZIVy/ypZBAZgAAAAEzoQaH21W1iCvv4woRL2vdxkPhi78Ovm7SNzvdp+ZbILnO4M5eEz5rG3ZZro6S9bmYrfIbC8y2cr/LSPrDM5Uky0j5tLAAAADCEDWRW8vvP37NXfU4bGgiULabY4UB8ZAFSk1TJN8+ofEvklfmw9TN6cmBQJB0nKcekz1lIyXxVONdlW7HMZ8ePzLTW7BHv88iH15T3/x9+DQAAgOf4+v7+Hl2GfjYbyHgSBoBuSvZ3mvytJLe3X/7/qBtIAACAGI9e1W6PVwAAAAAAyj061A4AQCpL2gEAAPaeFWr3ZAgAkMRfAQIAAMR4Vqh9z9MjAAAAAACFnh5qBwAgnr8RBAAAOPS4UPv++dDCdgCAQ26TAAAAIv13dAEAgEf4/efvkLQAAADQweNWtR+yYgsAYGN/g2T3GAAAgDNPDLV7SgQAAAAAoKInhtoPWdgOAPDDknYAAIAkDw21e1YEADhjCQIAAECqh4baD3mqBAA4vCOyTAEAACDsuaH2wydG0XYAAAAAAFI9N9T+Em0HAPhflrQDAADkeXSo/SXaDgDwf8TZAQAAsj091H5GtB0AeBRxdgAAgBJC7afPkKLtAMBDuO0BAAAo9GWx0tvX19fh57///O1cEgCAbgJBdneJAAAA8YTa/7+zaPtLwB0AuCNxdgAAgFqE2v9HINr+EnAHAO4ivGOM+0MAAIBUQu1b4Wj7S8AdAFiZIDsAAEALQu3HBNwBgJu5fPep20IAAIBsQu2nLqPtb2LuAMDMLiPsb+4JAQAASgi1h0RG2wEA1uVuEAAAoJxQ+zUBdwDgltwHAgAA1CLUHkvAHQC4DXeAAAAAdQm1JxNzBwDW5d4PAACgBaH2fGLuAMAq3PIBAAA0JdRek+A7ADADN3gAAACdCbUDAAAAAECR/4wuAAAAAAAArE2oHQAAAAAAigi1AwAAAABAEaF2AAAAAAAoItQOAAAAAABFhNoBAAAAAKCIUDsAAAAAABQRagcAAAAAgCJC7QAAAAAAUESoHQAAAAAAivw/CPa1JjlvFzEAAAAASUVORK5CYII=)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "MQ571N6bOaMp",
   "metadata": {
    "id": "MQ571N6bOaMp"
   },
   "source": [
    "In this assignment, we will primarily use the T5 model to generate summaries and translations of text. As you may have learned in Exercise Week 4, T5 is a Seq2Seq encoder-decoder model based on the Transformer architecture. It was developed by a team at Google [(Raffel et al., JMLR 2020)](https://arxiv.org/abs/1910.10683) to unify and test the extent of transferability of text-to-text tasks such as translation, summarization, question answering, and the like in a single model.\n",
    "\n",
    "It's been trained to take the input text as an encoder input. This input text typically has a prefix, which can be considered as an instruction on what to do with respect to the input text when the decoder starts generating.\n",
    "\n",
    "T5 then expects the decoder inputs to start with a `<pad>` token, after which we can let it generate by, for example, choosing the highest probability tokens. But is this really the best way to create our generations? Let's find out!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddedbbcd",
   "metadata": {
    "id": "ddedbbcd"
   },
   "source": [
    "## **PART 1: Natural Language Generation Decoding and Sampling Algorithms**\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "XdYMsi56KsvX",
   "metadata": {
    "id": "XdYMsi56KsvX"
   },
   "source": [
    "### 1.1) Implement decoding and sampling algorithms\n",
    "\n",
    "For this part of the assignment, you will be implementing decoding and sampling algorithms that are in `a3_decoding.py` and `a3_sampling.py` files. These files include a main function that print the behavior of your own implementation and the huggingface implementation.\n",
    "\n",
    "The NLG algorithms will be specifically for the `T5ForConditionalGeneration` model (provided by the `transformers` package) and its `t5-small` variant. While we provide a way to input the encoder and decoder input ids to the model, we highly recommend you closely look at the [documentation](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5ForConditionalGeneration) whenever you feel the need for clarification. **We do not allow you to use the huggingface `generate` function.**\n",
    "\n",
    "Remember that there are several ways to keep track of generation scores and that sampling can be affected by the specific package you choose. Therefore if the output does not match exactly huggingface's output for sampling algorithms, you don't have to worry.\n",
    "\n",
    "Please carefully read the detailed docstrings and the comments that start with `TODO` / `NOTE: caution`. While the `TODO`s will show where you should add code, the caution notes will point out what not to change.\n",
    "\n",
    "Important points to not miss:\n",
    "\n",
    "#### ⚠️ **Note: Do not use huggingface's `generate` function. The point of the exercise is for you to write parts of `generate`'s behavior.**\n",
    "\n",
    "#### ⚠️ **Note: Feel free to add testing cells below those already provided in this notebook. You are also welcome to write tests in the main function of `decoding.py` and `sampling.py`. Just make sure not to change the a3_tests.py file!**\n",
    "\n",
    "#### ⚠️ **Note: Decoding algorithms expectations - your greedy decoder output should match huggingface's output. Sometimes for beam search, due to floating point precision or simply the way huggingface counts scores (thresholding log probabilities), your scores may not align with theirs. Just aim for it to be around the same ballpark. We are not measuring how well you can replicate huggingface's implementation to the exact number. Therefore we will be flexible about this as long as your algorithm does the steps required. Also, a num_beams larger than 15 can run longer than a minute; you don't have to write the most efficient beam search!** ⚠️\n",
    "\n",
    "#### ⚠️ **Note: Sampling algorithms expectations - due to the random seed not being completely set right before sampling time in huggingface's implementation, your results may widely differ.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "QtP0aCXILjyf",
   "metadata": {
    "id": "QtP0aCXILjyf"
   },
   "source": [
    "Here is a reasonable order to implement the classes and features in these python files, although you are welcome to do it in a different order:\n",
    "- [ ] Implement the `GreedySearchDecoderForT5` class in `a3_decoding.py`\n",
    "- [ ] Implement the `BeamSearchDecoderForT5` class in `a3_decoding.py` without *length_penalty*\n",
    "- [ ] Add *length_penalty* to the `BeamSearchDecoderForT5` class in `decoding.py`\n",
    "- [ ] Implement the `TopKSamplerForT5` class in `a3_sampling.py` without *temperature*\n",
    "- [ ] Implement the `TopPSamplerForT5` class in `a3_sampling.py` without *temperature*\n",
    "- [ ] Add *temperature* functionality to both `TopKSamplerForT5` and `TopPSamplerForT5`\n",
    "\n",
    "You can modify the main function in these files to test your functions. We have also created the following cells to call your functions and see if they implement most features we request. These tests do not cover all cases. We don't expect you to throw errors when the inputs don't match their constraints, but whenever the constraints are not met, we return None and print statements to help you debug."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "gtEj0cd-MHBt",
   "metadata": {
    "id": "gtEj0cd-MHBt"
   },
   "source": [
    "### 1.2) Testing your implementation\n",
    "\n",
    "Let's first load up the relevant language modeling class and its tokenizers. Then we will create instances of our decoding and sampling classes in the testing subsection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "flRS42ajr0pn",
   "metadata": {
    "id": "flRS42ajr0pn",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[autoreload of a3_decoding failed: Traceback (most recent call last):\n",
      "  File \"C:\\Users\\mathi\\anaconda3\\lib\\site-packages\\IPython\\extensions\\autoreload.py\", line 245, in check\n",
      "    superreload(m, reload, self.old_objects)\n",
      "  File \"C:\\Users\\mathi\\anaconda3\\lib\\site-packages\\IPython\\extensions\\autoreload.py\", line 394, in superreload\n",
      "    module = reload(module)\n",
      "  File \"C:\\Users\\mathi\\anaconda3\\lib\\imp.py\", line 314, in reload\n",
      "    return importlib.reload(module)\n",
      "  File \"C:\\Users\\mathi\\anaconda3\\lib\\importlib\\__init__.py\", line 169, in reload\n",
      "    _bootstrap._exec(spec, module)\n",
      "  File \"<frozen importlib._bootstrap>\", line 613, in _exec\n",
      "  File \"<frozen importlib._bootstrap_external>\", line 846, in exec_module\n",
      "  File \"<frozen importlib._bootstrap_external>\", line 983, in get_code\n",
      "  File \"<frozen importlib._bootstrap_external>\", line 913, in source_to_code\n",
      "  File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n",
      "  File \"C:\\Users\\mathi\\Documents\\EPFL_Master\\MA4\\assignment-cs-522\\a3-Mathiponds\\a3_decoding.py\", line 82\n",
      "    model_inputs = self.prepare_next_inputs(\n",
      "    ^\n",
      "SyntaxError: invalid syntax\n",
      "]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "T5 Abstract:\n",
      "Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.\n",
      "--------------------------------------------------\n",
      "XSUM Article Example:\n",
      "Gundogan, 26, told BBC Sport he \"can see the finishing line\" after tearing cruciate knee ligaments in December, but will not rush his return.\n",
      "The German missed the 2014 World Cup following back surgery that kept him out for a year, and sat out Euro 2016 because of a dislocated kneecap.\n",
      "He said: \"It is heavy mentally to accept that.\"\n",
      "Gundogan will not be fit for the start of the Premier League season at Brighton on 12 August but said his recovery time is now being measured in \"weeks\" rather than months.\n",
      "He told BBC Sport: \"It is really hard always to fall and fight your way back. You feel good and feel ready, then you get the next kick.\n",
      "\"The worst part is behind me now. I want to feel ready when I am fully back. I want to feel safe and confident. I don't mind if it is two weeks or six.\"\n",
      "Gundogan made 15 appearances and scored five goals in his debut season for City following his Â£20m move from Borussia Dortmund.\n",
      "He is eager to get on the field again and was impressed at the club's 4-1 win over Real Madrid in a pre-season game in Los Angeles on Wednesday.\n",
      "Manager Pep Guardiola has made five new signings already this summer and continues to have an interest in Arsenal forward Alexis Sanchez and Monaco's Kylian Mbappe.\n",
      "Gundogan said: \"Optimism for the season is big. It is huge, definitely.\n",
      "\"We felt that last year as well but it was a completely new experience for all of us. We know the Premier League a bit more now and can't wait for the season to start.\"\n",
      "City complete their three-match tour of the United States against Tottenham in Nashville on Saturday.\n",
      "Chelsea manager Antonio Conte said earlier this week he did not feel Tottenham were judged by the same standards as his own side, City and Manchester United.\n",
      "Spurs have had the advantage in their recent meetings with City, winning three and drawing one of their last four Premier League games.\n",
      "And Gundogan thinks they are a major threat.\n",
      "He said: \"Tottenham are a great team. They have the style of football. They have young English players. Our experience last season shows it is really tough to beat them.\n",
      "\"They are really uncomfortable to play against.\n",
      "\"I am pretty sure, even if they will not say it loud, the people who know the Premier League know Tottenham are definitely a competitor for the title.\"\n",
      "--------------------------------------------------\n",
      "Opus Books Translation Example:\n",
      "{'en': 'They set out on their walk, gazing into the wood and thickets through which goats and pigs fled in hundreds.', 'fr': \"On se remit donc en marche, tout en fouillant du regard les bois et les taillis, Ã\\xa0 travers lesquels chÃ¨vres et porcs s'enfuyaient par centaines.\"}\n",
      "--------------------------------------------------\n",
      "Not putting inputs and model on any GPU device.\n"
     ]
    }
   ],
   "source": [
    "# 1) Load model and tokenizer\n",
    "model_name = \"t5-small\"\n",
    "tokenizer = T5Tokenizer.from_pretrained(model_name)\n",
    "model = T5ForConditionalGeneration.from_pretrained(model_name)\n",
    "\n",
    "# 2) Load relevant inputs\n",
    "with open(os.path.join(ROOT_PATH, \"part1_input_data.json\"), 'r') as read_file:\n",
    "    input_data = json.load(read_file)\n",
    "print(\"-\" * 50)\n",
    "t5_paper_abstract = input_data[\"t5_paper_abstract\"]\n",
    "print(\"T5 Abstract:\")\n",
    "print(t5_paper_abstract)\n",
    "print(\"-\" * 50)\n",
    "\n",
    "xsum_train_article_example = input_data[\"xsum_train_article_example\"]\n",
    "print(\"XSUM Article Example:\")\n",
    "print(xsum_train_article_example)\n",
    "print(\"-\" * 50)\n",
    "\n",
    "opusbooks_train_translation_example = input_data[\"opusbooks_train_translation_example\"]\n",
    "print(\"Opus Books Translation Example:\")\n",
    "print(opusbooks_train_translation_example)\n",
    "print(\"-\" * 50)\n",
    "#\n",
    "summary_prefix = \"summarize: \"\n",
    "translation_prefix = \"translate English to French: \"\n",
    "#\n",
    "abstract_inputs = tokenizer(\n",
    "    [summary_prefix + t5_paper_abstract], \n",
    "    max_length=MAX_T5_SEQ_LENGTH, \n",
    "    truncation=True, \n",
    "    return_tensors=\"pt\"\n",
    ")\n",
    "article_inputs = tokenizer(\n",
    "    [summary_prefix + xsum_train_article_example],\n",
    "    max_length=MAX_T5_SEQ_LENGTH, \n",
    "    truncation=True, \n",
    "    return_tensors=\"pt\"\n",
    ")  \n",
    "translation_inputs = tokenizer(\n",
    "    [translation_prefix + opusbooks_train_translation_example[\"en\"]], \n",
    "    max_length=MAX_T5_SEQ_LENGTH, \n",
    "    truncation=True, \n",
    "    return_tensors=\"pt\"\n",
    ")  \n",
    "\n",
    "all_inputs = [\n",
    "    (\"T5 Abstract\", abstract_inputs), \n",
    "    [\"XSUM article\", article_inputs], \n",
    "    [\"Eng-Fr translation\", translation_inputs]\n",
    "]\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    # TODO: if you want a specific GPU, you can choose which device here, \n",
    "    #       otherwise picks the default one\n",
    "    # device_name = 'cuda' # 'cuda:0', 'cuda:1'\n",
    "    # device = torch.device(device_name)\n",
    "    # print(f\"Putting inputs and model on the GPU device {device_name}.\")\n",
    "    model = model.cuda()\n",
    "    for _, inputs in all_inputs:\n",
    "        inputs[\"input_ids\"] = inputs[\"input_ids\"].cuda()\n",
    "        inputs[\"attention_mask\"] = inputs[\"attention_mask\"].cuda()\n",
    "    print(\"All inputs and models are on a GPU device.\")\n",
    "else:\n",
    "    print(\"Not putting inputs and model on any GPU device.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c5e5e57",
   "metadata": {},
   "source": [
    "Perfect! Now you can use the following tests to debug the behavior of your model. Feel free to play around with the parameters but do not change the a3_tests.py file. Do not hardcode answers, we will use other datapoints to test your implementation ;)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e44723c",
   "metadata": {},
   "source": [
    "#### 1) Greedy search test\n",
    "\n",
    "Greedy search is the first thing you can implement. The output here should match the huggingface implementation. Make sure you implement the *max_new_tokens* parameter behavior right!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "9915847e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "Greedy Tests\n",
      "--------------------------------------------------\n",
      "####################\n",
      "Input:  T5 Abstract\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>a new framework is introducing a framework that converts all text-based language problems into a text-to\n",
      "Output shape:  torch.Size([1, 26])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  summarize: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new <unk> Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.</s><pad>.. a systematic study compares pre-training, the study compares pre-training, the study compares pre-\n",
      "Output shape:  torch.Size([1, 258])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  XSUM article\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>german will not be fit for the start of the premier league season at Brighton. the 26-year-old has been\n",
      "Output shape:  torch.Size([1, 26])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>german will not be fit for the start of the premier league season at Brighton. the 26-year-old has been\n",
      "Output shape:  torch.Size([1, 26])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  Eng-Fr translation\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>Ils se sont rendus à pied, en regardant dans le bois et les épaisses où\n",
      "Output shape:  torch.Size([1, 26])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>Ils se sont rendus à pied, en regardant dans le bois et les épaisses où\n",
      "Output shape:  torch.Size([1, 26])\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 1) Relevant parameters to greedy search\n",
    "max_new_tokens = 25\n",
    "\n",
    "# 2) Run it on the 3 examples\n",
    "greedy_test(\n",
    "    model,\n",
    "    tokenizer,\n",
    "    all_inputs,\n",
    "    max_new_tokens\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18425d77",
   "metadata": {},
   "source": [
    "#### 2) Beam search test without length penalty\n",
    "\n",
    "Next, we implement beam search. First, implement it without length penalty and see if the log probability scores you get are roughly close to huggingface's for different beam sizes (particularly smaller ones). The sequence should be mostly the same, independent of the beam width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "id": "c6221f2b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "Beam Tests\n",
      "--------------------------------------------------\n",
      "####################\n",
      "Input:  T5 Abstract\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "1. score: -3.6086858682974707\n",
      "1. generated sequence: <pad>the effectiveness of transfer learning has given rise to a diversity of approaches,\n",
      "2. score: -5.180535542895086\n",
      "2. generated sequence: <pad>transfer learning has emerged as a powerful technique in natural language processing (N\n",
      "3. score: -5.191677702212473\n",
      "3. generated sequence: <pad>the effectiveness of transfer learning has given rise to a variety of approaches,\n",
      "4. score: -5.578663132765541\n",
      "4. generated sequence: <pad>transfer learning is a powerful technique in natural language processing (NLP)\n",
      "Best output shape:  torch.Size([4, 16])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "1. score: -3.5028603076934814\n",
      "1. generated sequence: <pad>the effectiveness of transfer learning has given rise to a diversity of approaches,\n",
      "2. score: -5.006743431091309\n",
      "2. generated sequence: <pad>transfer learning has emerged as a powerful technique in natural language processing (N\n",
      "3. score: -5.30509090423584\n",
      "3. generated sequence: <pad>the effectiveness of transfer learning has given rise to a variety of approaches,\n",
      "4. score: -5.653446197509766\n",
      "4. generated sequence: <pad>transfer learning has emerged as a powerful technique in natural language processing.\n",
      "Best output shape:  torch.Size([4, 16])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  XSUM article\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "1. score: -6.0659508097046455\n",
      "1. generated sequence: <pad>gundogan will not be fit for the start of the premier league season\n",
      "2. score: -7.030484471196814\n",
      "2. generated sequence: <pad>gundogan, 26, will not be fit for the start of the premier\n",
      "3. score: -7.164324505912646\n",
      "3. generated sequence: <pad>gundogan will not be fit for the start of the season at Brighton\n",
      "4. score: -7.283562092768534\n",
      "4. generated sequence: <pad>gundogan will not be fit for the start of the premier league at\n",
      "Best output shape:  torch.Size([4, 16])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "1. score: -5.924601078033447\n",
      "1. generated sequence: <pad>gundogan will not be fit for the start of the premier league season\n",
      "2. score: -7.014659881591797\n",
      "2. generated sequence: <pad>gundogan will not be fit for the start of the season at Brighton\n",
      "3. score: -7.1187028884887695\n",
      "3. generated sequence: <pad>gundogan will not be fit for the start of the premier league at\n",
      "4. score: -7.223356246948242\n",
      "4. generated sequence: <pad>gundogan says he can see the finishing line after tearing\n",
      "Best output shape:  torch.Size([4, 16])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  Eng-Fr translation\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "1. score: -10.706351754721254\n",
      "1. generated sequence: <pad>Ils se sont établis à leur marche,\n",
      "2. score: -10.726346222218126\n",
      "2. generated sequence: <pad>Ils se sont établis à la marche,\n",
      "3. score: -10.82033654814586\n",
      "3. generated sequence: <pad>Ils se sont établis en marche, en\n",
      "4. score: -11.094825563486665\n",
      "4. generated sequence: <pad>Ils se sont établis sur leur marche, en\n",
      "Best output shape:  torch.Size([4, 16])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "1. score: -9.328351020812988\n",
      "1. generated sequence: <pad>Ils se sont établis à leur promena\n",
      "2. score: -10.777024269104004\n",
      "2. generated sequence: <pad>Ils se sont établis en marche, en\n",
      "3. score: -10.785551071166992\n",
      "3. generated sequence: <pad>Ils se sont établis à leur marche,\n",
      "4. score: -11.053553581237793\n",
      "4. generated sequence: <pad>Ils se sont établis sur leur marche, en\n",
      "Best output shape:  torch.Size([4, 16])\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 1) Relevant parameters to beam search\n",
    "max_new_tokens = 15\n",
    "num_beams = 4\n",
    "num_return_sequences = 4\n",
    "length_penalty = 0.0\n",
    "\n",
    "# 2) Run it on the 3 examples\n",
    "beam_test(\n",
    "    model,\n",
    "    tokenizer,\n",
    "    all_inputs,\n",
    "    max_new_tokens,\n",
    "    num_beams,\n",
    "    length_penalty,\n",
    "    num_return_sequences,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7638911e",
   "metadata": {},
   "source": [
    "#### 3) Beam search test with length penalty\n",
    "\n",
    "Next, we implement length penalty for beam search. We do so by dividing the score (sum of the log probabilities) by the length of the generation exponentiated by the penalty. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "id": "e8fab8e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "Beam Tests\n",
      "--------------------------------------------------\n",
      "####################\n",
      "Input:  T5 Abstract\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "1. score: -0.08161449933979716\n",
      "1. generated sequence: <pad>the effectiveness of transfer learning has given rise to a diversity of approaches, methodologies, and practice. in this paper,\n",
      "2. score: -0.08371889214580516\n",
      "2. generated sequence: <pad>the effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. in this paper,\n",
      "3. score: -0.08412464181490924\n",
      "3. generated sequence: <pad>transfer learning has emerged as a powerful technique in natural language processing (NLP) the effectiveness of transfer learning has given rise\n",
      "Best output shape:  torch.Size([3, 26])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "1. score: -0.035840053111314774\n",
      "1. generated sequence: <pad>our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other\n",
      "2. score: -0.045790981501340866\n",
      "2. generated sequence: <pad>the effectiveness of transfer learning has given rise to a diversity of approaches, methodologies, and practice. in this paper,\n",
      "3. score: -0.04601394757628441\n",
      "3. generated sequence: <pad>the effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. in this paper,\n",
      "Best output shape:  torch.Size([3, 26])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  XSUM article\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "1. score: -0.08551909303401468\n",
      "1. generated sequence: <pad>german will not be fit for the start of the premier league season at Brighton on 12 august. the 26-year-\n",
      "2. score: -0.08856724188777135\n",
      "2. generated sequence: <pad>gundogan will not be fit for the start of the premier league season at Brighton. the 26-year-old\n",
      "3. score: -0.09430810158618154\n",
      "3. generated sequence: <pad>gundogan will not be fit for the start of the premier league season at Brighton on 12 august. the 26-\n",
      "Best output shape:  torch.Size([3, 26])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "1. score: -0.06569607555866241\n",
      "1. generated sequence: <pad>gundogan will not be fit for the start of the premier league season at Brighton on 12 august. the 26-\n",
      "2. score: -0.06793035566806793\n",
      "2. generated sequence: <pad>gundogan will not be fit for the start of the season at Brighton on 12 august. the 26-year-\n",
      "3. score: -0.07059860229492188\n",
      "3. generated sequence: <pad>gundogan says he can see the finishing line after tearing cruciate knee ligaments. the 26-\n",
      "Best output shape:  torch.Size([3, 26])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  Eng-Fr translation\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "1. score: -0.10339404978088332\n",
      "1. generated sequence: <pad>Ils se sont établis à leur promenade, gazant dans le bois et\n",
      "2. score: -0.1109945090912357\n",
      "2. generated sequence: <pad>Ils se sont établis à leur promenade en gazant dans le bois e\n",
      "3. score: -0.11662197847241385\n",
      "3. generated sequence: <pad>Ils se sont établis à leur promenade en regardant le bois et\n",
      "Best output shape:  torch.Size([3, 26])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "1. score: -0.09242898970842361\n",
      "1. generated sequence: <pad>Ils se sont établis à leur promenade, gazant dans le bois et\n",
      "2. score: -0.09994242340326309\n",
      "2. generated sequence: <pad>Ils se sont établis à leur promenade en gazant dans le bois e\n",
      "3. score: -0.10693950206041336\n",
      "3. generated sequence: <pad>Ils se sont établis à leur promenade, en gazant dans le bois\n",
      "Best output shape:  torch.Size([3, 26])\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 1) Relevant parameters to beam search + penalty\n",
    "max_new_tokens = 25\n",
    "num_beams = 5\n",
    "num_return_sequences = 3\n",
    "length_penalty = 1.5\n",
    "\n",
    "# 2) Run it on the 3 examples\n",
    "beam_test(\n",
    "    model,\n",
    "    tokenizer,\n",
    "    all_inputs,\n",
    "    max_new_tokens,\n",
    "    num_beams,\n",
    "    length_penalty,\n",
    "    num_return_sequences,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01eda21a",
   "metadata": {},
   "source": [
    "#### 4) Top-K sampling without temperature\n",
    "\n",
    "Next, we implement top-k sampling. First, implement it without temperature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "id": "cb55970f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "Top-k Tests\n",
      "--------------------------------------------------\n",
      "####################\n",
      "Input:  T5 Abstract\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>introducing integrated framework converting 'empowerance to translation in texts, data set for\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  summarize: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new <unk> Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.</s><pad>.. the study compares pre-training objectives, the a system study compares pre-training\n",
      "Output shape:  torch.Size([1, 253])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  XSUM article\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>german wants second-place injury and his first start on 16 febrs and 14 august at uplift\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>germany star will not move to the emirates this summer. gundogan suffered\n",
      "Output shape:  torch.Size([1, 21])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  Eng-Fr translation\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>Elle marche par pied. Après quelques semaines sur le chemin vers Le parc foresterie s’é\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>Là même, les gens se sont rendus à cette exposition qui re\n",
      "Output shape:  torch.Size([1, 21])\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 1) Relevant parameters to top-k sampling\n",
    "max_new_tokens = 20\n",
    "top_k = 15\n",
    "temperature = 1.0\n",
    "seed = SCIPER\n",
    "\n",
    "# 2) Run it on the 3 examples\n",
    "top_k_test(\n",
    "    model,\n",
    "    tokenizer,\n",
    "    all_inputs,\n",
    "    max_new_tokens,\n",
    "    top_k,\n",
    "    temperature,\n",
    "    seed\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6a7379d",
   "metadata": {},
   "source": [
    "#### 5) Top-P sampling without temperature\n",
    "\n",
    "Then, we implement top-p sampling. First, implement it without temperature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "id": "59235019",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "Top-p Tests\n",
      "--------------------------------------------------\n",
      "####################\n",
      "Input:  T5 Abstract\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>a new framework is introducing a framework that converts all text-based language problems into\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  summarize: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new <unk> Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.</s><pad>.. the the paper and the man at a single framework, we compare. the analysis of\n",
      "Output shape:  torch.Size([1, 253])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  XSUM article\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>german will not be fit for the start of the premier league season at Brighton. the 26-\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>german said the damage will only be reduced during the holiday. defender hopes to start start of\n",
      "Output shape:  torch.Size([1, 21])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  Eng-Fr translation\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>Ils se sont rendus à pied, en regardant dans le bois et\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>Là même, les chefs manquant à la levée de la maladie se faisaient\n",
      "Output shape:  torch.Size([1, 21])\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 1) Relevant parameters to top-p\n",
    "max_new_tokens = 20\n",
    "top_p = 0.92\n",
    "temperature = 1.0\n",
    "seed = SCIPER\n",
    "\n",
    "# 2) Run it on the 3 examples\n",
    "top_p_test(\n",
    "    model,\n",
    "    tokenizer,\n",
    "    all_inputs,\n",
    "    max_new_tokens,\n",
    "    top_p,\n",
    "    temperature,\n",
    "    seed\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caea7252",
   "metadata": {},
   "source": [
    "#### 6) Temperature for both sampling\n",
    "\n",
    "Now let's test what happens when you implement both sampling algorithms with temperature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "id": "c6be31cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "Top-k Tests\n",
      "--------------------------------------------------\n",
      "####################\n",
      "Input:  T5 Abstract\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>in txm study the effects from transcoding for real-term applications, uv\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  summarize: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new <unk> Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.</s><pad>.. the study compares pre-training objectives, the study compares pre-training, the results\n",
      "Output shape:  torch.Size([1, 253])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  XSUM article\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>his recoverance means tom hoddle can play at Brighton city on 1 September for the start of\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>germany star will not be fit for the start of the season at Brighton. the 26-\n",
      "Output shape:  torch.Size([1, 21])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  Eng-Fr translation\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad> Il était le ch<unk>aure en plein temps des centaine, à proximité, des\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>Ils se sont embarqués sur le chemin de la route, en voyant dans le\n",
      "Output shape:  torch.Size([1, 21])\n",
      "\n",
      "\n",
      "\n",
      "--------------------------------------------------\n",
      "Top-p Tests\n",
      "--------------------------------------------------\n",
      "####################\n",
      "Input:  T5 Abstract\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>a new framework is introducing a framework that converts all text-based language problems into\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  summarize: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new <unk> Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.</s><pad>.. the study compares pre-training objectives, the study compares pre-training, the results\n",
      "Output shape:  torch.Size([1, 253])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  XSUM article\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>german will not be fit for the start of the premier league season at Brighton. the 26-\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>germany star will not be fit for the start of the season at Brighton. the 26-\n",
      "Output shape:  torch.Size([1, 21])\n",
      "\n",
      "\n",
      "####################\n",
      "Input:  Eng-Fr translation\n",
      "####################\n",
      "~ Your Implementation ~\n",
      "Generated sequence:  <pad>Ils se sont rendus à pied, en regardant dans le bois et\n",
      "Output shape:  torch.Size([1, 21])\n",
      "--------------------\n",
      "~ Huggingface Implementation ~\n",
      "Generated sequence:  <pad>Ils se sont embarqués sur le chemin de la route, en voyant dans le\n",
      "Output shape:  torch.Size([1, 21])\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 1) Relevant parameters to sampling\n",
    "max_new_tokens = 20\n",
    "top_k = 15\n",
    "top_p = 0.92\n",
    "temperature = 0.7\n",
    "seed = SCIPER\n",
    "\n",
    "# 2) Run it on the 3 examples\n",
    "top_k_test(\n",
    "    model,\n",
    "    tokenizer,\n",
    "    all_inputs,\n",
    "    max_new_tokens,\n",
    "    top_k,\n",
    "    temperature,\n",
    "    seed\n",
    ")\n",
    "\n",
    "print()\n",
    "top_p_test(\n",
    "    model,\n",
    "    tokenizer,\n",
    "    all_inputs,\n",
    "    max_new_tokens,\n",
    "    top_p,\n",
    "    temperature,\n",
    "    seed\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c48088b",
   "metadata": {
    "id": "6c48088b"
   },
   "source": [
    "## **PART 2: Qualitative Evaluation of Generation Parameters**\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Gwc3Av4DN6iY",
   "metadata": {
    "id": "Gwc3Av4DN6iY"
   },
   "source": [
    "As seen in Exercise Week 7, automatic NLG evaluation metrics are difficult to interpret. Therefore, NLG often requires careful manual inspection of generations. For this reason, we want you to inspect the following generations qualitatively and answer the follow-up questions (there is a dedicated markdown cell for each question group). Depending on the parameter you are analyzing, a complete answer can range from 2 to 4 sentences.\n",
    "\n",
    "To observe the different qualitative behavior, we pick a task that T5 has been pretrained with: summarization. We will use the [XSum dataset](https://aclanthology.org/D18-1206/)'s test set and pick an article to summarize.\n",
    "\n",
    "#### ⚠️ **Note: There is no coding in this part. Instead, the outputs are pre-generated and written in the markdown cell below if the random seed is accidentally modified.**\n",
    "\n",
    "#### ⚠️ **Content Warning: The following article to be analyzed has sensitive content on a plane crash.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "id": "a7ddd9bb",
   "metadata": {
    "id": "a7ddd9bb"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4263f3745504780a6a3fd438b973652",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/792k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "db1ad2f706c4456289e9fd7df0b83a08",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/2.32k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "98a2bbffce514a7da3760033a47fd17a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.21k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "413c1d257f184743a5a8b4eba25a74b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/242M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a130ad9db98d4e2d95be573b950c6196",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/147 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:datasets.builder:Found cached dataset xsum (C:/Users/mathi/.cache/huggingface/datasets/xsum/default/1.2.0/082863bf4754ee058a5b6f6525d0cb2b18eadb62c7b370b095d1364050a52b71)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "The article that T5 summarizes:\n",
      "--------------------------------------------------\n",
      "Helio Hermito Zampier Neto, a 31-year-old defender for Chapecoense, was in a stable condition, doctors said, after surgery on a lung, a knee, a wrist and his head.\n",
      "Five other people survived the crash, which killed 71 people.\n",
      "Colombian officials say evidence is growing that the plane ran out of fuel.\n",
      "Miguel Quiroga, the pilot of the British-made Avro RJ85 aircraft, had been warned by an official at Santa Cruz airport, in Bolivia, where the plane took off, that he might not have enough fuel, Bolivian Deber newspaper said.\n",
      "But, despite the official's concerns, he went ahead with the flight to Medellin. The country's authorities have not yet commented.\n",
      "In a leaked tape, the pilot can be heard warning of a \"total electric failure\" and \"lack of fuel\".\n",
      "The flight missed a planned refuelling stop in Cobija, on the border between Brazil and Bolivia, because the airport did not operate at night, Brazil's O Globo newspaper reported.\n",
      "The pilot had the option to refuel in Bogota, it said, but headed straight to Medellin.\n",
      "Bolivia's President Evo Morales said he would take \"drastic measures\" to determine who was responsible for the crash. On Thursday, the country's aviation authority suspended the operating licence of charter airline LaMia, which was part-owned by the pilot and two other aviation officials.\n",
      "Colombian police released a video (in Spanish) with the moment crew member Erwin Tumiri was rescued alive. It showed him conscious, screaming for his colleagues. He is in hospital and reports in local media suggest he may be discharged this weekend.\n",
      "The other crew member who survived, Ximena Suarez, was said to be in good condition in hospital. The four other survivors were still in intensive care.\n",
      "Neto's father, Helam, said on Facebook that news of his recovery was giving the family \"renewed hope and faith\".\n",
      "\"My son is getting better and better. He has just undergone surgery on his leg and doctors say he will return to football,\" he said.\n",
      "\"We shall continue praying because we still need his discharge from hospital to see him closely.\"\n",
      "Doctors said 24-year-old goalkeeper Jakson Ragnar Follmann would not lose his left leg, after having his right one amputated.\n",
      "Defender Alan Ruschel, 27, had spinal surgery, but his movements were not affected, they added.\n",
      "Journalist Rafael Henzel was listed as stable.\n",
      "The bodies of the Brazilian victims were flown out of Medellin on Friday. Many of the victims were players and staff of Chapecoense, who were due to play in the final of the Copa Sudamericana against Medellin team Atletico Nacional.\n",
      "In the squad's home town of Chapeco, in southern Brazil, temporary structures have been set up in the football stadium for an open-air wake on Saturday. Some 100,000 people are expected to attend.\n",
      "The plane's flight recorders are to be examined in the UK. A full investigation into the crash is expected to take months.\n"
     ]
    }
   ],
   "source": [
    "# 1) Reload model and tokenizer in case you loaded something else earlier\n",
    "model_name = \"t5-small\"\n",
    "tokenizer = T5Tokenizer.from_pretrained(model_name)\n",
    "tokenizer.model_max_length = 512\n",
    "model = T5ForConditionalGeneration.from_pretrained(model_name)\n",
    "\n",
    "# 2) Load the test split of XSum, a summarization dataset\n",
    "summaries_test = load_dataset(\"xsum\", split='test')\n",
    "article_id = 102 # NOTE: caution - do not change this!\n",
    "summary_prefix = \"summarize: \" # NOTE: we add the \"summarize\" prefix to the \n",
    "#                                      input because of how T5 has been trained\n",
    "input_ids = tokenizer(\n",
    "    summary_prefix + summaries_test[article_id]['document'], \n",
    "    return_tensors=\"pt\", \n",
    "    max_length=512, \n",
    "    truncation=True # NOTE: the article gets truncated due to length\n",
    ").input_ids\n",
    "\n",
    "# 3) Display the article relevant to Part 2\n",
    "print(\"-\" * 50)\n",
    "print(\"The article that T5 summarizes:\")\n",
    "print(\"-\" * 50)\n",
    "print(summaries_test[article_id]['document'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5287558",
   "metadata": {
    "id": "a5287558"
   },
   "source": [
    "### 2.1) Beam size for beam-search\n",
    "\n",
    "Given the change in the beam size, analyze the following three generations and answer the relevant questions below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "P-Xj7dspqz_9",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "P-Xj7dspqz_9",
    "outputId": "5f311af6-597b-4307-ab94-6710ad24942d"
   },
   "outputs": [],
   "source": [
    "print(\"beam = 8\")\n",
    "beam_output = model.generate(\n",
    "    input_ids,\n",
    "    max_new_tokens=100,\n",
    "    num_beams=8,   \n",
    "    length_penalty=0.0,\n",
    "    do_sample=False\n",
    ")\n",
    "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))\n",
    "\n",
    "print(\"-\" * 50)\n",
    "print(\"beam = 2\")\n",
    "torch.manual_seed(seed)\n",
    "beam_output = model.generate(\n",
    "    input_ids,\n",
    "    max_new_tokens=100,\n",
    "    num_beams=2,\n",
    "    length_penalty=0.0,\n",
    "    do_sample=False\n",
    ")\n",
    "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))\n",
    "\n",
    "print(\"-\" * 50)\n",
    "print(\"beam = 1\")\n",
    "torch.manual_seed(seed)\n",
    "beam_output = model.generate(\n",
    "    input_ids,\n",
    "    max_new_tokens=100,\n",
    "    num_beams=1,\n",
    "    length_penalty=0.0,\n",
    "    do_sample=False\n",
    ")\n",
    "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f87fd538",
   "metadata": {},
   "source": [
    "**Our generations for the three beam size options:**\n",
    "\n",
    "---\n",
    "\n",
    "*beam = 8*\n",
    "\n",
    "defender Helio Hermito Zampier Neto, 31, is in a stable condition, doctors say. he had surgery on a lung, a knee, a wrist and his head. five others survived the crash, which killed 71 people.\n",
    "\n",
    "---\n",
    "\n",
    "*beam = 2*\n",
    "\n",
    "71 people were killed in the crash at a bolivian airport. the pilot had been warned that he might not have enough fuel. he went ahead with the flight to Medellin.\n",
    "\n",
    "---\n",
    "\n",
    "*beam = 1*\n",
    "\n",
    "71 people died in the crash at the airport in bolivia. the pilot had been warned that he might not have enough fuel. the pilot had the option to refuel in Bogota, it said.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c37afa7",
   "metadata": {},
   "source": [
    "#### Questions on (2.1)\n",
    "\n",
    "1. How did the choice for beam width (*num_beams*) affect the generated summary? Your observation can be about coherence, factual accuracy, conciseness, word choice, overall communication of the news story, format, etc. What is your hypothesis as to why this may be happening, or in other words, why might *num_beams* affect this?\n",
    "\n",
    "2. Which of these generations do you prefer for the summarization task and why? Is there something lacking from all three generations that you would have chosen to include in your summarization?\n",
    "\n",
    "3. If the task was something more creative, such as story generation, which *num_beams* between the options above would you prefer?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2600229c",
   "metadata": {},
   "source": [
    "#### Your Answers to (2.1)\n",
    "\n",
    "1. With beam 8, the generated snetence is too general and is very close to the first sentence of the article. Because of the large beam the algorithm can explore a lot ef sentences and take the most likely. However this eventually lead to no summarization or taking one of the sentences in the article\n",
    "    With beam 1 and 2, there is a mistake in the summarization  because the crash did not happend at the airport. However the main points of the article are there.\n",
    "\n",
    "2. We would prefer a beam size between 2 and 8. With beam size of 8 we don't learn why the crash happend but with beam size of 2 we have an error in the summarization.\n",
    "\n",
    "3. We would prefer taking number of beams relatevely high this will permit us to discover more possibilities. However, we do not want to take a beam too high because otherwise we will get too generic sentences. However with beam search it is hard to be creative because the algorithm is deterministic."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e80c9c00",
   "metadata": {
    "id": "e80c9c00"
   },
   "source": [
    "### 2.2) Length Penalty for beam-search\n",
    "\n",
    "Given the change in the length penalty, analyze the following three generations and answer the relevant questions below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41f455b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"penalty = 1.5\")\n",
    "beam_output = model.generate(\n",
    "    input_ids,\n",
    "    max_new_tokens=100,\n",
    "    num_beams=4,\n",
    "    length_penalty=1.5,\n",
    "    do_sample=False\n",
    ")\n",
    "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))\n",
    "\n",
    "print(\"-\" * 50)\n",
    "print(\"penalty = 0.75\")\n",
    "beam_output = model.generate(\n",
    "    input_ids,\n",
    "    max_new_tokens=100,\n",
    "    num_beams=4,\n",
    "    length_penalty=0.75,\n",
    "    do_sample=False\n",
    ")\n",
    "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))\n",
    "\n",
    "print(\"-\" * 50)\n",
    "print(\"penalty = -5.0\")\n",
    "beam_output = model.generate(\n",
    "    input_ids,\n",
    "    max_new_tokens=100,\n",
    "    num_beams=4,\n",
    "    length_penalty=-5.0,\n",
    "    do_sample=False\n",
    ")\n",
    "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aeb7df09",
   "metadata": {},
   "source": [
    "**Our generations for the three penalty options:**\n",
    "\n",
    "---\n",
    "\n",
    "*penalty = 1.5*\n",
    "\n",
    "71 people died in the crash at a bolivian airport. the pilot had been warned that he might not have enough fuel. the pilot had the option to refuel in Bogota, it said, but headed straight to Medellin. he was in a stable condition after surgery on a lung, a knee, a wrist and his head.\n",
    "\n",
    "---\n",
    "\n",
    "*penalty = 0.75*\n",
    "\n",
    "71 people died in the crash at a bolivian airport. the pilot had been warned that he might not have enough fuel. he went ahead with the flight to Medellin.\n",
    "\n",
    "---\n",
    "\n",
    "*penalty = -5.0*\n",
    "\n",
    "71 people died in the crash at a bolivian airport. the pilot had been warned he might not have enough fuel. he went ahead with the flight to Medellin.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af0ce0ae",
   "metadata": {},
   "source": [
    "#### Questions on (2.2)\n",
    "\n",
    "1. How did the choice for *length_penalty* affect the generated summary? Your observation can be about coherence, factual accuracy, conciseness, word choice, overall communication of the news story, format etc. What is your hypothesis as to why this may be happening, or in other words, why might *length_penalty* affect this?\n",
    "\n",
    "2. Given your observations for the previous question, which of these generations do you prefer for the summarization task and why? Is there something lacking from all three generations that you would have chosen to include in your summarization?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f623b6d3",
   "metadata": {},
   "source": [
    "#### Your Answers to (2.2)\n",
    "\n",
    "1. Length penalty is introduced because generation tends to output small generic outputs. A large length penalty will push the model to output longer sentences and smaller ones if the length penalty is negative. This we can see from the generated sentences above : the output and the sentences are longer with a higher length penalty. The generaed sentences with length penalty of 0.75 are shorter and concise and lowering the length penalty to negative values does not change much. This is due to the fact that generative models are already biased to generate short sentences and giving a negative length penalty does not increase this bias.\n",
    "\n",
    "2. If you want a concise summarization of the text, a low length penatly is a good idea. Number of survivors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d31521b",
   "metadata": {},
   "source": [
    "### 2.3) Top-k for top-k\n",
    "\n",
    "Here we will look at the output for three *k* values. Analyze the following generations given the change and answer the relevant questions below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b1926d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Model vocab size: \", model.config.vocab_size)\n",
    "\n",
    "print(\"-\" * 50)\n",
    "print(\"k = 32128\")\n",
    "torch.manual_seed(0)\n",
    "k_output = model.generate(\n",
    "        input_ids,\n",
    "        top_k = 32128,\n",
    "        max_new_tokens=100,\n",
    "        do_sample=True,\n",
    "    )\n",
    "print(tokenizer.decode(k_output[0], skip_special_tokens=True))\n",
    "\n",
    "print(\"-\" * 50)\n",
    "print(\"k = 500\")\n",
    "torch.manual_seed(0)\n",
    "k_output = model.generate(\n",
    "        input_ids,\n",
    "        top_k = 500,\n",
    "        max_new_tokens=100,\n",
    "        do_sample=True,\n",
    "    )\n",
    "print(tokenizer.decode(k_output[0], skip_special_tokens=True))\n",
    "\n",
    "print(\"-\" * 50)\n",
    "print(\"k = 15\")\n",
    "torch.manual_seed(0)\n",
    "k_output = model.generate(\n",
    "        input_ids,\n",
    "        top_k = 15,\n",
    "        max_new_tokens=100,\n",
    "        do_sample=True,\n",
    "    )\n",
    "print(tokenizer.decode(k_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c45c7de",
   "metadata": {},
   "source": [
    "**Our generations to the 3 top_k options:**\n",
    "\n",
    "---\n",
    "\n",
    "*k = 32128* (which is the model vocabulary size)\n",
    "\n",
    "wife of the defender says news reveals he does deserve more than worry. teammate's father says his sympathy has fallen beyond expectations. authorities have not said he was responsible for the crash. flight skipped planned refuelling stop in bolivia, official said.\n",
    "\n",
    "---\n",
    "\n",
    "*k = 500*\n",
    "\n",
    "officials say the defender's aircraft ran out of fuel during a convoy to Medellin. 71 died on the flight that missed planned refuelling stop at the airport. local media have released a video with it showing rescuers screaming for their colleagues.\n",
    "\n",
    "---\n",
    "\n",
    "*k = 15*\n",
    "\n",
    "officials say the defender's aircraft ran out of fuel during a crash. the air force has suspended the operating licence of the airline. the pilot was the first to fly the plane to be killed. five others survived the crash which killed 71 people.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b61af750",
   "metadata": {},
   "source": [
    "#### Questions on (2.3)\n",
    "\n",
    "1. How did the choice for *k* affect the generated summary? For example, what happens when you pick *k=vocab_size*? Your observation can be about coherence, factual accuracy, word choice, overall communication of the news story, format, etc. Re-read the complete article above. Is there any content **that is in the generation but not present in the article**? Why do you think these are generated, or more specifically, how does *k* affect this? \n",
    "\n",
    "2. If the problems persists for all the *k* options but to a different extent, what design component of the *top_k* algorithm could be the problem? How does *top_p* resolve this?\n",
    "\n",
    "3. Given your observation in the previous questions, if the task was something more creative, such as story generation, which *k* between the options above would you prefer?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d61c0e2d",
   "metadata": {},
   "source": [
    "#### Your Answers to (2.3)\n",
    "\n",
    "1. When picking *k=vocab_size*, the output does not have much sense. This is due to the fact that the tail of the distribution is big and therefore there is a high chance of picking a token that ahs low probability. And we can see that the token 'wife' was generated while we don't mention her in the article. Also in the second generation we speak about the rescuers which is not the most important thing of the article. However because we have a high k, the model can have diversity. \n",
    "\n",
    "2. We have at least two cases where top_k algorithm faces a problem. First is when the probability or a few sample is high but because k is bigger we also take a lot of other token as possible output even if they have a high probability. However, because the tail can be big, there is still a risk that low probability tokens are chosen. \n",
    "   The second case is when there is a lot of token with the same probability but we only take take the first k which will lead to a smaller diversity.\n",
    "   Top-p sampling solves that because it takes a number of tokens depending on their probability. In the first case, it would take only few tokens and in the second case more than k tokens.\n",
    "\n",
    "3. I would prefer taking k big that would lead to more diversity and risky outputs. However, we saw that taking a k as large as the length of the vocabulary leads to bad output. Therefore, we might find k large but still a lot smaller than the size of the vocabulary."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5a61197",
   "metadata": {
    "id": "c5a61197"
   },
   "source": [
    "### 2.4) Temperature for top-p\n",
    "\n",
    "Next, we investigate the effect of temperature on top-$p$ generations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Pj0RpwYmSizy",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Pj0RpwYmSizy",
    "outputId": "9085ec41-2e87-4871-abf2-5ab6aa38f6d3"
   },
   "outputs": [],
   "source": [
    "print(\"temperature=1.5\")\n",
    "torch.manual_seed(500)\n",
    "temp_output = model.generate(\n",
    "        input_ids,\n",
    "        top_p=0.85,\n",
    "        max_length=100,\n",
    "        top_k = 0,\n",
    "        do_sample=True,\n",
    "        temperature=1.5,\n",
    "        length_penalty=0.0\n",
    "    )\n",
    "print(tokenizer.decode(temp_output[0], skip_special_tokens=True))\n",
    "\n",
    "print(\"-\" * 50)\n",
    "print(\"temperature=0.8\")\n",
    "torch.manual_seed(500)\n",
    "temp_output = model.generate(\n",
    "        input_ids,\n",
    "        top_p=0.85,\n",
    "        top_k = 0,\n",
    "        max_length=100,\n",
    "        do_sample=True,\n",
    "        temperature=0.8,\n",
    "        length_penalty=0.0\n",
    "    )\n",
    "print(tokenizer.decode(temp_output[0], skip_special_tokens=True))\n",
    "\n",
    "print(\"-\" * 50)\n",
    "print(\"temperature=0.05\")\n",
    "torch.manual_seed(500)\n",
    "temp_output = model.generate(\n",
    "        input_ids,\n",
    "        top_p=0.85,\n",
    "        top_k = 0,\n",
    "        max_length=100,\n",
    "        do_sample=True,\n",
    "        temperature=0.05,\n",
    "        length_penalty=0.0\n",
    "    )\n",
    "print(tokenizer.decode(temp_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e3631a8",
   "metadata": {},
   "source": [
    "*temperature=1.5*\n",
    "\n",
    "arsenal and barcelona crash almost illegal in 140 states this weekend. authorities say air force die 134 vehicles, including 510 sparks blow-outs. helicopter makers after motor company could be disconnected and no power check was revealed. in formal earthquake surveillance sequence in surijer arena the plane led due undercover workers.\n",
    "\n",
    "---\n",
    "\n",
    "*temperature=0.8*\n",
    "\n",
    "\n",
    "officials say evidence growing that the plane ran out of fuel. the pilot had been warned by an official he might not have enough fuel. the pilot had the option to refuel in Bogota.\n",
    "\n",
    "---\n",
    "\n",
    "*temperature=0.05*\n",
    "\n",
    "71 people died in the crash at the airport in bolivia. the pilot had been warned that he might not have enough fuel. the pilot had the option to refuel in Bogota, it said.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5582743a",
   "metadata": {},
   "source": [
    "#### Questions on (2.4)\n",
    "\n",
    "1. How did the choice for *temperature* affect the generated summary? What do you observe for values less than 1 and higher than 1? Your observation can be about coherence, factual accuracy, word choice, overall communication of the news story, format, etc. Why do you think these are generated, or more specifically, how does the *temperature* choice affect this? \n",
    "\n",
    "2. If the temperature were to near infinity, what type of sampling behavior would we observe?\n",
    "\n",
    "3. If the temperature were to near 0, what type of sampling behavior would we observe?\n",
    "\n",
    "4. Given your observation in the previous questions, if the task was something more creative, such as story generation, which *temperature* between the options above would you prefer?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bdd7108",
   "metadata": {},
   "source": [
    "#### Your Answers to (2.4)\n",
    "\n",
    "1. Using point 2 and 3 we can analyse the outputs\n",
    "   We can see that when the temperature is high, 'Arsenal' and 'Barcelona' are generated while there is nothing about it in the article. This is due to the fact that a high temperature gives a flatter distribution and low probability output have a higher chance of being generated. Therefore the generated sentence is a bad one and do not summarize well the article.\n",
    "   For low temperature, the generated summary is really close to what we had with beam serach and what would have with greedy leeding to less diverse output. It is clear concise summary with the most important information given.\n",
    "\n",
    "\n",
    "2. A temperature that tends to infinity will set all the logits to 0. When we pass this to the exponential, every token will have a score of $1/size\\_vocab$ and lead to a final flat distribution that gives equal weight to all token and therefore leads to random outputs.\n",
    "\n",
    "\n",
    "3. A temperature that tends to 0 willmake the logits higher and by passing them to the exponential will give weight only to the most important token and therefore lead to greedy decoding, because we will sample from a distribution where the probability of one token is 1 and the other are 0. \n",
    "\n",
    "\n",
    "4. If we want creativity, we do not want our temperature going to 0 because this will lead to generic outputs that can be boring. On the other hand a too big temperature will lead to something random and not in adequation with the input of the story. Therefore we would like a temperature above 1 that will lead to more diversity and risky output but not too height otherwise the output will have no sense."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04d68032",
   "metadata": {},
   "source": [
    "## **PART 3: Reflection on Automatic NLG Evaluation Metrics**\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae462b8",
   "metadata": {},
   "source": [
    "#### ⚠️ **Note: There is no coding in this part. Instead, the outputs are pre-generated and the automatic evaluation metrics are already run.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdad6a6c",
   "metadata": {},
   "source": [
    "### 3.1) Description\n",
    "\n",
    "#### Flan-T5 and the datasets\n",
    "In this part, you will answer questions on automatic NLG metrics given a set of results on a finetuned version of T5. This model is called Flan-T5 and was released as a part of the [Scaling Instruction-Finetuned Language Models](https://arxiv.org/pdf/2210.11416.pdf) paper, which you do not have to read to complete this portion of the assignment. In short, it was finetuned on a mixture of tasks to generalize the model to unseen task prefixes (e.g., task prefixes other than \"summarize: \" or \"translate from English to French: \").\n",
    "\n",
    "Therefore, we can imagine using Flan-T5 without finetuning on a story generation task. To evaluate the small version of Flan-T5 on these tasks, we use a small portion of the validations set of the [ROCStories dataset](https://aclanthology.org/N16-1098/), a dataset of 5-sentence short stories. We also evaluate small Flan-T5 on the [XSum dataset](https://aclanthology.org/D18-1206/), which you've read about in the first and second parts of the assignment.\n",
    "\n",
    "We use the following prefixes/instructions to describe the task and the input to the model:\n",
    "- **Summarization with XSum:** For this dataset, we do not change our prefix \"summarize: \" that precedes the article that is input to the model's `generate` function (or in other words, as encoder inputs)\n",
    "- **Story Generation with ROCStories:** For this dataset, we give the prefix \"Write the story ending: \" and pass the first four sentences to the `generate` function. We expect it to output a single sentence that will conclude the story."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea9c81e9",
   "metadata": {},
   "source": [
    "#### NLG algorithms\n",
    "\n",
    "Once we have our dataset, we use three different generation algorithms to generate a maximum of 128 new tokens:\n",
    "1. **greedy:** The greedy decoding algorithm is greedily decoding the summary/story ending\n",
    "2. **beam:** The beam search algorithm uses *num_beams* of 4 and a *no_repeat_ngram_size* penalty of 2 (these variables should be familiar to you from Week 7's exercise).\n",
    "3. **top-$p$ sampling:** The top-$p$ sampling algorithm uses a *top_p* value of 0.9 with no *top_k* and no *temperature*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8705d3d5",
   "metadata": {},
   "source": [
    "#### Automatic Evaluation Metrics\n",
    "\n",
    "After generating the sentences, we evaluate them compared to the gold references with the following metrics that can range from 0 to 1 (which are multiplied by 100 for readability):\n",
    "\n",
    "**BLEU scores (BiLingual Evaluation Understudy):** The overall BLEU score measures the similarity between the gold reference and the generated candidate. 100 means they are a perfect match, and 0 means none of the n-gram precisions get fulfilled. \n",
    "\n",
    "- **bleu:** As you learned in the exercise, the BLEU score is calculated by comparing the $n$-grams and combining their average precisions from $n$=1 to $n$=4 with a brevity penalty.\n",
    "- **bleu-1:** Geometric mean of 1-gram precisions; it does not use the brevity penalty.\n",
    "- **bleu-4:** Geometric mean of 4-gram precisions; it does not use the brevity penalty.\n",
    "- **bleu-brevity-penalty:** The brevity penalty penalizes generated sentences that are *too short* compared to the closest reference length with exponential decay.\n",
    "\n",
    "**ROUGE scores (Recall-Oriented Understudy for Gisting Evaluation):** Similar to BLEU scores, the ROUGE scores look at the $n$-gram similarity of gold references and the generated candidate. However, while the BLEU score is calculated with *precision*, the ROUGE score focuses on $n$-gram *recall*, which we can interpret as the percentage of \"necessary/relevant\" information in the reference that is also present in the generated candidate. \n",
    "- **rouge-1:** ROUGE-1 is calculated as the overlap of unigrams between the generated candidate and gold reference.\n",
    "- **rouge-2:** ROUGE-2 is the same as ROUGE-1 but with bigrams.\n",
    "- **rouge-L:** ROUGE-L looks at the longest common subsequence (LCS) overlap between the generated candidate and gold reference. It's preferred over ROUGE-1 for text summarization (where the summary can be lengthy) as it captures large-span text similarity.\n",
    "\n",
    "**BERTScore:** As described in the [its paper](https://openreview.net/pdf?id=SkeHuCVFDr), BERTScore computes a similarity score for tokens in the candidate and tokens in the reference, where tokens are tokenized using a BERT-family model. However, instead of exact matches, they compute a soft token cosine similarity value using the respective BERT-family model's contextual embeddings. In our case, we use the huggingface default Roberta Large model to calculate these similarities. The soft-matching only counts the similarity of the most similar token in the reference or candidate, depending on whether it's a precision or recall measurement.\n",
    "- **bertscore-precision:**  Matches each token in the tokenized reference $\\tilde{x}$ to a token in the tokenized candidate $x$ to compute precision. Then divides by the length of the reference.\n",
    "- **bertscore-recall:** Matches each token in the tokenized candidate $x$ to a token in the tokenized reference $\\tilde{x}$ to compute recall. Then divides by the length of the candidate.\n",
    "- **bertscore-f1:** Calculates an F-1 score given the two prior measurements."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a22a6ef4",
   "metadata": {},
   "source": [
    "### 3.2) Task\n",
    "\n",
    "Now that you know what each task, generation style, and automatic evaluation metric is, we would like you to analyze the following results table and the generated instances in the file `part3_flan_t5_generations.json` to answer the follow-up questions below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7ff725c",
   "metadata": {},
   "source": [
    "![part3_flan_T5_automatic_eval.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABiMAAAJSCAIAAAAu7yEAAAAABGdBTUEAALGPC/xhBQAAAWFpQ0NQSUNDIFByb2ZpbGUAACiRdZBPSwJhEMafbQ3BPER/oEOEtxAsxJWigwfTkAUPixpWt3Xd1kDtZXcjOgSdOnWsPkJ4igjKS4fqEyRCQaduHToGdijZ5tVqtWiG4fnxMPO+wwADfpWxsgdApWqbmdRiYGV1LeB9hogpjGAeo6pmsbiipKkF39ofrXsIXJsz/K3aeWO66bPk7EHqpbF/e/m3vy98Rd3SSD+oohozbUAIEyvbNuO8Rzxm0lLEh5yNLtc4F7p81enJZRLEd8TDWkktEj8Rhwo9vtHDlfKW9rUD396vV5ezpBNUk0hiCWnKALKQEEWEMon8PzPRzkwCm2DYgYkNGCjBpuk4OQxl6MQyqtAwixBxBGEqid/69w1dbzcGLFzQV6eup4jAWY7Wn3O94BEw/gbcyEw11Z/LCi2PtS5FujxUBwaPHec1D3iDQPvBcd7rjtM+AcRH4Lr1CbZVZHr+uekfAAAAIGNIUk0AAHomAACAhAAA+gAAAIDoAAB1MAAA6mAAADqYAAAXcJy6UTwAAAAJcEhZcwAAHsMAAB7DAbyXl9oAACAASURBVHic7N3fsy3HQR/6JV9Bsg+JbGO5UpTRQZF8BYVVZ/unJGNhdI5CFbnJi/8BUlRSLvEA+jduHg0PqKikAL/f66pwEyjAki+WsRVbls+JLtgWdjlHNoRYBtu4zk4KUvs+jDwsZqZ7dc/0/P586lRpaVZPd6+1Zq+Z/d3dPXddXl4eAAAAAGCwN8zdAQAAAAA2QtIEAAAAQBmSJgAAAADKkDQBAAAAUIakCQAAAIAyJE0AAAAAlCFpAgAAAKAMSRMAAAAAZUiaAAAAAChD0gQAAABAGZImAAAAAMqQNAEAAABQhqQJAAAAgDIkTQAAAACUIWkCAAAAoIy75+7AKt1/9erx/37t9u2xmxu7iRl1vrr6HR71hR9/jicbinSpbG8bR1dtM8dA7+N5mqNi1UIHT6fG27j8tzfr1R07fkWJlfR+E6r6F/seZgkdEu3tW3rVqxP6Rl3+T3Rv7Z/iGV/jwt//ftc58cKd36LbO8yWoN9RlH6uzP3Uyn70w2vrXcPEv1gt5NugnzE6v+pfbFfd+elJmhat929Wq7DtV5fr5Luxgd/lfOJs2wZ+SPv52u3b91+96vJrYvv8Rl1OzLSK97/62awex39CU2KmyEve7bffqqV/aic/+sR6Umor0p8exzATWPX7v+rOz8XsOTgc5v76SG/d1xwsk5/NgzeBkTnAeshNf4b8iu4DWqPEv3QOrye9uXgZhyKshTFN2Y7D++qxP+QWt6v3M2Uy5nGZ3R5v+3zV/WzyvTr5+8+QObClbPKdT3E8dIIl2PahuO1XN56Tc/0SRZYR2O0lygLFP4h+wVDkoz8kfPrxoXM9TiKrOBQX0g2YnjFN8LpZfk1qnHRDZ6PGU36jg0Uxc+Tw/Zfv2wmWJn79kDtvrrOMS5TVSfnIUq5R0y9QixxFDkVYEUkTNNe1nbcPA8sAE3Mt2+ANgaUpdY0RKeMSZXWGjD/Kqiq9tvScyKEIy2f2XJ7G5IvECXQnp2yEbuWTWEP6vfDiDYVuMHSy2s7y6R2O9Cp9vYA1nlemuRvFqB964i4px3NjY7uTxY+K4kdR75/E5R/PuXd4Wd0LHFv6O7CQb+kZdb6o3B//xJ/93sdnqebiH2io8h7fqJFexYt11jb9l236RdFIH02/zvTuWI/yJ3Vess51HZJYYOwLmJMdTqyw7FfK0r6TE/uT+DtRSmg1xp8rcn9si1zRJZ6/spoL7ZK4V2KF6VUVPG1lVZjSkyl/ixnShw2TNK1b509j4jyO0Ajq0GkyUu3JsbIFf8wi42mHtJJ4gpzdwBNwwQ+9s/zJXXr3s1/50KcZH5Xdo/NlfxIPSzoOh/90L/wFlpL1Y3JIe/dW9y1d/AvqkHCoDPwiym10jOYi27MqTzTlO1b8y7ZHxxIbGuOX20jN4x1jiV0a4xt4mriqsaXfBUx6K7kVDv+4x/jpWJ2BL794zjjk0y/e3AQXsf0uUQb2pHeF/d6Qst+0U57sVuQNc3eA/uJfZ+M9O6TwQAW7Pbvevf1aeDmnIS2WCnf61RbZd+B5PXT67F3h8A4kPruE43l4Hxb7Au+/erXxr3c9vZ+NF1jpt3TvFvu9FRO//2Wb6xymMbYp37HiX7a9q+r90Qw38TEWd3LE0ElzfW9Pc/k3/Slv3lNkSutZPwuh77Tc2maMhgte0Q1/FT2+IlIKpJef8vgc6TIsVGDK34l6VLgZxjRl6PyiHGkszMnxlseHbOjS4WSXOkfadzYaqjPSjche6aNJO2tL716WuYY1TZxzD//QD/mHX49PfKSjot8Rm9iBsj+JWd0YQ8H3agkv8Pjwi1x2jPQbZu83cK5v6YmFOpl+Ldj7py/SYvo72bu5eLWhmvudQ+NtFX/Hyh548Zc80kfTrzMna57gGIv3pL7U6Wwovle94yxfF0UuYEKmP+X1+94rJf56F6v3oZjyY1v8ii6uR3OTXcT2PvmGutH7OzxU4ZAdy/4Ws4qrrLkY07R67aO2318Msv43pRvpPcn1tdagnlINdV4ib0nZD72zTJHPokcluUdF8SN2pJ/Efu5vDdi5f8DgnYHv1Ug/rb2lXz+lG+lgW+O39MAK2y9q7C+iib/3ElPFHjVnmfIdW+nrKmg559Yh2r9c9T6nFOnA8AuYxIZyKxz+cSd+751U9kpgOSY4FEtd0Y3X3NgXsb1PvmW7kV5hvx3LftNOebJbC0lTqkgKPn08kd5QqGSpPL76Jprgp+jkS958SDRcqQ/95B9CB34WPXpysqr0nmQd0iP9JC7ktFTkp3uZL7B+acf/6meHf6y53Qg9G9+3SCvjGfttLPJFlH58jtrcQk6jKSUL/kSXeuHjva4iJj7GcruU9cJDhScIL0p9H8YrmfKUt5yL1dnPyD2McSiO/btVqJKyP+8jXcQOaWWWCiM1T/ZNG+nDfpg9t26RH5LZz2EjdSDyg1rqUqMeWD77l0L8PZy9e8vR46goO/5/yT+JY1jji0r509ZIr2vGySZxS/sc9/ZzdNLEp9GR3ufJjn/HT6LIfK6T4r+DLfa7bqBZQrQJjttVf1LjHYqr+CZZ7EVs8bcoscIlfPksoQ9LI2mimOV8/zKNNX7ijbNm4yVs5vRQ/IWs8bPuoWzQvMCDbZOf48QvatTmtvQBLfD4X4tpjrGBX3eRcaBL+EPdQNv4SWx/CidHc6zR2g/F3CFLS/tenStaqgx5QwoOWVrah7IckqYkkQOoXXKHh9Q2Tsm1RQ1rorjIn2g2eRE20BImBa/341jOwbaxb+nt2eQHtJzjn1FVH+I2juFtvIqQzV/fbulQDFnO9+q8GVNtCW/IEvqwTNZpYqit/n5YPZj3dPW16Goy9Hbyndz2ZUq6Tf50T2wJB5vPceE2/AEt4finFvpje5FPITK0ZC02/JPYtvDPaEj3NnAoxi3he7X4D8uQCpfwhiyhDwtkTBODHP/YbO80PKoN/GVppd2unZzkv/YXOJCf7oJmPNg2/zlO/KJGnZq6yQ/o4Ms200jvRvxIi3wK6X+TX9RCNrn28JN4+PufUeRD73eNGjpUhtTWGRtt6VDsfbBt6bqiSIW5b8gY74+TXYOk6bT077W1BwdDLOEllx2guIEkiEPaUbH5P38NNM3xP3YrCxnAPOPB5nts4fbwAfmyXYLjT2FFv5BPZvM/icv/0BfevaXZ0nVFqd/g6sdzHUtL6MMSmD23bicnhW7Pll7aQubo9bacbqf35P5Td8AtO+h3OW8R0xvpYNuwfj9HE/+UbeCHeprvq+mP/818D4/R4UjUnn4psrp3chXmPW5TfkITe1K2WKMPQyrJNfEnklWti9iGIW9IqVfnYi9C0jSp0FC63HrSD9l5D+6C31CJA8pSSvZodN4Jzz2KjSrlHT75tTtNTyJHRfFlKUqVXIslHIpZUn6cRxr3tOT3agl96/duT/xFNNf33pSn0R4lU8x+IuhRchbjHWNZf4yJd2xg7FvqMnh6xTu5rtG1WZ9s50srchQVPBTjlZcqObCSyM/7xBexWYr/sBQ5RGtL+KbdD0nTCVm/eGT9JnOyZKJ2DbMc6O3XNV432pWP19bYlwKNs2bkjDJBcNND52cx0q/rPXrSLnbyMqX3m7yQn8QxTPnTPY3iof/EB1s/03yOA3/8+/0cTfxFNFJzE59GT24ZaJbjfzPfwwWPsZMRwCEzd0gZ/pA+jmCBH9D0p7wpL2trWb/XHMLvQ2N7+koFnT+tiVXV5U9un/JM1Fv6z/uoF7E9DsLiPyy5FQ58Q4p80y7tYm9RrNM0hcaM6MYBlzJfun3QN5b0C7Xbr8PpUrpxUuKPdPxtTKlhbCffgdDfeXK/3Gf/w2zKZxFR8DfAgUdFpOfpJ5gl/CSOochP96IkvqLp/8o36hGyis+xXycHfhHlGqm54R9Q+jfqjN9Xox7/834PT3xGG1J5etNDOtY5QW/gZfAEpv+qnPgbLOL+hKXB65KRenJ/BHp8LZTqTyjHmfKbZPgBUPYiNr0PxU9bpX76Et+QUX+Lmetib5mMaZrIGLMDhjw7jeI9nOUlTxPYrW78f8qvNLm7jNGTft3I6uryfxLHsNLX1eO4HbW54i3mWs7nmPiX8PSnEgtkmbi5eIXTX1eUbatsi9N/D891ydF7QNPAvXr8eb/H7os1/cc9zcXnxMUSL3eL/DiH/rg7QdPpRrqWLnURW/wsM9dpq8fbmLXL8i/25mJMU0yPv1PVKWn77wPtP9okHpcnw9Ee1RbU2Yd6e+QNOZx6dT2aW7XQSzt+dlF6fBYj/ekytyeRt3rIWXDen8QxDPnpXqyy3yEnD+niB1sP03+OU/4cTXxSGKO53h9Qv2/UKb+vpjz+p/8eXsgZLeS4htwLztA1W7vm9Cb6XQZPaZZTXrvRWd6W+IuKfPQ9ejv8a2HIodjvrD3eh9LvWrpzl/heWX0YeObtcdoa+Otk577xF1L2t5glXOwt0F2Xl5dz9wEA1qrg3Jm1G++t8CYDAKyIMU0AwGykSAAAG/OGuTsAACsWH8a/HwY0AQBQkTQBwCASkNqQt+L+wH2vh/UIAICpWacJAIba+bibgS8/JU7a5xsLALBGxjQBwFBykMOAN6HH/YYBAFgsY5oAoIxqbM7eYpGCr7rILbQBAJiXpAkAAACAMsyeAwAAAKAMSRMAAAAAZUiaAAAAAChD0gQAAABAGXenF33llVfG6wds1bVr53N3AQAAAPq7uLiTXtiYJgAAAADKyBjTVPn67VfH6AcAAAAAa5edNB0Oh+tP3ijeD9i8m7duzt0FANia8+/PUneeBYDiznutBmP2HAAAAABlSJoAAAAAKEPSBAAAAEAZkiYAAAAAypA0AQAAAFCGpAkAAACAMiRNAAAAAJQhaQIAAACgDEkTAAAAAGXcPXcHAOC082vn1YObt24OKVk/e+xknQAwi87TVlviiaxdW/oZcMgJNGvf82vno56Xq8449cOoJE0ArMDNWzerS8P4BWgkZopcrLvoBGDVEk+OkaeG7F5w38RkDVg4SRMA63AybEqJmSJPjf1HVADoJ356Sjw5dtZz8iSYsvuQ8/L0J1/nepiAdZoAWI2Uq8OsmKna6KITgPWqz2LtAUHHZ8CTJ8F+u6fsG28X2B5JEwBrUl2Yhq5oI5et8SvaULUAsHy5Q4pCux+fBxN3j4RNifs6+cL2mD0HwCodj7cf4yI1Hl21nz3e0uhPu1jnUwBQXOKJpp6i3mP30L5ZEkOulNNofUbunPoXOb+nn6OdzSFO0gTAyjQWbEr8o+vJlSBKXSZ2jrcKXYVbHAqABcpNjnrfSG7gze/SY6NSled2A/ZJ0gTA+rSDm/i8uTqZipccLjTWqTMOS7mbHgCc1GNSeVt1ujw+Kw05PeWefI8HBacvuRg/kyb2P73yft2AHbJOEwCrFL8kjaguo6t/43Ws838jTwFAP+nrMc2oyMm3/QILvuT0ykftBmyDMU0AbF9k8lpdoFRDPZ4CgIiUdGaBZ5lSJ9+Ul98eT5RVeSRUaozzClng+w8zkjQBsHq5l4BFZhkAwBIs+eRV8OQbD4P6drAP1wxwkqQJgFXqvLNMoviFLwAsSvq6RUu25JNvYn+Orzo6b2wHVKzTBMD6HP858XjBph5u3rrZXukTAJZs7dHGek++ofvcjbf+I6yRMU0ArEzoSq5zDl36LW9cIAKwIvElhNIXGKo1/oqTtXvjbLu6k2/uTfqqBwvpPCyQMU0ArEnnfAHXfADs1uznvtk7MJdqZNZ6x2fBeCRNAKxP7g2GXfkBsD0pAUfiGTAyXji3Mz32TTG8hz0qbxcLlVz7fEYoS9IEwGqcvBXxoXWxmPhnxqz7yMitAFiIyJkrfaDNkPHCnc+WPfmmnJ17Bz0nd2ynSy4D4CRJEwDrkH5hlxs2Ra6SQ7dkBoCFiNwZ4/gM2FngeHtkvHDxfSPbI0Y9KXcmSo0sLPKisv5kBZtnRXAAViDxds712qKNRUyPt0f2jTfdrg0AlqNzAe/jc1aPk+CQE2jvfdsn/XhVA/Od+FvUqDzlRQGSJgCWLjFmqsuEwqZD5hVqfDoeACxE/E8gkZE46RPT+kU8uftGXsioJ+VQP9M76fIAjt11eXmZWPSVV145HA5fv/3q9SdvjNkl2JSzsyvVA6cfACguK4kGALLU59mLizvpe1mnCQAAAIAyJE0AAAAAlCFpAgAAAKAMSRMAAAAAZUiaAAAAAChD0gQAAABAGZImAAAAAMqQNAEAAABQhqQJAAAAgDLuury8TCz6yiuvHA6Hr99+dcz+wNb8H//yX87dBQAAAOjv4uJOemFjmgAAAAAoQ9IEAAAAQBl399jn+pM3ivcDNu/mrZtzd4EdOb92Xj1w4DElBx7Tc9QxCwces3DgMb36qMtiTBMAAAAAZUiaAAAAAChD0gQAAABAGZImAAAAAMqQNAEAAABQhqQJAAAAgDIkTQAAAACUIWkCAAAAoAxJEwAAAABlSJoAAAAAKEPSBOzF+bXz82vnc/eCAnyUzM4RCG2+nDfDR8nsHIFrJ2kCAMjg8hcAxuM8uwGSJgAAAADKkDQBAAAAUIakCQAAAIAy7p67A8DKNCZO37x1s95YPT7+3+PC9bORenLb7V240eFQDSd7xezKHhihkqHyx8dJpPLco53FOv4o68ehA6PW+YknHjzpvSpSFUvgPMuizHuePTjV7ozz7GZImoBUndcEkRX7Qk9F6omcJxLLhwpnnQZc/q5F++POPTDaheOHdOio6OxJ45fAlHpYu9zvt85den8FFayKWTjPsjTLOc+GOuNUuzfOs2tx1+XlZWLRV1555XA4fP32q9efvDFml2BTzs6uVA/W/gUU+qtC519TU/4EEapnyPYenTz5N5CVinwEG5DyN/ysA2N44cj2xPq3YduvrnbyF60hB0+knnh/ilS1Rpt5jc6z67KZA6/Tcs6zkc7s81S74Zd2zHl2UerXeHFxJ30v6zQBGdrfof2+VXPrySpfqpMsX/EDo76yGXiIHv9v5Ck2adTvt7JNs0DOsyzKcs6z7QJOtbvlPLsKkibgtMg450P4GzZyeREp356enVi+XyfbNvCH1p2If0aNAyPxKOq89h3SEwfSfsS/gkJlEg/jFAWrYnrOsyzQcs6z8c44lnbCeXZdJE1AqlWcyNM72b7gZhuKH6jn186rf2WrZZPSE4F+9fSwiq9uKqv4sJxncZ5lRs6za2FFcGBquVcSU155+EPrVi35qAMoa8nfeM6zW7Xkow6YnqQJAP5O57Wvv8wzr8hhCbAuzrMskPNscZImYGozjm5t11yN1nYi2bzlHHUAY1vON57z7H4s56gDlkDSBPA6Q/pxDLBMjkm2wXcsjgGWyTFZnBXBgVSlhjSPOpPfuGs6jwEHBtMIHWn9vveKXPj6vW5FnGdZBedZZuQ8uxaSJuC0Ujf1PPkV3LjzSFb5fp2s1wVwhbQ6iffbzj3q+jUHKZeYoXvSF+EQXTXnWRbIeZZFcZ5dF0kTkCS0UmO/79z2NUe9pfMsklg+dDlb8BqdRRnjKIpUCylyvydD5Xt8HRWsiuk5z7JAzrMskPPsKlinCcgz8FKgWhw0VFX7azq3fKifx/WwDfVCs+3tnSWrx/HydcnOYg4hjrUHekSOn0Pg+yr9MD6pYFXMy3mWhXCeZV7Os6tmTBOQ6uatm53XFj3uNhI6EwwvH+pkvD8pxVig9M86/SiKFHMPZiqR74oe35O5X1lZHfO1ti7OsyyN8yyzcJ7dgLsuLy8Ti77yyiuHw+Hrt1+9/uSNMbsEm3J2dqV6sO2vobUPHF17/9sSl9WAshx46Qp+7WzvGyzLTo66tX/Ka+9/204OPJbGgZfOebaU+qi7uLiTvpcxTcBp5s8DwHicZwHYEkkTMNTaY/619x+AbVv7eWrt/Qcgl6QJOC10r5mDufQAMJjzLABb4t5zQJ70ez0sn8t3AJbGeRaAtZM0AUlCdwNZ6bVvwzZeBQDr5TwLwGZImoAMG7tS3NjLAVak4PePr7It2dinubGXA6yI8+y8rNMEAAAAQBmSJgAAAADKkDQBAAAAUMZdl5eXiUXPzq6M2hUAAAAAlubi4k56YWOaAAAAAChD0gQAAABAGXf32MdN/iDd+bXz6sFHn789b0/YlS9+/pPVg6sPv3/enrArb/izl6oH73zsg/P2hP145O1vrR688JXX5u0Ju/KJZz9ePXjoXY/P2xN25R/8t9fPs9efvDFvT9iPfssoGdMEAAAAQBmSJgAAAADKkDQBAAAAUIakCQAAAIAyJE0AAAAAlCFpAgAAAKAMSRMAAAAAZUiaAAAAAChD0gQAAABAGZImAAAAAMq4e+4OQE/n184Ph8PNWzeLF4aIn3/8amPLR5+/3XvfTvEKq0rSG2XDnrrxwOFweObZr+buUgvt2yjWkNUiq/bI29/a2PKf//Sb09dT7d6vadbo0QfvbWx54SuvTVBJY5cejbJeH3rv2xpbPva5b4xdT6lGWa+zsyvH/3txcWdgDQ2dFbZ36dHuScY0AaTqjIoS8yMoKx4GdZZv79K5EQ6HwyNvf2s7Hjp0ZUZj15PbIqv26IP3thOiQ1dsVLaSzl2yGmXV2olPaGO8kvR6sgqzSWdnV9qJT+fGsRs9nIqr+jGmCSBJnSgdjyeqNv7841dTBhmdLBOvR6RFrUfMVD1oDEeqtj9144Hj7T2GSrExdbjTGEZUbX/k7W9NHF40vB4x067UyU5jMFG1/dEH700ZZNSjks5dshpl1epw53g8UbXxQ+99W+Igo85KQvVkFWaT6mSnMZio2n52diVxkFFVPqvw8EYTGdMEcFpnzFT9b7VleAwkZiJR71FI7fBInEREOwaqt2QFQL3rETPtUzvZqbekDzJKrySUTPVolDUKhT4f+9w3qi1Zg4zaCVG9JWWiXKQwm9ROdsaYxZbeaNmRTZImgFTjrY4UCZJ+/vGrYibahESMJL4oUvpiSaXqsTzTTlSBTmgAUeLAot6VdD5lNNN+DBxDVAVDoUo6By4lFoZS4qOfxki4zJ5jC6oFv2u5K3+n7B5fU9yK45s36grckUW+jzOmUoOnWLXeq4CHdnnm2a9WqzWJrlgUq4CzBMKmzZPsML2ToU+1mtIE45tGJWli9Ro50SEn92nvm7U7lBJPstxpjspkKyhVTSTeqI6NKRXuDKlHzLRDRTKd3EriY6AgUVZcJduirCqQOnkPu+lzK0kT69aZClUbz6+dx9OiOmbqtzscwus39aihk4yJ2jQ3iavXCA89JW9ibJZnYi7HSzKJn3YutH4TLEe9knfoqXlHRUmaWL12HnTz1s3OwUpj7M4+NeKhgTGTOImTQjePG1XKjerYm1KjjUL1hG5Xx84VGXwUqaS97LfhTvvUWIq7SMwUX5hpSGGoTXkPu0RWBGfdEkcthZ4K7V5tlzeRyNpJTGOyiOeZZ7/qRnW0lRptdLIeMRPHitz9LVJJHSod/yvYNOs1/B5wWTW45Rw9XFzcGThXruwt52qSJrbJxDdG9dHnb9f/qi09wiYDmkg05bS1zozp+NnDVPP4WKzxFnKyPBMRoy7k1N5uNNM+fexz36j/VVuKpD8WcmIMnRnT8bOHnBSp+FQ7s+fYNaOWGO6jz9+uMqOff/yq2IjirI7EQkwzb07MRMPY8+bi2x998N5HH7xX6rRPH/vcN6qY6UPvfVu/9Me8OZZvvBWdjGkCGErAxEiMHmIhJlueCY5NEDNBxJDcR8zE8o26cLgxTeyaSXbMxdQ5skQiJ4OeGNvYMVO7zKjdYC3ETKyXmInlG/v+dJImgNMEQ+yE3Ipjpe4E545yZKnX4R6SEBWphM0rnvLU6zql1JlVGBp6R0X14k3jxUwHSRNblbgA0/m1c8OagMWKJz6JqdAzz371qRsPPHXjgc6SoiVCpoyZUgY6Sal2YsqYyUpMlCVmIsXFxZ2zsytnZ1c6g56xhxpNEzMdrNPE2sUTpUiKdDJgOr92nh5XpRRj1arRTKEbzOWOeDJCisWK311OLLUfdbhTJGYaXg87UU92Gx4zDazkYObdPlQpT+gGc/0mwaXHTImFoVP87nKdiVW9ceyY6SBpYgMaQU+dECUOVmonSp01VI/boZKYaW/aYVMofoKlaedHJ5d/am8RM+3BlAszQW2WhZmqkvUYqEY97Ec7bArFT5HdLcxElnZIFIqN+tUTipl6NNGD2XOs281bNzsHH6XETNW+1eP0GjpLypv24KPP365Cpc5oqT1AycAlphcKg6oJdIdAtNQoHy/M5tWT3eL3gzvOjzoTpR71sGd1shOPeI4jpHao1KOSw9EcupMl2aSPfe4bVfTTGS018qDOkKjeMR5ONcZPpRRmq6oJdIdAtNQOg0IhUbyedg0nSxbMoSRNrF476Elfeik0Uqmzhs7Clnnalc45dLIkVqFzWlxogFJWYYC1q8Kmxpa5OsP0OufQyXoYVefctx5BT6l6irvr8vIysWjde79aQ7o6mZJHMKUvfv6T1YOrD79/3p6wK2/4s5eqB+987IPz9oT9qMdMiQaY0iee/Xj14KF3PT5vT9iVf/DfXj/PXn/yxrw9YT/6LSJunSYAAAAAypA0AQAAAFCGpAkAAACAMiRNAAAAAJQhaQIAAACgDEkTAAAAAGVImgAAAAAoQ9IEAAAAQBmSJgAAAADKuLvHPi+9+GLxfsDmfeXWp+buAnt0++VPz90FduT+H/6H1YMvfOYP5+0JO/TK5xx1TOcf3H1X9eC//hcXeEznobdeqR489/Fn5+0JxBnTBAAAAEAZkiYAAAAAyugze+5d73lP8X7A5j147QNzd4Ed+eLnP1k9uPrw++ftCfvyZy9V/z1//8/M2xF26JHHn5i7C+zIf/q9P6gevON9Pz1vT9iVv/na6+vYXH/yxrw9gThjmgAAAAAoQ9IEAAAAQBmSJgAAAADKkDQBAAAAUIak6DVRYgAAIABJREFUCQAAAIAyJE0AAAAAlCFpAgAAAKAMSRMAAAAAZUiaAAAAAChD0gQAAABAGXfP3QH25fza+eFwuHnr5kjlp7TkvpHl5x+/2tjy0edvp++YWDjeaKSSdvcSd2TJnrrxQGPLM89+tWD59Ko662k3V6RpluDRB+9tbHnhK6/l7t5jl/QWG+Wz2mKxHnrbDze2fPkbf5m4V0rJUCud2hX26x4L97MP/0hjy++//Ocny3Rq79i7xfguiQ2xCmdnVw6Hw8XFncizIaG94m0l7t5uOre5FMY0AbvWmePEw53hLbbr79zIVnXmOJFwJ7d8pJ72Xj3qYaUeffDeduhz6EqCIjX0aDS9ns4ehrrNWjz0th/uDIASU6EJLLx79PCzD/9IZ4SUmCv1bjSrxc5OjtpDNuns7Eo7OercGN9evGPGNEG2ajQTG1CHO8cjg6qNP//41d7jjHJbjDQ6ZNgUy1QnO8cjg6qNT914oD1cKLf88HarjQYubUwd1jSGCFXbH33w3pNDh4bETMeVh1oc3kMWqM5rGkOEqu0Pve2HQ0OH+gU9JwcitVvs7OHJ7rFkdVjTGB9Ubf/Zh3/keHvKmKOUcUadjXa22K8863IyuIkPd+qhUVU8TuosfHZ2pezIJmOaII+YaTNCoc9Hn79dbQnFScMHH7WTI1nSTnTGPdX/VlsaI4xyy/dot1GAbWuHNfWWeJBUKmaq/rfa0llnpIesVzuvqbdMOZ4oMWaq/rfaYmTTerXDmnpL+rihITFT9b/VlkaLkfK5PWSBxhgfFG+rnRBVWzp7EiocKt+bpAlSnV87FzNtT1bEM3COW3yAkrBpP3KHC5UaXtRZj7FLexBfXCk9yukR+iTuUqqHLEp8laXO7aGpdqU6k94TVqrKaELxUNZYody4J3cgUmd5o5lWLTQ3bQnio6jGWKfJ7Dlm00htcpfWTtk9vmh31pLex81Vu0id1i432TnOmOKDnoaIz9pj7ebKmJbQCuvVYxXwHuVTuiFy2rbjMKjskKJQ5iVjIiSeWDWUjYeETSt1nDFFhhTth6SJebRjmvTcpzPimeZOcO40x/QxUGeqJY0iXY91lzqn5gmk1mtIQNMvZoJD3xxnpPRHqLQTBWOakRKfrAyL1UkfHNSZRo0xtmjsmkMkTcygMxWqNp5fO4+nOXXM1G/33mRM+xFZv2mWnkTujidv2ozQOkqlynfu27l7vUZ4aEd5066McdO30PpNMFIY1GNgVGj9Jnai1DJJofWYQm2Jn1YtPcqpV+AOPZUVV1VT9joX+Z4+XTomaWIe7eDm5q2b6fPRBu4ObQsfNJR4rzrWJXfQ0MBBRu38KB4eNbb3uOEdqxAatVQ2EmqEVmIm4qs4zdJKI5MSM21Pynii4WOOGkFVqKp2nmW40w4Nvw1cHTbFa44bI5myIjgzSBy1FHoqtLvlkyhojDWY+t3Srr4XXrsqNib3BnBZ5etQ6fhfqJ7jZ483ZnWPVTg5ammkSKgzeAp1ZoyhVcxrFfd0W0UnSTfXDd06261DpeN/kfJsz8XFncht4NKFVoNKXyVqpPWkJE0siBlqzKgKdI5jnTHCplDNnW11ZkzHz4Z2ZC1SQp8h5du7p2yJJEqd6zexAaEBTQVjphe+8lr977iJznbjW9iMRQ1oqkrW/45rYEvGHtB0+Pv50XG1J3tiNNNOdGZMx88ektOfulhVZ/2v8Wxir9ILp5A0sT7n185D/+buGhsxXth0XPPxv4NhSruXGx71Lt+5XXi0T/F5c+NNcOsMm443Hv8btSfMYpp5cwMJmzZmrolpobAp1JNqu2FNZBkyNmq8FZ2s0wSv6wyqDLParY8+f3uk4ULtsUgyJirPPPvV3DFKEiJ6O7k806he+Mpr7Ybac+hkTNuzipip8uVv/KWYaRvmXf/o91/+c8kRY4iHRKHFwtNrGEjSxPpIf9gA6RIwo5SxQpHIadShRtKlDZsyZlpRpMWo0mMmC3KzK2Pfn07SBK8TYO3N0uastfuztB4yXPxeb8PLDzd9i0ys7B3lTjZU8L51EqhVqwcHLSr6kUZtWz2SaOzwSERFlrEjnsQOjN0HSRMLkrjQ0vm1c6kQayQ5YhbVJLunbjwgPyIlZkoZ6FQ89JElbdsyYya2bbKYKVc1me5nH/6RpXWMnZgmZjpYEZxZxBOlSIp0MmBKXxfc8uHE7902XiqUvvzTXD1kPPHlt9vjiXLL99CoZIIWmUud5kwT6LQXXersTHsjG1OPG5oyZkocrFQVCC3GZMTTetWDjNLTnCHjkuLLeGfVbHjUHsTvLpc+4qlHPfXGCUZUSZqYRyPoqROixMFK7USps4bqcTtUEjNxrB3ljLoWeLv+k7efC/VQzLRe7SgnvrZ3bvmGUH4UqSRUWMy0UjMOGmrnR6G1wNtPuf3c2q0lrGmHTdYCX68Zw5p22NQZP4WSKQuH7007JOo3sS5UT5HKe7vr8vIysWjdXROX6K0OgxJv9NYZP0VyolANnSWzsq22xN3rDggFlikSKp38yFLiofazoRbTY6aUHn7x85+sHlx9+P2hMswlku905jjp5eNhUKiedvncHtbe8GcvVQ/O3/8zkWJMLHGsUEqaExmL1Ll7pOn0294lrly+/DhjbxLDmsgHFw+qIs9mJVyRfkZq+E+/9wfVg3e876dTWmEaiWFNO4dKzKcixSJNZ5WP9+FvvvZi9eD6kzcixViCeLITGovUuUukqsR6IsXiTR/6TrizThPzaIdN6YlPaKRSZw2dhaWl1DpnqI0aC370+dtZzU3fQ8bWOcIokuDklo/Uk1hJqRahcw5dKDl64SuvJZaEgjrn0AkuydU5UikSG1ULNiUWZns65771GHBUqp7ijGmCcRnTxCyMaWIWxjQxPWOamIUxTczCmCam129Mk3WaAAAAAChD0gQAAABAGZImAAAAAMqQNAEAAABQhqQJAAAAgDIkTQAAAACUIWkCAAAAoAxJEwAAAABlSJoAAAAAKOPuHvu89OKLxfsBm/ePv/nS3F1gR971oz/0+qNv35q1I+zLP7znB6oH//3/+6N5e8IOffGzn5y7C+zIf//rv3n9wbPPztsTduXxB95YPXju4w48Fs2YJgAAAADKkDQBAAAAUEaf2XPves97ivcDNu+nn3hi7i6wI88991z14Kc++MSsHWFfPv9Hf1g9eOLGjXl7wg79zI3rc3eBHfnMb/529eDqw++ftyfsy50/rv57/UnnWRbNmCYAAAAAypA0AQAAAFCGpAkAAACAMiRNAAAAAJQhaQIAAACgDEkTAAAAAGVImgAAAAAoQ9IEAAAAQBmSJgAAAADKkDQBAAAAUMbdc3eAfTm/dn44HG7eujlS+VmsopMUdO+b33j8v6/91Xf67RiSWGFVW3rrbMbb7n3T4XD4xmvf7r1vLbGSIS2yav/4H/2j4//96+99b3gl6fVUO/ZrlFV74z33HP/vd7773YLVtmtrNNdQqnXW6KkbDxwOh2ee/WqPvWopu/driA04O7ty/L8XF3dGradRrKF3650kTTBIFTOxE51R0SyJT2JoBbVGxnS8UYREWzseOvSKfkL1yI/o1Bn6hBKi4TVDRCMw6r3LUzceECHR1hn6VBuzEp9S9RQnaYL+xEy7Uoc7jVCp2n7vm994Mmw6WSClkoOYacc606KsHRuhUrX9bfe+KRQ29W6RVavjoUYeVG1Pz4k660mppDOfYvPqMKgRKlXb33jPPb3DpkjMVCTGYnuGxEzHuVK1MRI29WiIDajjoUYYVG0/O7uSGBJl1TNx/GSdJuhJzLRP7SSo1GgmMRNxw0OfdpwUH80kZtq5dhLUbzRTY6+//t73qi2hOEnMtHPt0MdoJiZWKmY6/t/QcKf83rEd7dCnXwxUqp6yJE0wiOWZKCI3P7I806687d43TRz6TN8iW5UbTomZGIlRS+TqMeWtc5fOjU/deEDMxLaZPcdsGmOCciOblN3ja3UPWcnbKuB7E1+M6bW/+s69b35j4qCk3Mr7lWQzjhOfaghSbgYUX4zpG699u8qV6gLDW2TV4osx/fX3vlflQScjpNyVmI4zpvigJzYpPovtO9/97hvvuafHBDqT48g1xuLcjdqOM6bqKanTrsRnsV1c3Dk7u5Iyga5UPSORNDGP9tSz9Oymc9ralNGPmInixEzETb9ot2XCmYWVwikoMWaqCox0wzvWZcjN5nL3skw406tSp1I3vIuTNDGDzqSm2nh+7Tye4NQxU7/dh7M8E2UlzpuzPNNuyZjYmMj6TXN0h81KWZ6pXms89JS8aVdKDSxqD1lqkDExvXqN8NBTxfMmSRPzaOdBN2/dTA9xBu7eWyjngn4ShymFbnsHsBaNqXByJUYVuo1dRPEb3rEuoSW9+1XS2CJaYlEG3vAukRXBmUHiqKXQU6Hdq+0T5E1iJmYhZgI2wxpMTCAxJPrOd79b/IZ3rNTwmOmZZ796/O/4KZjdxcWdyW5UJ2liQRae4FieibKyBjSJmYBV++vvfa/+V20RNjGS9IlvnRnT8bOHtFl4rF2pkUftGoxmYiE6M6bjZw+BuXW9mT3H+syyUpKYiVmImYDtqW5gdzgcUu5hB1msr0Su8WKmevtTNx546sYDUid2RdIEr+sMsCabkQdtVgEHtqoOm6Ag44/IZV4bjETSxPrMOLCo9xpS0JA1WCkSORn0BAANkcjJoCc6RSInq3pDD5ImeJ2QCABKqYYsmRzHkkmdgJ2o1mAaafHvTpImFiRxktr5tfOJU6GUm+UJqrbttb/6zr1vfuO9b35j5+ihkQYWxSs0molE33jt22+7901vu/dN33jt2+1n33bvm6oyk/eLharmtYVWUJIfMZLvfPe7b7znnjfec09n7pMYCcULyJVoi49UShzNZCUmslxc3Dk7u3J2dqUz90mPhErVMxL3nmMG8UQpEtmcTHPOr52nx1UpxWAMQiJg86o0KrQYk8SKJYjfXU4yRREm37EE8bvLjRFLSZqYRyPoqROixJFB7USps4bQet5iJnprL5lk3W7Wohq+FN8CtXZI1G8N71L1sBPt3GeCdb5DjYqZSFQFSe3Fnqw4TkQ79wklQQXrCRUuPvrJ7DlmcPPWzc7BRykxU7Vv9Ti9hs6S8iayVBPoDoFoqT1AycAlpheaCldNoDsEoiVT52iobwzXGQm1ByKFBijl1sPOVRPoDoFoqR36FAmD4o1CQ2SAUj2HrvOp0XvGqlQT3w6BSKgd+oTCoKx64oWLkzQxj3bQk77OUWikUmcNnYWtqUQ/VWzUSJpkSaxCFSc1kiYZEyGdc996ZEOl6mEnOqezjT2waJZG2aQqbGpsmaszLFzndLYeA4uy6inVaIq7Li8vE4vWHfJbOqSrQy55BFN67rnnqgc/9cEnZu0I+/L5P/rD6sETN27M2xP2o46xRANM6d/+5m9XD64+/P55e8KuXL3zx9WD6086zzKROgjKyqSs0wQAAABAGZImAAAAAMqQNAEAAABQhqQJAAAAgDIkTQAAAACUIWkCAAAAoAxJEwAAAABlSJoAAAAAKEPSBAAAAEAZd/fY56UXXyzeD9i83/v9j8/dBXbkTVd+sHpw8zPPz9sTduVzX7/z+oOP/j/z9oQd+re/+dtzd4E9uv3yp+fuAjty+/sPnv/qx+bsB5xiTBMAAAAAZUiaAAAAACijz+y5d73nPcX7AZv3yE8/MXcX2JEvv/hH1YPrT96Ytyfsygu/8R+qBw+c/9S8PWGHrj78/rm7wI7Uk+YceEzJgcdaGNMEAAAAQBmSJgAAAADKkDQBAAAAUIakCQAAAIAyJE0AAAAAlCFpAgAAAKAMSRMAAAAAZUiaAAAAAChD0gQAAABAGZImAAAAAMq4e+4OHM6vnR8Oh5u3bhYvPLZFdaZTZw/Pr50vuc+z8Ebx9h/54eP//dM//8vEkvHCRVos2y7zOju7klLs4uJOv/K5PYnv2G69R0Mszb/6wNXj//2tT92OFwhp7zik0XgHstpiYk/deOBwODzz7FdTitXi5bMKl6qkSKNMJuXAm+UYmP5QZ0oOvHTzJ01MqYpUOMkbtR/tBKfe2I5yQoWzQp+sFkPle7QL6UIJ19nZFWHTenVGSNXG8aKcrEZDIde/+sBVYdN6NX6lOd7Y/vUmq3CRFks1yqLMcgxMf6izNA68Y5KmLTMkBxI1IptIGNQoXG1MD306K4nUk1uehTsZ0zSinNzyKU6Ok6oLNGqutgubVqoOcRqRTbX9OMo5memk5z7pjeYWZgk6f2MJlWn8DlNtf+rGA8fbswoXabFUo0zp5IE3yzEw/aHOxBx4uazTBOxXaCRRteU4bwolPn/653/ZLnxSqMUe5bPaZeFyQ5wxYqZau+bcOX0sUDusyY1veiQ+WY1GCidO62MaKTFTrf3bS+T3mazCpSop0igTSD/wZjkGpj/UmYYDrwdJE0CqgQOIIlPkOrfnlmfVcuObHuVTdomv32Q000rFp8ilJ0dZWU9Wo6V6yASeuvFAVswERTjwmIUDr7dlJU3n186P/42xe7zmfu2mNF1vjxRLrySx/8f/23583J/0OhNLpn+UiSV7VN4olvhCJnijWKN67NLYjFHap5T1uYeXr1xc3BEY0XYyRRpjRSfDlFbn+DeuZ579asqis6Ey1fa6wqzCRVos1SgTSD/wZjkGpj/UmYYDb4gFrdPUmbwc0hYbCuUyibsPlN7zSP4V2jjvWkv9+pD4hvR71SmVx+OnMd7SJXxYbI9RS7uSm/70SItSdhFCETHe2CKjltbFHB9m4cBjFg68fpaSNEWSiJPRQB0r9Nt9oNyeR+Knk5XcvHWzGjjT+YriYUe1vV0mXmcPiW9Iv09t4LudckhM9kaxBH/653/59h/54dA63ClBT2j9Jkg39ry5g/yIwQw+ouI3LmbhwGMWDrwhFjR7LiWUGW/3IYo0PWP/i0t/LT1edWKcdPPWzS29pYynXs/7+N/hVHJ0XPJk4ZGkx2Es2djz5sqat3XmMsa8uXV1AADIspSkKXHUUuiplIE8I8nqeWhAU27/U7YM129GWMob0vtTS+lMZ8Y0KlPnVi20LlLWekmJheN3qRujRSjILefWq8poQuOSRhqvVKpRw6kAYHWWkjSFrPe395F6Hq+2X6MT5HGHuT/K9BXBI6Z5o5jS8aCk43+NZ9uyCkfajWxJYUDT2q1rQFPFgKZVa6c2EywE3qPRNgOaAGBFlrJO00Br/+W/R/9nWX9qjCbWW/9xK+uNRDl05TXV+k3pu1eF2+s9xQv3btS8Oaa3hJCLIX7rU7erfKed8tRPLbBR8+YAYI02kjSNoffsrbFVS1Mfb5nxLnvTtJsr0s+1h5IUFM9rQouFRwonttueQ5eVGYmZmJ6YaRva09kmSHCGNCpmAoCV2kjStMCwI8vs/Z/lxmpjtzVG/e5ARyn9oiIx05asZeqcmGljsoKbUllPjxrETACwXhtJmsaw/CihyjvGGNAUqnP578lh2ulsps7txwQRT7yJee9zxz7VS4CLmbZtlkwn0mg9AErMBAArtfQVwRPnOi1wSlRWl3L7v8Zoox3KLPBTg1FV0/H67Vg9EDMxGTHTlvyrD1yd/g5u/RoVM23JM89+9XA4PHXjgc5nq+1VmdzCRVos1SiLMssxMP2hztI48NqWkjTFQ4dIsHIyc0m/41i/4KN3zxMLDL9jWqJ6GaOBzaXsPterLlJnqTeK2bXXSzrWGGGUVTguN2yqKxczbcnCp87VzYmZtiQ39yky1imr0bpFMRMArNpSkqZDKwWof5NPHL/T/s2/s4bQstBDUoOBPW/slVjP8QsZaUGi3vuG3t52nbmvuncfxguG1ji+jIZ29BMJg7IKN9RRUWOXeqxTO0uyMBPTszDT9tTBTSP3qYcdjZHs5DZqYaYNa/8VPfR39dzCpSop0iiLMssxMP2hztI48Gp3XV5eJhatB9KPtCRQyn3NIoFFqP70wrlrHhXpebxL8c7Ue8WrjUxY6+zkyXbj/Ul8Q9qdSSmc+EpzP+JR36i6sLBgmSI5USj3ySqcXklWc/EdD4fDl1/8o+rB9SdvpFTCxEYa05Rebahkfa6PCzXxf/7Gf6gePHD+Uyn1MKXQ8KJQuJMe/aQsuhRvNHHoU7wJ801mcXLOReR3mPZePQoPrKRH+crtlz9dPbj68PtDZRhP/MAb7xiItDveoX7MgTevfR549V5Zf4Zc0Irg7YQiPewIjVQ6ORqoR1udFQ6sLav/jXaz2grFQI1nB4aJiW9Iv1fdrwN1tYk3j5vmjWIJOqfFhWLBrMKRFgfWANDDb33qdiPQmWAA0SyNsiidi4OEfqXJKlykxVKNsiizHAPTH+osjQPv2PxjmliUIQHKrsKX9BdrTBOzMKaJWRjTxPSMaWIWhpYwCwce0+s3pmlB6zQBAAAAsGqSJv7OrgYlDeGNAgAAgE6SJgAAAADKWNCK4Mwosvo1x7xRAAAAEGFME3+PGWGJvFEAAADQZkwTh0Oh3GQP4cseXiMAAAD0ZkwTAAAAAGVImgAAAAAoQ9IEAAAAQBl3XV5eJhY9O7syalcAAAAAWJqLizvphY1pAgAAAKAMSRMAAAAAZdzdYx83eod059fOqwevPvNr8/aEXXnhS69UDx65/8fm7Qm78tIP/GD14APvfte8PWE/7n3sserBNz/x3Lw9YVeef/Hz1YOfesdPztsTduXFv/1f1YPrT96YtyfsR79llIxpAgAAAKAMSRMAAAAAZUiaAAAAAChD0gQAAABAGZImAAAAAMqQNAEAAABQhqQJAAAAgDIkTQAAAACUIWkCAAAAoAxJEwAAAABl3D13B2Ctzq+dHw6Hm7duzt0RCrjvqV9sbHn1mV+LF+jU2CuxrcQaGjumtMWSXX366caW2x/5SHr5eOHejbYLdOrXOktw72OPNba89pnPFCmc3mi8kqzCrMVbn7je2PLNTzxXpHBiJZ01tBvq1KN1luCf/Nw/b2z5i9/9nSKF0xuNV5JVmNU5O7tyOBwuLu6kFKudLD+8nlItxkmagF0L5T73PfWLy4lyOju5qB6SJZTmXH366c4Ep7N8tTE98cltlO1px0b19naak1U4t9FqY2KjocKsRSjNeesT19sJTlbhrBarjTKjnWjHRvX2zjSns3yocFaj1cZ2PVmF2apG4lNvzI1+0usJlTyMkDdJmiBbNZqJDagTnM4RTMdRzslMJzH3qWrOSog6O9nuIWtRJz6dg4kiuU+/wUe5jZ5MnSRTK1WHOI3IptreyI+yCi+tURalDn0aEU+1vZEfZRUu0uLJCrMSLhaiDnEakU21vZ0fdZYPFT4pVPnwwqxIZ6ATKnMc8VQbs8KmHvU0Nqb0tgfrNEEeMdP2tMOaekvipLnxEp9QFpbbQ5amHdbUWxoRUmjsUrUlK29KbzREzLR27bCm3pIyUS5SuF+jAwuzFu2wpt6SMlEuUrhHi+nETKvWTojqLceBTiiW+ovf/Z1qS2L6ExqO1FlJVmFWp3fMdPy/ielPVj2hsUvVluJ5k6QJUp1fOxczbUl8eFF6cjRB1tPZGaOZ1ig+5W2kEKdUo1mpFosSn4DWOYYosXApszTK2OIT1jrHHCUWLtJiSlWsTnwCWu52yHJ2diUrr0kZcDRlPWWZPUdP9XrYx+FLY3nsRi7TuXh2fF3tyLOdlYfKp/Qkrv0ypU4ces2GK0jYxGRyl4WCk+HRvY89ljUnzgQ6ikgZqWRRp/2QMVHKccY0cJRQqZBo3rBJ0sQgobSlc3upm7VFKh+7J+40tyWlYprceqryKTeSmzfGYgyzJDWlGhUzrVdWQCPNoZSspGZpsc7S+kMiyRHzSgl3Sq3APdJK3qVImhgqMoCoc4jT+bXzIXlNvPLxeiJjoi133ly9jHfoqXiodLyj+Gk/bn/kI1effrq9QNKUg4zMmwP2w7w5DuH1m0L+4nd/55/83D8PLTfeXgQqvTBr0TvxaQ+GGqmei4s71fy+zhXB3XuO1WhHM42pdpNVPmpP2KqT0c/AAUcn73bXfrZg6yxTZPHvKmxqbx+v0fQCrF18mtuQwhCRNUNtsuls5s1t28kop7Ead1boU+dH7e0DC7NV7el1/UKf9HrqsKm9PavFFFYEZ5CTKyJ16p3yxHfsHLgUGotkrSUiRl3k+9Vnfi1yt7tQZ6q96n8T9JOJRcYNhZ4aPtTIYCWy7iKXVThLFV2F6h+vXeaSNW4o95ZzofIGK+1cj3u6Ze0SKty5Paswm1SHQcf/jp8ao55QzcVvPHcwpomRRPKd4eGOiWxMo/iApnj5V5/5tfue+sXOYU2dW8RMm9QeOlSHQZ2z59qz6oo02mjFgKbNW85CTu1lv8VMGzbeQk7tZb9TYiYDmvYgPmjo+Nkq9GnPcesUmm3XWUlWYTYsNOZopHrqLZ2z59qz6gaSNLFxRi2Ra2kT00I9iSRTrM7JQCc0pW7URtm8Rc2bq+5Gd+iKluqn2IZR58198xPPVbu0o6X6KXaoxxJI1QS3Q0700y5WVzKwMNsTinVCqymVqqdUvHWS2XPwuvNr5+1/c3eKqS0tZmIPIolPPAyqtvfLm8RMLCpmqrz2mc80mmhvYe0mWJ7pm594rrFLewu70nul7cRd4vVX2+sIKaswlBJfAaraXjZvMqaJjTPVjnSJMZM0ioJmSXwSG5VGbdgCY6aaaGnDplwFPHdHU+e2yg3dYBaSJnidTGrP6jWPRs2PRFQcCy3AtL1GWZR6GlpKoJNVeDxueLcB9bS1lCgnq3BuH2RJ+xFaDqldRg4FxZk9xyhC886KzEfLqsQMOE6aJmbqoeqPlb83SczELKaMmVJuJ1fXfO9jj1mJacMmi5ne+sR1KzFRSYmZYBalpqqNMeWtIEkThaWMDEocPdQOieI7NsqfbMVKTByOxhklxkxDxiXFk6Pcmo2QWq96VtrJxCe2LHjpAAAgAElEQVS+ElPW7Lb0RntUzirU4U56zDTxMknCpk2qRxKlx0wDl1XqETYZ7rQx9UilkzFTfF2kxBFPWZUUaZENiy+o1LueeCxVqtFjkibG0g5xOmOdKg9KLNyj/KErUaq3mDG3c3OFNe2wKdSTUDhloNN69Utw2mFT1kLgYiNmXJipnR913l2u86l6rJOpcys15cJMx/s2wqZ6rJMsaQ/6hTXt6KfHstxZlRRpkZUK5T65A5R61DO80UR3XV5eJhate+CXcw7fT3ZCB0Mk92nvEgmVOls5ORDpuHxWT7IkZlV1B4w9WZrEsKbxwaWHU6GSkXZD1YZ2iXTjhS+9Uj145P4fi/eTKSXGQ41IKLJXZ8n03XOrOumlH/jB6sEH3v2urB0ZVeJYofh8t87Cx/V3hkGR2trlQ4XjMVO9lxBhaRLHFlUfXFbh4/rbn3uoqsgR0iOHev7Fz1cPfuodP5m+F2NLDGsaOVRkr86SnTFWeiW5hY+9+Lf/q3pw/ckbkWIswcmBQqGIp71LvKrcehILt/fKGvRkRXBG0TnyKBTKZBWOlw9NuMudiAfj6RymFI+uXn3m17LKszGdc+gMUGIVOtOrUHL02mc+k1gSIr75iecaYZMUkrjOGW25o6KyKinSImt3cXGnEf30m7+WXk/nGKiyk+ZqxjSxKQucE2dME7MwpolZGNPE9IxpYhbGNDELY5qYXr8xTdZpYk2s4Q0AAABLJmliOxY4oAkAAAB2RdLEmtSLLmXdew4AAACYhhXBWavOaMmAJgAAAJiRpImVcS85AAAAWCxJE6skVwIAAIAFsk4TAAAAAGVImgAAAAAoo8/suZdefLF4P2DzXrj99bm7wB7956/917m7wI78wDt+snrwqZdfnrcn7NAfffHLc3eBHbnrDf9b9eDTf/KleXvCrvzg//726sFzH3923p5AnDFNAAAAAJQhaQIAAACgDEkTAAAAAGX0WafpXe95T/F+wOY9+uA/nbsL7MgLL/9x9eCR+39s3p6wKy99/8Hj733vnP1glx5/z7vn7gI78snnP1U9+MC1h+ftCbvy2Yv/UT24/uSNeXsCccY0AQAAAFCGpAkAAACAMiRNAAAAAJQhaQIAAACgDEkTAAAAAGVImgAAAAAoQ9IEAAAAQBmSJgAAAADKkDQBAAAAUMbdc3cAVub82nljy81bN2fpCVO671//m5Rir/77fzewqlAN7b1S2mJLrj799PH/3v7IR4ZXEqqnXWx402zDW975zsPh8K0vfCF3l2Px3XPLswdved/7DofDtz772R571eK7Nwr3aI71euuT/yzy7Dc//gfDK++spNHuwIZYtbOzK4fD4eLizpDda/F6GoWHtBshaYJU7Yyp3i5sYjyhZOq+f/1vhE070Zn7VBvTQ59QeHT16aclRyRqZ0D9dnnLO9/ZGR6F6g+VZyfaGVC/Xd7yvvd1hkeh+kPlIV0ow+rcHsmkIKQdGx3CuVVn4Wp78bBJ0gRJ6pipESpV24VNm3cy00nMfarYKD0hqmOmxi7VdmHTrjTyoPiwo4a6cGcljbApN8NiJ4bETMc5UbWxHR51Fo6UZyeGxEzHOVG1sR0edRaOlGd7xst3TsZMjUar7W998p8Jm3YllP5kaeREkfgpVLh42GSdJsjQjpPqLaERT+zB2IlPu/J6S+K0PtYrlPtUW7LyplAlufWwQ6Vipup/qy2ddbbjpHpLjz6wdqVipuP/7ayzHSfFy8NJ8Rl5h65sS8C0QwNjptDYpWpLZ+WhwsM70yBpgtOqFCk0aslopp0bL+uJD4AymolE8TFKxi6RrseoosRd4ss/Gc20cz1GFXXu0t4YX/7JaCaGa4dH8SFUwqb9ODu7UjbZOdncIbwek3WaWJA6fDkey9M5syz0bKOeeCuhpxqVh8qn9AR6yJ0NB7A6PVYB71EeGvqtAh4nPGIaA2fkmUC3eccZU2T80apJmhgkskh2aOPwlCdS+Ug9kUwRkRszVeUbI6E6KxFgUUSPUUudU/OMftqnfjFTLrEUDUNuNpe+l+CJShXrlLoTnIW9STTGSKIltFWRNDFUZADRGItnxyufsieQO2+uXsY79JRoibbbH/nI1aefbt8hbqR1u+s1wkNPyZt2ZYylkULrN0Gt1NJIx/UIlehUL8Ideio3MDq5PBMcyuU+Fxd3qll4nYt8T58uHZM0MZbOxbNLLZudVfmoPTmUG6vFugzMhobfS044tR912NTePrzySH6UcqM6NqxsJNQIrbLqnGZcFQsRWtK7XyWNLVl1jjF9jyUbfie40E3lYDx12NTenl7JGMmUFcEZ5OSKSJ16pzzxHTsHLsWX8R6eN7nlHLle/ff/LnIvuURuObcroRvDDb9hXKiG2x/5SORGdezKSPlO+mgpt5zbp+Ex07c++9njf8dPpVfCTnzz439Q8E5w8R07Z+rVDIkiV2iBp/SFn0ZaIkrSxChGvU3bAkcPLbBLjKrfeKLOjOn42UNmhGRA0x7UYVCV/tT/Gs8OcRwhdWZMjZJFGmXhig8j+tYXvlD/O24iffdSPWHJSg0jatfQr04DmjavM2M6fvaQnP7kzrZrVytmIlcdEl1c3Dn+13g2RfGpdmbPsXGjjjkyb45ZmDe3Q50jjAYmPtZdImTs2Wrf+sIXqibe8s53xlsxb25XxouZ6u1ved/73vK+951swrw5cmXFTN/8+B+E1oeqn4J07ZComlKXsu94KzpJmuB1nZlUJEUSMzELMdOuxMOg0GLhw2tmz6aZrVaHTSd7ImbaieXMVhMzkatHNtQeLWVpJ3LFQ6LQYuHpNQwkaWLjRkqCxEx7NmPWI2aiCDETKSJJ0AQZkJhptyKR0wQZkJiJISKRU+egJ+kSMxr7/nSSJnhdYmxUD30SM5FrSE5Ur98kZmKI41WfIgWEUAzXOyoqe887SFTknnesTu7iSmNbWn/YpOPVncZrRdLEKM6vnXcGMUVWTQpVPrxwSm3VAzETUxIzUcTJmAkOp/KdsYcaiZl2K57vJA41Sl+JqbP+lCagLR4MNcIjWRLzmiZmOrj3HMWlRDC5o4cSd2yUP9nK+bXzrOSrnjEnZtqzIeOS4neXC9Vcbxcz7VD8Xm9Z44/qwvHyBVtk56qoKDQFrzOxqjeKmRhDKK6qt4uZdih+d7nxUiHLfjNcFRWFVv7unBxXbxw7ZjpImhhPO8SJLLmdWLhH+UNXolRvyRoblVUeItph08mYaZqOsVjt6CfrxnM9EqJQi2ImcrXDps74ycJMFFGlRe3FnkLLP1mYiUo7+hkpZmoMbjpuznAn+mmHTZ3x09gLMzWYPUd5N2/drHKZrLu5dY5giudH6T0J1X+yhsa+8UblUBzrDIle/ff/rtoeGtnUruFkYTnUtlU3mDsEoqVG7tMZBtU7xsOpeq94i9AWConqG8x1RkvH5esC8RvSyaE4FgqJ6jl07V0ahesy8TvfyaG27Zsf/4Mq4kkZZ1QkDIq0KGaiUygkqm4wdwhES8fl6wKhMVDtXQaSNDGKzpFHoSAmq3C8fChOGhIzQVmdc+ikRUR0zmgbdWzR9C2yVZ1z6ARGjK0Kmxpb5uoMy9c5h27U0KcOm6Zpjg3rnEM32cCliLsuLy8Ti9a99ys6i7XAOW51yCVKYEovvPzH1YNH7v+xeXvCrrz0Qz9UPXj8ve+dtyfsx98tIi5KYEKffP5T1YMPXHt43p6wK5+9+B/Vg+tP3pi3J+xHv0XErdPEmuSu4Q0AAABMSdLEdixwQBMAAADsiqSJNakXXcq69xwAAAAwDSuCs1ZZN7YDAAAAJiBpYmXcSw4AAAAWS9LEKsmVAAAAYIGs0wQAAABAGZImAAAAAMqQNAEAAABQRp91ml568cXi/YDNe+FLr8zdBfbkDa//IeE/33513o6wKz94/vrtGj5167/M2xN26Ln/6/+euwvsyf/8n9V//9/bt+ftCLty5ckb1YPnPv7svD2BOGOaAAAAAChD0gQAAABAGX1mz73rPe8p3g/YvMce+Kdzd4Ed+cwrf1o9ePT+q/P2hF156fsPfvqxR+fsB7v0qPMsE3rhT75YPXjk/h+btyfsysvff3D9+9PoYJmMaQIAAACgDEkTAAAAAGVImgAAAAAoQ9IEAAAAQBmSJgAAAADKkDQBAAAAUIakCQAAAIAyJE0AAAAAlCFpAgAAAKAMSRMAAAAAZdw9dwdgHc6vnR8Oh5u3bs7dEUbxo0891djy9WeeKVI4semTNTQaHdIiC3HfL/1yY8urv/orBcsn1pNSSbVLv+ZYrDc/9NDhcPirL3+5c/tJ7R0jrSRWklWYdbnvwx9ubHn113+9SOHESrKaG9g0C3H16acbW25/5CMFy0d27BSpraohsTnW4uzsyuFwuLi4E3k2JLTX8Hra5dPbyiJpAnatHRvV29tpTlbhUjobTcynWKZ2ZlRv70xzcstntZtbCUCWUI5z34c/3A5xsgpntVhtFBvtRCj6ufr0051pTm55WKnOWOrs7MoYYZOkCdivOsRpRDbV9kZ+lFU4q/VSPWQV6rinke9U29u5T275rHZPVhIKuVi1gUOH3vzQQ1kDmsYozFrUoU8j4qm2N/KjrMJFWjxIoLaojo0aIVG1vR0e5ZZvO1kgUknieCjWJT7O6HBquFNuQ4n11L06Ll9tHCNssk4TsHftsKbekjJRLlI4IqtwpFFWqp3s1Fs6w53c8g2huCpeiZhpkxInx0V2FwaRq53j1FtSJspFCvdokZ1oJzv1ls5wJ7d8OjHT3pyMmWbRGTNV/1ttKd5tSROwU/EJaJ1jiBILn2w3MWYq2CgLEV/wqL09t3xcZ/nQfD0x0/a8+aGHhsdMpTrDHsSHC2WNLZITkSi+4FF7e275fv3p3C5m2p6zsyvLjJlqIy3J1MnsOXqqV8iuHlQaC2YfP9V+tlFPvJXQU43KQ+VTegITOM6Yqqgoa3BTZ4UiJwYKzdern5I6bcBxSFQNSsqNjcxuY/lOxlX3ffjDWUs+wRChGOs4Y6qelTptwHHGNNIooSGmzJgqkiYGaSQ48e2lbt8WqXzinrBqWQFN2TRHNrRbRUYh5ep35zjLhG/SwJyox+6dqVaonqzCLF9WoDNj+lM1nX6vOpYsdxTS2At+R+q31vgmpQc6nWlUjzyoVD3FSZoYKjKAqHOI0/m18yERT7zyKXsCPciYmF17yFKDjGmTBuY1PebNVbt07tgeHpVVGEqplwkPPSVvorfIMCUZ0yalhzv1Ityhp7JW+B5YT2j9puEkTYylHeI0ptpNVvmoPWGr4mskDSkMIbmDj/qVH1ID+zQ862nsW+dKnXVmFWa9sqKcCXKfITe8Yy3iqzINLz98R/amke/0vhNcbj0TjIGyIjiDnFwRqVPvlCe+Y+fApdCopeN1naBhvLvIZYkv5DReu8widy2kfuVf/dVfOf7XrypI91df/nI7IYrMnksvzKql30Uut3APr/76r7tX3R7kroVk7SRGVd/0rbFxlnrGWFJK0sQo4vnOSJVDWTMu5NTQDpXETFs16mpK7cJGM3FS7wFNnbHR8bOHo+lyWYXZjNkXcurMmBotjp1zMbEJFnIyoIm4zmzo+NlDWvQzpJ5q3+MaiodNZs+xcUYtkWtR8+a+/swzVRPtaKl+ig0Ye95cpPCrv/or9/3SL9/3S78sdQKmsbR5c+zBZPPmYHUuLu70nrUXIWmC13VmUsZP7c2iYqZKew6dBaE2ZoKYCWAhxExMT8wEcXXYVJCkiY0TFZFugTFTTbq0VWImlsx93yhLzMT0Jo6ZpFRQkTTB62RSe1aPGEoJdLIKj8cN79auXoc7MTbKLQ/zykqpRFqbV692lJIcZRUeQpi1bfWS3om5T2556KcaPTR8nlpWPaUaTWdFcEYRWh2pyKpJWZVYp4mTFhsz/ehTT1mJaaumjJmqXdxgDpjLZDFTfA1vudKuiJlgXsY0UdjNWzdPhjuJo4fa9cQrbzx1sifVs4Yy7dySZ8zVjRq4tDGLmjFnOh4hA8cZ/dWXv/zmhx5680MPddbQqDyrMOuy5Blzr/76r9/34Q/f9+EPd7YomVqvuRZmMnWOFNWiSKHlt9MHH2XVU6rRdMY0MZZ2yhNZcjuxcI/y1VONZ+stYqadW3jMVLfVGNlUj3WSQK3RLDFTaFiTgU5MoMqJ2lsioVJiYZZvrpipPawpNNApXl7MtEbW/2Yt2otw90t8suoJFS7OmCbKqwcTZd3NbeAIpnhPQvWfrCG3UdHVitTxTXyGWuPWbymFj0sODIO+/swzVT3tdsVMa1QnO/GIp86V+pXvjKVe/dVfue+XfrmzHgOaGCgUBlUjlQ5d+VFbVmFWoY5v4ilPY8pbSuHjko0wqBqmFKqnnRzFy7M69SS4+kGnOlfqV14sxUD1vd4SU55QcpRVT7xw8SWcJE2MonPkUSiFySocLx+Kk4rETDCLOmw63jJXZ1i1KmxqbJmrM+xEFT81wqPQAKWswhDSuVpTZHRSbnmA4apkpxH69Ih7suop1WiKuy4vLxOL1h3yKzqLtcA5cXXIJR1gSp955U+rB4/ef3XenrArL73xzdWDn37s0Xl7wn7UyZRogCm98CdfrB48cv+PzdsTduXlBx+sHlx/8sa8PWE/6iAoK5OyThNr0l5xCQAAAFgOSRPbscABTQAAALArkibWpF50KevecwAAAMA0rAjOWmXd2A4AAACYgKSJlXEvOQAAAFgsSROrJFcCAACABbJOEwAAAABlSJoAAAAAKEPSBAAAAEAZd11eXiYWPTu7MmpXAAAAAFiai4s76YWNaQIAAACgDEkTAAAAAGXc3WMfN5iHdOfXzqsHn/jif5y3J+zKy5/9YvXg2iPvmLcn7Mr3Xvnb6sH1J2/M2xP2o17ewXmWKX36+U9XD37i3Q/O2xN25Q2v3lM9+JnrT8zaEXbkTfe8pcdexjQBAAAAUIakCQAAAIAyJE0AAAAAlCFpAgAAAKAMSRMAAAAAZUiaAAAAAChD0gQAAABAGZImAAAAAMqQNAEAAABQhqQJAAAAgDLunrsDHc6vnR8Oh5u3bs7S9CztkmLIgTHjQcUqPPET/+L4fz/xxf9YqnDxSqod+zXKonzwx3/u+H//8Eu/W7b88EqKtMjSnJ1dOf7fi4s7ZcuHdgzprLB3oyzW8s+zjfK922U5PvTuXzj+3499/jfKlg/tGNKusL1jeqMs1pvuecvx/377u9+KFwhp75jeemjfeNP9WgxZYtI0lyqMAHalfWV5CKc5WYWLtMgmNRKc442daU5u+eGNFmmRpemMfqqNKXFPvXHU3GeWRhnV8s+zneWr7c7LK9UZ/VQbO6Oc3PLDhcKpD737F4RN69WZ48Sjnw2TNLEaQ0YkGc1EXONSMnKRGiqceD06pJLQpTAr1YhsOsOdIeV7V1Jv7Cz8wR//OWHTqjUim9DIo3r7cflqY2LukzJaKtSZ3o2yWIs9zxZplGVqRDaRcCdSPiX6SRktdVxmeIssWSNUasdPJ1OnN93zlh7J1MmhUhNnXtZpAvYr9BfOakvndXCocJYelYiZNiM0Mqja0o5+cssXaTRSmJUKjV2qtjTyps7E5/h/E2fGxfuTEjOVbZTpreU8Gynv/Ls6obFI1ZZQ3hQqX6Q/nVVFWkycjseihHKcakvijLnDaDHT9CRNAKfFh+6P+gfPJ37iX7jMZTLxKXLCpl3pHENUZGBRJDMar1EWbpbz7IwndxYiPkVueNjUzozGbpFV6xEYvemetywwZjosf/ZcY+2k+ByolML1ytChVZnq7Y3di/dkYJ/bJbN6GGmlYK/KvmmRVb2H7JvYz9z3hx06Ocb+5BVtFSodFzjOmCJ/AYYpmUC3cwNzn8jKUOM1yjaMcZ6Fk3pPZxt1pSe2p8fstuOMKXfw1NgWnTS1w6BQXtAZG6UHE7nd6NGTUfucW1Vc+tse6dVIb9rAhorsnvX+QBGuiYHJ9EuCsrQrn6BRaHN6ZWzjTcpjk/otopS+V2caNdLKTctNmjp/e682NjKI+CikUGCROOwlq/LOwqP2uXdVIelve6RXI71pod7mdnjI7j3eH5Ys9OfN5dwJbgl9oKw//NLvfvDHf649LCiylFJW+SKNsj0XF3fOzq6EVkeKRDzHM92GJ0GJay2VbZQZLf88y/Z87PO/8aF3/0J7INLEI4ystbQ33/7ut6qJbJ0rgqesAt6v0cSSVf1T3h1vuUnToSvCiMx6yyo8ZU/qku0komCfF1jVSG/awIaG7z7qkcb06ovg9vZZ+sMe1LlPe3uR8uNVwqrVYVN7e2f5dsmBI49Sdi/eKLNb73lWHLZeddjU3j5NB3qnWibcrVodNrW3x3ec+K5wnVlYv5XII5a7InjiaJT4xKXOVXVOVp7ew/YwnJRRPympRGKf+738iKyllDrLZ3Up/U3L1fh02gp+CqxaaNmjxvb4GknWTiJL6IZxpbYPqSR+S7usFlma0Hiizu11vnP8L15PqR5O3ChjW+l51pl91ULjiULZU3r5URkGtXahcUnLWTvp29/9VujueMUtekxTp1nGjwxfiKdzIEyo5AIHyMzSq8Q3fMins65PgeLq68jOUf2di4aGpgBAojqs6ZzIFprgll6+SKPxnrBGdVLTOXuuPauuXfLw/VFRQzpwcmhS2UaZ3drPswY0rVGd13TOngst7x2abde7A73HJRnQtFJ1nJQ7YmiaAU3x+kNT/4ZYX9LUabxQ4DhxiN/6bUYLzESm6dIqPh2Wr30dWQ31D22MPAWJQusxlSo/pJJ6Y+QpVio9xwlFQqH1nkbq3gSNMoHVnWfNm9uAzqW4O8OjenvniKfJxhmZN7cNnSOGljOmaUrLnT23HKFpa9W/6fvTW93n439zd2qozXw6TO/kvZAPXWP7G+XbWyAivgJ3e+ZabvkijVYbG+XbW1iR+HiiartxQxS30vOsmGnV4nlNaK7cxz7/G41d2ltGJWZau/i4pM47vm3eRsY0jT2Gpa5/meHFAofwTNmlhX86bI+rT3ZCtEQpVvUmy4znWTHTnhUJenpkRmKmPZt4LfApbSRpmkx7CfDEe9sPX+lpuAUGUnG5b1rvTwcGcmHKTsSHRwGMZILzbGhJKfZs7AwotKQUFDd9pLW+2XOd41bGG8wSmYQ1MMIIVdvjtUwwlic39JlmeNHwT6fgp8C2dd6hOVfKjXVc4DKvD/74z1mJaedmmUxnBh/znmfFTLv1oXf/wlx3fBMzsW3LTZriv+rXOcLJQKHIej0pNVQ9iQRh6X1OLJNSLPflD3+vsrqU/qbFK8zuZdFPgZXqcT1q2W8Giq+s1B4xlFu+SKORwqxUPMfJmtrWYx7c8KlzJt+t1FrOs3VPxEwbEFqJqRIao1QqbEofA1WXFDNtQ3wlptB4oinHGfXr4RDLTZoOrQShTigii0Cnl08UWQMoVHm7G6HKQ3X2UPDlp7/Sgl1Kf9OO9fh0Ovt5cgvb1r6u7bzrTedT9d9gsy5PU1pk29pRTjzcyS3fu5I6cmo8VY91MnVu1dphU+TGc4mFS5mlUaax5POsAcVb1Q6PQjee63y2Hus0RhJkYaYNa0c5S1sIPNTD4oHXXZeXl4lF6zP92AM96oyg8xf+UMwUqq1RPp5VhXbMrf9ksR51Rt729Koict/2eK+yupT4piWmVJ01DNk3XsPJZ+smXMQsUyTiaX9kocKRofjplYTqSay24eXPfrF6cO2Rd5ysk4lFQqLOHCe9fCQMymo0VPhkzPS9V/62enD9yRvxkkwvEtl0DhcKlW8Xjo856jFmKrGHjV2cZ5dpyefZxD/zdLb+6ec/XT34iXc/mFIJU4qMUerMd0LlQ6OfQiFRSoSUOH4qVMkbXr2nevAz159IqYcpRUKlzhwnPeIpVTK3h429sgYXL3pF8HbqEfo9PzQJKysUC4Usoac6K+/sycCSJxV5+cd7De9VVpcGvhXpn07xptmA0F2WQ4UTS5ZqkU3qnM4WCXFyyw+v5A+/9LsDm2OBOgcNRa4aLy7upBcuZZZGGZXzLNPrnEMXCYA+9vnfSC8MIZ0z1BZ1X7n/v727ebWsOvMAfAMOUtIEQkIySFpCJIktWoVX0ZgQGq0MHKYG/gH1l9WgJw0OSnokRGMPEvNhLKlSiqRNQlrsBpFAZxBrWD3Yyeaw18dZa++1P87Zz0OB56z7nrXWvbdA/fGutZfc4RZ7mljFFp6Od5b0NLEKPU2sQk8Ty9PTxCr0NLEKPU0sb1xP06bvaQIAAADghEiaAAAAAGhD0gQAAABAG5ImAAAAANqQNAEAAADQxiNrb4Ct8NQ5AAAAYCI9TQAAAAC0IWkCAAAAoA1JEwAAAABtjLmn6f333mu+Dzh7n93/v7W3wI589crXuhf/+8Fn6+6EXfnSF7/cvXj7rZ+tuxN26M6v3l97C+zR7+78ce0tsCNf/uJXuxf//h//tu5OIE9PEwAAAABtSJoAAAAAaGPM6blnnn22+T7g7L34ry+uvQV25Odv/6J78f0fPb/uTtiV++9+1L146frL6+6EHXryue+tvQV25P1f3e1ePHH5+Lo7YVc+vf/X7sW1F55edyeQp6cJAAAAgDYkTQAAAAC0IWkCAAAAoA1JEwAAAABtSJoAAAAAaEPSBAAAAEAbkiYAAAAA2pA0AQAAANCGpAkAAACANiRNAAAAALTxyNob2K5rV69dXFzcvXd3laVXWZcS4V+MFf+q0Mo3v/Ltw7ef/OVP+YKU8INTFi35eO2n2I5vff17h2///Onvj9akRD87etEp9WzflSuPHr598ODztvX5jw9EZ5u4Itv0ylOvHr5948PXWhVnPphSsnr5omzWjcubh29v37nVtr5kkvw8TVZka178zvXDt7/86K1WxdPnGZQNjF49StK0OV1sASwjGiHNHeKkFi1fsTD5Ypui+VE3OF+UU7toql7YdNJ2JqUAABNQSURBVLqioU83WJL49IPzRT+1O+QkRNOfVJpTVQwpYdzTD0bTnNr68kW78XCSJiuyNdEcpxsMQ5yq4laLLknSBDAMlcIc52gGVJ4T9ZMf1neDhZOImc7DILKJJjslPUdV0U/Joofjh/XdoLDp1A0im1TPUT9+WN8NlodN40Kiwh1yWgY5USZRShW/8tSrR8OmowX5SQpboti4PsQZRDbdeJj71NZPX7TJimzWIN+paiPKF4f6+ug8L37n+uH4wvGTe5qA/Ur1LnUj5YHOxJjp8O3RRcVMpy7VRtSNFB6X66cqDH2qFo3GTIdvqzbJRqRCn25kkOZEY6bDt3OkP1U75FSk2pG6kWiykypushkx036EYU0+vqmtr52k5FSdgOmkpXKcbmQQIVUVH5WaZ12SJoBJRkQ/0Viq6rCe65mYO+6JZli6mXYl2ovkFBszyR+Rmx42ZYKkV556Vcx0NvIH0GaKclZZFPI9SquHTU7PFRncnZS5+7mwsr9DOnUrUz8+mKF8J1XFJZXRPU/ZXmYnbVep+qUMuOebvLY3OpWc0Wu4HKdr7kudMoRNCJtYS8kButQHLxJx1WHGlOm04pwcHk87GhLduLzZ9jjb8iuyZ4MDdEuSNB0XBhDRZ41Fc4r8U8mqLv+umj9VXF4ZnTZVP2KevFarFP4cMr8IzwHkqNprvEfnRGImDs2U+KyYYbERbW/gjp59k1WxvHxE5bpxppAKcSqip/NmiqIkTUdkQqXDDCLfgpRJK1LRTyoKKZk/WlxeeXTbmbiq9tvPmL5K+c8hteiU/XMSPvnLn775lW+HtywVZjoTr0w6/Pjca7Edf/7099/6+vfCK5bKI54R5+ZGL3q4lvjppD148PmVK4+Gl3kfzZUOQ6KqhKi/Pjz1pfASqHE7ZMve+PC17mxa9JLvZfKdfI+SjAlo4pcfvfXid66HPUTRM25VxU30d4SnvtR8UUnTcWHKkDr1Vl45907C4r4yzE1abXvub3/cKkd/DpmWqDn2z9b0YVM4nv/glA6jcLn8bKl7xDlRfe4Tjh/97Oieo9pFw0rtTqeuj3LC8Wh9WDk69ImGR2GoVLtDTkIfNoXjC6y+ZKQFUfnjcpyTPj8KxycWt1XyoLrp3Ah+RMlFSPkzYodxRtXk5TsMe3bKG3bywprCJqxBfW1eM32V8p/D4AfI3qR6hebrIepDpcM/JYuKmc5Gqilp1ku+qxbtQ6XDPwtsklmlHt+Wbzs6/JOfJ+rwg4eD03fIqUi1FA3G83ckuTuJQl2gEz7orZMan8/yK7Ki1DPjMp1E5eOh/IPqouO//OitxR5Up6dppFW6XUZfe9SLNgSlKs+4naf2EnHOVapXqBsPT9UNCkZHP+EHu9aqOdZia/qkJnqQLTzgFtaM6CoasWh0RMx0uvqwprC9KKy8+EfPUeGK+Uak8KzciB2yfX1CFD09F73hO3XUbvTqGpr2KbxUe8XQR0PTHvTJTknHUFVxydLReQ7lJ0yd5ptC0tTMrNnEYe6TeSjbipaJZmZdRbq0W1W5zxzLHS46iLfETOdqlRynfNFUmJW674lTUR4epWKd1G1KrUyMt9imMOvpjtSlBjNfgqO6x7ddxKKl/kvLcG5uh6IdQ6m2o6ri1AypC5hqp2rO6bmTkXq2Wvdn+f2M1u/58M8WtjQYdKru7OVDnG583Xu43QJ+fvJNSd1487xplUXZlPz9StHHwy1s+ztkhHxLUfS43BsfvjaoD0fgqNt3bg3ynXBkbmKmXclfqj045lZVfFR4IC56RG55epqaWSCVCB91tx3LhDKzriJXotDyfUaZyEnT0364kBvYiSbRkqNzrJjyiJlY2BaipQFJ00kKrwAPnygXNf2mp+m2EOgc/hy28DMBgDMz+kF1EBIb0cQCAVB/Ok/MxLoGnVP5Rqo5OD03UthVNPcVQqn5J0YkqWlrv50zuKcJ5lZ7Im/wfLro4+oOX8N0DtPhqBqreOWpV93ExHQ3Lm9W3cRU8qy68syoJGZquyJ71t3hvfYukiRNR+SjjS7lKXmQWZOIpGSSwz6d6Mf73ZZEVE1qmnz7tauU/xzyc9btkpOSz31Sp9LmO63mHNwe5HOczPm4KUfnRi/adhusKB8eVTUflRdXLdpwh2xH9CamXrRNqVXYpAdq51Z5zFwfEsmJdih/udKgn6iq+KjCsKntoiUkTccN4oY+0QijijBSyRRXydzQlNlM5m3+S+MSlvm+/SmrFP4cUnOyB2HYNPeD55ZckW0Kc58F2ogKF00lUxqdzkAY5WQePFdYPHrRaHLUalE2JcyPok+Xi36p73USG1GiT3kGYVPf65SKgcJwqiquGtGLNHFFtimMcjJJUFVxaHAy7nCSTHKUWrT5wbovPHz4sLC0/zf9Tm606cOLaOJwNNkpKS5JRsIrmUqWKD9qVztn6rdf9e1ntF2l8OeQKQv3UzISnVyvyjZlIp7or6y88yhTmVq06i9Jfic/f/sX3Yvv/+j58jlZRiaySbULFTYTHe2KKl80VZ/fw/13P+pevHT95UwZq8hENoWhT6q4Kjk6Ok/5DgefkkRsU6ZNKfyVpYqjv9xMAjUlnCr87Pu/+vt/+D1x+fiIVZhVKrIpj5lSH4kmSoUJ0eGnylcc+PT+X7sX1154umRRlpTJicIcZ0RxYXKUmqR20fBTVc3FbgQ/LgyboplC6rhWbTCXyrZSXyrfzMTKvFbffttVCr+7zA+2+1LhheucomiT0dyx4Cd/+dPCK7Ip0aahuY+k1S76509/v/AOmVu0UynzX40PHnxeXtxk0dodchKiZ+hSOc4bH75WWAkZt+/cGkQ5+fgmenfSrIfgll+RBURPqKUSnKri/KLlk7RatISeJpiXniZWoaeJVehpYnl6mliFniZWoaeJ5Y3raXJPEwAAAABtSJoAAAAAaEPSBAAAAEAbkiYAAAAA2pA0AQAAANCGpAkAAACANiRNAAAAALQhaQIAAACgDUkTAAAAAG08MuIz77/3XvN9wNl786c/XXsL7MrD7h//+fbb6+6DXfnaP32ze/H2Wz9bdyfs0O/u/GHtLbBHv7vzx7W3wI7885e+3b344/v/ve5OIE9PEwAAAABtSJoAAAAAaGPM6blnnn22+T7g7D37g2fW3gI78t47d7oXl/7isaBP7n3WvXjp+svr7oQdeuq5f1l7C+zIu+/8tnvxxOXj6+6EXfnbH/5+PcLzP3xu3Z1Anp4mAAAAANqQNAEAAADQhqQJAAAAgDYkTQAAAAC0IWkCAAAAoA1JEwAAAABtSJoAAAAAaEPSBAAAAEAbkiYAAAAA2pA0AQAAANDGI2tvgKRrV69dXFzcvXd3laVXWXcjup98L/WjWPEXRHNPP/b84dsPPv5Nq+KSGUommb4oW3P1sRcO3977+Ndt64/OkJkkrJy4NNtx5cqjh28fPPi8bX2TeVotynb8+MmfHL598/7rrYpLZpgyyYgPshE3Lm8evr1951bb+imTDMoGxi3NRnz3G1cP3/7X/9xrVTzTolPWzZA0MTTIWfZm59/+DoWhTz8YpjlVxbUrpmZosiibEs1xusFoiFNbX7Wo2Gg/BvHN4WA0yknV1+Y+VfO0WpTtCEOfi3SIU1Vcu2JVZhSdhFMRzXG6wWiIU1vfZFHOT5jg9INhlFNVPOui3XjzsEnSBBH5TiVp1PkZRDbRcGdc8UBffDhJNxgNm6L1+Y9wKgYRT76NaER9tPhwkm4wDJtqMyxOyyCyiSY7h+OH9d1gVe5TNU+rRdmgQcqTSZRSxYVRUXSSqhlS2+PkDPKdVBtRPx6tv3F5syQnqppE/HTeBpFNKtwZURzq66PzDPKjquLp3NMEdcRM5yTVGdSNDCKkquLMcuEkH3z8m/wkqUU5RakcpxsJ86Pa+tSK4ST3Pv51+SSculTvUjcyyJuiic/h21Q+FV20cJ5Wi7IpqXakbiQa6KSKy5cLP/Lm/dczK6Ym4USlcpxuJJU3peqrNJmEE5VqI+pGBhFSVfFRqXlqi2vXzZM0QalrV6+JmZhOSMQq9ChRK9pDNKKxqGqeVotycvJH5KoOvjW5Wcn1TDuRby8qzImaTAKzyp/Lc0/TfhXeUV1e3N9mnYpO+vHBx5vvZOKew8qqHaY+GH774aJSJ2rVZkz5y5g++Pg3Tz/2vAN0HCVjoq1WuU/VPMImLgqOv02Ph9wCTqjwAN0Ck8Cho+HRd79xdY4LmApJmk5AmGiknnoWzT4yj0irykqqJk8Vz7rn2qlG8KQ5YA+ip+okVjuRuSN8vnlaLQpTiJlYTPQonyiK+SyfN0mati4alHSDg+Am34UUTXnC4vyKU3Yy655HTxVOm0qmZExnKdUWFG0mqiqukrq/ibN07+NfX33shfJ7uGvry+WvCU/Vy5tO1IMHn1+58mjqEu5MxHN4O9KUJKhqnlaLsro377/+4yd/EvYiLZzppO5vitZw6m7fuXXj8mbYQ7SRe7j7O8JTX1p9h4yT6iGKdh5VFZ8oSdMJiKYeqdaequIld9JXhqFPwz3P+u1zlvr8KByfWHzUYB4x03704VE43qQ+r7xTKZpthZkXp6IPm8LxaH1YOa7zqGqeVouyHX3YFI7PvfRg0ZKYSUPTeejDpnB8lf2kTHnaHRvU50fh+MTiuc2RcLkRfOsKL0LKnxFL3SvUpE/n7r27YZtSSS9SSQBUuOdx3z5cpB/3Fh2vKm61E85P6llvrcanb6Z/LN1gcPpyrCj17LboeJ/vHP7Jz5OZvHCeVouyKaleoWgMVFjcdieDPXAGUg+YS2VP5fVRIya5feeWB9Wdn9Sz26LjVcWzmmlFPU2napVWnel3HpWfStOLxAJSx9a68cFBuariEuHkrvfeg+iZtYt0x1Bt/VHRs3L9YH621FE+tq9PaqKn58JTdWHlxT+6omqXrpqn1aJsRKpXqBuP3vCdOmo3wuE8+RXDHXK6+mSnqmModdqudumjk+TjpNTRP7avz2uiB+IGB+WqipfRfEVJ01mZL5o5zH3yj35bkWSKccJwpzslN724ag/Cpl1J3cfUqr58G87E7Up5jpM6rZa676l8xcw8rRZla8IQpztSlxrMfGnKHqJhk5jpXEU7hlLtRam7k1IfmWkSzkCY13Sn5KYXz2S+m6GcnqNU6tha92f5/YzW7/nwz9qbYgX5m7y78T5CqioeR8C0B/lLtcMnvtXWjyBg2oP8VUfduL4hmsuHONHjcm/ef31QH46Mlupm4pzkL9VOHXMLz7JFT7flNZmEE5XPa7rxPkKqKp7VrBeQ62k6K3N3GPXzbzOa2VSDFQAAIyzfYZSJnDQ97UeTVEi0xKmY+zl3kibGCK8AD58oFzX9pqfpBFKsJd8YBTPJN0ZNKQZYRmHcIxWirXyH1LhJmswJU6RuiWpL0nSqol1FhXHP6OVST5Sb0uKU2vOIOef79mEV3eVNqZub5FbAHFpdilQ1j5uYWCUkyi8nt9qDObIkWEZ3qVPq+vBUy9IyMdOFe5q2Lx+49MHK0YSlyW1EJTN0O0kFYRc1ey6sKSlzGROh/OVKgxynqrjJipyl/M1KYT9Rbf30RZusyNbkb2LK3+I0pbjVPK0WZWHRm5h60Rxn+rXftStyZlI3MXVSkVCTG7vL7w7PFAutTlT+cqVB6FNV3FY/+QLPtpM0nYBBRNKHJpkrusvrC2VuaEpNHm4jNXlqzhFm+vY5b2H0k7nbu6q41SRNFmVTwignf7F3bf30SVLFYqaTFoZNmQfPFRZnVM3TalE2KIx+ok+Xi37px0/+ZERCVLIi5y2MclIPnot+9cblzfLQZ9wkqR2KmU5amB9l7vauKp6+6NwXMw184eHDh4Wl/b/p/R/7Mvp8JBq71J44G9Tns6rUB2vnP1o2Ys7MX7/yqTLKY6nCyn5XelW2KRPZhL+yEcUTJxlR33nvnTvdi8sfPJOqYS2ZfCea45TXZ8KgqkVrd9j75N5n3YuXrr+cKWMVmcgm2i6Uqg+L8z1H5fPUFg8+oldlmzIRT/nD4KK/3FQCVbViRj7heved33Yvnrh8vHxOlpFpLyoMfVLFVcnRiBVT9b2//eHv//P+/A+fy5SxikxOlDnFVl6cOSV3dJ7CDCu/RFVzsXuaTkAYNqXSjdTJtapwMHPvUvRLqcubwp1MrDyqybfP3kRPtKUSnKriJiu2WpRNiZ5QyyQ4tfXTJ2myIlsTbRrK/Fdjd3dSYXF+3eUXZTuiJ9pS8c2b918vrGy1ImcpekItk+DcvnOrvLjJJLU75CREj8WleoiqilstuiQ9TTAvPU2sQk8Tq9DTxPL0NLEKPU2sQk8TyxvX0+SeJgAAAADakDQBAAAA0IakCQAAAIA2JE0AAAAAtCFpAgAAAKANSRMAAAAAbUiaAAAAAGhD0gQAAABAG5ImAAAAANr4wsOHDwtLr1x5dNatAAAAALA1Dx58Xl6spwkAAACANiRNAAAAALRRcXoOAAAAADL0NAEAAADQhqQJAAAAgDYkTQAAAAC0IWkCAAAAoA1JEwAAAABtSJoAAAAAaEPSBAAAAEAbkiYAAAAA2pA0AQAAANCGpAkAAACANiRNAAAAALQhaQIAAACgjf8Hc2OO6bxJlt4AAAAASUVORK5CYII=)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9795d42",
   "metadata": {},
   "source": [
    " #### Questions on (3)\n",
    "\n",
    "1. What is your hypothesis as to why the BLEU-1 and BLEU-4 scores are higher in beam search than greedy search or top-$p$ sampling, for either of the tasks?\n",
    "\n",
    "2. Why do you think the ROUGE-1 and ROUGE-2 scores are lower for the ROC-stories dataset than the XSUM dataset? Given the nature of ROUGE and the tasks, how would you interpret this result? Find one example from the story generations that could illustrate this case (feel free to write the entry ID instead of the whole Input + Reference + Candidates) and explain why it applies to this.\n",
    "\n",
    "3. What is your hypothesis as to why the BERTScores (of all three generation styles) demonstrate Flan-T5 as an equally good story generator as a summarizer, compared to the BLEU and ROUGE observations you have made before? Similarly, why do you think top-$p$ is getting relatively better results with the BERTScore than the ROUGE or the BLEU scores? There can be several answers to this so feel free to provide several hypotheses (although one is enough!).\n",
    "\n",
    "4. What pitfall is a more significant concern for BERTScores than ROUGE and BLEU?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c903927",
   "metadata": {},
   "source": [
    "#### Your Answers to (3)\n",
    "\n",
    "1. Beam search considers the most probable sequence using a \"num_beams\" parameter, while greedy selects the most probable token one by one, resulting in a less likely final sentence. Top-p sampling introduces more diversity in the output. \n",
    "   Therefore, because BLUE-1 and BLUE-4 evaluate 1-gram and 4-gram precision, outputting the most probable sentence increases the likelihood of producing the same grams as the gold references. Additionally, there is a significant difference between greedy and beam search in the BLUE-4 with story generation. With greedy, choosing tokens one by one would result in fewer 4-gram correspondences with the gold references than with beam search, where the model can change its mind about a token and output a more probable n-gram.\n",
    "\n",
    "\n",
    "2. The Rouge score compares the overlap of grams between the generated candidate and the gold reference. The task of generation leads to more diversity than the task of summarization, resulting in higher overlapping scores in summarization. For example, in [this story](https://cs.rochester.edu/nlp/rocstories/) generation, an alternative ending where Tina is furious and unhappy is entirely possible, although there would be no overlapping grams with the gold reference, leading to a poor Rouge score.\n",
    "\n",
    "\n",
    "3. As mentioned in point 2, the task of generation leads to more diversity than the task of summarization. Additionally, while ROUGE and BLUE scores only look at n-grams and compare them, BERTscore uses its embedding to consider meaning. Therefore, if the model generates the end of a story that is similar to the gold references but with different words like \"the guy is furious\" and \"the man is enraged,\" we will have poor scores with ROUGE and BLUE but a good score with BERTscore. Therefore, ROUGE and BLUE are more affected by the diversity of the story generation, while BERTscore can still capture similar meanings.\n",
    "\n",
    "\n",
    "4. BERTscore places a higher emphasis on the meaning of the answers. As mentioned in point 3, while ROUGE and BLUE scores only consider n-grams and compare them, BERTscore uses its embedding to consider meaning. For example, \"the man is enraged\" and \"the guy is furious\" would lead to poor overlapping scores, even though they mean the same thing. However, BERTscore can capture this similarity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7513e3aa",
   "metadata": {},
   "source": [
    "## **PART 4: Checklist**\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2deea5a4",
   "metadata": {},
   "source": [
    "<div style=\"padding:15px 15px 15px 15px;border-left:3px solid #8e7cc3;background-color:#e4e1eb;border-radius: 15px;\">\n",
    "\n",
    "🎉 Excellent work! You just reached the end of the assignment. \n",
    "\n",
    "To give us the deliverables you will have to commit and push the following files to your github classroom repository:\n",
    "\n",
    "- [x] The python files:\n",
    "    - [x] `a3_decoding.py`\n",
    "    - [x] `a3_sampling.py`\n",
    "    - [ ] `a3_utils.py`, if you added any helper functions\n",
    "\n",
    "\n",
    "- [x] This jupyter notebook `a3_notebook.py` with \n",
    "    - [x] the answers to Part 2 written out in their corresponding cells.\n",
    "        - [x] Answers to (2.1) questions\n",
    "        - [x] Answers to (2.2) questions\n",
    "        - [x] Answers to (2.3) questions\n",
    "        - [x] Answers to (2.4) questions\n",
    "    - [x] the answers to Part 3 written out in its corresponding cell.\n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}