nlping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Импорт выгруженных сообщений из Telegram-канала"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "with open(\"channel_messages.json\", encoding='UTF-8') as jsonFile:\n",
    "    messages = json.load(jsonFile)\n",
    "    jsonFile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\"В настоящее время\\xa0ВОЗ\\xa0не рекомендует проводить поголовную вакцинацию детей и подростков, поскольку бремя тяжелых заболеваний в этих возрастных группах невелико, а высокий охват вакцинацией среди групп, подверженных наибольшему риску тяжелых заболеваний, еще не достигнут во всех странах\"', '⚡️ВОЗ не рекомендует проводить поголовную вакцинацию детей и подростков от COVID-19 – заявление', 'Спасибо!', 'Гемотест - не лучшая лаборатория, часто косячат с интересующими вас анализами. А так вам нужен abbot quant rbd - его много где делают', 'О госпитализациях с омикроном в Англии', 'https://www.imperial.ac.uk/mrc-global-infectious-disease-analysis/covid-19/report-50-severity-omicron/', 'Просто сделайте прививку. Даже если \"вдруг\", то хуже не будет', 'Здравствуйте, уважаемые! Помогите определиться по такому вопросу.\\nНе вакцинирована. Собираюсь в ближайшие дни. Ковидом вроде как не болела, но болел супруг. Я была в контакте в течение недели (сначала лечился дома, потом его положили в моногоспиталь), у меня никаких симптомов не появилось. ПЦР у меня взяли на 10 день от начала его болезни - отрицательно. Хочу перед прививкой сдать на антитела (а вдруг?). Сдавать буду в \"Гемотесте\". Какое из этих двух обследований мне лучше сдать?\\n1. Коронавирус SARS-CoV-2, нейтрализующие антитела IgG к S-белку (Код 20.158.)\\n2. Коронавирус SARS-CoV-2, нейтрализующие антитела IgG к RBD домену S-белка (Architect, Abbott) (Код 20.165.)? \\nИли любое из этих подойдёт.\\nБолел он в сентябре этого года.\\nСпасибо всем, кто ответит.', '3 дня лежала с темпой 37-38, сопли, от них кашель. Считаю, легко.\\nМуж без прививки. Те же симптомы, но дольше + горло, потеря запахов, вкусов.\\nДети бессимптомно. 2 из 3 тоже + по тестам', 'вы в итоге как долго болели и как перенесли?']\n"
     ]
    }
   ],
   "source": [
    "print(messages[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "для наглядности посмотрели на первые 7 сообщений"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"В настоящее время ВОЗ не рекомендует проводить поголовную вакцинацию детей и подростков, поскольку бремя тяжелых заболеваний в этих возрастных группах невелико, а высокий охват вакцинацией среди групп, подверженных наибольшему риску тяжелых заболеваний, еще не достигнут во всех странах\"\n",
      "⚡️ВОЗ не рекомендует проводить поголовную вакцинацию детей и подростков от COVID-19 – заявление\n",
      "Спасибо!\n",
      "Гемотест - не лучшая лаборатория, часто косячат с интересующими вас анализами. А так вам нужен abbot quant rbd - его много где делают\n",
      "О госпитализациях с омикроном в Англии\n",
      "https://www.imperial.ac.uk/mrc-global-infectious-disease-analysis/covid-19/report-50-severity-omicron/\n",
      "Просто сделайте прививку. Даже если \"вдруг\", то хуже не будет\n"
     ]
    }
   ],
   "source": [
    "for message in messages[:7]:\n",
    "    print(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Предобработка данных"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Нормализация\n",
    "убираем пунктуацию, ссылки, двойные пробелы"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pkg_resources\n",
    "import re, string\n",
    "import spacy\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simplify_punctuation_and_whitespace(sentence_list):\n",
    "    norm_sents = []\n",
    "    print(\"Normalizing whitespaces and punctuation\")\n",
    "    for sentence in tqdm(sentence_list):\n",
    "        sent = _replace_urls(sentence)\n",
    "        sent = _simplify_punctuation(sentence)\n",
    "        sent = _normalize_whitespace(sent)\n",
    "        norm_sents.append(sent)\n",
    "    return norm_sents\n",
    "\n",
    "def _replace_urls(text):\n",
    "    url_regex = r'(https?:\\/\\/(?:www\\.|(?!www))[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\\.[^\\s]{2,}|www\\.[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\\.[^\\s]{2,}|https?:\\/\\/(?:www\\.|(?!www))[a-zA-Z0-9]+\\.[^\\s]{2,}|www\\.[a-zA-Z0-9]+\\.[^\\s]{2,})'\n",
    "    text = re.sub(url_regex, \"<URL>\", text)\n",
    "    return text\n",
    "\n",
    "def _simplify_punctuation(text):\n",
    "    # This function simplifies doubled or more complex punctuation. The exception is '...'.\n",
    "    corrected = str(text)\n",
    "    corrected = re.sub(r'([!?,;])\\1+', r'\\1', corrected)\n",
    "    corrected = re.sub(r'\\.{2,}', r'...', corrected)\n",
    "    return corrected\n",
    "\n",
    "def _normalize_whitespace(text):\n",
    "    # This function normalizes whitespaces, removing duplicates\n",
    "    corrected = str(text)\n",
    "    corrected = re.sub(r\"//t\",r\"\\t\", corrected)\n",
    "    corrected = re.sub(r\"( )\\1+\",r\"\\1\", corrected)\n",
    "    corrected = re.sub(r\"(\\n)\\1+\",r\"\\1\", corrected)\n",
    "    corrected = re.sub(r\"(\\r)\\1+\",r\"\\1\", corrected)\n",
    "    corrected = re.sub(r\"(\\t)\\1+\",r\"\\1\", corrected)\n",
    "    return corrected.strip(\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Лемматизация"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lemmatize(sentence_list):\n",
    "    nlp = spacy.load(\"ru_core_news_sm\")\n",
    "    new_norm=[]\n",
    "    print(\"Lemmatizing Sentences\")\n",
    "    for sentence in tqdm(sentence_list):\n",
    "        new_norm.append(_lemmatize_text(sentence, nlp).strip())\n",
    "    return new_norm\n",
    "\n",
    "def _lemmatize_text(sentence, nlp):\n",
    "    sent = \"\"\n",
    "    doc = nlp(sentence)\n",
    "    for token in doc:\n",
    "        sent+=\" \"+token.lemma_\n",
    "    return sent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalization_pipeline(sentences):\n",
    "    sentences = simplify_punctuation_and_whitespace(sentences)\n",
    "    sentences = lemmatize(sentences)\n",
    "    return sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalizing whitespaces and punctuation\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8e2d7c76428f42bf99504c31bb7b178b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=5000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Lemmatizing Sentences\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8df67d065a2349c0b1dac0d4635fb5ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=5000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "messages_normed = normalization_pipeline(messages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Токенизация"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(text):\n",
    "    text = [string.lower() for string in text]\n",
    "    text = re.sub(r'[^а-яА-Я ^0-9 +]', '', str(text))\n",
    "    return text.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "данные до предобработки:\n",
      "['в', 'настоящее', 'время0воз0не', 'рекомендует', 'проводить', 'поголовную', 'вакцинацию', 'детей', 'и', 'подростков', 'поскольку', 'бремя', 'тяжелых', 'заболеваний', 'в']\n",
      "данные после нормализации и лемматизации:\n",
      "['в', 'настоящий', 'время', '0', 'воз', '0', 'не', 'рекомендовать', 'проводить', 'поголовный', 'вакцинация', 'ребнок', 'и', 'подросток', 'поскольку']\n"
     ]
    }
   ],
   "source": [
    "tokenized_original = tokenize(messages)\n",
    "print('данные до предобработки:')\n",
    "print(tokenized_original[:15])\n",
    "tokenized_normalized = tokenize(messages_normed)\n",
    "print('данные после нормализации и лемматизации:')\n",
    "print(tokenized_normalized[:15])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Удаление стоп-слов"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "удаляем русские стоп-слова: \n",
      "['и', 'в', 'во', 'не', 'что', 'он', 'на', 'я', 'с', 'со', 'как', 'а', 'то', 'все', 'она', 'так', 'его', 'но', 'да', 'ты', 'к', 'у', 'же', 'вы', 'за', 'бы', 'по', 'только', 'ее', 'мне', 'было', 'вот', 'от', 'меня', 'еще', 'нет', 'о', 'из', 'ему', 'теперь', 'когда', 'даже', 'ну', 'вдруг', 'ли', 'если', 'уже', 'или', 'ни', 'быть', 'был', 'него', 'до', 'вас', 'нибудь', 'опять', 'уж', 'вам', 'ведь', 'там', 'потом', 'себя', 'ничего', 'ей', 'может', 'они', 'тут', 'где', 'есть', 'надо', 'ней', 'для', 'мы', 'тебя', 'их', 'чем', 'была', 'сам', 'чтоб', 'без', 'будто', 'чего', 'раз', 'тоже', 'себе', 'под', 'будет', 'ж', 'тогда', 'кто', 'этот', 'того', 'потому', 'этого', 'какой', 'совсем', 'ним', 'здесь', 'этом', 'один', 'почти', 'мой', 'тем', 'чтобы', 'нее', 'сейчас', 'были', 'куда', 'зачем', 'всех', 'никогда', 'можно', 'при', 'наконец', 'два', 'об', 'другой', 'хоть', 'после', 'над', 'больше', 'тот', 'через', 'эти', 'нас', 'про', 'всего', 'них', 'какая', 'много', 'разве', 'три', 'эту', 'моя', 'впрочем', 'хорошо', 'свою', 'этой', 'перед', 'иногда', 'лучше', 'чуть', 'том', 'нельзя', 'такой', 'им', 'более', 'всегда', 'конечно', 'всю', 'между']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\Admin\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "nltk.download(\"stopwords\")\n",
    "import copy\n",
    "russian_stopwords = stopwords.words(\"russian\")\n",
    "print('удаляем русские стоп-слова: ')\n",
    "print(russian_stopwords)\n",
    "def remove_stopwords(text):\n",
    "    text_new = copy.deepcopy(text)\n",
    "    print('removing stop-words')\n",
    "    for word in tqdm(text):\n",
    "        if word in russian_stopwords:\n",
    "            text_new.remove(word)\n",
    "    return text_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "removing stop-words\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6cb66b2f31ca421280a44972152f7e08",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=81125.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "preprocessed_data = remove_stopwords(tokenized_normalized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Промежуточные результаты"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "количество слов до предобработки данных:  79342\n",
      "количество слов после предобработки данных:  49798\n",
      "количество уникальных слов до предобработки данных:  16630\n",
      "количество уникальных слов после нормализации и лемматизации:  9918\n",
      "количество уникальных слов после нормализации, лемматизации и удаления стоп-слов:  9780\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "print('количество слов до предобработки данных: ', len(tokenized_original))\n",
    "print('количество слов после предобработки данных: ', len(preprocessed_data))\n",
    "print('количество уникальных слов до предобработки данных: ', np.unique(tokenized_original).shape[0])\n",
    "print('количество уникальных слов после нормализации и лемматизации: ', np.unique(tokenized_normalized).shape[0])\n",
    "unique_words = np.unique(preprocessed_data)\n",
    "print('количество уникальных слов после нормализации, лемматизации и удаления стоп-слов: ', unique_words.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeUAAAE/CAYAAAB4ldsrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df3xU1Z3/8debH1UURcDoWsCGVrYqP4wQEUSQihW6VdFVVqwVUJQt1dZut7S6/XZ1t/bbWq1U1qK1tYBaKyyt1a+rXRWxoCIQFERgxVRBo66CoiIoCn6+f8xJOokhmUAkd+D9fDzmMXfOnHPm3MzNvHPuvZmriMDMzMxaXquWHoCZmZnlOJTNzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCPatPQAdtSBBx4YpaWlLT0MMzOzJlmyZMn6iCip77miDeXS0lIqKipaehhmZmZNImnt9p7z7mszM7OMcCibmZllhEPZzMwsI4r2mLKZWVN8+OGHVFVV8f7777f0UGwPsffee9O1a1fatm1bcBuHspntEaqqqthvv/0oLS1FUksPx3ZzEcEbb7xBVVUV3bt3L7idd1+b2R7h/fffp3Pnzg5k2yUk0blz5ybvmXEom9kew4Fsu9KObG8OZTOzXeCtt95i6tSpjdZbs2YNd9xxR0H1evXq1RxDK1hpaSnr16//xF9n0qRJ9OzZk0mTJn3irwUwdOjQzHzvhUPZzPZIUvPeGtPcoVxstm7dWnDdX/7ylzz55JNcc801LTqOluBQNjPbBS677DL+8pe/UFZWxqRJk4gIJk2aRK9evejduzczZ86sqTd//nzKysqYPHkya9asYfDgwfTt25e+ffvy+OOPN/g6jzzyCEOHDuWss87i8MMP59xzzyUigNoz3YqKCoYOHQrAlVdeydixYzn55JMpLS3lD3/4A9/97nfp3bs3I0aM4MMPP6zp/5prrqF///7079+fyspKANatW8eZZ57JMcccwzHHHMNjjz1W0++ECRM4+eSTGTNmTK1xbm/9TzvtNDZt2sSxxx5bU1atd+/evPXWW0QEnTt35tZbbwXgvPPO46GHHuL999/n/PPPp3fv3hx99NHMnTsXgOnTpzNq1ChOPfVUTj75ZN577z1Gjx5Nnz59OPvss3nvvfcA2LZtG+PGjasZ0+TJk5vwDjeTiCjKW79+/cLMrFArV66s9Ria99aYF154IXr27FnzePbs2XHSSSfF1q1b43//93+jW7du8corr8TcuXPjy1/+ck29TZs2xXvvvRcREatXr47qz766/VWbO3du7L///vHSSy/Ftm3bYsCAATF//vyIiPjMZz4T69ati4iIxYsXxwknnBAREVdccUUMGjQoPvjgg1i6dGm0a9cu7rvvvoiIOP300+Ouu+6qaX/VVVdFRMSMGTNqxnnOOefUvMbatWvj8MMPr+m3b9++sXnz5o+Nc3vrHxGx77771vsz/Md//Me49957Y/ny5VFeXh4XXnhhREQcdthhsXHjxrj22mtj3LhxERGxatWq6NatW7z33nsxbdq06NKlS7zxxhsREfGzn/0szj///IiIWLZsWbRu3ToWL14cFRUVcdJJJ9W83oYNG+odR1PU3e4iIoCK2E62eaZsZtYCHn30Uc455xxat27NwQcfzAknnMDixYs/Vu/DDz/koosuonfv3owaNYqVK1c22nf//v3p2rUrrVq1oqysjDVr1jTa5ktf+hJt27ald+/ebNu2jREjRgC52Wl++3POOafmfsGCBQA89NBDXHLJJZSVlXHaaafxzjvvsHHjRiA3823Xrt0Or3++wYMHM2/ePObNm8fEiRNZvnw5L7/8Mp06daJ9+/Y8+uijnHfeeQAcfvjhfOYzn2H16tUAfPGLX6RTp04AzJs3j69+9asA9OnThz59+gDw2c9+lueff55vfOMb/OlPf2L//fdv9OfW3AoKZUn/JGmFpGck/U7S3pI6SXpQ0nPpvmNe/cslVUp6VtLwvPJ+kpan56YonZomaS9JM1P5Qkmlzb2iZs2puY9H+vbJ39auhYqK3C0LIu1SbszkyZM5+OCDWbZsGRUVFXzwwQeNttlrr71qllu3bl1zHLVNmzZ89NFHAB/7V53qNq1ataJt27ZUnzncqlWrWsdh888orl7+6KOPWLBgAUuXLmXp0qW8/PLL7LfffgDsu+++9Y6x0PXPN2TIEObPn8/8+fMZOnQoJSUlzJ49m8GDBzfaZ91x1HdmdMeOHVm2bBlDhw7lF7/4BRdeeGGTx7izGg1lSV2AbwLlEdELaA2MBi4D5kRED2BOeoykI9PzPYERwFRJrVN3NwITgB7pNiKVjwc2RMRhwGTg6mZZOzOzjNhvv/1qZo+QC5iZM2eybds21q1bx7x58+jfv//H6r399tsccsghtGrVittuu41t27bt8BhKS0tZsmQJAL///e93qI/q47wzZ85k4MCBAJx88snccMMNNXWWLl3aaD/bW/+GdOvWjfXr1/Pcc8/x2c9+luOPP55rr722JpSHDBnCb3/7WwBWr17Niy++yOc///l6X7u63jPPPMPTTz8NwPr16/noo48488wz+eEPf8iTTz7Z6Ho0t0J3X7cB2klqA+wDvAKMBGak52cAp6flkcCdEbElIl4AKoH+kg4B9o+IBWmf+q112lT3NRsYpvr+jDEzK1KdO3dm0KBB9OrVi0mTJnHGGWfQp08fjjrqKE488UR++tOf8jd/8zf06dOHNm3acNRRRzF58mS+/vWvM2PGDAYMGMDq1au3O/MsxBVXXMGll17K4MGDad26deMN6rFlyxaOPfZYrr/++poToaZMmUJFRQV9+vThyCOP5Kabbmq0n+2tf2OOPfZY/vZv/xbI7c5++eWXOf744wH4+te/zrZt2+jduzdnn30206dPr7XXoNrEiRN599136dOnDz/96U9r/hh4+eWXGTp0KGVlZYwbN44f//jHBf9cmosK2YUg6VLgR8B7wAMRca6ktyLigLw6GyKio6QbgCci4vZUfgtwP7AG+ElEnJTKBwPfi4hTJD0DjIiIqvTcX4BjI6LWP8RJmkBups2hhx7ab+3atTu5+mY7xn8yFp/771/FgQceAUB5eQsPxvYYq1at4ogjjqhVJmlJRNS7FRay+7ojuZlsd+DTwL6SvtpQk3rKooHyhtrULoi4OSLKI6K8pKSk4YGbmZkVmUJ2X58EvBAR6yLiQ+APwHHAa2mXNOn+9VS/CuiW174rud3dVWm5bnmtNmkXeQfgzR1ZITMzs2JVSCi/CAyQtE86zjsMWAXcA4xNdcYCd6fle4DR6Yzq7uRO6FoUEa8CGyUNSP2MqdOmuq+zgIdjR07N2wktfWaobzt3MzPbHTR66caIWChpNvAksBV4CrgZaA/MkjSeXHCPSvVXSJoFrEz1L46I6tMFJwLTgXbkjjPfn8pvAW6TVEluhjy6WdbOzMysiBR0PeWIuAK4ok7xFnKz5vrq/4jciWF1yyuAj32DekS8Twp1MzOzPZW/0cvMzCwjHMpmZkVuV116cMqUKRxxxBGce+65O93XTTfdVHNBiWLTvn37T6zvgnZfm5ntdpr7DMFde25qs9m6dStt2hQWBVOnTuX++++ne/fuO/26X/va13a6j12hKT+f5uCZspnZLrBmzRqOOOIILrroInr27FlzCUGoPdNdv349paWlQO6Sg6effjqnnnoq3bt354YbbuC6667j6KOPZsCAAbz55l//c/T222/nuOOOo1evXixatAiATZs2ccEFF3DMMcdw9NFHc/fdd9f0m38pw7quu+46evXqRa9evfj5z38O5EL0+eef57TTTvvYJQ2nT5/OJZdcUvP4lFNO4ZFHHgFys8rvf//7HHXUUQwYMIDXXnsNyF3W8dprrwVgyZIlHHXUUQwcOLDmco6N9fvAAw8wcOBA+vbty6hRo3j33Xdrjen111+nX79+ACxbtgxJvPjiiwB87nOfY/Pmzaxdu5Zhw4bRp08fhg0bVvP8uHHj+Pa3v80XvvAFvve97/HCCy8wcOBAjjnmGH7wgx/UvMarr77KkCFDKCsro1evXsyfP7++t75JHMpmZrvIc889x8UXX8yKFSs44IADCvr+6WeeeYY77riDRYsW8f3vf5999tmHp556ioEDB9ba/btp0yYef/xxpk6dygUXXADAj370I0488UQWL17M3LlzmTRpEps2bQJgwYIFzJgxg4cffrjW6y1ZsoRp06axcOFCnnjiCX71q1/x1FNPcdNNN/HpT3+auXPn8k//9E8Fr/OmTZsYMGAAy5YtY8iQIfzqV7/6WJ3zzz+fKVOm1Fx1qjHr16/nqquu4qGHHuLJJ5+kvLyc6667rladgw46iPfff5933nmH+fPnU15ezvz581m7di0HHXQQ++yzD5dccgljxozh6aef5txzz+Wb3/xmTfvVq1fz0EMP8bOf/YxLL72UiRMnsnjx4lpfBXrHHXcwfPhwli5dyrJlyygrKyv457I9DmUzs12ke/fuNR/c/fr1K+iSil/4whfYb7/9KCkpoUOHDpx66qnA9i+pOGTIEN555x3eeustHnjgAX7yk59QVlbG0KFDef/992tmg/mXMsz36KOPcsYZZ7DvvvvSvn17/v7v/36nZoCf+tSnOOWUU7a7zm+//TZvvfUWJ5xwAkDNpRcb8sQTT7By5UoGDRpEWVkZM2bMoL6vXT7uuON47LHHmDdvHv/yL//CvHnzmD9/fs0FLBYsWMBXvvKVmtd99NFHa9qOGjWq5vvBH3vssZqfb/74jjnmGKZNm8aVV17J8uXLa66MtTN8TNnMbBepe0nF6t3XhVxSEXKXUcy/xOL2LqlY/Tgi+P3vf/+xKyUtXLiwWS+pmD/+uuuQfxnI/MtI5r/e9q4/tL1+I4IvfvGL/O53v2twXIMHD66ZHY8cOZKrr74aSTV/JNSVP45CLvU4ZMgQ5s2bx3/9139x3nnnMWnSJMaMGdPgmBrjmbKZWQsrLf3rJRVnz569Q31UX1Lx0UcfpUOHDnTo0IHhw4fzH//xHzVB+9RTTzXaz5AhQ/jjH//I5s2b2bRpE3fddVfNzLKh8S9dupSPPvqIl156qeaYdiEOOOAAOnToUDNLrb6kYkP9DhgwgMcee4zKykoANm/ezOrVq+tdl9tvv50ePXrQqlUrOnXqxH333cegQYOA3Ez6zjvvrHnd6qtN1TVo0KBa9apV7wq/6KKLGD9+fLNc6tEzZTOzFvad73yHf/iHf+C2227jxBNP3KE+OnbsyHHHHcc777zDb37zGwB+8IMf8K1vfYs+ffoQEZSWlnLvvfc22E/fvn0ZN25czeUML7zwQo4++ugG2wwaNIju3bvTu3dvevXqRd++fZs09mnTpnHBBRewzz77MHz48Eb7LSkpYfr06Zxzzjls2bIFgKuuuqrmko7Vqk+YGzJkCADHH388VVVVdOzYEcj9i9cFF1zANddcQ0lJCdOmTat3fNdffz1f+cpXuP766znzzDNryh955BGuueYa2rZtS/v27ZvlX7wKunRjFpWXl0dz/l+evz+5uO3qzdjbS/HxpRuLw5o1azjllFN45plnWnoozaLZL91oZmZmu4ZD2czMMqO0tHS3mSXvCIeymZlZRjiUzWyPkPvPmuI8h8aK046cs+VQNrM9QmXl3mzd+gYOZtsVIoI33niDvffeu0nt/C9RZrZHuPLKrlx5ZRWHHbaOVataejS2J9h7773p2rVrk9o4lM1sj7BhQ1suvTR3daMi/U9Q2wN497WZmVlGOJTNzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMcymZmZhnhUDYzM8uIRkNZ0uclLc27vSPpW5I6SXpQ0nPpvmNem8slVUp6VtLwvPJ+kpan56ZIuavSStpL0sxUvlBS6SexsmZmZlnWaChHxLMRURYRZUA/YDNwF3AZMCciegBz0mMkHQmMBnoCI4Cpklqn7m4EJgA90m1EKh8PbIiIw4DJwNXNs3pmZmbFo6m7r4cBf4mItcBIYEYqnwGcnpZHAndGxJaIeAGoBPpLOgTYPyIWRO7SGbfWaVPd12xgWPUs2szMbE/R1FAeDfwuLR8cEa8CpPuDUnkX4KW8NlWprEtarlteq01EbAXeBjrXfXFJEyRVSKpYt25dE4duZmaWbQWHsqRPAacB/9lY1XrKooHyhtrULoi4OSLKI6K8pKSkkWGYmZkVl6bMlL8EPBkRr6XHr6Vd0qT711N5FdAtr11X4JVU3rWe8lptJLUBOgBvNmFsZmZmRa8poXwOf911DXAPMDYtjwXuzisfnc6o7k7uhK5FaRf3RkkD0vHiMXXaVPd1FvBwOu5sZma2xyjoesqS9gG+CPxjXvFPgFmSxgMvAqMAImKFpFnASmArcHFEbEttJgLTgXbA/ekGcAtwm6RKcjPk0TuxTmZmZkVJxTohLS8vj4qKimbrz+d6F7ddvRl7eyluRfqxZ7sJSUsiory+5/yNXmZmZhnhUDYzM8sIh7KZmVlGOJTNzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMcymZmZhnhUDYzM8sIh7KZmVlGOJTNzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMcymZmZhnhUDYzM8sIh7KZmVlGOJTNzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMKCmVJB0iaLel/JK2SNFBSJ0kPSnou3XfMq3+5pEpJz0oanlfeT9Ly9NwUSUrle0mamcoXSipt7hU1MzPLukJnytcDf4qIw4GjgFXAZcCciOgBzEmPkXQkMBroCYwApkpqnfq5EZgA9Ei3Eal8PLAhIg4DJgNX7+R6mZmZFZ1GQ1nS/sAQ4BaAiPggIt4CRgIzUrUZwOlpeSRwZ0RsiYgXgEqgv6RDgP0jYkFEBHBrnTbVfc0GhlXPos3MzPYUhcyUPwusA6ZJekrSryXtCxwcEa8CpPuDUv0uwEt57atSWZe0XLe8VpuI2Aq8DXTeoTUyMzMrUoWEchugL3BjRBwNbCLtqt6O+ma40UB5Q21qdyxNkFQhqWLdunUNj9rMzKzIFBLKVUBVRCxMj2eTC+nX0i5p0v3refW75bXvCrySyrvWU16rjaQ2QAfgzboDiYibI6I8IspLSkoKGLqZmVnxaDSUI+J/gZckfT4VDQNWAvcAY1PZWODutHwPMDqdUd2d3Aldi9Iu7o2SBqTjxWPqtKnu6yzg4XTc2czMbI/RpsB63wB+K+lTwPPA+eQCfZak8cCLwCiAiFghaRa54N4KXBwR21I/E4HpQDvg/nSD3Elkt0mqJDdDHr2T62VmZlZ0VKwT0vLy8qioqGi2/nyud3Hb1Zuxt5fiVqQfe7abkLQkIsrre87f6GVmZpYRDmUzM7OMcCibmZllhEPZzMwsIxzKZmZmGeFQNjMzywiHspmZWUY4lM3MzDLCoWxmZpYRDmUzM7OMcCibmZllhEPZzMwsIxzKZmZmGeFQNjMzywiHspmZWUY4lM3MzDLCoWxmZpYRDmUzM7OMcCibmZllhEPZzMwsIxzKZmZmGeFQNjMzywiHspmZWUY4lM3MzDLCoWxmZpYRBYWypDWSlktaKqkilXWS9KCk59J9x7z6l0uqlPSspOF55f1SP5WSpkhSKt9L0sxUvlBSafOuppmZWfY1Zab8hYgoi4jy9PgyYE5E9ADmpMdIOhIYDfQERgBTJbVObW4EJgA90m1EKh8PbIiIw4DJwNU7vkpmZmbFaWd2X48EZqTlGcDpeeV3RsSWiHgBqAT6SzoE2D8iFkREALfWaVPd12xgWPUs2szMbE9RaCgH8ICkJZImpLKDI+JVgHR/UCrvAryU17YqlXVJy3XLa7WJiK3A20Dnpq2KmZlZcWtTYL1BEfGKpIOAByX9TwN165vhRgPlDbWp3XHuD4IJAIceemjDIzYzywjv9ytu8bE0+uQUNFOOiFfS/evAXUB/4LW0S5p0/3qqXgV0y2veFXgllXetp7xWG0ltgA7Am/WM4+aIKI+I8pKSkkKGbmZmVjQaDWVJ+0rar3oZOBl4BrgHGJuqjQXuTsv3AKPTGdXdyZ3QtSjt4t4oaUA6XjymTpvqvs4CHk7Hnc3MzPYYhey+Phi4K5131Qa4IyL+JGkxMEvSeOBFYBRARKyQNAtYCWwFLo6IbamvicB0oB1wf7oB3ALcJqmS3Ax5dDOsm5mZWVFRsU5Iy8vLo6Kiotn68zGf4rarN2NvL8XN24s1RXNvL5KW5P17cS3+Ri8zM7OMcCibmZllhEPZzMwsIxzKZmZmGeFQNjMzywiHspmZWUY4lM3MzDLCoWxmZpYRDmUzM7OMcCibmZllhEPZzMwsIxzKZmZmGeFQNjMzywiHspmZWUY4lM3MzDLCoWxmZpYRDmUzM7OMcCibmZllhEPZzMwsIxzKZmZmGeFQNjMzywiHspmZWUY4lM3MzDLCoWxmZpYRDmUzM7OMKDiUJbWW9JSke9PjTpIelPRcuu+YV/dySZWSnpU0PK+8n6Tl6bkpkpTK95I0M5UvlFTafKtoZmZWHJoyU74UWJX3+DJgTkT0AOakx0g6EhgN9ARGAFMltU5tbgQmAD3SbUQqHw9siIjDgMnA1Tu0NmZmZkWsoFCW1BX4MvDrvOKRwIy0PAM4Pa/8zojYEhEvAJVAf0mHAPtHxIKICODWOm2q+5oNDKueRZuZme0pCp0p/xz4LvBRXtnBEfEqQLo/KJV3AV7Kq1eVyrqk5brltdpExFbgbaBzwWthZma2G2g0lCWdArweEUsK7LO+GW40UN5Qm7pjmSCpQlLFunXrChyOmZlZcShkpjwIOE3SGuBO4ERJtwOvpV3SpPvXU/0qoFte+67AK6m8az3ltdpIagN0AN6sO5CIuDkiyiOivKSkpKAVNDMzKxaNhnJEXB4RXSOilNwJXA9HxFeBe4CxqdpY4O60fA8wOp1R3Z3cCV2L0i7ujZIGpOPFY+q0qe7rrPQaH5spm5mZ7c7a7ETbnwCzJI0HXgRGAUTECkmzgJXAVuDiiNiW2kwEpgPtgPvTDeAW4DZJleRmyKN3YlxmZmZFScU6IS0vL4+Kiopm68/nehe3Xb0Ze3spbt5erCmae3uRtCQiyut7zt/oZWZmlhEOZTMzs4xwKJuZmWWEQ9nMzCwjHMpmZmYZ4VA2MzPLCIeymZlZRjiUzczMMsKhbGZmlhEOZTMzs4xwKJuZmWWEQ9nMzCwjHMpmZmYZ4VA2MzPLCIeymZlZRjiUzczMMsKhbGZmlhEOZTMzs4xwKJuZmWWEQ9nMzCwjHMpmZmYZ4VA2MzPLCIeymZlZRjiUzczMMsKhbGZmlhGNhrKkvSUtkrRM0gpJ/5bKO0l6UNJz6b5jXpvLJVVKelbS8LzyfpKWp+emSFIq30vSzFS+UFJp86+qmZlZthUyU94CnBgRRwFlwAhJA4DLgDkR0QOYkx4j6UhgNNATGAFMldQ69XUjMAHokW4jUvl4YENEHAZMBq5uhnUzMzMrKo2GcuS8mx62TbcARgIzUvkM4PS0PBK4MyK2RMQLQCXQX9IhwP4RsSAiAri1TpvqvmYDw6pn0WZmZnuKgo4pS2otaSnwOvBgRCwEDo6IVwHS/UGpehfgpbzmVamsS1quW16rTURsBd4GOtczjgmSKiRVrFu3rrA1NDMzKxIFhXJEbIuIMqAruVlvrwaq1zfDjQbKG2pTdxw3R0R5RJSXlJQ0NmwzM7Oi0qSzryPiLeARcseCX0u7pEn3r6dqVUC3vGZdgVdSedd6ymu1kdQG6AC82ZSxmZmZFbtCzr4ukXRAWm4HnAT8D3APMDZVGwvcnZbvAUanM6q7kzuha1Haxb1R0oB0vHhMnTbVfZ0FPJyOO5uZme0x2hRQ5xBgRjqDuhUwKyLulbQAmCVpPPAiMAogIlZImgWsBLYCF0fEttTXRGA60A64P90AbgFuk1RJboY8ujlWzszMrJioWCek5eXlUVFR0Wz9+Vzv4rarN2NvL8XN24s1RXNvL5KWRER5fc/5G73MzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMcymZmZhnhUDYzM8sIh7KZmVlGOJTNzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMcymZmZhnhUDYzM8sIh7KZmVlGOJTNzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMcymZmZhnhUDYzM8sIh7KZmVlGOJTNzMwyotFQltRN0lxJqyStkHRpKu8k6UFJz6X7jnltLpdUKelZScPzyvtJWp6emyJJqXwvSTNT+UJJpc2/qmZmZtlWyEx5K/DPEXEEMAC4WNKRwGXAnIjoAcxJj0nPjQZ6AiOAqZJap75uBCYAPdJtRCofD2yIiMOAycDVzbBuZmZmRaXRUI6IVyPiybS8EVgFdAFGAjNStRnA6Wl5JHBnRGyJiBeASqC/pEOA/SNiQUQEcGudNtV9zQaGVc+izczM9hRNOqacdisfDSwEDo6IVyEX3MBBqVoX4KW8ZlWprEtarlteq01EbAXeBjrX8/oTJFVIqli3bl1Thm5mZpZ5BYeypPbA74FvRcQ7DVWtpywaKG+oTe2CiJsjojwiyktKShobspmZWVEpKJQltSUXyL+NiD+k4tfSLmnS/eupvArolte8K/BKKu9aT3mtNpLaAB2AN5u6MmZmZsWskLOvBdwCrIqI6/KeugcYm5bHAnfnlY9OZ1R3J3dC16K0i3ujpAGpzzF12lT3dRbwcDrubGZmtsdoU0CdQcB5wHJJS1PZvwA/AWZJGg+8CIwCiIgVkmYBK8mduX1xRGxL7SYC04F2wP3pBrnQv01SJbkZ8uidXC8zM7Oio2KdkJaXl0dFRUWz9edzvYvbrt6Mvb0UN28v1hTNvb1IWhIR5fU952/0MjMzywiHspmZWUY4lM3MzDLCoWxmZpYRDmUzM7OMcCibmZllRCH/p7xHiHq/6dOKR3H+a5+ZWT7PlM3MzDLCoWxmZpYRDmUzM7OMcCibmZllhEPZzMwsIxzKZmZmGeFQNjMzywiHspmZWUY4lM3MzOuDgVgAAArFSURBVDLCoWxmZpYRDmUzM7OMcCibmZllhEPZzMwsIxzKZmZmGeFQNjMzywiHspmZWUY4lM3MzDKi0VCW9BtJr0t6Jq+sk6QHJT2X7jvmPXe5pEpJz0oanlfeT9Ly9NwUSUrle0mamcoXSipt3lU0MzMrDoXMlKcDI+qUXQbMiYgewJz0GElHAqOBnqnNVEmtU5sbgQlAj3Sr7nM8sCEiDgMmA1fv6MqYmZkVs0ZDOSLmAW/WKR4JzEjLM4DT88rvjIgtEfECUAn0l3QIsH9ELIiIAG6t06a6r9nAsOpZtJmZ2Z5kR48pHxwRrwKk+4NSeRfgpbx6VamsS1quW16rTURsBd4GOu/guMzMzIpWm2bur74ZbjRQ3lCbj3cuTSC3C5xDDz10R8Zn1iyi3s3Wike9HzFmLW5HQ/k1SYdExKtp1/TrqbwK6JZXryvwSirvWk95fpsqSW2ADnx8dzkAEXEzcDNAeXm5f6vMrCj4j7hit+viZkd3X98DjE3LY4G788pHpzOqu5M7oWtR2sW9UdKAdLx4TJ021X2dBTycjjubmZntURqdKUv6HTAUOFBSFXAF8BNglqTxwIvAKICIWCFpFrAS2ApcHBHbUlcTyZ3J3Q64P90AbgFuk1RJboY8ulnWzMzMrMioWCel5eXlUVFR0Xwd+oTv4rart2NvL8XN24s1RTNvL5KWRER5fc/5G73MzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMcymZmZhnhUDYzM8sIh7KZmVlGOJTNzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMcymZmZhnhUDYzM8sIh7KZmVlGOJTNzMwywqFsZmaWEQ5lMzOzjHAom5mZZYRD2czMLCMcymZmZhnhUDYzM8sIh7KZmVlGOJTNzMwyIjOhLGmEpGclVUq6rKXHY2ZmtqtlIpQltQZ+AXwJOBI4R9KRLTsqMzOzXSsToQz0Byoj4vmI+AC4ExjZwmMyMzPbpbISyl2Al/IeV6UyMzOzPUablh5AonrK4mOVpAnAhPTwXUnPfqKj2r0cCKxv6UF8YlTfJmQ7wduLNYW3l6b5zPaeyEooVwHd8h53BV6pWykibgZu3lWD2p1IqoiI8pYehxUHby/WFN5emk9Wdl8vBnpI6i7pU8Bo4J4WHpOZmdkulYmZckRslXQJ8N9Aa+A3EbGihYdlZma2S2UilAEi4j7gvpYex27Mu/2tKby9WFN4e2kmivjY+VRmZmbWArJyTNnMzGyP51DeTUi6T9IBjdT5d0kn7WD/QyXdu2Ojs92BpDWSDkzLjzdDf+Mk3bDzI7OWIOlKSd8psG6ZpL/7pMfUVE1Zh13FoVzklNMqIv4uIt5qqG5E/GtEPLSrxmbZIalZzx+JiOOas789Vfpjd3ojdTJz7s9OKANaNJSrPytbcgyFyPwADSR9W9Iz6fYtSaWSVkmaCjwJdKszi/mBpP+R9KCk31X/JShpuqSz0vIaSf8m6UlJyyUdnsr7S3pc0lPp/vMttd5WW977/itJKyQ9IKldmoU8IelpSXdJ6pjqPyLp/0r6M3BpejxZ0rzUzzGS/iDpOUlX5b3OHyUtSa8xYTtjeTfd/7ukpen2sqRpqfyrkhal8l+m77dH0vmSVqcxDfqkf2bFqp73rp+kP6f35b8lHZJXr5D3tNZnSCq7WtLX8+pcKemf0/IkSYvTNvVveXW+r9yFgx4C6v1skDQqvc6yNK5PAf8OnJ22h7MldUrb2dNp2+2TN4bbJD2c1uGiVD5V0mlp+S5Jv0nL46vXczvrWN9nZb3rIOmbklamMd25k2/hjosI3zJ8A/oBy4F9gfbACuBo4CNgQF69NeS+VaccWAq0A/YDngO+k+pMB87Kq/+NtPx14NdpeX+gTVo+Cfh9Wh4K3NvSP489+QaUAluBsvR4FvBV4GnghFT278DP0/IjwNS89o8AV6flS8l9Qc8hwF7kvsCnc3quU7pvBzyTV74GODAtv1tnbB3SOPoBRwD/D2ibnpsKjEmv9SJQAnwKeAy4oaV/ri38ng4FptdTXvPeAW2Bx4GS9Phscv82WtB72sBnyNHAn/NecyVwKHAyubOpRW7idi8wJK+ffdLnRGX1Z0udsS8HuqTlA9L9uPz3GvgP4Iq0fCKwNC1fCSxL296B5L5++dPkvrvimlRnEfBEWp4GDG9gHUvJ+6xsaB3Sz26v/HG3xG132C2yuzseuCsiNgFI+gMwGFgbEU9sp/7dEfFeqv//Guj7D+l+CfD3abkDMENSD3Jfddp251fBmtELEbE0LS8BPkfuA+TPqWwG8J959WfWaV/9pTzLgRUR8SqApOfJfaveG8A3JZ2R6nUDeqTyekkS8FtgckQsUe47B/oBi3NP0Q54HTgWeCQi1qV2M4G/bcK67zYkLSQXnO2BTpKq39PvRcR/p+Xq9+7zQC/gwfTzbA28mtddY+9pvZ8hETFF0kGSPk3uD6UNEfGipG+SC+anUr/tyW0D+6V+Nqd+tvcFT48B0yXN4q+fMXUdD5wJEBEPS+osqUN6rvrz6z1Jc8ldsGg+8C3lrh64EuiY9hYMBL4JXFDfOqafTf5n5eAG1uFp4LeS/gj8cTvj/sQ5lLNve1+6uqmJ9euzJd1v46/bwg+BuRFxhqRScn+JW3ZsyVveBjR4ch8f306q239Up6+PgDaShpLbQzIwIjZLegTYu5HXuBKoiohp6bGAGRFxeX4lSadTz3fa74ki4ljIHVMGxkXEuHqqVb93Ihe2A7fTXYPvKQ1/JswGzgL+htzV+apf78cR8cv8immXcKPvX0R8TdKxwJeBpZLK6qnW0PUO6r5GRMTLyh2WGQHMAzoB/0Buj83G9Ifh9tT9HdjeOnyZ3B6B04AfSOoZEVsb6PcT4WPK2TcPOF3SPpL2Bc4g91fj9jwKnCppb0ntyW1oTdEBeDktj2vqYG2XexvYIGlwenwe8OcG6jemA7kZ02blzjMY0FBlSacAXyQ3W6k2BzhL0kGpTidJnwEWAkPTrKgtMGonxrkneRYokTQQQFJbST2b0L6hz5A7ye0aPotcQEPumxUvSJ8fSOqS3st5wBnKncewH3BqfS8m6XMRsTAi/pXcRSq6ARvJzbTzx3Ruqj8UWB8R76TnRqbPr87kdu8vTuULgG+ltvOB7+StR6Gfk/Wug3IngHWLiLnAd8n9sdu+3p/mJ8wz5YyLiCeVOztzUSr6NbChgfqL0y6ZZcBaoILcB3ehfkpu9/W3gYd3aNC2q40FbpK0D/A8cP5O9PUn4GuSniYXBvUdIsn3z+SO+S1Kk5V7IuJfJf0f4IH0YfchcHFEPCHpSnIfrq+SO/Gm9U6MdY8QER8od4LmlLSLtw3wc3LHTQtp/7HPkIh4Kj23IoXTy9W7vSPiAUlHAAvSe/ou8NXUz0xy56ysZfuTg2vS4S+R+wNtGblzCS5Lu+l/TG7vyrS0nW0mtw1XWwT8F7nj2z+MiOqLE80HTo6ISklryc2W5ze0jmlvX92fRX3r0Bq4Pf18Re5QTIP/zfJJ8Td67YYktY+Id9OH9DxgQkQ82dLjMjNrSPqj7d2IuLalx9JSPFPePd2cTojYm9yxPQeymVkR8EzZzMwsI3yil5mZWUY4lM3MzDLCoWxmZpYRDmUzM7OMcCibmZllhEPZzMwsI/4/fZVc42Xhdn8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "fig = plt.figure()\n",
    "ax = fig.add_axes([0,0,1,1])\n",
    "labels = ['original', 'normalized', '+removed stopwords']\n",
    "lengthes = [len(tokenized_original), len(tokenized_normalized), len(preprocessed_data)]\n",
    "ax.bar(labels, lengthes, color='b')\n",
    "uq_lengthes = [np.unique(tokenized_original).shape[0], np.unique(tokenized_normalized).shape[0], np.unique(preprocessed_data).shape[0]]\n",
    "ax.bar(labels, uq_lengthes, color='r')\n",
    "ax.legend(labels=['total number of words', 'number of unique words'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def toDict():\n",
    "    words_dict = {}\n",
    "    for word in unique_words:\n",
    "        words_dict[word] = 0\n",
    "    for word in preprocessed_data:\n",
    "        words_dict[word] += 1\n",
    "    return words_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sort_dict(text):\n",
    "    dict1 = toDict()\n",
    "    sorted_values = sorted(dict1.values()) \n",
    "    sorted_dict = {}\n",
    "    for i in sorted_values:\n",
    "        for k in dict1.keys():\n",
    "            if dict1[k] == i:\n",
    "                sorted_dict[k] = dict1[k]\n",
    "                break\n",
    "    return sorted_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'+10': 1, '01072022': 2, '025': 3, '044762': 4, '150': 5, '+1': 6, '120': 7, '24': 8, '1000': 9, '23': 10, '18': 11, '14': 12, '13': 13, '27': 14, 'антиваксеров': 15, '11': 16, '38': 17, 'ага': 18, '16': 19, '15': 20, '0900': 21, '000': 22, '60': 23, '50': 24, '2021': 25, '7': 26, 'близкий': 27, 'бустер': 28, 'волна': 29, '30': 30, 'лекарство': 31, 'жизнь': 32, '12': 33, 'больница': 34, 'взять': 35, '100': 36, 'всем': 37, '10': 38, 'видимо': 39, 'главный': 40, 'бау': 41, '20': 42, '4': 43, 'голиков': 44, 'группа': 45, 'брать': 46, 'воз': 47, 'мало': 48, 'значить': 49, 'мрнк': 50, 'вакцинировать': 51, 'госпитализация': 52, 'любой': 53, 'количество': 54, 'высокий': 55, 'сегодня': 56, 'лгкий': 57, 'данные': 58, 'место': 60, 'понимать': 61, 'именно': 62, 'защита': 63, 'понять': 64, 'интересный': 65, 'выявить': 67, 'вроде': 68, 'доза': 69, '5': 70, 'видеть': 71, 'вариант': 72, 'заражение': 73, '0': 74, 'декабрь': 75, 'почему': 76, 'маска': 77, 'пройти': 78, '19': 79, 'иметь': 80, 'антитело': 81, 'исследование': 82, 'давать': 83, 'анализ': 84, 'каждый': 85, 'второй': 86, '200': 87, 'вс': 88, 'пцр': 89, 'ничто': 91, 'должный': 93, 'болезнь': 94, '+': 95, 'неделя': 96, 'антител': 97, '3': 98, 'время': 101, 'очень': 102, 'заболеть': 105, 'переболеть': 107, 'вирус': 108, '1': 111, 'вопрос': 113, 'думать': 115, 'иммунитет': 121, 'вообще': 123, 'москва': 128, 'прививка': 129, 'сделать': 130, 'свой': 131, 'первый': 132, 'врач': 133, 'нужный': 134, 'россия': 135, 'хороший': 137, 'сказать': 139, '2': 141, 'говорить': 144, 'месяц': 146, 'знать': 150, 'случай': 151, 'код': 154, 'пока': 162, 'делать': 169, 'ещ': 171, 'вакцинация': 181, 'болеть': 182, 'коронавирус': 185, 'новый': 203, 'штамм': 206, 'день': 239, 'год': 279, 'спутник': 288, 'весь': 306, 'мочь': 346, 'вакцина': 349, 'омикрон': 404, 'это': 903}\n"
     ]
    }
   ],
   "source": [
    "sorted_dict = sort_dict(preprocessed_data)\n",
    "print(sorted_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def most_used_words(dict1):\n",
    "    total_num_words = sum(dict1.values())\n",
    "    most_used = []\n",
    "    word_count = []\n",
    "    total_most_used = 0\n",
    "    word_iterator = iter(sorted(dict1.items(), key=lambda x: x[1], reverse=True))\n",
    "    while total_most_used < total_num_words*0.8:\n",
    "        word_tuple = next(word_iterator)\n",
    "        most_used.append(word_tuple[0])\n",
    "        total_most_used+=word_tuple[1]\n",
    "        word_count.append(word_tuple[1])\n",
    "    return most_used, word_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdkAAAE/CAYAAAAKWrmdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df7ReV13n8fe3SWlpC9ralCltJR2tYMsMipGBkcEirgU6aHEWjMURi6JVB0Z+DEtb14yDw7CUGcdfjOh0ACkMCgVRKoMiRhCqQE1/0bShNDRpk/5K2qZtfif33j1/7O/2nNzcJDfN3c1z0/drrWc9z93P+bHPPvucz9nnefIkSilIkqSFd9zRroAkSccqQ1aSpE4MWUmSOjFkJUnqxJCVJKkTQ1aSpE6WHu0KAJx++ull+fLlR7sakiQdluuuu+6BUsqyA70/ESG7fPlyVq1adbSrIUnSYYmIOw/2vreLJUnqxJCVJKkTQ1aSpE4MWUmSOjFkJUnqxJCVJKkTQ1aSpE4MWUmSOjFkJUnqxJCVJKkTQ1aSpE4MWUmSOpmI/yBgoUXsX1bK418PSdITmyNZSZI6MWQlSerEkJUkqRNDVpKkTgxZSZI6MWQlSerEkJUkqRNDVpKkTgxZSZI6MWQlSerEkJUkqRNDVpKkTgxZSZI6MWQlSerEkJUkqRNDVpKkTgxZSZI6MWQlSerEkJUkqRNDVpKkTgxZSZI6MWQlSerEkJUkqRNDVpKkTgxZSZI6MWQlSepkXiEbEW+OiFsiYnVE/HFEnBgRp0XEZyLi9nw+dTT95RGxNiJui4iX9qu+JEmT65AhGxFnAb8ArCilPBtYAlwMXAasLKWcB6zMv4mI8/P9C4CXAe+OiCV9qi9J0uSa7+3ipcCTI2IpcBJwD3ARcGW+fyXwinx9EfDhUsruUso6YC3wvIWrsiRJi8MhQ7aUcjfwG8BdwL3AI6WUvwKeVkq5N6e5FzgjZzkL2DBaxMYskyTpCWU+t4tPpY5OzwWeDpwcET9+sFnmKCtzLPfSiFgVEas2b9483/pKkrRozOd28fcD60opm0spe4GPA/8SuD8izgTI5005/UbgnNH8Z1NvL++jlHJFKWVFKWXFsmXLjmQbJEmaSPMJ2buA50fESRERwEuANcDVwCU5zSXAJ/L11cDFEXFCRJwLnAdcu7DVliRp8i091ASllC9HxMeA64Ep4AbgCuAU4KqIeB01iF+V098SEVcBt+b0ry+lTHeqvyRJEytK2e/j0sfdihUryqpVqxZseTHHp8ITsJmSpGNMRFxXSllxoPf9xSdJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRO5hWyEfGNEfGxiPhqRKyJiBdExGkR8ZmIuD2fTx1Nf3lErI2I2yLipf2qL0nS5JrvSPZ3gL8spTwLeA6wBrgMWFlKOQ9YmX8TEecDFwMXAC8D3h0RSxa64pIkTbpDhmxEPBV4EfBegFLKnlLKw8BFwJU52ZXAK/L1RcCHSym7SynrgLXA8xa64pIkTbr5jGT/KbAZ+MOIuCEi3hMRJwNPK6XcC5DPZ+T0ZwEbRvNvzDJJkp5Q5hOyS4HnAr9fSvlOYDt5a/gAYo6yst9EEZdGxKqIWLV58+Z5VVaSpMVkPiG7EdhYSvly/v0xaujeHxFnAuTzptH054zmPxu4Z/ZCSylXlFJWlFJWLFu27LHWX5KkiXXIkC2l3AdsiIhnZtFLgFuBq4FLsuwS4BP5+mrg4og4ISLOBc4Drl3QWkuStAgsned0/wH4UEQ8CbgD+ElqQF8VEa8D7gJeBVBKuSUirqIG8RTw+lLK9ILXXJKkCTevkC2l3AismOOtlxxg+ncA7ziCekmStOj5i0+SJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdGLKSJHViyEqS1IkhK0lSJ4asJEmdzDtkI2JJRNwQEZ/Mv0+LiM9ExO35fOpo2ssjYm1E3BYRL+1RcUmSJt3hjGTfCKwZ/X0ZsLKUch6wMv8mIs4HLgYuAF4GvDsilixMdSVJWjzmFbIRcTbwr4H3jIovAq7M11cCrxiVf7iUsruUsg5YCzxvYap7ZCL2f0iS1Mt8R7K/DfwiMDMqe1op5V6AfD4jy88CNoym25hl+4iISyNiVUSs2rx582FXXJKkSXfIkI2IlwObSinXzXOZc40Py34FpVxRSllRSlmxbNmyeS5akqTFY+k8pvke4Icj4geBE4GnRsT/Be6PiDNLKfdGxJnAppx+I3DOaP6zgXsWstKSJC0GhxzJllIuL6WcXUpZTv1C09+UUn4cuBq4JCe7BPhEvr4auDgiToiIc4HzgGsXvOaSJE24+YxkD+TXgasi4nXAXcCrAEopt0TEVcCtwBTw+lLK9BHXVJKkRSZK2e/j0sfdihUryqpVqxZseXN9a7iUA5dLkvRYRMR1pZQVB3rfX3ySJKkTQ1aSpE4MWUmSOjFkJUnqxJCVJKkTQ1aSpE4MWUmSOjFkJUnqxJCVJKkTQ1aSpE6O5LeLjxmzf27Rn1qUJC0EQ/YgDF9J0pHwdrEkSZ0YspIkdWLISpLUiSErSVInhqwkSZ0YspIkdWLISpLUiSErSVInhqwkSZ0YspIkdWLISpLUib9dfJhm/54x+JvGkqS5OZKVJKkTQ1aSpE4MWUmSOjFkJUnqxC8+LRC/ECVJms2RrCRJnRiykiR14u3izryNLElPXI5kJUnqxJCVJKkTQ1aSpE4MWUmSOjFkJUnqxJCVJKkTQ1aSpE4MWUmSOjFkJUnqxJCVJKkTQ1aSpE787eKjxN80lqRjnyNZSZI6MWQlSerEkJUkqRNDVpKkTgxZSZI6MWQlSerEkJUkqZNDhmxEnBMRn42INRFxS0S8MctPi4jPRMTt+XzqaJ7LI2JtRNwWES/tuQGSJE2q+Yxkp4D/WEr5duD5wOsj4nzgMmBlKeU8YGX+Tb53MXAB8DLg3RGxpEflJUmaZIcM2VLKvaWU6/P1VmANcBZwEXBlTnYl8Ip8fRHw4VLK7lLKOmAt8LyFrvixKmL/hyRpcTqsz2QjYjnwncCXgaeVUu6FGsTAGTnZWcCG0Wwbs0xHwPCVpMVn3iEbEacAfwK8qZTy6MEmnaNsv1/ljYhLI2JVRKzavHnzfKshSdKiMa+QjYjjqQH7oVLKx7P4/og4M98/E9iU5RuBc0aznw3cM3uZpZQrSikrSikrli1b9ljrL0nSxJrPt4sDeC+wppTym6O3rgYuydeXAJ8YlV8cESdExLnAecC1C1dlSZIWh/n8V3ffA7wGuDkibsyyXwZ+HbgqIl4H3AW8CqCUcktEXAXcSv1m8utLKdMLXnNJkibcIUO2lHINc3/OCvCSA8zzDuAdR1AvSZIWPf/T9kVu9reM/Y/fJWlyGLLHqAOFr6EsSY8ff7tYkqRODFlJkjrxdrHm/PWoUg5cLkmaH0eykiR1YshKktSJt4t12LyNLEnz40hWkqROHMlqwTjClaR9GbLqbr7fXjaQJR1rDFlNHMNX0rHCkNWiYfhKWmwMWS1qfg4saZIZsjom+StWkiaBIStx+KFsWEuaD0NWWkCH818MGtTSsc+QlSaMo2fp2GHISouc4StNLkNWOkYZvtLR528XS5LUiSNZ6QnGn7mUHj+GrKSDMnylx86QlXTY/Aa0ND+GrKTuDF89URmyko4aR8Q61hmykhYNv7SlxcaQlXTM8mcudbQZspJ0EN7S1pEwZCVpAfk/OmnMkJWkCeSt7mODIStJxyhH1UefIStJOqgjDeuDjcKPdYasJOmoOdZH24asJGnRm9RQ9r+6kySpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpE0NWkqRODFlJkjoxZCVJ6sSQlSSpk24hGxEvi4jbImJtRFzWaz2SJE2qLiEbEUuA3wN+ADgfeHVEnN9jXZIkTapeI9nnAWtLKXeUUvYAHwYu6rQuSZImUq+QPQvYMPp7Y5ZJkvSEsbTTcmOOsrLPBBGXApfmn9si4rYO9TgdeKCub/7lsX/te5Y/pjpOSN0PVP6Eq/uByq37E6LuBypfzNv0RKr7kXrGQd8tpSz4A3gB8OnR35cDl/dY1yHqsWrSyyepLm7TZNXFui/u8kmqi3U/cHnvR6/bxf8AnBcR50bEk4CLgas7rUuSpInU5XZxKWUqIt4AfBpYAryvlHJLj3VJkjSpen0mSynlU8Cnei1/nq5YBOWTVJeFKp+kuhxu+STV5XDLJ6kuh1s+SXVZqPJJqsvhlk9SXQ63/HCX0VXkvWpJkrTA/FlFSZI6MWQnWER8c0R8MCKujYjVEXH6Y1jGByJiVUR8sEcde4uICyLiC9kGr57j/XMi4rMRsSYibomINz7O9XtxRHwxIr4UES9+rNNFxDsj4h8i4vrc1msm5VfSIuJ9EbEpIlYf4P2fzjo/GBEbH2tfnWddTsx13ZT7+1ez/IiPlZ4O1E8j4rSI+ExE3J7Pp47muTx/lva2iHjpYa7vaRGxMvvUmxd4W/brDxHxnOzfN0fEn0fEU0fvzesYOcw6vDIitkTEroi4LyL++UIst4uj8ZVmH4d+ACcCfwdcSN7W9zFnO50JPDdfPwX4GnD+0a7XUWqLfwb8CfBl6jf8lyzQcl8EPBdYPcd7rwP+EPjS49FXqf8G/5R8fXxu64sey7GS07//cdo3c/XTtwL3AOupPz97GfDOnOZ84CbgBOBc4OsLtT9HdboA+AJwLfDqI+kP2d++N1//FPD2jm25BNgB/HyeJ2+a5GP+qFdgHg36FmB1Pt4ELKf+sMXPAT8OXAfsAdYA3ws8RP21qZ3AXTnt6cCPZdnNwF9k+eW5vHuAGWAKuC9fz4zKZoCtwGbg7tzB7wHeAazL93fl8zrg0ZzvLcA1wHTW/wt5cP2fXP/2nG4vsJv6j5q353vTWbYV2JJlDwB35gHX1vlDWTaT0zw8mn4GuBXYlsvdAfw34PeBG7PNSs5/Y7bDV3O+7Tnv+qzDtwC3jLbzT4FvzWXuzedp4N7cB5tzuuncxmuAK7PsZuDjWa/rc/5d+dgN3J7bMT3ajjtH+2dv7oed+bgDuCGXNw3cP5p3NbAp5yn52Jvb2vrB3Vk2NVpfGT12ZN3avprObZzKfVFmPe4evW5tuYH686Jbs3wP8N2jda7Lsjbfo7PW/+l8vYsaKN/F0E/2jOb5nVnr3ksNwLm26858f+9oWTPs21btsXnW/K1/jvfTeB33ZH22jMo2Z1u046rMmu+r1P7QyrZTj+Edc9R9V653mtpf2nHX6n4X8OfU/rE32/1G4L/meWVmtIyvZdlzc7qZXOcW6jFyTe7nm6n9rAA/mNM9ksu9P9fzx7nudQx9tfW51hfG/Wwm2+XnqOeivaM6fyHrtYvah79EPR7XZfnnch/M5PM7qMfAcuqv7G1hOCYezXqWbOeduS0bsu5/ndMW6nHx3lGbt/PTnaN9uHf0/oY59k97jPt06yc7Zi27lX8q69Gm2T3H8m7Ourf5H8jpbsh91La1nTsvyDa7AVibbbIp2++FuS3X5+OXgfdTL742AK/M6a7Kdb082/DuXP/OXP9BL36OeogeImC/Kxv1ZOAU6kn+O6kH1d8ynGB3ZUf5T9SD+64sW5vl/yJ35teoQTxFPTn8LfVgKNSD6bPZ6HuAt+fzHoaw/lLujBlqYN7FcFC+MKfZOeoQW0evPzDqOHdm2UOj92dG27KDevAW6pX6lQwnlq3UA7mdDN9JPQC/nO8/ktu2Let+H8OFQuvUf8VwginAJ/J5fbZtOzhmctn3U8NzJjvY5tzOB3PZ45NGW+5U1vVW9j9QdmYbbKKOvPbkfP8q98PXs52mcxlXMxyM7cCbBr4P+O38+8zc1pL7uZ2sP8xwwN2R23I38Opcxkz2tW/K6XZRT1DbqQdUu6h4KJ9bO+wa1aOwb7CPT0BX5T5pwdK2o/XH8UlofBEwvsAowMcYTpjjNn84t6edXNpF2q0MfejRfP8PRu3fLoBWA+/KurR+8uBouz+Q89yR7+/K+v0RQx/5+Gi/35blt1EDfypft4ufD+U878vlbgH+PcNFy/Wj9nlTbt9eajC07diT9Wnt0y62/hpYxb79cZp6Trgd+H/ZJi2Y3gD8Za67bUub74ez7JF87MxtbAHz1nx/B/Ui/jeyfBv1PPK/qCPEB4Dd2cfaxdfGWfu39dvWt2/KZRTgv8+qV9vuB9i3z82MXq8ZlbXQ/muGvta2v11IzA6zndTzW1vGFupF9QbqBccuah9pbfYPuW/aRcTNo2XdkPXanPuwLfPzuY5PMlyIvIvhwuM44NcZwvdd+fo3gXcz9PG7c3vHx97dozp/Mtt3LXXQc2PW9UXAs4Bbct+cRz3vvJ9RyFJHyhtyv54OfHu25eqc793ATyzmkH0jeeWZf78d+AXqieFK6oFzH8NI9G+oB/HPZ/lmanD8aHaoNsL9AvXAuXLUWd5A7dit47Qr9zYCnMn3WnC2MGyh2sLpXoYDYXwSbieITcBPMJxg2gHU1ruVfU/Su3PHj8N4KudtYTweSU9nWVt2W86O0fv35/atzHW1i4j11AuP1oFbKNyUrzcBX2QI+R1Z7zaa2wN8lOGg3TPa/nYAtO3+fL63lv1HcNO5X26jHiTPmbWcknW+kPo/PbW/27ZupN7GHO+Dbbmu63Nbxhc4NzKcKMb7cq6r89llD8xRt22zpr+PIQjGAdBG/+PRaOtPrX7tIrDdQbkhn2+fY99uG00/uz1bcM3ehjZKH/eVVv9phpDbzv5t0R6/mvt0b07/2Xx9aZZ/aLS8diHzXoaLhXHAzW7L9mgh2/rWeDTT6r6Deo5odf0ANUTvoY7+t1FPrptG843bZtdofe2iYYY6Im6BtjX3+Q+O1r+aYSTbQraN+Mb1HR+/bZ+1C7et1P45k8t7kKFftPW0+R7Kdl7Nvhedre1a/237dy/1WG/npZ/JadsF09asa9v+TdS7fG0E+77c5vEdnd0Mffna3IZ2vrqB/Y+T3ex/nE9lu7TydpE3Qz0md7B/P9g9Km8XTL/L0I/aheB7qYOHB3O6doHaLjB+mfrxSqtD60/vp55X9lCPpbuzrR+mhuwbcl/vzDreBrztYDk26V98OtgvTH6U+lnFicA3MJzonwv8CnAGdfS7JJezBDibGiLPyL8/CpyUy7tvtOwp6shhC/UzkeOpB/kUdadBvdLaMZrnJ/P5xFz++CRcqAc7wKnUEwHUK7Dm+Hw+Oev7QP79JIb/wagd9LuoOxnqCWQXNTx2Zb0eHm1HC90l+bwVOI3aYV5M/bfS4/+8ofWJyNdLR8s9lRp4L8z3T2Tf235Q27hdgR6f87XR1/2jNvrufG9b1m0m3/9Ulj80qtNb832oJwuAp1P3zX9haOd2m/zknK4F7nbqSO3J1BHrXmqozgDTpZTvoH708JSs65n53O4OQD0pkPNtYrjYmsr6thPENLVPTeV6NwFPy20+keGkOJV/t33a/s16ez55tJ1QT9p7czuD2i8i26/1hZOo+wjqSWJlblO7WGrbsm7UnlsZRmnbRuVtW67O7TuJYbS8k3qHoF2QPYt9tRPWs3K+Fh6tHU/I8hY4b6H2460MJ/HtpZSgjigL9RboydQ+VbKdpnKZ0/lcgF+k7ufZluW8v8vQ37ZSL8yvowZQO1Z2UPv4e3KZ3zda13HZljCMdL+VekyNf3fgI/ncyl7FsO+Oo/b1yGUsoZ6rzsqy11NHxtPUOz4wHPc35zxLqR/htGNqN3N/kfXJOe+51OMD4NcY+lALNRjuJiwF/m3W60nAa6n77EFq37kzn1ub/CZDf53KdbVzd/t44EnUC2EYQvahXFdrox053zR1BP+FLP8Sw8j6AWo/Jcu2UM/57Ry1hNpnX5zrbOelDdQLrWdSLwieTb0Qfzjr/pTRtFDb9e057a+NyoN69+3rpZTvKKU8s5TyNg5i0kP288ArIuKkiDgZ+BGGhl9Nbdh/B/wsNdimqSeuR6gHbevI66gNfh/1pHUv9YBbzRAGz8nltgPoAYaGf5R6kK8Evi3fn6LuyDY6gOGKsn3rbhpo37Jbmu/fA7wty76VeqBDPTHtHr2+crSeFizt5NA65vG5rdPUk2sL1xYITWRbtYuNvTnN86gHwU+Ppm0nzPdQR7bt1s10rvNXqJ2VnPdEhs8LlzJ8ptLaZEeucwm17Qv1tn+7sn8BwwHyfOrFEfn8dGp7vyyn2UMdibS2WErdL0G9TXhcvt5Cbfeg9pfjqbeDllAPqK25rnH/v5hh9P0wdV/cyHBSPyWn+/tc7wlZflqWr83n8TKXAv85X7er8HYx2Oq6m31tG827m+F/s3omdZ89wnCib33miwwj0BY426l3AZi1PKgXWK2eT81thnohUqgjkbZPThlNO5PtcUKWnZ1l30TtB8dRj7F22/9ncr0tTDZT+wrUfbskl/fkXM9TqBezS4GIiGfkdu+h9sXxaGuKeixBbfvIv28evf+9DBevy7KuL6COTPcAv5Xt9ZSs+2pq/2gXLd/EEBDtYucBhr4Ata+czXBrfOwWhlH1/2YI/23ANzMcIydR9+e6rE9QA+146r6aZgjm3dRj/bis2+k575NG6/3ZfN7D8J2QM6n7qK2/nedaP2yj+7aNpzEcc7+d23ZWbsNZ1IDalNP/HUNQf5zaR8my2xkC9+RcR7tA/wvgg+RgpZTy1FzmUuqF17qcb0vWJ6jHQ9uO5dQL2OdTB0xLsnwN9aLpmdRjYyf1XPtnozZ6Sj72llJmgNewfx6+mXpRNj0qW0m9e7YE/vHb4c/gYI72LeF53DKe64tPq/N5Q77eSO18b8kGvo3httlG6ue47VbxHdSD/SHqZ7i3UW8JbM3GbLdx2i3b9nnhvfloB/G6rF/7Ukm7hdG+JNGCZnr0PJ11+D2G2zJtJLgl69L+Hn8hZetoGesZbo/vYrg1NE29rbM36zi+XdxuU4+3bxv1ZLgN+K+5jPUMJ4VtDAfe+FZqG2m0Uc2jDLd7Wp02zfq7fa7TbkvvyNdTuV+2j6Zdl8tby3Dbcm/ut3b7uY3+djOMDNtnZS3MWjt+jTriarf/pxhu8bb6jD/PabfkCvveXp0ZrafVYWo07/gxvs01viV5NfuO/D/CcMtu9mewe6knq7tHf7d+2Np7XO/WT8afr7XPDNu847Zq8z5MPXmP+/zs+s/epvHjfoZbyq3tt1KPu3ZLr41sN4/WuwP4yqw23sm+Xxrbw3CLuN0ubhdn49vubT+Mv4j2EPXLju3jl5uz7f6MeiG4hWH0vDen/wjDLcU7GO6ObGG4dfvpfL4423Q9+38m2y7Sxt+FONCj7f8p9r3luoV9b1lPZRtsZDgnrR8tf9yvbmI4v/w9w/dMPpXlHxtN2y7W7hnti78ELmH4Dsk7cz+29c7u85eN9vde9r0tPDPr9ezvLexm+GjkLOrnqGX03njedmv9bvY9Ju+mXoy3u4b35TIfoX73Yj21n7aP+Kapd0guzLIvUUerO9n3M9nV1OB9a+7T0/Oc/4ac9ivUuyDPX7SfyR4ifJcz3BdfQ73iOJ165bKKOhJbk9MtZ9+vm19B/bLF8mz07dRwXJM74weyoUtrQOro860Mtzh/ZLS89o2+dgXVTl57GL7A8xnqRUC7ddau8lpH2Zzb8JPUg2MVNSB2U0PuJmDtAbah1X/2NvwWQ3jemB1vJ/VWTAuo9o3rdqC2z4K2M4yS72Do/I8wfD7V1vPn+d7vMHybt33rr4X2WurB8FCWrcz99ZHsrLcCf5Dbdgp1JL86t/vfUEcV1wL/JNv7k8CFOf10bvtXGW75/dRB+s4rgQ8eZn+7EPjk6O9LcjvvoR6Ym3Kf35iPvQxfjjiXeuK6KbfzV6gjo1uo31hczvDZ1Ndy336R4WSxC3hDLiuoFwcP5n770azbVobPuaaAXxodJ+P+MT16/RBwc0433hebsw1XHKJNPnegaYAV5Ldj59OehzjOx8fuK7O9D9R31jOE106GkLqI+mWkFvobqV8E+6Nsh58eHedfz3a9IffDybmta7M+Pwa8d7S+00dtuD5fvzb7xjXU/voTub/aSPyjWY+PUG+/bs92b98deDP1tuQlDB+DPEo9Jtrnj+34bhcVu6n9rv0LiZLraBfp7YK8BXuh3h38HwwfgXxwtH+2MVx03g38UtbpToZg2jZ7f2Yb/j75WWW23Srgte18Odqfb8x2uTnb+k/Ib/U+hjx4NLf99KOdT/vU7WhX4Gg/2P8gfm3rDHNMez41cP7nQZb3NvLkPyp7+aiDXUi9/TJe5yks4L/Xa9twqLrMMd/6uTooNbjXA98213rmmP77qUHwpsOtwzy3b9sByi/OA+3CdoDPMc27qCexbzvMdf7jSWTWvn5rvv7crPfeD3yxU599NnDGrLL3MYT+mw5jWZ+bo+xt1KB5TCFLHeegyMkAAAEPSURBVNncCbzwcNpzgdpm/Rz97Zoj6W8LUKfXHKy95jH/nwF/Oqvs5dSBxGtnlT97dt9fyGMw+/VBQ/AA/fPpwLkLsfw55vlG6nn5kQOdw47mw98unqX9ckgp5SuPcf5nU/8d1qZR2dOBE0op6yLiDGoHHP9aylLgu0opXz6y2v/j8tqvn8wcrC6HsbwfAj5fSnlkVvkh2+pQ7THfOhyO/IzkGaWUz/dY/gHW+X2llL+ZVfY9pZS/O9A8kyAiXlNKWZS/BjaXiPgpaniP+9tFwFd69bfeDnIMfQuwcbxdEfENwDmzzi+P+zGogSErSVInk/7tYkmSFi1DVpKkTgxZSZI6MWQlSerEkJUkqZP/D/5Ig1KSMVO1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['это', 'омикрон', 'вакцина', 'мочь', 'весь', 'спутник', 'год', 'день', 'штамм', 'новый', 'коронавирус', 'болеть', 'вакцинация', 'ещ', 'делать', 'пока', 'код', 'случай', 'знать', 'месяц', 'говорить', '2', 'сказать', 'хороший', 'россия', 'нужный', 'врач', 'первый', 'свой', 'сделать', 'прививка', 'москва', 'вообще', 'иммунитет', 'думать', 'вопрос', '1', 'вирус', 'переболеть', 'заболеть', 'очень', 'время', '3', 'антител', 'неделя', '+', 'болезнь', 'должный', 'ничто', 'пцр', 'вс', '200', 'второй', 'каждый', 'анализ', 'давать', 'исследование', 'антитело', 'иметь', '19', 'пройти', 'маска', 'почему', 'декабрь', '0', 'заражение', 'вариант']\n"
     ]
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_axes([0,0,1,1])\n",
    "most_used, counts = most_used_words(sorted_dict)\n",
    "labels = most_used\n",
    "lengthes = counts\n",
    "ax.bar(labels, lengthes, color='b')\n",
    "plt.show()\n",
    "print(most_used)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Векторное представление слов"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2idx = {w: idx for (idx, w) in enumerate(unique_words)}\n",
    "idx2word = {idx: w for (idx, w) in enumerate(unique_words)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def CBOW(raw_text, window_size=2):  # continuous bag of words\n",
    "    data = []\n",
    "    for i in range(window_size, len(raw_text) - window_size):\n",
    "        context = [raw_text[i - window_size], raw_text[i - (window_size - 1)], raw_text[i + (window_size - 1)], raw_text[i + window_size]]\n",
    "        target = raw_text[i]\n",
    "        data.append((context, target))\n",
    "    return data\n",
    "class CBOW_Model(torch.nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim):\n",
    "        super(CBOW_Model, self).__init__()\n",
    "        self.embeddings = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.linear1 = nn.Linear(embedding_dim, 128)\n",
    "        self.activation_function1 = nn.ReLU()\n",
    "        self.linear2 = nn.Linear(128, vocab_size)\n",
    "      \n",
    "\n",
    "    def forward(self, inputs):\n",
    "        embeds = sum(self.embeddings(inputs)).view(1,-1)\n",
    "        out = self.linear1(embeds)\n",
    "        out = self.activation_function1(out)\n",
    "        out = self.linear2(out)\n",
    "        return out\n",
    "def make_context_vector(context, word2idx):\n",
    "    vec = []\n",
    "    for word in context:\n",
    "        vec.append(word2idx[word])\n",
    "    return torch.tensor(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['подросток', 'поскольку', 'тяжлый', 'заболевание'], 'бремя'), (['поскольку', 'бремя', 'заболевание', 'возрастной'], 'тяжлый'), (['бремя', 'тяжлый', 'возрастной', 'группа'], 'заболевание'), (['тяжлый', 'заболевание', 'группа', 'небольшой'], 'возрастной'), (['заболевание', 'возрастной', 'небольшой', 'высокий'], 'группа'), (['возрастной', 'группа', 'высокий', 'охват'], 'небольшой'), (['группа', 'небольшой', 'охват', 'вакцинация'], 'высокий'), (['небольшой', 'высокий', 'вакцинация', 'среди'], 'охват'), (['высокий', 'охват', 'среди', 'группа'], 'вакцинация'), (['охват', 'вакцинация', 'группа', 'подверженный'], 'среди')]\n"
     ]
    }
   ],
   "source": [
    "cbow = CBOW(preprocessed_data)\n",
    "print(cbow[10:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CBOW_Model(\n",
      "  (embeddings): Embedding(9780, 5)\n",
      "  (linear1): Linear(in_features=5, out_features=128, bias=True)\n",
      "  (activation_function1): ReLU()\n",
      "  (linear2): Linear(in_features=128, out_features=9780, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "model = CBOW_Model(len(unique_words), 5)\n",
    "print(model)\n",
    "loss_function = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d203249796094cdf8491ec347cdde35a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=10.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# training\n",
    "for epoch in tqdm(range(10)):\n",
    "    total_loss = 0\n",
    "    for context, target in cbow:\n",
    "        context_vector = make_context_vector(context, word2idx) \n",
    "        output = model(context_vector)\n",
    "        target = torch.tensor([word2idx[target]])\n",
    "        total_loss += loss_function(output, target)\n",
    "    #optimize at the end of each epoch\n",
    "    optimizer.zero_grad()\n",
    "    total_loss.backward()\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}