recommender-system-using-supervised-gnn_modified.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6698729e",
   "metadata": {},
   "source": [
    "\n",
    "# Recommender System using Supervised Graph Neural Networks\n",
    "\n",
    "## Overview\n",
    "This project implements a recommender system using supervised learning techniques with Graph Neural Networks (GNNs). The approach leverages the inherent structure of the data and supervised learning of **LightGCN** to enhance the recommendation quality without relying on labeled data.\n",
    "\n",
    "## Features\n",
    "- supervised learning with LightGCN (GNN)\n",
    "- Efficient handling of graph-structured data\n",
    "- Enhanced recommendation performance through advanced neural network techniques\n",
    "\n",
    "## Usage\n",
    "To run the project, execute the following command:\n",
    "```sh\n",
    "jupyter notebook recommender-system-using-supervised-gnn.ipynb\n",
    "```\n",
    "\n",
    "## Project Structure\n",
    "- `recommender-system-using-supervised-gnn.ipynb`: Main notebook containing the implementation of the recommender system.\n",
    "\n",
    "## Results\n",
    "The results of the recommender system are evaluated using standard metrics such as Precision, Recall, and F1-score. The supervised approach shows significant improvements over traditional methods.\n",
    "\n",
    "## Contributing\n",
    "Contributions are welcome! Please read the [contribution guidelines](CONTRIBUTING.md) first.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:16.930760Z",
     "iopub.status.busy": "2024-08-09T10:57:16.930366Z",
     "iopub.status.idle": "2024-08-09T10:57:29.062263Z",
     "shell.execute_reply": "2024-08-09T10:57:29.061322Z",
     "shell.execute_reply.started": "2024-08-09T10:57:16.930724Z"
    },
    "id": "SR6d7sjDFZIs",
    "outputId": "0b9011a8-7775-4b26-a2c0-052cbefb5256"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: torch-geometric in /opt/conda/lib/python3.10/site-packages (2.5.3)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (4.66.4)\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (1.26.4)\n",
      "Requirement already satisfied: scipy in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (1.11.4)\n",
      "Requirement already satisfied: fsspec in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (2024.5.0)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (3.1.2)\n",
      "Requirement already satisfied: aiohttp in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (3.9.1)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (2.32.3)\n",
      "Requirement already satisfied: pyparsing in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (3.1.1)\n",
      "Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (1.2.2)\n",
      "Requirement already satisfied: psutil>=5.8.0 in /opt/conda/lib/python3.10/site-packages (from torch-geometric) (5.9.3)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->torch-geometric) (23.2.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.10/site-packages (from aiohttp->torch-geometric) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->torch-geometric) (1.9.3)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from aiohttp->torch-geometric) (1.4.1)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.10/site-packages (from aiohttp->torch-geometric) (1.3.1)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->torch-geometric) (4.0.3)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2->torch-geometric) (2.1.3)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests->torch-geometric) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests->torch-geometric) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests->torch-geometric) (1.26.18)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests->torch-geometric) (2024.7.4)\n",
      "Requirement already satisfied: joblib>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from scikit-learn->torch-geometric) (1.4.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from scikit-learn->torch-geometric) (3.2.0)\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "!pip install torch-geometric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T11:00:02.750227Z",
     "iopub.status.busy": "2024-08-09T11:00:02.749617Z",
     "iopub.status.idle": "2024-08-09T11:00:02.757174Z",
     "shell.execute_reply": "2024-08-09T11:00:02.756080Z",
     "shell.execute_reply.started": "2024-08-09T11:00:02.750197Z"
    },
    "id": "-nTIfY-owmFf"
   },
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "import copy\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import model_selection, metrics, preprocessing\n",
    "from torch_geometric.utils import degree\n",
    "import torch\n",
    "from torch import nn, optim, Tensor\n",
    "# from torch_sparse import SparseTensor, matmul\n",
    "import torch.nn.functional as F\n",
    "from torch_geometric.nn.conv.gcn_conv import gcn_norm\n",
    "from torch_geometric.data import download_url, extract_zip\n",
    "from torch_geometric.utils import structured_negative_sampling\n",
    "from torch_geometric.nn.conv import MessagePassing\n",
    "from torch_geometric.typing import Adj\n",
    "from collections import defaultdict\n",
    "from scipy import sparse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.073242Z",
     "iopub.status.busy": "2024-08-09T10:57:29.073007Z",
     "iopub.status.idle": "2024-08-09T10:57:29.089551Z",
     "shell.execute_reply": "2024-08-09T10:57:29.088587Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.073221Z"
    },
    "id": "RwBZPk_iFjZX",
    "outputId": "bd819320-251c-42a0-a6a9-afb8ed4d6994"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 1 1]\n",
      " [1 0 1 0]\n",
      " [0 1 0 0]]\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "R = np.array([[0,0,1,1],[1,0,1,0],[0,1,0,0]])\n",
    "print(R)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.091988Z",
     "iopub.status.busy": "2024-08-09T10:57:29.091440Z",
     "iopub.status.idle": "2024-08-09T10:57:29.100728Z",
     "shell.execute_reply": "2024-08-09T10:57:29.099772Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.091963Z"
    },
    "id": "bTFwHHIIFwU3",
    "outputId": "f3b415c6-9ffa-489b-bb6d-8e0928596319"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 2)\t1\n",
      "  (0, 3)\t1\n",
      "  (1, 0)\t1\n",
      "  (1, 2)\t1\n",
      "  (2, 1)\t1\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "R_np_COO = sparse.coo_matrix(R)\n",
    "print(R_np_COO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.102070Z",
     "iopub.status.busy": "2024-08-09T10:57:29.101802Z",
     "iopub.status.idle": "2024-08-09T10:57:29.111170Z",
     "shell.execute_reply": "2024-08-09T10:57:29.110286Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.102047Z"
    },
    "id": "oUthOnT4F4og",
    "outputId": "3b7375bf-4ff8-4e13-991c-69182a0b7120"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([0, 0, 1, 1, 2], dtype=int32), array([2, 3, 0, 2, 1], dtype=int32)]\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "R_edge_index = [R_np_COO.row, R_np_COO.col]\n",
    "print(R_edge_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.112501Z",
     "iopub.status.busy": "2024-08-09T10:57:29.112187Z",
     "iopub.status.idle": "2024-08-09T10:57:29.121234Z",
     "shell.execute_reply": "2024-08-09T10:57:29.120362Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.112471Z"
    },
    "id": "Y9R3ccLcGG15",
    "outputId": "e55c404a-a96f-487d-83ff-f238abef520e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 1 1]\n",
      " [1 0 1 0]\n",
      " [0 1 0 0]]\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "print(R_np_COO.todense())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.122776Z",
     "iopub.status.busy": "2024-08-09T10:57:29.122355Z",
     "iopub.status.idle": "2024-08-09T10:57:29.153693Z",
     "shell.execute_reply": "2024-08-09T10:57:29.152835Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.122746Z"
    },
    "id": "j1mSFcuvGOt-",
    "outputId": "86a64086-2afd-4d9f-c7f8-064e3854a8b5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using existing file ml-latest-small.zip\n",
      "Extracting ./ml-latest-small.zip\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "url = 'https://files.grouplens.org/datasets/movielens/ml-latest-small.zip'\n",
    "extract_zip(download_url(url, '.'), '.')\n",
    "\n",
    "movie_path = './ml-latest-small/movies.csv'\n",
    "rating_path = './ml-latest-small/ratings.csv'\n",
    "user_path = './ml-latest-small/users.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.154981Z",
     "iopub.status.busy": "2024-08-09T10:57:29.154723Z",
     "iopub.status.idle": "2024-08-09T10:57:29.252207Z",
     "shell.execute_reply": "2024-08-09T10:57:29.251361Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.154959Z"
    },
    "id": "WyFniUNTZy96",
    "outputId": "a3cae97e-41dc-452d-c737-55cff9d73b92"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              userId        movieId         rating     timestamp\n",
      "count  100836.000000  100836.000000  100836.000000  1.008360e+05\n",
      "mean      326.127564   19435.295718       3.501557  1.205946e+09\n",
      "std       182.618491   35530.987199       1.042529  2.162610e+08\n",
      "min         1.000000       1.000000       0.500000  8.281246e+08\n",
      "25%       177.000000    1199.000000       3.000000  1.019124e+09\n",
      "50%       325.000000    2991.000000       3.500000  1.186087e+09\n",
      "75%       477.000000    8122.000000       4.000000  1.435994e+09\n",
      "max       610.000000  193609.000000       5.000000  1.537799e+09\n",
      "Movies: 9724\n",
      "Users: 610\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "df_movie = pd.read_csv(movie_path)\n",
    "df_rating = pd.read_csv(rating_path)\n",
    "\n",
    "print(df_rating.describe())\n",
    "df_rating.head()\n",
    "\n",
    "print(\"Movies:\", len(df_rating['movieId'].unique()))\n",
    "print(\"Users:\", len(df_rating['userId'].unique()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "p5JfYArzalhN"
   },
   "source": [
    "Label The movies cause movieIds are from 1 to 193609"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.253598Z",
     "iopub.status.busy": "2024-08-09T10:57:29.253313Z",
     "iopub.status.idle": "2024-08-09T10:57:29.272159Z",
     "shell.execute_reply": "2024-08-09T10:57:29.271429Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.253573Z"
    },
    "id": "wbGRQ0BaavXg"
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "user_labels = preprocessing.LabelEncoder()\n",
    "movie_labels = preprocessing.LabelEncoder()\n",
    "\n",
    "df_rating.userId = user_labels.fit_transform(df_rating.userId.values)\n",
    "df_rating.movieId = movie_labels.fit_transform(df_rating.movieId.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.275793Z",
     "iopub.status.busy": "2024-08-09T10:57:29.275416Z",
     "iopub.status.idle": "2024-08-09T10:57:29.299091Z",
     "shell.execute_reply": "2024-08-09T10:57:29.298224Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.275766Z"
    },
    "id": "09aC9Qa5a-RU",
    "outputId": "0cb76af3-1edd-49b4-8482-147d13bc627f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              userId        movieId         rating     timestamp\n",
      "count  100836.000000  100836.000000  100836.000000  1.008360e+05\n",
      "mean      325.127564    3101.735561       3.501557  1.205946e+09\n",
      "std       182.618491    2627.050983       1.042529  2.162610e+08\n",
      "min         0.000000       0.000000       0.500000  8.281246e+08\n",
      "25%       176.000000     900.000000       3.000000  1.019124e+09\n",
      "50%       324.000000    2252.000000       3.500000  1.186087e+09\n",
      "75%       476.000000    5095.250000       4.000000  1.435994e+09\n",
      "max       609.000000    9723.000000       5.000000  1.537799e+09\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "print(df_rating.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.300390Z",
     "iopub.status.busy": "2024-08-09T10:57:29.300145Z",
     "iopub.status.idle": "2024-08-09T10:57:29.308367Z",
     "shell.execute_reply": "2024-08-09T10:57:29.307438Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.300368Z"
    },
    "id": "8eddbo1rbIab",
    "outputId": "63913587-ac8e-4407-e4ec-b87eeb38cc39"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rating\n",
      "4.0    26818\n",
      "3.0    20047\n",
      "5.0    13211\n",
      "3.5    13136\n",
      "4.5     8551\n",
      "2.0     7551\n",
      "2.5     5550\n",
      "1.0     2811\n",
      "1.5     1791\n",
      "0.5     1370\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "print(df_rating.rating.value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "moTuyJUrbmyQ"
   },
   "source": [
    "## Load Edge Indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.309667Z",
     "iopub.status.busy": "2024-08-09T10:57:29.309395Z",
     "iopub.status.idle": "2024-08-09T10:57:29.318300Z",
     "shell.execute_reply": "2024-08-09T10:57:29.317568Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.309644Z"
    },
    "id": "jytaVM0dbNW2"
   },
   "outputs": [],
   "source": [
    "# Define functions\n",
    "def load_edge_csv(df, src_index_col, dest_index_col, link_index_col, threshold=3):\n",
    "  edge_index = None\n",
    "  sources = [user for user in df['userId']]\n",
    "  destinations = [movie for movie in df['movieId']]\n",
    "  link_vals = df[link_index_col].values\n",
    "\n",
    "  edge_attr = torch.from_numpy(df[link_index_col].values).view(-1, 1).to(torch.long) >= threshold\n",
    "  edge_index = [[],[]]\n",
    "  edge_values = []\n",
    "\n",
    "  for i in range(edge_attr.shape[0]):\n",
    "    if edge_attr[i]:\n",
    "      edge_index[0].append(sources[i])\n",
    "      edge_index[1].append(destinations[i])\n",
    "      edge_values.append(link_vals[i])\n",
    "\n",
    "  return edge_index, edge_values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.320207Z",
     "iopub.status.busy": "2024-08-09T10:57:29.319635Z",
     "iopub.status.idle": "2024-08-09T10:57:29.739326Z",
     "shell.execute_reply": "2024-08-09T10:57:29.738588Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.320175Z"
    },
    "id": "eOeI2a4pWhDJ"
   },
   "outputs": [],
   "source": [
    "# Code cell\n",
    "edge_index, edge_values = load_edge_csv(df_rating, 'userId', 'movieId', 'rating', threshold=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.740765Z",
     "iopub.status.busy": "2024-08-09T10:57:29.740392Z",
     "iopub.status.idle": "2024-08-09T10:57:29.745222Z",
     "shell.execute_reply": "2024-08-09T10:57:29.744340Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.740738Z"
    },
    "id": "S3UZhtp8XAgI",
    "outputId": "8bdacfc8-ba43-4a00-b0c0-8ec6760d2f75"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 x 99466\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "print(f'{len(edge_index)} x {len(edge_index[0])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.746547Z",
     "iopub.status.busy": "2024-08-09T10:57:29.746258Z",
     "iopub.status.idle": "2024-08-09T10:57:29.788584Z",
     "shell.execute_reply": "2024-08-09T10:57:29.787779Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.746500Z"
    },
    "id": "KVlJN_guXHC1",
    "outputId": "86e91f02-0d00-4d0e-e108-9e891115115d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 99466])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Code cell\n",
    "edge_index = torch.LongTensor(edge_index)\n",
    "edge_index.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.789977Z",
     "iopub.status.busy": "2024-08-09T10:57:29.789667Z",
     "iopub.status.idle": "2024-08-09T10:57:29.862835Z",
     "shell.execute_reply": "2024-08-09T10:57:29.862014Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.789953Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([4., 4., 4.,  ..., 5., 5., 3.], dtype=torch.float64)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Code cell\n",
    "edge_values = torch.tensor(edge_values)\n",
    "edge_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.863931Z",
     "iopub.status.busy": "2024-08-09T10:57:29.863702Z",
     "iopub.status.idle": "2024-08-09T10:57:29.874301Z",
     "shell.execute_reply": "2024-08-09T10:57:29.873513Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.863910Z"
    },
    "id": "ZCFxLxt7XUZ5",
    "outputId": "596f0b73-dd70-4f73-fcf4-a7e0f1892852"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of movies: 9724\n",
      "Number of users: 610\n",
      "Number of interactions: 99466\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "num_movies, num_users = len(df_rating['movieId'].unique()), len(df_rating['userId'].unique())\n",
    "num_interactions = edge_index.shape[1]\n",
    "print(f'Number of movies: {num_movies}')\n",
    "print(f'Number of users: {num_users}')\n",
    "print(f'Number of interactions: {num_interactions}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SqG-JB6-YWlw"
   },
   "source": [
    "### Splitting Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.875779Z",
     "iopub.status.busy": "2024-08-09T10:57:29.875424Z",
     "iopub.status.idle": "2024-08-09T10:57:29.980931Z",
     "shell.execute_reply": "2024-08-09T10:57:29.980088Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.875756Z"
    },
    "id": "znLNeHxrYThL",
    "outputId": "0326200c-1e2e-4510-971f-91e5eb396f6d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "79572\n",
      "9947\n",
      "9947\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "all_indices = list(range(num_interactions))\n",
    "train_indices, test_indices = train_test_split(all_indices, test_size=0.2, random_state=1)\n",
    "val_indices, test_indices = train_test_split(test_indices, test_size=0.5, random_state=1)\n",
    "\n",
    "train_edge_index = edge_index[:, train_indices]\n",
    "train_edge_values = edge_values[train_indices]\n",
    "val_edge_index = edge_index[:, val_indices]\n",
    "val_edge_values = edge_values[val_indices]\n",
    "test_edge_index = edge_index[:, test_indices]\n",
    "test_edge_values = edge_values[test_indices]\n",
    "print(len(train_edge_index[1]))\n",
    "print(len(val_edge_index[1]))\n",
    "print(len(test_edge_index[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "39j39FrVY_-I"
   },
   "source": [
    "### Conversion of Interaction matrix(R) to Adjacency Matrix(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.982322Z",
     "iopub.status.busy": "2024-08-09T10:57:29.982040Z",
     "iopub.status.idle": "2024-08-09T10:57:29.988940Z",
     "shell.execute_reply": "2024-08-09T10:57:29.988021Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.982299Z"
    },
    "id": "NOjq82zlY97I"
   },
   "outputs": [],
   "source": [
    "# Define functions\n",
    "def R_to_Adjacency_edge_index(edge_index, edge_values):\n",
    "  R = torch.zeros((num_users, num_movies))\n",
    "  for i in range(len(edge_index[0])):\n",
    "    R[edge_index[0][i], edge_index[1][i]] = edge_values[i]\n",
    "  R_T = torch.transpose(R, 0, 1)\n",
    "  A = torch.zeros((num_users+num_movies), (num_users+num_movies))\n",
    "  A[:num_users, num_users:] = R.clone()\n",
    "  A[num_users:, :num_users] = R_T.clone()\n",
    "  A_coo = A.to_sparse_coo()\n",
    "  return A_coo.indices(), A_coo.values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:29.990154Z",
     "iopub.status.busy": "2024-08-09T10:57:29.989905Z",
     "iopub.status.idle": "2024-08-09T10:57:30.000826Z",
     "shell.execute_reply": "2024-08-09T10:57:30.000103Z",
     "shell.execute_reply.started": "2024-08-09T10:57:29.990132Z"
    },
    "id": "wwX67R1OdFSS"
   },
   "outputs": [],
   "source": [
    "# Define functions\n",
    "def Adjacency_edge_index_to_R_edge_index(edge_index, edge_values):\n",
    "  sparse_input_edge_mat = torch.sparse_coo_tensor(edge_index, edge_values, size=(num_users + num_movies, num_users + num_movies))\n",
    "  A = sparse_input_edge_mat.to_dense()\n",
    "  R = A[:num_users, num_users:]\n",
    "  R_COO = R.to_sparse_coo()\n",
    "  R_edge_index = R_COO.indices()\n",
    "  return R_edge_index, R_COO.values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:30.002009Z",
     "iopub.status.busy": "2024-08-09T10:57:30.001751Z",
     "iopub.status.idle": "2024-08-09T10:57:32.413581Z",
     "shell.execute_reply": "2024-08-09T10:57:32.412576Z",
     "shell.execute_reply.started": "2024-08-09T10:57:30.001987Z"
    },
    "id": "VCu283xdaOt5"
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "train_adj_edge_index, train_adj_edge_values = R_to_Adjacency_edge_index(train_edge_index, train_edge_values)\n",
    "val_adj_edge_index, val_adj_edge_values = R_to_Adjacency_edge_index(val_edge_index, val_edge_values)\n",
    "test_adj_edge_index, test_adj_edge_values = R_to_Adjacency_edge_index(test_edge_index, test_edge_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:32.415084Z",
     "iopub.status.busy": "2024-08-09T10:57:32.414783Z",
     "iopub.status.idle": "2024-08-09T10:57:32.419952Z",
     "shell.execute_reply": "2024-08-09T10:57:32.419078Z",
     "shell.execute_reply.started": "2024-08-09T10:57:32.415058Z"
    },
    "id": "O3KGaz1wcKG3",
    "outputId": "386f64fe-7c3d-4204-8940-809552caef36"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 159144])\n",
      "torch.Size([2, 19894])\n",
      "torch.Size([2, 19894])\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "print(train_adj_edge_index.size())\n",
    "print(val_adj_edge_index.size())\n",
    "print(test_adj_edge_index.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aZ8n4KNcezso"
   },
   "source": [
    "### Training & computing BPR Loss\n",
    "Since this is Self-supervised Learning, we are relying on graph structure itself. So, we will random sample a batch of positives and negatives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:32.421250Z",
     "iopub.status.busy": "2024-08-09T10:57:32.420989Z",
     "iopub.status.idle": "2024-08-09T10:57:32.431838Z",
     "shell.execute_reply": "2024-08-09T10:57:32.430959Z",
     "shell.execute_reply.started": "2024-08-09T10:57:32.421223Z"
    },
    "id": "hyCxVrExcRAN"
   },
   "outputs": [],
   "source": [
    "# Define functions\n",
    "def sample_mini_batch(batch_size, edge_index): # Note: edge_index is in interactive matrix form\n",
    "  edges = structured_negative_sampling(edge_index)\n",
    "  edges = torch.stack(edges, dim=0)\n",
    "  mini_batch_sample_indices = random.choices(list(range(edges[0].shape[0])), k=batch_size)\n",
    "  mini_batch_edges = edges[:, mini_batch_sample_indices]\n",
    "  user_indices, positive_user_indices, negative_user_indices = mini_batch_edges[0], mini_batch_edges[1], mini_batch_edges[2]\n",
    "  return user_indices, positive_user_indices, negative_user_indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J93TM2PjkBxe"
   },
   "source": [
    "## LightGCN Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:32.433094Z",
     "iopub.status.busy": "2024-08-09T10:57:32.432860Z",
     "iopub.status.idle": "2024-08-09T10:57:32.446388Z",
     "shell.execute_reply": "2024-08-09T10:57:32.445536Z",
     "shell.execute_reply.started": "2024-08-09T10:57:32.433074Z"
    },
    "id": "Up9gGzmTiurS"
   },
   "outputs": [],
   "source": [
    "# Define functions\n",
    "class LightGCN(MessagePassing):\n",
    "  def __init__(self, num_users, num_movies, embedding_dim=64, K=3, add_self_loops=False): # K = No.of layers\n",
    "    super().__init__()\n",
    "    self.num_users = num_users\n",
    "    self.num_movies = num_movies\n",
    "    self.embedding_dim = embedding_dim\n",
    "    self.K = K\n",
    "    self.add_self_loops = add_self_loops\n",
    "\n",
    "    self.user_embedding = nn.Embedding(num_embeddings=self.num_users, embedding_dim=self.embedding_dim)\n",
    "    self.movie_embedding = nn.Embedding(num_embeddings=self.num_movies, embedding_dim=self.embedding_dim)\n",
    "    self.init_embedding()\n",
    "    self.out = nn.Linear(embedding_dim+embedding_dim, 1)\n",
    "\n",
    "  def init_embedding(self):\n",
    "    nn.init.normal_(self.user_embedding.weight, std=0.1)\n",
    "    nn.init.normal_(self.movie_embedding.weight, std=0.1)\n",
    "\n",
    "\n",
    "  def forward(self, edge_index, edge_values):\n",
    "    edge_index_norm = gcn_norm(edge_index, add_self_loops=self.add_self_loops)\n",
    "\n",
    "    emb_0 = torch.cat([self.user_embedding.weight, self.movie_embedding.weight])\n",
    "    embs = [emb_0]\n",
    "    emb_k = emb_0\n",
    "\n",
    "    for i in range(self.K):\n",
    "      emb_k = self.propagate(edge_index=edge_index_norm[0], x=emb_k, norm=edge_index_norm[1])\n",
    "      embs.append(emb_k)\n",
    "\n",
    "    embs = torch.stack(embs, dim=1)\n",
    "    lightgcn_all_embeddings = torch.mean(embs, dim=1)\n",
    "\n",
    "    user_embeddings, movie_embeddings = torch.split(lightgcn_all_embeddings, [self.num_users, self.num_movies])\n",
    "    \n",
    "    r_mat_edge_index, _ = Adjacency_edge_index_to_R_edge_index(edge_index, edge_values)\n",
    "    \n",
    "    src, dest = r_mat_edge_index[0], r_mat_edge_index[1]\n",
    "    user_embs = user_embeddings[src]\n",
    "    item_embs = movie_embeddings[dest]\n",
    "    out = torch.cat([user_embs, item_embs], dim=1)\n",
    "    return self.out(out)\n",
    "\n",
    "  def message(self, x_j, norm):\n",
    "    return norm.view(-1, 1) * x_j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dzkYw6axyJPW"
   },
   "source": [
    "### Evaluation Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T11:01:41.465240Z",
     "iopub.status.busy": "2024-08-09T11:01:41.464898Z",
     "iopub.status.idle": "2024-08-09T11:01:41.475418Z",
     "shell.execute_reply": "2024-08-09T11:01:41.474424Z",
     "shell.execute_reply.started": "2024-08-09T11:01:41.465212Z"
    },
    "id": "1esZxKYdyMeY"
   },
   "outputs": [],
   "source": [
    "# Define functions\n",
    "# Get recall\n",
    "def get_recall_at_k(input_edge_index, input_edge_values, pred_ratings, k=10, threshold=3.5):\n",
    "    with torch.no_grad():\n",
    "        user_item_rating_list = defaultdict(list)\n",
    "        for i in range(len(input_edge_index[0])):\n",
    "            src=input_edge_index[0][i].item()\n",
    "            dest=input_edge_index[1][i].item()\n",
    "            true_rating=input_edge_values[i].item()\n",
    "            pred_rating=pred_ratings[i].item()\n",
    "            \n",
    "            user_item_rating_list[src].append((pred_rating, true_rating))\n",
    "        recalls = dict()\n",
    "        precisions = dict()\n",
    "        for user_id, user_ratings in user_item_rating_list.items():\n",
    "            user_ratings.sort(key=lambda x: x[0], reverse=True)\n",
    "            n_rel = sum((true_r>=threshold) for (_, true_r) in user_ratings)\n",
    "            n_rec_k = sum((est>=threshold) for (est, _) in user_ratings[:k])\n",
    "            n_rel_and_rec_k = sum((est>=threshold and true_r>=threshold) for (est, true_r) in user_ratings[:k])\n",
    "            precisions[user_id] = n_rel_and_rec_k / n_rec_k if n_rec_k != 0 else 0\n",
    "            recalls[user_id] = n_rel_and_rec_k / n_rel if n_rel != 0 else 0\n",
    "        overall_recall = sum(rec for rec in recalls.values())/len(recalls)\n",
    "        overall_precision = sum(pre for pre in precisions.values())/len(precisions)\n",
    "        \n",
    "        return overall_recall, overall_precision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wHQuoykk6hhg"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_ISbGtDY7frA"
   },
   "source": [
    "### Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:32.461185Z",
     "iopub.status.busy": "2024-08-09T10:57:32.460878Z",
     "iopub.status.idle": "2024-08-09T10:57:32.473619Z",
     "shell.execute_reply": "2024-08-09T10:57:32.472781Z",
     "shell.execute_reply.started": "2024-08-09T10:57:32.461156Z"
    },
    "id": "TZw-1yxE6iyB"
   },
   "outputs": [],
   "source": [
    "# Code cell\n",
    "ITERATIONS = 10000\n",
    "EPOCHS = 10\n",
    "BATCH_SIZE = 1024\n",
    "K = 10\n",
    "LR = 1e-3\n",
    "ITERS_PER_EVAL = 200\n",
    "ITERS_PER_LR_DECAY = 200\n",
    "LAMBDA = 1e-6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qcNQepnK73qb"
   },
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T10:57:32.475391Z",
     "iopub.status.busy": "2024-08-09T10:57:32.474783Z",
     "iopub.status.idle": "2024-08-09T10:57:32.497584Z",
     "shell.execute_reply": "2024-08-09T10:57:32.496833Z",
     "shell.execute_reply.started": "2024-08-09T10:57:32.475365Z"
    },
    "id": "afWmlY9O76fj",
    "outputId": "29285360-ee02-4af1-f357-52bf517fd3c4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "model = LightGCN(num_users, num_movies, K=1)\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:58:41.798693Z",
     "iopub.status.busy": "2024-08-09T10:58:41.798077Z",
     "iopub.status.idle": "2024-08-09T10:58:41.806669Z",
     "shell.execute_reply": "2024-08-09T10:58:41.805816Z",
     "shell.execute_reply.started": "2024-08-09T10:58:41.798662Z"
    },
    "id": "Hfe-v18c8COW"
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "model = model.to(device)\n",
    "\n",
    "model.train()\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=LR)\n",
    "scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)\n",
    "\n",
    "edge_index = edge_index.to(device)\n",
    "train_adj_edge_index = train_adj_edge_index.to(device)\n",
    "val_adj_edge_index = val_adj_edge_index.to(device)\n",
    "test_adj_edge_index = test_adj_edge_index.to(device)\n",
    "\n",
    "train_adj_edge_values = train_adj_edge_values.to(device)\n",
    "val_adj_edge_values = val_adj_edge_values.to(device)\n",
    "test_adj_edge_values = test_adj_edge_values.to(device)\n",
    "loss_fn = nn.MSELoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T10:58:43.859160Z",
     "iopub.status.busy": "2024-08-09T10:58:43.858808Z",
     "iopub.status.idle": "2024-08-09T10:58:43.999508Z",
     "shell.execute_reply": "2024-08-09T10:58:43.998580Z",
     "shell.execute_reply.started": "2024-08-09T10:58:43.859130Z"
    }
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "r_mat_train_index, r_mat_train_values = Adjacency_edge_index_to_R_edge_index(train_adj_edge_index, train_adj_edge_values)\n",
    "r_mat_test_index, r_mat_test_values = Adjacency_edge_index_to_R_edge_index(test_adj_edge_index, test_adj_edge_values)\n",
    "r_mat_val_index, r_mat_val_values = Adjacency_edge_index_to_R_edge_index(val_adj_edge_index, val_adj_edge_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eOAJm_N79TsN"
   },
   "source": [
    "### Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 962,
     "referenced_widgets": [
      "96f7c37570b44beebbacbbe1bbb3bd4f",
      "944dfe5672b243a493631a16e554494e",
      "fb2f7178087d4d2eb6bf9a57ab1e501e",
      "a2d94911c6584013ae46b02ef53cb3c6",
      "b4daf3e1d2354d44a70bbaf98660d905",
      "1c3a7fd4cd394f55b245053b25c5a7ac",
      "40ba21bc2f8943bfb7bb47898d805765",
      "6b009c3db6b64c058fe17ebb4d25fc8a",
      "051ac13b99fc435d8d82de838cc5bf72",
      "35674005d4e540a8aadd5c6788d18fe5",
      "27b38b6ab74442f5b5d60b5a20d97b50"
     ]
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T11:01:57.887246Z",
     "iopub.status.busy": "2024-08-09T11:01:57.886364Z",
     "iopub.status.idle": "2024-08-09T11:03:51.056984Z",
     "shell.execute_reply": "2024-08-09T11:03:51.056080Z",
     "shell.execute_reply.started": "2024-08-09T11:01:57.887211Z"
    },
    "id": "iTTmpcGd9VMK",
    "outputId": "0851aa39-73ae-4da2-bceb-6a85fd347aeb"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7afcdc01d1df45958f945ad473aaad93",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Iteration 0/10000] train_loss: 13.422472953796387, val_Loss: 13.458664894104004, Recall: 0.0, Precision: 0.0\n",
      "[Iteration 200/10000] train_loss: 1.005832314491272, val_Loss: 1.383942723274231, Recall: 0.1707661789498549, Precision: 0.468895877850793\n",
      "[Iteration 400/10000] train_loss: 0.7693952918052673, val_Loss: 1.206430196762085, Recall: 0.3541587251001714, Precision: 0.6157676729588497\n",
      "[Iteration 600/10000] train_loss: 0.6509786248207092, val_Loss: 1.1792881488800049, Recall: 0.4360737275110559, Precision: 0.6269573989105723\n",
      "[Iteration 800/10000] train_loss: 0.5846289396286011, val_Loss: 1.1803618669509888, Recall: 0.4897558670550156, Precision: 0.6301863094908762\n",
      "[Iteration 1000/10000] train_loss: 0.5597918629646301, val_Loss: 1.1915977001190186, Recall: 0.5066265081751276, Precision: 0.6380902638598073\n",
      "[Iteration 1200/10000] train_loss: 0.551003634929657, val_Loss: 1.2008744478225708, Recall: 0.5161051417068063, Precision: 0.6434238971382935\n",
      "[Iteration 1400/10000] train_loss: 0.5474973917007446, val_Loss: 1.2066484689712524, Recall: 0.5172392128054607, Precision: 0.644591222495828\n",
      "[Iteration 1600/10000] train_loss: 0.5460148453712463, val_Loss: 1.2101248502731323, Recall: 0.5166844902961457, Precision: 0.6446360739650215\n",
      "[Iteration 1800/10000] train_loss: 0.5453898310661316, val_Loss: 1.2122336626052856, Recall: 0.5157040669154759, Precision: 0.6427361451900928\n",
      "[Iteration 2000/10000] train_loss: 0.5451316833496094, val_Loss: 1.2135076522827148, Recall: 0.5164387398125619, Precision: 0.6428635306471997\n",
      "[Iteration 2200/10000] train_loss: 0.5450272560119629, val_Loss: 1.2142654657363892, Recall: 0.5158240211978434, Precision: 0.6428430253240628\n",
      "[Iteration 2400/10000] train_loss: 0.5449857711791992, val_Loss: 1.214705467224121, Recall: 0.5158481757872153, Precision: 0.6429263586573961\n",
      "[Iteration 2600/10000] train_loss: 0.5449694395065308, val_Loss: 1.214954137802124, Recall: 0.5158481757872153, Precision: 0.6429263586573961\n",
      "[Iteration 2800/10000] train_loss: 0.5449632406234741, val_Loss: 1.2150899171829224, Recall: 0.5158481757872153, Precision: 0.6429263586573961\n",
      "[Iteration 3000/10000] train_loss: 0.5449608564376831, val_Loss: 1.2151613235473633, Recall: 0.5158481757872153, Precision: 0.6428152475462849\n",
      "[Iteration 3200/10000] train_loss: 0.5449599623680115, val_Loss: 1.2151973247528076, Recall: 0.5158481757872153, Precision: 0.6428152475462849\n",
      "[Iteration 3400/10000] train_loss: 0.5449596643447876, val_Loss: 1.2152142524719238, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 3600/10000] train_loss: 0.5449594855308533, val_Loss: 1.215221881866455, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 3800/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152245044708252, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 4000/10000] train_loss: 0.5449594259262085, val_Loss: 1.215225100517273, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 4200/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152246236801147, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 4400/10000] train_loss: 0.5449594259262085, val_Loss: 1.215224027633667, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 4600/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152235507965088, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 4800/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152230739593506, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 5000/10000] train_loss: 0.5449594855308533, val_Loss: 1.215222954750061, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 5200/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152225971221924, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 5400/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152224779129028, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 5600/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152223587036133, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 5800/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152223587036133, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 6000/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152223587036133, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 6200/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152223587036133, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 6400/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152221202850342, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 6600/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152222394943237, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 6800/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152222394943237, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 7000/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152221202850342, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 7200/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152221202850342, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 7400/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152221202850342, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 7600/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152221202850342, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 7800/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152221202850342, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 8000/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152221202850342, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 8200/10000] train_loss: 0.5449594855308533, val_Loss: 1.215221881866455, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 8400/10000] train_loss: 0.5449594855308533, val_Loss: 1.215221881866455, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 8600/10000] train_loss: 0.5449594855308533, val_Loss: 1.215221881866455, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 8800/10000] train_loss: 0.5449594855308533, val_Loss: 1.215221881866455, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 9000/10000] train_loss: 0.5449594855308533, val_Loss: 1.215221881866455, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 9200/10000] train_loss: 0.5449594855308533, val_Loss: 1.215221881866455, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 9400/10000] train_loss: 0.5449594855308533, val_Loss: 1.215221881866455, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 9600/10000] train_loss: 0.5449594855308533, val_Loss: 1.2152217626571655, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n",
      "[Iteration 9800/10000] train_loss: 0.5449594855308533, val_Loss: 1.215221881866455, Recall: 0.5158481757872153, Precision: 0.6426637323947698\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "train_losses = []\n",
    "val_losses = []\n",
    "val_recall_at_k = []\n",
    "val_precision_at_k = []\n",
    "\n",
    "for iter in tqdm(range(ITERATIONS)):\n",
    "  pred_ratings = model.forward(train_adj_edge_index, train_adj_edge_values)\n",
    "  train_loss = loss_fn(pred_ratings, r_mat_train_values.view(-1, 1))\n",
    "    \n",
    "  optimizer.zero_grad()\n",
    "  train_loss.backward()\n",
    "  optimizer.step()\n",
    "\n",
    "  # val set\n",
    "  if iter % ITERS_PER_EVAL == 0:\n",
    "    model.eval()\n",
    "\n",
    "    with torch.no_grad():\n",
    "      val_pred_ratings = model.forward(val_adj_edge_index, val_adj_edge_values)\n",
    "      val_loss = loss_fn(val_pred_ratings, r_mat_val_values.view(-1, 1)).sum()\n",
    "      recall_at_k, precision_at_k = get_recall_at_k(r_mat_val_index, r_mat_val_values, val_pred_ratings, k=20)\n",
    "      print(f\"[Iteration {iter}/{ITERATIONS}] train_loss: {train_loss}, val_Loss: {val_loss}, Recall: {recall_at_k}, Precision: {precision_at_k}\")\n",
    "\n",
    "      train_losses.append(train_loss.item())\n",
    "      val_losses.append(val_loss.item())\n",
    "      val_recall_at_k.append(recall_at_k)\n",
    "      val_precision_at_k.append(precision_at_k)\n",
    "\n",
    "    model.train()\n",
    "\n",
    "  if iter % ITERS_PER_LR_DECAY == 0 and iter != 0:\n",
    "    scheduler.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I1B0Ra3DFR04"
   },
   "source": [
    "## Plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 472
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T11:03:57.219589Z",
     "iopub.status.busy": "2024-08-09T11:03:57.218923Z",
     "iopub.status.idle": "2024-08-09T11:03:57.525365Z",
     "shell.execute_reply": "2024-08-09T11:03:57.524561Z",
     "shell.execute_reply.started": "2024-08-09T11:03:57.219557Z"
    },
    "id": "nbbGGvQREIZJ",
    "outputId": "bd3a6fc2-381e-4c96-a8e4-769af0a66c13"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAHHCAYAAABZbpmkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOkUlEQVR4nO3dd3gU5doG8Hu2JiE9lCSSEJqU0Isc4CAgwRAgih1ORIqfoIKAFJWjIEUJKipSRNRzhHOUqjRBwFAU6S10pEiAHAQjYBqBre/3x+4OWRIgZTczCffvuvbKzruzO89OEnLzzDs7khBCgIiIiKiC0ihdABEREZE3MewQERFRhcawQ0RERBUaww4RERFVaAw7REREVKEx7BAREVGFxrBDREREFRrDDhEREVVoDDtERERUoTHs0D2jf//+iImJKdFzJ0yYAEmSPFuQypw9exaSJGHevHllvm1JkjBhwgR5ed68eZAkCWfPnr3rc2NiYtC/f3+P1lOan5XSUPJ7QFSRMeyQ4iRJKtLtp59+UrrUe96wYcMgSRJOnz5923XefPNNSJKEQ4cOlWFlxff7779jwoQJOHDggNKlEJGX6ZQugOi///2v2/J//vMfpKSkFBhv0KBBqbbzxRdfwG63l+i5b731Ft54441Sbb8iSEpKwsyZM7FgwQKMHz++0HUWLlyIxo0bo0mTJiXeTt++fdG7d28YjcYSv8bd/P7775g4cSJiYmLQrFkzt8dK87NCROrDsEOKe/bZZ92Wd+7ciZSUlALjt8rLy4Ofn1+Rt6PX60tUHwDodDrodPx1adOmDerUqYOFCxcWGnZ27NiBtLQ0TJ06tVTb0Wq10Gq1pXqN0ijNzwoVn9Vqhd1uh8FgULoUqqB4GIvKhU6dOqFRo0bYt28fHnzwQfj5+eGf//wnAGDlypXo0aMHIiMjYTQaUbt2bUyePBk2m83tNW6dh+GaHzFt2jR8/vnnqF27NoxGI1q3bo09e/a4PbewOTuSJGHo0KFYsWIFGjVqBKPRiNjYWKxbt65A/T/99BNatWoFHx8f1K5dG3Pnzi3yPKBffvkFTz31FKKjo2E0GhEVFYVXX30V169fL/D+/P39ceHCBfTq1Qv+/v6oUqUKRo8eXWBfZGZmon///ggKCkJwcDD69euHzMzMu9YCOLo7v/76K/bv31/gsQULFkCSJPTp0wdmsxnjx49Hy5YtERQUhEqVKqFDhw7YvHnzXbdR2JwdIQTeeecdVK9eHX5+fujcuTOOHj1a4LlXr17F6NGj0bhxY/j7+yMwMBAJCQk4ePCgvM5PP/2E1q1bAwAGDBggHyp1zZUpbM7OtWvXMGrUKERFRcFoNKJevXqYNm0ahBBu6xXn56KoNm3ahA4dOqBSpUoIDg7Go48+iuPHj7utk5OTgxEjRiAmJgZGoxFVq1ZF165d3b5Pp06dwhNPPIHw8HD4+PigevXq6N27N7Kysu5aw65du9C9e3eEhISgUqVKaNKkCT755BP58U6dOqFTp04Fnnen37vp06fLv3epqanQ6XSYOHFigdc4ceIEJEnCrFmz5LHMzEyMGDFC/n7UqVMH7733XoGO3KJFi9CyZUsEBAQgMDAQjRs3dqub7g38ryqVG1euXEFCQgJ69+6NZ599FtWqVQPg+MPo7++PkSNHwt/fH5s2bcL48eORnZ2NDz744K6vu2DBAuTk5GDw4MGQJAnvv/8+Hn/8cZw5c+au/8PfunUrli1bhpdffhkBAQGYMWMGnnjiCZw/fx5hYWEAgNTUVHTr1g0RERGYOHEibDYbJk2ahCpVqhTpfS9duhR5eXl46aWXEBYWht27d2PmzJn43//+h6VLl7qta7PZEB8fjzZt2mDatGnYsGEDPvzwQ9SuXRsvvfQSAEdoePTRR7F161a8+OKLaNCgAZYvX45+/foVqZ6kpCRMnDgRCxYsQIsWLdy2vWTJEnTo0AHR0dG4fPkyvvzyS/Tp0wcvvPACcnJy8K9//Qvx8fHYvXt3gUNHdzN+/Hi888476N69O7p37479+/fj4YcfhtlsdlvvzJkzWLFiBZ566inUrFkTf/zxB+bOnYuOHTvi2LFjiIyMRIMGDTBp0iSMHz8egwYNQocOHQAA7dq1K3TbQgg88sgj2Lx5M55//nk0a9YM69evx5gxY3DhwgV8/PHHbusX5eeiqDZs2ICEhATUqlULEyZMwPXr1zFz5ky0b98e+/fvl4PEiy++iG+//RZDhw5Fw4YNceXKFWzduhXHjx9HixYtYDabER8fD5PJhFdeeQXh4eG4cOECVq9ejczMTAQFBd22hpSUFPTs2RMREREYPnw4wsPDcfz4caxevRrDhw8v1vtx+eqrr3Djxg0MGjQIRqMRERER6NixI5YsWYK3337bbd3FixdDq9XiqaeeAuDo6nbs2BEXLlzA4MGDER0dje3bt2Ps2LG4ePEipk+fLtfdp08fdOnSBe+99x4A4Pjx49i2bVuJ66ZyShCpzJAhQ8StP5odO3YUAMRnn31WYP28vLwCY4MHDxZ+fn7ixo0b8li/fv1EjRo15OW0tDQBQISFhYmrV6/K4ytXrhQAxPfffy+Pvf322wVqAiAMBoM4ffq0PHbw4EEBQMycOVMeS0xMFH5+fuLChQvy2KlTp4ROpyvwmoUp7P0lJycLSZLEuXPn3N4fADFp0iS3dZs3by5atmwpL69YsUIAEO+//748ZrVaRYcOHQQA8dVXX921ptatW4vq1asLm80mj61bt04AEHPnzpVf02QyuT3vr7/+EtWqVRMDBw50Gwcg3n77bXn5q6++EgBEWlqaEEKIjIwMYTAYRI8ePYTdbpfX++c//ykAiH79+sljN27ccKtLCMf32mg0uu2bPXv23Pb93vqz4tpn77zzjtt6Tz75pJAkye1noKg/F4Vx/Uzmr6lZs2aiatWq4sqVK26vp9FoxHPPPSePBQUFiSFDhtz2tVNTUwUAsXTp0jvWcCur1Spq1qwpatSoIf766y+3x/J/Lzp27Cg6duxY4Pm3+70LDAwUGRkZbuvOnTtXABCHDx92G2/YsKF46KGH5OXJkyeLSpUqiZMnT7qt98YbbwitVivOnz8vhBBi+PDhIjAwUFit1uK8ZaqAeBiLyg2j0YgBAwYUGPf19ZXv5+Tk4PLly+jQoQPy8vLw66+/3vV1n3nmGYSEhMjLrv/lnzlz5q7PjYuLQ+3ateXlJk2aIDAwUH6uzWbDhg0b0KtXL0RGRsrr1alTBwkJCXd9fcD9/V27dg2XL19Gu3btIIRAampqgfVffPFFt+UOHTq4vZcffvgBOp1O7vQAjjkyr7zySpHqARzzrP73v/9hy5Yt8tiCBQtgMBjk/31rtVp5DobdbsfVq1dhtVrRqlWrQg+B3cmGDRtgNpvxyiuvuB36GzFiRIF1jUYjNBrHP202mw1XrlyBv78/6tWrV+ztuvzwww/QarUYNmyY2/ioUaMghMDatWvdxu/2c1FUFy9exIEDB9C/f3+Ehoa6vV7Xrl3xww8/yGPBwcHYtWsXfv/990Jfy9W5Wb9+PfLy8opcQ2pqKtLS0jBixAgEBwe7PVaaj2N44oknCnQ3H3/8ceh0OixevFgeO3LkCI4dO4ZnnnlGHlu6dCk6dOiAkJAQXL58Wb7FxcXBZrPJP5fBwcG4du0aUlJSSlwnVQwMO1Ru3HfffYVOYDx69Cgee+wxBAUFITAwEFWqVJEnNxdlLkJ0dLTbsiv4/PXXX8V+ruv5rudmZGTg+vXrqFOnToH1ChsrzPnz5+U/dq55OB07dgRQ8P35+PgU+AOSvx4AOHfuHCIiIuDv7++2Xr169YpUDwD07t0bWq0WCxYsAADcuHEDy5cvR0JCgltwnD9/Ppo0aQIfHx+EhYWhSpUqWLNmTZG+L/mdO3cOAFC3bl238SpVqrhtD3AEq48//hh169aF0WhE5cqVUaVKFRw6dKjY282//cjISAQEBLiNu84QdNXncrefi+JsFyj8e9OgQQNcvnwZ165dAwC8//77OHLkCKKiovDAAw9gwoQJbuGqZs2aGDlyJL788ktUrlwZ8fHxmD179l33yW+//QYAaNSoUbFqv5uaNWsWGKtcuTK6dOmCJUuWyGOLFy+GTqfD448/Lo+dOnUK69atQ5UqVdxucXFxABy/dwDw8ssv4/7770dCQgKqV6+OgQMHlmruFJVfDDtUbuTvcLhkZmaiY8eOOHjwICZNmoTvv/8eKSkp8vH5opw+fLuzfsQtE089/dyisNls6Nq1K9asWYPXX38dK1asQEpKijyR9tb3V1ZnMLkmv3733XewWCz4/vvvkZOTg6SkJHmdr7/+Gv3790ft2rXxr3/9C+vWrUNKSgoeeughr57WPWXKFIwcORIPPvggvv76a6xfvx4pKSmIjY0ts9PJvf1zUZinn34aZ86cwcyZMxEZGYkPPvgAsbGxbl2nDz/8EIcOHcI///lPXL9+HcOGDUNsbCz+97//lXr7t+vy3Do53qWw32fAEaRPnjwpf/7RkiVL0KVLF1SuXFlex263o2vXrkhJSSn09sQTTwBw/JweOHAAq1atkudcJSQkFHl+GlUcnKBM5dpPP/2EK1euYNmyZXjwwQfl8bS0NAWruqlq1arw8fEp9EP47vTBfC6HDx/GyZMnMX/+fDz33HPyeGna8jVq1MDGjRuRm5vr1t05ceJEsV4nKSkJ69atw9q1a7FgwQIEBgYiMTFRfvzbb79FrVq1sGzZMrc/hLdOPi1qzYDjf/S1atWSx//8888C3ZJvv/0WnTt3xr/+9S+38czMTLc/mMU5BFOjRg1s2LABOTk5bt0d12FSV32e5nrdwr43v/76KypXroxKlSrJYxEREXj55Zfx8ssvIyMjAy1atMC7777rdsi0cePGaNy4Md566y1s374d7du3x2effYZ33nmn0Bpch+OOHDkid04KExISUuhhulu7XnfTq1cvDB48WD6UdfLkSYwdO7ZATbm5uXesx8VgMCAxMRGJiYmw2+14+eWXMXfuXIwbN67I3VUq/9jZoXLN9T/o/P9jNpvN+PTTT5UqyY1Wq0VcXBxWrFjhNpfi9OnTBeZ53O75gPv7E0KU6tTZ7t27w2q1Ys6cOfKYzWbDzJkzi/U6vXr1gp+fHz799FOsXbsWjz/+OHx8fO5Y+65du7Bjx45i1xwXFwe9Xo+ZM2e6vZ7rrJv8tFptgQ7K0qVLceHCBbcxV0goyin33bt3h81mczv1GQA+/vhjSJJU5PlXxRUREYFmzZph/vz5bnUeOXIEP/74I7p37w7A8f279XBU1apVERkZCZPJBADIzs6G1Wp1W6dx48bQaDTyOoVp0aIFatasienTpxfYV/n3c+3atfHrr7/izz//lMcOHjyIbdu2Fes9BwcHIz4+HkuWLMGiRYtgMBjQq1cvt3Wefvpp7NixA+vXry/w/MzMTPl9Xrlyxe0xjUYjf9jlnd4zVTzs7FC51q5dO4SEhKBfv37ypQz++9//evVwQXFNmDABP/74I9q3b4+XXnpJ/qPZqFGju16qoH79+qhduzZGjx6NCxcuIDAwEN99912x537kl5iYiPbt2+ONN97A2bNn0bBhQyxbtqzY81n8/f3Rq1cved5O/kNYANCzZ08sW7YMjz32GHr06IG0tDR89tlnaNiwIXJzc4u1LdfnBSUnJ6Nnz57o3r07UlNTsXbtWrdujWu7kyZNwoABA9CuXTscPnwY33zzjVtHCHD8cQ4ODsZnn32GgIAAVKpUCW3atCl0LkliYiI6d+6MN998E2fPnkXTpk3x448/YuXKlRgxYoTbZGRP++CDD5CQkIC2bdvi+eefl089DwoKkq8nlpOTg+rVq+PJJ59E06ZN4e/vjw0bNmDPnj348MMPATg+q2fo0KF46qmncP/998NqteK///0vtFqtfNinMBqNBnPmzEFiYiKaNWuGAQMGICIiAr/++iuOHj0qB46BAwfio48+Qnx8PJ5//nlkZGTgs88+Q2xsLLKzs4v1np955hk8++yz+PTTTxEfH19gYvSYMWOwatUq9OzZE/3790fLli1x7do1HD58GN9++y3Onj2LypUr4//+7/9w9epVPPTQQ6hevTrOnTuHmTNnolmzZqX+RHYqZxQ5B4zoDm536nlsbGyh62/btk387W9/E76+viIyMlK89tprYv369QKA2Lx5s7ze7U6B/eCDDwq8Jm45Ffp2p54XdqpvjRo13E6FFkKIjRs3iubNmwuDwSBq164tvvzySzFq1Cjh4+Nzm71w07Fjx0RcXJzw9/cXlStXFi+88IJ8KnP+U5T79esnKlWqVOD5hdV+5coV0bdvXxEYGCiCgoJE37595VOTi3LqucuaNWsEABEREVHgdG+73S6mTJkiatSoIYxGo2jevLlYvXp1ge+DEHc/9VwIIWw2m5g4caKIiIgQvr6+olOnTuLIkSMF9veNGzfEqFGj5PXat28vduzYUeip0StXrhQNGzaUPwbA9d4LqzEnJ0e8+uqrIjIyUuj1elG3bl3xwQcfuJ1+7XovRf25uFVhp54LIcSGDRtE+/btha+vrwgMDBSJiYni2LFj8uMmk0mMGTNGNG3aVAQEBIhKlSqJpk2bik8//VRe58yZM2LgwIGidu3awsfHR4SGhorOnTuLDRs23LEml61bt4quXbvKr9+kSZMCp9J//fXXolatWsJgMIhmzZqJ9evXF+v3ziU7O1v4+voKAOLrr78udJ2cnBwxduxYUadOHWEwGETlypVFu3btxLRp04TZbBZCCPHtt9+Khx9+WFStWlUYDAYRHR0tBg8eLC5evFik90wVhySEiv4LTHQP6dWrF44ePYpTp04pXQoRUYXGOTtEZeDWSzucOnUKP/zwQ6Efr09ERJ7Fzg5RGYiIiED//v1Rq1YtnDt3DnPmzIHJZEJqamqBz44hIiLP4gRlojLQrVs3LFy4EJcuXYLRaETbtm0xZcoUBh0iojLAzg4RERFVaJyzQ0RERBUaww4RERFVaIrO2dmyZQs++OAD7Nu3DxcvXsTy5csLfFKmy4svvoi5c+fi448/LvRKx7djt9vx+++/IyAgoFRX6CUiIqKyI4RATk4OIiMjodGUrjejaNi5du0amjZtioEDB7pd0fZWy5cvx86dOxEZGVnsbfz++++IiooqTZlERESkkPT0dFSvXr1Ur6Fo2ElISLjrNWUuXLiAV155BevXr0ePHj2KvQ3XRfvS09MRGBhYojqJiIiobGVnZyMqKsrt4rslpepTz+12O/r27YsxY8YgNja2SM8xmUxuF3jLyckBAAQGBjLsEBERlTOemIKi6gnK7733HnQ6HYYNG1bk5yQnJyMoKEi+8RAWERHRvU21YWffvn345JNPMG/evGKlurFjxyIrK0u+paene7FKIiIiUjvVhp1ffvkFGRkZiI6Ohk6ng06nw7lz5zBq1CjExMTc9nlGo1E+ZMVDV0RERKTaOTt9+/ZFXFyc21h8fDz69u2LAQMGKFQVERHRTTabDRaLRekyyiW9Xg+tVlsm21I07OTm5uL06dPyclpaGg4cOIDQ0FBER0cjLCzMbX29Xo/w8HDUq1evrEslIiKSCSFw6dIlZGZmKl1KuRYcHIzw8HCvfw6eomFn79696Ny5s7w8cuRIAEC/fv0wb948haoiIiK6M1fQqVq1Kvz8/PihtcUkhEBeXh4yMjIAABEREV7dnqJhp1OnTijOdUjPnj3rvWKIiIiKwGazyUHn1iMQVHS+vr4AgIyMDFStWtWrh7RUO0GZiIhIjVxzdPz8/BSupPxz7UNvz3ti2CEiIioBHroqvbLahww7REREVKEx7BAREVGxxcTEYPr06UqXUSSq/ZwdIiIi8qxOnTqhWbNmHgkpe/bsQaVKlUpfVBlg2Cmp65nAjSzAJxDwDVG6GiIiolITQsBms0Gnu3s8qFKlShlU5Bk8jFVC19eMBT5pgtxtc5UuhYiI6K769++Pn3/+GZ988gkkSYIkSfL1J9euXYuWLVvCaDRi69at+O233/Doo4+iWrVq8Pf3R+vWrbFhwwa317v1MJYkSfjyyy/x2GOPwc/PD3Xr1sWqVavK+F0WjmGnhA78fh0AcOz8ZYUrISIipQkhkGe2KnIr6ufVffLJJ2jbti1eeOEFXLx4ERcvXkRUVBQA4I033sDUqVNx/PhxNGnSBLm5uejevTs2btyI1NRUdOvWDYmJiTh//vwdtzFx4kQ8/fTTOHToELp3746kpCRcvXq11Pu3tHgYq4TsWiMAQLLeULgSIiJS2nWLDQ3Hr1dk28cmxcPPcPc/50FBQTAYDPDz80N4eDgA4NdffwUATJo0CV27dpXXDQ0NRdOmTeXlyZMnY/ny5Vi1ahWGDh162230798fffr0AQBMmTIFM2bMwO7du9GtW7cSvTdPYWenpLQGx1ebWdk6iIiISqlVq1Zuy7m5uRg9ejQaNGiA4OBg+Pv74/jx43ft7DRp0kS+X6lSJQQGBsqXhFASOzslJHTOzo7NpHAlRESkNF+9FscmxSu27dK69ayq0aNHIyUlBdOmTUOdOnXg6+uLJ598Embznf+Dr9fr3ZYlSYLdbi91faXFsFNScthhZ4eI6F4nSVKRDiUpzWAwwGaz3XW9bdu2oX///njssccAODo95fn6lDyMVVI6HwCAhp0dIiIqJ2JiYrBr1y6cPXsWly9fvm3XpW7duli2bBkOHDiAgwcP4h//+IcqOjQlxbBTQpKzs6Oxs7NDRETlw+jRo6HVatGwYUNUqVLltnNwPvroI4SEhKBdu3ZITExEfHw8WrRoUcbVeo76e24qJbk6O3Z2doiIqHy4//77sWPHDrex/v37F1gvJiYGmzZtchsbMmSI2/Kth7UKOwU+MzOzRHV6Gjs7JSTpHZ0dLTs7REREqsawU0JavaOzw7BDRESkbgw7JaRxdnZ0dovClRAREdGdMOyUkNbg6OzoBDs7REREasawU0IaHcMOERFRecCwU0I6oy8AQM+wQ0REpGoMOyWkc05Q1gvO2SEiIlIzhp0S0hqdYQcMO0RERGrGsFNCeucEZQPDDhERkaox7JSQ3jlnxwArUI6vF0JERFRUMTExmD59utJlFBvDTgm5wg4ACF4MlIiISLUYdkrIYPST71tMNxSshIiIiO6EYaeEjEajfN9ivq5gJURERHf3+eefIzIyEvZbpl48+uijGDhwIH777Tc8+uijqFatGvz9/dG6dWts2LBBoWo9i2GnhAw6LW4IPQDAcoNhh4joniYEYL6mzK2Qq40X5qmnnsKVK1ewefNmeezq1atYt24dkpKSkJubi+7du2Pjxo1ITU1Ft27dkJiYiPPnz3trr5UZndIFlFcajQQz9PCBBRYzD2MREd3TLHnAlEhltv3P3wFDpbuuFhISgoSEBCxYsABdunQBAHz77beoXLkyOnfuDI1Gg6ZNm8rrT548GcuXL8eqVaswdOhQr5VfFtjZKQUznJ0dU57ClRAREd1dUlISvvvuO5hMjhNrvvnmG/Tu3RsajQa5ubkYPXo0GjRogODgYPj7++P48ePs7NzrLJIj7FjZ2SEiurfp/RwdFqW2XUSJiYkQQmDNmjVo3bo1fvnlF3z88ccAgNGjRyMlJQXTpk1DnTp14OvriyeffBJmc/m/LBLDTimYYQAAWE2cs0NEdE+TpCIdSlKaj48PHn/8cXzzzTc4ffo06tWrhxYtWgAAtm3bhv79++Oxxx4DAOTm5uLs2bMKVus5DDulYJX0gACsFn7ODhERlQ9JSUno2bMnjh49imeffVYer1u3LpYtW4bExERIkoRx48YVOHOrvOKcnVKwSI7Ojo2nnhMRUTnx0EMPITQ0FCdOnMA//vEPefyjjz5CSEgI2rVrh8TERMTHx8tdn/KOnZ1SsGr0gB2ws7NDRETlhEajwe+/F5xfFBMTg02bNrmNDRkyxG25vB7WYmenFKzOzo7dwgnKREREasWwUwo2DcMOERGR2jHslIIr7AiGHSIiItVi2CkFu6uzY+WcHSIiIrVi2CkFu9ZxMVDBCcpERPccUcRrUtHtldU+VDTsbNmyBYmJiYiMjIQkSVixYoX8mMViweuvv47GjRujUqVKiIyMxHPPPVfoDHKluDo7YGeHiOieodc7Pj0/L4+XCiot1z507VNvUfTU82vXrqFp06YYOHAgHn/8cbfH8vLysH//fowbNw5NmzbFX3/9heHDh+ORRx7B3r17FarYnXB2dmDjnB0ionuFVqtFcHAwMjIyAAB+fn6QJEnhqsoXIQTy8vKQkZGB4OBgaLVar25P0bCTkJCAhISEQh8LCgpCSkqK29isWbPwwAMP4Pz584iOji6LEu9IaF2dnfJ/3RAiIiq68PBwAJADD5VMcHCwvC+9qVx9qGBWVhYkSUJwcPBt1zGZTPLVXAEgOzvba/UInQ8AQGJnh4joniJJEiIiIlC1alVYLBalyymX9Hq91zs6LuUm7Ny4cQOvv/46+vTpg8DAwNuul5ycjIkTJ5ZNUc7OjmRjZ4eI6F6k1WrL7A82lVy5OBvLYrHg6aefhhACc+bMueO6Y8eORVZWlnxLT0/3XmE6x5wdjY0TlImIiNRK9Z0dV9A5d+4cNm3adMeuDgAYjUYYjcYyqU2SD2Oxs0NERKRWqg47rqBz6tQpbN68GWFhYUqX5E7vCFVaO8MOERGRWikadnJzc3H69Gl5OS0tDQcOHEBoaCgiIiLw5JNPYv/+/Vi9ejVsNhsuXboEAAgNDYXBYFCqbJnkOozFsENERKRaioadvXv3onPnzvLyyJEjAQD9+vXDhAkTsGrVKgBAs2bN3J63efNmdOrUqazKvC2N8zCW1s45O0RERGqlaNjp1KnTHT8qWu0fxa01uMIOTzskIiJSq3JxNpZaafSOsKMXPIxFRESkVgw7paDhBGUiIiLVY9gpBZ3BFwA7O0RERGrGsFMKrjk7OnDODhERkVox7JSC1tnZMQiGHSIiIrVi2CkFnbOzo2dnh4iISLUYdkrBYHR0doywACo/TZ6IiOhexbBTCq7ODgCA18ciIiJSJYadUnB1dgBAWK4rWAkRERHdDsNOKRjzhR2rhZeMICIiUiOGnVIw6LUwCT0AwHKDnR0iIiI1YtgpBYNOA5Pz8mIWM8MOERGRGjHslIJWI8EMZ2fHxLBDRESkRgw7pWRxhh2r6YbClRAREVFhGHZKySwZAABWno1FRESkSgw7pSR3dszs7BAREakRw04pWTTOzg7DDhERkSox7JSSVXJ0dmwMO0RERKrEsFNKVuecHbuFYYeIiEiNGHZKyaZxhR1+gjIREZEaMeyU0s2ww84OERGRGjHslJIr7Agrww4REZEaMeyUkt0VdtjZISIiUiWGnVKya12dHbPClRAREVFhGHZKya4xOu7wMBYREZEqMeyUknB2dsDODhERkSox7JSS0Po47tjY2SEiIlIjhp3S0jk6O5KNnR0iIiI1YtgpJaFzdHY0Nn6oIBERkRox7JSWlp0dIiIiNWPYKSVJz84OERGRmjHslJKkc5x6rrWzs0NERKRGDDulpHHO2WHYISIiUieGnVLS6B2dHQ3DDhERkSox7JSSxjlnRycYdoiIiNSIYaeUNAZn2GFnh4iISJUYdkpJy84OERGRqjHslJLW1dkRFoUrISIiosIw7JSSzhl29Aw7REREqsSwU0quzo4BDDtERERqxLBTSnqDn+MrLIAQCldDREREt1I07GzZsgWJiYmIjIyEJElYsWKF2+NCCIwfPx4RERHw9fVFXFwcTp06pUyxt6E3Oi8XAQHY2N0hIiJSG0XDzrVr19C0aVPMnj270Mfff/99zJgxA5999hl27dqFSpUqIT4+Hjdu3CjjSm9Pb/S9ucDrYxEREamOTsmNJyQkICEhodDHhBCYPn063nrrLTz66KMAgP/85z+oVq0aVqxYgd69e5dlqbdlcHZ2AABWE2AMUK4YIiIiKkC1c3bS0tJw6dIlxMXFyWNBQUFo06YNduzYcdvnmUwmZGdnu928yajXwSQcmdFqvu7VbREREVHxqTbsXLp0CQBQrVo1t/Fq1arJjxUmOTkZQUFB8i0qKsqrdRp0GpihBwBYTAw7REREaqPasFNSY8eORVZWlnxLT0/36vYMWg1MrrBjVs9cIiIiInJQbdgJDw8HAPzxxx9u43/88Yf8WGGMRiMCAwPdbt6k02pgcU59YmeHiIhIfVQbdmrWrInw8HBs3LhRHsvOzsauXbvQtm1bBSsryAwDAMDKzg4REZHqKHo2Vm5uLk6fPi0vp6Wl4cCBAwgNDUV0dDRGjBiBd955B3Xr1kXNmjUxbtw4REZGolevXsoVXQiL5DiMZWVnh4iISHUUDTt79+5F586d5eWRI0cCAPr164d58+bhtddew7Vr1zBo0CBkZmbi73//O9atWwcfH5/bvaQizJIBEOzsEBERqZGiYadTp04Qd7jEgiRJmDRpEiZNmlSGVRWfVdIDArDx1HMiIiLVUe2cnfLEKjnm7Ngs/ARlIiIitWHY8QCrc86O3cLDWERERGrDsOMBNo0RAGBnZ4eIiEh1GHY8wK5xdHYEOztERESqw7DjAXJnx8rODhERkdow7HiAXeOYoCysPBuLiIhIbRh2PMCudYQdWM3KFkJEREQFMOx4gF3rOIwFK+fsEBERqQ3DjgcIZ9gR7OwQERGpDsOOJ+gch7EkTlAmIiJSHYYdDxBax7W6JDvDDhERkdow7HiCc4KyxsawQ0REpDYMO56gd3Z2bJyzQ0REpDYMOx4g6RwTlLXs7BAREakOw44HuMKOxm5RuBIiIiK6FcOOB2j0zs4OJygTERGpDsOOB2icc3a07OwQERGpDsOOB7jCjk6ws0NERKQ2DDseoNH7AgB0gp0dIiIitWHY8QCdwTFnRyd46jkREZHaMOx4gNbg6Ozo2dkhIiJSHYYdD9AZHHN29OzsEBERqQ7DjgdoXWEHVkAIhashIiKi/Bh2PEBvdJ56DjtgtypcDREREeXHsOMBBqPvzQUrTz8nIiJSE4YdD9Ab/G4uMOwQERGpCsOOBxgNOpiF1rHAi4ESERGpCsOOBxi0GpihBwDYzDcUroaIiIjyY9jxAINOA5Mz7FhM1xWuhoiIiPJj2PEAo+5mZ8diylO4GiIiIsqPYccDdNp8nR0exiIiIlIVhh0PsTjDjtXEsENERKQmDDseYpGcYcfCsENERKQmDDseYpUMAACbmROUiYiI1IRhx0NcnR2bhZ+zQ0REpCYMOx7i6uzYOUGZiIhIVRh2PMSqcR7G4pwdIiIiVWHY8RC7q7PDsENERKQqDDseYnN2dgTn7BAREakKw46H2LTOsGNlZ4eIiEhNGHY8xK4xAgCElZ0dIiIiNVF12LHZbBg3bhxq1qwJX19f1K5dG5MnT4YQQunSChDs7BAREamSTukC7uS9997DnDlzMH/+fMTGxmLv3r0YMGAAgoKCMGzYMKXLc2PXOjo7sJqVLYSIiIjcqDrsbN++HY8++ih69OgBAIiJicHChQuxe/duhSsryNXZkWzs7BAREamJqg9jtWvXDhs3bsTJkycBAAcPHsTWrVuRkJBw2+eYTCZkZ2e73cqCYGeHiIhIlVTd2XnjjTeQnZ2N+vXrQ6vVwmaz4d1330VSUtJtn5OcnIyJEyeWYZVOOkfY0dg4QZmIiEhNVN3ZWbJkCb755hssWLAA+/fvx/z58zFt2jTMnz//ts8ZO3YssrKy5Ft6enrZFOsMO5KdnR0iIiI1UXVnZ8yYMXjjjTfQu3dvAEDjxo1x7tw5JCcno1+/foU+x2g0wmg0lmWZAABJ5wOAnR0iIiK1UXVnJy8vDxqNe4larRZ2u12him5P0rsOY7GzQ0REpCaq7uwkJibi3XffRXR0NGJjY5GamoqPPvoIAwcOVLq0AiTnBGUtD2MRERGpiqrDzsyZMzFu3Di8/PLLyMjIQGRkJAYPHozx48crXVoBGoPjMJZWMOwQERGpiarDTkBAAKZPn47p06crXcpdaZwTlHXs7BAREamKqufslCcagy8AHsYiIiJSG4YdD9E6JyjreBiLiIhIVRh2PETr7OzohEXhSoiIiCg/hh0P0TknKBvY2SEiIlIVhh0P0eodnR092NkhIiJSE4YdD9H7OE89hx2wWRWuhoiIiFwYdjxE55yzAwDgJSOIiIhUg2HHQwzOOTsAACvDDhERkVow7HiI0WCARWgdCww7REREqlGisDN//nysWbNGXn7ttdcQHByMdu3a4dy5cx4rrjwx6DQwuz6Q2npD2WKIiIhIVqKwM2XKFPj6Ouao7NixA7Nnz8b777+PypUr49VXX/VogeWFQaeBCXoAgN3Czg4REZFalOjaWOnp6ahTpw4AYMWKFXjiiScwaNAgtG/fHp06dfJkfeWGQadBrjPsWMzXYVS4HiIiInIoUWfH398fV65cAQD8+OOP6Nq1KwDAx8cH169f91x15YhRp4FJOMOOiYexiIiI1KJEnZ2uXbvi//7v/9C8eXOcPHkS3bt3BwAcPXoUMTExnqyv3NBpJJjlzk6ewtUQERGRS4k6O7Nnz0bbtm3x559/4rvvvkNYWBgAYN++fejTp49HCywvJEmCWXKEHauZnR0iIiK1KFFnJzg4GLNmzSowPnHixFIXVJ5ZXWGHh7GIiIhUo0SdnXXr1mHr1q3y8uzZs9GsWTP84x//wF9//eWx4sobi2QAANgsDDtERERqUaKwM2bMGGRnZwMADh8+jFGjRqF79+5IS0vDyJEjPVpgeWJ1hh07D2MRERGpRokOY6WlpaFhw4YAgO+++w49e/bElClTsH//fnmy8r3Iys4OERGR6pSos2MwGJCX5zjjaMOGDXj44YcBAKGhoXLH515k07g+VJBhh4iISC1K1Nn5+9//jpEjR6J9+/bYvXs3Fi9eDAA4efIkqlev7tECyxObxvFRgnZeG4uIiEg1StTZmTVrFnQ6Hb799lvMmTMH9913HwBg7dq16Natm0cLLE9cnR3Bzg4REZFqlKizEx0djdWrVxcY//jjj0tdUHlm1zo6O4LXxiIiIlKNEoUdALDZbFixYgWOHz8OAIiNjcUjjzwCrVbrseLKG7vGMUFZ8KrnREREqlGisHP69Gl0794dFy5cQL169QAAycnJiIqKwpo1a1C7dm2PFlleuDo7sJqVLYSIiIhkJZqzM2zYMNSuXRvp6enYv38/9u/fj/Pnz6NmzZoYNmyYp2ssN4TW0dmBjZ0dIiIitShRZ+fnn3/Gzp07ERoaKo+FhYVh6tSpaN++vceKK2+Es7MjsbNDRESkGiXq7BiNRuTk5BQYz83NhcFgKHVR5ZYr7Ng4QZmIiEgtShR2evbsiUGDBmHXrl0QQkAIgZ07d+LFF1/EI4884ukayw2hY9ghIiJSmxKFnRkzZqB27dpo27YtfHx84OPjg3bt2qFOnTqYPn26h0ssPyRn2NEw7BAREalGiebsBAcHY+XKlTh9+rR86nmDBg1Qp04djxZX3shhx845O0RERGpR5LBzt6uZb968Wb7/0UcflbyicuxmZ4dhh4iISC2KHHZSU1OLtJ4kSSUupryT9D4AAC07O0RERKpR5LCTv3NDhdO4wo5g2CEiIlKLEk1QpsJp2NkhIiJSHYYdD3KFHR07O0RERKrBsONBOoMz7NgtCldCRERELgw7HqR1dnb0YGeHiIhILRh2PEjr7OzoBTs7REREasGw40F6oy8AQAcbYLcpXA0REREBDDsepTP43lyw8pIRREREaqD6sHPhwgU8++yzCAsLg6+vLxo3boy9e/cqXVah9EafmwvWG8oVQkRERLISXRurrPz1119o3749OnfujLVr16JKlSo4deoUQkJClC6tUAaDAVahgU6yA7xkBBERkSqoOuy89957iIqKwldffSWP1axZU8GK7syo08AMPXQwsbNDRESkEqo+jLVq1Sq0atUKTz31FKpWrYrmzZvjiy++uONzTCYTsrOz3W5lxaDTwAS9Y8HKzg4REZEaqDrsnDlzBnPmzEHdunWxfv16vPTSSxg2bBjmz59/2+ckJycjKChIvkVFRZVZvQatBmZns8xuuV5m2yUiIqLbk4QQQukibsdgMKBVq1bYvn27PDZs2DDs2bMHO3bsKPQ5JpMJJtPNM6Gys7MRFRWFrKwsBAYGerXeXJMVV9+tj2jNnzD1/xHGmDZe3R4REVFFlZ2djaCgII/8/VZ1ZyciIgINGzZ0G2vQoAHOnz9/2+cYjUYEBga63cqKo7PjOIxlMbGzQ0REpAaqDjvt27fHiRMn3MZOnjyJGjVqKFTRnem1kjxnx2rmBGUiIiI1UHXYefXVV7Fz505MmTIFp0+fxoIFC/D5559jyJAhSpdWKEmSYJFcYYedHSIiIjVQddhp3bo1li9fjoULF6JRo0aYPHkypk+fjqSkJKVLuy2LZAAA2NjZISIiUgVVf84OAPTs2RM9e/ZUuowis0p6QAA2C8MOERGRGqi6s1MeWZ2dHTs7O0RERKrAsONhNo3zMBY7O0RERKrAsONhcmfHwqueExERqQHDjoe5OjuCnR0iIiJVYNjxMFfYsVvZ2SEiIlIDhh0Ps2vZ2SEiIlIThh0Ps2uMjjvs7BAREakCw46HCWdnBzaGHSIiIjVg2PEwofNx3GFnh4iISBUYdjxMOCcoS+zsEBERqQLDjocJnWPOjmQzK1wJERERAQw7HifJYYedHSIiIjVg2PEwV9jRsLNDRESkCgw7HiaHHTs7O0RERGrAsONpzrOxtHZ2doiIiNSAYcfDNAZHZ0fLw1hERESqwLDjYRpXZ0cw7BAREakBw46HafSOsKPjYSwiIiJVYNjxMK3B1dmxKFwJERERAQw7Hqd1dnb0PIxFRESkCgw7HqY1+gIAdOzsEBERqQLDjofpnGdj6WEF7HaFqyEiIiKGHQ/TGXxvLvCSEURERIpj2PEwvdHn5oL1hnKFEBEREQCGHY8z6IywCcmxYOUkZSIiIqUx7HiY0aCFGXrHAjs7REREimPY8TCDVgOTK+zwkhFERESKY9jxMINOw84OERGRijDseJhBp4FJOMKO3cKzsYiIiJTGsONhRp0GZugAABbzdYWrISIiIoYdDzPoNDDBAACwmnkYi4iISGkMOx5m0N7s7NjY2SEiIlIcw46HSZIEs+To7FjY2SEiIlIcw44XWJ1nY9kYdoiIiBTHsOMFFmdnx86wQ0REpDiGHS+waZydHQvn7BARESmNYccLrK7ODj9nh4iISHEMO15g07jCDg9jERERKY1hxwvksGNlZ4eIiEhpDDteYNcaAQCCE5SJiIgUx7DjBa7OjrCxs0NERKS0chV2pk6dCkmSMGLECKVLuSOhdYQd8DAWERGR4spN2NmzZw/mzp2LJk2aKF3KXQnnYSyGHSIiIuWVi7CTm5uLpKQkfPHFFwgJCVG6nLtydXYkhh0iIiLFlYuwM2TIEPTo0QNxcXF3XddkMiE7O9vtVtaEzgcAIHHODhERkeJ0ShdwN4sWLcL+/fuxZ8+eIq2fnJyMiRMnermqu3B1dhh2iIiIFKfqzk56ejqGDx+Ob775Bj4+PkV6ztixY5GVlSXf0tPTvVxlQZLc2TGX+baJiIjInao7O/v27UNGRgZatGghj9lsNmzZsgWzZs2CyWSCVqt1e47RaITRaCzrUt3pHNvX2NnZISIiUpqqw06XLl1w+PBht7EBAwagfv36eP311wsEHbVwdXa0dovClRAREZGqw05AQAAaNWrkNlapUiWEhYUVGFcTjd7Z2eGcHSIiIsWpes5OeSXpnZ0dwTk7RERESlN1Z6cwP/30k9Il3JWrs6OzM+wQEREpjZ0dL9DoOWeHiIhILRh2vEBrcIQdPQ9jERERKY5hxwu0Bl8AgE6ws0NERKQ0hh0v0DoPY+lhAex2hashIiK6tzHseIHemO/TnvkpykRERIpi2PECvT5f2LHeUK4QIiIiYtjxBr3BCLuQHAvs7BARESmKYccLDHotzK6PMGJnh4iISFEMO15g1Glhgt6xYGVnh4iISEkMO15g0GlglsMOOztERERKYtjxAoNOc7Ozw4uBEhERKYphxwuMOg1MwhF2hIWdHSIiIiUx7HhB/sNYFjPDDhERkZIYdrzAoNXA5Dwby2q+rnA1RERE9zaGHS8w6jQwwQAAsLKzQ0REpCiGHS+QJAlW52EsG8MOERGRohh2vMQiMewQERGpAcOOl1glx2EsG8/GIiIiUhTDjpdYNc6wY+bn7BARESmJYcdLbBrn5+xYeTYWERGRkhh2vMQmGQEAdgs7O0REREpi2PESu9bR2bFzzg4REZGiGHa8xKZxdHaElZ0dIiIiJTHseIld65igzKueExERKYthx0vsWkdnB1azsoUQERHd4xh2vETIYYedHSIiIiUx7HiJHHZs7OwQEREpiWHHW5xzdjQ2TlAmIiJSEsOOl0g6H8dXdnaIiIgUxbDjLTp2doiIiNSAYcdbnJ0djZ2dHSIiIiUx7HiJpHdMUGZnh4iISFkMO16icXZ2tOzsEBERKYphx0s0eoYdIiIiNWDY8RLJFXaEReFKiIiI7m0MO16iMzjm7OjZ2SEiIlIUw46XuA5j6QTDDhERkZIYdrxEa3CEHT0sgBAKV0NERHTvYtjxEp3B9+YCP0WZiIhIMQw7XqJ3dnYA8MrnRERECmLY8RL3sMPODhERkVJUHXaSk5PRunVrBAQEoGrVqujVqxdOnDihdFlFYtBrYRJ6xwI7O0RERIpRddj5+eefMWTIEOzcuRMpKSmwWCx4+OGHce3aNaVLuyujTgMTnGGHc3aIiIgUo1O6gDtZt26d2/K8efNQtWpV7Nu3Dw8++KBCVRWNQaeBybV72dkhIiJSjKrDzq2ysrIAAKGhobddx2QywWS6efHN7Oxsr9dVGINWAxMMjgUrLwZKRESkFFUfxsrPbrdjxIgRaN++PRo1anTb9ZKTkxEUFCTfoqKiyrDKm4x6LczC1dlh2CEiIlJKuQk7Q4YMwZEjR7Bo0aI7rjd27FhkZWXJt/T09DKq0J2js+OYsyMYdoiIiBRTLg5jDR06FKtXr8aWLVtQvXr1O65rNBphNBrLqLLbM+g0MDvDjtV83TVVmYiIiMqYqjs7QggMHToUy5cvx6ZNm1CzZk2lSyqy/GdjWS2coExERKQUVXd2hgwZggULFmDlypUICAjApUuXAABBQUHw9fW9y7OVZdBq5Dk7VtN1hashIiK6d6m6szNnzhxkZWWhU6dOiIiIkG+LFy9WurS70mgkWCTH2Vg2dnaIiIgUo+rOjijnVwu3So7DWDYzww4REZFSVN3ZKe8sGmdnx8yzsYiIiJTCsONFNudhLLuFc3aIiIiUwrDjRVaNK+yws0NERKQUhh0vsjvDjuAEZSIiIsUw7HiRzdXZ4ScoExERKYZhx4tcnR1e9ZyIiEg5DDteZNc6LlvBa2MREREph2HHi4Qz7PCq50RERMph2PEiV9gJzDoB/HlS4WqIiIjuTQw7XpThWwt2ISE49zQw+wHgu/8DLp9SuiwiIqJ7CsOOF6UHNENP87tIq9wZgAAOL3WGnhcYeoiIiMoIw44XGXUaHBMxWNPwA2DwFqBeD0DYgcNLHKFn2SDg8mmlyyQiIqrQVH0h0PLOoHNkSbPVDkQ0BfosAH4/APz8HnDiB+DQYke3p/HTQKPHAb/KgF+o42YMBCTJuwUKAdjMjgnUNgtgM7nft1sBu82xbLfeXLbfsizsjpt835ZvWdx8HK77hY3BeV/c+aurbuRfH4U8lm+swHgx9s/dVyr+6xIRVTSxjwFRDyhdxW0x7HiRQesIOyab/eZgZDOgz0Lg91Tg5/edoWeR45afRgf4hQG+oY6vfqGAoVIhf/RvCQJ2myOwWG/k+3rjlmVnqLFbvLwHiIjonlD5foade5VR7ww7FnvBByOb3ww922c65vDkXQWuXwUseY6uSe4fjltZ0egArRHQ6gGtwXHTaJ3jesdX17JG77yvBSQNIDm/ysuam8uQ8o3luw/JuSzdvF/gK24/Btwcl+/fwm1Mus14IY/fcb2S8HKXjohISeFNlK7gjhh2vMig1QIAzLZCwo5LZHPgyX+7j5nzHKEn7yqQd8V5c4aguwUDSQvojIDe1/FV51P4V63R+VXvDDgGQMMpXEREVPEw7HiR25ydYj3Rz3ELqu6FqoiIiO4t/K+8F7nCjqm4YYeIiIg8hmHHi4zOsHP0QhbOX8lTuBoiIqJ7E8OOF7WsEQI/gxZnLl9Dt0+24D87zsJu56nKREREZYlhx4saRARi3fAH0aZmKPLMNoxfeRT/+HIn0q+yy0NERFRWGHa8LDrMDwtf+BsmPhILX70WO89cRfz0LfgvuzxERERlgmGnDGg0Evq1i8G6ER3wgLPLM27lUSR9uYtdHiIiIi9j2ClDNcIqYdELf8PbiQ3ho9dgx5krji7PznPs8hAREXmJJERJLhpUfmRnZyMoKAhZWVkIDAxUuhzZ2cvX8Nq3h7D77FUAjvk9jzSNRI/GEYgO81O4OiIiImV58u83w46C7HaBedvP4v31v+JGvktKNL4vCD2aRKBH4whEhTL4EBHRvYdhpxjUHHZcrl4zY/3RS1hz6CK2/3YZ+Y9oNakehB6NI9CdwYeIiO4hDDvFUB7CTn6Xc01y8Nl55opb8KlXLQCx9wWiYUQgGkYGIjYiCEF+euWKJSIi8hKGnWIob2Env8u5Jqw74gg+u9Lcg4/LfcG+aBh5MwDVreqP0EoGBPjoodXwSttERFQ+MewUQ3kOO/n9mWPCgfRMHPs9G8cuZuHYxWykX71+2/UlCQj00SPYT49gXz2C/AwI9nUsB/nqYdRpoNc6bgadBgatBnqdBINWC71Wgl6ngVaSoNVI0EgSNBKg1UiQnGNaSYIkARrnV0kCJDjv56sB+cYk55XZ3R93PC//cml56nWIiKhogv0M8Dd69triDDvFUFHCTmGyrlvw68VsHLuY7QxB2Ui7fA15ZpvSpRER0T1kymON8Y820R59TU/+/fZsDKMyFeSrR5taYWhTK8xt3Gy1I+u6BVnXzcjMszhu1y3IzDM7xy2w2OwwWe2w2AQsVjvMNjssNjvMVudXmx02u+OMMbsQsAkBIQCbXcBmFxDOMbsAHHHZ8bgAIIRwfr15H45V8n9B/px9a+IuLIKLAmuVTMWO90REZU+r8k/tY9ipgAw6DaoEGFElwKh0KURERIpTeRYjIiIiKh2GHSIiIqrQGHaIiIioQmPYISIiogqNYYeIiIgqNIYdIiIiqtAYdoiIiKhCY9ghIiKiCq1chJ3Zs2cjJiYGPj4+aNOmDXbv3q10SURERFROqD7sLF68GCNHjsTbb7+N/fv3o2nTpoiPj0dGRobSpREREVE5oPqw89FHH+GFF17AgAED0LBhQ3z22Wfw8/PDv//9b6VLIyIionJA1WHHbDZj3759iIuLk8c0Gg3i4uKwY8eOQp9jMpmQnZ3tdiMiIqJ7l6rDzuXLl2Gz2VCtWjW38WrVquHSpUuFPic5ORlBQUHyLSoqqixKJSIiIpVSddgpibFjxyIrK0u+paenK10SERERKUindAF3UrlyZWi1Wvzxxx9u43/88QfCw8MLfY7RaITRaJSXhRAAwMNZRERE5Yjr77br73hpqDrsGAwGtGzZEhs3bkSvXr0AAHa7HRs3bsTQoUOL9Bo5OTkAwMNZRERE5VBOTg6CgoJK9RqqDjsAMHLkSPTr1w+tWrXCAw88gOnTp+PatWsYMGBAkZ4fGRmJ9PR0BAQEQJIkj9WVnZ2NqKgopKenIzAw0GOvS4Xj/i5b3N9lj/u8bHF/l62S7G8hBHJychAZGVnq7as+7DzzzDP4888/MX78eFy6dAnNmjXDunXrCkxavh2NRoPq1at7rb7AwED+opQh7u+yxf1d9rjPyxb3d9kq7v4ubUfHRfVhBwCGDh1a5MNWRERERPlVuLOxiIiIiPJj2Ckho9GIt99+2+3ML/Ie7u+yxf1d9rjPyxb3d9lSen9LwhPndBERERGpFDs7REREVKEx7BAREVGFxrBDREREFRrDDhEREVVoDDslNHv2bMTExMDHxwdt2rTB7t27lS5J9ZKTk9G6dWsEBASgatWq6NWrF06cOOG2zo0bNzBkyBCEhYXB398fTzzxRIFro50/fx49evSAn58fqlatijFjxsBqtbqt89NPP6FFixYwGo2oU6cO5s2b5+23p3pTp06FJEkYMWKEPMb97VkXLlzAs88+i7CwMPj6+qJx48bYu3ev/LgQAuPHj0dERAR8fX0RFxeHU6dOub3G1atXkZSUhMDAQAQHB+P5559Hbm6u2zqHDh1Chw4d4OPjg6ioKLz//vtl8v7UxGazYdy4cahZsyZ8fX1Ru3ZtTJ482e06StzfJbdlyxYkJiYiMjISkiRhxYoVbo+X5b5dunQp6tevDx8fHzRu3Bg//PBD8d+QoGJbtGiRMBgM4t///rc4evSoeOGFF0RwcLD4448/lC5N1eLj48VXX30ljhw5Ig4cOCC6d+8uoqOjRW5urrzOiy++KKKiosTGjRvF3r17xd/+9jfRrl07+XGr1SoaNWok4uLiRGpqqvjhhx9E5cqVxdixY+V1zpw5I/z8/MTIkSPFsWPHxMyZM4VWqxXr1q0r0/erJrt37xYxMTGiSZMmYvjw4fI497fnXL16VdSoUUP0799f7Nq1S5w5c0asX79enD59Wl5n6tSpIigoSKxYsUIcPHhQPPLII6JmzZri+vXr8jrdunUTTZs2FTt37hS//PKLqFOnjujTp4/8eFZWlqhWrZpISkoSR44cEQsXLhS+vr5i7ty5Zfp+lfbuu++KsLAwsXr1apGWliaWLl0q/P39xSeffCKvw/1dcj/88IN48803xbJlywQAsXz5crfHy2rfbtu2TWi1WvH++++LY8eOibfeekvo9Xpx+PDhYr0fhp0SeOCBB8SQIUPkZZvNJiIjI0VycrKCVZU/GRkZAoD4+eefhRBCZGZmCr1eL5YuXSqvc/z4cQFA7NixQwjh+AXUaDTi0qVL8jpz5swRgYGBwmQyCSGEeO2110RsbKzbtp555hkRHx/v7bekSjk5OaJu3boiJSVFdOzYUQ473N+e9frrr4u///3vt33cbreL8PBw8cEHH8hjmZmZwmg0ioULFwohhDh27JgAIPbs2SOvs3btWiFJkrhw4YIQQohPP/1UhISEyPvfte169ep5+i2pWo8ePcTAgQPdxh5//HGRlJQkhOD+9qRbw05Z7tunn35a9OjRw62eNm3aiMGDBxfrPfAwVjGZzWbs27cPcXFx8phGo0FcXBx27NihYGXlT1ZWFgAgNDQUALBv3z5YLBa3fVu/fn1ER0fL+3bHjh1o3Lix27XR4uPjkZ2djaNHj8rr5H8N1zr36vdnyJAh6NGjR4F9wv3tWatWrUKrVq3w1FNPoWrVqmjevDm++OIL+fG0tDRcunTJbV8FBQWhTZs2bvs7ODgYrVq1kteJi4uDRqPBrl275HUefPBBGAwGeZ34+HicOHECf/31l7ffpmq0a9cOGzduxMmTJwEABw8exNatW5GQkACA+9ubynLfeurfF4adYrp8+TJsNluBC5FWq1YNly5dUqiq8sdut2PEiBFo3749GjVqBAC4dOkSDAYDgoOD3dbNv28vXbpU6L53PXandbKzs3H9+nVvvB3VWrRoEfbv34/k5OQCj3F/e9aZM2cwZ84c1K1bF+vXr8dLL72EYcOGYf78+QBu7q87/dtx6dIlVK1a1e1xnU6H0NDQYn1P7gVvvPEGevfujfr160Ov16N58+YYMWIEkpKSAHB/e1NZ7tvbrVPcfV8uLgRKFc+QIUNw5MgRbN26VelSKqz09HQMHz4cKSkp8PHxUbqcCs9ut6NVq1aYMmUKAKB58+Y4cuQIPvvsM/Tr10/h6iqeJUuW4JtvvsGCBQsQGxuLAwcOYMSIEYiMjOT+pgLY2SmmypUrQ6vVFjhj5Y8//kB4eLhCVZUvQ4cOxerVq7F582ZUr15dHg8PD4fZbEZmZqbb+vn3bXh4eKH73vXYndYJDAyEr6+vp9+Oau3btw8ZGRlo0aIFdDoddDodfv75Z8yYMQM6nQ7VqlXj/vagiIgINGzY0G2sQYMGOH/+PICb++tO/3aEh4cjIyPD7XGr1YqrV68W63tyLxgzZozc3WncuDH69u2LV199Ve5icn97T1nu29utU9x9z7BTTAaDAS1btsTGjRvlMbvdjo0bN6Jt27YKVqZ+QggMHToUy5cvx6ZNm1CzZk23x1u2bAm9Xu+2b0+cOIHz58/L+7Zt27Y4fPiw2y9RSkoKAgMD5T80bdu2dXsN1zr32venS5cuOHz4MA4cOCDfWrVqhaSkJPk+97fntG/fvsBHKZw8eRI1atQAANSsWRPh4eFu+yo7Oxu7du1y29+ZmZnYt2+fvM6mTZtgt9vRpk0beZ0tW7bAYrHI66SkpKBevXoICQnx2vtTm7y8PGg07n/CtFot7HY7AO5vbyrLfeuxf1+KNZ2ZhBCOU8+NRqOYN2+eOHbsmBg0aJAIDg52O2OFCnrppZdEUFCQ+Omnn8TFixflW15enrzOiy++KKKjo8WmTZvE3r17Rdu2bUXbtm3lx12nQj/88MPiwIEDYt26daJKlSqFngo9ZswYcfz4cTF79ux78lTowuQ/G0sI7m9P2r17t9DpdOLdd98Vp06dEt98843w8/MTX3/9tbzO1KlTRXBwsFi5cqU4dOiQePTRRws9Xbd58+Zi165dYuvWraJu3bpup+tmZmaKatWqib59+4ojR46IRYsWCT8/vwp/KvSt+vXrJ+677z751PNly5aJypUri9dee01eh/u75HJyckRqaqpITU0VAMRHH30kUlNTxblz54QQZbdvt23bJnQ6nZg2bZo4fvy4ePvtt3nqeVmaOXOmiI6OFgaDQTzwwANi586dSpekegAKvX311VfyOtevXxcvv/yyCAkJEX5+fuKxxx4TFy9edHuds2fPioSEBOHr6ysqV64sRo0aJSwWi9s6mzdvFs2aNRMGg0HUqlXLbRv3slvDDve3Z33//feiUaNGwmg0ivr164vPP//c7XG73S7GjRsnqlWrJoxGo+jSpYs4ceKE2zpXrlwRffr0Ef7+/iIwMFAMGDBA5OTkuK1z8OBB8fe//10YjUZx3333ialTp3r9valNdna2GD58uIiOjhY+Pj6iVq1a4s0333Q7jZn7u+Q2b95c6L/X/fr1E0KU7b5dsmSJuP/++4XBYBCxsbFizZo1xX4/khD5Pm6SiIiIqILhnB0iIiKq0Bh2iIiIqEJj2CEiIqIKjWGHiIiIKjSGHSIiIqrQGHaIiIioQmPYISIiogqNYYeIPKpTp04YMWKE0mW4kSQJK1asULoMIlIIP1SQiDzq6tWr0Ov1CAgIQExMDEaMGFFm4WfChAlYsWIFDhw44DZ+6dIlhISEwGg0lkkdRKQuOqULIKKKJTQ01OOvaTabYTAYSvz8e/Xq1ETkwMNYRORRrsNYnTp1wrlz5/Dqq69CkiRIkiSvs3XrVnTo0AG+vr6IiorCsGHDcO3aNfnxmJgYTJ48Gc899xwCAwMxaNAgAMDrr7+O+++/H35+fqhVqxbGjRsnXzF53rx5mDhxIg4ePChvb968eQAKHsY6fPgwHnroIfj6+iIsLAyDBg1Cbm6u/Hj//v3Rq1cvTJs2DREREQgLC8OQIUPcrs5MROUHww4RecWyZctQvXp1TJo0CRcvXsTFixcBAL/99hu6deuGJ554AocOHcLixYuxdetWDB061O3506ZNQ9OmTZGamopx48YBAAICAjBv3jwcO3YMn3zyCb744gt8/PHHAIBnnnkGo0aNQmxsrLy9Z555pkBd165dQ3x8PEJCQrBnzx4sXboUGzZsKLD9zZs347fffsPmzZsxf/58zJs3Tw5PRFS+8DAWEXlFaGgotFotAgIC3A4jJScnIykpSZ7HU7duXcyYMQMdO3bEnDlz4OPjAwB46KGHMGrUKLfXfOutt+T7MTExGD16NBYtWoTXXnsNvr6+8Pf3h06nu+NhqwULFuDGjRv4z3/+g0qVKgEAZs2ahcTERLz33nuoVq0aACAkJASzZs2CVqtF/fr10aNHD2zcuBEvvPCCR/YPEZUdhh0iKlMHDx7EoUOH8M0338hjQgjY7XakpaWhQYMGAIBWrVoVeO7ixYsxY8YM/Pbbb8jNzYXVakVgYGCxtn/8+HE0bdpUDjoA0L59e9jtdpw4cUIOO7GxsdBqtfI6EREROHz4cLG2RUTqwLBDRGUqNzcXgwcPxrBhwwo8Fh0dLd/PH0YAYMeOHUhKSsLEiRMRHx+PoKAgLFq0CB9++KFX6tTr9W7LkiTBbrd7ZVtE5F0MO0TkNQaDATabzW2sRYsWOHbsGOrUqVOs19q+fTtq1KiBN998Ux47d+7cXbd3qwYNGmDevHm4du2aHKi2bdsGjUaDevXqFasmIiofOEGZiLwmJiYGW7ZswYULF3D58mUAjjOqtm/fjqFDh+LAgQM4deoUVq5cWWCC8K3q1q2L8+fPY9GiRfjtt98wY8YMLF++vMD20tLScODAAVy+fBkmk6nA6yQlJcHHxwf9+vXDkSNHsHnzZrzyyivo27evfAiLiCoWhh0i8ppJkybh7NmzqF27NqpUqQIAaNKkCX7++WecPHkSHTp0QPPmzTF+/HhERkbe8bUeeeQRvPrqqxg6dCiaNWuG7du3y2dpuTzxxBPo1q0bOnfujCpVqmDhwoUFXsfPzw/r16/H1atX0bp1azz55JPo0qULZs2a5bk3TkSqwk9QJiIiogqNnR0iIiKq0Bh2iIiIqEJj2CEiIqIKjWGHiIiIKjSGHSIiIqrQGHaIiIioQmPYISIiogqNYYeIiIgqNIYdIiIiqtAYdoiIiKhCY9ghIiKiCo1hh4iIiCq0/wcKTfuXDF3QJgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Train the model\n",
    "iters = [iter * ITERS_PER_EVAL for iter in range(len(train_losses))]\n",
    "plt.plot(iters, train_losses, label='train')\n",
    "plt.plot(iters, val_losses, label='val')\n",
    "plt.xlabel('iteration')\n",
    "plt.ylabel('loss')\n",
    "plt.title('Training and Validation loss curves')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 472
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T11:04:12.649655Z",
     "iopub.status.busy": "2024-08-09T11:04:12.648971Z",
     "iopub.status.idle": "2024-08-09T11:04:12.913834Z",
     "shell.execute_reply": "2024-08-09T11:04:12.912978Z",
     "shell.execute_reply.started": "2024-08-09T11:04:12.649623Z"
    },
    "id": "8rCmF7RYFTT5",
    "outputId": "ca317b91-9aee-45e7-9d9d-a0c1fb6b1d0d"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj8AAAHHCAYAAABQhTneAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABJyElEQVR4nO3deXhTZd7G8TvpkrZ0o7S0LIWyCYIIsoODoPa1KoIKOgWrYHHQdxQVcRtHZdHLKS6DuDCizgAuIIiivi4DYsUZWRQFARGmKrKKLUtpS1vokjzvH9BAbHGgJDlp+/1cVy7pyZOcX07F3J7nd85jM8YYAQAANBB2qwsAAADwJ8IPAABoUAg/AACgQSH8AACABoXwAwAAGhTCDwAAaFAIPwAAoEEh/AAAgAaF8AMAABoUwg8QwLZv3y6bzaa5c+e6t02ZMkU2m+2UXm+z2TRlyhSv1jR48GANHjzYq+9ZX6SkpOjGG290//zZZ5/JZrPps88+s6wmANURfgAvGTZsmCIiInTo0KGTjsnIyFBoaKgOHDjgx8pO3+bNmzVlyhRt377d6lLcqoJE1SMoKEhNmzbVNddcoy1btlhdHoA6hPADeElGRoYOHz6sd955p8bnS0tL9d577+nSSy9VkyZNar2fhx56SIcPH67160/F5s2bNXXq1BrDz8cff6yPP/7Yp/v/LXfccYdee+01/f3vf1dGRoY+/PBDDRw4ULm5uZbVBKBuIfwAXjJs2DBFRUVp/vz5NT7/3nvvqaSkRBkZGWe0n+DgYIWFhZ3Re5yJ0NBQhYaGWrb/gQMH6vrrr1dmZqaefvppPf300zpw4IBeffVVy2oKdMYYnwdmoC4h/ABeEh4eruHDhys7O1t79+6t9vz8+fMVFRWlYcOGKT8/X/fcc4+6du2qyMhIRUdH67LLLtOGDRv+635q6vkpKyvTXXfdpYSEBPc+du/eXe21O3bs0K233qqOHTsqPDxcTZo00bXXXutxhmfu3Lm69tprJUkXXnihe5qpqm+lpp6fvXv36qabblJiYqLCwsLUrVs3vfLKKx5jqvqXnnrqKb300ktq166dHA6Hevfura+++uq/fu6TGThwoCRp69atHtt//vlnjR07VomJiXI4HOrSpYtmz55d7fVHjhzRlClTdNZZZyksLEzNmjXT8OHDPd7vqaee0oABA9SkSROFh4erZ8+eeuutt2pdc01+/vln3XTTTWrevLkcDofatGmjP/7xjyovL5d08l6vuXPnymazefwOU1JSdMUVV2jp0qXq1auXwsPD9eKLL+qcc87RhRdeWO09XC6XWrRooWuuucZj24wZM9SlSxeFhYUpMTFRt9xyiw4ePOjx2q+//lppaWmKj49XeHi42rRpo7Fjx3rpqAC+EWx1AUB9kpGRoVdeeUVvvvmmxo8f796en5+vpUuXatSoUQoPD9d3332nd999V9dee63atGmjvLw8vfjiixo0aJA2b96s5s2bn9Z+//CHP+j111/XddddpwEDBujTTz/VkCFDqo376quvtGrVKo0cOVItW7bU9u3b9cILL2jw4MHavHmzIiIidMEFF+iOO+7Qs88+qz//+c86++yzJcn9z187fPiwBg8erB9//FHjx49XmzZttGjRIt14440qKCjQnXfe6TF+/vz5OnTokG655RbZbDY98cQTGj58uH766SeFhISc1ueW5P7Sb9y4sXtbXl6e+vXrJ5vNpvHjxyshIUH//Oc/ddNNN6moqEgTJkyQJDmdTl1xxRXKzs7WyJEjdeedd+rQoUNatmyZNm3apHbt2kmSnnnmGQ0bNkwZGRkqLy/XggULdO211+qDDz6o8Tifrj179qhPnz4qKCjQzTffrE6dOunnn3/WW2+9pdLS0lqdacvJydGoUaN0yy23aNy4cerYsaPS09M1ZcoU5ebmKikpyT12xYoV2rNnj0aOHOnedsstt2ju3LnKzMzUHXfcoW3btun555/XN998o5UrVyokJER79+7VJZdcooSEBP3pT39SbGystm/frsWLF5/xMQF8ygDwmsrKStOsWTPTv39/j+2zZs0ykszSpUuNMcYcOXLEOJ1OjzHbtm0zDofDPPLIIx7bJJk5c+a4t02ePNmc+Fd3/fr1RpK59dZbPd7vuuuuM5LM5MmT3dtKS0ur1bx69Wojybz66qvubYsWLTKSzPLly6uNHzRokBk0aJD75xkzZhhJ5vXXX3dvKy8vN/379zeRkZGmqKjI47M0adLE5Ofnu8e+9957RpJ5//33q+3rRMuXLzeSzOzZs82+ffvMnj17zJIlS0z79u2NzWYza9ascY+96aabTLNmzcz+/fs93mPkyJEmJibGfRxmz55tJJnp06dX25/L5XL/+dfHrby83Jxzzjnmoosu8tjeunVrM2bMmGo113QcTzR69Ghjt9vNV199ddI6fv17rzJnzhwjyWzbts2jDklmyZIlHmNzcnKMJPPcc895bL/11ltNZGSk+3N+/vnnRpKZN2+ex7glS5Z4bH/nnXeMpBrrBgIZ016AFwUFBWnkyJFavXq1xzTE/PnzlZiYqIsvvliS5HA4ZLcf/evndDp14MABRUZGqmPHjlq3bt1p7fOjjz6SdLQR+ERVZzdOFB4e7v5zRUWFDhw4oPbt2ys2Nva093vi/pOSkjRq1Cj3tpCQEN1xxx0qLi7Wv/71L4/x6enpHmdpqqatfvrpp1Pa39ixY5WQkKDmzZvr0ksvVWFhoV577TX17t1b0tH+lrfffltDhw6VMUb79+93P9LS0lRYWOj+rG+//bbi4+N1++23V9vPiVNMJx63gwcPqrCwUAMHDqz1MTuRy+XSu+++q6FDh6pXr16/WcfpaNOmjdLS0jy2nXXWWerevbsWLlzo3uZ0OvXWW29p6NCh7s+5aNEixcTE6H/+5388jl/Pnj0VGRmp5cuXS5JiY2MlSR988IEqKipqVSdgBcIP4GVVDc1Vjc+7d+/W559/rpEjRyooKEjS0S+8p59+Wh06dJDD4VB8fLwSEhK0ceNGFRYWntb+duzYIbvd7p6iqdKxY8dqYw8fPqxJkyYpOTnZY78FBQWnvd8T99+hQwd3mKtSNU22Y8cOj+2tWrXy+LkqCP26l+RkJk2apGXLlumdd97R6NGjVVhY6LHvffv2qaCgQC+99JISEhI8HpmZmZLk7snaunWrOnbsqODg3+4A+OCDD9SvXz+FhYUpLi5OCQkJeuGFF2p9zE60b98+FRUV6Zxzzjnj9zpRmzZtatyenp6ulStX6ueff5Z09BYCe/fuVXp6unvMDz/8oMLCQjVt2rTaMSwuLnYfv0GDBmnEiBGaOnWq4uPjdeWVV2rOnDkqKyvz6mcBvI2eH8DLevbsqU6dOumNN97Qn//8Z73xxhsyxnhc5fWXv/xFDz/8sMaOHatHH31UcXFxstvtmjBhglwul89qu/322zVnzhxNmDBB/fv3V0xMjGw2m0aOHOnT/Z6oKgD+mjHmlF7ftWtXpaamSpKuuuoqlZaWaty4cfrd736n5ORk9+e4/vrrNWbMmBrf49xzzz3lej///HMNGzZMF1xwgf72t7+pWbNmCgkJ0Zw5c056ZZ8vnOwMkNPprHH7iWerTpSenq4HHnhAixYt0oQJE/Tmm28qJiZGl156qXuMy+VS06ZNNW/evBrfIyEhwV3TW2+9pS+++ELvv/++li5dqrFjx+qvf/2rvvjiC0VGRp7ORwT8hvAD+EBGRoYefvhhbdy4UfPnz1eHDh3c0zKS9NZbb+nCCy/UP/7xD4/XFRQUKD4+/rT21bp1a7lcLvdZjCo5OTnVxr711lsaM2aM/vrXv7q3HTlyRAUFBR7jTmeqpXXr1tq4caNcLpfHGZj//Oc/7ud9adq0aXrnnXf02GOPadasWe4r3pxOpzsknUy7du305ZdfqqKi4qTN1m+//bbCwsK0dOlSORwO9/Y5c+Z4pf6EhARFR0dr06ZNvzmu6gxZQUGBe7pJqn5m7b9p06aN+vTpo4ULF2r8+PFavHixrrrqKo/P1q5dO33yySc6//zzTxqiTtSvXz/169dPjz32mObPn6+MjAwtWLBAf/jDH06rNsBfmPYCfKDqLM+kSZO0fv36avf2CQoKqnamY9GiRe6piNNx2WWXSZKeffZZj+0zZsyoNram/T733HPVzh40atRIkqqFoppcfvnlys3N9egjqays1HPPPafIyEgNGjToVD5GrbVr104jRozQ3LlzlZubq6CgII0YMUJvv/12jYFi37597j+PGDFC+/fv1/PPP19tXNVxCgoKks1m8zhG27dv17vvvuuV+u12u6666iq9//77+vrrr09aR9W05r///W/3cyUlJdVuKXAq0tPT9cUXX2j27Nnav3+/x5SXJP3+97+X0+nUo48+Wu21lZWV7n8vDh48WO3fp+7du0sSU18IaJz5AXygTZs2GjBggN577z1JqhZ+rrjiCj3yyCPKzMzUgAED9O2332revHlq27btae+re/fuGjVqlP72t7+psLBQAwYMUHZ2tn788cdqY6+44gq99tpriomJUefOnbV69Wp98skn1e443b17dwUFBenxxx9XYWGhHA6HLrroIjVt2rTae95888168cUXdeONN2rt2rVKSUnRW2+9pZUrV2rGjBmKioo67c90uu699169+eabmjFjhqZNm6Zp06Zp+fLl6tu3r8aNG6fOnTsrPz9f69at0yeffKL8/HxJ0ujRo/Xqq69q4sSJWrNmjQYOHKiSkhJ98sknuvXWW3XllVdqyJAhmj59ui699FJdd9112rt3r2bOnKn27dtr48aNXqn/L3/5iz7++GMNGjRIN998s84++2z98ssvWrRokVasWKHY2FhdcsklatWqlW666Sbde++9CgoK0uzZs5WQkKCdO3ee1v5+//vf65577tE999yjuLi4amfIBg0apFtuuUVZWVlav369LrnkEoWEhOiHH37QokWL9Mwzz+iaa67RK6+8or/97W+6+uqr1a5dOx06dEgvv/yyoqOjdfnll3vl2AA+Ydl1ZkA9N3PmTCPJ9OnTp9pzR44cMXfffbdp1qyZCQ8PN+eff75ZvXp1tcvIT+VSd2OMOXz4sLnjjjtMkyZNTKNGjczQoUPNrl27ql3qfvDgQZOZmWni4+NNZGSkSUtLM//5z3+qXaJtjDEvv/yyadu2rQkKCvK4XPvXNRpjTF5envt9Q0NDTdeuXT1qPvGzPPnkk9WOx6/rrEnVZeOLFi2q8fnBgweb6OhoU1BQ4K7ptttuM8nJySYkJMQkJSWZiy++2Lz00kserystLTUPPvigadOmjXvcNddcY7Zu3eoe849//MN06NDBOBwO06lTJzNnzpwafw+1vdTdGGN27NhhRo8ebRISEozD4TBt27Y1t912mykrK3OPWbt2renbt68JDQ01rVq1MtOnTz/ppe5Dhgz5zf2df/75RpL5wx/+cNIxL730kunZs6cJDw83UVFRpmvXrua+++4ze/bsMcYYs27dOjNq1CjTqlUr43A4TNOmTc0VV1xhvv766//6eQEr2Yw5xS5DAACAeoCeHwAA0KAQfgAAQINC+AEAAA0K4QcAADQohB8AANCgEH4AAECD0uBucuhyubRnzx5FRUXVerVkAADgX8YYHTp0SM2bN6+2kPLpanDhZ8+ePUpOTra6DAAAUAu7du1Sy5Ytz+g9Glz4qbrV/q5duxQdHW1xNQAA4FQUFRUpOTnZK0vmNLjwUzXVFR0dTfgBAKCO8UbLCg3PAACgQSH8AACABoXwAwAAGhTCDwAAaFAIPwAAoEEh/AAAgAaF8AMAABoUwg8AAGhQCD8AAKBBIfwAAIAGhfADAAAaFMIPAABoUBrcwqbwDZfL6NCRSpVWVCo8JEhhIUFyBNu9sgAdAADeRPjBKfl2d6HW7shXfkm58kvLj/7T/ajQwdJyOV3G4zU2mxQeEuQOQ+GhQe6fo8NDFBsRopjwEMVW/Tki1P1zdHiIyitdKi6rVHFZpUpO/OeRShWXV6q0zCm7TQoPDVZEaJAiQo/uIyI0SOEhwWrkOPrnkKCjJzjNsfKqqjTGs14AgHc0cgTrrMQoq8s4KcIPflNpeaWeWJKjuau2n9L4YLtNlcdCkDFSablTpeVOH1YIAAg0PVrFavGt51tdxkkRfnBSX/50QPe9vVE7DpRKkgZ3TFDLxuGKa+RQXESI4iIdiosIVVyjo4/GjULkCA5SpdOlI5UuHS536kiFU4crnDp8LAQdqXCqpLxSRYcrVXC4XIWlFSo8XKGC0oqjPx+uVGFpuYqOVCo02K5GjiBFOkIU6QhSI0ewIo89Gh17GGPcAetweaVKyqv2VXl0W4VTFZWuatNvVT8yKwcA3pcYHWZ1Cb+J8INqqs72vLJ6u4yRmseEadqIc3XBWQmn9PrgILsig+yKdPCvFwAg8PDtBA9rtuXr3rc2uM/2pPdK1oNXnK3osBCLKwMAwDsIP5AkHS536smlOZqzapuMkZrFhClreFcN7tjU6tIAAPAqwg+0dke+7lm0Udv2l0iSft+rpR66ojNnewAA9RLhp4H7LGevbnrlazldRknRYcoa0VUXcrYHAFCPBcQdnmfOnKmUlBSFhYWpb9++WrNmzUnHzp07VzabzeMRFhbYXeWB6oe8Q7p9/jdyuowu7ZKkpXddQPABANR7loefhQsXauLEiZo8ebLWrVunbt26KS0tTXv37j3pa6Kjo/XLL7+4Hzt27PBjxfXDgeIyjX3lKx0qq1SfNnF6dtR5iglnmgsAUP9ZHn6mT5+ucePGKTMzU507d9asWbMUERGh2bNnn/Q1NptNSUlJ7kdiYqIfK677yiqd+t/X12pX/mG1iovQrOt7KjTY8n8VAADwC0u/8crLy7V27Vqlpqa6t9ntdqWmpmr16tUnfV1xcbFat26t5ORkXXnllfruu+9OOrasrExFRUUej4bMGKM/L96kr7YfVFRYsGbf2EtxjUKtLgsAAL+xNPzs379fTqez2pmbxMRE5ebm1viajh07avbs2Xrvvff0+uuvy+VyacCAAdq9e3eN47OyshQTE+N+JCcne/1z1CWz/vWT3l63W0F2m2Ze10Ptmwbu2isAAPhCnZvr6N+/v0aPHq3u3btr0KBBWrx4sRISEvTiiy/WOP6BBx5QYWGh+7Fr1y4/Vxw4ln6XqyeW/keSNHlo51O+YzMAAPWJpZe6x8fHKygoSHl5eR7b8/LylJSUdErvERISovPOO08//vhjjc87HA45HI4zrrWu+25PoSYsWC9jpNH9W2t0/xSrSwIAwBKWnvkJDQ1Vz549lZ2d7d7mcrmUnZ2t/v37n9J7OJ1Offvtt2rWrJmvyqzz9hYd0R9e+VqHK5wa2CFek67obHVJAABYxvKbHE6cOFFjxoxRr1691KdPH82YMUMlJSXKzMyUJI0ePVotWrRQVlaWJOmRRx5Rv3791L59exUUFOjJJ5/Ujh079Ic//MHKjxGwjlQ4Ne61tfql8IjaJTTS89f1UHBQnZvtBADAaywPP+np6dq3b58mTZqk3Nxcde/eXUuWLHE3Qe/cuVN2+/Ev64MHD2rcuHHKzc1V48aN1bNnT61atUqdO3M249eMMbpn0QZt2FWg2IgQzb6xN/fyAQA0eDZjjLG6CH8qKipSTEyMCgsLFR0dbXU5PvX+hj26/Y1vFBJk02s39VW/tk2sLgkAgFrx5vc38x/1WPaWo43kY89vQ/ABAOAYwk89ZYzRyq0HJEmDWa8LAAA3wk899cPeYu07VKawELt6tI61uhwAAAIG4aeeWvnjfklS75Q4OYKDLK4GAIDAQfipp6rCz/nt4y2uBACAwEL4qYcqnS598VO+JOl3hB8AADwQfuqhDbsLVVxWqdiIEHVuVr8v5wcA4HQRfuqhqimvAe2ayG63WVwNAACBhfBTDx0PP0x5AQDwa4Sfeqa0vFLrdh6URL8PAAA1IfzUM19tP6gKp1GL2HC1bhJhdTkAAAQcwk89c/wS9yay2ej3AQDg1wg/9Qz39wEA4LcRfuqR/JJyfbenSBLNzgAAnAzhpx5ZtfXoWZ9OSVFKiHJYXA0AAIGJ8FOPrPzx6CrunPUBAODkCD/1SFW/z+86NLG4EgAAAhfhp57YlV+qnfmlCrbb1KcN4QcAgJMh/NQTVWd9uifHKtIRbHE1AAAELsJPPbFy67F+Hy5xBwDgNxF+6gGXy2hVVb8P4QcAgN9E+KkHcvIO6UBJucJDgtQ9OdbqcgAACGiEn3qgqt+nb9s4hQbzKwUA4LfwTVkPrKha0oL7+wAA8F8Rfuq48kqX1mzLl8R6XgAAnArCTx23fleBSsudimsUqk5JUVaXAwBAwCP81HFV/T4D2jWR3W6zuBoAAAIf4aeOqwo/THkBAHBqCD91WHFZpdbvKpDE/X0AADhVhJ86bM22A6p0GbWKi1ByXITV5QAAUCcQfuqwFT8cXdLi/PYsZAoAwKki/NRhq7bS7wMAwOki/NRR+w6V6T+5hyRJ/dty5gcAgFNF+Kmjqs76dG4WrSaRDourAQCg7iD81FFf/FR1V2fO+gAAcDoIP3XUf3KLJEnntoy1thAAAOoYwk8dZIzR98f6fTqypAUAAKeF8FMH7T54WCXlToUE2dQmvpHV5QAAUKcQfuqg7/OOnvVplxCpkCB+hQAAnA6+OeugnGPh56xEprwAADhdhJ86iH4fAABqj/BTB1Xd3LAjZ34AADhthJ86psLp0k/7SiRx5gcAgNog/NQxOw6UqNzpUkRokFrEhltdDgAAdQ7hp47JyS2WdLTZ2W63WVwNAAB1D+Gnjsk5dmdn+n0AAKgdwk8d477MnX4fAABqhfBTx3yfd3TaqxPhBwCAWiH81CGHy53afuDolV7c4BAAgNoh/NQhP+4tljFSXKNQxUeGWl0OAAB1EuGnDjm+rEWkbDau9AIAoDYIP3VI1YKmnZKiLa4EAIC6i/BTh1Qta0G/DwAAtUf4qUOOL2gaaXElAADUXQERfmbOnKmUlBSFhYWpb9++WrNmzSm9bsGCBbLZbLrqqqt8W2AAKCytUG7REUmc+QEA4ExYHn4WLlyoiRMnavLkyVq3bp26deumtLQ07d279zdft337dt1zzz0aOHCgnyq11vd7j571aREbrqiwEIurAQCg7rI8/EyfPl3jxo1TZmamOnfurFmzZikiIkKzZ88+6WucTqcyMjI0depUtW3b1o/VWud4vw9TXgAAnAlLw095ebnWrl2r1NRU9za73a7U1FStXr36pK975JFH1LRpU910003/dR9lZWUqKiryeNRFVf0+LGsBAMCZsTT87N+/X06nU4mJiR7bExMTlZubW+NrVqxYoX/84x96+eWXT2kfWVlZiomJcT+Sk5PPuG4r5Lgvcyf8AABwJiyf9jodhw4d0g033KCXX35Z8fHxp/SaBx54QIWFhe7Hrl27fFyl9xlj3Pf4odkZAIAzE2zlzuPj4xUUFKS8vDyP7Xl5eUpKSqo2fuvWrdq+fbuGDh3q3uZyuSRJwcHBysnJUbt27Txe43A45HA4fFC9/+w9VKaC0grZbVK7BHp+AAA4E5ae+QkNDVXPnj2VnZ3t3uZyuZSdna3+/ftXG9+pUyd9++23Wr9+vfsxbNgwXXjhhVq/fn2dndL6b3KO9fukxDdSWEiQxdUAAFC3WXrmR5ImTpyoMWPGqFevXurTp49mzJihkpISZWZmSpJGjx6tFi1aKCsrS2FhYTrnnHM8Xh8bGytJ1bbXJ9/T7wMAgNdYHn7S09O1b98+TZo0Sbm5uerevbuWLFniboLeuXOn7PY61ZrkdTksawEAgNfYjDHG6iL8qaioSDExMSosLFR0dN1YIHTY8yu0cXehXsjoocu6NrO6HAAA/M6b398N+5RKHeByHb/SqyPTXgAAnDHCT4DbdbBURypcCg22q3WTRlaXAwBAnUf4CXBV/T4dmkYqyG6zuBoAAOo+wk+Aqwo/HWl2BgDAKwg/AS6Hfh8AALyK8BPg3MtaEH4AAPAKwk8AK6906ad9JZKY9gIAwFsIPwHsp/3FqnQZRYUFq1lMmNXlAABQLxB+AtiJzc42G1d6AQDgDYSfAEa/DwAA3kf4CWA5ucWS6PcBAMCbCD8BLCevSBKXuQMA4E2EnwBVUlapXfmHJbGaOwAA3kT4CVA/7D065ZUQ5VBco1CLqwEAoP4g/ASo749d6dWJKS8AALyK8BOg/nMs/DDlBQCAdxF+AlTVZe5c6QUAgHcRfgJUDvf4AQDAJwg/ASi/pFz7DpVJks5KjLS4GgAA6hfCTwCqWtaiVVyEIkKDLa4GAID6hfATgNzLWtDvAwCA1xF+AlBVv0/HJKa8AADwNsJPAPqey9wBAPAZwk8Ayi06IklKjouwuBIAAOofwk8AOlhSLklqHMGyFgAAeBvhJ8CUVTpVUu6UJMURfgAA8DrCT4ApKK2QJAXZbYoK4zJ3AAC8jfATYPLdU14hstttFlcDAED9Q/gJMFX9PrFMeQEA4BOEnwBz8Ni0F/0+AAD4BuEnwOSXHpv2ahRicSUAANRPhJ8AUzXtFdeIMz8AAPgC4SfA5NPzAwCATxF+AszBY9Ne9PwAAOAbhJ8AU9Xw3JhpLwAAfILwE2CO9/zQ8AwAgC8QfgJMPut6AQDgU4SfAFPV80P4AQDANwg/AeRIhVOlxxY1pecHAADfIPwEkBMXNY1mUVMAAHyC8BNATuz3sdlY1BQAAF8g/ASQ4/0+XOkFAICvEH4CiDv80O8DAIDPEH4CiPseP1zpBQCAzxB+Akh+CXd3BgDA1wg/AcS9rhd3dwYAwGcIPwGEGxwCAOB7hJ8AwtIWAAD4HuEngByf9iL8AADgK4SfAHKQhmcAAHyO8BNAjk970fAMAICvEH4CxJEKpw5XsKgpAAC+RvgJEFX9PsF2m6IcLGoKAICvEH4ChHvKqxGLmgIA4EsBEX5mzpyplJQUhYWFqW/fvlqzZs1Jxy5evFi9evVSbGysGjVqpO7du+u1117zY7W+UdXszNIWAAD4luXhZ+HChZo4caImT56sdevWqVu3bkpLS9PevXtrHB8XF6cHH3xQq1ev1saNG5WZmanMzEwtXbrUz5V7V9W0VyzNzgAA+JTl4Wf69OkaN26cMjMz1blzZ82aNUsRERGaPXt2jeMHDx6sq6++WmeffbbatWunO++8U+eee65WrFjh58q9i3v8AADgH5aGn/Lycq1du1apqanubXa7XampqVq9evV/fb0xRtnZ2crJydEFF1xQ45iysjIVFRV5PALRiT0/AADAdywNP/v375fT6VRiYqLH9sTEROXm5p70dYWFhYqMjFRoaKiGDBmi5557Tv/zP/9T49isrCzFxMS4H8nJyV79DN5y8Fj4oecHAADfsnzaqzaioqK0fv16ffXVV3rsscc0ceJEffbZZzWOfeCBB1RYWOh+7Nq1y7/FnqKDpUcbnun5AQDAtyy9oUx8fLyCgoKUl5fnsT0vL09JSUknfZ3dblf79u0lSd27d9eWLVuUlZWlwYMHVxvrcDjkcDi8Wrcv0PMDAIB/WHrmJzQ0VD179lR2drZ7m8vlUnZ2tvr373/K7+NyuVRWVuaLEv2Gnh8AAPzD8lsJT5w4UWPGjFGvXr3Up08fzZgxQyUlJcrMzJQkjR49Wi1atFBWVpakoz08vXr1Urt27VRWVqaPPvpIr732ml544QUrP8YZo+cHAAD/sDz8pKena9++fZo0aZJyc3PVvXt3LVmyxN0EvXPnTtntx09QlZSU6NZbb9Xu3bsVHh6uTp066fXXX1d6erpVH8Er8pn2AgDAL2zGGGN1Ef5UVFSkmJgYFRYWKjo62upyJEmHy506e9ISSdK3Uy5RVBhNzwAAnMib39918mqv+qaq2TkkyKZIFjUFAMCnCD8BwN3sHMGipgAA+BrhJwBwmTsAAP5D+AkA3OAQAAD/IfwEAPdl7pz5AQDA5065u3b48OGn/KaLFy+uVTEN1Yk9PwAAwLdOOfzExMT4so4GjZ4fAAD855TDz5w5c3xZR4NW1fPDmR8AAHyPnp8AcNC9rhcNzwAA+Nopn/k577zzTvkeNOvWrat1QQ0RPT8AAPjPKYefq666yodlNGz0/AAA4D+nHH4mT57syzoaNM78AADgP/T8WOxwuVNllS5JUmPO/AAA4HO1WkXT6XTq6aef1ptvvqmdO3eqvLzc4/n8/HyvFNcQ5B+b8goNsqtRaJDF1QAAUP/V6szP1KlTNX36dKWnp6uwsFATJ07U8OHDZbfbNWXKFC+XWL+deKUXi5oCAOB7tQo/8+bN08svv6y7775bwcHBGjVqlP7+979r0qRJ+uKLL7xdY71Gvw8AAP5Vq/CTm5urrl27SpIiIyNVWFgoSbriiiv04Ycfeq+6BqDqSi/CDwAA/lGr8NOyZUv98ssvkqR27drp448/liR99dVXcjgc3quuAWBRUwAA/KtW4efqq69Wdna2JOn222/Xww8/rA4dOmj06NEaO3asVwus7/Krlrbg7s4AAPhFra72mjZtmvvP6enpat26tVatWqUOHTpo6NChXiuuIXCf+WHaCwAAv6hV+Pm1fv36qV+/ft54qwbH3fPDtBcAAH5Rq2mvrKwszZ49u9r22bNn6/HHHz/johoSGp4BAPCvWoWfF198UZ06daq2vUuXLpo1a9YZF9WQ5JdU9fwQfgAA8IdaX+rerFmzatsTEhLcV4Hh1NDzAwCAf9Uq/CQnJ2vlypXVtq9cuVLNmzc/46IaCmOMe3kLrvYCAMA/atXwPG7cOE2YMEEVFRW66KKLJEnZ2dm67777dPfdd3u1wPrscIVT5VWLmnLmBwAAv6hV+Ln33nt14MAB3Xrrre5FTcPCwnT//ffrgQce8GqB9VnV0hahwXZFsKgpAAB+UavwY7PZ9Pjjj+vhhx/Wli1bFB4erg4dOnB359N08Fizc1xEKIuaAgDgJ7Xq+amSm5ur/Px8tWvXTg6HQ8YYb9XVIORzjx8AAPyuVuHnwIEDuvjii3XWWWfp8ssvd1/hddNNN9HzcxoKSqvW9aLZGQAAf6lV+LnrrrsUEhKinTt3KiIiwr09PT1dS5Ys8Vpx9V1Vz08szc4AAPhNrXp+Pv74Yy1dulQtW7b02N6hQwft2LHDK4U1BNzjBwAA/6vVmZ+SkhKPMz5V8vPzaXo+DfT8AADgf7UKPwMHDtSrr77q/tlms8nlcumJJ57QhRde6LXi6ruDpVVXe9HzAwCAv9Rq2uvJJ5/URRddpK+//lrl5eW677779N133yk/P7/GOz+jZlXTXpz5AQDAf047/FRUVOiOO+7Q+++/r2XLlikqKkrFxcUaPny4brvtthrX/ELNqhqeubszAAD+c9rhJyQkRBs3blTjxo314IMP+qKmBuOg+1J3wg8AAP5Sq56f66+/Xv/4xz+8XUuDYoxx3+GZaS8AAPynVj0/lZWVmj17tj755BP17NlTjRo18nh++vTpXimuPistd6rceXRRUy51BwDAf2oVfjZt2qQePXpIkr7//nuP51ij6tRU9fs4gu0KZ1FTAAD8plbhZ/ny5d6uo8Gh3wcAAGuc0cKmqD2u9AIAwBqEH4sUVN3gkDM/AAD4FeHHIscXNeXuzgAA+BPhxyL0/AAAYA3Cj0Xo+QEAwBqEH4vQ8wMAgDUIPxbJZ1FTAAAsQfixSFXPT2MangEA8CvCj0Xo+QEAwBqEHwsYY7jaCwAAixB+LFBS7lSF00jizA8AAP5G+LHAwWNTXmEhLGoKAIC/EX4sUNXvE8dZHwAA/C4gws/MmTOVkpKisLAw9e3bV2vWrDnp2JdfflkDBw5U48aN1bhxY6Wmpv7m+ECUX8pl7gAAWMXy8LNw4UJNnDhRkydP1rp169StWzelpaVp7969NY7/7LPPNGrUKC1fvlyrV69WcnKyLrnkEv38889+rrz2Cmh2BgDAMpaHn+nTp2vcuHHKzMxU586dNWvWLEVERGj27Nk1jp83b55uvfVWde/eXZ06ddLf//53uVwuZWdn+7ny2ssvOXp3Z5qdAQDwP0vDT3l5udauXavU1FT3NrvdrtTUVK1evfqU3qO0tFQVFRWKi4ur8fmysjIVFRV5PKx2sIQbHAIAYBVLw8/+/fvldDqVmJjosT0xMVG5ubmn9B7333+/mjdv7hGgTpSVlaWYmBj3Izk5+YzrPlP0/AAAYB3Lp73OxLRp07RgwQK98847CgsLq3HMAw88oMLCQvdj165dfq6yOnp+AACwTrCVO4+Pj1dQUJDy8vI8tufl5SkpKek3X/vUU09p2rRp+uSTT3TuueeedJzD4ZDD4fBKvd7C0hYAAFjH0jM/oaGh6tmzp0ezclXzcv/+/U/6uieeeEKPPvqolixZol69evmjVK86SMMzAACWsfTMjyRNnDhRY8aMUa9evdSnTx/NmDFDJSUlyszMlCSNHj1aLVq0UFZWliTp8ccf16RJkzR//nylpKS4e4MiIyMVGRlp2ec4Hcd7fmh4BgDA3ywPP+np6dq3b58mTZqk3Nxcde/eXUuWLHE3Qe/cuVN2+/ETVC+88ILKy8t1zTXXeLzP5MmTNWXKFH+WXivGGPfVXvT8AADgfzZjjLG6CH8qKipSTEyMCgsLFR0d7ff9HzpSoa5TPpYk/efRSxUWwtpeAAD8N978/q7TV3vVRVX9PhGhQQQfAAAsQPjxM3e/D83OAABYgvDjZ+67O9PsDACAJQg/fnaQMz8AAFiK8ONn+VzpBQCApQg/fsaZHwAArEX48bN87u4MAIClCD9+dnxRUxqeAQCwAuHHz9yLmtLzAwCAJQg/fkbPDwAA1iL8+Bk9PwAAWIvw40fGmBN6fgg/AABYgfDjR0VHKlXpOrqObGwEDc8AAFiB8ONHB4rLJEmRjmAWNQUAwCKEHz+qutKrSSRTXgAAWIXw40cHWNoCAADLEX78yH3mh/ADAIBlCD9+VNXzw5kfAACsQ/jxowPunh+HxZUAANBwEX78iGkvAACsR/jxo3wangEAsBzhx4/2FxN+AACwGuHHj/JLjjY8x9PzAwCAZQg/fmKMYdoLAIAAQPjxk0NllapwHl3Xi/ADAIB1CD9+cuBYv0+j0CDW9QIAwEKEHz+p6veJY10vAAAsRfjxk6ozP00a0ewMAICVCD9+wg0OAQAIDIQfP2FFdwAAAgPhx0+qpr3o+QEAwFqEHz9x3+CQnh8AACxF+PETpr0AAAgMhB8/cd/dmWkvAAAsRfjxk+OXuhN+AACwEuHHD05c16sJi5oCAGApwo8fFJdVqtzpksSZHwAArEb48YOqsz4RrOsFAIDlCD9+sL+YK70AAAgUhB8/oN8HAIDAQfjxg6obHNLvAwCA9Qg/fsANDgEACByEHz/gHj8AAAQOwo8fHO/5IfwAAGA1wo8fHJ/2ouEZAACrEX78gIZnAAACB+HHDw5wnx8AAAIG4cfHjDFc7QUAQAAh/PhYSblT5ZXH1vWi4RkAAMsRfnws/9iUV3hIkCJCgy2uBgAAEH587MCxZmemvAAACAyEHx9z3+CQKS8AAAIC4cfH3Dc45MwPAAABgfDjY9zgEACAwGJ5+Jk5c6ZSUlIUFhamvn37as2aNScd+91332nEiBFKSUmRzWbTjBkz/FdoLblvcMi0FwAAAcHS8LNw4UJNnDhRkydP1rp169StWzelpaVp7969NY4vLS1V27ZtNW3aNCUlJfm52trhBocAAAQWS8PP9OnTNW7cOGVmZqpz586aNWuWIiIiNHv27BrH9+7dW08++aRGjhwph6NuTCMdoOcHAICAYln4KS8v19q1a5Wamnq8GLtdqampWr16tdf2U1ZWpqKiIo+HP7GiOwAAgcWy8LN//345nU4lJiZ6bE9MTFRubq7X9pOVlaWYmBj3Izk52WvvfSryaXgGACCgWN7w7GsPPPCACgsL3Y9du3b5bd/GGO0vZkV3AAACiWXrLcTHxysoKEh5eXke2/Py8rzazOxwOCzrDyotd6qMdb0AAAgolp35CQ0NVc+ePZWdne3e5nK5lJ2drf79+1tVlldVTXmFhdhZ1wsAgABh6TfyxIkTNWbMGPXq1Ut9+vTRjBkzVFJSoszMTEnS6NGj1aJFC2VlZUk62iS9efNm959//vlnrV+/XpGRkWrfvr1ln+Nkjl/pRb8PAACBwtLwk56ern379mnSpEnKzc1V9+7dtWTJEncT9M6dO2W3Hz85tWfPHp133nnun5966ik99dRTGjRokD777DN/l/9fHShmUVMAAAKN5XMx48eP1/jx42t87teBJiUlRcYYP1TlHQe4zB0AgIBT76/2stLxy9wJPwAABArCjw+xojsAAIGH8OND+909PzQ8AwAQKAg/PsSZHwAAAg/hx4dY1wsAgMBD+PGhA8U0PAMAEGgIPz6Uz00OAQAIOIQfHyktr9ThCqckKY5pLwAAAgbhx0eqprwcwXY1Cg2yuBoAAFCF8OMjJ17pZbPZLK4GAABUIfz4iPvuzkx5AQAQUAg/PsINDgEACEyEHx+pOvMTz2XuAAAEFMKPj7CoKQAAgYnw4yMH6PkBACAgEX585MCxnh/W9QIAILAQfnyEuzsDABCYCD8+wrQXAACBifDjIyfe5BAAAAQOwo8PHC53qrT82LpehB8AAAIK4ccHDpQcbXYODbIr0hFscTUAAOBEhB8fcE95RbKuFwAAgYbw4wMHuMEhAAABi/DjAweKCT8AAAQqwo8P5Jdwg0MAAAIV4ccHDrh7frjBIQAAgYbw4wP5THsBABCwCD8+cIAbHAIAELAIPz7A1V4AAAQuwo8PuBue6fkBACDgEH58oKrnh2kvAAACD+HHy45UOFVSta4XK7oDABBwCD9eVtXvExJkUxTregEAEHAIP152fMrLwbpeAAAEIMKPl1Wt6M6VXgAABCbCj5eduKI7AAAIPIQfL2NRUwAAAhvhx8uO392Ze/wAABCICD9edvwGh5z5AQAgEBF+vCyfpS0AAAhohB8v20/PDwAAAY3w42X5rOgOAEBA4xbEXnb8UncangHAn5xOpyoqKqwuA2cgJCREQUFBPt8P4ceLjlQ4VVxWKYlpLwDwp+LiYu3evVvGGKtLwRmw2Wxq2bKlIiMjfbofwo8X5Z+wrld0GIcWAPzB6XRq9+7dioiIUEJCAksL1VHGGO3bt0+7d+9Whw4dfHoGiG9oL6oKP40jQvnLBwB+UlFRIWOMEhISFB4ebnU5OAMJCQnavn27KioqfBp+aHj2ogP0+wCAZfifzrrPX79Dwo8XuW9wSL8PAAABi/DjRazrBQDwl5SUFM2YMcPqMn7T4MGDNWHCBPfPgVIz4ceLDnB3ZwAAAh7hx4vyj535iWddLwBAAGvo90Mi/HjR8TM/NDwDAE7upZdeUvPmzeVyuTy2X3nllRo7dqy2bt2qK6+8UomJiYqMjFTv3r31ySef1Hp/NptNL7zwgoYNG6ZGjRrpsccekyS999576tGjh8LCwtS2bVtNnTpVlZWV7tcVFBTolltuUWJiosLCwnTOOefogw8+kCQdOHBAo0aNUosWLRQREaGuXbvqjTfeqHWN/sSl7l5U1fDMtBcAWMcYo8MVTkv2HR4SdEpXLF177bW6/fbbtXz5cl188cWSpPz8fC1ZskQfffSRiouLdfnll+uxxx6Tw+HQq6++qqFDhyonJ0etWrWqVW1TpkzRtGnTNGPGDAUHB+vzzz/X6NGj9eyzz2rgwIHaunWrbr75ZknS5MmT5XK5dNlll+nQoUN6/fXX1a5dO23evNl9CfqRI0fUs2dP3X///YqOjtaHH36oG264Qe3atVOfPn1qVaO/EH686Pil7oQfALDK4QqnOk9aasm+Nz+SpojQ//7V2rhxY1122WWaP3++O/y89dZbio+P14UXXii73a5u3bq5xz/66KN655139H//938aP358rWq77rrrlJmZ6f557Nix+tOf/qQxY8ZIktq2batHH31U9913nyZPnqxPPvlEa9as0ZYtW3TWWWe5x1Rp0aKF7rnnHvfPt99+u5YuXao333wz4MNPQEx7zZw5UykpKQoLC1Pfvn21Zs2a3xy/aNEiderUSWFhYeratas++ugjP1X626p6frjUHQDw32RkZOjtt99WWdnRWYN58+Zp5MiRstvtKi4u1j333KOzzz5bsbGxioyM1JYtW7Rz585a769Xr14eP2/YsEGPPPKIIiMj3Y9x48bpl19+UWlpqdavX6+WLVu6g8+vOZ1OPfroo+ratavi4uIUGRmppUuXnlGN/mL5mZ+FCxdq4sSJmjVrlvr27asZM2YoLS1NOTk5atq0abXxq1at0qhRo5SVlaUrrrhC8+fP11VXXaV169bpnHPOseATHFVW6dShY+t6NaHnBwAsEx4SpM2PpFm271M1dOhQGWP04Ycfqnfv3vr888/19NNPS5LuueceLVu2TE899ZTat2+v8PBwXXPNNSovL691bY0aNfL4ubi4WFOnTtXw4cOrjQ0LC/uvd8t+8skn9cwzz2jGjBnq2rWrGjVqpAkTJpxRjf5iefiZPn26xo0b5z4VN2vWLH344YeaPXu2/vSnP1Ub/8wzz+jSSy/VvffeK+noqcBly5bp+eef16xZs/xa+4kOlhztnA+22xQdbvlhBYAGy2azndLUk9XCwsI0fPhwzZs3Tz/++KM6duyoHj16SJJWrlypG2+8UVdffbWko0Fl+/btXt1/jx49lJOTo/bt29f4/Lnnnqvdu3fr+++/r/Hsz8qVK3XllVfq+uuvlyS5XC59//336ty5s1fr9AVLp73Ky8u1du1apaamurfZ7XalpqZq9erVNb5m9erVHuMlKS0t7aTjy8rKVFRU5PHwhf3FR09bNm7Eul4AgFOTkZHh/h/+jIwM9/YOHTpo8eLFWr9+vTZs2KDrrruu2pVhZ2rSpEl69dVXNXXqVH333XfasmWLFixYoIceekiSNGjQIF1wwQUaMWKEli1bpm3btumf//ynlixZ4q5x2bJlWrVqlbZs2aJbbrlFeXl5Xq3RVywNP/v375fT6VRiYqLH9sTEROXm5tb4mtzc3NMan5WVpZiYGPcjOTnZO8X/Smm5U1GOYPp9AACn7KKLLlJcXJxycnJ03XXXubdPnz5djRs31oABAzR06FClpaW5zwp5S1pamj744AN9/PHH6t27t/r166enn35arVu3do95++231bt3b40aNUqdO3fWfffdJ6fz6JV0Dz30kHr06KG0tDQNHjxYSUlJuuqqq7xao68E/nnBM/TAAw9o4sSJ7p+Liop8EoD6tInTt1PTVOn0bjIHANRfdrtde/bsqbY9JSVFn376qce22267zePn05kGM8bUuD0tLU1paSfvj4qLi9Ps2bNP+ty77777m/v97LPPPH729tRdbVkafuLj4xUUFFTtNFleXp6SkpJqfE1SUtJpjXc4HHI4/NeAHBwUEBfQAQCAk7D0mzo0NFQ9e/ZUdna2e5vL5VJ2drb69+9f42v69+/vMV6Sli1bdtLxAADUZ/PmzfO4XP3ER5cuXawuLyBZPu01ceJEjRkzRr169VKfPn00Y8YMlZSUuK/+Gj16tFq0aKGsrCxJ0p133qlBgwbpr3/9q4YMGaIFCxbo66+/1ksvvWTlxwAAwBLDhg1T3759a3wuJCTEz9XUDZaHn/T0dO3bt0+TJk1Sbm6uunfvriVLlribmnfu3Cm7/fgJqgEDBmj+/Pl66KGH9Oc//1kdOnTQu+++a+k9fgAAsEpUVJSioqKsLqNOsZmTdUHVU0VFRYqJiVFhYaGio6OtLgcAcIaOHDmibdu2qU2bNgoLC7O6HJyB3/pdevP7m+5cAEC90MD+X75e8tfvkPADAKjTqlYZrwvLKuC3Vf0Oq36nvmJ5zw8AAGciODhYERER2rdvn0JCQjz6RFF3uFwu7du3TxEREQoO9m08IfwAAOo0m82mZs2aadu2bdqxY4fV5eAM2O12tWrVyufLRBF+AAB1XmhoqDp06MDUVx0XGhrqlzN3hB8AQL1gt9u52gunhIlRAADQoBB+AABAg0L4AQAADUqD6/mpuoFSUVGRxZUAAIBTVfW97Y0bITa48HPo0CFJUnJyssWVAACA03Xo0CHFxMSc0Xs0uLW9XC6X9uzZo6ioKK/fR6CoqEjJycnatWsX64b5Acfbvzje/sXx9i+Ot3/V5ngbY3To0CE1b978jC+Hb3Bnfux2u1q2bOnTfURHR/OXx4843v7F8fYvjrd/cbz963SP95me8alCwzMAAGhQCD8AAKBBIfx4kcPh0OTJk+VwOKwupUHgePsXx9u/ON7+xfH2L6uPd4NreAYAAA0bZ34AAECDQvgBAAANCuEHAAA0KIQfAADQoBB+vGTmzJlKSUlRWFiY+vbtqzVr1lhdUsDLyspS7969FRUVpaZNm+qqq65STk6Ox5gjR47otttuU5MmTRQZGakRI0YoLy/PY8zOnTs1ZMgQRUREqGnTprr33ntVWVnpMeazzz5Tjx495HA41L59e82dO9fXHy/gTZs2TTabTRMmTHBv43h7188//6zrr79eTZo0UXh4uLp27aqvv/7a/bwxRpMmTVKzZs0UHh6u1NRU/fDDDx7vkZ+fr4yMDEVHRys2NlY33XSTiouLPcZs3LhRAwcOVFhYmJKTk/XEE0/45fMFEqfTqYcfflht2rRReHi42rVrp0cffdRjHSiO95n597//raFDh6p58+ay2Wx69913PZ735/FdtGiROnXqpLCwMHXt2lUfffTR6X0YgzO2YMECExoaambPnm2+++47M27cOBMbG2vy8vKsLi2gpaWlmTlz5phNmzaZ9evXm8svv9y0atXKFBcXu8f87//+r0lOTjbZ2dnm66+/Nv369TMDBgxwP19ZWWnOOecck5qaar755hvz0Ucfmfj4ePPAAw+4x/z0008mIiLCTJw40WzevNk899xzJigoyCxZssSvnzeQrFmzxqSkpJhzzz3X3Hnnne7tHG/vyc/PN61btzY33nij+fLLL81PP/1kli5dan788Uf3mGnTppmYmBjz7rvvmg0bNphhw4aZNm3amMOHD7vHXHrppaZbt27miy++MJ9//rlp3769GTVqlPv5wsJCk5iYaDIyMsymTZvMG2+8YcLDw82LL77o189rtccee8w0adLEfPDBB2bbtm1m0aJFJjIy0jzzzDPuMRzvM/PRRx+ZBx980CxevNhIMu+8847H8/46vitXrjRBQUHmiSeeMJs3bzYPPfSQCQkJMd9+++0pfxbCjxf06dPH3Hbbbe6fnU6nad68ucnKyrKwqrpn7969RpL517/+ZYwxpqCgwISEhJhFixa5x2zZssVIMqtXrzbGHP3LaLfbTW5urnvMCy+8YKKjo01ZWZkxxpj77rvPdOnSxWNf6enpJi0tzdcfKSAdOnTIdOjQwSxbtswMGjTIHX443t51//33m9/97ncnfd7lcpmkpCTz5JNPurcVFBQYh8Nh3njjDWOMMZs3bzaSzFdffeUe889//tPYbDbz888/G2OM+dvf/mYaN27sPv5V++7YsaO3P1JAGzJkiBk7dqzHtuHDh5uMjAxjDMfb234dfvx5fH//+9+bIUOGeNTTt29fc8stt5xy/Ux7naHy8nKtXbtWqamp7m12u12pqalavXq1hZXVPYWFhZKkuLg4SdLatWtVUVHhcWw7deqkVq1auY/t6tWr1bVrVyUmJrrHpKWlqaioSN999517zInvUTWmof5+brvtNg0ZMqTaMeF4e9f//d//qVevXrr22mvVtGlTnXfeeXr55Zfdz2/btk25ubkexyomJkZ9+/b1ON6xsbHq1auXe0xqaqrsdru+/PJL95gLLrhAoaGh7jFpaWnKycnRwYMHff0xA8aAAQOUnZ2t77//XpK0YcMGrVixQpdddpkkjrev+fP4euO/MYSfM7R//345nU6PLwNJSkxMVG5urkVV1T0ul0sTJkzQ+eefr3POOUeSlJubq9DQUMXGxnqMPfHY5ubm1njsq577rTFFRUU6fPiwLz5OwFqwYIHWrVunrKysas9xvL3rp59+0gsvvKAOHTpo6dKl+uMf/6g77rhDr7zyiqTjx+u3/tuRm5urpk2bejwfHBysuLi40/qdNAR/+tOfNHLkSHXq1EkhISE677zzNGHCBGVkZEjiePuaP4/vycaczvFvcKu6IzDddttt2rRpk1asWGF1KfXWrl27dOedd2rZsmUKCwuzupx6z+VyqVevXvrLX/4iSTrvvPO0adMmzZo1S2PGjLG4uvrnzTff1Lx58zR//nx16dJF69ev14QJE9S8eXOON6rhzM8Zio+PV1BQULUrYvLy8pSUlGRRVXXL+PHj9cEHH2j58uVq2bKle3tSUpLKy8tVUFDgMf7EY5uUlFTjsa967rfGREdHKzw83NsfJ2CtXbtWe/fuVY8ePRQcHKzg4GD961//0rPPPqvg4GAlJiZyvL2oWbNm6ty5s8e2s88+Wzt37pR0/Hj91n87kpKStHfvXo/nKysrlZ+ff1q/k4bg3nvvdZ/96dq1q2644Qbddddd7rOcHG/f8ufxPdmY0zn+hJ8zFBoaqp49eyo7O9u9zeVyKTs7W/3797ewssBnjNH48eP1zjvv6NNPP1WbNm08nu/Zs6dCQkI8jm1OTo527tzpPrb9+/fXt99+6/EXatmyZYqOjnZ/8fTv39/jParGNLTfz8UXX6xvv/1W69evdz969eqljIwM95853t5z/vnnV7t1w/fff6/WrVtLktq0aaOkpCSPY1VUVKQvv/zS43gXFBRo7dq17jGffvqpXC6X+vbt6x7z73//WxUVFe4xy5YtU8eOHdW4cWOffb5AU1paKrvd8ystKChILpdLEsfb1/x5fL3y35hTbo3GSS1YsMA4HA4zd+5cs3nzZnPzzTeb2NhYjytiUN0f//hHExMTYz777DPzyy+/uB+lpaXuMf/7v/9rWrVqZT799FPz9ddfm/79+5v+/fu7n6+69PqSSy4x69evN0uWLDEJCQk1Xnp97733mi1btpiZM2c2yEuva3Li1V7GcLy9ac2aNSY4ONg89thj5ocffjDz5s0zERER5vXXX3ePmTZtmomNjTXvvfee2bhxo7nyyitrvDT4vPPOM19++aVZsWKF6dChg8elwQUFBSYxMdHccMMNZtOmTWbBggUmIiKiQVx6faIxY8aYFi1auC91X7x4sYmPjzf33XefewzH+8wcOnTIfPPNN+abb74xksz06dPNN998Y3bs2GGM8d/xXblypQkODjZPPfWU2bJli5k8eTKXulvlueeeM61atTKhoaGmT58+5osvvrC6pIAnqcbHnDlz3GMOHz5sbr31VtO4cWMTERFhrr76avPLL794vM/27dvNZZddZsLDw018fLy5++67TUVFhceY5cuXm+7du5vQ0FDTtm1bj300ZL8OPxxv73r//ffNOeecYxwOh+nUqZN56aWXPJ53uVzm4YcfNomJicbhcJiLL77Y5OTkeIw5cOCAGTVqlImMjDTR0dEmMzPTHDp0yGPMhg0bzO9+9zvjcDhMixYtzLRp03z+2QJNUVGRufPOO02rVq1MWFiYadu2rXnwwQc9LpnmeJ+Z5cuX1/jf7DFjxhhj/Ht833zzTXPWWWeZ0NBQ06VLF/Phhx+e1mexGXPC7S8BAADqOXp+AABAg0L4AQAADQrhBwAANCiEHwAA0KAQfgAAQINC+AEAAA0K4QcAADQohB8APjV48GBNmDDB6jI82Gw2vfvuu1aXAcAi3OQQgE/l5+crJCREUVFRSklJ0YQJE/wWhqZMmaJ3331X69ev99iem5urxo0by+Fw+KUOAIEl2OoCANRvcXFxXn/P8vJyhYaG1vr1DXn1bQBMewHwsappr8GDB2vHjh266667ZLPZZLPZ3GNWrFihgQMHKjw8XMnJybrjjjtUUlLifj4lJUWPPvqoRo8erejoaN18882SpPvvv19nnXWWIiIi1LZtWz388MPu1aDnzp2rqVOnasOGDe79zZ07V1L1aa9vv/1WF110kcLDw9WkSRPdfPPNKi4udj9/44036qqrrtJTTz2lZs2aqUmTJrrttts8Vp4GUHcQfgD4xeLFi9WyZUs98sgj+uWXX/TLL79IkrZu3apLL71UI0aM0MaNG7Vw4UKtWLFC48eP93j9U089pW7duumbb77Rww8/LEmKiorS3LlztXnzZj3zzDN6+eWX9fTTT0uS0tPTdffdd6tLly7u/aWnp1erq6SkRGlpaWrcuLG++uorLVq0SJ988km1/S9fvlxbt27V8uXL9corr2ju3LnuMAWgbmHaC4BfxMXFKSgoSFFRUR7TTllZWcrIyHD3AXXo0EHPPvusBg0apBdeeEFhYWGSpIsuukh33323x3s+9NBD7j+npKTonnvu0YIFC3TfffcpPDxckZGRCg4O/s1prvnz5+vIkSN69dVX1ahRI0nS888/r6FDh+rxxx9XYmKiJKlx48Z6/vnnFRQUpE6dOmnIkCHKzs7WuHHjvHJ8APgP4QeApTZs2KCNGzdq3rx57m3GGLlcLm3btk1nn322JKlXr17VXrtw4UI9++yz2rp1q4qLi1VZWano6OjT2v+WLVvUrVs3d/CRpPPPP18ul0s5OTnu8NOlSxcFBQW5xzRr1kzffvvtae0LQGAg/ACwVHFxsW655Rbdcccd1Z5r1aqV+88nhhNJWr16tTIyMjR16lSlpaUpJiZGCxYs0F//+lef1BkSEuLxs81mk8vl8sm+APgW4QeA34SGhsrpdHps69GjhzZv3qz27duf1nutWrVKrVu31oMPPujetmPHjv+6v187++yzNXfuXJWUlLgD1sqVK2W329WxY8fTqglA3UDDMwC/SUlJ0b///W/9/PPP2r9/v6SjV2ytWrVK48eP1/r16/XDDz/ovffeq9Zw/GsdOnTQzp07tWDBAm3dulXPPvus3nnnnWr727Ztm9avX6/9+/errKys2vtkZGQoLCxMY8aM0aZNm7R8+XLdfvvtuuGGG9xTXgDqF8IPAL955JFHtH37drVr104JCQmSpHPPPVf/+te/9P3332vgwIE677zzNGnSJDVv3vw332vYsGG66667NH78eHXv3l2rVq1yXwVWZcSIEbr00kt14YUXKiEhQW+88Ua194mIiNDSpUuVn5+v3r1765prrtHFF1+s559/3nsfHEBA4Q7PAACgQeHMDwAAaFAIPwAAoEEh/AAAgAaF8AMAABoUwg8AAGhQCD8AAKBBIfwAAIAGhfADAAAaFMIPAABoUAg/AACgQSH8AACABoXwAwAAGpT/Bz2YavRsZWH6AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Code cell\n",
    "f2 = plt.figure()\n",
    "plt.plot(iters, val_recall_at_k, label='val_recall')\n",
    "plt.xlabel('iteration')\n",
    "plt.ylabel('recall')\n",
    "plt.title('Validation Recall curves')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GzS_43m3FkAZ"
   },
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-08-09T11:07:31.214718Z",
     "iopub.status.busy": "2024-08-09T11:07:31.214090Z",
     "iopub.status.idle": "2024-08-09T11:07:31.869000Z",
     "shell.execute_reply": "2024-08-09T11:07:31.868041Z",
     "shell.execute_reply.started": "2024-08-09T11:07:31.214687Z"
    },
    "id": "YK1VDllNFdNf",
    "outputId": "f911ec34-5978-4826-c12b-d0f7e05202bf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recall: 0.5161722698902388, Precision: 0.6290730778910735\n"
     ]
    }
   ],
   "source": [
    "# Code cell\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    pred_ratings = model.forward(test_adj_edge_index, test_adj_edge_values)\n",
    "    test_recall, test_precision = get_recall_at_k(r_mat_test_index, r_mat_test_values, pred_ratings, 20)\n",
    "    print(f\"Recall: {test_recall}, Precision: {test_precision}\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kaggle": {
   "accelerator": "gpu",
   "dataSources": [],
   "dockerImageVersionId": 30747,
   "isGpuEnabled": true,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "051ac13b99fc435d8d82de838cc5bf72": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1c3a7fd4cd394f55b245053b25c5a7ac": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "27b38b6ab74442f5b5d60b5a20d97b50": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "35674005d4e540a8aadd5c6788d18fe5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "40ba21bc2f8943bfb7bb47898d805765": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6b009c3db6b64c058fe17ebb4d25fc8a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "944dfe5672b243a493631a16e554494e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1c3a7fd4cd394f55b245053b25c5a7ac",
      "placeholder": "​",
      "style": "IPY_MODEL_40ba21bc2f8943bfb7bb47898d805765",
      "value": "100%"
     }
    },
    "96f7c37570b44beebbacbbe1bbb3bd4f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_944dfe5672b243a493631a16e554494e",
       "IPY_MODEL_fb2f7178087d4d2eb6bf9a57ab1e501e",
       "IPY_MODEL_a2d94911c6584013ae46b02ef53cb3c6"
      ],
      "layout": "IPY_MODEL_b4daf3e1d2354d44a70bbaf98660d905"
     }
    },
    "a2d94911c6584013ae46b02ef53cb3c6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_35674005d4e540a8aadd5c6788d18fe5",
      "placeholder": "​",
      "style": "IPY_MODEL_27b38b6ab74442f5b5d60b5a20d97b50",
      "value": " 10000/10000 [05:55&lt;00:00, 64.43it/s]"
     }
    },
    "b4daf3e1d2354d44a70bbaf98660d905": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fb2f7178087d4d2eb6bf9a57ab1e501e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6b009c3db6b64c058fe17ebb4d25fc8a",
      "max": 10000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_051ac13b99fc435d8d82de838cc5bf72",
      "value": 10000
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}