lectures/17_recommender-systems.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 17: Recommender Systems\n",
    "\n",
    "UBC 2022-23\n",
    "\n",
    "Instructor: Varada Kolhatkar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import sys\n",
    "import time\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "sys.path.append(\"code/.\")\n",
    "import matplotlib.pyplot as plt\n",
    "from plotting_functions import *\n",
    "from plotting_functions_unsup import *\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.model_selection import cross_validate, train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "plt.rcParams[\"font.size\"] = 16\n",
    "import matplotlib.cm as cm\n",
    "\n",
    "# plt.style.use(\"seaborn\")\n",
    "%matplotlib inline\n",
    "pd.set_option(\"display.max_colwidth\", 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Learning outcomes <a name=\"lo\"></a>\n",
    "\n",
    "From this lecture, students are expected to be able to:\n",
    "\n",
    "- State the problem of recommender systems. \n",
    "- Describe components of a utility matrix. \n",
    "- Create a utility matrix given ratings data. \n",
    "- Describe a common approach to evaluate recommender systems. \n",
    "- Implement some baseline approaches to complete the utility matrix. \n",
    "- Explain the idea of collaborative filtering. \n",
    "- Explain some serious consequences of recommendation systems. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Announcements\n",
    "\n",
    "- Homework 6 is due Tuesday, November 8th at 11:59pm. \n",
    "- We have scheduled an extra OH at 4pm today on Zoom. \n",
    "- Midterm grades have been released. \n",
    "- No class on Thursday and no tutorials on Friday. \n",
    "- Nothing is due next week. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Recommender systems motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is a recommender system? \n",
    "\n",
    "- A recommender or a recommendation system **recommends** a particular product or service to users they are likely to consume. \n",
    "\n",
    "![](img/recommendation_system.png)\n",
    "\n",
    "<!-- <img src=\"img/recommendation_system.png\" alt=\"\" height=\"900\" width=\"900\">  -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: Recommender Systems\n",
    "- A client goes to Amazon to buy products. \n",
    "- Amazon has some information about the client. They also have information about other clients buying similar products. \n",
    "- What should they recommend to the client, so that they buy more products? \n",
    "- There's no \"right\" answer (no label). \n",
    "- The whole idea is to understand user behavior in order to recommend them products they are likely to consume. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Why should we care about recommendation systems? \n",
    "\n",
    "- Almost everything we buy or consume today is in some way or the other influenced by recommendation systems. \n",
    "    - Music (Spotify), videos (YouTube), news, books and products (Amazon), movies (Netflix), jokes, restaurants, dating , friends (Facebook), professional connections (Linkedin)\n",
    "- Recommendation systems are at the core of the success of many companies. \n",
    "    - Amazon\n",
    "    - [Netflix](https://help.netflix.com/en/node/100639)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### An example Capstone project: [QxMD](https://qxmd.com/)\n",
    "\n",
    "- Present personalized journal article recommendations to health care professionals.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What kind of data we need to build recommendation systems? \n",
    "\n",
    "- **User ratings data** (most common)\n",
    "- **Features related to items or users** \n",
    "- Customer purchase history data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Main approaches\n",
    "\n",
    "- Collaborative filtering \n",
    "    - \"Unsupervised\" learning \n",
    "    - We only have labels $y_{ij}$ (rating of user $i$ for item $j$). \n",
    "    - We learn features.  \n",
    "- Content-based recommenders \n",
    "    - Supervised learning\n",
    "    - Extract features $x_i$ of users and/or items and building a model to predict rating $y_i$ given $x_i$. \n",
    "    - Apply model to predict for new users/items. \n",
    "- Hybrid \n",
    "    - Combining collaborative filtering with content-based filtering\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Netflix prize\n",
    "\n",
    "![](img/netflix.png)\n",
    "\n",
    "[Source](https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Netflix prize\n",
    "\n",
    "- 100M ratings from 0.5M users on 18k movies.\n",
    "- Grand prize was \\$1M for first team to reduce squared error at least by 10%.\n",
    "- Winning entry (and most entries) used collaborative filtering:\n",
    "    - Methods that only looks at ratings, not features of movies/users.\n",
    "- A simple collaborative filtering method that does really well:\n",
    "   - Now adopted by many companies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Recommender systems problem "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Problem formulation\n",
    "\n",
    "- Most often the data for recommender systems come in as **ratings** for a set of items from a set of users. \n",
    "- We have two entities: $N$ **users** and $M$ **items**. \n",
    "- **Users** are consumers. \n",
    "- **Items** are the products or services offered.  \n",
    "    - E.g., movies (Netflix), books (Amazon), songs (spotify), people (tinder)  \n",
    "    \n",
    "![](img/utility_matrix.png)\n",
    "\n",
    "<!-- <img src=\"img/utility_matrix.png\" alt=\"\" height=\"900\" width=\"900\">  -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Utility matrix \n",
    "\n",
    "- A **utility matrix** is the matrix that captures **interactions** between $N$ **users** and $M$ **items**. \n",
    "- The interaction may come in different forms: \n",
    "    - ratings, clicks, purchases\n",
    "\n",
    "![](img/utility_matrix.png)\n",
    "\n",
    "<!-- <img src=\"img/utility_mat.png\" alt=\"\" height=\"900\" width=\"900\">  -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Utility matrix\n",
    "\n",
    "- Below is a toy utility matrix. Here $N$ = 6 and $M$ = 5.  \n",
    "- Each entry $y_{ij}$ ($i^{th}$ row and $j^{th}$ column) denotes the rating given by the user $i$ to item $j$. \n",
    "- We represent users in terms of items and items in terms of users. \n",
    "\n",
    "![](img/utility_matrix.png)\n",
    "\n",
    "<!-- <img src=\"img/utility_matrix.png\" alt=\"\" height=\"900\" width=\"900\">  -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sparsity of utility matrix\n",
    "\n",
    "- The utility matrix is very sparse because usually users only interact with a few items. \n",
    "- For example: \n",
    "    - all Netflix users will have rated only a small percentage of content available on Netflix\n",
    "    - all amazon clients will have rated only a small fraction of items among all items available on Amazon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What do we predict? \n",
    "Given a utility matrix of $N$ users and $M$ items, **complete the utility matrix**. In other words, **predict missing values in the matrix**. \n",
    "\n",
    "![](img/utility_matrix.png)\n",
    "\n",
    "<!-- <img src=\"img/utility_matrix.png\" alt=\"\" height=\"900\" width=\"900\">  -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Once we have predicted ratings, we can recommend items to users they are likely to rate higher. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example dataset: [Jester 1.7M jokes ratings dataset](https://www.kaggle.com/vikashrajluhaniwal/jester-17m-jokes-ratings-dataset?select=jester_ratings.csv)\n",
    "\n",
    "- We'll use a sample of [Jester 1.7M jokes ratings dataset](https://www.kaggle.com/vikashrajluhaniwal/jester-17m-jokes-ratings-dataset) to demonstrate different recommendation systems. \n",
    "\n",
    "The dataset comes with two CSVs\n",
    "- A CSV containing ratings (-10.0 to +10.0) of 150 jokes from 59,132 users. \n",
    "- A CSV containing joke IDs and the actual text of jokes. \n",
    "\n",
    "> Some jokes might be offensive. Please do not look too much into the actual text data if you are sensitive to such language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Recommendation systems are most effective when you have a large amount of data.\n",
    "- But we are only taking a sample here for speed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "filename = \"data/jester_ratings.csv\"\n",
    "ratings_full = pd.read_csv(filename)\n",
    "ratings = ratings_full[ratings_full[\"userId\"] <= 4000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>jokeId</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0.219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>-9.281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>-9.281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>-6.781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>0.875</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  jokeId  rating\n",
       "0  1       5       0.219 \n",
       "1  1       7      -9.281 \n",
       "2  1       8      -9.281 \n",
       "3  1       13     -6.781 \n",
       "4  1       15      0.875 "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_key = \"userId\"\n",
    "item_key = \"jokeId\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Dataset stats "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 141362 entries, 0 to 141361\n",
      "Data columns (total 3 columns):\n",
      " #   Column  Non-Null Count   Dtype  \n",
      "---  ------  --------------   -----  \n",
      " 0   userId  141362 non-null  int64  \n",
      " 1   jokeId  141362 non-null  int64  \n",
      " 2   rating  141362 non-null  float64\n",
      "dtypes: float64(1), int64(2)\n",
      "memory usage: 4.3 MB\n"
     ]
    }
   ],
   "source": [
    "ratings.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of ratings: 141362\n",
      "Average rating:  1.200\n",
      "Number of users (N): 3635\n",
      "Number of items (M): 140\n",
      "Fraction non-nan ratings: 0.278\n"
     ]
    }
   ],
   "source": [
    "def get_stats(ratings, item_key=\"jokeId\", user_key=\"userId\"):\n",
    "    print(\"Number of ratings:\", len(ratings))\n",
    "    print(\"Average rating:  %0.3f\" % (np.mean(ratings[\"rating\"])))\n",
    "    N = len(np.unique(ratings[user_key]))\n",
    "    M = len(np.unique(ratings[item_key]))\n",
    "    print(\"Number of users (N): %d\" % N)\n",
    "    print(\"Number of items (M): %d\" % M)\n",
    "    print(\"Fraction non-nan ratings: %0.3f\" % (len(ratings) / (N * M)))\n",
    "    return N, M\n",
    "\n",
    "\n",
    "N, M = get_stats(ratings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Creating utility matrix\n",
    "\n",
    "- Let's construct utility matrix with `number of users` rows and `number of items` columns from the ratings data. \n",
    "\n",
    "> Note we are constructing a non-sparse matrix for demonstration purpose here. In real life it's recommended that you work with sparse matrices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_mapper = dict(zip(np.unique(ratings[user_key]), list(range(N))))\n",
    "item_mapper = dict(zip(np.unique(ratings[item_key]), list(range(M))))\n",
    "user_inverse_mapper = dict(zip(list(range(N)), np.unique(ratings[user_key])))\n",
    "item_inverse_mapper = dict(zip(list(range(M)), np.unique(ratings[item_key])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def create_Y_from_ratings(\n",
    "    data, N, M, user_mapper, item_mapper, user_key=\"userId\", item_key=\"jokeId\"\n",
    "):  # Function to create a dense utility matrix\n",
    "    Y = np.zeros((N, M))\n",
    "    Y.fill(np.nan)\n",
    "    for index, val in data.iterrows():\n",
    "        n = user_mapper[val[user_key]]\n",
    "        m = item_mapper[val[item_key]]\n",
    "        Y[n, m] = val[\"rating\"]\n",
    "\n",
    "    return Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Utility matrix for the example problem\n",
    "- Rows represent users.\n",
    "- Columns represent items (jokes in our case).\n",
    "- Each cell gives the rating given by the user to the corresponding joke. \n",
    "- Users are features for jokes and jokes are features for users.\n",
    "- We want to predict the missing entries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3635, 140)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_mat = create_Y_from_ratings(ratings, N, M, user_mapper, item_mapper)\n",
    "Y_mat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>130</th>\n",
       "      <th>131</th>\n",
       "      <th>132</th>\n",
       "      <th>133</th>\n",
       "      <th>134</th>\n",
       "      <th>135</th>\n",
       "      <th>136</th>\n",
       "      <th>137</th>\n",
       "      <th>138</th>\n",
       "      <th>139</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.219</td>\n",
       "      <td>-9.281</td>\n",
       "      <td>-9.281</td>\n",
       "      <td>-6.781</td>\n",
       "      <td>0.875</td>\n",
       "      <td>-9.656</td>\n",
       "      <td>-9.031</td>\n",
       "      <td>-7.469</td>\n",
       "      <td>-8.719</td>\n",
       "      <td>-9.156</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-9.688</td>\n",
       "      <td>9.938</td>\n",
       "      <td>9.531</td>\n",
       "      <td>9.938</td>\n",
       "      <td>0.406</td>\n",
       "      <td>3.719</td>\n",
       "      <td>9.656</td>\n",
       "      <td>-2.688</td>\n",
       "      <td>-9.562</td>\n",
       "      <td>-9.125</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-9.844</td>\n",
       "      <td>-9.844</td>\n",
       "      <td>-7.219</td>\n",
       "      <td>-2.031</td>\n",
       "      <td>-9.938</td>\n",
       "      <td>-9.969</td>\n",
       "      <td>-9.875</td>\n",
       "      <td>-9.812</td>\n",
       "      <td>-9.781</td>\n",
       "      <td>-6.844</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-5.812</td>\n",
       "      <td>-4.500</td>\n",
       "      <td>-4.906</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.906</td>\n",
       "      <td>4.750</td>\n",
       "      <td>-5.906</td>\n",
       "      <td>-0.406</td>\n",
       "      <td>-4.031</td>\n",
       "      <td>3.875</td>\n",
       "      <td>6.219</td>\n",
       "      <td>5.656</td>\n",
       "      <td>6.094</td>\n",
       "      <td>5.406</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3630</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-9.812</td>\n",
       "      <td>-0.062</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3631</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-9.844</td>\n",
       "      <td>7.531</td>\n",
       "      <td>-9.719</td>\n",
       "      <td>-9.344</td>\n",
       "      <td>3.875</td>\n",
       "      <td>9.812</td>\n",
       "      <td>8.938</td>\n",
       "      <td>8.375</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3632</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.906</td>\n",
       "      <td>3.969</td>\n",
       "      <td>-2.312</td>\n",
       "      <td>-0.344</td>\n",
       "      <td>-8.844</td>\n",
       "      <td>4.188</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3633</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-8.875</td>\n",
       "      <td>-9.156</td>\n",
       "      <td>-9.156</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3634</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-6.312</td>\n",
       "      <td>1.281</td>\n",
       "      <td>-3.531</td>\n",
       "      <td>2.125</td>\n",
       "      <td>-5.812</td>\n",
       "      <td>5.562</td>\n",
       "      <td>-6.062</td>\n",
       "      <td>0.125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.188</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3635 rows × 140 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          0      1      2      3      4      5      6      7      8      9  \\\n",
       "0     0.219 -9.281 -9.281 -6.781  0.875 -9.656 -9.031 -7.469 -8.719 -9.156   \n",
       "1    -9.688  9.938  9.531  9.938  0.406  3.719  9.656 -2.688 -9.562 -9.125   \n",
       "2    -9.844 -9.844 -7.219 -2.031 -9.938 -9.969 -9.875 -9.812 -9.781 -6.844   \n",
       "3    -5.812 -4.500 -4.906 NaN    NaN    NaN    NaN    NaN    NaN    NaN      \n",
       "4     6.906  4.750 -5.906 -0.406 -4.031  3.875  6.219  5.656  6.094  5.406   \n",
       "...     ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "3630 NaN    -9.812 -0.062 NaN    NaN    NaN    NaN    NaN    NaN    NaN      \n",
       "3631 NaN    -9.844  7.531 -9.719 -9.344  3.875  9.812  8.938  8.375 NaN      \n",
       "3632 NaN    -1.906  3.969 -2.312 -0.344 -8.844  4.188 NaN    NaN    NaN      \n",
       "3633 NaN    -8.875 -9.156 -9.156 NaN    NaN    NaN    NaN    NaN    NaN      \n",
       "3634 NaN    -6.312  1.281 -3.531  2.125 -5.812  5.562 -6.062  0.125 NaN      \n",
       "\n",
       "      ...  130  131    132  133  134  135  136  137  138  139  \n",
       "0     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "1     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "2     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "4     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "...   ...  ..   ..   ..     ..   ..   ..   ..   ..   ..   ..   \n",
       "3630  ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3631  ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3632  ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3633  ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3634  ... NaN  NaN   4.188 NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "\n",
       "[3635 rows x 140 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(Y_mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Baseline Approaches\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Recall that our goal is to predict missing entries in the utility matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>130</th>\n",
       "      <th>131</th>\n",
       "      <th>132</th>\n",
       "      <th>133</th>\n",
       "      <th>134</th>\n",
       "      <th>135</th>\n",
       "      <th>136</th>\n",
       "      <th>137</th>\n",
       "      <th>138</th>\n",
       "      <th>139</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.219</td>\n",
       "      <td>-9.281</td>\n",
       "      <td>-9.281</td>\n",
       "      <td>-6.781</td>\n",
       "      <td>0.875</td>\n",
       "      <td>-9.656</td>\n",
       "      <td>-9.031</td>\n",
       "      <td>-7.469</td>\n",
       "      <td>-8.719</td>\n",
       "      <td>-9.156</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-9.688</td>\n",
       "      <td>9.938</td>\n",
       "      <td>9.531</td>\n",
       "      <td>9.938</td>\n",
       "      <td>0.406</td>\n",
       "      <td>3.719</td>\n",
       "      <td>9.656</td>\n",
       "      <td>-2.688</td>\n",
       "      <td>-9.562</td>\n",
       "      <td>-9.125</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-9.844</td>\n",
       "      <td>-9.844</td>\n",
       "      <td>-7.219</td>\n",
       "      <td>-2.031</td>\n",
       "      <td>-9.938</td>\n",
       "      <td>-9.969</td>\n",
       "      <td>-9.875</td>\n",
       "      <td>-9.812</td>\n",
       "      <td>-9.781</td>\n",
       "      <td>-6.844</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-5.812</td>\n",
       "      <td>-4.500</td>\n",
       "      <td>-4.906</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.906</td>\n",
       "      <td>4.750</td>\n",
       "      <td>-5.906</td>\n",
       "      <td>-0.406</td>\n",
       "      <td>-4.031</td>\n",
       "      <td>3.875</td>\n",
       "      <td>6.219</td>\n",
       "      <td>5.656</td>\n",
       "      <td>6.094</td>\n",
       "      <td>5.406</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3630</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-9.812</td>\n",
       "      <td>-0.062</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3631</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-9.844</td>\n",
       "      <td>7.531</td>\n",
       "      <td>-9.719</td>\n",
       "      <td>-9.344</td>\n",
       "      <td>3.875</td>\n",
       "      <td>9.812</td>\n",
       "      <td>8.938</td>\n",
       "      <td>8.375</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3632</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.906</td>\n",
       "      <td>3.969</td>\n",
       "      <td>-2.312</td>\n",
       "      <td>-0.344</td>\n",
       "      <td>-8.844</td>\n",
       "      <td>4.188</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3633</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-8.875</td>\n",
       "      <td>-9.156</td>\n",
       "      <td>-9.156</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3634</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-6.312</td>\n",
       "      <td>1.281</td>\n",
       "      <td>-3.531</td>\n",
       "      <td>2.125</td>\n",
       "      <td>-5.812</td>\n",
       "      <td>5.562</td>\n",
       "      <td>-6.062</td>\n",
       "      <td>0.125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.188</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3635 rows × 140 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          0      1      2      3      4      5      6      7      8      9  \\\n",
       "0     0.219 -9.281 -9.281 -6.781  0.875 -9.656 -9.031 -7.469 -8.719 -9.156   \n",
       "1    -9.688  9.938  9.531  9.938  0.406  3.719  9.656 -2.688 -9.562 -9.125   \n",
       "2    -9.844 -9.844 -7.219 -2.031 -9.938 -9.969 -9.875 -9.812 -9.781 -6.844   \n",
       "3    -5.812 -4.500 -4.906 NaN    NaN    NaN    NaN    NaN    NaN    NaN      \n",
       "4     6.906  4.750 -5.906 -0.406 -4.031  3.875  6.219  5.656  6.094  5.406   \n",
       "...     ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "3630 NaN    -9.812 -0.062 NaN    NaN    NaN    NaN    NaN    NaN    NaN      \n",
       "3631 NaN    -9.844  7.531 -9.719 -9.344  3.875  9.812  8.938  8.375 NaN      \n",
       "3632 NaN    -1.906  3.969 -2.312 -0.344 -8.844  4.188 NaN    NaN    NaN      \n",
       "3633 NaN    -8.875 -9.156 -9.156 NaN    NaN    NaN    NaN    NaN    NaN      \n",
       "3634 NaN    -6.312  1.281 -3.531  2.125 -5.812  5.562 -6.062  0.125 NaN      \n",
       "\n",
       "      ...  130  131    132  133  134  135  136  137  138  139  \n",
       "0     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "1     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "2     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "4     ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "...   ...  ..   ..   ..     ..   ..   ..   ..   ..   ..   ..   \n",
       "3630  ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3631  ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3632  ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3633  ... NaN  NaN  NaN    NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3634  ... NaN  NaN   4.188 NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "\n",
       "[3635 rows x 140 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(Y_mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Evaluation\n",
    "\n",
    "- We'll try a number of methods to do this. \n",
    "- Although there is no notion of \"accurate\" recommendations, we need a way to evaluate our predictions so that we'll be able to compare different methods.\n",
    "- Although we are doing unsupervised learning, we'll split the data and evaluate our predictions as follows.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Data splitting \n",
    "\n",
    "- We split the ratings into train and validation sets. \n",
    "- It's easier to split the ratings data instead of splitting the utility matrix.\n",
    "- Don't worry about `y`; we're not really going to use it.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((113089, 3), (28273, 3))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = ratings.copy()\n",
    "y = ratings[user_key]\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=42\n",
    ")\n",
    "\n",
    "X_train.shape, X_valid.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we will create utility matrices for train and validation splits. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "train_mat = create_Y_from_ratings(X_train, N, M, user_mapper, item_mapper)\n",
    "valid_mat = create_Y_from_ratings(X_valid, N, M, user_mapper, item_mapper)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3635, 140), (3635, 140))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_mat.shape, valid_mat.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `train_mat` has only ratings from the train set and `valid_mat` has only ratings from the valid set.\n",
    "- During training we assume that we do not have access to some of the available ratings. We predict these ratings and evaluate them against ratings in the validation set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Questions for you\n",
    "\n",
    "- How do train and validation utility matrices differ? \n",
    "- Why are utility matrices for train and validation sets are of the same shape?\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "**Answer:**\n",
    "\n",
    "- The training matrix `train_mat` is of shape N by M but only has ratings from `X_train` and all other ratings missing. \n",
    "- The validation matrix `valid_mat` is also of shape N by M but it only has ratings `X_valid` and all other ratings missing. \n",
    "- They have the same shape because both have the same number of users and items; that's how we have constructed them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Evaluation\n",
    "\n",
    "- Now that we have train and validation sets, how do we evaluate our predictions?\n",
    "- You can calculate the error between actual ratings and predicted ratings with metrics of your choice. \n",
    "    - Most common ones are MSE or RMSE. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- The `error` function below calculates RMSE and `evaluate` function prints train and validation RMSE.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def error(X1, X2):\n",
    "    \"\"\"\n",
    "    Returns the root mean squared error.\n",
    "    \"\"\"\n",
    "    return np.sqrt(np.nanmean((X1 - X2) ** 2))\n",
    "\n",
    "\n",
    "def evaluate(pred_X, train_X, valid_X, model_name=\"Global average\"):\n",
    "    print(\"%s train RMSE: %0.2f\" % (model_name, error(pred_X, train_X)))\n",
    "    print(\"%s valid RMSE: %0.2f\" % (model_name, error(pred_X, valid_X)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Baselines\n",
    "\n",
    "Let's first try some simple approaches to predict missing entries. \n",
    "\n",
    "1. Global average baseline\n",
    "2. [$k$-Nearest Neighbours imputation](https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html)    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Global average baseline\n",
    "\n",
    "- Let's examine RMSE of the global average baseline. \n",
    "- In this baseline we predict everything as the global average rating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>130</th>\n",
       "      <th>131</th>\n",
       "      <th>132</th>\n",
       "      <th>133</th>\n",
       "      <th>134</th>\n",
       "      <th>135</th>\n",
       "      <th>136</th>\n",
       "      <th>137</th>\n",
       "      <th>138</th>\n",
       "      <th>139</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>...</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>...</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>...</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>...</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>...</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "      <td>1.20741</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 140 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         0        1        2        3        4        5        6        7  \\\n",
       "0  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "1  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "2  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "3  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "4  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "\n",
       "         8        9  ...      130      131      132      133      134  \\\n",
       "0  1.20741  1.20741  ...  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "1  1.20741  1.20741  ...  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "2  1.20741  1.20741  ...  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "3  1.20741  1.20741  ...  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "4  1.20741  1.20741  ...  1.20741  1.20741  1.20741  1.20741  1.20741   \n",
       "\n",
       "       135      136      137      138      139  \n",
       "0  1.20741  1.20741  1.20741  1.20741  1.20741  \n",
       "1  1.20741  1.20741  1.20741  1.20741  1.20741  \n",
       "2  1.20741  1.20741  1.20741  1.20741  1.20741  \n",
       "3  1.20741  1.20741  1.20741  1.20741  1.20741  \n",
       "4  1.20741  1.20741  1.20741  1.20741  1.20741  \n",
       "\n",
       "[5 rows x 140 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avg = np.nanmean(train_mat)\n",
    "pred_g = np.zeros(train_mat.shape) + avg\n",
    "pd.DataFrame(pred_g).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global average train RMSE: 5.75\n",
      "Global average valid RMSE: 5.77\n"
     ]
    }
   ],
   "source": [
    "evaluate(pred_g, train_mat, valid_mat, model_name=\"Global average\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [$k$-nearest neighbours imputation](https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html)\n",
    "\n",
    "- Can we try $k$-nearest neighbours type imputation? \n",
    "- Impute missing values using the mean value from $k$ nearest neighbours found in the training set. \n",
    "- Calculate distances between examples using features where neither value is missing. \n",
    "\n",
    "![](img/utility_matrix.png)\n",
    "\n",
    "<!-- <img src=\"img/utility_matrix.png\" alt=\"\" height=\"900\" width=\"900\">  -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.impute import KNNImputer\n",
    "\n",
    "imputer = KNNImputer(n_neighbors=10)\n",
    "train_mat_imp = imputer.fit_transform(train_mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>130</th>\n",
       "      <th>131</th>\n",
       "      <th>132</th>\n",
       "      <th>133</th>\n",
       "      <th>134</th>\n",
       "      <th>135</th>\n",
       "      <th>136</th>\n",
       "      <th>137</th>\n",
       "      <th>138</th>\n",
       "      <th>139</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-5.9406</td>\n",
       "      <td>-9.2810</td>\n",
       "      <td>-9.2810</td>\n",
       "      <td>-6.7810</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>-9.6560</td>\n",
       "      <td>-9.0310</td>\n",
       "      <td>-7.4690</td>\n",
       "      <td>-8.7190</td>\n",
       "      <td>-9.1560</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.5311</td>\n",
       "      <td>1.8968</td>\n",
       "      <td>0.6905</td>\n",
       "      <td>-3.1218</td>\n",
       "      <td>1.2843</td>\n",
       "      <td>-2.6063</td>\n",
       "      <td>-0.1812</td>\n",
       "      <td>-1.3937</td>\n",
       "      <td>1.7625</td>\n",
       "      <td>-0.4092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.3405</td>\n",
       "      <td>9.9380</td>\n",
       "      <td>9.5310</td>\n",
       "      <td>9.9380</td>\n",
       "      <td>0.4060</td>\n",
       "      <td>3.7190</td>\n",
       "      <td>9.6560</td>\n",
       "      <td>-2.6880</td>\n",
       "      <td>4.3438</td>\n",
       "      <td>-9.1250</td>\n",
       "      <td>...</td>\n",
       "      <td>2.2437</td>\n",
       "      <td>3.1719</td>\n",
       "      <td>5.0251</td>\n",
       "      <td>5.1812</td>\n",
       "      <td>8.2407</td>\n",
       "      <td>5.9311</td>\n",
       "      <td>5.8375</td>\n",
       "      <td>6.3812</td>\n",
       "      <td>1.1687</td>\n",
       "      <td>6.2532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-9.8440</td>\n",
       "      <td>-3.5750</td>\n",
       "      <td>-7.2190</td>\n",
       "      <td>-2.0310</td>\n",
       "      <td>-9.9380</td>\n",
       "      <td>-9.9690</td>\n",
       "      <td>-9.8750</td>\n",
       "      <td>-9.8120</td>\n",
       "      <td>-9.7810</td>\n",
       "      <td>-6.8440</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.4186</td>\n",
       "      <td>-3.1156</td>\n",
       "      <td>-1.5655</td>\n",
       "      <td>-5.6250</td>\n",
       "      <td>0.3720</td>\n",
       "      <td>-4.0439</td>\n",
       "      <td>-6.0500</td>\n",
       "      <td>-5.5563</td>\n",
       "      <td>-5.4125</td>\n",
       "      <td>-5.5874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-5.8120</td>\n",
       "      <td>-2.4624</td>\n",
       "      <td>-4.9060</td>\n",
       "      <td>-2.7781</td>\n",
       "      <td>-0.0532</td>\n",
       "      <td>-3.8594</td>\n",
       "      <td>1.7031</td>\n",
       "      <td>-0.3687</td>\n",
       "      <td>1.8469</td>\n",
       "      <td>0.0593</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.2968</td>\n",
       "      <td>2.1469</td>\n",
       "      <td>2.9532</td>\n",
       "      <td>1.6845</td>\n",
       "      <td>-0.0844</td>\n",
       "      <td>-0.0156</td>\n",
       "      <td>1.2595</td>\n",
       "      <td>3.8219</td>\n",
       "      <td>3.1971</td>\n",
       "      <td>5.0249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.3157</td>\n",
       "      <td>4.7500</td>\n",
       "      <td>1.8658</td>\n",
       "      <td>-0.4060</td>\n",
       "      <td>1.7937</td>\n",
       "      <td>3.8750</td>\n",
       "      <td>6.2190</td>\n",
       "      <td>1.9220</td>\n",
       "      <td>6.0940</td>\n",
       "      <td>5.4060</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.2844</td>\n",
       "      <td>1.1313</td>\n",
       "      <td>4.0157</td>\n",
       "      <td>3.0344</td>\n",
       "      <td>4.0406</td>\n",
       "      <td>0.5218</td>\n",
       "      <td>4.3594</td>\n",
       "      <td>4.0968</td>\n",
       "      <td>3.9250</td>\n",
       "      <td>3.9657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3630</th>\n",
       "      <td>-0.7750</td>\n",
       "      <td>-9.8120</td>\n",
       "      <td>-0.0620</td>\n",
       "      <td>-2.8218</td>\n",
       "      <td>-3.5282</td>\n",
       "      <td>-4.8281</td>\n",
       "      <td>2.2718</td>\n",
       "      <td>-2.8782</td>\n",
       "      <td>-1.0125</td>\n",
       "      <td>0.0688</td>\n",
       "      <td>...</td>\n",
       "      <td>-6.6844</td>\n",
       "      <td>3.0531</td>\n",
       "      <td>2.8687</td>\n",
       "      <td>1.5281</td>\n",
       "      <td>4.5002</td>\n",
       "      <td>-0.1878</td>\n",
       "      <td>2.0031</td>\n",
       "      <td>4.0908</td>\n",
       "      <td>2.3563</td>\n",
       "      <td>5.0406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3631</th>\n",
       "      <td>2.5188</td>\n",
       "      <td>-5.0625</td>\n",
       "      <td>-0.4001</td>\n",
       "      <td>-9.7190</td>\n",
       "      <td>-9.3440</td>\n",
       "      <td>-1.6408</td>\n",
       "      <td>-4.1187</td>\n",
       "      <td>8.9380</td>\n",
       "      <td>8.3750</td>\n",
       "      <td>-0.9314</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.0344</td>\n",
       "      <td>7.9155</td>\n",
       "      <td>3.4282</td>\n",
       "      <td>4.2968</td>\n",
       "      <td>6.7968</td>\n",
       "      <td>7.3999</td>\n",
       "      <td>1.8500</td>\n",
       "      <td>5.8219</td>\n",
       "      <td>5.1812</td>\n",
       "      <td>2.8437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3632</th>\n",
       "      <td>0.1749</td>\n",
       "      <td>-1.9060</td>\n",
       "      <td>3.9690</td>\n",
       "      <td>-1.3844</td>\n",
       "      <td>-0.3440</td>\n",
       "      <td>-8.8440</td>\n",
       "      <td>4.1880</td>\n",
       "      <td>-1.5564</td>\n",
       "      <td>5.0593</td>\n",
       "      <td>0.3343</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.0126</td>\n",
       "      <td>2.8344</td>\n",
       "      <td>2.4499</td>\n",
       "      <td>2.9312</td>\n",
       "      <td>2.3750</td>\n",
       "      <td>-0.4062</td>\n",
       "      <td>1.4375</td>\n",
       "      <td>3.9750</td>\n",
       "      <td>-1.2220</td>\n",
       "      <td>2.8375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3633</th>\n",
       "      <td>-2.7749</td>\n",
       "      <td>-6.4907</td>\n",
       "      <td>-6.1594</td>\n",
       "      <td>-9.1560</td>\n",
       "      <td>-7.1437</td>\n",
       "      <td>-6.5406</td>\n",
       "      <td>3.8718</td>\n",
       "      <td>-1.7782</td>\n",
       "      <td>-3.7406</td>\n",
       "      <td>-0.6406</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.6938</td>\n",
       "      <td>4.8061</td>\n",
       "      <td>4.9968</td>\n",
       "      <td>-0.1626</td>\n",
       "      <td>2.4187</td>\n",
       "      <td>-0.7750</td>\n",
       "      <td>4.6781</td>\n",
       "      <td>1.7658</td>\n",
       "      <td>0.4595</td>\n",
       "      <td>0.1843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3634</th>\n",
       "      <td>-0.0812</td>\n",
       "      <td>-6.3120</td>\n",
       "      <td>1.2810</td>\n",
       "      <td>-3.5310</td>\n",
       "      <td>2.1250</td>\n",
       "      <td>-5.8120</td>\n",
       "      <td>5.5620</td>\n",
       "      <td>0.2218</td>\n",
       "      <td>0.1250</td>\n",
       "      <td>-1.1874</td>\n",
       "      <td>...</td>\n",
       "      <td>-3.8156</td>\n",
       "      <td>4.1812</td>\n",
       "      <td>4.1880</td>\n",
       "      <td>3.7280</td>\n",
       "      <td>3.0750</td>\n",
       "      <td>2.1033</td>\n",
       "      <td>2.8156</td>\n",
       "      <td>5.5312</td>\n",
       "      <td>3.8283</td>\n",
       "      <td>4.1219</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3635 rows × 140 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           0       1       2       3       4       5       6       7       8  \\\n",
       "0    -5.9406 -9.2810 -9.2810 -6.7810  0.8750 -9.6560 -9.0310 -7.4690 -8.7190   \n",
       "1     2.3405  9.9380  9.5310  9.9380  0.4060  3.7190  9.6560 -2.6880  4.3438   \n",
       "2    -9.8440 -3.5750 -7.2190 -2.0310 -9.9380 -9.9690 -9.8750 -9.8120 -9.7810   \n",
       "3    -5.8120 -2.4624 -4.9060 -2.7781 -0.0532 -3.8594  1.7031 -0.3687  1.8469   \n",
       "4     1.3157  4.7500  1.8658 -0.4060  1.7937  3.8750  6.2190  1.9220  6.0940   \n",
       "...      ...     ...     ...     ...     ...     ...     ...     ...     ...   \n",
       "3630 -0.7750 -9.8120 -0.0620 -2.8218 -3.5282 -4.8281  2.2718 -2.8782 -1.0125   \n",
       "3631  2.5188 -5.0625 -0.4001 -9.7190 -9.3440 -1.6408 -4.1187  8.9380  8.3750   \n",
       "3632  0.1749 -1.9060  3.9690 -1.3844 -0.3440 -8.8440  4.1880 -1.5564  5.0593   \n",
       "3633 -2.7749 -6.4907 -6.1594 -9.1560 -7.1437 -6.5406  3.8718 -1.7782 -3.7406   \n",
       "3634 -0.0812 -6.3120  1.2810 -3.5310  2.1250 -5.8120  5.5620  0.2218  0.1250   \n",
       "\n",
       "           9  ...     130     131     132     133     134     135     136  \\\n",
       "0    -9.1560  ... -4.5311  1.8968  0.6905 -3.1218  1.2843 -2.6063 -0.1812   \n",
       "1    -9.1250  ...  2.2437  3.1719  5.0251  5.1812  8.2407  5.9311  5.8375   \n",
       "2    -6.8440  ... -4.4186 -3.1156 -1.5655 -5.6250  0.3720 -4.0439 -6.0500   \n",
       "3     0.0593  ... -2.2968  2.1469  2.9532  1.6845 -0.0844 -0.0156  1.2595   \n",
       "4     5.4060  ... -0.2844  1.1313  4.0157  3.0344  4.0406  0.5218  4.3594   \n",
       "...      ...  ...     ...     ...     ...     ...     ...     ...     ...   \n",
       "3630  0.0688  ... -6.6844  3.0531  2.8687  1.5281  4.5002 -0.1878  2.0031   \n",
       "3631 -0.9314  ... -4.0344  7.9155  3.4282  4.2968  6.7968  7.3999  1.8500   \n",
       "3632  0.3343  ... -4.0126  2.8344  2.4499  2.9312  2.3750 -0.4062  1.4375   \n",
       "3633 -0.6406  ... -4.6938  4.8061  4.9968 -0.1626  2.4187 -0.7750  4.6781   \n",
       "3634 -1.1874  ... -3.8156  4.1812  4.1880  3.7280  3.0750  2.1033  2.8156   \n",
       "\n",
       "         137     138     139  \n",
       "0    -1.3937  1.7625 -0.4092  \n",
       "1     6.3812  1.1687  6.2532  \n",
       "2    -5.5563 -5.4125 -5.5874  \n",
       "3     3.8219  3.1971  5.0249  \n",
       "4     4.0968  3.9250  3.9657  \n",
       "...      ...     ...     ...  \n",
       "3630  4.0908  2.3563  5.0406  \n",
       "3631  5.8219  5.1812  2.8437  \n",
       "3632  3.9750 -1.2220  2.8375  \n",
       "3633  1.7658  0.4595  0.1843  \n",
       "3634  5.5312  3.8283  4.1219  \n",
       "\n",
       "[3635 rows x 140 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(train_mat_imp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN imputer train RMSE: 0.00\n",
      "KNN imputer valid RMSE: 4.79\n"
     ]
    }
   ],
   "source": [
    "evaluate(train_mat_imp, train_mat, valid_mat, model_name=\"KNN imputer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Finding [nearest neighbors](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html)\n",
    "\n",
    "- We can look at nearest neighbours of a query item. \n",
    "- Here our columns are jokes, and users are features for jokes, and we'll have to find nearest neighbours of columns vectors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>130</th>\n",
       "      <th>131</th>\n",
       "      <th>132</th>\n",
       "      <th>133</th>\n",
       "      <th>134</th>\n",
       "      <th>135</th>\n",
       "      <th>136</th>\n",
       "      <th>137</th>\n",
       "      <th>138</th>\n",
       "      <th>139</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-5.9406</td>\n",
       "      <td>-9.2810</td>\n",
       "      <td>-9.2810</td>\n",
       "      <td>-6.7810</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>-9.6560</td>\n",
       "      <td>-9.0310</td>\n",
       "      <td>-7.4690</td>\n",
       "      <td>-8.7190</td>\n",
       "      <td>-9.1560</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.5311</td>\n",
       "      <td>1.8968</td>\n",
       "      <td>0.6905</td>\n",
       "      <td>-3.1218</td>\n",
       "      <td>1.2843</td>\n",
       "      <td>-2.6063</td>\n",
       "      <td>-0.1812</td>\n",
       "      <td>-1.3937</td>\n",
       "      <td>1.7625</td>\n",
       "      <td>-0.4092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.3405</td>\n",
       "      <td>9.9380</td>\n",
       "      <td>9.5310</td>\n",
       "      <td>9.9380</td>\n",
       "      <td>0.4060</td>\n",
       "      <td>3.7190</td>\n",
       "      <td>9.6560</td>\n",
       "      <td>-2.6880</td>\n",
       "      <td>4.3438</td>\n",
       "      <td>-9.1250</td>\n",
       "      <td>...</td>\n",
       "      <td>2.2437</td>\n",
       "      <td>3.1719</td>\n",
       "      <td>5.0251</td>\n",
       "      <td>5.1812</td>\n",
       "      <td>8.2407</td>\n",
       "      <td>5.9311</td>\n",
       "      <td>5.8375</td>\n",
       "      <td>6.3812</td>\n",
       "      <td>1.1687</td>\n",
       "      <td>6.2532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-9.8440</td>\n",
       "      <td>-3.5750</td>\n",
       "      <td>-7.2190</td>\n",
       "      <td>-2.0310</td>\n",
       "      <td>-9.9380</td>\n",
       "      <td>-9.9690</td>\n",
       "      <td>-9.8750</td>\n",
       "      <td>-9.8120</td>\n",
       "      <td>-9.7810</td>\n",
       "      <td>-6.8440</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.4186</td>\n",
       "      <td>-3.1156</td>\n",
       "      <td>-1.5655</td>\n",
       "      <td>-5.6250</td>\n",
       "      <td>0.3720</td>\n",
       "      <td>-4.0439</td>\n",
       "      <td>-6.0500</td>\n",
       "      <td>-5.5563</td>\n",
       "      <td>-5.4125</td>\n",
       "      <td>-5.5874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-5.8120</td>\n",
       "      <td>-2.4624</td>\n",
       "      <td>-4.9060</td>\n",
       "      <td>-2.7781</td>\n",
       "      <td>-0.0532</td>\n",
       "      <td>-3.8594</td>\n",
       "      <td>1.7031</td>\n",
       "      <td>-0.3687</td>\n",
       "      <td>1.8469</td>\n",
       "      <td>0.0593</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.2968</td>\n",
       "      <td>2.1469</td>\n",
       "      <td>2.9532</td>\n",
       "      <td>1.6845</td>\n",
       "      <td>-0.0844</td>\n",
       "      <td>-0.0156</td>\n",
       "      <td>1.2595</td>\n",
       "      <td>3.8219</td>\n",
       "      <td>3.1971</td>\n",
       "      <td>5.0249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.3157</td>\n",
       "      <td>4.7500</td>\n",
       "      <td>1.8658</td>\n",
       "      <td>-0.4060</td>\n",
       "      <td>1.7937</td>\n",
       "      <td>3.8750</td>\n",
       "      <td>6.2190</td>\n",
       "      <td>1.9220</td>\n",
       "      <td>6.0940</td>\n",
       "      <td>5.4060</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.2844</td>\n",
       "      <td>1.1313</td>\n",
       "      <td>4.0157</td>\n",
       "      <td>3.0344</td>\n",
       "      <td>4.0406</td>\n",
       "      <td>0.5218</td>\n",
       "      <td>4.3594</td>\n",
       "      <td>4.0968</td>\n",
       "      <td>3.9250</td>\n",
       "      <td>3.9657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3630</th>\n",
       "      <td>-0.7750</td>\n",
       "      <td>-9.8120</td>\n",
       "      <td>-0.0620</td>\n",
       "      <td>-2.8218</td>\n",
       "      <td>-3.5282</td>\n",
       "      <td>-4.8281</td>\n",
       "      <td>2.2718</td>\n",
       "      <td>-2.8782</td>\n",
       "      <td>-1.0125</td>\n",
       "      <td>0.0688</td>\n",
       "      <td>...</td>\n",
       "      <td>-6.6844</td>\n",
       "      <td>3.0531</td>\n",
       "      <td>2.8687</td>\n",
       "      <td>1.5281</td>\n",
       "      <td>4.5002</td>\n",
       "      <td>-0.1878</td>\n",
       "      <td>2.0031</td>\n",
       "      <td>4.0908</td>\n",
       "      <td>2.3563</td>\n",
       "      <td>5.0406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3631</th>\n",
       "      <td>2.5188</td>\n",
       "      <td>-5.0625</td>\n",
       "      <td>-0.4001</td>\n",
       "      <td>-9.7190</td>\n",
       "      <td>-9.3440</td>\n",
       "      <td>-1.6408</td>\n",
       "      <td>-4.1187</td>\n",
       "      <td>8.9380</td>\n",
       "      <td>8.3750</td>\n",
       "      <td>-0.9314</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.0344</td>\n",
       "      <td>7.9155</td>\n",
       "      <td>3.4282</td>\n",
       "      <td>4.2968</td>\n",
       "      <td>6.7968</td>\n",
       "      <td>7.3999</td>\n",
       "      <td>1.8500</td>\n",
       "      <td>5.8219</td>\n",
       "      <td>5.1812</td>\n",
       "      <td>2.8437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3632</th>\n",
       "      <td>0.1749</td>\n",
       "      <td>-1.9060</td>\n",
       "      <td>3.9690</td>\n",
       "      <td>-1.3844</td>\n",
       "      <td>-0.3440</td>\n",
       "      <td>-8.8440</td>\n",
       "      <td>4.1880</td>\n",
       "      <td>-1.5564</td>\n",
       "      <td>5.0593</td>\n",
       "      <td>0.3343</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.0126</td>\n",
       "      <td>2.8344</td>\n",
       "      <td>2.4499</td>\n",
       "      <td>2.9312</td>\n",
       "      <td>2.3750</td>\n",
       "      <td>-0.4062</td>\n",
       "      <td>1.4375</td>\n",
       "      <td>3.9750</td>\n",
       "      <td>-1.2220</td>\n",
       "      <td>2.8375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3633</th>\n",
       "      <td>-2.7749</td>\n",
       "      <td>-6.4907</td>\n",
       "      <td>-6.1594</td>\n",
       "      <td>-9.1560</td>\n",
       "      <td>-7.1437</td>\n",
       "      <td>-6.5406</td>\n",
       "      <td>3.8718</td>\n",
       "      <td>-1.7782</td>\n",
       "      <td>-3.7406</td>\n",
       "      <td>-0.6406</td>\n",
       "      <td>...</td>\n",
       "      <td>-4.6938</td>\n",
       "      <td>4.8061</td>\n",
       "      <td>4.9968</td>\n",
       "      <td>-0.1626</td>\n",
       "      <td>2.4187</td>\n",
       "      <td>-0.7750</td>\n",
       "      <td>4.6781</td>\n",
       "      <td>1.7658</td>\n",
       "      <td>0.4595</td>\n",
       "      <td>0.1843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3634</th>\n",
       "      <td>-0.0812</td>\n",
       "      <td>-6.3120</td>\n",
       "      <td>1.2810</td>\n",
       "      <td>-3.5310</td>\n",
       "      <td>2.1250</td>\n",
       "      <td>-5.8120</td>\n",
       "      <td>5.5620</td>\n",
       "      <td>0.2218</td>\n",
       "      <td>0.1250</td>\n",
       "      <td>-1.1874</td>\n",
       "      <td>...</td>\n",
       "      <td>-3.8156</td>\n",
       "      <td>4.1812</td>\n",
       "      <td>4.1880</td>\n",
       "      <td>3.7280</td>\n",
       "      <td>3.0750</td>\n",
       "      <td>2.1033</td>\n",
       "      <td>2.8156</td>\n",
       "      <td>5.5312</td>\n",
       "      <td>3.8283</td>\n",
       "      <td>4.1219</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3635 rows × 140 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           0       1       2       3       4       5       6       7       8  \\\n",
       "0    -5.9406 -9.2810 -9.2810 -6.7810  0.8750 -9.6560 -9.0310 -7.4690 -8.7190   \n",
       "1     2.3405  9.9380  9.5310  9.9380  0.4060  3.7190  9.6560 -2.6880  4.3438   \n",
       "2    -9.8440 -3.5750 -7.2190 -2.0310 -9.9380 -9.9690 -9.8750 -9.8120 -9.7810   \n",
       "3    -5.8120 -2.4624 -4.9060 -2.7781 -0.0532 -3.8594  1.7031 -0.3687  1.8469   \n",
       "4     1.3157  4.7500  1.8658 -0.4060  1.7937  3.8750  6.2190  1.9220  6.0940   \n",
       "...      ...     ...     ...     ...     ...     ...     ...     ...     ...   \n",
       "3630 -0.7750 -9.8120 -0.0620 -2.8218 -3.5282 -4.8281  2.2718 -2.8782 -1.0125   \n",
       "3631  2.5188 -5.0625 -0.4001 -9.7190 -9.3440 -1.6408 -4.1187  8.9380  8.3750   \n",
       "3632  0.1749 -1.9060  3.9690 -1.3844 -0.3440 -8.8440  4.1880 -1.5564  5.0593   \n",
       "3633 -2.7749 -6.4907 -6.1594 -9.1560 -7.1437 -6.5406  3.8718 -1.7782 -3.7406   \n",
       "3634 -0.0812 -6.3120  1.2810 -3.5310  2.1250 -5.8120  5.5620  0.2218  0.1250   \n",
       "\n",
       "           9  ...     130     131     132     133     134     135     136  \\\n",
       "0    -9.1560  ... -4.5311  1.8968  0.6905 -3.1218  1.2843 -2.6063 -0.1812   \n",
       "1    -9.1250  ...  2.2437  3.1719  5.0251  5.1812  8.2407  5.9311  5.8375   \n",
       "2    -6.8440  ... -4.4186 -3.1156 -1.5655 -5.6250  0.3720 -4.0439 -6.0500   \n",
       "3     0.0593  ... -2.2968  2.1469  2.9532  1.6845 -0.0844 -0.0156  1.2595   \n",
       "4     5.4060  ... -0.2844  1.1313  4.0157  3.0344  4.0406  0.5218  4.3594   \n",
       "...      ...  ...     ...     ...     ...     ...     ...     ...     ...   \n",
       "3630  0.0688  ... -6.6844  3.0531  2.8687  1.5281  4.5002 -0.1878  2.0031   \n",
       "3631 -0.9314  ... -4.0344  7.9155  3.4282  4.2968  6.7968  7.3999  1.8500   \n",
       "3632  0.3343  ... -4.0126  2.8344  2.4499  2.9312  2.3750 -0.4062  1.4375   \n",
       "3633 -0.6406  ... -4.6938  4.8061  4.9968 -0.1626  2.4187 -0.7750  4.6781   \n",
       "3634 -1.1874  ... -3.8156  4.1812  4.1880  3.7280  3.0750  2.1033  2.8156   \n",
       "\n",
       "         137     138     139  \n",
       "0    -1.3937  1.7625 -0.4092  \n",
       "1     6.3812  1.1687  6.2532  \n",
       "2    -5.5563 -5.4125 -5.5874  \n",
       "3     3.8219  3.1971  5.0249  \n",
       "4     4.0968  3.9250  3.9657  \n",
       "...      ...     ...     ...  \n",
       "3630  4.0908  2.3563  5.0406  \n",
       "3631  5.8219  5.1812  2.8437  \n",
       "3632  3.9750 -1.2220  2.8375  \n",
       "3633  1.7658  0.4595  0.1843  \n",
       "3634  5.5312  3.8283  4.1219  \n",
       "\n",
       "[3635 rows x 140 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(train_mat_imp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (Optional) $k$-nearest neighbours on a query joke\n",
    "- Let's transpose the matrix.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "item_user_mat = train_mat_imp.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>jokeId</th>\n",
       "      <th>jokeText</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>A man visits the doctor. The doctor says \"I have bad news for you.You have\\ncancer and Alzheimer's disease\". \\nThe man replies \"Well,thank God I don't have cancer!\"\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>This couple had an excellent relationship going until one day he came home\\nfrom work to find his girlfriend packing. He asked her why she was leaving him\\nand she told him that she had heard awful things about him. \\n\\n\"What could they possibly have said to make you move out?\" \\n\\n\"They told me that you were a pedophile.\" \\n\\nHe replied, \"That's an awfully big word for a ten year old.\" \\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Q. What's 200 feet long and has 4 teeth? \\n\\nA. The front row at a Willie Nelson Concert.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Q. What's the difference between a man and a toilet? \\n\\nA. A toilet doesn't follow you around after you use it.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Q.\\tWhat's O. J. Simpson's Internet address? \\nA.\\tSlash, slash, backslash, slash, slash, escape.\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   jokeId  \\\n",
       "0  1        \n",
       "1  2        \n",
       "2  3        \n",
       "3  4        \n",
       "4  5        \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                   jokeText  \n",
       "0  A man visits the doctor. The doctor says \"I have bad news for you.You have\\ncancer and Alzheimer's disease\". \\nThe man replies \"Well,thank God I don't have cancer!\"\\n                                                                                                                                                                                                                                    \n",
       "1  This couple had an excellent relationship going until one day he came home\\nfrom work to find his girlfriend packing. He asked her why she was leaving him\\nand she told him that she had heard awful things about him. \\n\\n\"What could they possibly have said to make you move out?\" \\n\\n\"They told me that you were a pedophile.\" \\n\\nHe replied, \"That's an awfully big word for a ten year old.\" \\n  \n",
       "2  Q. What's 200 feet long and has 4 teeth? \\n\\nA. The front row at a Willie Nelson Concert.\\n                                                                                                                                                                                                                                                                                                               \n",
       "3  Q. What's the difference between a man and a toilet? \\n\\nA. A toilet doesn't follow you around after you use it.\\n                                                                                                                                                                                                                                                                                        \n",
       "4  Q.\\tWhat's O. J. Simpson's Internet address? \\nA.\\tSlash, slash, backslash, slash, slash, escape.\\n                                                                                                                                                                                                                                                                                                       "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jokes_df = pd.read_csv(\"data/jester_items.csv\")\n",
    "jokes_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "id_joke_map = dict(zip(jokes_df.jokeId, jokes_df.jokeText))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query joke:  Q: If a person who speaks three languages is called \"tri-lingual,\" and\n",
      "a person who speaks two languages is called \"bi-lingual,\" what do call\n",
      "a person who only speaks one language?\n",
      "\n",
      "A: American! \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>top recommendations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Q: What is the difference between George  Washington, Richard Nixon,\\nand Bill Clinton?\\n\\nA: Washington couldn't tell a lie, Nixon couldn't   tell the truth, and\\nClinton doesn't know the difference.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A man in a hot air balloon realized he was lost. He reduced altitude and spotted a woman below. He descended a bit more and shouted, \"Excuse me, can you help me? I promised a friend I would meet him an hour ago, but I don't know where I am.\" The woman below replied, \"You are in a hot air balloon hovering approximately 30 feet above the ground. You are between 40 and 41 degrees north latitude and between 59 and 60 degrees west longitude.\" \"You must be an engineer,\" said the balloonist. \"I am,\" replied the woman. \"How did you know?\" \"Well,\" answered the balloonist, \"everything you told me is technically correct, but I have no idea what to make of your information, and the fact is, I am still lost. Frankly, you've not been much help so far.\" The woman below responded, \"You must be in management.\" \"I am,\" replied the balloonist, \"but how did you know?\" \"Well,\" said the woman, \"you don't know where you are or where you are going. You have risen to where you are due to a large quantity of hot air. You made a promise that you have no idea how to keep, and you expect people beneath you to solve your problems. The fact is, you are in exactly the same position you were in before we met, but now, somehow, it's my fault!\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If pro- is the opposite of con- then congress must be the opposite\\nof progress.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Arnold Swartzeneger and Sylvester Stallone are making a movie about\\nthe lives of the great composers.  \\nStallone says \"I want to be Mozart.\" \\nSwartzeneger says: \"In that case...  I'll be Bach.\"\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             top recommendations\n",
       "0  Q: What is the difference between George  Washington, Richard Nixon,\\nand Bill Clinton?\\n\\nA: Washington couldn't tell a lie, Nixon couldn't   tell the truth, and\\nClinton doesn't know the difference.\\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   \n",
       "1  A man in a hot air balloon realized he was lost. He reduced altitude and spotted a woman below. He descended a bit more and shouted, \"Excuse me, can you help me? I promised a friend I would meet him an hour ago, but I don't know where I am.\" The woman below replied, \"You are in a hot air balloon hovering approximately 30 feet above the ground. You are between 40 and 41 degrees north latitude and between 59 and 60 degrees west longitude.\" \"You must be an engineer,\" said the balloonist. \"I am,\" replied the woman. \"How did you know?\" \"Well,\" answered the balloonist, \"everything you told me is technically correct, but I have no idea what to make of your information, and the fact is, I am still lost. Frankly, you've not been much help so far.\" The woman below responded, \"You must be in management.\" \"I am,\" replied the balloonist, \"but how did you know?\" \"Well,\" said the woman, \"you don't know where you are or where you are going. You have risen to where you are due to a large quantity of hot air. You made a promise that you have no idea how to keep, and you expect people beneath you to solve your problems. The fact is, you are in exactly the same position you were in before we met, but now, somehow, it's my fault!\"\n",
       "2  If pro- is the opposite of con- then congress must be the opposite\\nof progress.\\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "3  Arnold Swartzeneger and Sylvester Stallone are making a movie about\\nthe lives of the great composers.  \\nStallone says \"I want to be Mozart.\" \\nSwartzeneger says: \"In that case...  I'll be Bach.\"\\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import NearestNeighbors\n",
    "\n",
    "\n",
    "def get_topk_recommendations(X, query_ind=0, metric=\"cosine\", k=5):\n",
    "    query_idx = item_inverse_mapper[query_ind]\n",
    "    model = NearestNeighbors(n_neighbors=k, metric=\"cosine\")\n",
    "    model.fit(X)\n",
    "    neigh_ind = model.kneighbors([X[query_ind]], k, return_distance=False).flatten()\n",
    "    neigh_ind = np.delete(neigh_ind, np.where(query_ind == query_ind))\n",
    "    recs = [id_joke_map[item_inverse_mapper[i]] for i in neigh_ind]\n",
    "    print(\"Query joke: \", id_joke_map[query_idx])\n",
    "\n",
    "    return pd.DataFrame(data=recs, columns=[\"top recommendations\"])\n",
    "\n",
    "\n",
    "get_topk_recommendations(item_user_mat, query_ind=8, metric=\"cosine\", k=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Question**\n",
    "- Instead of imputation, what would be the consequences if we replace `nan` with zeros so that we can calculate distances between vectors? \n",
    "\n",
    "<br><br>\n",
    "**Answer**\n",
    "\n",
    "It's not a good idea replace ratings with 0, because 0 can be an actual rating value in our case. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What to do with predictions? \n",
    "- Once you have predictions, you can sort them based on ratings and recommend items with highest ratings.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Break (5 min)\n",
    "\n",
    "![](img/eva-coffee.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Collaborative filtering "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Collaborative filtering \n",
    "- One of the most popular approach for recommendation systems. \n",
    "- Approach used by the winning entry (and most of the entries) in the Netflix competition. \n",
    "- An unsupervised approach\n",
    "    - Only uses the user-item interactions given in the ratings matrix. \n",
    "- **Intuition**\n",
    "    - We may have similar users and similar items which can help us predict missing entries. \n",
    "    - Leverage social information to provide recommendations. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Problem \n",
    "\n",
    "- Given a utility matrix with many missing entries, how can we predict missing ratings?  \n",
    "\n",
    "$$\n",
    "\\begin{bmatrix} \n",
    "? & ? & \\checkmark  & ? & \\checkmark\\\\\n",
    "\\checkmark & ? & ?  & ? & ?\\\\\n",
    "? & \\checkmark & \\checkmark  & ? & \\checkmark\\\\\n",
    "? & ? & ?  & ? & ?\\\\\n",
    "? & ? & ? & \\checkmark & ?\\\\\n",
    "? & \\checkmark & \\checkmark  & ? & \\checkmark\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "> Note: rating prediction $\\neq$ Classification or regression "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Classification or regression\n",
    "\n",
    "- We have $X$ and targets for some rows in $X$. \n",
    "- We want to predict the last column (target column).  \n",
    "\n",
    "$$\n",
    "\\begin{bmatrix} \n",
    "\\checkmark & \\checkmark & \\checkmark  & \\checkmark & \\checkmark\\\\\n",
    "\\checkmark & \\checkmark & \\checkmark  & \\checkmark & \\checkmark\\\\\n",
    "\\checkmark & \\checkmark & \\checkmark  & \\checkmark & \\checkmark\\\\\n",
    "\\checkmark & \\checkmark & \\checkmark  & \\checkmark & ?\\\\\n",
    "\\checkmark & \\checkmark & \\checkmark  & \\checkmark & ?\\\\\n",
    "\\checkmark & \\checkmark & \\checkmark  & \\checkmark & ?\\\\\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Rating prediction \n",
    "\n",
    "- Ratings data has many missing values in the utility matrix. There is no special target column. We want to predict the missing entries in the matrix. \n",
    "- Since our goal is to **predict** ratings, usually the utility matrix is referred to as $Y$ matrix. \n",
    "\n",
    "$$\n",
    "\\begin{bmatrix} \n",
    "? & ? & \\checkmark  & ? & \\checkmark\\\\\n",
    "\\checkmark & ? & ?  & ? & ?\\\\\n",
    "? & \\checkmark & \\checkmark  & ? & \\checkmark\\\\\n",
    "? & ? & ?  & ? & ?\\\\\n",
    "? & ? & ? & \\checkmark & ?\\\\\n",
    "? & \\checkmark & \\checkmark  & ? & \\checkmark\n",
    "\\end{bmatrix}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We don't have sufficient background to understand how collaborative filtering works under-the-hood.\n",
    "- Let's look at an example to understand this at a high level. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sam</td>\n",
       "      <td>Lion King</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Sam</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Sam</td>\n",
       "      <td>The Little Mermaid</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sam</td>\n",
       "      <td>Bambi</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sam</td>\n",
       "      <td>The Social Dilemma</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Eva</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Eva</td>\n",
       "      <td>The Social Dilemma</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Eva</td>\n",
       "      <td>Man on Wire</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pat</td>\n",
       "      <td>The Little Mermaid</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pat</td>\n",
       "      <td>Lion King</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Pat</td>\n",
       "      <td>Bambi</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Jim</td>\n",
       "      <td>The Social Dilemma</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Jim</td>\n",
       "      <td>Malcolm x</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jim</td>\n",
       "      <td>Man on Wire</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id            movie_id  rating\n",
       "0   Sam     Lion King           5     \n",
       "1   Sam     Toy Story           4     \n",
       "2   Sam     The Little Mermaid  5     \n",
       "3   Sam     Bambi               5     \n",
       "4   Sam     The Social Dilemma  1     \n",
       "5   Eva     Toy Story           1     \n",
       "6   Eva     The Social Dilemma  5     \n",
       "7   Eva     Man on Wire         5     \n",
       "8   Pat     The Little Mermaid  4     \n",
       "9   Pat     Lion King           5     \n",
       "10  Pat     Bambi               5     \n",
       "11  Jim     The Social Dilemma  5     \n",
       "12  Jim     Malcolm x           4     \n",
       "13  Jim     Man on Wire         5     "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_ratings = pd.read_csv(\"data/toy-movie-ratings.csv\")\n",
    "toy_ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of users (N)                : 4\n",
      "Number of movies (M)               : 7\n"
     ]
    }
   ],
   "source": [
    "N_toy = len(np.unique(toy_ratings[\"user_id\"]))\n",
    "M_toy = len(np.unique(toy_ratings[\"movie_id\"]))\n",
    "print(\"Number of users (N)                : %d\" % N_toy)\n",
    "print(\"Number of movies (M)               : %d\" % M_toy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_mapper_toy = dict(zip(np.unique(toy_ratings[\"user_id\"]), list(range(N_toy))))\n",
    "item_mapper_toy = dict(zip(np.unique(toy_ratings[\"movie_id\"]), list(range(M_toy))))\n",
    "user_inverse_mapper_toy = dict(\n",
    "    zip(list(range(N_toy)), np.unique(toy_ratings[\"user_id\"]))\n",
    ")\n",
    "item_inverse_mapper_toy = dict(\n",
    "    zip(list(range(M_toy)), np.unique(toy_ratings[\"movie_id\"]))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bambi</th>\n",
       "      <th>Lion King</th>\n",
       "      <th>Malcolm x</th>\n",
       "      <th>Man on Wire</th>\n",
       "      <th>The Little Mermaid</th>\n",
       "      <th>The Social Dilemma</th>\n",
       "      <th>Toy Story</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Eva</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jim</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pat</th>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Bambi  Lion King  Malcolm x  Man on Wire  The Little Mermaid  \\\n",
       "Eva NaN    NaN        NaN         5.0         NaN                   \n",
       "Jim NaN    NaN         4.0        5.0         NaN                   \n",
       "Pat  5.0    5.0       NaN        NaN           4.0                  \n",
       "Sam  5.0    5.0       NaN        NaN           5.0                  \n",
       "\n",
       "     The Social Dilemma  Toy Story  \n",
       "Eva  5.0                 1.0        \n",
       "Jim  5.0                NaN         \n",
       "Pat NaN                 NaN         \n",
       "Sam  1.0                 4.0        "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_toy = create_Y_from_ratings(\n",
    "    toy_ratings, N_toy, M_toy, user_mapper_toy, item_mapper_toy, user_key=\"user_id\", item_key=\"movie_id\"\n",
    ")\n",
    "utility_mat_toy = pd.DataFrame(\n",
    "    Y_toy, columns=item_mapper_toy.keys(), index=user_mapper_toy.keys()\n",
    ")\n",
    "utility_mat_toy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In this toy example, we see clear groups of movies and users.\n",
    "    - For movies: Children movies and documentaries \n",
    "    - For users: Children movie lovers and documentary lovers  \n",
    "- There are some unsupervised models which identify such latent features. \n",
    "- I'll show you how to use a package which implements this popular algorithm for collaborative filtering. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Rating prediction using the surprise package\n",
    "\n",
    "- We'll be using a package called [Surprise](https://surprise.readthedocs.io/en/stable/index.html). \n",
    "- The collaborative filtering algorithm we use in this package is called `SVD`. \n",
    "\n",
    "```\n",
    "pip install scikit-surprise\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try it out on our Jester dataset utility matrix.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import surprise\n",
    "from surprise import SVD, Dataset, Reader, accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "reader = Reader()\n",
    "data = Dataset.load_from_df(ratings, reader)  # Load the data\n",
    "\n",
    "# I'm being sloppy here. Probably there is a way to create validset from our already split data.\n",
    "trainset, validset = surprise.model_selection.train_test_split(\n",
    "    data, test_size=0.2, random_state=42\n",
    ")  # Split the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 5.2893\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "5.28926338380112"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k = 10\n",
    "algo = SVD(n_factors=k, random_state=42)\n",
    "algo.fit(trainset)\n",
    "svd_preds = algo.test(validset)\n",
    "accuracy.rmse(svd_preds, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- No big improvement over the global baseline (RMSE=5.77). \n",
    "- Probably because we are only considering a sample. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Cross-validation for recommender systems\n",
    "\n",
    "- We can also carry out cross-validation and grid search with this package. \n",
    "- Let's look at an example of cross-validation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating RMSE, MAE of algorithm SVD on 5 split(s).\n",
      "\n",
      "                  Fold 1  Fold 2  Fold 3  Fold 4  Fold 5  Mean    Std     \n",
      "RMSE (testset)    5.3015  5.3121  5.2793  5.2826  5.2507  5.2853  0.0211  \n",
      "MAE (testset)     4.2206  4.2267  4.2014  4.1952  4.1699  4.2028  0.0202  \n",
      "Fit time          0.18    0.18    0.16    0.16    0.16    0.17    0.01    \n",
      "Test time         0.06    0.06    0.11    0.06    0.06    0.07    0.02    \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_rmse</th>\n",
       "      <th>test_mae</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>test_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.301509</td>\n",
       "      <td>4.220646</td>\n",
       "      <td>0.182725</td>\n",
       "      <td>0.058555</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.312086</td>\n",
       "      <td>4.226680</td>\n",
       "      <td>0.175239</td>\n",
       "      <td>0.056918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5.279349</td>\n",
       "      <td>4.201399</td>\n",
       "      <td>0.157127</td>\n",
       "      <td>0.113663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.282595</td>\n",
       "      <td>4.195177</td>\n",
       "      <td>0.164283</td>\n",
       "      <td>0.058140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.250745</td>\n",
       "      <td>4.169886</td>\n",
       "      <td>0.160570</td>\n",
       "      <td>0.055707</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   test_rmse  test_mae  fit_time  test_time\n",
       "0  5.301509   4.220646  0.182725  0.058555 \n",
       "1  5.312086   4.226680  0.175239  0.056918 \n",
       "2  5.279349   4.201399  0.157127  0.113663 \n",
       "3  5.282595   4.195177  0.164283  0.058140 \n",
       "4  5.250745   4.169886  0.160570  0.055707 "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from surprise.model_selection import cross_validate\n",
    "\n",
    "pd.DataFrame(cross_validate(algo, data, measures=[\"RMSE\", \"MAE\"], cv=5, verbose=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Jester dataset is available as one of the built-in datasets in this package and you can load it as follows and run cross-validation as follows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating RMSE, MAE of algorithm SVD on 5 split(s).\n",
      "\n",
      "                  Fold 1  Fold 2  Fold 3  Fold 4  Fold 5  Mean    Std     \n",
      "RMSE (testset)    4.5782  4.5814  4.5723  4.6351  4.5787  4.5892  0.0232  \n",
      "MAE (testset)     3.3261  3.3252  3.3162  3.3489  3.3211  3.3275  0.0113  \n",
      "Fit time          2.74    3.01    2.93    3.25    3.19    3.02    0.18    \n",
      "Test time         1.67    1.71    1.54    1.48    1.65    1.61    0.08    \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_rmse</th>\n",
       "      <th>test_mae</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>test_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.578230</td>\n",
       "      <td>3.326136</td>\n",
       "      <td>2.743036</td>\n",
       "      <td>1.672764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.581433</td>\n",
       "      <td>3.325161</td>\n",
       "      <td>3.008567</td>\n",
       "      <td>1.706807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.572321</td>\n",
       "      <td>3.316228</td>\n",
       "      <td>2.930031</td>\n",
       "      <td>1.544924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.635125</td>\n",
       "      <td>3.348926</td>\n",
       "      <td>3.247309</td>\n",
       "      <td>1.483578</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.578656</td>\n",
       "      <td>3.321067</td>\n",
       "      <td>3.187807</td>\n",
       "      <td>1.653728</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   test_rmse  test_mae  fit_time  test_time\n",
       "0  4.578230   3.326136  2.743036  1.672764 \n",
       "1  4.581433   3.325161  3.008567  1.706807 \n",
       "2  4.572321   3.316228  2.930031  1.544924 \n",
       "3  4.635125   3.348926  3.247309  1.483578 \n",
       "4  4.578656   3.321067  3.187807  1.653728 "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = Dataset.load_builtin(\"jester\")\n",
    "\n",
    "pd.DataFrame(cross_validate(algo, data, measures=[\"RMSE\", \"MAE\"], cv=5, verbose=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Content-based filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is content-based filtering? \n",
    "\n",
    "- Supervised machine learning approach\n",
    "- In collaborative filtering we assumed that we only have ratings data. \n",
    "- Usually there is some information on items and users available. \n",
    "- Examples\n",
    "    - Netflix can describe movies as action, romance, comedy, documentaries. \n",
    "    - Amazon could describe books according to topics: math, languages, history. \n",
    "    - Tinder could describe people according to age, location, employment.\n",
    "- Can we use this information to predict ratings in the utility matrix?   \n",
    "    - Yes!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Item and user features\n",
    "\n",
    "- In collaborative filtering we assumed that we only have ratings data. \n",
    "- Usually there is some information available on items and users. \n",
    "- Examples\n",
    "    - Netflix can describe movies as action, romance, comedy, documentaries. \n",
    "    - Amazon could describe books according to topics: math, languages, history. \n",
    "    - Tinder could describe people according to age, location, employment.\n",
    "- Can we use this information to predict ratings in the utility matrix?   \n",
    "    - Yes!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Toy example: Movie recommendation\n",
    "\n",
    "- Let's consider movie recommendation problem with the following toy data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Ratings data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sam</td>\n",
       "      <td>Lion King</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Sam</td>\n",
       "      <td>Jerry Maguire</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Sam</td>\n",
       "      <td>Roman Holidays</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sam</td>\n",
       "      <td>Downfall</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Eva</td>\n",
       "      <td>Titanic</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Eva</td>\n",
       "      <td>Jerry Maguire</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Eva</td>\n",
       "      <td>Inception</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Eva</td>\n",
       "      <td>Man on Wire</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Eva</td>\n",
       "      <td>The Social Dilemma</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pat</td>\n",
       "      <td>Titanic</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Pat</td>\n",
       "      <td>Lion King</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Pat</td>\n",
       "      <td>Bambi</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Pat</td>\n",
       "      <td>Cast Away</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Pat</td>\n",
       "      <td>Jerry Maguire</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Pat</td>\n",
       "      <td>Downfall</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Pat</td>\n",
       "      <td>A Beautiful Mind</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Jim</td>\n",
       "      <td>Titanic</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Jim</td>\n",
       "      <td>Lion King</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Jim</td>\n",
       "      <td>The Social Dilemma</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Jim</td>\n",
       "      <td>Malcolm x</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Jim</td>\n",
       "      <td>Man on Wire</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id            movie_id  rating\n",
       "0   Sam     Lion King           4     \n",
       "1   Sam     Jerry Maguire       4     \n",
       "2   Sam     Roman Holidays      5     \n",
       "3   Sam     Downfall            1     \n",
       "4   Eva     Titanic             2     \n",
       "5   Eva     Jerry Maguire       1     \n",
       "6   Eva     Inception           4     \n",
       "7   Eva     Man on Wire         5     \n",
       "8   Eva     The Social Dilemma  5     \n",
       "9   Pat     Titanic             3     \n",
       "10  Pat     Lion King           4     \n",
       "11  Pat     Bambi               4     \n",
       "12  Pat     Cast Away           3     \n",
       "13  Pat     Jerry Maguire       5     \n",
       "14  Pat     Downfall            2     \n",
       "15  Pat     A Beautiful Mind    3     \n",
       "16  Jim     Titanic             2     \n",
       "17  Jim     Lion King           3     \n",
       "18  Jim     The Social Dilemma  5     \n",
       "19  Jim     Malcolm x           4     \n",
       "20  Jim     Man on Wire         5     "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_ratings = pd.read_csv(\"data/toy_ratings.csv\")\n",
    "toy_ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of users (N)                : 4\n",
      "Number of movies (M)               : 12\n"
     ]
    }
   ],
   "source": [
    "N = len(np.unique(toy_ratings[\"user_id\"]))\n",
    "M = len(np.unique(toy_ratings[\"movie_id\"]))\n",
    "print(\"Number of users (N)                : %d\" % N)\n",
    "print(\"Number of movies (M)               : %d\" % M)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "user_key = \"user_id\"\n",
    "item_key = \"movie_id\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_mapper = dict(zip(np.unique(toy_ratings[user_key]), list(range(N))))\n",
    "item_mapper = dict(zip(np.unique(toy_ratings[item_key]), list(range(M))))\n",
    "user_inverse_mapper = dict(zip(list(range(N)), np.unique(toy_ratings[user_key])))\n",
    "item_inverse_mapper = dict(zip(list(range(M)), np.unique(toy_ratings[item_key])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Utility matrix\n",
    "\n",
    "Let's create a dense utility matrix for our toy dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def create_Y_from_ratings(data, N, M):\n",
    "    Y = np.zeros((N, M))\n",
    "    Y.fill(np.nan)\n",
    "    for index, val in data.iterrows():\n",
    "        n = user_mapper[val[user_key]]\n",
    "        m = item_mapper[val[item_key]]\n",
    "        Y[n, m] = val[\"rating\"]\n",
    "\n",
    "    return Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Utility matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A Beautiful Mind</th>\n",
       "      <th>Bambi</th>\n",
       "      <th>Cast Away</th>\n",
       "      <th>Downfall</th>\n",
       "      <th>Inception</th>\n",
       "      <th>Jerry Maguire</th>\n",
       "      <th>Lion King</th>\n",
       "      <th>Malcolm x</th>\n",
       "      <th>Man on Wire</th>\n",
       "      <th>Roman Holidays</th>\n",
       "      <th>The Social Dilemma</th>\n",
       "      <th>Titanic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Eva</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jim</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pat</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A Beautiful Mind  Bambi  Cast Away  Downfall  Inception  Jerry Maguire  \\\n",
       "Eva NaN               NaN    NaN        NaN        4.0        1.0             \n",
       "Jim NaN               NaN    NaN        NaN       NaN        NaN              \n",
       "Pat  3.0               4.0    3.0        2.0      NaN         5.0             \n",
       "Sam NaN               NaN    NaN         1.0      NaN         4.0             \n",
       "\n",
       "     Lion King  Malcolm x  Man on Wire  Roman Holidays  The Social Dilemma  \\\n",
       "Eva NaN        NaN         5.0         NaN              5.0                  \n",
       "Jim  3.0        4.0        5.0         NaN              5.0                  \n",
       "Pat  4.0       NaN        NaN          NaN             NaN                   \n",
       "Sam  4.0       NaN        NaN           5.0            NaN                   \n",
       "\n",
       "     Titanic  \n",
       "Eva  2.0      \n",
       "Jim  2.0      \n",
       "Pat  3.0      \n",
       "Sam NaN       "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y = create_Y_from_ratings(toy_ratings, N, M)\n",
    "utility_mat = pd.DataFrame(Y, columns=item_mapper.keys(), index=user_mapper.keys())\n",
    "utility_mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg = np.nanmean(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Goal**: Predict missing entries in the utility matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "import surprise\n",
    "from surprise import SVD, Dataset, Reader, accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's predict ratings with collaborative filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "reader = Reader()\n",
    "data = Dataset.load_from_df(toy_ratings, reader)  # Load the data\n",
    "\n",
    "trainset, validset = surprise.model_selection.train_test_split(\n",
    "    data, test_size=0.01, random_state=42\n",
    ")  # Split the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "k = 2\n",
    "algo = SVD(n_factors=k, random_state=42)\n",
    "algo.fit(trainset)\n",
    "preds = algo.test(trainset.build_testset())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "rating_preds = defaultdict(list)\n",
    "for uid, iid, true_r, est, _ in preds:\n",
    "    rating_preds[uid].append((iid, est))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(list,\n",
       "            {'Sam': [('Lion King', 3.5442874862516582),\n",
       "              ('Jerry Maguire', 3.471958396420975),\n",
       "              ('Downfall', 3.141157981632877),\n",
       "              ('Roman Holidays', 3.6555436348053982)],\n",
       "             'Jim': [('Lion King', 3.6494404051925047),\n",
       "              ('The Social Dilemma', 3.8739407581035588),\n",
       "              ('Titanic', 3.295718235231984),\n",
       "              ('Man on Wire', 3.8839492577938532),\n",
       "              ('Malcolm x', 3.6435176323135128)],\n",
       "             'Pat': [('A Beautiful Mind', 3.4463313322323263),\n",
       "              ('Bambi', 3.540418795140043),\n",
       "              ('Jerry Maguire', 3.4582870107738803),\n",
       "              ('Titanic', 3.1872411557123517),\n",
       "              ('Cast Away', 3.4442142132704827),\n",
       "              ('Lion King', 3.5286392016604875),\n",
       "              ('Downfall', 3.133747883605952)],\n",
       "             'Eva': [('The Social Dilemma', 3.6665140635371194),\n",
       "              ('Jerry Maguire', 3.3423360343482957),\n",
       "              ('Titanic', 3.113324069881786),\n",
       "              ('Man on Wire', 3.685575559931666)]})"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rating_preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Movie features\n",
    "\n",
    "- Suppose we also have movie features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Action</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>Children</th>\n",
       "      <th>Documentary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A Beautiful Mind</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bambi</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cast Away</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Downfall</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Inception</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jerry Maguire</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lion King</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Malcolm x</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Man on Wire</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Roman Holidays</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>The Social Dilemma</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Titanic</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Action  Romance  Drama  Comedy  Children  Documentary\n",
       "A Beautiful Mind    0       1        1      0       0         0          \n",
       "Bambi               0       0        1      0       1         0          \n",
       "Cast Away           0       1        1      0       0         0          \n",
       "Downfall            0       0        0      0       0         1          \n",
       "Inception           1       0        1      0       0         0          \n",
       "Jerry Maguire       0       1        1      1       0         0          \n",
       "Lion King           0       0        1      0       1         0          \n",
       "Malcolm x           0       0        0      0       0         1          \n",
       "Man on Wire         0       0        0      0       0         1          \n",
       "Roman Holidays      0       1        1      1       0         0          \n",
       "The Social Dilemma  0       0        0      0       0         1          \n",
       "Titanic             0       1        1      0       0         0          "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_feats_df = pd.read_csv(\"data/toy_movie_feats.csv\", index_col=0)\n",
    "movie_feats_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12, 6)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z = movie_feats_df.to_numpy()\n",
    "Z.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How can we use these features to predict missing ratings? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Overall idea\n",
    "\n",
    "- Using the ratings data and movie features, we'll build **profiles for different users**. \n",
    "- Let's consider an example user **Pat**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Pat's ratings\n",
    "\n",
    "- We don't know anything about Pat but we know her ratings to movies. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A Beautiful Mind      3.0\n",
       "Bambi                 4.0\n",
       "Cast Away             3.0\n",
       "Downfall              2.0\n",
       "Inception            NaN \n",
       "Jerry Maguire         5.0\n",
       "Lion King             4.0\n",
       "Malcolm x            NaN \n",
       "Man on Wire          NaN \n",
       "Roman Holidays       NaN \n",
       "The Social Dilemma   NaN \n",
       "Titanic               3.0\n",
       "Name: Pat, dtype: float64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utility_mat.loc[\"Pat\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We also know about movies and their features. \n",
    "- If Pat gave a high rating to _Lion King_, it means that she liked the features of the movie. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Action         0\n",
       "Romance        0\n",
       "Drama          1\n",
       "Comedy         0\n",
       "Children       1\n",
       "Documentary    0\n",
       "Name: Lion King, dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_feats_df.loc[\"Lion King\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Supervised approach to rating prediction \n",
    "\n",
    "- We treat ratings prediction problem as a set of regression problems. \n",
    "- Given movie information, we create user profile for each user.\n",
    "- Build regression model for each user and learn regression weights for each user. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We build a profile for users based on \n",
    "    - the movies they have watched\n",
    "    - their rating for the movies\n",
    "    - the features of the movies\n",
    "- We train a personalized regression model for each user using this information. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Supervised approach to rating prediction \n",
    "\n",
    "For each user $i$ create a user profile as follows. \n",
    "\n",
    "- Consider all movies rated by $i$ and create `X` and `y` for the user: \n",
    "    - Each row in `X` contains the movie features of movie $j$ rated by $i$. \n",
    "    - Each value in `y` is the corresponding rating given to the movie $j$ by user $i$. \n",
    "- Fit a regression model using `X` and `y`. \n",
    "- Apply the model to predict ratings for new items! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Let's build user profiles \n",
    "\n",
    "- Build `X` and `y` for all users. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "\n",
    "def get_lr_data_per_user(ratings_df, d):\n",
    "    lr_y = defaultdict(list)\n",
    "    lr_X = defaultdict(list)\n",
    "    lr_items = defaultdict(list)\n",
    "\n",
    "    for index, val in ratings_df.iterrows():\n",
    "        n = user_mapper[val[user_key]]\n",
    "        m = item_mapper[val[item_key]]\n",
    "        lr_X[n].append(Z[m])\n",
    "        lr_y[n].append(val[\"rating\"])\n",
    "        lr_items[n].append(m)\n",
    "\n",
    "    for n in lr_X:\n",
    "        lr_X[n] = np.array(lr_X[n])\n",
    "        lr_y[n] = np.array(lr_y[n])\n",
    "\n",
    "    return lr_X, lr_y, lr_items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "d = movie_feats_df.shape[1]\n",
    "X_train_usr, y_train_usr, rated_items = get_lr_data_per_user(toy_ratings, d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What's going is be shape of each `X` and `y`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examine user profiles \n",
    "\n",
    "- Let's examine some user profiles. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def get_user_profile(user_name):\n",
    "    X = X_train_usr[user_mapper[user_name]]\n",
    "    y = y_train_usr[user_mapper[user_name]]\n",
    "    items = rated_items[user_mapper[user_name]]\n",
    "    movie_names = [item_inverse_mapper[item] for item in items]\n",
    "    print(\"Profile for user: \", user_name)\n",
    "    profile_df = pd.DataFrame(X, columns=movie_feats_df.columns, index=movie_names)\n",
    "    profile_df[\"ratings\"] = y\n",
    "    return profile_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Pat's profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Profile for user:  Pat\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Action</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>Children</th>\n",
       "      <th>Documentary</th>\n",
       "      <th>ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Titanic</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lion King</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bambi</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cast Away</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jerry Maguire</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Downfall</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A Beautiful Mind</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Action  Romance  Drama  Comedy  Children  Documentary  \\\n",
       "Titanic           0       1        1      0       0         0             \n",
       "Lion King         0       0        1      0       1         0             \n",
       "Bambi             0       0        1      0       1         0             \n",
       "Cast Away         0       1        1      0       0         0             \n",
       "Jerry Maguire     0       1        1      1       0         0             \n",
       "Downfall          0       0        0      0       0         1             \n",
       "A Beautiful Mind  0       1        1      0       0         0             \n",
       "\n",
       "                  ratings  \n",
       "Titanic           3        \n",
       "Lion King         4        \n",
       "Bambi             4        \n",
       "Cast Away         3        \n",
       "Jerry Maguire     5        \n",
       "Downfall          2        \n",
       "A Beautiful Mind  3        "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_user_profile(\"Pat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pat seems to like Children's movies and movies with Comedy. \n",
    "- Seems like she's not so much into romantic movies.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Eva's profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Profile for user:  Eva\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Action</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>Children</th>\n",
       "      <th>Documentary</th>\n",
       "      <th>ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Titanic</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jerry Maguire</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Inception</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Man on Wire</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>The Social Dilemma</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Action  Romance  Drama  Comedy  Children  Documentary  \\\n",
       "Titanic             0       1        1      0       0         0             \n",
       "Jerry Maguire       0       1        1      1       0         0             \n",
       "Inception           1       0        1      0       0         0             \n",
       "Man on Wire         0       0        0      0       0         1             \n",
       "The Social Dilemma  0       0        0      0       0         1             \n",
       "\n",
       "                    ratings  \n",
       "Titanic             2        \n",
       "Jerry Maguire       1        \n",
       "Inception           4        \n",
       "Man on Wire         5        \n",
       "The Social Dilemma  5        "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_user_profile(\"Eva\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Eva hasn't rated many movies. There are not many rows. \n",
    "- Eva seems to like documentaries and action movies. \n",
    "- Seems like she's not so much into romantic movies.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Regression models for users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "\n",
    "def train_for_usr(user_name, model=Ridge()):\n",
    "    X = X_train_usr[user_mapper[user_name]]\n",
    "    y = y_train_usr[user_mapper[user_name]]\n",
    "    model.fit(X, y)\n",
    "    return model\n",
    "\n",
    "\n",
    "def predict_for_usr(model, movie_names):\n",
    "    feat_vecs = movie_feats_df.loc[movie_names].values\n",
    "    preds = model.predict(feat_vecs)\n",
    "    return preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Regression model for Pat \n",
    "\n",
    "- What are the regression weights learned for Pat? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coefficients for Pat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Action</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Romance</th>\n",
       "      <td>-0.020833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Drama</th>\n",
       "      <td>0.437500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Comedy</th>\n",
       "      <td>0.854167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Children</th>\n",
       "      <td>0.458333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Documentary</th>\n",
       "      <td>-0.437500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Coefficients for Pat\n",
       "Action       0.000000            \n",
       "Romance     -0.020833            \n",
       "Drama        0.437500            \n",
       "Comedy       0.854167            \n",
       "Children     0.458333            \n",
       "Documentary -0.437500            "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_name = \"Pat\"\n",
    "pat_model = train_for_usr(user_name)\n",
    "col = \"Coefficients for %s\" % user_name\n",
    "pd.DataFrame(pat_model.coef_, index=movie_feats_df.columns, columns=[col])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Predictions for Pat\n",
    "- How would Pat rate some movies she hasn't seen? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Action</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>Children</th>\n",
       "      <th>Documentary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Roman Holidays</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Malcolm x</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Action  Romance  Drama  Comedy  Children  Documentary\n",
       "Roman Holidays  0       1        1      1       0         0          \n",
       "Malcolm x       0       0        0      0       0         1          "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_to_pred = [\"Roman Holidays\", \"Malcolm x\"]\n",
    "pred_df = movie_feats_df.loc[movies_to_pred]\n",
    "pred_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Action</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>Children</th>\n",
       "      <th>Documentary</th>\n",
       "      <th>Pat's predicted ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Roman Holidays</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.145833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Malcolm x</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.437500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Action  Romance  Drama  Comedy  Children  Documentary  \\\n",
       "Roman Holidays  0       1        1      1       0         0             \n",
       "Malcolm x       0       0        0      0       0         1             \n",
       "\n",
       "                Pat's predicted ratings  \n",
       "Roman Holidays  4.145833                 \n",
       "Malcolm x       2.437500                 "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_name = \"Pat\"\n",
    "preds = predict_for_usr(pat_model, movies_to_pred)\n",
    "pred_df[user_name + \"'s predicted ratings\"] = preds\n",
    "pred_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Regression model for Eva \n",
    "\n",
    "- What are the regression weights learned for Eva? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coefficients for Eva</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Action</th>\n",
       "      <td>0.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Romance</th>\n",
       "      <td>-1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Drama</th>\n",
       "      <td>-0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Comedy</th>\n",
       "      <td>-0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Children</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Documentary</th>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Coefficients for Eva\n",
       "Action       0.333333            \n",
       "Romance     -1.000000            \n",
       "Drama       -0.666667            \n",
       "Comedy      -0.666667            \n",
       "Children     0.000000            \n",
       "Documentary  0.666667            "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_name = \"Eva\"\n",
    "eva_model = train_for_usr(user_name)\n",
    "col = \"Coefficients for %s\" % user_name\n",
    "pd.DataFrame(eva_model.coef_, index=movie_feats_df.columns, columns=[col])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Predictions for Eva\n",
    "\n",
    "- What are the predicted ratings for Eva for a list of movies?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Action</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>Children</th>\n",
       "      <th>Documentary</th>\n",
       "      <th>Pat's predicted ratings</th>\n",
       "      <th>Eva's predicted ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Roman Holidays</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.145833</td>\n",
       "      <td>1.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Malcolm x</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.437500</td>\n",
       "      <td>4.666667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Action  Romance  Drama  Comedy  Children  Documentary  \\\n",
       "Roman Holidays  0       1        1      1       0         0             \n",
       "Malcolm x       0       0        0      0       0         1             \n",
       "\n",
       "                Pat's predicted ratings  Eva's predicted ratings  \n",
       "Roman Holidays  4.145833                 1.666667                 \n",
       "Malcolm x       2.437500                 4.666667                 "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_name = \"Eva\"\n",
    "preds = predict_for_usr(eva_model, movies_to_pred)\n",
    "pred_df[user_name + \"'s predicted ratings\"] = preds\n",
    "pred_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Completing the utility matrix with content-based filtering\n",
    "\n",
    "Here is the original utility matrix.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A Beautiful Mind</th>\n",
       "      <th>Bambi</th>\n",
       "      <th>Cast Away</th>\n",
       "      <th>Downfall</th>\n",
       "      <th>Inception</th>\n",
       "      <th>Jerry Maguire</th>\n",
       "      <th>Lion King</th>\n",
       "      <th>Malcolm x</th>\n",
       "      <th>Man on Wire</th>\n",
       "      <th>Roman Holidays</th>\n",
       "      <th>The Social Dilemma</th>\n",
       "      <th>Titanic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Eva</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jim</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pat</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A Beautiful Mind  Bambi  Cast Away  Downfall  Inception  Jerry Maguire  \\\n",
       "Eva NaN               NaN    NaN        NaN        4.0        1.0             \n",
       "Jim NaN               NaN    NaN        NaN       NaN        NaN              \n",
       "Pat  3.0               4.0    3.0        2.0      NaN         5.0             \n",
       "Sam NaN               NaN    NaN         1.0      NaN         4.0             \n",
       "\n",
       "     Lion King  Malcolm x  Man on Wire  Roman Holidays  The Social Dilemma  \\\n",
       "Eva NaN        NaN         5.0         NaN              5.0                  \n",
       "Jim  3.0        4.0        5.0         NaN              5.0                  \n",
       "Pat  4.0       NaN        NaN          NaN             NaN                   \n",
       "Sam  4.0       NaN        NaN           5.0            NaN                   \n",
       "\n",
       "     Titanic  \n",
       "Eva  2.0      \n",
       "Jim  2.0      \n",
       "Pat  3.0      \n",
       "Sam NaN       "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utility_mat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Using predictions per user, we can fill in missing entries in the utility matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "models = dict()\n",
    "pred_lin_reg = np.zeros((N, M))\n",
    "\n",
    "for n in range(N):\n",
    "    models[n] = Ridge()\n",
    "    models[n].fit(X_train_usr[n], y_train_usr[n])\n",
    "    pred_lin_reg[n] = models[n].predict(Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A Beautiful Mind</th>\n",
       "      <th>Bambi</th>\n",
       "      <th>Cast Away</th>\n",
       "      <th>Downfall</th>\n",
       "      <th>Inception</th>\n",
       "      <th>Jerry Maguire</th>\n",
       "      <th>Lion King</th>\n",
       "      <th>Malcolm x</th>\n",
       "      <th>Man on Wire</th>\n",
       "      <th>Roman Holidays</th>\n",
       "      <th>The Social Dilemma</th>\n",
       "      <th>Titanic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Eva</th>\n",
       "      <td>2.333333</td>\n",
       "      <td>3.333333</td>\n",
       "      <td>2.333333</td>\n",
       "      <td>4.666667</td>\n",
       "      <td>3.666667</td>\n",
       "      <td>1.666667</td>\n",
       "      <td>3.333333</td>\n",
       "      <td>4.666667</td>\n",
       "      <td>4.666667</td>\n",
       "      <td>1.666667</td>\n",
       "      <td>4.666667</td>\n",
       "      <td>2.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jim</th>\n",
       "      <td>2.575000</td>\n",
       "      <td>3.075000</td>\n",
       "      <td>2.575000</td>\n",
       "      <td>4.450000</td>\n",
       "      <td>3.150000</td>\n",
       "      <td>2.575000</td>\n",
       "      <td>3.075000</td>\n",
       "      <td>4.450000</td>\n",
       "      <td>4.450000</td>\n",
       "      <td>2.575000</td>\n",
       "      <td>4.450000</td>\n",
       "      <td>2.575000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pat</th>\n",
       "      <td>3.291667</td>\n",
       "      <td>3.770833</td>\n",
       "      <td>3.291667</td>\n",
       "      <td>2.437500</td>\n",
       "      <td>3.312500</td>\n",
       "      <td>4.145833</td>\n",
       "      <td>3.770833</td>\n",
       "      <td>2.437500</td>\n",
       "      <td>2.437500</td>\n",
       "      <td>4.145833</td>\n",
       "      <td>2.437500</td>\n",
       "      <td>3.291667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>3.810811</td>\n",
       "      <td>3.675676</td>\n",
       "      <td>3.810811</td>\n",
       "      <td>1.783784</td>\n",
       "      <td>3.351351</td>\n",
       "      <td>4.270270</td>\n",
       "      <td>3.675676</td>\n",
       "      <td>1.783784</td>\n",
       "      <td>1.783784</td>\n",
       "      <td>4.270270</td>\n",
       "      <td>1.783784</td>\n",
       "      <td>3.810811</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A Beautiful Mind     Bambi  Cast Away  Downfall  Inception  \\\n",
       "Eva  2.333333          3.333333  2.333333   4.666667  3.666667    \n",
       "Jim  2.575000          3.075000  2.575000   4.450000  3.150000    \n",
       "Pat  3.291667          3.770833  3.291667   2.437500  3.312500    \n",
       "Sam  3.810811          3.675676  3.810811   1.783784  3.351351    \n",
       "\n",
       "     Jerry Maguire  Lion King  Malcolm x  Man on Wire  Roman Holidays  \\\n",
       "Eva  1.666667       3.333333   4.666667   4.666667     1.666667         \n",
       "Jim  2.575000       3.075000   4.450000   4.450000     2.575000         \n",
       "Pat  4.145833       3.770833   2.437500   2.437500     4.145833         \n",
       "Sam  4.270270       3.675676   1.783784   1.783784     4.270270         \n",
       "\n",
       "     The Social Dilemma   Titanic  \n",
       "Eva  4.666667            2.333333  \n",
       "Jim  4.450000            2.575000  \n",
       "Pat  2.437500            3.291667  \n",
       "Sam  1.783784            3.810811  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(pred_lin_reg, columns=item_mapper.keys(), index=user_mapper.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### More comments on content-based filtering\n",
    "\n",
    "- The feature matrix for movies can contain different types of features.\n",
    "    - Example: Plot of the movie (text features), actors (categorical features), year of the movie, budget and revenue of the movie (numerical features). \n",
    "    - You'll apply our usual preprocessing techniques to these features. \n",
    "- If you have enough data, you could also carry out hyperparameter tuning with cross-validation for each model.\n",
    "- Finally, although we have been talking about linear models above, you can use any regression model of your choice. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Advantages of content-based filtering \n",
    "\n",
    "- We don't need many users to provide ratings for an item. \n",
    "- Each user is modeled separately, so you might be able to capture uniqueness of taste. \n",
    "- Since you can obtain the features of the items, you can immediately recommend new items. \n",
    "    - This would not have been possible with collaborative filtering. \n",
    "- Recommendations are interpretable.\n",
    "    - You can explain to the user why you are recommending an item because you have learned weights. \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Disadvantages of content-based filtering \n",
    "\n",
    "- Feature acquisition and feature engineering\n",
    "    - What features should we use to explain the difference in ratings? \n",
    "    - Obtaining those features for each item might be very expensive. \n",
    "- Less diversity: hardly recommend an item outside the user's profile. \n",
    "- Cold start: When a new user shows up, you don't have any information about them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hybrid filtering\n",
    "\n",
    "- Combining advantages of collaborative filtering and content-based filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Final comments and summary <a name=\"1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Formulating the problem of recommender systems \n",
    "\n",
    "- We are given ratings data. \n",
    "- We use this data to create **utility matrix** which encodes interactions between users and items. \n",
    "- The utility matrix has many missing entries. \n",
    "- We defined recommendation systems problem as **matrix completion problem**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What did we cover? \n",
    "\n",
    "- There is a big world of recommendation systems out there. We talked about some basic traditional approaches to recommender systems. \n",
    "    - collaborative filtering \n",
    "    - content-based filtering "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to know more advanced approaches to recommender systems, watch this 4-hour summer school tutorial by Xavier Amatriain, Research/Engineering Director @ Netflix.  \n",
    "\n",
    "- [Part1](https://www.youtube.com/watch?v=bLhq63ygoU8)\n",
    "- [Part2](https://www.youtube.com/watch?v=mRToFXlNBpQ)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Evaluation \n",
    "\n",
    "- We split the data similar to supervised systems. \n",
    "- We evaluate recommendation systems using traditional regression metrics such as MSE or RMSE. \n",
    "- But real evaluation of recommender system can be very tricky because there is no ground truth. \n",
    "- We have been using RMSE due to the lack of a better measure.  \n",
    "- What we actually want to measure is the interest that our user has in the recommended items. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Beyond error rate in recommendation systems \n",
    "\n",
    "- If a system gives the best RMSE it doesn't necessarily mean that it's going to give best recommendations. \n",
    "- In recommendation systems we do not have ground truth.\n",
    "- Just training your model and evaluating it offline is not ideal. \n",
    "- Other aspects such as simplicity, interpretation, code maintainability are equally (if not more) important than best validation error. \n",
    "- Winning system of Netflix Challenge was never adopted.\n",
    "    - Big mess of ensembles was not really maintainable \n",
    "- There are other considerations. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other issues important in recommender systems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Are these good recommendations? \n",
    "\n",
    "You are looking for water shoes and at the moment you are looking at [VIFUUR Water Sports Shoes](https://www.amazon.ca/VIFUUR-Barefoot-Quick-Dry-Blue-38-39/dp/B0753DL15Y), are these good recommendations? \n",
    "\n",
    "![](img/reco-diversity.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now suppose you've recently bought VIFUUR Water Sports Shoes and rated them highly. Are these good recommendations now? \n",
    "- Not really. Even though you really liked them you don't need them anymore. You want some non-Water Sports Shoes recommendations.\n",
    "- **Diversity** is about how different are the recommendations. \n",
    "    - Another example: Even if you really really like Star Wars, you might want non-Star-Wars suggestions.    \n",
    "- But be careful. We need a balance here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Are these good recommendations? \n",
    "\n",
    "![](img/freshness.png)\n",
    "\n",
    "- Some of these books don't have many ratings but it might be a good idea to recommend \"fresh\" things. \n",
    "- **Freshness**: people tend to get more excited about new/surprising things.    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- But again you need a balance here. What would happen if you keep surprising the user all the time? \n",
    "- There might be **trust** issues. \n",
    "- Another aspect of trust is explaining your recommendation, i.e., telling the user why you made a recommendation. This gives the user an opportunity to understand why your recommendations could be interesting to them.   \n",
    "    \n",
    "![](img/recommendation-explanation.png)    \n",
    "\n",
    "[Source](https://sudonull.com/post/65548-10-lessons-of-the-Quora-recommendation-system-Retail-Rocket-Blog)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Persistence**: how long should recommendations last?\n",
    "- If you keep not clicking on a recommendation, should it remain a recommendation?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Social recommendation**: what did your friends watch?\n",
    "- Many recommenders\tare\tnow\tconnected to social\tnetworks.\n",
    "- \"Login using you Facebook\taccount\".\n",
    "- Often, people\tlike similar movies\tto their friends.\n",
    "- If we get a new user, then recommendations are based on friend's preferences. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Types of data \n",
    "\n",
    "- Explicit data: ratings, thumbs up, etc. \n",
    "- Implicit data: collected from the users' behaviour (e.g., mouse clicks, purchases, time spent doing something)\n",
    "- Trust implicit data that costs something, like time or even money. \n",
    "    - this makes it harder to fraud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Some thoughts on recommendation systems  \n",
    "- Be mindful of the consequences of recommendation systems. \n",
    "    - Recommendation systems can have terrible consequences. \n",
    "- Companies such as Amazon,  Netflix, Facebook, Google (YouTube), which extensively use recommendation systems, are profit-driven and so they design these systems to maximize user attention; their focus is not necessarily human well-being. \n",
    "- There are tons of news and research articles on serious consequences of recommendation systems.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Some thoughts on recommendation systems  \n",
    "\n",
    "- Some weird stories which got media attention.   \n",
    "[How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did](https://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/?sh=3171af136668)\n",
    "- More serious consequences are in political contexts. \n",
    "    - [Facebook Admits It Was Used to Incite Violence in Myanmar](https://www.nytimes.com/2018/11/06/technology/myanmar-facebook.html)\n",
    "    - [YouTube Extremism and the Long Tail](https://www.theatlantic.com/politics/archive/2018/03/youtube-extremism-and-the-long-tail/555350/)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### My advice\n",
    "\n",
    "- Ask hard and uncomfortable questions to yourself (and to your employer if possible) before implementing and deploying such systems.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Resources \n",
    "\n",
    "- [Collaborative filtering for recommendation systems in Python, by N. Hug](https://www.youtube.com/watch?v=z0dx-YckFko)\n",
    "- [An interesting talk: The paradox of choice](https://www.ted.com/talks/barry_schwartz_the_paradox_of_choice)\n",
    "- [How Netflix’s Recommendations System Works](https://help.netflix.com/en/node/100639)\n",
    "- [Hands on Recommendation Systems with Python](https://learning.oreilly.com/library/view/hands-on-recommendation-systems/9781788993753/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}