Movie_Recommendation_System.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Minimal Recommendation Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "### References\n",
    "Credit where credit is due. Please check the references at the bottom of this document.\n",
    "\n",
    "### Dataset\n",
    "MovieLens from GroupLens Research: [grouplens.org](http://www.grouplens.org/)\n",
    "\n",
    "The MovieLens 1M data set contains 1 million ratings collected from 6000 users\n",
    "on 4000 movies.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Roadmap\n",
    "\n",
    "What exactly are we going to do? Here's a high-level overview:\n",
    "\n",
    "- quick introduction to the recommendation problem (theory)\n",
    "- known solutions & challenges to the recommendation problem (theory) \n",
    "- load data, setup evaluation functions, test dummy solution (practice) \n",
    "- minimal reco engine v1.0 (practice) \n",
    "- more formulas! (theory) \n",
    "- pandas aggregation & minimal reco engine v1.1 (practice) \n",
    "- challenge (practice) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Recommendation Problem\n",
    "\n",
    "Recommenders have been around since at least 1992. Today we see different flavours of recommenders, deployed across different verticals: \n",
    "\n",
    "- Amazon\n",
    "- Netflix\n",
    "- Facebook\n",
    "- Last.fm.\n",
    "\n",
    "What exactly do they do?\n",
    "\n",
    "### Definitions from the literature\n",
    "\n",
    "- *In a typical recommender system people provide recommendations as inputs, which\n",
    "the system then aggregates and directs to appropriate recipients.* -- Resnick\n",
    "and Varian, 1997\n",
    "\n",
    "- *Collaborative filtering simply means that people collaborate to help one\n",
    "another perform filtering by recording their reactions to documents they read.*\n",
    "-- Goldberg et al, 1992\n",
    "\n",
    "- *In its most common formulation, the recommendation problem is reduced to the\n",
    "problem of estimating ratings for the items that have not been seen by a\n",
    "user. Intuitively, this estimation is usually based on the ratings given by this\n",
    "user to other items and on some other information [...] Once we can estimate\n",
    "ratings for the yet unrated items, we can recommend to the user the item(s) with\n",
    "the highest estimated rating(s).* -- Adomavicius and Tuzhilin, 2005\n",
    "\n",
    "- *Driven by computer algorithms, recommenders help consumers\n",
    "by selecting products they will probably like and might buy\n",
    "based on their browsing, searches, purchases, and preferences.* -- Konstan and Riedl, 2012"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notation\n",
    "\n",
    "- $U$ is the set of users in our domain. Its size is $|U|$.\n",
    "- $I$ is the set of items in our domain. Its size is $|I|$.\n",
    "- $I(u)$ is the set of items that user $u$ has rated.\n",
    "- $-I(u)$ is the complement of $I(u)$ i.e., the set of items not yet seen by user $u$.\n",
    "- $U(i)$ is the set of users that have rated item $i$.\n",
    "- $-U(i)$ is the complement of $U(i)$.\n",
    "- $S(u,i)$ is a function that measures the utility of item $i$ for user $u$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Goal of a recommendation system\n",
    "\n",
    "$\n",
    "i^{*} = argmax_{i \\in -I(u)} S(u,i), \\forall{u \\in U}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem statement\n",
    "\n",
    "The recommendation problem in its most basic form is quite simple to define:\n",
    "\n",
    "```\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| user_id, movie_id | m_1 | m_2 | m_3 | m_4 | m_5 |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_1               | ?   | ?   | 4   | ?   | 1   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_2               | 3   | ?   | ?   | 2   | 2   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_3               | 3   | ?   | ?   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_4               | ?   | 1   | 2   | 1   | 1   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_5               | ?   | ?   | ?   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_6               | 2   | ?   | 2   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_7               | ?   | ?   | ?   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_8               | 3   | 1   | 5   | ?   | ?   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "| u_9               | ?   | ?   | ?   | ?   | 2   |\n",
    "|-------------------+-----+-----+-----+-----+-----|\n",
    "```\n",
    "\n",
    "*Given a partially filled matrix of ratings ($|U|x|I|$), estimate the missing values.*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MovieLens dataset: loading and first look\n",
    "\n",
    "Loading of the MovieLens dataset is based on the intro chapter of 'Python\n",
    "for Data Analysis\".\n",
    "\n",
    "The MovieLens data is spread across three files. We'll load each file using the `pd.read_table` function:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:41.119803Z",
     "start_time": "2019-05-23T04:26:35.251262Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('always')\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# set some print options\n",
    "np.set_printoptions(precision=4)\n",
    "np.set_printoptions(threshold=5)\n",
    "np.set_printoptions(suppress=True)\n",
    "pd.set_option('precision', 3, 'notebook_repr_html', True, )\n",
    "\n",
    "# init random gen\n",
    "np.random.seed(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:52.623270Z",
     "start_time": "2019-05-23T04:26:41.124817Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  timestamp\n",
       "0        1      1193       5  978300760\n",
       "1        1       661       3  978302109\n",
       "2        1       914       3  978301968\n",
       "3        1      3408       4  978300275\n",
       "4        1      2355       5  978824291"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users = pd.read_table('data/ml-1m/users.dat',\n",
    "                      sep='::', header=None, \n",
    "                      names=['user_id', 'gender', 'age', 'occupation', 'zip'])\n",
    "\n",
    "ratings = pd.read_table('data/ml-1m/ratings.dat',\n",
    "                        sep='::', header=None, \n",
    "                        names=['user_id', 'movie_id', 'rating', 'timestamp'])\n",
    "\n",
    "movies = pd.read_table('data/ml-1m/movies.dat',\n",
    "                       sep='::', header=None, \n",
    "                       names=['movie_id', 'title', 'genres'])\n",
    "\n",
    "# show how one of them looks\n",
    "ratings.head(5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `pd.merge` we get it all into  one big DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:53.461492Z",
     "start_time": "2019-05-23T04:26:52.626354Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978298413</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978220179</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>12</td>\n",
       "      <td>32793</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978199279</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>7</td>\n",
       "      <td>22903</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978158471</td>\n",
       "      <td>M</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>95350</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  timestamp gender  age  occupation    zip  \\\n",
       "0        1      1193       5  978300760      F    1          10  48067   \n",
       "1        2      1193       5  978298413      M   56          16  70072   \n",
       "2       12      1193       4  978220179      M   25          12  32793   \n",
       "3       15      1193       4  978199279      M   25           7  22903   \n",
       "4       17      1193       5  978158471      M   50           1  95350   \n",
       "\n",
       "                                    title genres  \n",
       "0  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "1  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "2  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "3  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "4  One Flew Over the Cuckoo's Nest (1975)  Drama  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens = pd.merge(pd.merge(ratings, users), movies)\n",
    "movielens.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation mechanism\n",
    "\n",
    "Before we start building our minimal reco engine we need a basic mechanism to evaluate the performance of our engine. For that we will:\n",
    "\n",
    "- split the data into train and test sets\n",
    "- introduce a performance criterion\n",
    "- write an `evaluate` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation: split ratings into train and test sets\n",
    "\n",
    "This subsection will generate training and testing sets for evaluation. \n",
    "\n",
    "- take a smaller sample from the full 1M dataset for speed reasons;\n",
    "- make sure that we have at least 2 ratings per user in that subset;\n",
    "- split the result into training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:53.829465Z",
     "start_time": "2019-05-23T04:26:53.464500Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 10)\n",
      "3698\n",
      "2275\n"
     ]
    }
   ],
   "source": [
    "# let's work with a smaller subset for speed reasons\n",
    "movielens = movielens.ix[np.random.choice(movielens.index, size=10000, replace=False)]\n",
    "print (movielens.shape)\n",
    "print (movielens.user_id.nunique())\n",
    "print (movielens.movie_id.nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T05:54:24.358314Z",
     "start_time": "2019-05-23T05:54:24.119723Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8442, 11)\n"
     ]
    }
   ],
   "source": [
    "user_ids_larger_1 = pd.value_counts(movielens.user_id, sort=False) > 1\n",
    "user_ids_larger_1 = user_ids_larger_1[user_ids_larger_1].index\n",
    "\n",
    "movielens = movielens.select(lambda l: movielens.loc[l, 'user_id'] in user_ids_larger_1)\n",
    "print (movielens.shape)\n",
    "assert np.all(movielens.user_id.value_counts() > 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now generate train and test subsets by marking 20% of each users's ratings, using groupby and apply."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:59.037576Z",
     "start_time": "2019-05-23T04:26:54.023982Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8442, 11)\n",
      "(5801, 11)\n",
      "(2641, 11)\n"
     ]
    }
   ],
   "source": [
    "def assign_to_set(df):\n",
    "    sampled_ids = np.random.choice(df.index,\n",
    "                                   size=np.int64(np.ceil(df.index.size * 0.2)),\n",
    "                                   replace=False)\n",
    "    df.ix[sampled_ids, 'for_testing'] = True\n",
    "    return df\n",
    "\n",
    "movielens['for_testing'] = False\n",
    "grouped = movielens.groupby('user_id', group_keys=False).apply(assign_to_set)\n",
    "movielens_train = movielens[grouped.for_testing == False]\n",
    "movielens_test = movielens[grouped.for_testing == True]\n",
    "print (movielens.shape)\n",
    "print (movielens_train.shape)\n",
    "print (movielens_test.shape)\n",
    "assert len(movielens_train.index & movielens_test.index) == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store these two sets in text files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:59.303964Z",
     "start_time": "2019-05-23T04:26:59.042272Z"
    }
   },
   "outputs": [],
   "source": [
    "movielens_train.to_csv('data/my_generated_movielens_train.csv')\n",
    "movielens_test.to_csv('data/my_generated_movielens_test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation: performance criterion\n",
    "\n",
    "Performance evaluation of recommendation systems is an entire topic all in\n",
    "itself. Some of the options include:\n",
    "\n",
    "- RMSE: $\\sqrt{\\frac{\\sum(\\hat y - y)^2}{n}}$\n",
    "- Precision / Recall / F-scores\n",
    "- ROC curves\n",
    "- Cost curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:59.316998Z",
     "start_time": "2019-05-23T04:26:59.309981Z"
    }
   },
   "outputs": [],
   "source": [
    "def compute_rmse(y_pred, y_true):\n",
    "    \"\"\" Compute Root Mean Squared Error. \"\"\"\n",
    "    \n",
    "    return np.sqrt(np.mean(np.power(y_pred - y_true, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation: the 'evaluate' method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:59.347105Z",
     "start_time": "2019-05-23T04:26:59.320008Z"
    }
   },
   "outputs": [],
   "source": [
    "def evaluate(estimate_f):\n",
    "    \"\"\" RMSE-based predictive performance evaluation with pandas. \"\"\"\n",
    "    \n",
    "    ids_to_estimate = zip(movielens_test.user_id, movielens_test.movie_id)\n",
    "    estimated = np.array([estimate_f(u,i) for (u,i) in ids_to_estimate])\n",
    "    real = movielens_test.rating.values\n",
    "    return compute_rmse(estimated, real)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:59.376156Z",
     "start_time": "2019-05-23T04:26:59.350136Z"
    }
   },
   "outputs": [],
   "source": [
    "def my_estimate_function(user_id, movie_id):\n",
    "    return 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:59.410247Z",
     "start_time": "2019-05-23T04:26:59.379680Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for my estimate function: 1.2571642479137273\n"
     ]
    }
   ],
   "source": [
    "print ('RMSE for my estimate function: %s' % evaluate(my_estimate_function))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:59.442331Z",
     "start_time": "2019-05-23T04:26:59.414257Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "overall_mean =  3.5264609550077575\n",
      "RMSE for my_estimate_func2: 1.129753606013628\n"
     ]
    }
   ],
   "source": [
    "overall_mean = movielens_train.rating.mean()\n",
    "print('overall_mean = ', overall_mean)\n",
    "def my_estimate_func2(user_id, movie_id):\n",
    "    return overall_mean\n",
    "print ('RMSE for my_estimate_func2: %s' % evaluate(my_estimate_func2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:26:59.465414Z",
     "start_time": "2019-05-23T04:26:59.445339Z"
    }
   },
   "outputs": [],
   "source": [
    "user_info = users.set_index('user_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Well-known Solutions to the Recommendation Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content-based filtering\n",
    "\n",
    "*Recommend based on the user's rating history.* \n",
    "\n",
    "Generic expression (notice how this is kind of a 'row-based' approach):\n",
    "\n",
    "$$ \n",
    "\\newcommand{\\aggr}{\\mathop{\\rm aggr}\\nolimits}\n",
    "r_{u,i} = \\aggr_{i' \\in I(u)} [r_{u,i'}]\n",
    "$$\n",
    "\n",
    "\n",
    "A simple example using the mean as an aggregation function:\n",
    "\n",
    "$$ \n",
    "r_{u,i} = \\bar r_u = \\frac{\\sum_{i' \\in I(u)} r_{u,i'}}{|I(u)|} \n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collaborative filtering\n",
    "\n",
    "*Recommend based on other user's rating histories.* \n",
    "\n",
    "Generic expression (notice how this is kind of a 'col-based' approach):\n",
    "\n",
    "$$ \n",
    "\\newcommand{\\aggr}{\\mathop{\\rm aggr}\\nolimits}\n",
    "r_{u,i} = \\aggr_{u' \\in U(i)} [r_{u',i}] \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple example using the mean as an aggregation function:\n",
    "\n",
    "$$ \n",
    "r_{u,i} = \\bar r_i = \\frac{\\sum_{u' \\in U(i)} r_{u',i}}{|U(i)|} \n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hybrid solutions\n",
    "\n",
    "The literature has lots of examples of systems that try to combine the strengths\n",
    "of the two main approaches. This can be done in a number of ways:\n",
    "\n",
    "- Combine the predictions of a content-based system and a collaborative system.\n",
    "- Incorporate content-based techniques into a collaborative approach.\n",
    "- Incorporarte collaborative techniques into a content-based approach.\n",
    "- Unifying model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenges\n",
    "\n",
    "#### Availability of item metadata\n",
    "\n",
    "Content-based techniques are limited by the amount of metadata that is available\n",
    "to describe an item. There are domains in which feature extraction methods are\n",
    "expensive or time consuming, e.g., processing multimedia data such as graphics,\n",
    "audio/video streams. In the context of grocery items for example, it's often the\n",
    "case that item information is only partial or completely missing. Examples\n",
    "include:\n",
    "\n",
    "- Ingredients\n",
    "- Nutrition facts\n",
    "- Brand\n",
    "- Description\n",
    "- County of origin\n",
    "\n",
    "#### New user problem\n",
    "\n",
    "A user has to have rated a sufficient number of items before a recommender\n",
    "system can have a good idea of what their preferences are. In a content-based\n",
    "system, the aggregation function needs ratings to aggregate.\n",
    "\n",
    "#### New item problem\n",
    "\n",
    "Collaborative filters rely on an item being rated by many users to compute\n",
    "aggregates of those ratings. Think of this as the exact counterpart of the new\n",
    "user problem for content-based systems.\n",
    "\n",
    "#### Data sparsity\n",
    "\n",
    "When looking at the more general versions of content-based and collaborative\n",
    "systems, the success of the recommender system depends on the availability of a\n",
    "critical mass of user/item iteractions. We get a first glance at the data\n",
    "sparsity problem by quantifying the ratio of existing ratings vs $|U|x|I|$. A\n",
    "highly sparse matrix of interactions makes it difficult to compute similarities\n",
    "between users and items. As an example, for a user whose tastes are unusual\n",
    "compared to the rest of the population, there will not be any other users who\n",
    "are particularly similar, leading to poor recommendations.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal reco engine v1.0: simple mean ratings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content-based filtering using mean ratings\n",
    "\n",
    "With this table-like representation of the ratings data, a basic content-based\n",
    "filter becomes a one-liner function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:01.314289Z",
     "start_time": "2019-05-23T04:26:59.468455Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for estimate1: 1.2684467037493927\n"
     ]
    }
   ],
   "source": [
    "def content_mean(user_id, movie_id):\n",
    "    \"\"\" Simple content-filtering based on mean ratings. \"\"\"\n",
    "    \n",
    "    user_condition = movielens_train.user_id == user_id\n",
    "    return movielens_train.loc[user_condition, 'rating'].mean()\n",
    "\n",
    "print ('RMSE for estimate1: %s' % evaluate(content_mean))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple collaborative filtering using mean ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:04.882742Z",
     "start_time": "2019-05-23T04:27:01.316333Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0\n",
      "RMSE for my estimate function: 1.1454717831225134\n"
     ]
    }
   ],
   "source": [
    "# write an 'estimate' function that computes the mean rating of a particular user\n",
    "def collab_mean(user_id, movie_id):\n",
    "    item_condition = movielens_train.movie_id == movie_id\n",
    "    user_condition = movielens_train.user_id != user_id\n",
    "    ratings_by_others = movielens_train.loc[item_condition & user_condition, 'rating']\n",
    "    if ratings_by_others.empty:\n",
    "        return 3.0\n",
    "    else:\n",
    "        return ratings_by_others.mean()\n",
    "\n",
    "    \n",
    "# try it out for a user_id, movie_id pair\n",
    "print(collab_mean(4653, 2648))\n",
    "print ('RMSE for my estimate function: %s' % evaluate(collab_mean))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More formulas!\n",
    "\n",
    "Here are some basic ways in which we can generalize the simple mean-based algorithms we discussed before. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generalizations of the aggregation function for content-based filtering: incorporating similarities\n",
    "\n",
    "Possibly incorporating metadata about items, which makes the term 'content' make more sense now.\n",
    "\n",
    "$$ r_{u,i} = k \\sum_{i' \\in I(u)} sim(i, i') \\; r_{u,i'} $$\n",
    "\n",
    "$$ r_{u,i} = \\bar r_u + k \\sum_{i' \\in I(u)} sim(i, i') \\; (r_{u,i'} - \\bar r_u) $$\n",
    "\n",
    "Here $k$ is a normalizing factor,\n",
    "\n",
    "$$ k = \\frac{1}{\\sum_{i' \\in I(u)} |sim(i,i')|} $$\n",
    "\n",
    "and $\\bar r_u$ is the average rating of user u:\n",
    "\n",
    "$$ \\bar r_u = \\frac{\\sum_{i \\in I(u)} r_{u,i}}{|I(u)|} $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generalizations of the aggregation function for collaborative filtering: incorporating similarities\n",
    "\n",
    "Possibly incorporating metadata about users.\n",
    "\n",
    "$$ r_{u,i} = k \\sum_{u' \\in U(i)} sim(u, u') \\; r_{u',i} $$\n",
    "\n",
    "$$ r_{u,i} = \\bar r_u + k \\sum_{u' \\in U(i)} sim(u, u') \\; (r_{u',i} - \\bar r_u) $$\n",
    "\n",
    "Here $k$ is a normalizing factor,\n",
    "\n",
    "$$ k = \\frac{1}{\\sum_{u' \\in U(i)} |sim(u,u')|} $$\n",
    "\n",
    "and $\\bar r_u$ is the average rating of user u:\n",
    "\n",
    "$$ \\bar r_u = \\frac{\\sum_{i \\in I(u)} r_{u,i}}{|I(u)|} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregation in pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Groupby\n",
    "\n",
    "The idea of groupby is that of *split-apply-combine*:\n",
    "\n",
    "- split data in an object according to a given key;\n",
    "- apply a function to each subset;\n",
    "- combine results into a new object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:04.899785Z",
     "start_time": "2019-05-23T04:27:04.885861Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender\n",
       "F    3.588\n",
       "M    3.507\n",
       "Name: rating, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens_train.groupby('gender')['rating'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:04.965987Z",
     "start_time": "2019-05-23T04:27:04.902827Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender  age\n",
       "F       1      3.605\n",
       "        18     3.429\n",
       "        25     3.609\n",
       "        35     3.648\n",
       "        45     3.537\n",
       "        50     3.671\n",
       "        56     4.069\n",
       "M       1      3.384\n",
       "        18     3.438\n",
       "        25     3.430\n",
       "        35     3.614\n",
       "        45     3.555\n",
       "        50     3.762\n",
       "        56     3.704\n",
       "Name: rating, dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens_train.groupby(['gender', 'age'])['rating'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pivoting\n",
    "\n",
    "Let's start with a simple pivoting example that does not involve any\n",
    "aggregation. We can extract a ratings matrix as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:05.229660Z",
     "start_time": "2019-05-23T04:27:04.968974Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>movie_id</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>14</th>\n",
       "      <th>...</th>\n",
       "      <th>3930</th>\n",
       "      <th>3932</th>\n",
       "      <th>3935</th>\n",
       "      <th>3938</th>\n",
       "      <th>3943</th>\n",
       "      <th>3945</th>\n",
       "      <th>3948</th>\n",
       "      <th>3949</th>\n",
       "      <th>3950</th>\n",
       "      <th>3952</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 1934 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "movie_id  1     2     3     4     5     6     7     10    11    14    ...   \\\n",
       "user_id                                                               ...    \n",
       "8          NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...    \n",
       "10         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...    \n",
       "11         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...    \n",
       "\n",
       "movie_id  3930  3932  3935  3938  3943  3945  3948  3949  3950  3952  \n",
       "user_id                                                               \n",
       "8          NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "10         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "11         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "\n",
       "[3 rows x 1934 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# transform the ratings frame into a ratings matrix\n",
    "ratings_mtx_df = movielens_train.pivot_table(values='rating',\n",
    "                                             index='user_id',\n",
    "                                             columns='movie_id')\n",
    "ratings_mtx_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:05.247707Z",
     "start_time": "2019-05-23T04:27:05.231664Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>movie_id</th>\n",
       "      <th>1196</th>\n",
       "      <th>1197</th>\n",
       "      <th>1198</th>\n",
       "      <th>1199</th>\n",
       "      <th>1200</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "movie_id  1196  1197  1198  1199  1200\n",
       "user_id                               \n",
       "11         NaN   NaN   NaN   NaN   NaN\n",
       "13         5.0   NaN   NaN   NaN   NaN\n",
       "15         NaN   NaN   NaN   NaN   NaN"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# grab another subsquare of the ratings matrix to actually diplay some real entries!\n",
    "ratings_mtx_df.loc[11:16, 1196:1200]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The more interesting case with `pivot_table` is as an interface to\n",
    "`groupby`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:05.292826Z",
     "start_time": "2019-05-23T04:27:05.250917Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gender</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.605</td>\n",
       "      <td>3.384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3.429</td>\n",
       "      <td>3.438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>3.609</td>\n",
       "      <td>3.430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>3.648</td>\n",
       "      <td>3.614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>3.537</td>\n",
       "      <td>3.555</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>3.671</td>\n",
       "      <td>3.762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>4.069</td>\n",
       "      <td>3.704</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "gender      F      M\n",
       "age                 \n",
       "1       3.605  3.384\n",
       "18      3.429  3.438\n",
       "25      3.609  3.430\n",
       "35      3.648  3.614\n",
       "45      3.537  3.555\n",
       "50      3.671  3.762\n",
       "56      4.069  3.704"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens_train.pivot_table(values='rating', index='age', columns='gender', aggfunc='mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass in a list of functions, such as `[np.mean, np.std]`, to compute mean ratings and a measure of disagreement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:05.360004Z",
     "start_time": "2019-05-23T04:27:05.295868Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">mean</th>\n",
       "      <th colspan=\"2\" halign=\"left\">std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "      <th>F</th>\n",
       "      <th>M</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.605</td>\n",
       "      <td>3.384</td>\n",
       "      <td>1.285</td>\n",
       "      <td>1.303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3.429</td>\n",
       "      <td>3.438</td>\n",
       "      <td>1.200</td>\n",
       "      <td>1.149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>3.609</td>\n",
       "      <td>3.430</td>\n",
       "      <td>1.081</td>\n",
       "      <td>1.142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>3.648</td>\n",
       "      <td>3.614</td>\n",
       "      <td>1.019</td>\n",
       "      <td>1.039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>3.537</td>\n",
       "      <td>3.555</td>\n",
       "      <td>0.998</td>\n",
       "      <td>1.063</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>3.671</td>\n",
       "      <td>3.762</td>\n",
       "      <td>1.138</td>\n",
       "      <td>1.037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>4.069</td>\n",
       "      <td>3.704</td>\n",
       "      <td>0.961</td>\n",
       "      <td>1.079</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         mean           std       \n",
       "gender      F      M      F      M\n",
       "age                               \n",
       "1       3.605  3.384  1.285  1.303\n",
       "18      3.429  3.438  1.200  1.149\n",
       "25      3.609  3.430  1.081  1.142\n",
       "35      3.648  3.614  1.019  1.039\n",
       "45      3.537  3.555  0.998  1.063\n",
       "50      3.671  3.762  1.138  1.037\n",
       "56      4.069  3.704  0.961  1.079"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movielens_train.pivot_table(values='rating', index='age', columns='gender', aggfunc=[np.mean, np.std])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal reco engine v1.1: implicit sim functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to need a user index from the users portion of the dataset. This will allow us to retrieve information given a specific user_id in a more convenient way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:05.378053Z",
     "start_time": "2019-05-23T04:27:05.363014Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>55117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>7</td>\n",
       "      <td>02460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>55455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        gender  age  occupation    zip\n",
       "user_id                               \n",
       "1            F    1          10  48067\n",
       "2            M   56          16  70072\n",
       "3            M   25          15  55117\n",
       "4            M   45           7  02460\n",
       "5            M   25          20  55455"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_info = users.set_index('user_id')\n",
    "user_info.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this in hand, we can now ask what the gender of a particular user_id is like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:05.400111Z",
     "start_time": "2019-05-23T04:27:05.381060Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'M'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_id = 3\n",
    "user_info.loc[user_id, 'gender']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collaborative-based filtering using implicit sim functions\n",
    "\n",
    "Using the pandas aggregation framework we will build a collaborative filter that estimates ratings using an implicit `sim(u,u')` function to compare different users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:41.236587Z",
     "start_time": "2019-05-23T04:27:05.403119Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for collab_gender: 1.1953061054016687\n"
     ]
    }
   ],
   "source": [
    "def collab_gender(user_id, movie_id):\n",
    "    \"\"\" Collaborative filtering using an implicit sim(u,u') based on gender. \"\"\"\n",
    "    \n",
    "    user_condition = movielens_train.user_id != user_id\n",
    "    movie_condition = movielens_train.movie_id == movie_id\n",
    "    ratings_by_others = movielens_train.loc[user_condition & movie_condition]\n",
    "    if ratings_by_others.empty: \n",
    "        return 3.0\n",
    "    \n",
    "    means_by_gender = ratings_by_others.pivot_table('rating', index='movie_id', columns='gender')\n",
    "    user_gender = user_info.ix[user_id, 'gender']\n",
    "    if user_gender in means_by_gender.columns: \n",
    "        return means_by_gender.ix[movie_id, user_gender]\n",
    "    else:\n",
    "        return means_by_gender.ix[movie_id].mean()\n",
    "\n",
    "print ('RMSE for collab_gender: %s' % evaluate(collab_gender))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point it seems worthwhile to write a `learn` function to pre-compute whatever datastructures we need at estimation time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:41.785038Z",
     "start_time": "2019-05-23T04:27:41.238592Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for CollabGenderReco: 1.1953061054016687\n"
     ]
    }
   ],
   "source": [
    "class CollabGenderReco:\n",
    "    \"\"\" Collaborative filtering using an implicit sim(u,u'). \"\"\"\n",
    "\n",
    "    def learn(self):\n",
    "        \"\"\" Prepare datastructures for estimation. \"\"\"\n",
    "        \n",
    "        self.means_by_gender = movielens_train.pivot_table('rating', index='movie_id', columns='gender')\n",
    "\n",
    "    def estimate(self, user_id, movie_id):\n",
    "        \"\"\" Mean ratings by other users of the same gender. \"\"\"\n",
    "        \n",
    "        if movie_id not in self.means_by_gender.index: \n",
    "            return 3.0\n",
    "        \n",
    "        user_gender = user_info.ix[user_id, 'gender']\n",
    "        if ~np.isnan(self.means_by_gender.ix[movie_id, user_gender]):\n",
    "            return self.means_by_gender.ix[movie_id, user_gender]\n",
    "        else:\n",
    "            return self.means_by_gender.ix[movie_id].mean()\n",
    "\n",
    "reco = CollabGenderReco()\n",
    "reco.learn()\n",
    "print ('RMSE for CollabGenderReco: %s' % evaluate(reco.estimate))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:42.197196Z",
     "start_time": "2019-05-23T04:27:41.789079Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for my estimate function: 1.2151497127725077\n"
     ]
    }
   ],
   "source": [
    "class CollabAgeReco:\n",
    "    def learn(self):\n",
    "        self.mean_by_age = movielens_train.pivot_table(values='rating', index='movie_id', columns='age')\n",
    "        \n",
    "    def estimate(self, user_id, movie_id):\n",
    "        if movie_id not in self.mean_by_age.index:\n",
    "            return 3.0\n",
    "        user_age = user_info.loc[user_id, 'age']\n",
    "        if np.isnan(self.mean_by_age.loc[movie_id, user_age]):\n",
    "            return self.mean_by_age.loc[movie_id].mean()\n",
    "        else:\n",
    "            return self.mean_by_age.loc[movie_id, user_age] \n",
    "reco = CollabAgeReco()\n",
    "reco.learn()\n",
    "print ('RMSE for my estimate function: %s' % evaluate(reco.estimate))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:43.156729Z",
     "start_time": "2019-05-23T04:27:42.201609Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for my estimate function: 1.1457982962042659\n"
     ]
    }
   ],
   "source": [
    "class CollabZipReco:\n",
    "    def learn(self):\n",
    "        self.mean_by_zip = movielens_train.pivot_table(values='rating', index='movie_id', columns='zip')\n",
    "        \n",
    "    def estimate(self, user_id, movie_id):\n",
    "        if movie_id not in self.mean_by_zip.index:\n",
    "            return 3.0\n",
    "        user_zip = user_info.loc[user_id, 'zip']\n",
    "        if np.isnan(self.mean_by_zip.loc[movie_id, user_zip]):\n",
    "            return self.mean_by_zip.loc[movie_id].mean()\n",
    "        else:\n",
    "            return self.mean_by_zip.loc[movie_id, user_zip] \n",
    "reco = CollabZipReco()\n",
    "reco.learn()\n",
    "print ('RMSE for my estimate function: %s' % evaluate(reco.estimate))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:43.727225Z",
     "start_time": "2019-05-23T04:27:43.160683Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for my estimate function: 1.2018899397294673\n"
     ]
    }
   ],
   "source": [
    "class CollabOccupationReco:\n",
    "    def learn(self):\n",
    "        self.mean_by_occupation = movielens_train.pivot_table(values='rating', index='movie_id', columns='occupation')\n",
    "        \n",
    "    def estimate(self, user_id, movie_id):\n",
    "        if movie_id not in self.mean_by_occupation.index:\n",
    "            return 3.0\n",
    "        user_occupation = user_info.loc[user_id, 'occupation']\n",
    "        if np.isnan(self.mean_by_occupation.loc[movie_id, user_occupation]):\n",
    "            return self.mean_by_occupation.loc[movie_id].mean()\n",
    "        else:\n",
    "            return self.mean_by_occupation.loc[movie_id, user_occupation] \n",
    "reco = CollabOccupationReco()\n",
    "reco.learn()\n",
    "print ('RMSE for my estimate function: %s' % evaluate(reco.estimate))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal reco engine v1.2: custom similarity functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A few similarity functions\n",
    "\n",
    "These were all written to operate on two pandas Series, each one representing the rating history of two different users. You can also apply them to any two feature vectors that describe users or items. In all cases, the higher the return value, the more similar two Series are. You might need to add checks for edge cases, such as divisions by zero, etc.\n",
    "\n",
    "- Euclidean 'similarity'\n",
    "\n",
    "$$ sim(x,y) = \\frac{1}{1 + \\sqrt{\\sum (x - y)^2}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:43.736273Z",
     "start_time": "2019-05-23T04:27:43.732200Z"
    }
   },
   "outputs": [],
   "source": [
    "def euclidean(s1, s2):\n",
    "    \"\"\"Take two pd.Series objects and return their euclidean 'similarity'.\"\"\"\n",
    "    diff = s1 - s2\n",
    "    return 1 / (1 + np.sqrt(np.sum(diff ** 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Cosine similarity\n",
    "\n",
    "$$ sim(x,y) = \\frac{(x . y)}{\\sqrt{(x . x) (y . y)}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:43.766326Z",
     "start_time": "2019-05-23T04:27:43.741221Z"
    }
   },
   "outputs": [],
   "source": [
    "def cosine(s1, s2):\n",
    "    \"\"\"Take two pd.Series objects and return their cosine similarity.\"\"\"\n",
    "    return np.sum(s1 * s2) / np.sqrt(np.sum(s1 ** 2) * np.sum(s2 ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pearson correlation\n",
    "\n",
    "$$ sim(x,y) = \\frac{(x - \\bar x).(y - \\bar y)}{\\sqrt{(x - \\bar x).(x - \\bar x) * (y - \\bar y)(y - \\bar y)}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:43.804389Z",
     "start_time": "2019-05-23T04:27:43.772303Z"
    }
   },
   "outputs": [],
   "source": [
    "def pearson(s1, s2):\n",
    "    \"\"\"Take two pd.Series objects and return a pearson correlation.\"\"\"\n",
    "    s1_c = s1 - s1.mean()\n",
    "    s2_c = s2 - s2.mean()\n",
    "    return np.sum(s1_c * s2_c) / np.sqrt(np.sum(s1_c ** 2) * np.sum(s2_c ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Jaccard similarity\n",
    "\n",
    "$$ sim(x,y) = \\frac{(x . y)}{(x . x) + (y . y) - (x . y)} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:27:43.839480Z",
     "start_time": "2019-05-23T04:27:43.808398Z"
    }
   },
   "outputs": [],
   "source": [
    "def jaccard(s1, s2):\n",
    "    dotp = np.sum(s1 * s2)\n",
    "    return dotp / (np.sum(s1 ** 2) + np.sum(s2 ** 2) - dotp)\n",
    "\n",
    "def binjaccard(s1, s2):\n",
    "    dotp = (s1.index & s2.index).size\n",
    "    return dotp / (s1.sum() + s2.sum() - dotp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collaborative-based filtering using custom sim functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:28:21.001282Z",
     "start_time": "2019-05-23T04:27:43.843492Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for CollabPearsonReco: 1.149746741087946\n"
     ]
    }
   ],
   "source": [
    "class CollabPearsonReco:\n",
    "    \"\"\" Collaborative filtering using a custom sim(u,u'). \"\"\"\n",
    "\n",
    "    def learn(self):\n",
    "        \"\"\" Prepare datastructures for estimation. \"\"\"\n",
    "        \n",
    "        #self.all_user_profiles = movielens.pivot_table('rating', index='movie_id', columns='user_id')\n",
    "        self.all_user_profiles = movielens_train.pivot_table('rating', index='movie_id', columns='user_id')\n",
    "\n",
    "    def estimate(self, user_id, movie_id):\n",
    "        \"\"\" Ratings weighted by correlation similarity. \"\"\"\n",
    "        \n",
    "        user_condition = movielens_train.user_id != user_id\n",
    "        movie_condition = movielens_train.movie_id == movie_id\n",
    "        ratings_by_others = movielens_train.loc[user_condition & movie_condition]\n",
    "        if ratings_by_others.empty: \n",
    "            return 3.0\n",
    "        \n",
    "        ratings_by_others.set_index('user_id', inplace=True)\n",
    "        their_ids = ratings_by_others.index\n",
    "        their_ratings = ratings_by_others.rating\n",
    "        their_profiles = self.all_user_profiles[their_ids]\n",
    "        user_profile = self.all_user_profiles[user_id]\n",
    "        sims = their_profiles.apply(lambda profile: pearson(profile, user_profile), axis=0)\n",
    "        ratings_sims = pd.DataFrame({'sim': sims, 'rating': their_ratings})\n",
    "        ratings_sims = ratings_sims[ratings_sims.sim > 0]\n",
    "        if ratings_sims.empty:\n",
    "            return their_ratings.mean()\n",
    "        else:\n",
    "            return np.average(ratings_sims.rating, weights=ratings_sims.sim)\n",
    "        \n",
    "reco = CollabPearsonReco()\n",
    "reco.learn()\n",
    "print ('RMSE for CollabPearsonReco: %s' % evaluate(reco.estimate))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:28:21.016287Z",
     "start_time": "2019-05-23T04:28:21.005273Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "overall_mean =  3.5264609550077575\n",
      "RMSE for my_estimate_func2: 1.129753606013628\n"
     ]
    }
   ],
   "source": [
    "overall_mean = movielens_train.rating.mean()\n",
    "print('overall_mean = ', overall_mean)\n",
    "def my_estimate_func2(user_id, movie_id):\n",
    "    return overall_mean\n",
    "print ('RMSE for my_estimate_func2: %s' % evaluate(my_estimate_func2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:28:21.053385Z",
     "start_time": "2019-05-23T04:28:21.020298Z"
    }
   },
   "outputs": [],
   "source": [
    "movielens=movielens_train.append(movielens_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:28:57.443358Z",
     "start_time": "2019-05-23T04:28:21.056656Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for CollabPearsonReco: 1.1520936099301031\n"
     ]
    }
   ],
   "source": [
    "def pearson(s1, s2):\n",
    "    \"\"\"Take two pd.Series objects and return a pearson correlation.\"\"\"\n",
    "    s1_c = s1 - s1.mean()\n",
    "    s2_c = s2 - s2.mean()\n",
    "    return np.sum(s1_c * s2_c) / np.sqrt(np.sum(s1_c ** 2) * np.sum(s2_c ** 2))\n",
    "\n",
    "class CollabPearsonReco:\n",
    "    def learn(self):\n",
    "        self.all_user_profiles = movielens_train.pivot_table(values='rating', index='movie_id', columns='user_id')\n",
    "        #self.all_user_profiles = movielens.pivot_table(values='rating', index='movie_id', columns='user_id')\n",
    "        \n",
    "    def estimate(self, user_id, movie_id):\n",
    "        rating_by_others = movielens_train.loc[(movielens_train.user_id!=user_id) & (movielens_train.movie_id==movie_id)]\n",
    "        if rating_by_others.empty:\n",
    "            return overall_mean\n",
    "        \n",
    "        rating_by_others.set_index('user_id', inplace=True)\n",
    "        their_id = rating_by_others.index\n",
    "        their_rating = rating_by_others.rating\n",
    "        their_profiles = self.all_user_profiles[their_id]\n",
    "        sim_score = their_profiles.apply(lambda x: pearson(x, self.all_user_profiles[user_id]), axis=0)\n",
    "        df = pd.DataFrame({'sim_score':sim_score, 'rating':their_rating})\n",
    "        df = df[df.sim_score>0]\n",
    "        if df.sim_score.sum()==0:\n",
    "            return their_rating.mean()\n",
    "        else:\n",
    "            return np.average(df.rating, weights=df.sim_score)\n",
    "\n",
    "reco = CollabPearsonReco()\n",
    "reco.learn()\n",
    "print ('RMSE for CollabPearsonReco: %s' % evaluate(reco.estimate))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:29:19.036877Z",
     "start_time": "2019-05-23T04:28:57.446419Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for CollabEuclideanReco: 1.1457140553803442\n"
     ]
    }
   ],
   "source": [
    "def euclidean(s1, s2):\n",
    "    \"\"\"Take two pd.Series objects and return their euclidean 'similarity'.\"\"\"\n",
    "    diff = s1 - s2\n",
    "    return 1 / (1 + np.sqrt(np.sum(diff ** 2)))\n",
    "\n",
    "class CollabEuclideanReco:\n",
    "    def learn(self):\n",
    "        self.all_user_profiles = movielens_train.pivot_table(values='rating', index='movie_id', columns='user_id')\n",
    "        #self.all_user_profiles = movielens.pivot_table(values='rating', index='movie_id', columns='user_id')\n",
    "        \n",
    "    def estimate(self, user_id, movie_id):\n",
    "        rating_by_others = movielens_train.loc[(movielens_train.user_id!=user_id) & (movielens_train.movie_id==movie_id)]\n",
    "        if rating_by_others.empty:\n",
    "            return 3.0\n",
    "        \n",
    "        rating_by_others.set_index('user_id', inplace=True)\n",
    "        their_id = rating_by_others.index\n",
    "        their_rating = rating_by_others.rating\n",
    "        their_profiles = self.all_user_profiles[their_id]\n",
    "        sim_score = their_profiles.apply(lambda x: euclidean(x, self.all_user_profiles[user_id]), axis=0)\n",
    "        df = pd.DataFrame({'sim_score':sim_score, 'rating':their_rating})\n",
    "        df = df[df.sim_score>0]\n",
    "        if df.sim_score.sum()==0:\n",
    "            return their_rating.mean()\n",
    "        else:\n",
    "            return np.average(df.rating, weights=df.sim_score)\n",
    "\n",
    "reco = CollabEuclideanReco()\n",
    "reco.learn()\n",
    "print ('RMSE for CollabEuclideanReco: %s' % evaluate(reco.estimate))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-23T04:29:49.426343Z",
     "start_time": "2019-05-23T04:29:19.041886Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for CollabCosineReco: 1.1518874729220447\n"
     ]
    }
   ],
   "source": [
    "def cosine(s1, s2):\n",
    "    \"\"\"Take two pd.Series objects and return their cosine similarity.\"\"\"\n",
    "    return np.sum(s1 * s2) / np.sqrt(np.sum(s1 ** 2) * np.sum(s2 ** 2))\n",
    "\n",
    "class CollabCosineReco:\n",
    "    def learn(self):\n",
    "        self.all_user_profiles = movielens_train.pivot_table(values='rating', index='movie_id', columns='user_id')\n",
    "        #self.all_user_profiles = movielens.pivot_table(values='rating', index='movie_id', columns='user_id')\n",
    "        \n",
    "    def estimate(self, user_id, movie_id):\n",
    "        rating_by_others = movielens_train.loc[(movielens_train.user_id!=user_id) & (movielens_train.movie_id==movie_id)]\n",
    "        if rating_by_others.empty:\n",
    "            return 3.0\n",
    "        \n",
    "        rating_by_others.set_index('user_id', inplace=True)\n",
    "        their_id = rating_by_others.index\n",
    "        their_rating = rating_by_others.rating\n",
    "        their_profiles = self.all_user_profiles[their_id]\n",
    "        sim_score = their_profiles.apply(lambda x: cosine(x, self.all_user_profiles[user_id]), axis=0)\n",
    "        df = pd.DataFrame({'sim_score':sim_score, 'rating':their_rating})\n",
    "        df = df[df.sim_score>0]\n",
    "        if df.sim_score.sum()==0:\n",
    "            return their_rating.mean()\n",
    "        else:\n",
    "            return np.average(df.rating, weights=df.sim_score)\n",
    "\n",
    "reco = CollabCosineReco()\n",
    "reco.learn()\n",
    "print ('RMSE for CollabCosineReco: %s' % evaluate(reco.estimate))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References and further reading\n",
    "\n",
    "- Goldberg, D., D. Nichols, B. M. Oki, and D. Terry. “Using Collaborative Filtering to Weave an Information Tapestry.” Communications of the ACM 35, no. 12 (1992): 61–70.\n",
    "- Resnick, Paul, and Hal R. Varian. “Recommender Systems.” Commun. ACM 40, no. 3 (March 1997): 56–58. doi:10.1145/245108.245121.\n",
    "- Adomavicius, Gediminas, and Alexander Tuzhilin. “Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions.” IEEE Transactions on Knowledge and Data Engineering 17, no. 6 (2005): 734–749. doi:http://doi.ieeecomputersociety.org/10.1109/TKDE.2005.99.\n",
    "- Adomavicius, Gediminas, Ramesh Sankaranarayanan, Shahana Sen, and Alexander Tuzhilin. “Incorporating Contextual Information in Recommender Systems Using a Multidimensional Approach.” ACM Trans. Inf. Syst. 23, no. 1 (2005): 103–145. doi:10.1145/1055709.1055714.\n",
    "- Koren, Y., R. Bell, and C. Volinsky. “Matrix Factorization Techniques for Recommender Systems.” Computer 42, no. 8 (2009): 30–37.\n",
    "- William Wesley McKinney. Python for Data Analysis. O’Reilly, 2012.\n",
    "- Toby Segaran. Programming Collective Intelligence. O’Reilly, 2007.\n",
    "- Zhou, Tao, Zoltan Kuscsik, Jian-Guo Liu, Matus Medo, Joseph R Wakeling, and Yi-Cheng Zhang. “Solving the Apparent Diversity-accuracy Dilemma of Recommender Systems.” arXiv:0808.2670 (August 19, 2008). doi:10.1073/pnas.1000488107.\n",
    "- Shani, G., D. Heckerman, and R. I Brafman. “An MDP-based Recommender System.” Journal of Machine Learning Research 6, no. 2 (2006): 1265.\n",
    "- Joseph A. Konstan, John Riedl. \"Deconstructing Recommender Systems.\" IEEE Spectrum, October 2012."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}