5 - Singular Value Decomposition.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Singular Value Decomposition.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "AI8iUPuAgW34",
        "VLyHpP9GcNI8",
        "ZecYlb32aPRG",
        "7Gnr6yC9VqxW"
      ],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dean-sh/Movie-Ratings-Collaborating-Filltering/blob/master/Singular%20Value%20Decomposition.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AI8iUPuAgW34",
        "colab_type": "text"
      },
      "source": [
        "MovieLens Recommendations - Dean Shabi, Dedi Kovatch, July 2019\n",
        "=============================================\n",
        "\n",
        "## Final Project for TCDS - Technion Data Science Specialization. \n",
        "\n",
        "MovieLens data sets were collected by the GroupLens Research Project\n",
        "at the University of Minnesota.\n",
        " \n",
        "This data set consists of:\n",
        "\t* 100,000 ratings (1-5) from 943 users on 1682 movies. \n",
        "\t* Each user has rated at least 20 movies. \n",
        "        * Simple demographic info for the users (age, gender, occupation, zip)\n",
        "\n",
        "The data was collected through the MovieLens web site\n",
        "(movielens.umn.edu) during the seven-month period from September 19th, \n",
        "1997 through April 22nd, 1998. This data has been cleaned up - users\n",
        "who had less than 20 ratings or did not have complete demographic\n",
        "information were removed from this data set. Detailed descriptions of\n",
        "the data file can be found at the end of this file.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VLyHpP9GcNI8",
        "colab_type": "text"
      },
      "source": [
        "## Data Description\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "79B5aQBncSJn",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "Here are brief descriptions of the data.\n",
        "\n",
        "ml-data.tar.gz   -- Compressed tar file.  To rebuild the u data files do this:\n",
        "                gunzip ml-data.tar.gz\n",
        "                tar xvf ml-data.tar\n",
        "                mku.sh\n",
        "\n",
        "u.data     -- The full u data set, 100000 ratings by 943 users on 1682 items.\n",
        "              Each user has rated at least 20 movies.  Users and items are\n",
        "              numbered consecutively from 1.  The data is randomly\n",
        "              ordered. This is a tab separated list of \n",
        "\t         user id | item id | rating | timestamp. \n",
        "              The time stamps are unix seconds since 1/1/1970 UTC   \n",
        "\n",
        "u.info     -- The number of users, items, and ratings in the u data set.\n",
        "\n",
        "u.item     -- Information about the items (movies); this is a tab separated\n",
        "              list of\n",
        "              movie id | movie title | release date | video release date |\n",
        "              IMDb URL | unknown | Action | Adventure | Animation |\n",
        "              Children's | Comedy | Crime | Documentary | Drama | Fantasy |\n",
        "              Film-Noir | Horror | Musical | Mystery | Romance | Sci-Fi |\n",
        "              Thriller | War | Western |\n",
        "              The last 19 fields are the genres, a 1 indicates the movie\n",
        "              is of that genre, a 0 indicates it is not; movies can be in\n",
        "              several genres at once.\n",
        "              The movie ids are the ones used in the u.data data set.\n",
        "\n",
        "u.genre    -- A list of the genres.\n",
        "\n",
        "u.user     -- Demographic information about the users; this is a tab\n",
        "              separated list of\n",
        "              user id | age | gender | occupation | zip code\n",
        "              The user ids are the ones used in the u.data data set.\n",
        "\n",
        "u.occupation -- A list of the occupations.\n",
        "\n",
        "u1.base    -- The data sets u1.base and u1.test through u5.base and u5.test\n",
        "u1.test       are 80%/20% splits of the u data into training and test data.\n",
        "u2.base       Each of u1, ..., u5 have disjoint test sets; this if for\n",
        "u2.test       5 fold cross validation (where you repeat your experiment\n",
        "u3.base       with each training and test set and average the results).\n",
        "u3.test       These data sets can be generated from u.data by mku.sh.\n",
        "u4.base\n",
        "u4.test\n",
        "u5.base\n",
        "u5.test\n",
        "\n",
        "ua.base    -- The data sets ua.base, ua.test, ub.base, and ub.test\n",
        "ua.test       split the u data into a training set and a test set with\n",
        "ub.base       exactly 10 ratings per user in the test set.  The sets\n",
        "ub.test       ua.test and ub.test are disjoint.  These data sets can\n",
        "              be generated from u.data by mku.sh.\n",
        "\n",
        "allbut.pl  -- The script that generates training and test sets where\n",
        "              all but n of a users ratings are in the training data.\n",
        "\n",
        "mku.sh     -- A shell script to generate all the u data sets from u.data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZecYlb32aPRG",
        "colab_type": "text"
      },
      "source": [
        "## Imports\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "eSXwQhcbxpTA",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import collections\n",
        "import seaborn as sns\n",
        "%matplotlib inline\n",
        "# from mpl_toolkits.mplot3d import Axes3D\n",
        "from IPython import display\n",
        "from matplotlib import pyplot as plt\n",
        "import sklearn\n",
        "import sklearn.manifold\n",
        "# import tensorflow as tf\n",
        "# tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "\n",
        "# # Add some convenience functions to Pandas DataFrame.\n",
        "# pd.options.display.max_rows = 10\n",
        "# pd.options.display.float_format = '{:.3f}'.format\n",
        "\n",
        "\n",
        "\n",
        "# # Install Altair and activate its colab renderer.\n",
        "# print(\"Installing Altair...\")\n",
        "# !pip install git+git://github.com/altair-viz/altair.git\n",
        "# import altair as alt\n",
        "# alt.data_transformers.enable('default', max_rows=None)\n",
        "# alt.renderers.enable('colab')\n",
        "# print(\"Done installing Altair.\")\n",
        "\n",
        "# # Install spreadsheets and import authentication module.\n",
        "# USER_RATINGS = False\n",
        "# !pip install --upgrade -q gspread\n",
        "# from google.colab import auth\n",
        "# import gspread\n",
        "# from oauth2client.client import GoogleCredentials"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Gnr6yC9VqxW",
        "colab_type": "text"
      },
      "source": [
        "## **Importing dataset, preprocessing**\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gq9LYL4ef_Ms",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# download the MovieLens Data, and create DataFrames containing movies, users, and ratings.\n",
        "\n",
        "print(\"Downloading movielens data...\")\n",
        "import zipfile\n",
        "import urllib.request\n",
        "\n",
        "urllib.request.urlretrieve(\"http://files.grouplens.org/datasets/movielens/ml-100k.zip\", \"movielens.zip\")\n",
        "zip_ref = zipfile.ZipFile('movielens.zip', \"r\")\n",
        "zip_ref.extractall()\n",
        "print(\"Done. Dataset contains:\")\n",
        "print(zip_ref.read('ml-100k/u.info'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ny6Op3qUe3bc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Load each data set (users, movies, and ratings).\n",
        "users_cols = ['user_id', 'age', 'sex', 'occupation', 'zip_code']\n",
        "users = pd.read_csv(\n",
        "    'ml-100k/u.user', sep='|', names=users_cols, encoding='latin-1')\n",
        "\n",
        "ratings_cols = ['user_id', 'movie_id', 'rating', 'unix_timestamp']\n",
        "ratings = pd.read_csv(\n",
        "    'ml-100k/u.data', sep='\\t', names=ratings_cols, encoding='latin-1')\n",
        "\n",
        "# The movies file contains a binary feature for each genre.\n",
        "genre_cols = [\n",
        "    \"genre_unknown\", \"Action\", \"Adventure\", \"Animation\", \"Children\", \"Comedy\",\n",
        "    \"Crime\", \"Documentary\", \"Drama\", \"Fantasy\", \"Film-Noir\", \"Horror\",\n",
        "    \"Musical\", \"Mystery\", \"Romance\", \"Sci-Fi\", \"Thriller\", \"War\", \"Western\"\n",
        "]\n",
        "movies_cols = [\n",
        "    'movie_id', 'title', 'release_date', \"video_release_date\", \"imdb_url\"] + genre_cols\n",
        "\n",
        "movies = pd.read_csv(\n",
        "    'ml-100k/u.item', sep='|', names=movies_cols, encoding='latin-1')\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dXNpqJsDgeqj",
        "colab_type": "text"
      },
      "source": [
        "Some Preproccessing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xjlkpHekgXyq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Since the ids start at 1, we shift them to start at 0.\n",
        "users[\"user_id\"] = users[\"user_id\"].apply(lambda x: str(x-1))\n",
        "movies[\"movie_id\"] = movies[\"movie_id\"].apply(lambda x: str(x-1))\n",
        "movies[\"year\"] = movies['release_date'].apply(lambda x: str(x).split('-')[-1])\n",
        "ratings[\"movie_id\"] = ratings[\"movie_id\"].apply(lambda x: str(x-1))\n",
        "ratings[\"user_id\"] = ratings[\"user_id\"].apply(lambda x: str(x-1))\n",
        "ratings[\"rating\"] = ratings[\"rating\"].apply(lambda x: float(x))\n",
        "\n",
        "# Compute the number of movies to which a genre is assigned.\n",
        "genre_occurences = movies[genre_cols].sum().to_dict()\n",
        "\n",
        "# Since some movies can belong to more than one genre, we create different\n",
        "# 'genre' columns as follows:\n",
        "# - all_genres: all the active genres of the movie.\n",
        "# - genre: randomly sampled from the active genres.\n",
        "def mark_genres(movies, genres):\n",
        "    def get_random_genre(gs):\n",
        "        active = [genre for genre, g in zip(genres, gs) if g==1]\n",
        "        if len(active) == 0:\n",
        "            return 'Other'\n",
        "        return np.random.choice(active)\n",
        "    def get_all_genres(gs):\n",
        "        active = [genre for genre, g in zip(genres, gs) if g==1]\n",
        "        if len(active) == 0:\n",
        "            return 'Other'\n",
        "        return '-'.join(active)\n",
        "    movies['genre'] = [\n",
        "        get_random_genre(gs) for gs in zip(*[movies[genre] for genre in genres])]\n",
        "    movies['all_genres'] = [\n",
        "        get_all_genres(gs) for gs in zip(*[movies[genre] for genre in genres])]\n",
        "\n",
        "mark_genres(movies, genre_cols)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "apgLczpJgWcf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create one merged DataFrame containing all the movielens data.\n",
        "movielens = ratings.merge(movies, on='movie_id').merge(users, on='user_id')\n",
        "\n",
        "# Utility to split the data into training and test sets.\n",
        "def split_dataframe(df, holdout_fraction=0.1):\n",
        "    \"\"\"Splits a DataFrame into training and test sets.\n",
        "    Args:\n",
        "    df: a dataframe.\n",
        "    holdout_fraction: fraction of dataframe rows to use in the test set.\n",
        "    Returns:\n",
        "    train: dataframe for training\n",
        "    test: dataframe for testing\n",
        "    \"\"\"\n",
        "    test = df.sample(frac=holdout_fraction, replace=False)\n",
        "    train = df[~df.index.isin(test.index)]\n",
        "    return train, test"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XJMhuEXXgXdB",
        "colab_type": "code",
        "outputId": "4e49f141-5aa3-4bf6-84f2-68c491c77716",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "movies.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movie_id</th>\n",
              "      <th>title</th>\n",
              "      <th>release_date</th>\n",
              "      <th>video_release_date</th>\n",
              "      <th>imdb_url</th>\n",
              "      <th>genre_unknown</th>\n",
              "      <th>Action</th>\n",
              "      <th>Adventure</th>\n",
              "      <th>Animation</th>\n",
              "      <th>Children</th>\n",
              "      <th>Comedy</th>\n",
              "      <th>Crime</th>\n",
              "      <th>Documentary</th>\n",
              "      <th>Drama</th>\n",
              "      <th>Fantasy</th>\n",
              "      <th>Film-Noir</th>\n",
              "      <th>Horror</th>\n",
              "      <th>Musical</th>\n",
              "      <th>Mystery</th>\n",
              "      <th>Romance</th>\n",
              "      <th>Sci-Fi</th>\n",
              "      <th>Thriller</th>\n",
              "      <th>War</th>\n",
              "      <th>Western</th>\n",
              "      <th>year</th>\n",
              "      <th>genre</th>\n",
              "      <th>all_genres</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>01-Jan-1995</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?Toy%20Story%2...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1995</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>Animation-Children-Comedy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>GoldenEye (1995)</td>\n",
              "      <td>01-Jan-1995</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?GoldenEye%20(...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1995</td>\n",
              "      <td>Adventure</td>\n",
              "      <td>Action-Adventure-Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>Four Rooms (1995)</td>\n",
              "      <td>01-Jan-1995</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?Four%20Rooms%...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1995</td>\n",
              "      <td>Thriller</td>\n",
              "      <td>Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>Get Shorty (1995)</td>\n",
              "      <td>01-Jan-1995</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?Get%20Shorty%...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1995</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>Action-Comedy-Drama</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>Copycat (1995)</td>\n",
              "      <td>01-Jan-1995</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?Copycat%20(1995)</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1995</td>\n",
              "      <td>Crime</td>\n",
              "      <td>Crime-Drama-Thriller</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  movie_id              title  ...      genre                 all_genres\n",
              "0        0   Toy Story (1995)  ...     Comedy  Animation-Children-Comedy\n",
              "1        1   GoldenEye (1995)  ...  Adventure  Action-Adventure-Thriller\n",
              "2        2  Four Rooms (1995)  ...   Thriller                   Thriller\n",
              "3        3  Get Shorty (1995)  ...     Comedy        Action-Comedy-Drama\n",
              "4        4     Copycat (1995)  ...      Crime       Crime-Drama-Thriller\n",
              "\n",
              "[5 rows x 27 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MgCRtomqgXHp",
        "colab_type": "code",
        "outputId": "f890e6f0-1681-42fb-cd98-4417a0abb50a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "users.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>user_id</th>\n",
              "      <th>age</th>\n",
              "      <th>sex</th>\n",
              "      <th>occupation</th>\n",
              "      <th>zip_code</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>24</td>\n",
              "      <td>M</td>\n",
              "      <td>technician</td>\n",
              "      <td>85711</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>53</td>\n",
              "      <td>F</td>\n",
              "      <td>other</td>\n",
              "      <td>94043</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>23</td>\n",
              "      <td>M</td>\n",
              "      <td>writer</td>\n",
              "      <td>32067</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>24</td>\n",
              "      <td>M</td>\n",
              "      <td>technician</td>\n",
              "      <td>43537</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>33</td>\n",
              "      <td>F</td>\n",
              "      <td>other</td>\n",
              "      <td>15213</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  user_id  age sex  occupation zip_code\n",
              "0       0   24   M  technician    85711\n",
              "1       1   53   F       other    94043\n",
              "2       2   23   M      writer    32067\n",
              "3       3   24   M  technician    43537\n",
              "4       4   33   F       other    15213"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UHVdwsCBgWyl",
        "colab_type": "code",
        "outputId": "eb7bac9f-29b5-467c-ba41-c41603ccd7a6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "ratings.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>user_id</th>\n",
              "      <th>movie_id</th>\n",
              "      <th>rating</th>\n",
              "      <th>unix_timestamp</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>195</td>\n",
              "      <td>241</td>\n",
              "      <td>3.0</td>\n",
              "      <td>881250949</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>185</td>\n",
              "      <td>301</td>\n",
              "      <td>3.0</td>\n",
              "      <td>891717742</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>21</td>\n",
              "      <td>376</td>\n",
              "      <td>1.0</td>\n",
              "      <td>878887116</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>243</td>\n",
              "      <td>50</td>\n",
              "      <td>2.0</td>\n",
              "      <td>880606923</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>165</td>\n",
              "      <td>345</td>\n",
              "      <td>1.0</td>\n",
              "      <td>886397596</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  user_id movie_id  rating  unix_timestamp\n",
              "0     195      241     3.0       881250949\n",
              "1     185      301     3.0       891717742\n",
              "2      21      376     1.0       878887116\n",
              "3     243       50     2.0       880606923\n",
              "4     165      345     1.0       886397596"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SlgCWxb9h3Er",
        "colab_type": "code",
        "outputId": "9d57673f-c3e4-4bce-d051-da98d1149c46",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "movielens.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>user_id</th>\n",
              "      <th>movie_id</th>\n",
              "      <th>rating</th>\n",
              "      <th>unix_timestamp</th>\n",
              "      <th>title</th>\n",
              "      <th>release_date</th>\n",
              "      <th>video_release_date</th>\n",
              "      <th>imdb_url</th>\n",
              "      <th>genre_unknown</th>\n",
              "      <th>Action</th>\n",
              "      <th>Adventure</th>\n",
              "      <th>Animation</th>\n",
              "      <th>Children</th>\n",
              "      <th>Comedy</th>\n",
              "      <th>Crime</th>\n",
              "      <th>Documentary</th>\n",
              "      <th>Drama</th>\n",
              "      <th>Fantasy</th>\n",
              "      <th>Film-Noir</th>\n",
              "      <th>Horror</th>\n",
              "      <th>Musical</th>\n",
              "      <th>Mystery</th>\n",
              "      <th>Romance</th>\n",
              "      <th>Sci-Fi</th>\n",
              "      <th>Thriller</th>\n",
              "      <th>War</th>\n",
              "      <th>Western</th>\n",
              "      <th>year</th>\n",
              "      <th>genre</th>\n",
              "      <th>all_genres</th>\n",
              "      <th>age</th>\n",
              "      <th>sex</th>\n",
              "      <th>occupation</th>\n",
              "      <th>zip_code</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>195</td>\n",
              "      <td>241</td>\n",
              "      <td>3.0</td>\n",
              "      <td>881250949</td>\n",
              "      <td>Kolya (1996)</td>\n",
              "      <td>24-Jan-1997</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?Kolya%20(1996)</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1997</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>49</td>\n",
              "      <td>M</td>\n",
              "      <td>writer</td>\n",
              "      <td>55105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>195</td>\n",
              "      <td>256</td>\n",
              "      <td>2.0</td>\n",
              "      <td>881251577</td>\n",
              "      <td>Men in Black (1997)</td>\n",
              "      <td>04-Jul-1997</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?Men+in+Black+...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1997</td>\n",
              "      <td>Sci-Fi</td>\n",
              "      <td>Action-Adventure-Comedy-Sci-Fi</td>\n",
              "      <td>49</td>\n",
              "      <td>M</td>\n",
              "      <td>writer</td>\n",
              "      <td>55105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>195</td>\n",
              "      <td>110</td>\n",
              "      <td>4.0</td>\n",
              "      <td>881251793</td>\n",
              "      <td>Truth About Cats &amp; Dogs, The (1996)</td>\n",
              "      <td>26-Apr-1996</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?Truth%20About...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1996</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>Comedy-Romance</td>\n",
              "      <td>49</td>\n",
              "      <td>M</td>\n",
              "      <td>writer</td>\n",
              "      <td>55105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>195</td>\n",
              "      <td>24</td>\n",
              "      <td>4.0</td>\n",
              "      <td>881251955</td>\n",
              "      <td>Birdcage, The (1996)</td>\n",
              "      <td>08-Mar-1996</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?Birdcage,%20T...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1996</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>49</td>\n",
              "      <td>M</td>\n",
              "      <td>writer</td>\n",
              "      <td>55105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>195</td>\n",
              "      <td>381</td>\n",
              "      <td>4.0</td>\n",
              "      <td>881251843</td>\n",
              "      <td>Adventures of Priscilla, Queen of the Desert, ...</td>\n",
              "      <td>01-Jan-1994</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://us.imdb.com/M/title-exact?Adventures%20...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1994</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>Comedy-Drama</td>\n",
              "      <td>49</td>\n",
              "      <td>M</td>\n",
              "      <td>writer</td>\n",
              "      <td>55105</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  user_id movie_id  rating  unix_timestamp  ... age sex  occupation zip_code\n",
              "0     195      241     3.0       881250949  ...  49   M      writer    55105\n",
              "1     195      256     2.0       881251577  ...  49   M      writer    55105\n",
              "2     195      110     4.0       881251793  ...  49   M      writer    55105\n",
              "3     195       24     4.0       881251955  ...  49   M      writer    55105\n",
              "4     195      381     4.0       881251843  ...  49   M      writer    55105\n",
              "\n",
              "[5 rows x 34 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nb7OxcMvmFpU",
        "colab_type": "text"
      },
      "source": [
        "# Matrix Factorization via Singular Value Decomposition\n",
        "\n",
        "Based on great work done by **Nick Becker**, RAPIDS Team at NVIDIA\n",
        "\n",
        "*   https://beckernick.github.io/matrix-factorization-recommender/\n",
        "*   https://github.com/beckernick/matrix_factorization_recommenders\n",
        "\n",
        "This gave us great recommendations, however the RMSE we got was pretty high (probably due to some normalization factor we didn't account for)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W7jQ6gsq2_6Z",
        "colab_type": "text"
      },
      "source": [
        "Matrix factorization is the breaking down of one matrix in a product of multiple matrices. It's extremely well studied in mathematics, and it's highly useful. There are many different ways to factor matrices, but singular value decomposition is particularly useful for making recommendations.\n",
        "\n",
        "So what is singular value decomposition (SVD)? At a high level, SVD is an algorithm that decomposes a matrix $R$ into the best lower rank (i.e. smaller/simpler) approximation of the original matrix $R$. Mathematically, it decomposes R into a two unitary matrices and a diagonal matrix:\n",
        "\n",
        "$$\\begin{equation}\n",
        "R = U\\Sigma V^{T}\n",
        "\\end{equation}$$\n",
        "\n",
        "where R is users's ratings matrix, $U$ is the user \"features\" matrix, $\\Sigma$ is the diagonal matrix of singular values (essentially weights), and $V^{T}$ is the movie \"features\" matrix. $U$ and $V^{T}$ are orthogonal, and represent different things. $U$ represents how much users \"like\" each feature and $V^{T}$ represents how relevant each feature is to each movie.\n",
        "\n",
        "To get the lower rank approximation, we take these matrices and keep only the top $k$ features, which we think of as the underlying tastes and preferences vectors.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HRZ_GFV1mFpU",
        "colab_type": "text"
      },
      "source": [
        "## Setting Up the Ratings Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KjXQi5BwmFpV",
        "colab_type": "code",
        "outputId": "d8e4c8bb-0333-4211-9003-72e0db6a0f3e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import zipfile\n",
        "import urllib.request\n",
        "\n",
        "print(\"Downloading movielens data...\")\n",
        "    \n",
        "urllib.request.urlretrieve(\"http://files.grouplens.org/datasets/movielens/ml-latest-small.zip\", \"movielens.zip\")\n",
        "\n",
        "zip_ref = zipfile.ZipFile('movielens.zip', \"r\")\n",
        "zip_ref.extractall()\n",
        "\n",
        "\n",
        "ratings_df = pd.read_csv('ml-latest-small/ratings.csv', names=['user_id', 'movie_id', 'rating', 'timestamp'], sep=',', encoding='latin-1', header = None)\n",
        "ratings_df.drop([0], inplace=True)\n",
        "ratings_df=ratings_df.apply(pd.to_numeric)\n",
        "# ratings_df['UserID'] = ratings_df['UserID'].apply(pd.to_numeric)\n",
        "# ratings_df['UserID'] = ratings_df['UserID'].apply(pd.to_numeric)\n",
        "\n",
        "\n",
        "movies_df = pd.read_csv('ml-latest-small/movies.csv',names= ['movie_id', 'title', 'genres'], sep=',', encoding='latin-1')\n",
        "movies_df.drop([0], inplace=True)\n",
        "movies_df['movie_id'] = movies_df['movie_id'].apply(pd.to_numeric)\n",
        "# movies_df.drop('Genres', axis = 1, inplace = True)\n",
        "\n",
        "# Create one merged DataFrame containing all the movielens data.\n",
        "movielens18 = ratings_df.merge(movies_df, on='movie_id')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading movielens data...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "chdoZbZY32Pq"
      },
      "source": [
        "I'll also take a look at the movies and ratings dataframes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LAKvwm-qmFpe",
        "colab_type": "code",
        "outputId": "f714c0ac-acbd-4538-ccf7-dfc7c2841a4b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "movies_df.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(9742, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lm3XdZyL77iT",
        "colab_type": "code",
        "outputId": "53fb6822-1f1e-4851-eea5-3ff1989d2252",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "movielens18.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>user_id</th>\n",
              "      <th>movie_id</th>\n",
              "      <th>rating</th>\n",
              "      <th>timestamp</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>4.0</td>\n",
              "      <td>964982703</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>4.0</td>\n",
              "      <td>847434962</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>7</td>\n",
              "      <td>1</td>\n",
              "      <td>4.5</td>\n",
              "      <td>1106635946</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>15</td>\n",
              "      <td>1</td>\n",
              "      <td>2.5</td>\n",
              "      <td>1510577970</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>17</td>\n",
              "      <td>1</td>\n",
              "      <td>4.5</td>\n",
              "      <td>1305696483</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   user_id  ...                                       genres\n",
              "0        1  ...  Adventure|Animation|Children|Comedy|Fantasy\n",
              "1        5  ...  Adventure|Animation|Children|Comedy|Fantasy\n",
              "2        7  ...  Adventure|Animation|Children|Comedy|Fantasy\n",
              "3       15  ...  Adventure|Animation|Children|Comedy|Fantasy\n",
              "4       17  ...  Adventure|Animation|Children|Comedy|Fantasy\n",
              "\n",
              "[5 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "id": "nGQJRKjimFpm",
        "colab_type": "code",
        "outputId": "3116fc6c-4ad2-4eaa-c431-4eed07a2cbaa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "R_df = ratings_df.pivot(index = 'user_id', columns ='movie_id', values = 'rating').fillna(0)\n",
        "R_df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>movie_id</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "      <th>14</th>\n",
              "      <th>15</th>\n",
              "      <th>16</th>\n",
              "      <th>17</th>\n",
              "      <th>18</th>\n",
              "      <th>19</th>\n",
              "      <th>20</th>\n",
              "      <th>21</th>\n",
              "      <th>22</th>\n",
              "      <th>23</th>\n",
              "      <th>24</th>\n",
              "      <th>25</th>\n",
              "      <th>26</th>\n",
              "      <th>27</th>\n",
              "      <th>28</th>\n",
              "      <th>29</th>\n",
              "      <th>30</th>\n",
              "      <th>31</th>\n",
              "      <th>32</th>\n",
              "      <th>34</th>\n",
              "      <th>36</th>\n",
              "      <th>38</th>\n",
              "      <th>39</th>\n",
              "      <th>40</th>\n",
              "      <th>41</th>\n",
              "      <th>42</th>\n",
              "      <th>43</th>\n",
              "      <th>...</th>\n",
              "      <th>185135</th>\n",
              "      <th>185435</th>\n",
              "      <th>185473</th>\n",
              "      <th>185585</th>\n",
              "      <th>186587</th>\n",
              "      <th>187031</th>\n",
              "      <th>187541</th>\n",
              "      <th>187593</th>\n",
              "      <th>187595</th>\n",
              "      <th>187717</th>\n",
              "      <th>188189</th>\n",
              "      <th>188301</th>\n",
              "      <th>188675</th>\n",
              "      <th>188751</th>\n",
              "      <th>188797</th>\n",
              "      <th>188833</th>\n",
              "      <th>189043</th>\n",
              "      <th>189111</th>\n",
              "      <th>189333</th>\n",
              "      <th>189381</th>\n",
              "      <th>189547</th>\n",
              "      <th>189713</th>\n",
              "      <th>190183</th>\n",
              "      <th>190207</th>\n",
              "      <th>190209</th>\n",
              "      <th>190213</th>\n",
              "      <th>190215</th>\n",
              "      <th>190219</th>\n",
              "      <th>190221</th>\n",
              "      <th>191005</th>\n",
              "      <th>193565</th>\n",
              "      <th>193567</th>\n",
              "      <th>193571</th>\n",
              "      <th>193573</th>\n",
              "      <th>193579</th>\n",
              "      <th>193581</th>\n",
              "      <th>193583</th>\n",
              "      <th>193585</th>\n",
              "      <th>193587</th>\n",
              "      <th>193609</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>user_id</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 9724 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "movie_id  1       2       3       4       ...  193583  193585  193587  193609\n",
              "user_id                                   ...                                \n",
              "1            4.0     0.0     4.0     0.0  ...     0.0     0.0     0.0     0.0\n",
              "2            0.0     0.0     0.0     0.0  ...     0.0     0.0     0.0     0.0\n",
              "3            0.0     0.0     0.0     0.0  ...     0.0     0.0     0.0     0.0\n",
              "4            0.0     0.0     0.0     0.0  ...     0.0     0.0     0.0     0.0\n",
              "5            4.0     0.0     0.0     0.0  ...     0.0     0.0     0.0     0.0\n",
              "\n",
              "[5 rows x 9724 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cx3JXNidmFpp",
        "colab_type": "text"
      },
      "source": [
        "de-mean the data (normalize by each users mean) and convert it from a dataframe to a numpy array."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YlZlCFVgjLOd",
        "colab_type": "code",
        "outputId": "3fe659da-be4d-42ff-b2fd-8db552c9b8a1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "R = R_df.as_matrix()\n",
        "\n",
        "Z = R>0\n",
        "m, n = R.shape\n",
        "Ymean = np.zeros(m)\n",
        "Ynorm = np.zeros(R.shape)\n",
        "\n",
        "for i in range(m):\n",
        "    idx = Z[i, :] == 1\n",
        "    Ymean[i] = np.mean(R[i, idx])\n",
        "    Ynorm[i, idx] = R[i, idx] - Ymean[i]    "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n",
            "  \"\"\"Entry point for launching an IPython kernel.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KL2P2nQ9mFpp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "R_demeaned = R - Ymean[:,np.newaxis]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tJy5WybXmFps",
        "colab_type": "text"
      },
      "source": [
        "## Singular Value Decomposition\n",
        "\n",
        "Scipy and Numpy both have functions to do the singular value decomposition. I'm going to use the Scipy function `svds` because it let's me choose how many latent factors I want to use to approximate the original ratings matrix (instead of having to truncate it after)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RGOP36rnmFpt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from scipy.sparse.linalg import svds\n",
        "K = 10\n",
        "U, sigma, Vt = svds(R_demeaned, k = 10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i4l6wB903Ogv",
        "colab_type": "code",
        "outputId": "ea4e9231-eb25-4ca7-b8cd-a7da34bce92e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        }
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movie_id</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji (1995)</td>\n",
              "      <td>Adventure|Children|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>Grumpier Old Men (1995)</td>\n",
              "      <td>Comedy|Romance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>Waiting to Exhale (1995)</td>\n",
              "      <td>Comedy|Drama|Romance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5</td>\n",
              "      <td>Father of the Bride Part II (1995)</td>\n",
              "      <td>Comedy</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movie_id  ...                                       genres\n",
              "1         1  ...  Adventure|Animation|Children|Comedy|Fantasy\n",
              "2         2  ...                   Adventure|Children|Fantasy\n",
              "3         3  ...                               Comedy|Romance\n",
              "4         4  ...                         Comedy|Drama|Romance\n",
              "5         5  ...                                       Comedy\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nA7LhvKM31k0"
      },
      "source": [
        "Done. The function returns exactly what I detailed earlier in this post, except that the $\\Sigma$ returned is just the values instead of a diagonal matrix. This is useful, but since I'm going to leverage matrix multiplication to get predictions I'll convert it to the diagonal matrix form."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fJcwQHcYmFpw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sigma = np.diag(sigma)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tk_pBo_YmFpy",
        "colab_type": "text"
      },
      "source": [
        "## Making Predictions from the Decomposed Matrices"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I8R0CcTskojp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pd.DataFrame(np.dot(np.dot(U, sigma), Vt)).head() + Ymean[:,np.newaxis]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "04437d18-6393-47f8-d474-1f8e5d059689",
        "id": "dLsQaC0Ei4MX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 244
        }
      },
      "source": [
        "pd.DataFrame(np.dot(np.dot(U, sigma), Vt)).head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "      <th>14</th>\n",
              "      <th>15</th>\n",
              "      <th>16</th>\n",
              "      <th>17</th>\n",
              "      <th>18</th>\n",
              "      <th>19</th>\n",
              "      <th>20</th>\n",
              "      <th>21</th>\n",
              "      <th>22</th>\n",
              "      <th>23</th>\n",
              "      <th>24</th>\n",
              "      <th>25</th>\n",
              "      <th>26</th>\n",
              "      <th>27</th>\n",
              "      <th>28</th>\n",
              "      <th>29</th>\n",
              "      <th>30</th>\n",
              "      <th>31</th>\n",
              "      <th>32</th>\n",
              "      <th>33</th>\n",
              "      <th>34</th>\n",
              "      <th>35</th>\n",
              "      <th>36</th>\n",
              "      <th>37</th>\n",
              "      <th>38</th>\n",
              "      <th>39</th>\n",
              "      <th>...</th>\n",
              "      <th>9684</th>\n",
              "      <th>9685</th>\n",
              "      <th>9686</th>\n",
              "      <th>9687</th>\n",
              "      <th>9688</th>\n",
              "      <th>9689</th>\n",
              "      <th>9690</th>\n",
              "      <th>9691</th>\n",
              "      <th>9692</th>\n",
              "      <th>9693</th>\n",
              "      <th>9694</th>\n",
              "      <th>9695</th>\n",
              "      <th>9696</th>\n",
              "      <th>9697</th>\n",
              "      <th>9698</th>\n",
              "      <th>9699</th>\n",
              "      <th>9700</th>\n",
              "      <th>9701</th>\n",
              "      <th>9702</th>\n",
              "      <th>9703</th>\n",
              "      <th>9704</th>\n",
              "      <th>9705</th>\n",
              "      <th>9706</th>\n",
              "      <th>9707</th>\n",
              "      <th>9708</th>\n",
              "      <th>9709</th>\n",
              "      <th>9710</th>\n",
              "      <th>9711</th>\n",
              "      <th>9712</th>\n",
              "      <th>9713</th>\n",
              "      <th>9714</th>\n",
              "      <th>9715</th>\n",
              "      <th>9716</th>\n",
              "      <th>9717</th>\n",
              "      <th>9718</th>\n",
              "      <th>9719</th>\n",
              "      <th>9720</th>\n",
              "      <th>9721</th>\n",
              "      <th>9722</th>\n",
              "      <th>9723</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2.731634</td>\n",
              "      <td>0.824226</td>\n",
              "      <td>0.863543</td>\n",
              "      <td>-0.128215</td>\n",
              "      <td>0.117660</td>\n",
              "      <td>1.620033</td>\n",
              "      <td>0.022565</td>\n",
              "      <td>-0.117652</td>\n",
              "      <td>0.050279</td>\n",
              "      <td>1.912932</td>\n",
              "      <td>0.552621</td>\n",
              "      <td>0.254022</td>\n",
              "      <td>0.005336</td>\n",
              "      <td>-0.102017</td>\n",
              "      <td>-0.180319</td>\n",
              "      <td>0.948455</td>\n",
              "      <td>0.061236</td>\n",
              "      <td>-0.096068</td>\n",
              "      <td>0.687209</td>\n",
              "      <td>0.151289</td>\n",
              "      <td>1.261252</td>\n",
              "      <td>0.181749</td>\n",
              "      <td>-0.017190</td>\n",
              "      <td>0.252383</td>\n",
              "      <td>0.495318</td>\n",
              "      <td>-0.188269</td>\n",
              "      <td>-0.199703</td>\n",
              "      <td>-0.273025</td>\n",
              "      <td>0.547097</td>\n",
              "      <td>-0.012557</td>\n",
              "      <td>-0.111906</td>\n",
              "      <td>2.252317</td>\n",
              "      <td>1.131754</td>\n",
              "      <td>0.199436</td>\n",
              "      <td>0.003295</td>\n",
              "      <td>0.714506</td>\n",
              "      <td>-0.106971</td>\n",
              "      <td>-0.008130</td>\n",
              "      <td>0.051073</td>\n",
              "      <td>-0.122895</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.157027</td>\n",
              "      <td>-0.115065</td>\n",
              "      <td>-0.071877</td>\n",
              "      <td>-0.075409</td>\n",
              "      <td>-0.071877</td>\n",
              "      <td>-0.087704</td>\n",
              "      <td>-0.120529</td>\n",
              "      <td>-0.110677</td>\n",
              "      <td>-0.104763</td>\n",
              "      <td>-0.090134</td>\n",
              "      <td>-0.092262</td>\n",
              "      <td>-0.082204</td>\n",
              "      <td>-0.164606</td>\n",
              "      <td>-0.127641</td>\n",
              "      <td>-0.058698</td>\n",
              "      <td>-0.179804</td>\n",
              "      <td>-0.112345</td>\n",
              "      <td>-0.112530</td>\n",
              "      <td>-0.123898</td>\n",
              "      <td>-0.149409</td>\n",
              "      <td>-0.113312</td>\n",
              "      <td>-0.100775</td>\n",
              "      <td>-0.114916</td>\n",
              "      <td>-0.111973</td>\n",
              "      <td>-0.112902</td>\n",
              "      <td>-0.111787</td>\n",
              "      <td>-0.111973</td>\n",
              "      <td>-0.111787</td>\n",
              "      <td>-0.111787</td>\n",
              "      <td>-0.126525</td>\n",
              "      <td>-0.123168</td>\n",
              "      <td>-0.121489</td>\n",
              "      <td>-0.124847</td>\n",
              "      <td>-0.124847</td>\n",
              "      <td>-0.123168</td>\n",
              "      <td>-0.124847</td>\n",
              "      <td>-0.123168</td>\n",
              "      <td>-0.123168</td>\n",
              "      <td>-0.123168</td>\n",
              "      <td>-0.133069</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.178901</td>\n",
              "      <td>-0.038069</td>\n",
              "      <td>-0.037811</td>\n",
              "      <td>-0.006254</td>\n",
              "      <td>0.016541</td>\n",
              "      <td>0.076676</td>\n",
              "      <td>-0.073422</td>\n",
              "      <td>-0.002933</td>\n",
              "      <td>-0.018228</td>\n",
              "      <td>-0.082428</td>\n",
              "      <td>0.002699</td>\n",
              "      <td>-0.049389</td>\n",
              "      <td>-0.039679</td>\n",
              "      <td>-0.037711</td>\n",
              "      <td>-0.055149</td>\n",
              "      <td>0.211399</td>\n",
              "      <td>-0.075330</td>\n",
              "      <td>-0.094330</td>\n",
              "      <td>-0.072832</td>\n",
              "      <td>0.051227</td>\n",
              "      <td>-0.037693</td>\n",
              "      <td>-0.040162</td>\n",
              "      <td>-0.023834</td>\n",
              "      <td>-0.059590</td>\n",
              "      <td>0.045661</td>\n",
              "      <td>-0.070228</td>\n",
              "      <td>0.005690</td>\n",
              "      <td>-0.095775</td>\n",
              "      <td>-0.204458</td>\n",
              "      <td>-0.022554</td>\n",
              "      <td>0.016811</td>\n",
              "      <td>0.005587</td>\n",
              "      <td>-0.089603</td>\n",
              "      <td>0.000461</td>\n",
              "      <td>0.006093</td>\n",
              "      <td>-0.017727</td>\n",
              "      <td>-0.009136</td>\n",
              "      <td>-0.070792</td>\n",
              "      <td>-0.004064</td>\n",
              "      <td>-0.043874</td>\n",
              "      <td>...</td>\n",
              "      <td>0.031704</td>\n",
              "      <td>0.010337</td>\n",
              "      <td>0.000233</td>\n",
              "      <td>0.016881</td>\n",
              "      <td>0.000233</td>\n",
              "      <td>-0.002226</td>\n",
              "      <td>0.013059</td>\n",
              "      <td>0.091506</td>\n",
              "      <td>0.004994</td>\n",
              "      <td>0.000808</td>\n",
              "      <td>-0.000043</td>\n",
              "      <td>-0.044401</td>\n",
              "      <td>-0.005832</td>\n",
              "      <td>0.006365</td>\n",
              "      <td>0.002879</td>\n",
              "      <td>-0.005297</td>\n",
              "      <td>-0.004993</td>\n",
              "      <td>-0.004451</td>\n",
              "      <td>-0.001224</td>\n",
              "      <td>-0.006367</td>\n",
              "      <td>-0.008008</td>\n",
              "      <td>-0.003448</td>\n",
              "      <td>-0.004725</td>\n",
              "      <td>-0.006077</td>\n",
              "      <td>-0.003367</td>\n",
              "      <td>-0.006620</td>\n",
              "      <td>-0.006077</td>\n",
              "      <td>-0.006620</td>\n",
              "      <td>-0.006620</td>\n",
              "      <td>-0.000455</td>\n",
              "      <td>-0.002066</td>\n",
              "      <td>-0.002871</td>\n",
              "      <td>-0.001260</td>\n",
              "      <td>-0.001260</td>\n",
              "      <td>-0.002066</td>\n",
              "      <td>-0.001260</td>\n",
              "      <td>-0.002066</td>\n",
              "      <td>-0.002066</td>\n",
              "      <td>-0.002066</td>\n",
              "      <td>0.005347</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.023583</td>\n",
              "      <td>-0.001087</td>\n",
              "      <td>0.009023</td>\n",
              "      <td>-0.006277</td>\n",
              "      <td>-0.024101</td>\n",
              "      <td>0.065603</td>\n",
              "      <td>-0.024909</td>\n",
              "      <td>-0.005670</td>\n",
              "      <td>0.005685</td>\n",
              "      <td>0.055995</td>\n",
              "      <td>-0.022128</td>\n",
              "      <td>0.021467</td>\n",
              "      <td>0.000627</td>\n",
              "      <td>-0.007039</td>\n",
              "      <td>-0.006482</td>\n",
              "      <td>-0.006187</td>\n",
              "      <td>-0.038940</td>\n",
              "      <td>0.000050</td>\n",
              "      <td>0.002065</td>\n",
              "      <td>0.009784</td>\n",
              "      <td>0.016307</td>\n",
              "      <td>-0.008969</td>\n",
              "      <td>-0.002138</td>\n",
              "      <td>-0.000635</td>\n",
              "      <td>-0.007776</td>\n",
              "      <td>-0.011981</td>\n",
              "      <td>-0.015251</td>\n",
              "      <td>-0.016733</td>\n",
              "      <td>0.021421</td>\n",
              "      <td>0.003982</td>\n",
              "      <td>-0.023048</td>\n",
              "      <td>0.052589</td>\n",
              "      <td>-0.016008</td>\n",
              "      <td>-0.024877</td>\n",
              "      <td>0.003988</td>\n",
              "      <td>-0.047013</td>\n",
              "      <td>-0.001045</td>\n",
              "      <td>0.000345</td>\n",
              "      <td>0.006930</td>\n",
              "      <td>-0.005408</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.001495</td>\n",
              "      <td>0.000430</td>\n",
              "      <td>-0.000285</td>\n",
              "      <td>0.000377</td>\n",
              "      <td>-0.000285</td>\n",
              "      <td>-0.000713</td>\n",
              "      <td>-0.002556</td>\n",
              "      <td>0.010296</td>\n",
              "      <td>0.008870</td>\n",
              "      <td>-0.000708</td>\n",
              "      <td>-0.000762</td>\n",
              "      <td>0.010046</td>\n",
              "      <td>-0.002890</td>\n",
              "      <td>-0.001361</td>\n",
              "      <td>0.000034</td>\n",
              "      <td>-0.003361</td>\n",
              "      <td>-0.001517</td>\n",
              "      <td>-0.001573</td>\n",
              "      <td>-0.000727</td>\n",
              "      <td>-0.002418</td>\n",
              "      <td>-0.000932</td>\n",
              "      <td>-0.000974</td>\n",
              "      <td>-0.001406</td>\n",
              "      <td>-0.001406</td>\n",
              "      <td>-0.001684</td>\n",
              "      <td>-0.001351</td>\n",
              "      <td>-0.001406</td>\n",
              "      <td>-0.001351</td>\n",
              "      <td>-0.001351</td>\n",
              "      <td>-0.001205</td>\n",
              "      <td>-0.001213</td>\n",
              "      <td>-0.001217</td>\n",
              "      <td>-0.001209</td>\n",
              "      <td>-0.001209</td>\n",
              "      <td>-0.001213</td>\n",
              "      <td>-0.001209</td>\n",
              "      <td>-0.001213</td>\n",
              "      <td>-0.001213</td>\n",
              "      <td>-0.001213</td>\n",
              "      <td>-0.003140</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.479939</td>\n",
              "      <td>0.196467</td>\n",
              "      <td>0.192636</td>\n",
              "      <td>-0.035120</td>\n",
              "      <td>0.104789</td>\n",
              "      <td>0.194374</td>\n",
              "      <td>0.267949</td>\n",
              "      <td>-0.133225</td>\n",
              "      <td>-0.115444</td>\n",
              "      <td>-0.010298</td>\n",
              "      <td>0.718759</td>\n",
              "      <td>-0.133585</td>\n",
              "      <td>-0.007747</td>\n",
              "      <td>0.036157</td>\n",
              "      <td>-0.009510</td>\n",
              "      <td>0.395341</td>\n",
              "      <td>1.088663</td>\n",
              "      <td>0.172450</td>\n",
              "      <td>0.192556</td>\n",
              "      <td>-0.160884</td>\n",
              "      <td>0.877455</td>\n",
              "      <td>-0.066031</td>\n",
              "      <td>-0.167028</td>\n",
              "      <td>0.099026</td>\n",
              "      <td>0.699756</td>\n",
              "      <td>0.005758</td>\n",
              "      <td>-0.080493</td>\n",
              "      <td>0.334972</td>\n",
              "      <td>0.899928</td>\n",
              "      <td>0.085349</td>\n",
              "      <td>-0.010672</td>\n",
              "      <td>1.062285</td>\n",
              "      <td>0.975442</td>\n",
              "      <td>0.721076</td>\n",
              "      <td>-0.077426</td>\n",
              "      <td>1.067818</td>\n",
              "      <td>-0.097435</td>\n",
              "      <td>0.034366</td>\n",
              "      <td>-0.016771</td>\n",
              "      <td>-0.036849</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.128765</td>\n",
              "      <td>-0.109915</td>\n",
              "      <td>-0.073742</td>\n",
              "      <td>-0.075341</td>\n",
              "      <td>-0.073742</td>\n",
              "      <td>-0.077813</td>\n",
              "      <td>-0.088481</td>\n",
              "      <td>-0.180858</td>\n",
              "      <td>-0.180034</td>\n",
              "      <td>-0.044795</td>\n",
              "      <td>-0.049857</td>\n",
              "      <td>-0.138352</td>\n",
              "      <td>-0.064030</td>\n",
              "      <td>-0.088939</td>\n",
              "      <td>-0.066516</td>\n",
              "      <td>-0.055062</td>\n",
              "      <td>-0.096042</td>\n",
              "      <td>-0.096166</td>\n",
              "      <td>-0.099938</td>\n",
              "      <td>-0.072999</td>\n",
              "      <td>-0.098290</td>\n",
              "      <td>-0.070107</td>\n",
              "      <td>-0.075434</td>\n",
              "      <td>-0.095793</td>\n",
              "      <td>-0.096415</td>\n",
              "      <td>-0.095669</td>\n",
              "      <td>-0.095793</td>\n",
              "      <td>-0.095669</td>\n",
              "      <td>-0.095669</td>\n",
              "      <td>-0.096665</td>\n",
              "      <td>-0.096388</td>\n",
              "      <td>-0.096250</td>\n",
              "      <td>-0.096527</td>\n",
              "      <td>-0.096527</td>\n",
              "      <td>-0.096388</td>\n",
              "      <td>-0.096527</td>\n",
              "      <td>-0.096388</td>\n",
              "      <td>-0.096388</td>\n",
              "      <td>-0.096388</td>\n",
              "      <td>-0.105531</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.256434</td>\n",
              "      <td>0.974787</td>\n",
              "      <td>0.403596</td>\n",
              "      <td>0.106501</td>\n",
              "      <td>0.518697</td>\n",
              "      <td>0.736876</td>\n",
              "      <td>0.617943</td>\n",
              "      <td>0.101136</td>\n",
              "      <td>0.094213</td>\n",
              "      <td>1.135083</td>\n",
              "      <td>1.039142</td>\n",
              "      <td>-0.009352</td>\n",
              "      <td>0.057612</td>\n",
              "      <td>0.268524</td>\n",
              "      <td>0.224823</td>\n",
              "      <td>0.492558</td>\n",
              "      <td>0.722043</td>\n",
              "      <td>0.036372</td>\n",
              "      <td>0.588009</td>\n",
              "      <td>0.010699</td>\n",
              "      <td>1.037831</td>\n",
              "      <td>0.473947</td>\n",
              "      <td>0.152440</td>\n",
              "      <td>0.153748</td>\n",
              "      <td>0.740593</td>\n",
              "      <td>0.207954</td>\n",
              "      <td>0.100080</td>\n",
              "      <td>0.087360</td>\n",
              "      <td>0.101443</td>\n",
              "      <td>-0.024128</td>\n",
              "      <td>0.434146</td>\n",
              "      <td>1.249361</td>\n",
              "      <td>1.437849</td>\n",
              "      <td>0.846536</td>\n",
              "      <td>-0.031665</td>\n",
              "      <td>1.015884</td>\n",
              "      <td>-0.017989</td>\n",
              "      <td>0.179374</td>\n",
              "      <td>-0.014745</td>\n",
              "      <td>0.105071</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.016617</td>\n",
              "      <td>-0.023214</td>\n",
              "      <td>-0.020868</td>\n",
              "      <td>-0.018384</td>\n",
              "      <td>-0.020868</td>\n",
              "      <td>-0.020560</td>\n",
              "      <td>-0.017447</td>\n",
              "      <td>-0.024693</td>\n",
              "      <td>-0.043651</td>\n",
              "      <td>-0.014485</td>\n",
              "      <td>-0.015134</td>\n",
              "      <td>-0.031037</td>\n",
              "      <td>-0.008812</td>\n",
              "      <td>-0.019438</td>\n",
              "      <td>-0.020831</td>\n",
              "      <td>-0.005335</td>\n",
              "      <td>-0.018894</td>\n",
              "      <td>-0.018477</td>\n",
              "      <td>-0.021794</td>\n",
              "      <td>-0.012288</td>\n",
              "      <td>-0.022236</td>\n",
              "      <td>-0.017732</td>\n",
              "      <td>-0.017760</td>\n",
              "      <td>-0.019728</td>\n",
              "      <td>-0.017642</td>\n",
              "      <td>-0.020145</td>\n",
              "      <td>-0.019728</td>\n",
              "      <td>-0.020145</td>\n",
              "      <td>-0.020145</td>\n",
              "      <td>-0.019929</td>\n",
              "      <td>-0.020162</td>\n",
              "      <td>-0.020279</td>\n",
              "      <td>-0.020046</td>\n",
              "      <td>-0.020046</td>\n",
              "      <td>-0.020162</td>\n",
              "      <td>-0.020046</td>\n",
              "      <td>-0.020162</td>\n",
              "      <td>-0.020162</td>\n",
              "      <td>-0.020162</td>\n",
              "      <td>-0.020979</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 9724 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "       0         1         2     ...      9721      9722      9723\n",
              "0  2.731634  0.824226  0.863543  ... -0.123168 -0.123168 -0.133069\n",
              "1  0.178901 -0.038069 -0.037811  ... -0.002066 -0.002066  0.005347\n",
              "2  0.023583 -0.001087  0.009023  ... -0.001213 -0.001213 -0.003140\n",
              "3  1.479939  0.196467  0.192636  ... -0.096388 -0.096388 -0.105531\n",
              "4  1.256434  0.974787  0.403596  ... -0.020162 -0.020162 -0.020979\n",
              "\n",
              "[5 rows x 9724 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I6giAY0tmFpz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt) + preds_df"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iLvgLIzYmFp3",
        "colab_type": "text"
      },
      "source": [
        "### Making Movie Recommendations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l8bzSP5HlJ59",
        "colab_type": "code",
        "outputId": "1c05eab8-252a-4229-dbc1-ae516a08ab46",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "Ymean[:,np.newaxis].shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(610, 1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "TNNVvxX4mFp4",
        "colab_type": "code",
        "outputId": "64128273-6967-4f8a-c00e-1771d12f5da0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "preds_df = pd.DataFrame(all_user_predicted_ratings, columns = R_df.columns)\n",
        "print(preds_df.shape)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(610, 9724)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MCfC6oBKmFp8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def recommend_movies(predictions_df, userID, movies_df, original_ratings_df, num_recommendations=5):\n",
        "    \n",
        "    # Get and sort the user's predictions\n",
        "    user_row_number = userID -1\n",
        "    sorted_user_predictions = predictions_df.iloc[user_row_number].sort_values(ascending=False)\n",
        "\n",
        "    # Get the user's data and merge in the movie information.\n",
        "    user_data = original_ratings_df[original_ratings_df.user_id == userID]\n",
        "    user_full = (user_data.merge(movies_df, how = 'left', left_on = 'movie_id', right_on = 'movie_id').\n",
        "                     sort_values(['rating'], ascending=False))\n",
        "\n",
        "    print ('User {0} has already rated {1} movies.'.format(userID, user_full.shape[0]))\n",
        "    print ('Recommending highest {0} predicted ratings movies not already rated.'.format(num_recommendations))\n",
        "    \n",
        "    # Recommend the highest predicted rating movies that the user hasn't seen yet.\n",
        "    recommendations = (movies_df[~movies_df['movie_id'].isin(user_full['movie_id'])]. #all the movies not in the user_full recommendations\n",
        "         merge(pd.DataFrame(sorted_user_predictions).reset_index(), how = 'left',\n",
        "               left_on = 'movie_id',\n",
        "               right_on = 'movie_id').\n",
        "         rename(columns = {user_row_number: 'Predictions'}).\n",
        "         sort_values('Predictions', ascending = False).\n",
        "                       iloc[:num_recommendations, :-1]\n",
        "                      )\n",
        "\n",
        "    return user_full, recommendations"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8c3WNh_VmFp-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "already_rated, predictions = recommend_movies(preds_df, 25, movies_df, ratings_df, 10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "WKrJyHYTmFqC",
        "colab_type": "code",
        "outputId": "9d3cc007-17e8-4a83-b22d-47a8b77a5bb2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        }
      },
      "source": [
        "already_rated.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>user_id</th>\n",
              "      <th>movie_id</th>\n",
              "      <th>rating</th>\n",
              "      <th>timestamp</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>25</td>\n",
              "      <td>68157</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470515</td>\n",
              "      <td>Inglourious Basterds (2009)</td>\n",
              "      <td>Action|Drama|War</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>25</td>\n",
              "      <td>60069</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470523</td>\n",
              "      <td>WALLÂ·E (2008)</td>\n",
              "      <td>Adventure|Animation|Children|Romance|Sci-Fi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>25</td>\n",
              "      <td>180095</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470476</td>\n",
              "      <td>Wonder (2017)</td>\n",
              "      <td>Drama</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>25</td>\n",
              "      <td>177593</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470532</td>\n",
              "      <td>Three Billboards Outside Ebbing, Missouri (2017)</td>\n",
              "      <td>Crime|Drama</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>25</td>\n",
              "      <td>122912</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470461</td>\n",
              "      <td>Avengers: Infinity War - Part I (2018)</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>25</td>\n",
              "      <td>116797</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470507</td>\n",
              "      <td>The Imitation Game (2014)</td>\n",
              "      <td>Drama|Thriller|War</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>25</td>\n",
              "      <td>91529</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470498</td>\n",
              "      <td>Dark Knight Rises, The (2012)</td>\n",
              "      <td>Action|Adventure|Crime|IMAX</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>25</td>\n",
              "      <td>79132</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470428</td>\n",
              "      <td>Inception (2010)</td>\n",
              "      <td>Action|Crime|Drama|Mystery|Sci-Fi|Thriller|IMAX</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>25</td>\n",
              "      <td>68954</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470493</td>\n",
              "      <td>Up (2009)</td>\n",
              "      <td>Adventure|Animation|Children|Drama</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>25</td>\n",
              "      <td>260</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1535470429</td>\n",
              "      <td>Star Wars: Episode IV - A New Hope (1977)</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    user_id  ...                                           genres\n",
              "13       25  ...                                 Action|Drama|War\n",
              "12       25  ...      Adventure|Animation|Children|Romance|Sci-Fi\n",
              "23       25  ...                                            Drama\n",
              "22       25  ...                                      Crime|Drama\n",
              "19       25  ...                          Action|Adventure|Sci-Fi\n",
              "18       25  ...                               Drama|Thriller|War\n",
              "17       25  ...                      Action|Adventure|Crime|IMAX\n",
              "16       25  ...  Action|Crime|Drama|Mystery|Sci-Fi|Thriller|IMAX\n",
              "14       25  ...               Adventure|Animation|Children|Drama\n",
              "1        25  ...                          Action|Adventure|Sci-Fi\n",
              "\n",
              "[10 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "Mrrbep46mFqH",
        "colab_type": "code",
        "outputId": "5fa078cc-79c3-4df1-dd56-791a2b7ae0a3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        }
      },
      "source": [
        "predictions"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movie_id</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>275</th>\n",
              "      <td>318</td>\n",
              "      <td>Shawshank Redemption, The (1994)</td>\n",
              "      <td>Crime|Drama</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>312</th>\n",
              "      <td>356</td>\n",
              "      <td>Forrest Gump (1994)</td>\n",
              "      <td>Comedy|Drama|Romance|War</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2220</th>\n",
              "      <td>2959</td>\n",
              "      <td>Fight Club (1999)</td>\n",
              "      <td>Action|Crime|Drama|Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>895</th>\n",
              "      <td>1196</td>\n",
              "      <td>Star Wars: Episode V - The Empire Strikes Back...</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>907</th>\n",
              "      <td>1210</td>\n",
              "      <td>Star Wars: Episode VI - Return of the Jedi (1983)</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>507</th>\n",
              "      <td>593</td>\n",
              "      <td>Silence of the Lambs, The (1991)</td>\n",
              "      <td>Crime|Horror|Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>656</th>\n",
              "      <td>858</td>\n",
              "      <td>Godfather, The (1972)</td>\n",
              "      <td>Crime|Drama</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>255</th>\n",
              "      <td>296</td>\n",
              "      <td>Pulp Fiction (1994)</td>\n",
              "      <td>Comedy|Crime|Drama|Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3187</th>\n",
              "      <td>4306</td>\n",
              "      <td>Shrek (2001)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy|Ro...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8358</th>\n",
              "      <td>109487</td>\n",
              "      <td>Interstellar (2014)</td>\n",
              "      <td>Sci-Fi|IMAX</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      movie_id  ...                                             genres\n",
              "275        318  ...                                        Crime|Drama\n",
              "312        356  ...                           Comedy|Drama|Romance|War\n",
              "2220      2959  ...                        Action|Crime|Drama|Thriller\n",
              "895       1196  ...                            Action|Adventure|Sci-Fi\n",
              "907       1210  ...                            Action|Adventure|Sci-Fi\n",
              "507        593  ...                              Crime|Horror|Thriller\n",
              "656        858  ...                                        Crime|Drama\n",
              "255        296  ...                        Comedy|Crime|Drama|Thriller\n",
              "3187      4306  ...  Adventure|Animation|Children|Comedy|Fantasy|Ro...\n",
              "8358    109487  ...                                        Sci-Fi|IMAX\n",
              "\n",
              "[10 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "EvIXZS7b6KBx"
      },
      "source": [
        "###Train Test Split"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kXVh1g1V9EZc",
        "colab_type": "code",
        "outputId": "88bdd805-0a4a-46e6-f7bf-620daa0bf3d3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "movielens18.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(100836, 6)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "0603cca2-780c-45e3-88f9-ede765f6dea5",
        "id": "omhiuMPQ6KB3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "train_df = movielens18.sample(frac=0.9, random_state=0)\n",
        "test_df = movielens18.drop(train_df.index.tolist())\n",
        "train_df.shape, test_df.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((90752, 6), (10084, 6))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 100
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yAIh_rA5pQSM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.metrics import mean_absolute_error\n",
        "\n",
        "# df = fetch_ml20m_ratings()\n",
        "# movielens18.drop(columns = ['title', 'genres'], inplace=True)\n",
        "# movielens18.columns = df.columns\n",
        "\n",
        "train = movielens18.sample(frac=0.8, random_state=7)\n",
        "val = movielens18.drop(train.index.tolist()).sample(frac=0.5, random_state=8)\n",
        "test = movielens18.drop(train.index.tolist()).drop(val.index.tolist())\n",
        "\n",
        "svd = SVD(learning_rate=0.01, regularization=0.1, n_epochs=100,\n",
        "          n_factors=80, min_rating=1, max_rating=5)\n",
        "\n",
        "svd.fit(X=train, X_val=val, early_stopping=True, shuffle=False)\n",
        "\n",
        "pred = svd.predict(test)\n",
        "mae = mean_absolute_error(test[\"rating\"], pred)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "qjxZ1ffa6KB9",
        "colab": {}
      },
      "source": [
        "min_user_clicks = 10\n",
        "filter_users = train_df['user_id'].value_counts() > min_user_clicks\n",
        "filter_users = filter_users[filter_users].index.tolist()\n",
        "\n",
        "min_item_clicks = 10\n",
        "filter_items = train_df['movie_id'].value_counts() > min_item_clicks\n",
        "filter_items = filter_items[filter_items].index.tolist()\n",
        "\n",
        "train_filtered = train_df[(train_df['user_id'].isin(filter_users)) & train_df['movie_id'].isin(filter_items)]\n",
        "print('The original data frame shape:\\t{}'.format(train_df.shape))\n",
        "print('The new data frame shape:\\t{}'.format(train_filtered.shape))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Ltv9ZAXE6KB_",
        "colab": {}
      },
      "source": [
        "#Check that all users appears in both Train & Test\n",
        "\n",
        "train_users_idx = list(train_filtered.user_id.unique())\n",
        "test_df_idx = list(test_df.user_id.unique())\n",
        "iters = list(set(train_users_idx) & set(test_df_idx))\n",
        "\n",
        "\n",
        "train_filtered = train_filtered[train_filtered.user_id.isin(iters)]\n",
        "test_df = test_df[test_df.user_id.isin(iters)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Mu5Ob0dw6KCB",
        "colab": {}
      },
      "source": [
        "#Check that all movies in Test appears in Train\n",
        "\n",
        "train_movies_inx = list(train_filtered.movie_id.unique())\n",
        "test_movies_inx = list(test_df.movie_id.unique())\n",
        "uniques_movies_test = list(np.setdiff1d(test_movies_inx, train_movies_inx))\n",
        "flt_lst = list(np.setdiff1d(test_movies_inx, uniques_movies_test))\n",
        "\n",
        "\n",
        "test_df = test_df[test_df['movie_id'].isin(flt_lst)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kFOXDGpE6KCC",
        "colab": {}
      },
      "source": [
        "train_users = list(train_filtered.user_id.unique())\n",
        "test_users = list(test_df.user_id.unique())\n",
        "counter=0\n",
        "for user in test_users:\n",
        "    if user not in train_users:\n",
        "        counter =+ 1\n",
        "print(\"Number of non overlaps in users = {}\".format(counter))\n",
        "\n",
        "train_movies = list(train_filtered.movie_id.unique())\n",
        "test_movies = list(test_df.movie_id.unique())\n",
        "counter=0\n",
        "for movie in test_movies:\n",
        "    if movie not in train_movies:\n",
        "        counter =counter + 1\n",
        "\n",
        "print(\"Number movies in test that are not in train = {}\".format(counter))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RpKy21EQ6KCF",
        "colab": {}
      },
      "source": [
        "pivot_train = train_filtered.pivot_table(values = 'rating', index = 'movie_id', columns = 'user_id')\n",
        "pivot_train.fillna(0, inplace = True)\n",
        "\n",
        "pivot_test = test_df.pivot_table(values = 'rating', index = 'movie_id', columns = 'user_id')\n",
        "pivot_test.fillna(0, inplace = True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MBGquUd06KCR",
        "colab": {}
      },
      "source": [
        "print(\"Train set contains {} ratings\".format(np.sum(np.sum(pivot_train>0))))\n",
        "print(\"Test set contains {} ratings\".format(np.sum(np.sum(pivot_test>0))))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yVgThXaT6KCT",
        "colab": {}
      },
      "source": [
        "#Checking if movies in the test set are included in the train set\n",
        "\n",
        "s_test = set(pivot_test.columns)\n",
        "s_train = set(pivot_train.columns)\n",
        "\n",
        "inter = s_test.issubset(s_train)\n",
        "inter, len(s_train), len(s_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oN1j0Del9r0F",
        "colab_type": "text"
      },
      "source": [
        "### Training the algorithm on the train set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "colab_type": "code",
        "outputId": "1cc35b27-5c1a-434b-9bf2-0da20165c375",
        "id": "BdN9Wvbx-G6A",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 277
        }
      },
      "source": [
        "R_df = pivot_train.copy()\n",
        "R_df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>user_id</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "      <th>14</th>\n",
              "      <th>15</th>\n",
              "      <th>16</th>\n",
              "      <th>17</th>\n",
              "      <th>18</th>\n",
              "      <th>19</th>\n",
              "      <th>20</th>\n",
              "      <th>21</th>\n",
              "      <th>22</th>\n",
              "      <th>23</th>\n",
              "      <th>24</th>\n",
              "      <th>26</th>\n",
              "      <th>27</th>\n",
              "      <th>28</th>\n",
              "      <th>29</th>\n",
              "      <th>31</th>\n",
              "      <th>32</th>\n",
              "      <th>33</th>\n",
              "      <th>34</th>\n",
              "      <th>35</th>\n",
              "      <th>36</th>\n",
              "      <th>37</th>\n",
              "      <th>38</th>\n",
              "      <th>39</th>\n",
              "      <th>40</th>\n",
              "      <th>41</th>\n",
              "      <th>42</th>\n",
              "      <th>...</th>\n",
              "      <th>570</th>\n",
              "      <th>571</th>\n",
              "      <th>572</th>\n",
              "      <th>573</th>\n",
              "      <th>574</th>\n",
              "      <th>575</th>\n",
              "      <th>576</th>\n",
              "      <th>577</th>\n",
              "      <th>578</th>\n",
              "      <th>579</th>\n",
              "      <th>580</th>\n",
              "      <th>581</th>\n",
              "      <th>582</th>\n",
              "      <th>583</th>\n",
              "      <th>584</th>\n",
              "      <th>585</th>\n",
              "      <th>586</th>\n",
              "      <th>587</th>\n",
              "      <th>588</th>\n",
              "      <th>589</th>\n",
              "      <th>590</th>\n",
              "      <th>591</th>\n",
              "      <th>592</th>\n",
              "      <th>593</th>\n",
              "      <th>594</th>\n",
              "      <th>595</th>\n",
              "      <th>596</th>\n",
              "      <th>597</th>\n",
              "      <th>598</th>\n",
              "      <th>599</th>\n",
              "      <th>600</th>\n",
              "      <th>601</th>\n",
              "      <th>602</th>\n",
              "      <th>603</th>\n",
              "      <th>604</th>\n",
              "      <th>605</th>\n",
              "      <th>606</th>\n",
              "      <th>607</th>\n",
              "      <th>608</th>\n",
              "      <th>610</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>movie_id</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.5</td>\n",
              "      <td>3.5</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.5</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.5</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>3.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.5</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 595 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "user_id   1    2    3    4    5    6    7    ...  603  604  605  606  607  608  610\n",
              "movie_id                                     ...                                   \n",
              "1         4.0  0.0  0.0  0.0  4.0  0.0  4.5  ...  4.0  3.0  4.0  2.5  0.0  2.5  5.0\n",
              "2         0.0  0.0  0.0  0.0  0.0  4.0  0.0  ...  0.0  0.0  3.5  0.0  0.0  2.0  0.0\n",
              "3         0.0  0.0  0.0  0.0  0.0  5.0  0.0  ...  0.0  0.0  0.0  0.0  0.0  2.0  0.0\n",
              "5         0.0  0.0  0.0  0.0  0.0  5.0  0.0  ...  0.0  3.0  0.0  0.0  0.0  0.0  0.0\n",
              "6         4.0  0.0  0.0  0.0  0.0  4.0  0.0  ...  4.0  3.0  0.0  0.0  0.0  0.0  5.0\n",
              "\n",
              "[5 rows x 595 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "6a100e6c-8d90-4763-a3a2-b6ec62e5340d",
        "id": "Rt5Z3Cc_l1Cr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "R = R_df.as_matrix()\n",
        "\n",
        "Z = R>0\n",
        "m, n = R.shape\n",
        "Ymean = np.zeros(m)\n",
        "Ynorm = np.zeros(R.shape)\n",
        "\n",
        "for i in range(m):\n",
        "    idx = Z[i, :] == 1\n",
        "    Ymean[i] = np.mean(R[i, idx])\n",
        "    Ynorm[i, idx] = R[i, idx] - Ymean[i]    "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n",
            "  \"\"\"Entry point for launching an IPython kernel.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "DvtLH8JDl1C4",
        "colab": {}
      },
      "source": [
        "R_demeaned = R - Ymean[:,np.newaxis]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-utzkXnO-G6D",
        "colab": {}
      },
      "source": [
        "from scipy.sparse.linalg import svds"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hnN2JAjl6KCb",
        "colab": {}
      },
      "source": [
        "'''\n",
        "The best way I've found to do this..\n",
        "Looping over the test pivot and locating the corresponding cell in the train set,\n",
        "by INDEX, where the test-pivot has a rating.\n",
        "'''\n",
        "def accuracy(pivot_test, preds_df, k):\n",
        "    true_val = []\n",
        "    prediction_val = []\n",
        "\n",
        "    movie_intersect = list(np.setdiff1d(pivot_test.index, preds_df.index))\n",
        "    user_intersect = list(np.setdiff1d(pivot_test.columns, preds_df.columns))\n",
        "\n",
        "    for i in pivot_test.index: #movie_id\n",
        "        for j in pivot_test.columns: #user_id\n",
        "            if (pivot_test.loc[i,j]!=0) and (i not in movie_intersect):\n",
        "                real_value = pivot_test.loc[i,j]\n",
        "                prediction = preds_df.loc[i,j]\n",
        "#                 print('movie_id = {},user_id = {}'.format(i,j))\n",
        "#                 print(\"real value = {}, prediction = {}\".format(real_value, prediction))\n",
        "                prediction_val.append(preds_df.loc[i,j] + Ymean[:,np.newaxis])\n",
        "                true_val.append(pivot_test.loc[i,j])\n",
        "\n",
        "    true_val = np.array(true_val)\n",
        "    validation_val = np.array(prediction_val)\n",
        "    RMSE = np.round(np.sqrt(np.square(true_val - prediction_val).mean()),4)\n",
        "    MAE = np.round(np.abs(true_val - prediction_val).mean(),4)\n",
        "\n",
        "    print('RMSE: {}, MAE: {}, K: {}'.format(RMSE, MAE, k))\n",
        "    return (RMSE, MAE)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZxaMTQGNDbVz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "results = pd.DataFrame(columns=['RMSE', 'MAE','K'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dYt30XeMp2rU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pivot_train"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kih3qrFPuMcG",
        "colab_type": "code",
        "outputId": "93c6f7b2-f90c-4454-846a-41814eee730b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 247
        }
      },
      "source": [
        "preds_df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>user_id</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "      <th>14</th>\n",
              "      <th>15</th>\n",
              "      <th>16</th>\n",
              "      <th>17</th>\n",
              "      <th>18</th>\n",
              "      <th>19</th>\n",
              "      <th>20</th>\n",
              "      <th>21</th>\n",
              "      <th>22</th>\n",
              "      <th>23</th>\n",
              "      <th>24</th>\n",
              "      <th>26</th>\n",
              "      <th>27</th>\n",
              "      <th>28</th>\n",
              "      <th>29</th>\n",
              "      <th>31</th>\n",
              "      <th>32</th>\n",
              "      <th>33</th>\n",
              "      <th>34</th>\n",
              "      <th>35</th>\n",
              "      <th>36</th>\n",
              "      <th>37</th>\n",
              "      <th>38</th>\n",
              "      <th>39</th>\n",
              "      <th>40</th>\n",
              "      <th>41</th>\n",
              "      <th>42</th>\n",
              "      <th>...</th>\n",
              "      <th>570</th>\n",
              "      <th>571</th>\n",
              "      <th>572</th>\n",
              "      <th>573</th>\n",
              "      <th>574</th>\n",
              "      <th>575</th>\n",
              "      <th>576</th>\n",
              "      <th>577</th>\n",
              "      <th>578</th>\n",
              "      <th>579</th>\n",
              "      <th>580</th>\n",
              "      <th>581</th>\n",
              "      <th>582</th>\n",
              "      <th>583</th>\n",
              "      <th>584</th>\n",
              "      <th>585</th>\n",
              "      <th>586</th>\n",
              "      <th>587</th>\n",
              "      <th>588</th>\n",
              "      <th>589</th>\n",
              "      <th>590</th>\n",
              "      <th>591</th>\n",
              "      <th>592</th>\n",
              "      <th>593</th>\n",
              "      <th>594</th>\n",
              "      <th>595</th>\n",
              "      <th>596</th>\n",
              "      <th>597</th>\n",
              "      <th>598</th>\n",
              "      <th>599</th>\n",
              "      <th>600</th>\n",
              "      <th>601</th>\n",
              "      <th>602</th>\n",
              "      <th>603</th>\n",
              "      <th>604</th>\n",
              "      <th>605</th>\n",
              "      <th>606</th>\n",
              "      <th>607</th>\n",
              "      <th>608</th>\n",
              "      <th>610</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3.451891</td>\n",
              "      <td>0.058611</td>\n",
              "      <td>0.007331</td>\n",
              "      <td>2.637868</td>\n",
              "      <td>1.099987</td>\n",
              "      <td>3.230932</td>\n",
              "      <td>2.198768</td>\n",
              "      <td>1.193644</td>\n",
              "      <td>0.458786</td>\n",
              "      <td>0.541504</td>\n",
              "      <td>1.229292</td>\n",
              "      <td>0.177793</td>\n",
              "      <td>0.473869</td>\n",
              "      <td>0.738335</td>\n",
              "      <td>1.672910</td>\n",
              "      <td>1.059495</td>\n",
              "      <td>1.665207</td>\n",
              "      <td>3.663445</td>\n",
              "      <td>3.391498</td>\n",
              "      <td>3.579316</td>\n",
              "      <td>2.991586</td>\n",
              "      <td>0.399030</td>\n",
              "      <td>0.528706</td>\n",
              "      <td>1.105328</td>\n",
              "      <td>0.417550</td>\n",
              "      <td>2.416811</td>\n",
              "      <td>0.737759</td>\n",
              "      <td>0.500821</td>\n",
              "      <td>2.068553</td>\n",
              "      <td>2.782120</td>\n",
              "      <td>1.729464</td>\n",
              "      <td>0.793354</td>\n",
              "      <td>0.362697</td>\n",
              "      <td>0.122071</td>\n",
              "      <td>0.626887</td>\n",
              "      <td>1.287499</td>\n",
              "      <td>2.118650</td>\n",
              "      <td>1.572042</td>\n",
              "      <td>-0.011347</td>\n",
              "      <td>1.154391</td>\n",
              "      <td>...</td>\n",
              "      <td>3.487177</td>\n",
              "      <td>-0.038712</td>\n",
              "      <td>1.532049</td>\n",
              "      <td>5.176255</td>\n",
              "      <td>0.576355</td>\n",
              "      <td>0.277724</td>\n",
              "      <td>0.045875</td>\n",
              "      <td>1.171996</td>\n",
              "      <td>0.056506</td>\n",
              "      <td>1.627237</td>\n",
              "      <td>2.119013</td>\n",
              "      <td>0.540326</td>\n",
              "      <td>0.587333</td>\n",
              "      <td>0.791465</td>\n",
              "      <td>1.663337</td>\n",
              "      <td>0.019029</td>\n",
              "      <td>2.753613</td>\n",
              "      <td>1.486589</td>\n",
              "      <td>0.874798</td>\n",
              "      <td>0.928583</td>\n",
              "      <td>3.031569</td>\n",
              "      <td>1.030969</td>\n",
              "      <td>1.573785</td>\n",
              "      <td>2.113194</td>\n",
              "      <td>2.120092</td>\n",
              "      <td>0.456212</td>\n",
              "      <td>3.905150</td>\n",
              "      <td>2.109267</td>\n",
              "      <td>0.163412</td>\n",
              "      <td>1.939983</td>\n",
              "      <td>2.508578</td>\n",
              "      <td>1.832005</td>\n",
              "      <td>2.196829</td>\n",
              "      <td>0.668947</td>\n",
              "      <td>1.430875</td>\n",
              "      <td>2.476204</td>\n",
              "      <td>-1.218595</td>\n",
              "      <td>1.408496</td>\n",
              "      <td>1.771635</td>\n",
              "      <td>1.495228</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.894914</td>\n",
              "      <td>-0.092731</td>\n",
              "      <td>-0.071086</td>\n",
              "      <td>-0.153370</td>\n",
              "      <td>0.607579</td>\n",
              "      <td>3.121587</td>\n",
              "      <td>0.844036</td>\n",
              "      <td>1.391994</td>\n",
              "      <td>-0.040986</td>\n",
              "      <td>0.064775</td>\n",
              "      <td>0.618743</td>\n",
              "      <td>0.037675</td>\n",
              "      <td>-0.331725</td>\n",
              "      <td>0.779259</td>\n",
              "      <td>0.774952</td>\n",
              "      <td>0.026932</td>\n",
              "      <td>0.314524</td>\n",
              "      <td>2.036401</td>\n",
              "      <td>2.183381</td>\n",
              "      <td>1.420304</td>\n",
              "      <td>1.639731</td>\n",
              "      <td>-0.117252</td>\n",
              "      <td>-0.253789</td>\n",
              "      <td>0.263450</td>\n",
              "      <td>0.381263</td>\n",
              "      <td>0.952063</td>\n",
              "      <td>0.144212</td>\n",
              "      <td>0.033835</td>\n",
              "      <td>0.410589</td>\n",
              "      <td>0.832148</td>\n",
              "      <td>0.306754</td>\n",
              "      <td>0.061445</td>\n",
              "      <td>0.333529</td>\n",
              "      <td>-0.116311</td>\n",
              "      <td>0.579957</td>\n",
              "      <td>1.402147</td>\n",
              "      <td>0.230994</td>\n",
              "      <td>1.575549</td>\n",
              "      <td>-0.115456</td>\n",
              "      <td>0.396768</td>\n",
              "      <td>...</td>\n",
              "      <td>1.311270</td>\n",
              "      <td>0.203133</td>\n",
              "      <td>0.310089</td>\n",
              "      <td>1.649999</td>\n",
              "      <td>0.720692</td>\n",
              "      <td>-0.225939</td>\n",
              "      <td>-0.149671</td>\n",
              "      <td>0.555731</td>\n",
              "      <td>-0.062473</td>\n",
              "      <td>0.610019</td>\n",
              "      <td>1.250978</td>\n",
              "      <td>0.258769</td>\n",
              "      <td>0.213432</td>\n",
              "      <td>0.328823</td>\n",
              "      <td>2.397240</td>\n",
              "      <td>-0.493491</td>\n",
              "      <td>1.055446</td>\n",
              "      <td>0.830552</td>\n",
              "      <td>0.747009</td>\n",
              "      <td>0.701016</td>\n",
              "      <td>2.812340</td>\n",
              "      <td>-0.184432</td>\n",
              "      <td>2.111701</td>\n",
              "      <td>0.304915</td>\n",
              "      <td>1.519398</td>\n",
              "      <td>-0.068936</td>\n",
              "      <td>1.571269</td>\n",
              "      <td>0.132260</td>\n",
              "      <td>-0.058371</td>\n",
              "      <td>2.277476</td>\n",
              "      <td>2.868166</td>\n",
              "      <td>0.359764</td>\n",
              "      <td>2.048927</td>\n",
              "      <td>-0.693193</td>\n",
              "      <td>1.512298</td>\n",
              "      <td>1.680663</td>\n",
              "      <td>0.940717</td>\n",
              "      <td>0.817510</td>\n",
              "      <td>2.109111</td>\n",
              "      <td>0.809867</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.835231</td>\n",
              "      <td>0.176799</td>\n",
              "      <td>0.135655</td>\n",
              "      <td>-0.103934</td>\n",
              "      <td>0.244541</td>\n",
              "      <td>2.452694</td>\n",
              "      <td>-0.046309</td>\n",
              "      <td>0.305062</td>\n",
              "      <td>0.118281</td>\n",
              "      <td>-0.349651</td>\n",
              "      <td>0.364915</td>\n",
              "      <td>0.166561</td>\n",
              "      <td>-0.138204</td>\n",
              "      <td>0.425359</td>\n",
              "      <td>-0.267285</td>\n",
              "      <td>-0.400672</td>\n",
              "      <td>-0.010952</td>\n",
              "      <td>0.602568</td>\n",
              "      <td>1.522893</td>\n",
              "      <td>0.051047</td>\n",
              "      <td>-0.466322</td>\n",
              "      <td>-0.073531</td>\n",
              "      <td>-0.250858</td>\n",
              "      <td>-0.004646</td>\n",
              "      <td>0.083104</td>\n",
              "      <td>0.355020</td>\n",
              "      <td>0.321876</td>\n",
              "      <td>0.010884</td>\n",
              "      <td>0.450327</td>\n",
              "      <td>1.284789</td>\n",
              "      <td>0.325924</td>\n",
              "      <td>-0.270504</td>\n",
              "      <td>0.249414</td>\n",
              "      <td>0.187452</td>\n",
              "      <td>0.237916</td>\n",
              "      <td>0.576842</td>\n",
              "      <td>-0.014429</td>\n",
              "      <td>0.662791</td>\n",
              "      <td>-0.148310</td>\n",
              "      <td>2.446762</td>\n",
              "      <td>...</td>\n",
              "      <td>0.280190</td>\n",
              "      <td>0.180126</td>\n",
              "      <td>0.344350</td>\n",
              "      <td>-0.196989</td>\n",
              "      <td>0.112100</td>\n",
              "      <td>0.036494</td>\n",
              "      <td>0.135883</td>\n",
              "      <td>0.440409</td>\n",
              "      <td>0.094003</td>\n",
              "      <td>-0.093739</td>\n",
              "      <td>0.278460</td>\n",
              "      <td>-0.085237</td>\n",
              "      <td>-0.240761</td>\n",
              "      <td>0.069985</td>\n",
              "      <td>0.562625</td>\n",
              "      <td>0.258966</td>\n",
              "      <td>0.128109</td>\n",
              "      <td>0.696193</td>\n",
              "      <td>0.370148</td>\n",
              "      <td>0.452293</td>\n",
              "      <td>0.896718</td>\n",
              "      <td>0.020287</td>\n",
              "      <td>0.741429</td>\n",
              "      <td>0.035030</td>\n",
              "      <td>0.439975</td>\n",
              "      <td>0.079905</td>\n",
              "      <td>-0.522108</td>\n",
              "      <td>0.628086</td>\n",
              "      <td>0.031336</td>\n",
              "      <td>2.302958</td>\n",
              "      <td>1.877882</td>\n",
              "      <td>-0.518013</td>\n",
              "      <td>1.039225</td>\n",
              "      <td>-0.445847</td>\n",
              "      <td>0.859887</td>\n",
              "      <td>0.517374</td>\n",
              "      <td>0.465730</td>\n",
              "      <td>0.291600</td>\n",
              "      <td>2.158124</td>\n",
              "      <td>-1.147367</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>-0.005933</td>\n",
              "      <td>0.141640</td>\n",
              "      <td>0.076914</td>\n",
              "      <td>0.209794</td>\n",
              "      <td>0.457974</td>\n",
              "      <td>2.790473</td>\n",
              "      <td>0.130251</td>\n",
              "      <td>0.405387</td>\n",
              "      <td>0.105704</td>\n",
              "      <td>0.138638</td>\n",
              "      <td>0.272641</td>\n",
              "      <td>0.223936</td>\n",
              "      <td>-0.114563</td>\n",
              "      <td>0.538392</td>\n",
              "      <td>-0.213365</td>\n",
              "      <td>-0.298493</td>\n",
              "      <td>0.024003</td>\n",
              "      <td>0.768882</td>\n",
              "      <td>0.185059</td>\n",
              "      <td>0.101214</td>\n",
              "      <td>0.353634</td>\n",
              "      <td>0.048443</td>\n",
              "      <td>0.105409</td>\n",
              "      <td>0.092231</td>\n",
              "      <td>0.171479</td>\n",
              "      <td>0.251265</td>\n",
              "      <td>0.184331</td>\n",
              "      <td>0.065643</td>\n",
              "      <td>0.679358</td>\n",
              "      <td>1.594514</td>\n",
              "      <td>0.368817</td>\n",
              "      <td>-0.269517</td>\n",
              "      <td>0.313365</td>\n",
              "      <td>0.017324</td>\n",
              "      <td>0.230821</td>\n",
              "      <td>0.685795</td>\n",
              "      <td>-0.081723</td>\n",
              "      <td>0.942186</td>\n",
              "      <td>0.075291</td>\n",
              "      <td>0.969324</td>\n",
              "      <td>...</td>\n",
              "      <td>0.160051</td>\n",
              "      <td>0.066241</td>\n",
              "      <td>0.259483</td>\n",
              "      <td>0.199997</td>\n",
              "      <td>0.164557</td>\n",
              "      <td>-0.005609</td>\n",
              "      <td>0.082995</td>\n",
              "      <td>0.126856</td>\n",
              "      <td>0.095371</td>\n",
              "      <td>-0.232842</td>\n",
              "      <td>-0.137198</td>\n",
              "      <td>-0.030546</td>\n",
              "      <td>0.034345</td>\n",
              "      <td>0.133730</td>\n",
              "      <td>0.810954</td>\n",
              "      <td>0.167301</td>\n",
              "      <td>0.465687</td>\n",
              "      <td>0.323077</td>\n",
              "      <td>0.467235</td>\n",
              "      <td>0.485481</td>\n",
              "      <td>0.639930</td>\n",
              "      <td>-0.129197</td>\n",
              "      <td>0.836729</td>\n",
              "      <td>0.168541</td>\n",
              "      <td>0.402371</td>\n",
              "      <td>0.096907</td>\n",
              "      <td>0.048056</td>\n",
              "      <td>0.007177</td>\n",
              "      <td>0.084540</td>\n",
              "      <td>1.284753</td>\n",
              "      <td>1.376774</td>\n",
              "      <td>-0.167675</td>\n",
              "      <td>1.334417</td>\n",
              "      <td>-0.529334</td>\n",
              "      <td>1.032968</td>\n",
              "      <td>0.651285</td>\n",
              "      <td>0.646068</td>\n",
              "      <td>-0.234083</td>\n",
              "      <td>0.370818</td>\n",
              "      <td>-0.210490</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.057611</td>\n",
              "      <td>-0.054667</td>\n",
              "      <td>0.036791</td>\n",
              "      <td>0.559091</td>\n",
              "      <td>0.602180</td>\n",
              "      <td>3.332881</td>\n",
              "      <td>1.133493</td>\n",
              "      <td>0.413266</td>\n",
              "      <td>0.024486</td>\n",
              "      <td>-0.378158</td>\n",
              "      <td>1.351953</td>\n",
              "      <td>-0.048798</td>\n",
              "      <td>-0.039464</td>\n",
              "      <td>0.733045</td>\n",
              "      <td>0.558456</td>\n",
              "      <td>0.239070</td>\n",
              "      <td>1.195875</td>\n",
              "      <td>3.026044</td>\n",
              "      <td>-0.143045</td>\n",
              "      <td>-1.669538</td>\n",
              "      <td>0.549019</td>\n",
              "      <td>-0.059388</td>\n",
              "      <td>1.428677</td>\n",
              "      <td>0.780960</td>\n",
              "      <td>0.134476</td>\n",
              "      <td>0.170390</td>\n",
              "      <td>3.511240</td>\n",
              "      <td>0.811889</td>\n",
              "      <td>1.423887</td>\n",
              "      <td>3.137265</td>\n",
              "      <td>1.258200</td>\n",
              "      <td>-0.118721</td>\n",
              "      <td>0.206144</td>\n",
              "      <td>-0.007862</td>\n",
              "      <td>0.141090</td>\n",
              "      <td>0.844985</td>\n",
              "      <td>0.805467</td>\n",
              "      <td>0.958085</td>\n",
              "      <td>0.449198</td>\n",
              "      <td>2.103651</td>\n",
              "      <td>...</td>\n",
              "      <td>1.851180</td>\n",
              "      <td>0.093355</td>\n",
              "      <td>1.299100</td>\n",
              "      <td>2.193886</td>\n",
              "      <td>-0.001891</td>\n",
              "      <td>-0.091884</td>\n",
              "      <td>-0.116470</td>\n",
              "      <td>1.492557</td>\n",
              "      <td>-0.071569</td>\n",
              "      <td>-0.429140</td>\n",
              "      <td>3.284227</td>\n",
              "      <td>-0.435802</td>\n",
              "      <td>-0.423406</td>\n",
              "      <td>-0.313958</td>\n",
              "      <td>0.543579</td>\n",
              "      <td>0.934421</td>\n",
              "      <td>0.854894</td>\n",
              "      <td>-0.280835</td>\n",
              "      <td>1.100163</td>\n",
              "      <td>0.832717</td>\n",
              "      <td>2.924837</td>\n",
              "      <td>0.034789</td>\n",
              "      <td>1.065256</td>\n",
              "      <td>0.661926</td>\n",
              "      <td>1.473646</td>\n",
              "      <td>0.182405</td>\n",
              "      <td>0.188328</td>\n",
              "      <td>1.601696</td>\n",
              "      <td>-0.144806</td>\n",
              "      <td>3.175701</td>\n",
              "      <td>-0.477687</td>\n",
              "      <td>-0.254032</td>\n",
              "      <td>2.508251</td>\n",
              "      <td>2.747775</td>\n",
              "      <td>1.319326</td>\n",
              "      <td>-0.241857</td>\n",
              "      <td>1.135724</td>\n",
              "      <td>0.941970</td>\n",
              "      <td>3.116881</td>\n",
              "      <td>2.827764</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 595 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "user_id       1         2         3    ...       607       608       610\n",
              "1        3.451891  0.058611  0.007331  ...  1.408496  1.771635  1.495228\n",
              "2        0.894914 -0.092731 -0.071086  ...  0.817510  2.109111  0.809867\n",
              "3        0.835231  0.176799  0.135655  ...  0.291600  2.158124 -1.147367\n",
              "4       -0.005933  0.141640  0.076914  ... -0.234083  0.370818 -0.210490\n",
              "5        1.057611 -0.054667  0.036791  ...  0.941970  3.116881  2.827764\n",
              "\n",
              "[5 rows x 595 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TaYJcSLdnJaH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "U, sigma, Vt = svds(R_demeaned, k = 50, maxiter= 500)\n",
        "sigma = np.diag(sigma)\n",
        "all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt) + Ymean[:,np.newaxis]\n",
        "preds_df = pd.DataFrame(all_user_predicted_ratings, columns = R_df.columns)\n",
        "preds_df.index +=1\n",
        "# RMSE, MAE = accuracy(pivot_test, preds_df, K)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bOouUUXpyNv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# flattened_preds = pd.DataFrame(preds_df.to_records())\n",
        "preds_df['movies'] = preds_df.index\n",
        "preds_melted = pd.melt(preds_df, id_vars=['movies'], value_vars=preds_df.columns[:595])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aHhcUpQgy6nt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "preds_melted.head()\n",
        "preds_melted.columns = ['movie_id', 'user_id', 'rating_predicted']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uVBewlaxxHNV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "test_df = pd.merge(test_df, preds_melted,  how='inner', left_on=['user_id','movie_id'], right_on = ['user_id','movie_id'],)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a_4n8sld04U6",
        "colab_type": "code",
        "outputId": "b00e168b-5683-4510-831d-c8979d253c19",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "test_df.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3717, 9)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 127
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cRQPFM4FymUX",
        "colab_type": "code",
        "outputId": "e9097c01-4c5c-4026-c172-e9ed96ca409f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "sum(test_df.rating_predicted.isna())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f4TAkr_AoG6s",
        "colab_type": "code",
        "outputId": "09aaca05-4f91-430d-88b3-6416b65c915a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "from sklearn.metrics import mean_absolute_error\n",
        "\n",
        "mean_absolute_error(test_df.rating, test_df.rating_predicted)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3.1374667166646915"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 128
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jd9D7fB998iH",
        "colab": {}
      },
      "source": [
        "for K in range(1,2):\n",
        "    U, sigma, Vt = svds(R_demeaned, k = K, maxiter= 500)\n",
        "    sigma = np.diag(sigma)\n",
        "    all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt) + Ymean[:,np.newaxis]\n",
        "    preds_df = pd.DataFrame(all_user_predicted_ratings, columns = R_df.columns)\n",
        "    RMSE, MAE = accuracy(pivot_test, preds_df, K)\n",
        "    results.append([RMSE, MAE, K])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D-dRWfMXUf5v",
        "colab_type": "text"
      },
      "source": [
        "**The results are pretty bad (too bad, random model givs 1.5 RMSE)..**\n",
        "\n",
        "We think there's a normalization factor we didn't account for, since the recomendations are okay."
      ]
    }
  ]
}