examples/06_benchmarks/movielens.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmark with Movielens dataset\n",
    "\n",
    "This illustrative comparison applies to collaborative filtering algorithms available in this repository such as Spark ALS, Surprise SVD, SAR and others using the Movielens dataset. These algorithms are usable in a variety of recommendation tasks, including product or news recommendations.\n",
    "\n",
    "The main purpose of this notebook is not to produce comprehensive benchmarking results on multiple datasets. Rather, it is intended to illustrate on how one could evaluate different recommender algorithms using tools in this repository.\n",
    "\n",
    "## Experimentation setup:\n",
    "\n",
    "* Objective\n",
    "  * To compare how each collaborative filtering algorithm perform in predicting ratings and recommending relevant items.\n",
    "\n",
    "* Environment\n",
    "  * The comparison is run on a [Azure Data Science Virtual Machine](https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/). \n",
    "  * The virtual machine size is Standard NC6 (6 vcpus, 55 GB memory, 1K80 GPU).\n",
    "  * It should be noted that the single node DSVM is not supposed to run scalable benchmarking analysis. Either scaling up or out the computing instances is necessary to run the benchmarking in an run-time efficient way without any memory issue.\n",
    "  * **NOTE ABOUT THE DEPENDENCIES TO INSTALL**: This notebook uses CPU, GPU and PySpark algorithms, so make sure you install the `full environment` as detailed in the [SETUP.md](../../SETUP.md). \n",
    "  \n",
    "* Datasets\n",
    "  * [Movielens 100K](https://grouplens.org/datasets/movielens/100k/).\n",
    "  * [Movielens 1M](https://grouplens.org/datasets/movielens/1m/).\n",
    "\n",
    "* Data split\n",
    "  * The data is split into train and test sets.\n",
    "  * The split ratios are 75-25 for train and test datasets.\n",
    "  * The splitting is stratified based on items. \n",
    "\n",
    "* Model training\n",
    "  * A recommendation model is trained by using each of the collaborative filtering algorithms. \n",
    "  * Empirical parameter values reported [here](http://mymedialite.net/examples/datasets.html) are used in this notebook.  More exhaustive hyper parameter tuning would be required to further optimize results.\n",
    "\n",
    "* Evaluation metrics\n",
    "  * Ranking metrics:\n",
    "    * Precision@k.\n",
    "    * Recall@k.\n",
    "    * Normalized discounted cumulative gain@k (NDCG@k).\n",
    "    * Mean-average-precision (MAP). \n",
    "    * In the evaluation metrics above, k = 10. \n",
    "  * Rating metrics:\n",
    "    * Root mean squared error (RMSE).\n",
    "    * Mean average error (MAE).\n",
    "    * R squared.\n",
    "    * Explained variance.\n",
    "  * Run time performance\n",
    "    * Elapsed for training a model and using a model for predicting/recommending k items. \n",
    "    * The time may vary across different machines. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Globals settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.11 | packaged by conda-forge | (default, Nov 27 2020, 18:57:37) \n",
      "[GCC 9.3.0]\n",
      "Pandas version: 1.1.5\n",
      "PySpark version: 2.4.5\n",
      "Surprise version: 1.1.1\n",
      "PyTorch version: 1.4.0\n",
      "Fast AI version: 1.0.46\n",
      "Cornac version: 1.12.0\n",
      "Tensorflow version: 1.15.4\n",
      "CUDA version: CUDA Version 10.2.89\n",
      "CuDNN version: No CUDNN in this machine\n",
      "Number of cores: 48\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    
    "import os\n",
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import pyspark\n",
    "import torch\n",
    "import fastai\n",
    "import tensorflow as tf\n",
    "tf.get_logger().setLevel('ERROR') # only show error messages\n",
    "import surprise\n",
    "\n",
    "from recommenders.utils.general_utils import get_number_processors\n",
    "from recommenders.utils.gpu_utils import get_cuda_version, get_cudnn_version\n",
    "from recommenders.datasets import movielens\n",
    "from recommenders.datasets.python_splitters import python_stratified_split\n",
    "\n",
    "from benchmark_utils import * \n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Pandas version: {}\".format(pd.__version__))\n",
    "print(\"PySpark version: {}\".format(pyspark.__version__))\n",
    "print(\"Surprise version: {}\".format(surprise.__version__))\n",
    "print(\"PyTorch version: {}\".format(torch.__version__))\n",
    "print(\"Fast AI version: {}\".format(fastai.__version__))\n",
    "print(\"Cornac version: {}\".format(cornac.__version__))\n",
    "print(\"Tensorflow version: {}\".format(tf.__version__))\n",
    "print(\"CUDA version: {}\".format(get_cuda_version()))\n",
    "print(\"CuDNN version: {}\".format(get_cudnn_version()))\n",
    "n_cores = get_number_processors()\n",
    "print(\"Number of cores: {}\".format(n_cores))\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hide fastai progress bar\n",
    "hide_fastai_progress_bar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fix random seeds to make sure out runs are reproducible\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.cuda.manual_seed_all(SEED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "environments = {\n",
    "    \"als\": \"pyspark\",\n",
    "    \"sar\": \"python_cpu\",\n",
    "    \"svd\": \"python_cpu\",\n",
    "    \"fastai\": \"python_gpu\",\n",
    "    \"ncf\": \"python_gpu\",\n",
    "    \"bpr\": \"python_cpu\",\n",
    "    \"bivae\": \"python_gpu\",\n",
    "    \"lightgcn\": \"python_gpu\",\n",
    "}\n",
    "\n",
    "metrics = {\n",
    "    \"als\": [\"rating\", \"ranking\"],\n",
    "    \"sar\": [\"ranking\"],\n",
    "    \"svd\": [\"rating\", \"ranking\"],\n",
    "    \"fastai\": [\"rating\", \"ranking\"],\n",
    "    \"ncf\": [\"ranking\"],\n",
    "    \"bpr\": [\"ranking\"],\n",
    "    \"bivae\": [\"ranking\"],\n",
    "    \"lightgcn\": [\"ranking\"]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Algorithm parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "als_params = {\n",
    "    \"rank\": 10,\n",
    "    \"maxIter\": 15,\n",
    "    \"implicitPrefs\": False,\n",
    "    \"alpha\": 0.1,\n",
    "    \"regParam\": 0.05,\n",
    "    \"coldStartStrategy\": \"drop\",\n",
    "    \"nonnegative\": False,\n",
    "    \"userCol\": DEFAULT_USER_COL,\n",
    "    \"itemCol\": DEFAULT_ITEM_COL,\n",
    "    \"ratingCol\": DEFAULT_RATING_COL,\n",
    "}\n",
    "\n",
    "sar_params = {\n",
    "    \"similarity_type\": \"jaccard\",\n",
    "    \"time_decay_coefficient\": 30,\n",
    "    \"time_now\": None,\n",
    "    \"timedecay_formula\": True,\n",
    "    \"col_user\": DEFAULT_USER_COL,\n",
    "    \"col_item\": DEFAULT_ITEM_COL,\n",
    "    \"col_rating\": DEFAULT_RATING_COL,\n",
    "    \"col_timestamp\": DEFAULT_TIMESTAMP_COL,\n",
    "}\n",
    "\n",
    "svd_params = {\n",
    "    \"n_factors\": 150,\n",
    "    \"n_epochs\": 15,\n",
    "    \"lr_all\": 0.005,\n",
    "    \"reg_all\": 0.02,\n",
    "    \"random_state\": SEED,\n",
    "    \"verbose\": False\n",
    "}\n",
    "\n",
    "fastai_params = {\n",
    "    \"n_factors\": 40, \n",
    "    \"y_range\": [0,5.5], \n",
    "    \"wd\": 1e-1,\n",
    "    \"max_lr\": 5e-3,\n",
    "    \"epochs\": 15\n",
    "}\n",
    "\n",
    "ncf_params = {\n",
    "    \"model_type\": \"NeuMF\",\n",
    "    \"n_factors\": 4,\n",
    "    \"layer_sizes\": [16, 8, 4],\n",
    "    \"n_epochs\": 15,\n",
    "    \"batch_size\": 1024,\n",
    "    \"learning_rate\": 1e-3,\n",
    "    \"verbose\": 10\n",
    "}\n",
    "\n",
    "bpr_params = {\n",
    "    \"k\": 200,\n",
    "    \"max_iter\": 200,\n",
    "    \"learning_rate\": 0.01,\n",
    "    \"lambda_reg\": 1e-3,\n",
    "    \"seed\": SEED,\n",
    "    \"verbose\": False\n",
    "}\n",
    "\n",
    "bivae_params = {\n",
    "    \"k\": 100,\n",
    "    \"encoder_structure\": [200],\n",
    "    \"act_fn\": \"tanh\",\n",
    "    \"likelihood\": \"pois\",\n",
    "    \"n_epochs\": 500,\n",
    "    \"batch_size\": 1024,\n",
    "    \"learning_rate\": 0.001,\n",
    "    \"seed\": SEED,\n",
    "    \"use_gpu\": True,\n",
    "    \"verbose\": False\n",
    "}\n",
    "\n",
    "lightgcn_param = {\n",
    "    \"yaml_file\": os.path.join(\"..\",\"..\",\"recommenders\", \"recommender\", \"deeprec\", \"config\", \"lightgcn.yaml\"),\n",
    "    \"n_layers\": 3,\n",
    "    \"batch_size\": 1024,\n",
    "    \"epochs\": 15,\n",
    "    \"learning_rate\": 0.005,\n",
    "    \"eval_epoch\": 5,\n",
    "    \"top_k\": DEFAULT_K,\n",
    "}\n",
    "\n",
    "params = {\n",
    "    \"als\": als_params,\n",
    "    \"sar\": sar_params,\n",
    "    \"svd\": svd_params,\n",
    "    \"fastai\": fastai_params,\n",
    "    \"ncf\": ncf_params,\n",
    "    \"bpr\": bpr_params,\n",
    "    \"bivae\": bivae_params,\n",
    "    \"lightgcn\": lightgcn_param,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "prepare_training_data = {\n",
    "    \"als\": prepare_training_als,\n",
    "    \"sar\": prepare_training_sar,\n",
    "    \"svd\": prepare_training_svd,\n",
    "    \"fastai\": prepare_training_fastai,\n",
    "    \"ncf\": prepare_training_ncf,\n",
    "    \"bpr\": prepare_training_cornac,\n",
    "    \"bivae\": prepare_training_cornac,\n",
    "    \"lightgcn\": prepare_training_lightgcn,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "prepare_metrics_data = {\n",
    "    \"als\": lambda train, test: prepare_metrics_als(train, test),\n",
    "    \"fastai\": lambda train, test: prepare_metrics_fastai(train, test),    \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = {\n",
    "    \"als\": lambda params, data: train_als(params, data),\n",
    "    \"svd\": lambda params, data: train_svd(params, data),\n",
    "    \"sar\": lambda params, data: train_sar(params, data), \n",
    "    \"fastai\": lambda params, data: train_fastai(params, data),\n",
    "    \"ncf\": lambda params, data: train_ncf(params, data),\n",
    "    \"bpr\": lambda params, data: train_bpr(params, data),\n",
    "    \"bivae\": lambda params, data: train_bivae(params, data),\n",
    "    \"lightgcn\": lambda params, data: train_lightgcn(params, data),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "rating_predictor = {\n",
    "    \"als\": lambda model, test: predict_als(model, test),\n",
    "    \"svd\": lambda model, test: predict_svd(model, test),\n",
    "    \"fastai\": lambda model, test: predict_fastai(model, test),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranking_predictor = {\n",
    "    \"als\": lambda model, test, train: recommend_k_als(model, test, train),\n",
    "    \"sar\": lambda model, test, train: recommend_k_sar(model, test, train),\n",
    "    \"svd\": lambda model, test, train: recommend_k_svd(model, test, train),\n",
    "    \"fastai\": lambda model, test, train: recommend_k_fastai(model, test, train),\n",
    "    \"ncf\": lambda model, test, train: recommend_k_ncf(model, test, train),\n",
    "    \"bpr\": lambda model, test, train: recommend_k_cornac(model, test, train),\n",
    "    \"bivae\": lambda model, test, train: recommend_k_cornac(model, test, train),\n",
    "    \"lightgcn\": lambda model, test, train: recommend_k_lightgcn(model, test, train),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "rating_evaluator = {\n",
    "    \"als\": lambda test, predictions: rating_metrics_pyspark(test, predictions),\n",
    "    \"svd\": lambda test, predictions: rating_metrics_python(test, predictions),\n",
    "    \"fastai\": lambda test, predictions: rating_metrics_python(test, predictions)\n",
    "}\n",
    "    \n",
    "    \n",
    "ranking_evaluator = {\n",
    "    \"als\": lambda test, predictions, k: ranking_metrics_pyspark(test, predictions, k),\n",
    "    \"sar\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n",
    "    \"svd\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n",
    "    \"fastai\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n",
    "    \"ncf\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n",
    "    \"bpr\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n",
    "    \"bivae\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n",
    "    \"lightgcn\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_summary(data, algo, k, train_time, time_rating, rating_metrics, time_ranking, ranking_metrics):\n",
    "    summary = {\"Data\": data, \"Algo\": algo, \"K\": k, \"Train time (s)\": train_time, \"Predicting time (s)\": time_rating, \"Recommending time (s)\": time_ranking}\n",
    "    if rating_metrics is None:\n",
    "        rating_metrics = {\n",
    "            \"RMSE\": np.nan,\n",
    "            \"MAE\": np.nan,\n",
    "            \"R2\": np.nan,\n",
    "            \"Explained Variance\": np.nan,\n",
    "        }\n",
    "    if ranking_metrics is None:\n",
    "        ranking_metrics = {\n",
    "            \"MAP\": np.nan,\n",
    "            \"nDCG@k\": np.nan,\n",
    "            \"Precision@k\": np.nan,\n",
    "            \"Recall@k\": np.nan,\n",
    "        }\n",
    "    summary.update(rating_metrics)\n",
    "    summary.update(ranking_metrics)\n",
    "    return summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_sizes = [\"100k\", \"1m\"] # Movielens data size: 100k, 1m, 10m, or 20m\n",
    "algorithms = [\"als\", \"svd\", \"sar\", \"ncf\", \"fastai\", \"bpr\", \"bivae\", \"lightgcn\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4.81k/4.81k [00:01<00:00, 2.41kKB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of Movielens 100k: (100000, 4)\n",
      "\n",
      "Computing als algorithm on Movielens 100k\n",
      "Training time: 6.6739s\n",
      "Rating prediction time: 0.0716s\n",
      "Ranking prediction time: 0.0910s\n",
      "\n",
      "Computing svd algorithm on Movielens 100k\n",
      "Training time: 4.2722s\n",
      "Rating prediction time: 0.2882s\n",
      "Ranking prediction time: 14.9461s\n",
      "\n",
      "Computing sar algorithm on Movielens 100k\n",
      "Training time: 0.3608s\n",
      "Ranking prediction time: 0.0853s\n",
      "\n",
      "Computing ncf algorithm on Movielens 100k\n",
      "Training time: 63.8578s\n",
      "Ranking prediction time: 4.8731s\n",
      "\n",
      "Computing fastai algorithm on Movielens 100k\n",
      "Training time: 96.2693s\n",
      "Rating prediction time: 0.0533s\n",
      "Ranking prediction time: 3.2034s\n",
      "\n",
      "Computing bpr algorithm on Movielens 100k\n",
      "Training time: 6.6029s\n",
      "Ranking prediction time: 1.5408s\n",
      "\n",
      "Computing bivae algorithm on Movielens 100k\n",
      "Training time: 10.4133s\n",
      "Ranking prediction time: 1.5687s\n",
      "\n",
      "Computing lightgcn algorithm on Movielens 100k\n",
      "Already create adjacency matrix.\n",
      "Already normalize adjacency matrix.\n",
      "Using xavier initialization.\n",
      "Epoch 1 (train)1.5s: train loss = 0.47708 = (mf)0.47684 + (embed)0.00024\n",
      "Epoch 2 (train)0.9s: train loss = 0.28938 = (mf)0.28875 + (embed)0.00063\n",
      "Epoch 3 (train)0.9s: train loss = 0.25478 = (mf)0.25398 + (embed)0.00080\n",
      "Epoch 4 (train)0.9s: train loss = 0.23985 = (mf)0.23888 + (embed)0.00097\n",
      "Epoch 5 (train)0.8s + (eval)0.2s: train loss = 0.23025 = (mf)0.22914 + (embed)0.00110, recall = 0.16051, ndcg = 0.34891, precision = 0.30297, map = 0.09295\n",
      "Epoch 6 (train)0.9s: train loss = 0.22287 = (mf)0.22165 + (embed)0.00122\n",
      "Epoch 7 (train)0.9s: train loss = 0.21368 = (mf)0.21236 + (embed)0.00132\n",
      "Epoch 8 (train)0.9s: train loss = 0.20252 = (mf)0.20107 + (embed)0.00145\n",
      "Epoch 9 (train)0.9s: train loss = 0.19008 = (mf)0.18847 + (embed)0.00161\n",
      "Epoch 10 (train)0.9s + (eval)0.2s: train loss = 0.18022 = (mf)0.17844 + (embed)0.00178, recall = 0.18071, ndcg = 0.39217, precision = 0.34019, map = 0.10928\n",
      "Epoch 11 (train)0.9s: train loss = 0.17680 = (mf)0.17487 + (embed)0.00194\n",
      "Epoch 12 (train)1.0s: train loss = 0.17439 = (mf)0.17233 + (embed)0.00206\n",
      "Epoch 13 (train)0.9s: train loss = 0.16669 = (mf)0.16450 + (embed)0.00219\n",
      "Epoch 14 (train)0.9s: train loss = 0.16579 = (mf)0.16349 + (embed)0.00230\n",
      "Epoch 15 (train)0.9s + (eval)0.2s: train loss = 0.16282 = (mf)0.16041 + (embed)0.00241, recall = 0.19053, ndcg = 0.40763, precision = 0.35292, map = 0.11737\n",
      "Training time: 14.7697s\n",
      "Ranking prediction time: 0.0577s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5.78k/5.78k [00:01<00:00, 2.89kKB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of Movielens 1m: (1000209, 4)\n",
      "\n",
      "Computing als algorithm on Movielens 1m\n",
      "Training time: 3.3558s\n",
      "Rating prediction time: 0.0262s\n",
      "Ranking prediction time: 0.0547s\n",
      "\n",
      "Computing svd algorithm on Movielens 1m\n",
      "Training time: 43.4842s\n",
      "Rating prediction time: 3.4605s\n",
      "Ranking prediction time: 210.9069s\n",
      "\n",
      "Computing sar algorithm on Movielens 1m\n",
      "Training time: 3.3966s\n",
      "Ranking prediction time: 2.1385s\n",
      "\n",
      "Computing ncf algorithm on Movielens 1m\n",
      "Training time: 818.8540s\n",
      "Ranking prediction time: 52.5052s\n",
      "\n",
      "Computing fastai algorithm on Movielens 1m\n",
      "Training time: 800.6882s\n",
      "Rating prediction time: 0.4382s\n",
      "Ranking prediction time: 54.6923s\n",
      "\n",
      "Computing bpr algorithm on Movielens 1m\n",
      "Training time: 70.1201s\n",
      "Ranking prediction time: 30.3135s\n",
      "\n",
      "Computing bivae algorithm on Movielens 1m\n",
      "Training time: 157.1440s\n",
      "Ranking prediction time: 29.6179s\n",
      "\n",
      "Computing lightgcn algorithm on Movielens 1m\n",
      "Already create adjacency matrix.\n",
      "Already normalize adjacency matrix.\n",
      "Using xavier initialization.\n",
      "Epoch 1 (train)33.9s: train loss = 0.34573 = (mf)0.34513 + (embed)0.00060\n",
      "Epoch 2 (train)27.2s: train loss = 0.26890 = (mf)0.26748 + (embed)0.00142\n",
      "Epoch 3 (train)27.2s: train loss = 0.22710 = (mf)0.22481 + (embed)0.00228\n",
      "Epoch 4 (train)27.2s: train loss = 0.20468 = (mf)0.20173 + (embed)0.00295\n",
      "Epoch 5 (train)28.2s + (eval)2.3s: train loss = 0.18749 = (mf)0.18387 + (embed)0.00362, recall = 0.12346, ndcg = 0.37514, precision = 0.33876, map = 0.07333\n",
      "Epoch 6 (train)28.0s: train loss = 0.17191 = (mf)0.16761 + (embed)0.00430\n",
      "Epoch 7 (train)27.3s: train loss = 0.16209 = (mf)0.15716 + (embed)0.00493\n",
      "Epoch 8 (train)28.0s: train loss = 0.15376 = (mf)0.14825 + (embed)0.00551\n",
      "Epoch 9 (train)27.6s: train loss = 0.14711 = (mf)0.14105 + (embed)0.00606\n",
      "Epoch 10 (train)25.7s + (eval)2.0s: train loss = 0.14080 = (mf)0.13424 + (embed)0.00657, recall = 0.13965, ndcg = 0.40881, precision = 0.37071, map = 0.08435\n",
      "Epoch 11 (train)25.5s: train loss = 0.13659 = (mf)0.12955 + (embed)0.00704\n",
      "Epoch 12 (train)26.1s: train loss = 0.13223 = (mf)0.12474 + (embed)0.00749\n",
      "Epoch 13 (train)26.4s: train loss = 0.12833 = (mf)0.12039 + (embed)0.00794\n",
      "Epoch 14 (train)25.7s: train loss = 0.12402 = (mf)0.11564 + (embed)0.00838\n",
      "Epoch 15 (train)27.1s + (eval)2.0s: train loss = 0.12167 = (mf)0.11285 + (embed)0.00882, recall = 0.14387, ndcg = 0.41690, precision = 0.37853, map = 0.08785\n",
      "Training time: 417.1986s\n",
      "Ranking prediction time: 0.5851s\n",
      "\n",
      "Computation finished\n",
      "CPU times: user 58min 40s, sys: 8min 2s, total: 1h 6min 43s\n",
      "Wall time: 1h 3min 3s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# For each data size and each algorithm, a recommender is evaluated. \n",
    "cols = [\"Data\", \"Algo\", \"K\", \"Train time (s)\", \"Predicting time (s)\", \"RMSE\", \"MAE\", \"R2\", \"Explained Variance\", \"Recommending time (s)\", \"MAP\", \"nDCG@k\", \"Precision@k\", \"Recall@k\"]\n",
    "df_results = pd.DataFrame(columns=cols)\n",
    "\n",
    "for data_size in data_sizes:\n",
    "    # Load the dataset\n",
    "    df = movielens.load_pandas_df(\n",
    "        size=data_size,\n",
    "        header=[DEFAULT_USER_COL, DEFAULT_ITEM_COL, DEFAULT_RATING_COL, DEFAULT_TIMESTAMP_COL]\n",
    "    )\n",
    "    print(\"Size of Movielens {}: {}\".format(data_size, df.shape))\n",
    "    \n",
    "    # Split the dataset\n",
    "    df_train, df_test = python_stratified_split(df,\n",
    "                                                ratio=0.75, \n",
    "                                                min_rating=1, \n",
    "                                                filter_by=\"item\", \n",
    "                                                col_user=DEFAULT_USER_COL, \n",
    "                                                col_item=DEFAULT_ITEM_COL\n",
    "                                                )\n",
    "   \n",
    "    # Loop through the algos\n",
    "    for algo in algorithms:\n",
    "        print(f\"\\nComputing {algo} algorithm on Movielens {data_size}\")\n",
    "          \n",
    "        # Data prep for training set\n",
    "        train = prepare_training_data.get(algo, lambda x,y:(x,y))(df_train, df_test)\n",
    "        \n",
    "        # Get model parameters\n",
    "        model_params = params[algo]\n",
    "          \n",
    "        # Train the model\n",
    "        model, time_train = trainer[algo](model_params, train)\n",
    "        print(f\"Training time: {time_train}s\")\n",
    "                \n",
    "        # Predict and evaluate\n",
    "        train, test = prepare_metrics_data.get(algo, lambda x,y:(x,y))(df_train, df_test)\n",
    "        \n",
    "        if \"rating\" in metrics[algo]:   \n",
    "            # Predict for rating\n",
    "            preds, time_rating = rating_predictor[algo](model, test)\n",
    "            print(f\"Rating prediction time: {time_rating}s\")\n",
    "            \n",
    "            # Evaluate for rating\n",
    "            ratings = rating_evaluator[algo](test, preds)\n",
    "        else:\n",
    "            ratings = None\n",
    "            time_rating = np.nan\n",
    "        \n",
    "        if \"ranking\" in metrics[algo]:\n",
    "            # Predict for ranking\n",
    "            top_k_scores, time_ranking = ranking_predictor[algo](model, test, train)\n",
    "            print(f\"Ranking prediction time: {time_ranking}s\")\n",
    "            \n",
    "            # Evaluate for rating\n",
    "            rankings = ranking_evaluator[algo](test, top_k_scores, DEFAULT_K)\n",
    "        else:\n",
    "            rankings = None\n",
    "            time_ranking = np.nan\n",
    "            \n",
    "        # Record results\n",
    "        summary = generate_summary(data_size, algo, DEFAULT_K, time_train, time_rating, ratings, time_ranking, rankings)\n",
    "        df_results.loc[df_results.shape[0] + 1] = summary\n",
    "        \n",
    "print(\"\\nComputation finished\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Data</th>\n",
       "      <th>Algo</th>\n",
       "      <th>K</th>\n",
       "      <th>Train time (s)</th>\n",
       "      <th>Predicting time (s)</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>MAE</th>\n",
       "      <th>R2</th>\n",
       "      <th>Explained Variance</th>\n",
       "      <th>Recommending time (s)</th>\n",
       "      <th>MAP</th>\n",
       "      <th>nDCG@k</th>\n",
       "      <th>Precision@k</th>\n",
       "      <th>Recall@k</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100k</td>\n",
       "      <td>als</td>\n",
       "      <td>10</td>\n",
       "      <td>6.6739</td>\n",
       "      <td>0.0716</td>\n",
       "      <td>0.963173</td>\n",
       "      <td>0.752521</td>\n",
       "      <td>0.258453</td>\n",
       "      <td>0.254536</td>\n",
       "      <td>0.0910</td>\n",
       "      <td>0.004091</td>\n",
       "      <td>0.041346</td>\n",
       "      <td>0.045387</td>\n",
       "      <td>0.015172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100k</td>\n",
       "      <td>svd</td>\n",
       "      <td>10</td>\n",
       "      <td>4.2722</td>\n",
       "      <td>0.2882</td>\n",
       "      <td>0.938681</td>\n",
       "      <td>0.742690</td>\n",
       "      <td>0.291967</td>\n",
       "      <td>0.291971</td>\n",
       "      <td>14.9461</td>\n",
       "      <td>0.012873</td>\n",
       "      <td>0.095930</td>\n",
       "      <td>0.091198</td>\n",
       "      <td>0.032783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100k</td>\n",
       "      <td>sar</td>\n",
       "      <td>10</td>\n",
       "      <td>0.3608</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0853</td>\n",
       "      <td>0.113028</td>\n",
       "      <td>0.388321</td>\n",
       "      <td>0.333828</td>\n",
       "      <td>0.183179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100k</td>\n",
       "      <td>ncf</td>\n",
       "      <td>10</td>\n",
       "      <td>63.8578</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.8731</td>\n",
       "      <td>0.094349</td>\n",
       "      <td>0.369862</td>\n",
       "      <td>0.330117</td>\n",
       "      <td>0.165953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>100k</td>\n",
       "      <td>fastai</td>\n",
       "      <td>10</td>\n",
       "      <td>96.2693</td>\n",
       "      <td>0.0533</td>\n",
       "      <td>0.943093</td>\n",
       "      <td>0.744332</td>\n",
       "      <td>0.285296</td>\n",
       "      <td>0.287658</td>\n",
       "      <td>3.2034</td>\n",
       "      <td>0.025590</td>\n",
       "      <td>0.148119</td>\n",
       "      <td>0.130541</td>\n",
       "      <td>0.053903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>100k</td>\n",
       "      <td>bpr</td>\n",
       "      <td>10</td>\n",
       "      <td>6.6029</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5408</td>\n",
       "      <td>0.132478</td>\n",
       "      <td>0.441997</td>\n",
       "      <td>0.388229</td>\n",
       "      <td>0.212522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>100k</td>\n",
       "      <td>bivae</td>\n",
       "      <td>10</td>\n",
       "      <td>10.4133</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5687</td>\n",
       "      <td>0.146126</td>\n",
       "      <td>0.475077</td>\n",
       "      <td>0.411771</td>\n",
       "      <td>0.219145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>100k</td>\n",
       "      <td>lightgcn</td>\n",
       "      <td>10</td>\n",
       "      <td>14.7697</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0577</td>\n",
       "      <td>0.117371</td>\n",
       "      <td>0.407634</td>\n",
       "      <td>0.352916</td>\n",
       "      <td>0.190530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1m</td>\n",
       "      <td>als</td>\n",
       "      <td>10</td>\n",
       "      <td>3.3558</td>\n",
       "      <td>0.0262</td>\n",
       "      <td>0.860522</td>\n",
       "      <td>0.679683</td>\n",
       "      <td>0.412199</td>\n",
       "      <td>0.406352</td>\n",
       "      <td>0.0547</td>\n",
       "      <td>0.002039</td>\n",
       "      <td>0.025166</td>\n",
       "      <td>0.031373</td>\n",
       "      <td>0.009830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1m</td>\n",
       "      <td>svd</td>\n",
       "      <td>10</td>\n",
       "      <td>43.4842</td>\n",
       "      <td>3.4605</td>\n",
       "      <td>0.883017</td>\n",
       "      <td>0.695366</td>\n",
       "      <td>0.374910</td>\n",
       "      <td>0.374911</td>\n",
       "      <td>210.9069</td>\n",
       "      <td>0.008828</td>\n",
       "      <td>0.089320</td>\n",
       "      <td>0.082856</td>\n",
       "      <td>0.021582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1m</td>\n",
       "      <td>sar</td>\n",
       "      <td>10</td>\n",
       "      <td>3.3966</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.1385</td>\n",
       "      <td>0.066214</td>\n",
       "      <td>0.313502</td>\n",
       "      <td>0.279692</td>\n",
       "      <td>0.111135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1m</td>\n",
       "      <td>ncf</td>\n",
       "      <td>10</td>\n",
       "      <td>818.8540</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>52.5052</td>\n",
       "      <td>0.063974</td>\n",
       "      <td>0.348130</td>\n",
       "      <td>0.318900</td>\n",
       "      <td>0.109476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1m</td>\n",
       "      <td>fastai</td>\n",
       "      <td>10</td>\n",
       "      <td>800.6882</td>\n",
       "      <td>0.4382</td>\n",
       "      <td>0.874729</td>\n",
       "      <td>0.695722</td>\n",
       "      <td>0.386589</td>\n",
       "      <td>0.389092</td>\n",
       "      <td>54.6923</td>\n",
       "      <td>0.026299</td>\n",
       "      <td>0.183996</td>\n",
       "      <td>0.167550</td>\n",
       "      <td>0.055682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1m</td>\n",
       "      <td>bpr</td>\n",
       "      <td>10</td>\n",
       "      <td>70.1201</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>30.3135</td>\n",
       "      <td>0.083546</td>\n",
       "      <td>0.390911</td>\n",
       "      <td>0.357777</td>\n",
       "      <td>0.142510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1m</td>\n",
       "      <td>bivae</td>\n",
       "      <td>10</td>\n",
       "      <td>157.1440</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29.6179</td>\n",
       "      <td>0.092682</td>\n",
       "      <td>0.433709</td>\n",
       "      <td>0.396058</td>\n",
       "      <td>0.148564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1m</td>\n",
       "      <td>lightgcn</td>\n",
       "      <td>10</td>\n",
       "      <td>417.1986</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.5851</td>\n",
       "      <td>0.087846</td>\n",
       "      <td>0.416899</td>\n",
       "      <td>0.378532</td>\n",
       "      <td>0.143865</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Data      Algo   K Train time (s) Predicting time (s)      RMSE       MAE  \\\n",
       "1   100k       als  10         6.6739              0.0716  0.963173  0.752521   \n",
       "2   100k       svd  10         4.2722              0.2882  0.938681  0.742690   \n",
       "3   100k       sar  10         0.3608                 NaN       NaN       NaN   \n",
       "4   100k       ncf  10        63.8578                 NaN       NaN       NaN   \n",
       "5   100k    fastai  10        96.2693              0.0533  0.943093  0.744332   \n",
       "6   100k       bpr  10         6.6029                 NaN       NaN       NaN   \n",
       "7   100k     bivae  10        10.4133                 NaN       NaN       NaN   \n",
       "8   100k  lightgcn  10        14.7697                 NaN       NaN       NaN   \n",
       "9     1m       als  10         3.3558              0.0262  0.860522  0.679683   \n",
       "10    1m       svd  10        43.4842              3.4605  0.883017  0.695366   \n",
       "11    1m       sar  10         3.3966                 NaN       NaN       NaN   \n",
       "12    1m       ncf  10       818.8540                 NaN       NaN       NaN   \n",
       "13    1m    fastai  10       800.6882              0.4382  0.874729  0.695722   \n",
       "14    1m       bpr  10        70.1201                 NaN       NaN       NaN   \n",
       "15    1m     bivae  10       157.1440                 NaN       NaN       NaN   \n",
       "16    1m  lightgcn  10       417.1986                 NaN       NaN       NaN   \n",
       "\n",
       "          R2  Explained Variance Recommending time (s)       MAP    nDCG@k  \\\n",
       "1   0.258453            0.254536                0.0910  0.004091  0.041346   \n",
       "2   0.291967            0.291971               14.9461  0.012873  0.095930   \n",
       "3        NaN                 NaN                0.0853  0.113028  0.388321   \n",
       "4        NaN                 NaN                4.8731  0.094349  0.369862   \n",
       "5   0.285296            0.287658                3.2034  0.025590  0.148119   \n",
       "6        NaN                 NaN                1.5408  0.132478  0.441997   \n",
       "7        NaN                 NaN                1.5687  0.146126  0.475077   \n",
       "8        NaN                 NaN                0.0577  0.117371  0.407634   \n",
       "9   0.412199            0.406352                0.0547  0.002039  0.025166   \n",
       "10  0.374910            0.374911              210.9069  0.008828  0.089320   \n",
       "11       NaN                 NaN                2.1385  0.066214  0.313502   \n",
       "12       NaN                 NaN               52.5052  0.063974  0.348130   \n",
       "13  0.386589            0.389092               54.6923  0.026299  0.183996   \n",
       "14       NaN                 NaN               30.3135  0.083546  0.390911   \n",
       "15       NaN                 NaN               29.6179  0.092682  0.433709   \n",
       "16       NaN                 NaN                0.5851  0.087846  0.416899   \n",
       "\n",
       "    Precision@k  Recall@k  \n",
       "1      0.045387  0.015172  \n",
       "2      0.091198  0.032783  \n",
       "3      0.333828  0.183179  \n",
       "4      0.330117  0.165953  \n",
       "5      0.130541  0.053903  \n",
       "6      0.388229  0.212522  \n",
       "7      0.411771  0.219145  \n",
       "8      0.352916  0.190530  \n",
       "9      0.031373  0.009830  \n",
       "10     0.082856  0.021582  \n",
       "11     0.279692  0.111135  \n",
       "12     0.318900  0.109476  \n",
       "13     0.167550  0.055682  \n",
       "14     0.357777  0.142510  \n",
       "15     0.396058  0.148564  \n",
       "16     0.378532  0.143865  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (reco_full)",
   "language": "python",
   "name": "reco_full"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}