recom.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_folder = \"data/\"\n",
    "\n",
    "user_id_path = data_folder + \"user_id_mappings.npy\"\n",
    "anime_id_path = data_folder + \"anime_id_mappings.npy\"\n",
    "anime_title_path = data_folder + \"anime_title_mappings.pickle\"\n",
    "pivoted_array_path = data_folder + \"pivoted_array.npy\"\n",
    "model_path = \"model\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_id_mappings = np.load(user_id_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "anime_id_mappings = np.load(anime_id_path, allow_pickle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(anime_title_path, 'rb') as f:\n",
    "  anime_title_mappings = pickle.load(f)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since computing the pivot matrix is extremely intensive, we compute it once and store it to disk, and simply load it here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "pivoted_array = np.load(pivoted_array_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cosine Similarity"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why and how we use cosine similarity?\n",
    "- Cosine similarity is our first choice because of the abundance of data in the given dataset.\n",
    "- We begin by asking the new user for 5 Anime ratings of their choice.\n",
    "- We use these 5 ratings to find the closest existing user in the database (*this is where cosine similarity is used*).\n",
    "- We assume that this closest existing user has rated at least 10 other Anime's in addition to the 5 given by the new user.\n",
    "- We rank these animes by the user score and use them as recommendations for the new user.\n",
    "- If the assumption is false, we resort to using ALS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transform_new_ratings(new_user_ratings):\n",
    "\tnew_user_array = np.zeros(pivoted_array.shape[1])\n",
    "\tfor anime_id, score in new_user_ratings:\n",
    "\t\tnew_user_array[anime_id - 1] = score\n",
    "  \n",
    "\treturn new_user_array"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "`cosine_similarity(u,v)`: calculates the cosine similarity measures the cosine of the angle between two vectors as a measure of similarity between these vectors in a high-dimensional space. It ranges from -1 to 1, where:\n",
    "  - 1 indicates that the vectors are identical.\n",
    "  - -1 indicates that the vectors are diametrically opposed (i.e., they point in opposite directions).\n",
    "  - 0 indicates orthogonality (i.e., the vectors are perpendicular to each other).\n",
    "  - Values between 0 and 1 represent varying degrees of similarity between the vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cosine_similarity(u, v):\n",
    "\tnorm_u = np.linalg.norm(u)\n",
    "\tnorm_v = np.linalg.norm(v)\n",
    "\tif norm_u == 0 or norm_v == 0:\n",
    "\t\treturn 0\n",
    "\treturn np.dot(u, v) / (norm_u * norm_v)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`find_closest_user(pivoted_array, new_user_array, new_user_only)`: finds the user most similar to the new user in terms of preferences by calculating the cosine similarity between the `new_user_only` vector and every other vector in the user-item association matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_closest_user(pivoted_array, new_user_array, new_user_only):\n",
    "\tsimilarities = []\n",
    "\tuser_id = 0\n",
    " \n",
    "\tfor row in pivoted_array:\n",
    "\t\tratings = row\n",
    "\t\tratings = ratings[:pivoted_array.shape[1]] \n",
    "  \n",
    "\t\tratings = ratings[[x[0] for x in new_user_only]]\n",
    "\t\tnew_user_array = np.array([x[1] for x in new_user_only])\n",
    "\n",
    "\t\tsimilarity = cosine_similarity(new_user_array, ratings)\n",
    "\t\tsimilarities.append((user_id, similarity))\n",
    "  \n",
    "\t\t# if (similarity > 0.8):\n",
    "\t\t# \tprint(user_id, ratings)\n",
    "\n",
    "\t\tuser_id += 1\n",
    "  \n",
    "\treturn max(similarities, key = lambda x: x[1])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`top_20_for_user(pivoted_array, user_index)`: finds the top 20 Anime titles that the user has rated the highest (there may not be 20, but we deal with that later)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_20_for_user(pivoted_array, user_index):\n",
    "  user_ratings = pivoted_array[user_index]\n",
    "  \n",
    "  # for i, r in [(59, 10), (249, 10), (178, 10), (1698, 10), (1914, 10)]:\n",
    "  #   print('\\t', user_ratings[i])\n",
    "  \n",
    "  top_20 = sorted(range(len(user_ratings)), key=lambda x: user_ratings[x], reverse=True)[:20]\n",
    "  \n",
    "  # print(top_20)\n",
    "  \n",
    "  return_length = 20\n",
    "\n",
    "  for i in range(19, -1, -1):\n",
    "    # print(user_ratings[top_20[i]])\n",
    "    if user_ratings[top_20[i]] == 0:\n",
    "      return_length -= 1\n",
    "  \n",
    "  return top_20[:return_length]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The user must enter five Anime ID's and their corresponding ratings. Mappings between 'anime_id' and 'title' can be found in `data/anime_title_mappings.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New user ratings: [(7252, 8), (8695, 7), (8015, 8), (559, 6), (99, 8)]\n"
     ]
    }
   ],
   "source": [
    "new_user_ratings = []\n",
    "\n",
    "for i in range(5):\n",
    "    anime_id = int(input(\"Enter the anime ID: \"))\n",
    "    rating = int(input(\"Enter the rating out of 10: \"))\n",
    "    \n",
    "    # check if the anime ID exists in the mappings\n",
    "    if anime_id in anime_id_mappings:\n",
    "        # get the index of the anime ID from the mappings\n",
    "        anime_index = np.where(anime_id_mappings == anime_id)[0][0]\n",
    "        new_user_ratings.append((anime_index, rating))\n",
    "    else:\n",
    "        print(\"Anime ID not found in the mappings.\")\n",
    "\n",
    "print(\"New user ratings:\", new_user_ratings)\n",
    "\n",
    "new_user_only = new_user_ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_user_ratings = transform_new_ratings(new_user_ratings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(7252, 8), (8695, 7), (8015, 8), (559, 6), (99, 8)]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_user_only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "closest_user = find_closest_user(pivoted_array, new_user_ratings, new_user_only)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "106326\n"
     ]
    }
   ],
   "source": [
    "closest_user_index = closest_user[0]\n",
    "print(closest_user_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([8, 8, 0, ..., 0, 8, 0], dtype=uint8)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the closest user's highest rated animes\n",
    "pivoted_array[closest_user_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "do_als = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_20 = top_20_for_user(pivoted_array, closest_user_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now remove already watched from the top_20 for new user\n",
    "new_user_recommendations = list(filter(lambda x: x not in [a[0] for a in new_user_only], top_20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4090, 4775, 5785, 6680, 6753, 7702, 7926, 11, 176, 517, 765, 791, 1386, 1523, 1551, 2613, 3262, 3299, 3321, 3360]\n"
     ]
    }
   ],
   "source": [
    "print(new_user_recommendations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(top_20) != 20 or len(new_user_recommendations) < 10:\n",
    "    do_als = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommmendation for new user:\n",
      "0. Angel Beats!\n",
      "1. Steins;Gate\n",
      "2. Yahari Ore no Seishun Love Comedy wa Machigatteiru.\n",
      "3. Shigatsu wa Kimi no Uso\n",
      "4. Yahari Ore no Seishun Love Comedy wa Machigatteiru. Zoku\n",
      "5. Kimi no Na wa.\n",
      "6. Yahari Ore no Seishun Love Comedy wa Machigatteiru. Zoku OVA\n",
      "7. One Piece\n",
      "8. Sen to Chihiro no Kamikakushi\n",
      "9. Digimon Adventure\n",
      "10. Suzumiya Haruhi no Yuuutsu\n",
      "11. Nana\n",
      "12. Death Note\n",
      "13. Byousoku 5 Centimeter\n",
      "14. Clannad Movie\n",
      "15. Ookami to Koushinryou\n",
      "16. Natsume Yuujinchou\n",
      "17. Clannad: After Story\n",
      "18. Toradora!\n",
      "19. Suzumiya Haruhi no Yuuutsu (2009)\n"
     ]
    }
   ],
   "source": [
    "print(\"Recommmendation for new user:\")\n",
    "for i, anime_index in enumerate(new_user_recommendations):\n",
    "  print(f\"{i}.\", anime_title_mappings[anime_id_mappings[anime_index]])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ALS"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### When will ALS be used?\n",
    "- Keeping in mind that we have found the closest existing user to the new user based on the new user's rating of five Anime's.\n",
    "- If the closest user doesn't have at least 10 new recommendations, we fall back to using ALS.\n",
    "- ALS is used to predict 5 Anime's for the closest user found using cosine similarity.\n",
    "- Essentially, we use ALS if cosine similarity fails to give new recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "if not do_als:\n",
    "    exit(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.recommendation import ALS, ALSModel\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql import functions as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = SparkSession.builder.appName('spark').master('local').getOrCreate()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "model = ALSModel.load(model_path)\n",
    "\n",
    "als = ALS(userCol=\"user_id\", itemCol=\"anime_id\", ratingCol=\"my_score\", coldStartStrategy=\"drop\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = r\"files\\final_animedataset.csv\"\n",
    "\n",
    "df = spark.read.csv(path, header=True).select(\n",
    "    F.col('user_id'), F.col('anime_id'), F.col('my_score'), F.col('title'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.dropna(subset=[\"user_id\", \"anime_id\", \"my_score\", \"title\"])\n",
    "\n",
    "df = df.withColumn(\"my_score\", F.col(\"my_score\").cast(\"double\")) \\\n",
    "       .withColumn(\"anime_id\", F.col(\"anime_id\").cast(\"int\")) \\\n",
    "       .withColumn(\"user_id\", F.col(\"user_id\").cast(\"int\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "anime_mapping_df = df.select(\"anime_id\", \"title\").distinct()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+\n",
      "|user_id|\n",
      "+-------+\n",
      "|5173589|\n",
      "+-------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "user_id = int(user_id_mappings[closest_user_index])\n",
    "\n",
    "user_id_list = [(user_id,)]\n",
    "\n",
    "user_df = spark.createDataFrame(user_id_list, [\"user_id\"])\n",
    "\n",
    "user_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_subset_recs = model.recommendForUserSubset(user_df, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+--------------------+\n",
      "|user_id|     recommendations|\n",
      "+-------+--------------------+\n",
      "|5173589|[{4282, 9.707604}...|\n",
      "+-------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "user_subset_recs.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(user_id=5173589, recommendations=[Row(anime_id=4282, rating=9.70760440826416), Row(anime_id=5460, rating=9.638671875), Row(anime_id=6438, rating=9.539900779724121), Row(anime_id=8609, rating=9.52868366241455), Row(anime_id=2514, rating=9.521927833557129)])\n"
     ]
    }
   ],
   "source": [
    "for row in user_subset_recs.collect():\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec = user_subset_recs.collect()[0]['recommendations']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Row(anime_id=4282, rating=9.70760440826416), Row(anime_id=5460, rating=9.638671875), Row(anime_id=6438, rating=9.539900779724121), Row(anime_id=8609, rating=9.52868366241455), Row(anime_id=2514, rating=9.521927833557129)]\n"
     ]
    }
   ],
   "source": [
    "print(rec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4282, 5460, 6438, 8609, 2514]\n"
     ]
    }
   ],
   "source": [
    "anime_ids_list = []\n",
    "\n",
    "for row in rec:\n",
    "    anime_id = row['anime_id']\n",
    "    anime_ids_list.append(anime_id)\n",
    "\n",
    "print(anime_ids_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommmendation for new user using ALS:\n",
      "1. Kara no Kyoukai 5: Mujun Rasen\n",
      "2. Detective Conan Movie 13: The Raven Chaser\n",
      "3. Detective Conan OVA 09: The Stranger in 10 Years...\n",
      "4. Detective Conan OVA 10: Kid in Trap Island\n",
      "5. Detective Conan OVA 04: Conan and Kid and Crystal Mother\n"
     ]
    }
   ],
   "source": [
    "print(\"Recommmendation for new user using ALS:\")\n",
    "for i, a_id in enumerate(anime_ids_list):\n",
    "  print(f\"{i + 1}.\", anime_title_mappings[a_id])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}