13 - Project - Information Retrieval (IR).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6f42ffed",
   "metadata": {},
   "source": [
    "# Project: Information Retrieval (IR)\n",
    "- Calculate the TF-IDF of the corpus form 'files/holmes'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "803e001a",
   "metadata": {},
   "source": [
    "### Step 1: Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "401bf1c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import math\n",
    "from nltk import word_tokenize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa401848",
   "metadata": {},
   "source": [
    "### Step 2: Read the corpus\n",
    "- Read all the Sherlock Holmes texts in files/holmes/\n",
    "- Create a dictionary (dict) calleds corpus\n",
    "- Use os.listdir(...) ([docs](https://docs.python.org/3/library/os.html)) to iterate over all the filenames in 'files/holmes'\n",
    "- For each filename open the file and read the content and add it to the **corpus[filename]**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "782c0503",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ffb7167",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8b4ec6ab",
   "metadata": {},
   "source": [
    "### Step 3: Tokenize the content\n",
    "- Iterate over **filename** in **corpus**\n",
    "- For each filename assign **corpus[filename]** to be the list of word (in lower) for word in word_tokenize(...) of the content of filename if word is alpha.\n",
    "    - HINT: Use list comprehension\n",
    "    - HINT: Use **.isalpha()**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a099db29",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7e668f7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "dd72c541",
   "metadata": {},
   "source": [
    "### Step 4: Get all words\n",
    "- Create a set **words**\n",
    "    - HINT: **words = set()**\n",
    "- For each **filename** in **corpus** update the set **words** with the content\n",
    "    - HINT: apply **update(...)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "236a2030",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f68db39",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2e74793c",
   "metadata": {},
   "source": [
    "### Step 5: Calculate term frequency (TF)\n",
    "- Createa empty dictionary (dict) called **tf**\n",
    "- Iterate over **filename** in **corpus**\n",
    "- For each filename add **tf[filename]** with the word frequency.\n",
    "    - HINT: Use dict comprehension with **word** in **words**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20e4017d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28b0167b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f4bf275e",
   "metadata": {},
   "source": [
    "### Step 6: Calculate the inverse document frequency (IDF)\n",
    "- Create an empty dictionary called **idf**\n",
    "- Iterate **word** in **words**\n",
    "- For each **word** calculate the number of documents word is in the corpus\n",
    "    - HINT: **freq = sum(word in corpus[filename] for filename in corpus)**\n",
    "- Update **idf[word]** to be the logarithm of number of documents divided by the calcualted frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eecb6845",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdde41de",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e84cef46",
   "metadata": {},
   "source": [
    "### Step 7: Calculate the Term Frequence-Inverse Document Frequency (TF-IDF)\n",
    "- Create a dictionary tfidf\n",
    "- Iterate over **filename** in **corpus**\n",
    "- For each **filename** calculate the TF-IDF for each word and add it as pairs **(word, tf-idf)**\n",
    "    - HINT: Use list comprehension **[(word, tf[filename][word] * idf[word]) for word in words]**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbe279e5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ccdc583",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3694d85d",
   "metadata": {},
   "source": [
    "### Step 8: Sort the values\n",
    "- Iterate over **filename** in **corpus**\n",
    "- For each **filename** sort the values in **tfidf** by second item in reverse order\n",
    "    - HINT: Use **sorted** ([docs](https://docs.python.org/3/howto/sorting.html)) with **key=lambda x: x[1]** and **reverse=True**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dc84f1d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c13989a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d2c17b08",
   "metadata": {},
   "source": [
    "### Step 9: Print the top five words\n",
    "- Iterate **filename** in **corpus**\n",
    "- For each **filename** print the filename and iterate over the first file elements of **tfidf[filename]** and print the **term** and **score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f09efec",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9971bb74",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7c198cd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}