from_scratch_text_representation (1).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "from-scratch-text-representation.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "RGrwaPLPfScq",
        "GLLtl7nsoJbW",
        "KNLw7qqeddS6"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **Natural Language Processing From Scratch**\n",
        "## Text Representation\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "LvXWQjVyd4fT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this lesson we will see in some details how we can best represent text in our application. Let's start by importing the modules we will be using:"
      ],
      "metadata": {
        "id": "t99wEzl0eVOv"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "hz7Mc-WddaGM"
      },
      "outputs": [],
      "source": [
        "import string\n",
        "from collections import Counter\n",
        "from pprint import pprint\n",
        "import gzip\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We choose a well known nursery rhyme, that has the added distinction of having been the first audio ever recorded, to be the short snippet of text that we will ise in our examples:"
      ],
      "metadata": {
        "id": "Un0adwyXe1lb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"\"\"Mary had a little lamb, little lamb,\n",
        "    little lamb. Mary had a little lamb\n",
        "    whose fleece was white as snow.\n",
        "    And everywhere that Mary went\n",
        "    Mary went, Mary went. Everywhere\n",
        "    that Mary went,\n",
        "    The lamb was sure to go\"\"\""
      ],
      "metadata": {
        "id": "NpO32egufH8Q"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Tokenization**\n",
        "The first step in any analysis is to tokenize the text. What this means is that we will extract all the individual words in the text. For the sake of simplicity, we will assume that our text is well formed and that our words are delimited either by white spece or punctuation characters."
      ],
      "metadata": {
        "id": "RGrwaPLPfScq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def extract_words(text):\n",
        "    temp = text.split() # Split the text on whitespace\n",
        "    text_words = []\n",
        "\n",
        "    for word in temp:\n",
        "        # Remove any punctuation characters present in the beginning of the word\n",
        "        while word[0] in string.punctuation:\n",
        "            word = word[1:]\n",
        "        \n",
        "        # Remove any punctuation characters present in the end of the word\n",
        "        while word[-1] in string.punctuation:\n",
        "            word = word[:-1]\n",
        "        \n",
        "        # Append this word into our list of words.\n",
        "        text_words.append(word.lower())\n",
        "\n",
        "    return text_words\n",
        "\n",
        "        "
      ],
      "metadata": {
        "id": "X21BRSZCfx_a"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text.split()"
      ],
      "metadata": {
        "id": "ckMD6UCXgYgq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "328e6ecc-dec1-44ad-d9ba-428b8ae2e678"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Mary',\n",
              " 'had',\n",
              " 'a',\n",
              " 'little',\n",
              " 'lamb,',\n",
              " 'little',\n",
              " 'lamb,',\n",
              " 'little',\n",
              " 'lamb.',\n",
              " 'Mary',\n",
              " 'had',\n",
              " 'a',\n",
              " 'little',\n",
              " 'lamb',\n",
              " 'whose',\n",
              " 'fleece',\n",
              " 'was',\n",
              " 'white',\n",
              " 'as',\n",
              " 'snow.',\n",
              " 'And',\n",
              " 'everywhere',\n",
              " 'that',\n",
              " 'Mary',\n",
              " 'went',\n",
              " 'Mary',\n",
              " 'went,',\n",
              " 'Mary',\n",
              " 'went.',\n",
              " 'Everywhere',\n",
              " 'that',\n",
              " 'Mary',\n",
              " 'went,',\n",
              " 'The',\n",
              " 'lamb',\n",
              " 'was',\n",
              " 'sure',\n",
              " 'to',\n",
              " 'go']"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "After this step we now have our text represented as an array of individual, lowercase words."
      ],
      "metadata": {
        "id": "vYwcVgaphLGp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_words = extract_words(text)\n",
        "pprint(text_words)"
      ],
      "metadata": {
        "id": "nhA9NmHDhWrs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4a856335-0845-4a7d-b5da-f1058c380df8"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['mary',\n",
            " 'had',\n",
            " 'a',\n",
            " 'little',\n",
            " 'lamb',\n",
            " 'little',\n",
            " 'lamb',\n",
            " 'little',\n",
            " 'lamb',\n",
            " 'mary',\n",
            " 'had',\n",
            " 'a',\n",
            " 'little',\n",
            " 'lamb',\n",
            " 'whose',\n",
            " 'fleece',\n",
            " 'was',\n",
            " 'white',\n",
            " 'as',\n",
            " 'snow',\n",
            " 'and',\n",
            " 'everywhere',\n",
            " 'that',\n",
            " 'mary',\n",
            " 'went',\n",
            " 'mary',\n",
            " 'went',\n",
            " 'mary',\n",
            " 'went',\n",
            " 'everywhere',\n",
            " 'that',\n",
            " 'mary',\n",
            " 'went',\n",
            " 'the',\n",
            " 'lamb',\n",
            " 'was',\n",
            " 'sure',\n",
            " 'to',\n",
            " 'go']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is a wasteful way to represent text. We can be much more efficient by representing each word by a number. "
      ],
      "metadata": {
        "id": "kH2QHBkPioPv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "word_dict = {}\n",
        "word_list = []\n",
        "vocabulary_size = 0\n",
        "text_tokens = []\n",
        "\n",
        "for word in text_words:\n",
        "    # If we are seeing this word for the first time, create an id for it and add it to our word dictionary\n",
        "    if word not in word_dict:\n",
        "        word_dict[word] = vocabulary_size\n",
        "        word_list.append(word)\n",
        "        vocabulary_size += 1\n",
        "\n",
        "    # Add the token corresponding to the current word to the tokenized text.\n",
        "    text_tokens.append(word_dict[word])"
      ],
      "metadata": {
        "id": "8UC7TyWgi5KB"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "When we were tokenizing our text, we also generated a dictionary **word_dict** that maps words to integers and a **word_list**  that maps each integer to the corresponding word. "
      ],
      "metadata": {
        "id": "TCYNbuPdkUrB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pprint(word_list)\n",
        "pprint(word_dict)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3nmMxN5gk7Ye",
        "outputId": "12faf303-53ce-4080-d268-fc3a65ab5449"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['mary',\n",
            " 'had',\n",
            " 'a',\n",
            " 'little',\n",
            " 'lamb',\n",
            " 'whose',\n",
            " 'fleece',\n",
            " 'was',\n",
            " 'white',\n",
            " 'as',\n",
            " 'snow',\n",
            " 'and',\n",
            " 'everywhere',\n",
            " 'that',\n",
            " 'went',\n",
            " 'the',\n",
            " 'sure',\n",
            " 'to',\n",
            " 'go']\n",
            "{'a': 2,\n",
            " 'and': 11,\n",
            " 'as': 9,\n",
            " 'everywhere': 12,\n",
            " 'fleece': 6,\n",
            " 'go': 18,\n",
            " 'had': 1,\n",
            " 'lamb': 4,\n",
            " 'little': 3,\n",
            " 'mary': 0,\n",
            " 'snow': 10,\n",
            " 'sure': 16,\n",
            " 'that': 13,\n",
            " 'the': 15,\n",
            " 'to': 17,\n",
            " 'was': 7,\n",
            " 'went': 14,\n",
            " 'white': 8,\n",
            " 'whose': 5}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "These two datastructures already proved their usefulness when we converted our text to a list of tokens."
      ],
      "metadata": {
        "id": "CD73KVMYmIUp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(text_tokens)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cXhsuIpOmb8Y",
        "outputId": "dae27f1b-6bcf-4e3c-ab94-ecbab1b60ca8"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0, 1, 2, 3, 4, 3, 4, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 14, 0, 14, 0, 14, 12, 13, 0, 14, 15, 4, 7, 16, 17, 18]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "word_dict"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CBaF8F6cpGLH",
        "outputId": "eb5656d5-eab7-4bbf-b25a-f645139fb6d1"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'a': 2,\n",
              " 'and': 11,\n",
              " 'as': 9,\n",
              " 'everywhere': 12,\n",
              " 'fleece': 6,\n",
              " 'go': 18,\n",
              " 'had': 1,\n",
              " 'lamb': 4,\n",
              " 'little': 3,\n",
              " 'mary': 0,\n",
              " 'snow': 10,\n",
              " 'sure': 16,\n",
              " 'that': 13,\n",
              " 'the': 15,\n",
              " 'to': 17,\n",
              " 'was': 7,\n",
              " 'went': 14,\n",
              " 'white': 8,\n",
              " 'whose': 5}"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Unfortunately, while this representation is convenient for memory reasons it has some severe limitations. Perhaps the most important of which is the fact that computers naturally assume that numbers can be operated on mathematically (by addition, subtraction, etc) in a way that doesn't match our understanding of words."
      ],
      "metadata": {
        "id": "VRqJv9z3nZHv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **One-hot encoding**\n",
        "One typical way of overcoming this difficulty is to represent each word by a one-hot encoded vector where every element is zero except the one corresponding to a specific word."
      ],
      "metadata": {
        "id": "GLLtl7nsoJbW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def one_hot(word, word_dict):\n",
        "    # Generate a one-hot encoded vector corresponding to word\n",
        "    vector = np.zeros(len(word_dict))\n",
        "    vector[word_dict[word]] = 1\n",
        "    return vector"
      ],
      "metadata": {
        "id": "VbiFE8YGoli-"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "one_hot('mary', word_dict)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jPlSQVNlpaYC",
        "outputId": "7b910390-8403-401e-fc07-7051d1ecf619"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0.])"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "So, for example, the word 'fleece' would be represented by:"
      ],
      "metadata": {
        "id": "HwLua5d8p8tC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fleece_hot = one_hot('fleece', word_dict)\n",
        "print(fleece_hot)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-G9PQUh4qEYl",
        "outputId": "076fa840-9c1b-427d-b94a-2dde7424ceb4"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This vector has every element set to zero, except element 6, since:"
      ],
      "metadata": {
        "id": "rJKqx4Uaqwpy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(word_dict['fleece'])\n",
        "fleece_hot[6] == 1"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zM80kKLsq4OS",
        "outputId": "2ce1fecf-e3ff-4978-dca0-8db9a03e829b"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "6\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Bag of words**\n",
        "We can now use the one-hot encoded vector for each word to produce a vector representation of our original text, by simply adding up all the one-hot encoded vectors."
      ],
      "metadata": {
        "id": "KNLw7qqeddS6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_vector1 = np.zeros(vocabulary_size)\n",
        "\n",
        "for word in text_words:\n",
        "    hot_word = one_hot(word, word_dict)\n",
        "    text_vector1 += hot_word\n",
        "print(text_vector1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qjf_ejMjdwVb",
        "outputId": "90502c61-1834-4977-d6e7-97d6876d40d1"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(text_vector1.shape)\n",
        "print(word_list)\n",
        "print(len(word_list))\n",
        "v = np.zeros(5)\n",
        "print(np.zeros(5))\n",
        "v2 = [0, 0,0,0,3]\n",
        "print([0, 0, 0, 0, 3])\n",
        "print(v + v2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "McInTO5jeDDM",
        "outputId": "560587df-ebf7-49b3-ea67-f33947a6b42d"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(19,)\n",
            "['mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'went', 'the', 'sure', 'to', 'go']\n",
            "19\n",
            "[0. 0. 0. 0. 0.]\n",
            "[0, 0, 0, 0, 3]\n",
            "[0. 0. 0. 0. 3.]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(text_words)\n",
        "print(text_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cGNlWpsJepQh",
        "outputId": "f374724c-6632-4419-e2b2-5bd041e40f72"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb', 'mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went', 'everywhere', 'that', 'mary', 'went', 'the', 'lamb', 'was', 'sure', 'to', 'go']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In practice, we can also easily skip the encoding step at the word level by using the word_dict defined above:"
      ],
      "metadata": {
        "id": "IIk1ypk6gTEa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_vector = np.zeros(vocabulary_size)\n",
        "for word in text_words:\n",
        "    text_vector[word_dict[word]] += 1\n",
        "\n",
        "print(text_vector)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WmZ5bQmeghpO",
        "outputId": "8a8ce656-4c94-48cb-e24f-5b2e4fb2da5c"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is known as the bag of words representation of the textt. It should be noted that these vectors simply contains the number of times each word appears in our document, so we can easily tell that the word *mary* appears exactly 6 times in our little nursery rhyme."
      ],
      "metadata": {
        "id": "I-S-ztR1hUFv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_vector[word_dict['mary']]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ji-fQiXwh8qX",
        "outputId": "b6a7f214-c62f-4a79-9a6d-e7cccfa6691b"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "6.0"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A more pythonic and efficient way of producing the same result is to use the standard *Counter* module:"
      ],
      "metadata": {
        "id": "FMx4PnLLiSQv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "word_counts = Counter(text_words)\n",
        "pprint(word_counts)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-sShTb80ixBu",
        "outputId": "b06c32a2-603e-46a5-d2e8-20383fe076c3"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Counter({'mary': 6,\n",
            "         'lamb': 5,\n",
            "         'little': 4,\n",
            "         'went': 4,\n",
            "         'had': 2,\n",
            "         'a': 2,\n",
            "         'was': 2,\n",
            "         'everywhere': 2,\n",
            "         'that': 2,\n",
            "         'whose': 1,\n",
            "         'fleece': 1,\n",
            "         'white': 1,\n",
            "         'as': 1,\n",
            "         'snow': 1,\n",
            "         'and': 1,\n",
            "         'the': 1,\n",
            "         'sure': 1,\n",
            "         'to': 1,\n",
            "         'go': 1})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "From which we can easily generate the text_vector and word_dict data structures:"
      ],
      "metadata": {
        "id": "xH8KVV6LjARh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "items = list(word_counts.items())\n",
        "# Extract word dictionary and vector representation\n",
        "word_dict2 = dict([[items[i][0], i] for i in range(len(items))])\n",
        "text_vector2 = [items[i][1] for i in range(len(items))]"
      ],
      "metadata": {
        "id": "jBWPABn7jL_5"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "type(items[0])\n",
        "print(items[0])\n",
        "print(items[0][0])\n",
        "print(text_vector2)\n",
        "print(word_dict2)\n",
        "print(text_vector)\n",
        "print(word_dict)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EB5DbC1AkAC0",
        "outputId": "f48c0b60-bb74-4001-bf4d-f6f24ff5a6b6"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "('mary', 6)\n",
            "mary\n",
            "[6, 2, 2, 4, 5, 1, 1, 2, 1, 1, 1, 1, 2, 2, 4, 1, 1, 1, 1]\n",
            "{'mary': 0, 'had': 1, 'a': 2, 'little': 3, 'lamb': 4, 'whose': 5, 'fleece': 6, 'was': 7, 'white': 8, 'as': 9, 'snow': 10, 'and': 11, 'everywhere': 12, 'that': 13, 'went': 14, 'the': 15, 'sure': 16, 'to': 17, 'go': 18}\n",
            "[6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]\n",
            "{'mary': 0, 'had': 1, 'a': 2, 'little': 3, 'lamb': 4, 'whose': 5, 'fleece': 6, 'was': 7, 'white': 8, 'as': 9, 'snow': 10, 'and': 11, 'everywhere': 12, 'that': 13, 'went': 14, 'the': 15, 'sure': 16, 'to': 17, 'go': 18}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for word in word_dict.keys():\n",
        "    if text_vector[word_dict[word]] != text_vector2[word_dict2[word]]:\n",
        "        print('Error!')"
      ],
      "metadata": {
        "id": "pyVQixBGlK7b"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Term Frequency**"
      ],
      "metadata": {
        "id": "70a1MKAOiNwc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "items"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t_xM74_mjbDc",
        "outputId": "c3b50b09-f650-470e-c9b9-3ebf51e55f69"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('mary', 6),\n",
              " ('had', 2),\n",
              " ('a', 2),\n",
              " ('little', 4),\n",
              " ('lamb', 5),\n",
              " ('whose', 1),\n",
              " ('fleece', 1),\n",
              " ('was', 2),\n",
              " ('white', 1),\n",
              " ('as', 1),\n",
              " ('snow', 1),\n",
              " ('and', 1),\n",
              " ('everywhere', 2),\n",
              " ('that', 2),\n",
              " ('went', 4),\n",
              " ('the', 1),\n",
              " ('sure', 1),\n",
              " ('to', 1),\n",
              " ('go', 1)]"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Intuitively, we expect the frequency with which a given word is mentioned should correspond to the relevance of that word for the piece of text we are considering. 'mary' is a pretty important word in our little nursery rhyme and indeed it is the one that that occurs the most often:"
      ],
      "metadata": {
        "id": "jfd9e3iXjw_Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sorted(items, key=lambda x: x[1], reverse=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2lpe1r1yjeKd",
        "outputId": "757b4beb-1c16-4a1a-aa24-19dd771200ae"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('mary', 6),\n",
              " ('lamb', 5),\n",
              " ('little', 4),\n",
              " ('went', 4),\n",
              " ('had', 2),\n",
              " ('a', 2),\n",
              " ('was', 2),\n",
              " ('everywhere', 2),\n",
              " ('that', 2),\n",
              " ('whose', 1),\n",
              " ('fleece', 1),\n",
              " ('white', 1),\n",
              " ('as', 1),\n",
              " ('snow', 1),\n",
              " ('and', 1),\n",
              " ('the', 1),\n",
              " ('sure', 1),\n",
              " ('to', 1),\n",
              " ('go', 1)]"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data = []\n",
        "\n",
        "for line in gzip.open('/content/drive/MyDrive/NLP/text8.gz', 'rt'):\n",
        "    data.extend(line.strip().split())"
      ],
      "metadata": {
        "id": "0E-V8SRsVANW"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(data))\n",
        "print(data[0])\n",
        "print(type(data))\n",
        "print(data[:10])\n",
        "dir(data)\n",
        "print(data.index('anarchism'))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0omtYqfAVUYV",
        "outputId": "7bd18f18-d0eb-47e9-9d22-c37910625160"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "17005207\n",
            "anarchism\n",
            "<class 'list'>\n",
            "['anarchism', 'originated', 'as', 'a', 'term', 'of', 'abuse', 'first', 'used', 'against']\n",
            "0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "counts = Counter(data)\n",
        "\n",
        "sorted_counts = sorted(list(counts.items()), key=lambda x: x[1], reverse=True)\n",
        "\n",
        "for word, count in sorted_counts[:10]:\n",
        "    print(word, count)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KCN5ty0FWoti",
        "outputId": "05bc99f4-24bc-4756-9a1a-a355a452ebbb"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "the 1061396\n",
            "of 593677\n",
            "and 416629\n",
            "one 411764\n",
            "in 372201\n",
            "a 325873\n",
            "to 316376\n",
            "zero 264975\n",
            "nine 250430\n",
            "two 192644\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(counts.most_common(19))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2uM0peUPZquC",
        "outputId": "077c4a17-8a2d-45eb-85ba-d79016d49e3c"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('the', 1061396), ('of', 593677), ('and', 416629), ('one', 411764), ('in', 372201), ('a', 325873), ('to', 316376), ('zero', 264975), ('nine', 250430), ('two', 192644), ('is', 183153), ('as', 131815), ('eight', 125285), ('for', 118445), ('s', 116710), ('five', 115789), ('three', 114775), ('was', 112807), ('by', 111831)]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(counts))\n",
        "print(type(counts))\n",
        "print(counts.values().__len__())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cfEpjZSfW-rI",
        "outputId": "72f7725d-ab7f-4aaa-cc62-8bb2338c46d7"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "253854\n",
            "<class 'collections.Counter'>\n",
            "253854\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Grouping words with same frequency\n",
        "dist = Counter(counts.values())\n",
        "print(type(dist))\n",
        "dist = list(dist.items())\n",
        "dist.sort(key=lambda x: x[0])\n",
        "\n",
        "dist = np.array(dist)\n",
        "print(type(dist))\n",
        "print(type(dist.T))\n",
        "print(type(dist.T[0]))\n",
        "print(type(dist.T))\n",
        "norm = np.dot(dist.T[0], dist.T[1])\n",
        "print(type(norm))\n",
        "\n",
        "plt.loglog(dist.T[0], dist.T[1]/norm)\n",
        "plt.xlabel('count')\n",
        "plt.ylabel('P(count)')\n",
        "plt.title('Word frequency distribution')"
      ],
      "metadata": {
        "id": "G86RUev_aDis",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 421
        },
        "outputId": "31727ef6-4aec-4cfd-ac44-b61bec797d73"
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'collections.Counter'>\n",
            "<class 'numpy.ndarray'>\n",
            "<class 'numpy.ndarray'>\n",
            "<class 'numpy.ndarray'>\n",
            "<class 'numpy.ndarray'>\n",
            "<class 'numpy.int64'>\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'Word frequency distribution')"
            ]
          },
          "metadata": {},
          "execution_count": 52
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEaCAYAAAAPGBBTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1f3/8dcnISGyyy6bIJsCKmDE3bqLCmL9Wqtoq60Vaau2te1X/FW/trUuXbStQuvy1UKrglT9WtyrVcEFlaCguAARUQLIFpB9SfL5/TE3OIRkkklmcmdu3s/HYx7MnLlz7ufMkPnMPefcc83dERERqUlO2AGIiEhmU6IQEZGElChERCQhJQoREUlIiUJERBJSohARkYSUKCTjmNkvzezBBM9/38xWmdlmM+vQmLFlIjNbamanBPf/n5n9bwrr3mxmBwT3J5vZb1JY991mdkOq6pP0UaKQWpnZdWb2bJWyxTWUXZDmWPKAO4DT3L2Vu69L5/6yjbvf4u7fq207M3vFzGrdLniPlzQ0LjO71Mxeq1L3eHe/qaF1S/opUUhdzAKONrNcADPbD8gDhlUp6xdsW2dm1izJWLoABcAHKapPqqH3UeIpUUhdzCGWGIYGj48DXgYWVin7xN1XmFk3M5thZqVmVmxml1dWFHQrPWpmD5rZRuBSM+tjZjPNbJOZvQB0rC4IMxsQ7BNgg5m9FJS7mf3QzBYDi4OyUWY2z8w2mNkbZnZIXD3DzOydYH+PmNm0yi6V6n75BvX3C+43N7M/mNnnQffX3Wa2T/DcCWZWYmY/NbPVZrbSzL4TV88+Zna7mX1mZl+a2WtB2dNmdlWVfb5nZl+v4X34VlDHOjP7RZXndnfbmVlB8D6vC96HOWbWxcxuDj6viUHX0sQE7+Putgc6mtkLwXs308z2D7brHWzbLC6WV8zse2Z2EHA3cFSwvw3B83t0ZZnZ5cH/l9Lg/0+3Kp/B+OCodYOZTTIzq+79kdRTopBauftO4C3g+KDoeOBV4LUqZZVHE9OAEqAbcB5wi5mdFFflGOBRoB3wEPAwMJdYgrgJuKSGOBYBg4OH7dw9vs5zgCOAQWY2DHgAuALoANwDzAi+5POBJ4B/AO2BfwL/lcTbcRswgFiC7Ad0B/4n7vmuQNug/DJgkpntGzz3B+Aw4Ohg3/8NVABTgIsrKzCzQ4PXP11152Y2CPgr8C1i728HoEcNsV4SxNIz2G48sM3df0Hs87sy6Fq6Mu41u9/HGuq8iNhn1BGYR+zzS8jdPwr2PTvYX7tq2nUScCtwPrAf8Bmx/0fxRgGHA4cE251e274lNZQopK5m8lVSOI7YF82rVcpmmllP4BjgWnff7u7zgP8Fvh1X12x3f8LdK4BOxP74b3D3He4+C3iyHvHd6u6l7r4NGAfc4+5vuXu5u08BdgBHBrc84E/uvsvdHyV2xFSr4BfsOOAnwb42AbcA8eMyu4BfB3U/A2wGBppZDvBd4EfuvjyI6w133wHMAAaYWf+gjm8BjwQJuqrzgKfcfVbw2huIJZvq7CKWIPoF+5vr7htraWb8+1idp+P2/QtiRwk9a6mzLi4CHnD3d4K6rwvq7h23zW3uvsHdPyd2RDt072okHZQopK5mAceaWXugk7svBt4gNnbRHhgSbNMNqPwSrfQZsV/IlZbF3e8GrHf3LVW2T1Z8nfsDPw26KDYEXR09g311A5b7nqth1nV/nYAWwNy4ep8Lyiutc/eyuMdbgVbEfoEXAJ9UrdTdtwOPABcHCeVCYkc81ekW39bgfatpQP8fwPPANDNbYWa/s9hkgESW1fV5d98MlAYxNVQ34j6HoO517Pn/5ou4+5XvqzQCJQqpq9nEujEuB14HCH6drgjKVrj7p8Hj9mbWOu61vYDlcY/jv6RXAvuaWcsq2ycrvs5lwM3u3i7u1sLdpwb7616lfzt+f1uIJQMAzKxr3HNrgW3A4Lh627p7Xb6w1gLbgb41PD+F2K/qk4Gt7j67hu1WEkt6lfG1IHbUsJfgqOZX7j6IWHfXKL46sqtp2ejalpOO33crYl1oK4i9bxD33hHrhqtrvSuIJfjKulsSa9fyGl8hjUaJQuok6IooAq4h1uVU6bWgbFaw3TJiRxq3BoOphxDrq6/2vAh3/yyo91dmlm9mxwKjGxjufcB4MzvCYlqa2VlB8poNlAFXm1memZ0LjIh77XxgsJkNNbMC4JdxsVYEdf/RzDoDmFl3M6u1rzx47QPAHRYb7M81s6PMrHnw/GxiXUi3U/PRBMTGdkaZ2bHBeMuvqeHv2MxONLODLTYzbSOxrqjKbqpVwAG1xV2NM+P2fRPwprsvc/c1xL7ULw7a9l32TIqrgB7B66ozFfhO8L43J9al95a7L61HjJJiShSSjJlAZ2LJodKrQVn8tNgLgd7EfiX+H3Cju7+YoN6xxAZQS4Ebgb83JEh3LyJ2lDMRWA8UA5cGz+0Ezg0elwLfBB6Pe+0iYl++LxKb+bPHDCjg2qC+Ny02a+tFYGAdQ/sZ8D6xMZFS4Lfs+Tf4d+BgakiqQXwfAD8kNgFgZdC+kho270ossWwEPiL2+VUmoT8D55nZejO7s47xE+z3xiD+w4gbhCf2nv+cWJfRYGI/GCq9RGxK8xdmtraadr1IbLzlsaBdfdlz7EdCZLpwkTR1ZjYZKHH360OO49vAOHc/Nsw4RKrSEYVIBgjGGn4A3Bt2LCJVKVGIhCwY41hDrB//4ZDDEdmLup5ERCQhHVGIiEhCShQiIpJQVqwQaWbnAGcBbYD73f3fibbv2LGj9+7duzFCExGJjLlz5651905Vy9OeKMzsAWJnhK529yFx5SOJzeXOBf7X3W+rqQ53fwJ4Ilhc7Q9AwkTRu3dvioqKUhG+iEiTYWbVLmfTGEcUk4md+LT7JKrgTNFJwKnEThaaY2YziCWNW6u8/rvuvjq4f33wOhERaSRpTxTuPqvKCpAQWzKhuPLKWWY2DRjj7rcSO/rYQ7Auz23As+7+TnX7MbNxxFb2pFev+iwVJCIi1QlrMLs7e65SWcKeq0RWdRVwCrElB8ZXt4G73+vuhe5e2KnTXl1sIiJST1kxmO3udwK1rkdjZqOB0f369attUxERqaOwjiiWE7dcMbErdDV4OWF3f9Ldx7Vt27ahVYmISCCsRDEH6G+xayXnE1slckZDKzWz0WZ275dfftngAEVEJCbticLMphK7BsBAi114/rLgCmBXErv61kfA9GD55AZp6BHFyi+38crC1bVvKCLShDTGrKcLayh/Bngmlftq6BjFL2d8wKuL1/LkVcfSt5OusigiAhFbwqOhRxS/OnsIzZvl8MOH3mH7rvIURycikp0ilSgaqmvbAm4//1A+/mITv3n6w7DDERHJCJFKFKkYzD7pwC6MO/4AHnzzc55+b2UKoxMRyU6RShSpmh77s9MGMrRnOyY89h6fr9uaouhERLJTpBJFquQ3y+GuC4eBwZVT32FnWUXYIYmIhCZSiSKV51H0bN+C3593CO+VfMlvn/s4BdGJiGSnSCWKVJ+ZPXLIflxy1P7c/9qnvPDhqpTUKSKSbSKVKNLhujMPYnC3Nvzsn/NZvmFb2OGIiDQ6JYpaFOTlMnHscMrKK7h66rvsKtd4hYg0LZFKFOla66lPx5bccu7BzP1sPX98YVFK6xYRyXSRShTpXD12zNDuXHB4T/7yyifMXLQm5fWLiGSqSCWKdLtx9GAGdGnFNY/MY/XG7WGHIyLSKJQokrBPfi6Txg5ny84yfjRtHuUVHnZIIiJpp0SRpP5dWvPrMUOYvWQdE18qDjscEZG0i1SiaKwLF33jsB58fVh3/vyfRcz+ZF1a9yUiErZIJYrGuhSqmXHTOUPo3aElP5r2Lus270jr/kREwhSpRNGYWjVvxl1jh7Fh2y6umT6fCo1XiEhEKVE0wOBubbnhrIOYuWgN9766JOxwRETSQomigS4+cn/OGNKV3z+/kLmfrQ87HBGRlFOiaCAz47b/OoT92hZw9dR3Wb9lZ9ghiYiklBJFCrTdJ49JY4ezetN2fjJ9nsYrRCRSIpUoGmt6bHUO7dmO/xk1iFcWrmHiyzq/QkSiI1KJorGmx9bk4iP355yh3fjji4t4dbHWgxKRaIhUogibmXHLuQfTv3MrfjRtHit0/QoRiQAlihRrkd+Mv158GDt2lfODh3S9bRHJfkoUadC3Uyt+/41DmbdsA7c881HY4YiINIgSRZqcefB+XHZsHya/sZQZ81eEHY6ISL0pUaTRhDMOpHD/fZnw2HssXrUp7HBEROpFiSKN8nJzmDh2OC3yc/n+Q++wZUdZ2CGJiCQt4xOFmR1kZneb2aNm9v2w40lW17YF3HnBMJas2cyEx9/HXSfjiUh2SWuiMLMHzGy1mS2oUj7SzBaaWbGZTUhUh7t/5O7jgfOBY9IZb7oc3a8jPz1tIE/OX8GUN5aGHY6ISFLSfUQxGRgZX2BmucAk4AxgEHChmQ0ys4PN7Kkqt87Ba84GngaeSXO8afP9r/XllIM6c/MzH/HO51o8UESyR1oThbvPAkqrFI8Ait19ibvvBKYBY9z9fXcfVeW2OqhnhrufAVxU077MbJyZFZlZ0Zo1mXdWdE6Ocfs3htK1bQE/fOgdXexIRLJGGGMU3YFlcY9LgrJqmdkJZnanmd1DgiMKd7/X3QvdvbBTp06pizaF2rbI468XHca6LTv50bR5lGvxQBHJAhk/mO3ur7j71e5+hbtPSrRtmIsC1tWQ7m25acxgXitey59fXBR2OCIitQojUSwHesY97hGUNVjYiwLW1TcP78X5hT2486ViXv54ddjhiIgkFEaimAP0N7M+ZpYPXADMSEXF2XBEUenXY4YwaL82/PiReSwr3Rp2OCIiNUr39NipwGxgoJmVmNll7l4GXAk8D3wETHf3D1Kxv2w5ogAoyMvlrxcPp8KdHzz0Dtt3lYcdkohItSxKJ4CZ2WhgdL9+/S5fvHhx2OHUyb8/+IJx/5jL2CN6ccvXDw47HBFpwsxsrrsXVi3P+MHsZGTTEUWl0wZ3ZfzX+vLwW5/z6NySsMMREdlLpBJFtvrZaQM4pl8H/t//vc/7JZk/viIiTUukEkU2DWbHa5abw10XDqdTq+Zc8Y8inYwnIhklUokiG7ueKrVvmc8934qdjPfDh9+hrFxXxhORzBCpRJHthnRvy63nHsybS0q57dmPww5HRASIWKLI1q6neOcO78GlR/fmf1/7lH/NS8l5iCIiDRKpRJHNXU/xfnHWQYzo3Z5rH3uPhV/oyngiEq5IJYqoyMvNYeJFw2jVvBlXTdXJeCISLiWKDNW5dQF3nD+URas2c9NTH4Ydjog0YZFKFFEYo4h3/IBOXHH8ATz01uc8t2Bl2OGISBMVqUQRlTGKeD89bSCH9mjLfz/6Hss3bAs7HBFpgiKVKKIov1kOd144jAqHH097V+dXiEijU6LIAvt3aMnNXx/CnKXruf6JBURpIUcRyXzNwg5A6mbM0O4Ur97MXS8V063dPlx9cv+wQxKRJiJSiSJumfGwQ0mLa04dwPIN27jjhUXs17aAbxT2rP1FIiINFKmupygOZsczM2479xCO69+R6x5/n5mL1oQdkog0AZFKFE1BfrMc/nLRcPp3ac0PHpzLguXRmAosIplLiSILtS7IY/J3Dqddi3y+M3mOrrktImmlRJGlurQpYPJ3DmfHrnIu/dvbbNi6M+yQRCSilCiyWP8urbnv24UsK93G1dPmUVGhabMiknpKFFnuiAM6cOPZg5i1aA1/eaU47HBEJIIilSiittZTXY0d0YuzD+3GHS8sYvYn68IOR0QiJlKJIurTY2tiZtxy7sH07tCSq6e9yxdfbg87JBGJkEgliqasVfNm/PXiw9i6o4zLpsxh686ysEMSkYhQooiQgV1bM3HscD5auZEfa3BbRFJEiSJiTjywMzeMGsS/P1zFb5/7OOxwRCQCIrXWk8RcenRvlqzZwj2zltCnY0suGNEr7JBEJIspUUSQmXHj6EF8VrqV659YwP4dWnJU3w5hhyUiWUpdTxHVLDeHiWOHsX+HFlw97V3WbNoRdkgikqWyIlGYWUszKzKzUWHHkk3aFOQxcexwNm7bxTXTNbgtIvWT1kRhZg+Y2WozW1ClfKSZLTSzYjObUIeqrgWmpyfKaDtovzbcOHowry5ey92zPgk7HBHJQukeo5gMTAT+XllgZrnAJOBUoASYY2YzgFzg1iqv/y5wKPAhUJDmWCPrwhE9ef2Ttdz+70WM6N2ewt7tww5JRLJIWo8o3H0WUFqleARQ7O5L3H0nMA0Y4+7vu/uoKrfVwAnAkcBY4HIzqzZmMxsXdE8VrVmjC/rEMzNuPfdgurfbh6umvstn67aEHZKIZJEwxii6A8viHpcEZdVy91+4+4+Bh4H73L2ihu3udfdCdy/s1KlTSgOOgjYFefzlouFs21XO2RNf541P1oYdkohkiawYzAZw98nu/lSibZrqooB1NaR7W/71w2Po3Lo5l00u4v0SvU8iUrswEsVyoGfc4x5BWYM11UUBk7F/h5Y8dPkRtG+Zz3enzGH1Ji0gKCKJhZEo5gD9zayPmeUDFwAzUlGxjijqpnPrAh649HBKt+zk/lc/DTscEclw6Z4eOxWYDQw0sxIzu8zdy4ArgeeBj4Dp7v5BKvanI4q6G9i1NWcevB8Pv/U5m7bvCjscEclg5h6dk7DMbDQwul+/fpcvXrw47HAy3nslGzh74usM6d6GLq0L+MvFw2neLDfssEQkJGY2190Lq5ZnzWB2XeiIIjmH9GjHWYfsx/otu/jPx6t5bG5KhopEJGIilSgkeZPGDue1a0/k0J7t+Msrxewqr3b2sYg0YUknimDdpYzsn9Bgdv2YGT8+uT8l67dx7aPvMX/ZBnaUlYcdlohkiFrHKIIzoS8ALgIOB3YAzYG1wNPAPe5enOY4k1JYWOhFRUVhh5FV3J27XirmjhcWATCsVzumjTtSYxYiTUhDxiheBvoC1wFd3b2nu3cGjgXeBH5rZhenNFppdGbG1Sf3Z/oVR/GLMw/i3c838IfnF4YdlohkgLosCniKu+81f9LdS4HHgMfMLC/lkdVD3KynsEPJWiP6tGdEn/YsWPEl095exjWnDmSffB1ViDRltR5RVCYJM/tH1ecqy6pLJGHQrKfUueDwXmzaUcavn/qQD1ZozEekKUtmMHtw/INgQPuw1IYjmeLIA9rTt1NLpr79OeMfnMvOMs2GEmmqak0UZnadmW0CDjGzjcFtE7Aa+FfaI5RQmBn/HH80f/zmoSwr3cbDb30WdkgiEpK6dD3d6u6tgd+7e5vg1trdO7j7dY0QY51pemxqtW+ZzzlDu3N03w7c+VKxlvoQaaLq3PXk7teZWXczO9rMjq+8pTO4ZGmMIvXMjGtHHkjplp2ceeerzFla9TpUIhJ1dU4UZnYb8DpwPfDz4PazNMUlGeTQnu2451uH0Swnh0seeJt/zVtOmc7gFmkykrlm9teBge6+I13BSOY6fXBXhvVsx+X/mMuPps3jhQ9XMXHs8LDDEpFGkMyspyVARpwvIeHo3KaAR8cfxZih3Xh2wRes26zfDCJNQTKJYiswz8zuMbM7K2/pCqw+NJidfnm5OVxxfF/KK5xpc5bV/gIRyXrJdD3NIEVXoksXd38SeLKwsPDysGOJsoP2a80pB3Xh9n8vZJ+8XCrc+dZR+2tdKJGIqnOicPcp6QxEsoeZ8ecLhjL+wbn8+qkPAejYqjnnDOsecmQikg7JzHr61MyWVL2lMzjJXC2bN+OP3xy6+3HRZ5o2KxJVyYxRFBJbZvxw4DjgTuDBdAQl2aFjq+bMvf4UjuvfkX9/sIoFyzU2JBJFyZxwty7uttzd/wSclcbYJAt0aNWcH58ygNWbdjBm0ussK90adkgikmLJdD0Nj7sVmtl4khsMTzvNegrHYfvvy/QrjqK8wnlQa0KJRE4yX/S3x90vA5YC56c0mgbSrKfwjOjTnjOGdOWemUs4um9HvjagU9ghiUiKJDPr6cR0BiLZ74/fHMqHK2dx89Mfckzf42iWm/Ql2UUkAyXT9dTWzO4ws6LgdruZafU92a0gL5cJIw9k0arNjH/wHcbe9ybPLfgi7LBEpIGS+cn3ALCJWHfT+cBG4G/pCEqy18ghXTlovza8+NEq3vhkHeMfnMuaTVrqQySbJZMo+rr7je6+JLj9CjggXYFJdjIz/nLRcG7/xqG7y078wysUaXlykayVTKLYZmbHVj4ws2OAbakPSbJdn44t+a/DevDxTSMB2LyjjPPunq2psyJZKplE8X1gkpktNbOlwERgfFqikkgoyMvlgI4tdz/+ySPzmPr25yFGJCL1Ye6e3AvM2gC4+8a0RJQChYWFXlRUFHYYAmzavouZi9Zw5cPvApCXayy++cyQoxKR6pjZXHcvrFqezKynW8ysnbtvdPeNZravmf0mtWFWu98TzOxVM7vbzE5I9/4ktVoX5HHmkP12P95V7ry2eC2rN20PMSoRSUYyXU9nuPuGygfuvh5I+NPQzB4ws9VmtqBK+UgzW2hmxWY2oZb9OrAZKABKkohXMkROjvHED4/hxtGDALj4/rc4+Q8zQ45KROoqmUSRa2bNKx+Y2T5A8wTbA0wGRsYXmFkuMAk4AxgEXGhmg8zsYDN7qsqtM/Cqu58BXAv8Kol4JYMM7dmOi47YnwtH9AJg044yps9ZRrJdnyLS+JJZwuMh4D9mVnnuxHeAhNeocPdZZta7SvEIoNjdlwCY2TRgjLvfCoxKUN16EiQmMxsHjAPo1atXorAkJPnNcrj13IOZv2wDH67cyH8/9h59OrXk8N7tww5NRBJIZgmP35rZfOCUoOgmd3++HvvsDsRfQ7MEOKKmjc3sXOB0oB2xmVY1xXcvcC/EBrPrEZc0krOHduPDlbG5EN+4ezYt83O575JCju7bMeTIRKQ6tSYKMzMP+gfc/TnguUTbpJq7Pw48XpdtzWw0MLpfv37pCEVS5HvH9uG2Zz/e/XjLznImvVzMgC6t6diqtt5MEWlsdRmjeNnMrjKzPfpzzCzfzE4ysynAJUnscznQM+5xj6Cswdz9SXcf17atlqDKZNUtFvh68ToKf/NiCNGISG3qkihGAuXAVDNbaWYfmtmnwGLgQuBP7j45iX3OAfqbWR8zywcuAGYkGXe1dD2K7LezrCLsEESkiqROuDOzPKAjsC1+qmyC7acCJwSvWQXc6O73m9mZwJ+AXOABd7+5HrHXSCfcZb4vt+7CcQrycvn2/W/zdrAW1PVnHcSYod3p1FpdUCKNraYT7mpNFGZWQGypjn7Ae8S+2MvSEmUDxY1RXL548eKww5E6enL+Cq6a+u4eZe/ecCr7tswPKSKRpqkhZ2ZPAQqB94mdYHd74s3DozGK7DS4W5u9yr7/0NwQIhGR6tRleuwgdz8YwMzuB95Ob0jS1HRpU7BX2ZtLSllWupWe7VuEEJGIxKvLEcWuyjuZ2uVUSYPZ2all89jvlRb5uXuUn/6nWWzcvotd5RrgFglTXcYoyoEtlQ+BfYCtwX139737DUKmwezs8/anpXTfdx8Wr9rEpX+bs9fzS287K4SoRJqWeo9RuHuuu7cJbq3dvVnc/YxLEpKdRvRpT/d2+3DCwM4c068DBXl7/tfsPeFpfvXkByFFJ9K0JbMoYMZT11M0PPS9I7n65P57lf/t9aWNH4yIRCtRaNZTdPSqYRB75qI1jRyJiEQqUUh0HHlAh2rLL3ngbXpPeJoFy3XUKNJYlCgkI3Vs1ZxHxh1Z4/Oj7nqNRas2NWJEIk1XpBKFxiiipXfHlgmfX1a6tZEiEWnaIpUoNEYRLftUOa+iqsumFHHXf7RUi0i6RSpRSLS0KcjjxWuO5/EfHA3AT08dsNc2t7+wiDKdkCeSVkoUktH6dW7N8F77svS2szikZ7tqtxn26xcor9BFDUXSRYlCssauGq5VsWlHGddMn9fI0Yg0HXW+ZnY20KVQo61jgmtU/GveCtoU5LFm0w5OHdSF/zqsRyNGJhJtSV24KFtorafoKlpaynl3z651O60NJZK8hlyPQiRjFPZuzz3fOozzC3twVA0n5UFskUERSQ0lCsk6pw/uyu/OO5STDuxc4zbn3xM76thRVs7WnRm9Or5IxlOikKz13WP71LrN2Xe9zqD/eb4RohGJLiUKyVq5OZbw+affW8nCuGU+HptbwqSXi9MdlkjkRCpRaAmPpuedG05lwa9Op3mzvf8r//Dhd/Z4/NN/zuf3zy9srNBEIiNSiUJLeDQ97Vvm06p5M/70zaEJt9tZwzkYIlK7SCUKabpOOqjmgW2Asfe92UiRiESPEoVEQvNmiRcQLPps/e77c5aW7jUT6pQ7ZjLu7zr3RqQ6ShTS5Hzj7tlc88j8PcqKV2/m3x+uCikikcymRCFN0oIVmvAgUldKFBIZr/73iVw4oledtq0IVpud+9l6pryxNI1RiWQ/rfUkkVNR4bxavJZLHng76ddqjShpyrTWkzQZOTnG1wZ0CjsMkcjI+ERhZjlmdrOZ3WVml4Qdj2SPK752QNghiERCWhOFmT1gZqvNbEGV8pFmttDMis1sQi3VjAF6ALuAknTFKtHTq32LpF9zyzMf6dKqIlWk+4hiMjAyvsDMcoFJwBnAIOBCMxtkZgeb2VNVbp2BgcAb7n4N8P00xysRkpeb/H/ve2ct4bjfvcz2XeVpiEgkO6U1Ubj7LKDqhQFGAMXuvsTddwLTgDHu/r67j6pyW03sKKLybKka/3rNbJyZFZlZ0Zo1a9LRHMkyJ9RznGLll9v504uLUxyNSPYKY4yiO7As7nFJUFaTx4HTzewuYFZNG7n7ve5e6O6FnTppIFOgc5sCJo0dXq/XPrdgZYqjEcleGT+Y7e5b3f0yd7/K3Scl2larx0pVZx2yX71et3TdVt74ZC0AqzdtZ8PWnakMSySrhJEolgM94x73CMoaTKvHSnVG1TNZjL3vLZ55fyUjbv4PQ3/9QoqjEskeaT/hzsx6A0+5+5DgcTNgEXAysQQxBxjr7h+kYF+jgdH9+vW7fPFi9TFLzK7yCjZtL6OsooIRN/+nwd5WHEYAAAxMSURBVPW9c8OptG+Zn4LIRDJLKCfcmdlUYDYw0MxKzOwydy8DrgSeBz4CpqciSYCOKKR6ebk5tG+ZT+fWBVx8ZN2W+Ejk07Wb93g86eViildvqmFrkeyX7llPF7r7fu6e5+493P3+oPwZdx/g7n3d/eZU7U9jFFKbw/bfN2V1bd5RxvotO/n98ws57+7ZKatXJNM0CzuAVHL3J4EnCwsLLw87FslMOZb4Ott1UdlbO+TG52mZH7sOxo5dOklPoivjZz2JpFJKEgVQuiU2C2rLTp2YJ9EXqUShriepTW5Oao4orpk+b4+yFOQfkYwVqUShwWypTbMUJIrH5pbsPqIQaQoilShEanPSgZ353rF9uGHUoHrX8UjRMiqqTCvXAYVEWaQGs+POowg7FMlQzXJzuH7UIJ54t2HneC5YvnGPx1t2lnPsb19i3xb5fLDiS/7vB8dwaM92ANzx74W0b5nPBys2cvJBXRg5pGuD9i3S2CJ1RKGuJwlTyfptvL/8Syocfv7o/N3ld75UzC+f/JB/zi1h/INzQ4xQpH4idUQh0hCdWjdnzaYdKamrwuHWZz6id8eWKalPJExKFNIkHd23AwBXHH8A98xawvEDOpFj8MrC1CxRv3bzDu6ZtSQldYmELVKJQmMUUled2xSw9LazALjuzIMAuHrquymrv6w8vWuoiTQmjVGIBFJ5LsTmHWU1Prdguc7zkewSqUQh0hCNNcV1etGy2jcSySBKFCIB0+nVItVSohAJtAgW+Eu36UXL6D3haT5bt4X5yzZw2h9nsiVBV5VI2CKVKLTWkzTEhDMOpHVB+ud3bA9Wmp34UjG3PvsRi1ZtZn7JhrTvV6S+IpUoNJgtDdG6II/3f3k6S287iwFdWoUdjkjGiFSiEEmVNF8hWCSrKFGIVGNg19Zp38c/55bw5pJSAMbe9xY7yva8tsXwm17glDtmpj0OkdooUYhU43fnHcKEMw5kv7YFjbbPrTv2TBSlW3ZSvHpzDVuLNB4lCpFqtMhvxviv9WX2dSc32j7V2yWZKlKJQrOeRERSL1KJQrOeJJu9+/l6Fn6xiWWlW/G40fSdZRXM/ayUVRu3s2n7rjrVtW7zDnaVV6QrVGliIrUooEg2u2xK0e7747/Wd/f9cya9zocrv7pQUuVihjUpK6/gsN+8yDlDu/GnC4alPlBpciJ1RCGSTs2b5fDatSc2yr4enVuy+358kqiL8uBo5On3V6Y0Jmm6lChE6sgdeuzborH2Vu9X5gRrVulcEEkVJQqRDNSQL/nKpQ0rlCkkRZQoROrIG3ECayr2pDQhqaJEIVJHjfkD3VOwMx1QSKpo1pNIHTXm9+76rTVPg/3xtHd5c0kpB3RqSY4ZbVvk0alVc44f0JEPlm9kR9lX02Jve/ZjzhjSlS+37aIgL5fOrZtjBvt3aLlXve98vp6Du7clL1e/H2VPGZ8ozOw44CJisQ5y96NDDkmaqFT8yk+FJ+atAOCLjdv3KJ/8xtK9tr175ifcPfOTvcqrTrFdtGoT5/7lDb5zTG9uHD04dcFKJKT1p4OZPWBmq81sQZXykWa20MyKzWxCojrc/VV3Hw88BUxJZ7wi1Zl/42nAV0cUi28+g5MO7BxeQGmwfstOAD5YntxUXGka0n1EMRmYCPy9ssDMcoFJwKlACTDHzGYAucCtVV7/XXdfHdwfC1yW5nhF9tK82Z6/p/Jyc+jSpvEWC2wMOTmxuVLlGXLUJJklrYnC3WeZWe8qxSOAYndfAmBm04Ax7n4rMKq6esysF/Clu2+qaV9mNg4YB9CrV6+GBy8SqLyU9p7fodH6Qq0890JTaqU6YYxadQeWxT0uCcoSuQz4W6IN3P1edy9098JOnTo1MESRr9juMxO+ErXv0+CAgoqKiDVMUiLjB7MB3P3GumxnZqOB0f369UtzRNKU2N55InKJIlddT5JAGEcUy4GecY97BGUNptVjJR2qyRORU9n1pAVnpTphJIo5QH8z62Nm+cAFwIxUVKzrUUg65FRzSNGYZ2k3hsojikyZAiyZJd3TY6cCs4GBZlZiZpe5exlwJfA88BEw3d0/SMX+dEQh6VBd11PUVLaxXGMUUo10z3q6sIbyZ4BnUr0/jVFIOlh1RxQR+z6tbI/GKKQ6kTpXX0cUIvVTOS1WeUKqkxWznkQyTbZ/n17zyLw9Hm/YFltb6tO1W/Z6TrLLD07sR7/OrVJaZ6QShbqeJF2O6NOebx/Ve/fj8V/ry+vFaymvcNq1yGPRqs385JQB/PHFReEFmYQ5n5XW6znJfJt3lKW8ToviLIfCwkIvKiqqfUMREdnNzOa6e2HV8kiNUYiISOpFKlHoPAoRkdSLVKLQrCcRkdSLVKIQEZHUU6IQEZGEIpUoNEYhIpJ6kUoUGqMQEUm9SCUKERFJvUiecGdmXwKL44raAl/W8X5HYG09dx1fX322qfpcMo8r78eXhdWW6srrEntN9/WZ1B5nbdvoM9nzvj6T6ve7v7vvfYlQd4/cDbi3pse13QeKUrXfZLdJFHdd21WlLJS2VFeuz0SfiT6T7P1Motr19GSCx3W5n6r9JrtNorhre/xkDdvUV0PaUl25PpOG02dS/XP6TBouYR2R7HpqCDMr8mrWOslGUWlLVNoB0WlLVNoB0WlLOtsR1SOKhrg37ABSKCptiUo7IDptiUo7IDptSVs7dEQhIiIJ6YhCREQSUqIQEZGElChERCQhJYpamFlLM5tiZveZ2UVhx1NfZnaAmd1vZo+GHUtDmdk5wefxiJmdFnY89WVmB5nZ3Wb2qJl9P+x4Gir4Wykys1Fhx1JfZnaCmb0afC4nhB1PQ5hZjpndbGZ3mdklDamrSSYKM3vAzFab2YIq5SPNbKGZFZvZhKD4XOBRd78cOLvRg00gmXa4+xJ3vyycSGuXZFueCD6P8cA3w4i3Jkm24yN3Hw+cDxwTRryJJPl3AnAtML1xo6xdku1wYDNQAJQ0dqy1SbItY4AewC4a2pb6nsmXzTfgeGA4sCCuLBf4BDgAyAfmA4OA64ChwTYPhx17fdsR9/yjYcedwrbcDgwPO/aGtIPYj49ngbFhx96QtgCnAhcAlwKjwo69Ae3ICZ7vAjwUduwNbMsE4Ipgmwb93TfJIwp3nwWUVikeARR77Jf3TmAasYxcQiwrQ4YdgSXZjoyWTFss5rfAs+7+TmPHmkiyn4m7z3D3M4CM69ZMsi0nAEcCY4HLzSxj/laSaYe7VwTPrweaN2KYdVKP7671wTblDdlvs4a8OGK6A8viHpcARwB3AhPN7CxSd9p/OlXbDjPrANwMDDOz69z91lCiS05Nn8lVwClAWzPr5+53hxFcEmr6TE4g1rXZHHgmhLjqo9q2uPuVAGZ2KbA27gs3U9X0mZwLnA60AyaGEVg91PR38mfgLjM7DpjVkB0oUdTC3bcA3wk7joZy93XE+vSznrvfSSyBZzV3fwV4JeQwUsrdJ4cdQ0O4++PA42HHkQruvhVIybhkxhweZoDlQM+4xz2CsmwTlXZAdNoSlXZAdNoSlXZAI7RFieIrc4D+ZtbHzPKJDczNCDmm+ohKOyA6bYlKOyA6bYlKO6Ax2hL2KH5IMwemAiv5atrYZUH5mcAiYjMIfhF2nE2lHVFqS1TaEaW2RKUdYbZFiwKKiEhC6noSEZGElChERCQhJQoREUlIiUJERBJSohARkYSUKEREJCElCpEMZWY/NrMWYcchovMoRDKUmS0FCt19bdixSNOmIwqRBjCzb5vZe2Y238z+YWa9zeyloOw/ZtYr2G6ymZ0X97rNwb8nmNkrwVXuPjazh4Jl1K8GugEvm9nL4bROJEarx4rUk5kNBq4Hjnb3tWbWHpgCTHH3KWb2XWKr3J5TS1XDgMHACuB14Bh3v9PMrgFO1BGFhE1HFCL1dxLwz8ovcncvBY4CHg6e/wdwbB3qedvdSzx2DYd5QO80xCpSb0oUIo2jjODvLbj6W37cczvi7pejI33JMEoUIvX3EvCN4OqBBF1PbxBb5hlilzd9Nbi/FDgsuH82kFeH+jcBrVMVrEh96ZeLSD25+wdmdjMw08zKgXeJXab1b2b2c2ANX10d8T7gX2Y2H3gO2FKHXdwLPGdmK9z9xNS3QKRuND1WREQSUteTiIgkpEQhIiIJKVGIiEhCShQiIpKQEoWIiCSkRCEiIgkpUYiISEJKFCIiktD/B0K5y61Dgn++AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(counts.values()))\n",
        "# print(counts.values())\n",
        "# print(type(counts.keys()))\n",
        "print(type(counts.keys()))\n",
        "print(type(counts))\n",
        "print(type(counts.items()))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lqR1sjsytJYU",
        "outputId": "63dad180-a399-4853-8eb7-ffd8774e0726"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "253854\n",
            "<class 'dict_keys'>\n",
            "<class 'collections.Counter'>\n",
            "<class 'dict_items'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "stopwords = set([word for word, count in sorted_counts[:100]])\n",
        "\n",
        "clean_data = []\n",
        "\n",
        "for word in data:\n",
        "    if word not in stopwords:\n",
        "        clean_data.append(word)\n",
        "\n",
        "print(f'Original size: {len(data)}')\n",
        "print(f'clean size: {len(clean_data)}')\n",
        "print(f'Reduction: {1-len(clean_data)/len(data)}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wun9xSOOvm7k",
        "outputId": "7d09d665-2d5f-42e9-87d2-4c9e5117958c"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original size: 17005207\n",
            "clean size: 9006229\n",
            "Reduction: 0.470384041782026\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# words that appear across multiple documents\n",
        "print(type(text), len(text))\n",
        "print(text)\n",
        "corpus_text = text.split('.')\n",
        "print(corpus_text)\n",
        "print(type(corpus_text))\n",
        "print(len(corpus_text))\n",
        "corpus_words = []\n",
        "\n",
        "for document in corpus_text:\n",
        "    doc_words = extract_words(document)\n",
        "    corpus_words.append(doc_words)\n",
        "\n",
        "print(corpus_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nHVXpMGpwW22",
        "outputId": "3ce42790-5592-4fed-da78-3254322eb360"
      },
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'str'> 231\n",
            "Mary had a little lamb, little lamb,\n",
            "    little lamb. Mary had a little lamb\n",
            "    whose fleece was white as snow.\n",
            "    And everywhere that Mary went\n",
            "    Mary went, Mary went. Everywhere\n",
            "    that Mary went,\n",
            "    The lamb was sure to go\n",
            "['Mary had a little lamb, little lamb,\\n    little lamb', ' Mary had a little lamb\\n    whose fleece was white as snow', '\\n    And everywhere that Mary went\\n    Mary went, Mary went', ' Everywhere\\n    that Mary went,\\n    The lamb was sure to go']\n",
            "<class 'list'>\n",
            "4\n",
            "[['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'], ['mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow'], ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'], ['everywhere', 'that', 'mary', 'went', 'the', 'lamb', 'was', 'sure', 'to', 'go']]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pprint(corpus_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Faa2AtP8yjcs",
        "outputId": "59057b82-2afe-43ee-a1db-e4ec06642c5e"
      },
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],\n",
            " ['mary',\n",
            "  'had',\n",
            "  'a',\n",
            "  'little',\n",
            "  'lamb',\n",
            "  'whose',\n",
            "  'fleece',\n",
            "  'was',\n",
            "  'white',\n",
            "  'as',\n",
            "  'snow'],\n",
            " ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],\n",
            " ['everywhere',\n",
            "  'that',\n",
            "  'mary',\n",
            "  'went',\n",
            "  'the',\n",
            "  'lamb',\n",
            "  'was',\n",
            "  'sure',\n",
            "  'to',\n",
            "  'go']]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# word that appears in documents\n",
        "document_count = {}\n",
        "\n",
        "for document in corpus_words:\n",
        "    word_set = set(document)\n",
        "\n",
        "    for word in word_set:\n",
        "        document_count[word] = document_count.get(word, 0) + 1\n",
        "pprint(document_count)\n",
        "\n",
        "# pprint(sorted(document_count, key=lambda x: x[0], reverse=False))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BHJH6aJOytOb",
        "outputId": "d1bfaadc-829e-4ece-83a1-04f8b93a9325"
      },
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'a': 2,\n",
            " 'and': 1,\n",
            " 'as': 1,\n",
            " 'everywhere': 2,\n",
            " 'fleece': 1,\n",
            " 'go': 1,\n",
            " 'had': 2,\n",
            " 'lamb': 3,\n",
            " 'little': 2,\n",
            " 'mary': 4,\n",
            " 'snow': 1,\n",
            " 'sure': 1,\n",
            " 'that': 2,\n",
            " 'the': 1,\n",
            " 'to': 1,\n",
            " 'was': 2,\n",
            " 'went': 2,\n",
            " 'white': 1,\n",
            " 'whose': 1}\n",
            "['a',\n",
            " 'as',\n",
            " 'and',\n",
            " 'everywhere',\n",
            " 'fleece',\n",
            " 'go',\n",
            " 'had',\n",
            " 'little',\n",
            " 'lamb',\n",
            " 'mary',\n",
            " 'snow',\n",
            " 'sure',\n",
            " 'that',\n",
            " 'the',\n",
            " 'to',\n",
            " 'whose',\n",
            " 'was',\n",
            " 'white',\n",
            " 'went']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def inv_doc_freq(corpus_words):\n",
        "    number_docs = len(corpus_words)\n",
        "    print(type(corpus_words))\n",
        "\n",
        "    document_count = {}\n",
        "\n",
        "    for document in corpus_words:\n",
        "        word_set = set(document)\n",
        "\n",
        "        for word in word_set:\n",
        "            document_count[word] = document_count.get(word, 0) + 1\n",
        "\n",
        "    IDF = {}\n",
        "\n",
        "    for word in document_count:\n",
        "        IDF[word] = np.log(number_docs / document_count[word])\n",
        "\n",
        "    return IDF"
      ],
      "metadata": {
        "id": "3xaqGjKn0EH9"
      },
      "execution_count": 64,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "IDF = inv_doc_freq(corpus_words)\n",
        "pprint(IDF)\n",
        "# pprint(sorted(IDF, key=lambda x: x[0], reverse=False))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eQQzs4MN0-dT",
        "outputId": "174bb850-986a-474a-eb6f-c088e4206a71"
      },
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'list'>\n",
            "{'a': 0.6931471805599453,\n",
            " 'and': 1.3862943611198906,\n",
            " 'as': 1.3862943611198906,\n",
            " 'everywhere': 0.6931471805599453,\n",
            " 'fleece': 1.3862943611198906,\n",
            " 'go': 1.3862943611198906,\n",
            " 'had': 0.6931471805599453,\n",
            " 'lamb': 0.28768207245178085,\n",
            " 'little': 0.6931471805599453,\n",
            " 'mary': 0.0,\n",
            " 'snow': 1.3862943611198906,\n",
            " 'sure': 1.3862943611198906,\n",
            " 'that': 0.6931471805599453,\n",
            " 'the': 1.3862943611198906,\n",
            " 'to': 1.3862943611198906,\n",
            " 'was': 0.6931471805599453,\n",
            " 'went': 0.6931471805599453,\n",
            " 'white': 1.3862943611198906,\n",
            " 'whose': 1.3862943611198906}\n",
            "['a',\n",
            " 'as',\n",
            " 'and',\n",
            " 'everywhere',\n",
            " 'fleece',\n",
            " 'go',\n",
            " 'had',\n",
            " 'little',\n",
            " 'lamb',\n",
            " 'mary',\n",
            " 'snow',\n",
            " 'sure',\n",
            " 'that',\n",
            " 'the',\n",
            " 'to',\n",
            " 'whose',\n",
            " 'was',\n",
            " 'white',\n",
            " 'went']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def tf_idf(corpus_words):\n",
        "    IDF = inv_doc_freq(corpus_words)\n",
        "\n",
        "    TFIDF = []\n",
        "\n",
        "    for document in corpus_words:\n",
        "         TFIDF.append(Counter(document))\n",
        "\n",
        "    for document in TFIDF:\n",
        "        for word in document:\n",
        "            document[word] = document[word] * IDF[word]\n",
        "    return TFIDF"
      ],
      "metadata": {
        "id": "eOrAL_mm2tTN"
      },
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tf_idf(corpus_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YvZxuiyL3OHx",
        "outputId": "ac7f2412-a792-407d-c9c2-77901e098de4"
      },
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'list'>\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Counter({'a': 0.6931471805599453,\n",
              "          'had': 0.6931471805599453,\n",
              "          'lamb': 0.8630462173553426,\n",
              "          'little': 2.0794415416798357,\n",
              "          'mary': 0.0}),\n",
              " Counter({'a': 0.6931471805599453,\n",
              "          'as': 1.3862943611198906,\n",
              "          'fleece': 1.3862943611198906,\n",
              "          'had': 0.6931471805599453,\n",
              "          'lamb': 0.28768207245178085,\n",
              "          'little': 0.6931471805599453,\n",
              "          'mary': 0.0,\n",
              "          'snow': 1.3862943611198906,\n",
              "          'was': 0.6931471805599453,\n",
              "          'white': 1.3862943611198906,\n",
              "          'whose': 1.3862943611198906}),\n",
              " Counter({'and': 1.3862943611198906,\n",
              "          'everywhere': 0.6931471805599453,\n",
              "          'mary': 0.0,\n",
              "          'that': 0.6931471805599453,\n",
              "          'went': 2.0794415416798357}),\n",
              " Counter({'everywhere': 0.6931471805599453,\n",
              "          'go': 1.3862943611198906,\n",
              "          'lamb': 0.28768207245178085,\n",
              "          'mary': 0.0,\n",
              "          'sure': 1.3862943611198906,\n",
              "          'that': 0.6931471805599453,\n",
              "          'the': 1.3862943611198906,\n",
              "          'to': 1.3862943611198906,\n",
              "          'was': 0.6931471805599453,\n",
              "          'went': 0.6931471805599453})]"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "V = set('aeiouy')\n",
        "C = set('bcdfghjklmnpqrstvwxz')"
      ],
      "metadata": {
        "id": "XauxU7-f4rts"
      },
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_stem(suffix, word):\n",
        "    # Extract the stem of a word\n",
        "\n",
        "    if word.lower().endswith(suffix.lower()):\n",
        "        return word[:-len(suffix)]\n",
        "    return None"
      ],
      "metadata": {
        "id": "8IG4DIaG44Xi"
      },
      "execution_count": 70,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def measure(orig_word):\n",
        "    # \n",
        "    word = orig_word.lower()\n",
        "\n",
        "    optV = False\n",
        "    optC = False\n",
        "    VC = False\n",
        "    m = 0\n",
        "\n",
        "    pos = 0\n",
        "\n",
        "    while pos < len(word) and word[pos] in C:\n",
        "        pos += 1\n",
        "        optc = True\n",
        "\n",
        "    while pos < len(word):\n",
        "        while pos < len(word) and word[pos] in V:\n",
        "            pos += 1\n",
        "            optV = True\n",
        "\n",
        "        while pos < len(word) and word[pos] in C:\n",
        "            pos += 1\n",
        "            optV = False\n",
        "        if not optV:\n",
        "            m += 1\n",
        "    return m"
      ],
      "metadata": {
        "id": "UcC5bPwhrgnb"
      },
      "execution_count": 73,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "word = 'crepusculars'\n",
        "print(measure(word))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2-APz2P2srsD",
        "outputId": "4ac18fd6-8673-4243-fd42-68283c7b66a7"
      },
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "4\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def ends_with(char, stem):\n",
        "    return stem[-1] == char\n",
        "def double_consonant(stem):\n",
        "    if len(stem) < 2:\n",
        "        return False\n",
        "    if stem[-1] in C and stem[-2] == stem[-1]:\n",
        "        return True\n",
        "    return False\n",
        "def contains_vowel(stem):\n",
        "    return len(set(stem) & V) > 0"
      ],
      "metadata": {
        "id": "8Sl_3kHgs5pr"
      },
      "execution_count": 76,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def apply_rule(condition, suffix, replacement, word):\n",
        "    # apply porter stemmer rule\n",
        "    # if condition is True replace suffix by replacement in word\n",
        "    stem = get_stem(suffix, word)\n",
        "    if stem is not None and condition is True:\n",
        "        word = stem\n",
        "\n",
        "        if replacement is not None:\n",
        "            word += replacement\n",
        "    return word"
      ],
      "metadata": {
        "id": "PfYDmFrJtlxw"
      },
      "execution_count": 80,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "word = 'plastered'\n",
        "suffix = 'ed'\n",
        "stem = get_stem(suffix, word)\n",
        "apply_rule(contains_vowel(stem), suffix, None, word)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "2mFy0ZccuNsC",
        "outputId": "8d0b130e-497d-4d78-f52b-2919d06a51bc"
      },
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'plaster'"
            ]
          },
          "metadata": {},
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "word = 'bled'\n",
        "suffix = 'ed'\n",
        "stem = get_stem(suffix, word)\n",
        "apply_rule(contains_vowel(stem), suffix, None, word)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "-WXtfIZeulkk",
        "outputId": "4f67cb1f-7ab3-4e4a-a53d-cd51df255036"
      },
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'bled'"
            ]
          },
          "metadata": {},
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "word = 'adoption'\n",
        "suffix = 'ion'\n",
        "stem = get_stem(suffix, word)\n",
        "apply_rule(measure(stem) > 1 and (ends_with('s', stem) or ends_with('t', stem)), suffix, None, word)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "7hzTqQ42u2hZ",
        "outputId": "bc914557-9fbf-4186-c45b-cc319e75e569"
      },
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'adopt'"
            ]
          },
          "metadata": {},
          "execution_count": 84
        }
      ]
    }
  ]
}