Assignment_1_18QE30008.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NLP_Assignment_1.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.10"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fk0evCZ7U9WO"
      },
      "source": [
        "# **Assignment 1 on Natural Language Processing**\n",
        "\n",
        "### Date : 4th Sept, 2020\n",
        "\n",
        "#### Instructor : Prof. Sudeshna Sarkar\n",
        "\n",
        "#### Teaching Assistants : Alapan Kuila, Aniruddha Roy, Anusha Potnuru, Uppada Vishnu"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Il_b_LFKXi8t"
      },
      "source": [
        " # NLTK Library"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ss5CZjC2Xt0i"
      },
      "source": [
        "The [NLTK](https://www.nltk.org/) Python framework is generally used as an education and research tool. Tokenization, Stemming, Lemmatization, Punctuation, Character count, word count are some of these packages which will be discussed in this tutorial.\n",
        "\n",
        "**Installing Nltk** <br>\n",
        "Nltk can be installed using PIP or Conda package managers.For detailed installation instructions follow this [link](https://www.nltk.org/install.html).\n",
        "\n",
        "To ensure we are all on the same page, the coding environment will be in **python3**. We suggest downloading Anaconda3 and creating a separate environment to do this assignment. \n",
        "The link to anaconda3 for Windows and Linux is available here https://docs.anaconda.com/anaconda/install/. \n",
        "The steps to install NLTK is available on the link: \n",
        "```bash\n",
        "sudo pip3 install nltk \n",
        "python3 \n",
        "nltk.download()\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "r4txbU5-RlMv"
      },
      "source": [
        "**Note for Question and answers:**\n",
        "\n",
        "Write your answers to the point in the text box below labelled as **Answer here**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "52_aJRSqaHgC"
      },
      "source": [
        "# Tokenizing words and Sentences using Nltk"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "o5_ElYaeaMbR"
      },
      "source": [
        "**Tokenization** is the process by which big quantity of text is divided into smaller parts called tokens. <br>It is crucial to understand the pattern in the text in order to perform various NLP tasks.These tokens are very useful for finding such patterns.<br>\n",
        "\n",
        "Natural Language toolkit has very important module tokenize which further comprises of sub-modules\n",
        "\n",
        "1. word tokenize\n",
        "2. sentence tokenize"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "sby_OS3qZ_fz",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        },
        "outputId": "b9e4d1ca-1894-4055-c5d0-b63097d96a8b"
      },
      "source": [
        "# Importing modules\n",
        "import nltk\n",
        "nltk.download('punkt') # For tokenizers\n",
        "nltk.download('inaugural') # For dataset\n",
        "from nltk.tokenize import word_tokenize,sent_tokenize"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package inaugural to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/inaugural.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ew9Aq5WHXSn-",
        "colab": {}
      },
      "source": [
        "# Sample corpus.\n",
        "from nltk.corpus import inaugural\n",
        "corpus = inaugural.raw('1789-Washington.txt')\n",
        "# print(corpus)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "V2wbXKzVW0GO"
      },
      "source": [
        "### **TASK**:\n",
        "\n",
        "For the given corpus, \n",
        "1. Print the number of sentences and tokens. \n",
        "2. Print the average number of tokens per sentence.\n",
        "3. Print the number of unique tokens\n",
        "4. Print the number of tokens after stopword removal using the stopwords from nltk.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D6ZPqUIqch7B",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        },
        "outputId": "059ea1bc-d1c0-4bbf-f11b-531e1af3429f"
      },
      "source": [
        "from nltk.tokenize import word_tokenize, sent_tokenize\n",
        "nltk.download('stopwords')\n",
        "from nltk.corpus import stopwords"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jrtu9HcHXFe6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 101
        },
        "outputId": "e933a595-aa92-4435-a2d9-6d2567c58372"
      },
      "source": [
        "# TODO\n",
        "\n",
        "sentences = sent_tokenize(corpus)\n",
        "words = word_tokenize(corpus)\n",
        "print(\"Number of sentences: {}\\nNumber of tokens: {}\".format(len(sentences), len(words)))\n",
        "\n",
        "count = 0\n",
        "for sentence in sentences:\n",
        "    count += len(word_tokenize(sentence))\n",
        "print(\"Average number of tokens per sentence: {:.4f}\".format(count / len(sentences)))\n",
        "\n",
        "count = 0\n",
        "wordSet = set()\n",
        "for word in words:\n",
        "    if word not in wordSet:\n",
        "        count += 1\n",
        "        wordSet.add(word)\n",
        "print(\"Number of unique tokens: {}\".format(count))\n",
        "\n",
        "stop_words = set(stopwords.words(\"english\"))\n",
        "print(\"Number of tokens after removing stopwords: {}\".format(len([word for word in words if word not in stop_words])))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of sentences: 23\n",
            "Number of tokens: 1537\n",
            "Average number of tokens per sentence: 66.8261\n",
            "Number of unique tokens: 626\n",
            "Number of tokens after removing stopwords: 800\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "UViYY9_3t2UE"
      },
      "source": [
        "# Stemming and Lemmatization with NLTK"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "g55XX9KDLgO7"
      },
      "source": [
        "**What is Stemming?** <br>\n",
        "Stemming is a kind of normalization for words. Normalization is a technique where a set of words in a sentence are converted into a sequence to shorten its lookup. The words which have the same meaning but have some variation according to the context or sentence are normalized.<br>\n",
        "Hence Stemming is a way to find the root word from any variations of respective word\n",
        "\n",
        "There are many stemmers provided by Nltk like **PorterStemmer**, **SnowballStemmer**, **LancasterStemmer**.<br>\n",
        "\n",
        "We will try and see differences between Porterstemmer and Snowballstemmer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SS4Ij__XLfTB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        },
        "outputId": "34ab1024-4831-433a-a433-78d51bfa2619"
      },
      "source": [
        "from nltk.stem import PorterStemmer\n",
        "from nltk.stem import SnowballStemmer # Note that SnowballStemmer has language as parameter.\n",
        "\n",
        "words = [\"grows\",\"leaves\",\"fairly\",\"cats\",\"trouble\",\"misunderstanding\",\"friendships\",\"easily\", \"rational\", \"relational\"]\n",
        "\n",
        "# TODO\n",
        "# create an instance of both the stemmers and perform stemming on above words\n",
        "\n",
        "stemmer = PorterStemmer()\n",
        "stemmed_words = [stemmer.stem(word) for word in words]\n",
        "print(\"PorterStemmer stemmed words: {}\".format(stemmed_words))\n",
        "\n",
        "stemmer = SnowballStemmer(language = 'english')\n",
        "stemmed_words = [stemmer.stem(word) for word in words]\n",
        "print(\"SnowballStemmmer stemmed words: {}\".format(stemmed_words))\n",
        "\n",
        "# TODO\n",
        "# Complete the function which takes a sentence/corpus and gets its stemmed version.\n",
        "\n",
        "def stemSentence(sentence=None):\n",
        "    stemmer = PorterStemmer()\n",
        "    words = [stemmer.stem(word) for word in word_tokenize(sentence)]\n",
        "    stemmed_sent = \"\"\n",
        "    for word in words:\n",
        "        stemmed_sent += (word + \" \")\n",
        "    return stemmed_sent\n",
        "\n",
        "print(\"\\n\" + stemSentence(\"The quick brown fox jumps over the lazy dog\")) # Test sentences here"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "PorterStemmer stemmed words: ['grow', 'leav', 'fairli', 'cat', 'troubl', 'misunderstand', 'friendship', 'easili', 'ration', 'relat']\n",
            "SnowballStemmmer stemmed words: ['grow', 'leav', 'fair', 'cat', 'troubl', 'misunderstand', 'friendship', 'easili', 'ration', 'relat']\n",
            "\n",
            "the quick brown fox jump over the lazi dog \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0JuE8CuDQSno"
      },
      "source": [
        "**What is Lemmatization?** <br>\n",
        "Lemmatization is the algorithmic process of finding the lemma of a word depending on their meaning. Lemmatization usually refers to the morphological analysis of words, which aims to remove inflectional endings. It helps in returning the base or dictionary form of a word, which is known as the lemma.<br>\n",
        "\n",
        "*The NLTK Lemmatization method is based on WorldNet's built-in morph function.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "31kQlMqnteOU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        },
        "outputId": "e57fd022-6795-4595-dd0c-1bf023aec299"
      },
      "source": [
        "#imports\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "nltk.download('wordnet') # Since Lemmatization method is based on WorldNet's built-in morph function."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/wordnet.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "noyl1YNsQp98",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        },
        "outputId": "a33cb3a3-5e4f-4483-b8e6-12ffe7bc2f35"
      },
      "source": [
        "words = [\"grows\",\"leaves\",\"fairly\",\"cats\",\"trouble\",\"running\",\"friendships\",\"easily\", \"was\", \"relational\",\"has\"]\n",
        "\n",
        "#TODO\n",
        "# Create an instance of the Lemmatizer and perform Lemmatization on above words\n",
        "# You can also give Parts-of-speech(pos) to the Lemmatizer for example \"v\" (verb). Check the differences in the outputs.\n",
        "\n",
        "lemmatizer = WordNetLemmatizer()\n",
        "lemmatized_words = [lemmatizer.lemmatize(word, pos = 'n') for word in words]\n",
        "print(\"Lemmatized words with pos = 'n': {}\".format(lemmatized_words))\n",
        "\n",
        "lemmatized_words = [lemmatizer.lemmatize(word, pos = 'v') for word in words]\n",
        "print(\"Lemmatized words with pos = 'v': {}\".format(lemmatized_words))\n",
        "\n",
        "#TODO\n",
        "# Complete the function which takes a sentence/corpus and gets its lemmatized version.\n",
        "\n",
        "def lemmatizeSentence(sentence=None):\n",
        "    lemmatizer = WordNetLemmatizer()\n",
        "    words = [lemmatizer.lemmatize(word) for word in word_tokenize(sentence)]\n",
        "    lemmatized_sent = \"\"\n",
        "    for word in words:\n",
        "        lemmatized_sent += (word + \" \")\n",
        "    return lemmatized_sent\n",
        "\n",
        "print(\"\\n\" + lemmatizeSentence(\"The quick brown fox jumps over the lazy dog\")) # Test sentences here"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Lemmatized words with pos = 'n': ['grows', 'leaf', 'fairly', 'cat', 'trouble', 'running', 'friendship', 'easily', 'wa', 'relational', 'ha']\n",
            "Lemmatized words with pos = 'v': ['grow', 'leave', 'fairly', 'cat', 'trouble', 'run', 'friendships', 'easily', 'be', 'relational', 'have']\n",
            "\n",
            "The quick brown fox jump over the lazy dog \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rce8GJ3WxSXz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        },
        "outputId": "f40d27db-c1a7-410c-e42c-a004b598699e"
      },
      "source": [
        "print(stemSentence(\"loving\"))\n",
        "print(stemSentence(\"love\"))\n",
        "print(lemmatizeSentence(\"loving\"))\n",
        "print(lemmatizeSentence(\"love\"))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "love \n",
            "love \n",
            "loving \n",
            "love \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VJW6HsycSAlU"
      },
      "source": [
        "**Question:** Give example of two words which have same stem but different lemma? Show the stem and lemma of both words in the code below \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zcq6bUEaSAt1"
      },
      "source": [
        "**Answer here:** <br> <br>\n",
        "Words -> university, universe <br> \n",
        "*Stems* -> univers, univers <br>\n",
        "*Lemmas* -> university, universe <br>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "a8OtIEmFkGBM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        },
        "outputId": "cbdbd197-2a3a-4374-80bc-2bc458a25113"
      },
      "source": [
        "#TODO\n",
        "# Write code to print the stem and lemma of both your words\n",
        "print(\"Words -> university , universe\")\n",
        "print(\"Stems -> {}, {}\\nLemmas -> {}, {}\".format(stemSentence(\"university\"), stemSentence(\"universe\"), lemmatizeSentence(\"university\"), lemmatizeSentence(\"universe\")))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Words -> university , universe\n",
            "Stems -> univers , univers \n",
            "Lemmas -> university , universe \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "e0tz3SIGSA2b"
      },
      "source": [
        "**Question:** Write a comparison between stemming and lemmatization?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aks8QaQ_SA_N"
      },
      "source": [
        "**Answer here:** <br> <br>\n",
        "Stemming: <br>\n",
        "Stemming involves rule-based approaches to reduce a word to its root, i.e. its stem. The input word is passed thorugh a set of conditionals to determine what part of it should be dropped in the stemmed result. Stemmed words can often suffer from understemming and overstemming, where too little or too much of the word is chopped off, and hence is generally considered inferior to lemmatization for most tasks. Stemmed words may or may not be actual words.\n",
        "\n",
        "Lemmatization: <br>\n",
        "Lemmatization is more nuanced than stemming, and requires additional information about the part of speech (POS) of the word inputted in order to work. If this is not provided, the lemmatizer uses the default POS for the word. The same word may be lemmatized differently if the POS for it provided differently. Lemmatization generally results in a meaningful result which is the dictionary form of the input word."
      ]
    }
  ]
}