Assignment_3_18QE30008.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "18QE30008_NLP_Assignment_3.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zyJ25uz0kSaw"
      },
      "source": [
        "# Assignment 3 on Natural Language Processing\n",
        "\n",
        "## Date : 30th Sept, 2020\n",
        "\n",
        "### Instructor : Prof. Sudeshna Sarkar\n",
        "\n",
        "### Teaching Assistants : Alapan Kuila, Aniruddha Roy, Anusha Potnuru, Uppada Vishnu"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ao1nhg9RknmF"
      },
      "source": [
        "The central idea of this assignment is to use Naive Bayes classifier and LSTM based classifier and compare the models by accuracy on IMDB dataset.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ONM5Q4SCe9Mr"
      },
      "source": [
        "Please submit with outputs. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EXK2Dq5GQ7Gk"
      },
      "source": [
        "### **-18QE30008, Divyanshu Sheth**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BTaIf1eV6HMb",
        "outputId": "214d5596-c32d-44fd-a77c-440c64d2f034",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 390
        }
      },
      "source": [
        "# Gets the dataset\n",
        "import os\n",
        "import shutil\n",
        "import zipfile\n",
        "! rm -r sample_data\n",
        "! wget \"https://www.dropbox.com/s/a0na1oqn9ysb7fz/archive.zip?dl=1\" 1> /dev/null\n",
        "os.rename(os.path.join(os.getcwd(), 'archive.zip?dl=1'), os.path.join(os.getcwd(), 'archive.zip'))\n",
        "with zipfile.ZipFile(os.path.join(os.getcwd(), 'archive.zip'), 'r') as zip_ref:\n",
        "    zip_ref.extractall(os.getcwd())\n",
        "! rm archive.zip"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "rm: cannot remove 'sample_data': No such file or directory\n",
            "--2020-10-11 12:31:29--  https://www.dropbox.com/s/a0na1oqn9ysb7fz/archive.zip?dl=1\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.65.1, 2620:100:6021:1::a27d:4101\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.65.1|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/dl/a0na1oqn9ysb7fz/archive.zip [following]\n",
            "--2020-10-11 12:31:29--  https://www.dropbox.com/s/dl/a0na1oqn9ysb7fz/archive.zip\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc694408d3851b5f09a23645d991.dl.dropboxusercontent.com/cd/0/get/BBDGnjDpg6_WL-PaBRzKmNLnCwPOLLLFqPI0iKts6PkMIMbq7EmFadrMp77Vcbj-cjuPCUhvI0Vx_GxZbNqZEgZ_7es1NA9-uz-4OCPuJt3d_Au01hEcFGijRVrMNPkCoSo/file?dl=1# [following]\n",
            "--2020-10-11 12:31:30--  https://uc694408d3851b5f09a23645d991.dl.dropboxusercontent.com/cd/0/get/BBDGnjDpg6_WL-PaBRzKmNLnCwPOLLLFqPI0iKts6PkMIMbq7EmFadrMp77Vcbj-cjuPCUhvI0Vx_GxZbNqZEgZ_7es1NA9-uz-4OCPuJt3d_Au01hEcFGijRVrMNPkCoSo/file?dl=1\n",
            "Resolving uc694408d3851b5f09a23645d991.dl.dropboxusercontent.com (uc694408d3851b5f09a23645d991.dl.dropboxusercontent.com)... 162.125.66.15, 2620:100:6021:15::a27d:410f\n",
            "Connecting to uc694408d3851b5f09a23645d991.dl.dropboxusercontent.com (uc694408d3851b5f09a23645d991.dl.dropboxusercontent.com)|162.125.66.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 26962657 (26M) [application/binary]\n",
            "Saving to: ‘archive.zip?dl=1’\n",
            "\n",
            "archive.zip?dl=1    100%[===================>]  25.71M  23.6MB/s    in 1.1s    \n",
            "\n",
            "2020-10-11 12:31:32 (23.6 MB/s) - ‘archive.zip?dl=1’ saved [26962657/26962657]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ElRkQElWUMjG",
        "outputId": "96cd8d36-092b-4fcf-a59a-e769c9cf907c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 118
        }
      },
      "source": [
        "# Imports\n",
        "import re\n",
        "import math\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.model_selection import train_test_split\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "import keras\n",
        "from sklearn.metrics import classification_report, accuracy_score\n",
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.tokenize import word_tokenize \n",
        "from nltk.stem import WordNetLemmatizer\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from collections import Counter, defaultdict\n",
        "nltk.download('wordnet')\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')\n",
        "stopw = stopwords.words('english')"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fhHRim2AUm4z",
        "outputId": "42423743-18b2-4fdf-ba87-ee1017a8e348",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 252
        }
      },
      "source": [
        "# Load the IMDB dataset\n",
        "dataset = pd.read_csv('IMDB Dataset.csv')\n",
        "print(dataset)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                                  review sentiment\n",
            "0      One of the other reviewers has mentioned that ...  positive\n",
            "1      A wonderful little production. <br /><br />The...  positive\n",
            "2      I thought this was a wonderful way to spend ti...  positive\n",
            "3      Basically there's a family where a little boy ...  negative\n",
            "4      Petter Mattei's \"Love in the Time of Money\" is...  positive\n",
            "...                                                  ...       ...\n",
            "49995  I thought this movie did a down right good job...  positive\n",
            "49996  Bad plot, bad dialogue, bad acting, idiotic di...  negative\n",
            "49997  I am a Catholic taught in parochial elementary...  negative\n",
            "49998  I'm going to have to disagree with the previou...  negative\n",
            "49999  No one expects the Star Trek movies to be high...  negative\n",
            "\n",
            "[50000 rows x 2 columns]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lK_Hn2f6VMP7"
      },
      "source": [
        "# Preprocessing\n",
        "PrePrecessing that needs to be done on lower cased corpus\n",
        "\n",
        "1. Remove html tags\n",
        "2. Remove URLS\n",
        "3. Remove non alphanumeric character\n",
        "4. Remove Stopwords\n",
        "5. Perform stemming and lemmatization\n",
        "\n",
        "You can use regex from re. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5B5lHZPsVOXv"
      },
      "source": [
        "def preprocess(raw_text):   \n",
        "  cleantext = raw_text.lower()\n",
        "  cleanr = re.compile('<.*?>')\n",
        "  cleantext = re.sub(cleanr, ' ', cleantext)    \n",
        "  cleantext = re.sub(r'http\\S+', '', cleantext)  \n",
        "  cleantext = re.sub(r'[^\\w\\s]', '', cleantext)  \n",
        "  lemmatizer = WordNetLemmatizer()  \n",
        "  cleanwords = [lemmatizer.lemmatize(word) for word in word_tokenize(cleantext) if word not in stopw]  \n",
        "  cleantext = ' '.join(cleanwords)  \n",
        "  return cleantext"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lp1831JUQvqO"
      },
      "source": [
        "def ppDataset(dataset):   # preprocess the entire dataset\n",
        "  for i in range(len(dataset)):\n",
        "    dataset.iloc[i][0] = preprocess(dataset.iloc[i][0])\n",
        "  return dataset"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0DTgGnRTZFOU",
        "outputId": "74742e27-12a4-47ec-835a-49994f2803f5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        }
      },
      "source": [
        "dataset = ppDataset(dataset)\n",
        "print(\"Preprocessed dataset:\\n\", dataset)"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Preprocessed dataset:\n",
            "                                                   review sentiment\n",
            "0      one reviewer mentioned watching 1 oz episode y...  positive\n",
            "1      wonderful little production filming technique ...  positive\n",
            "2      thought wonderful way spend time hot summer we...  positive\n",
            "3      basically there family little boy jake think t...  negative\n",
            "4      petter matteis love time money visually stunni...  positive\n",
            "...                                                  ...       ...\n",
            "49995  thought movie right good job wasnt creative or...  positive\n",
            "49996  bad plot bad dialogue bad acting idiotic direc...  negative\n",
            "49997  catholic taught parochial elementary school nu...  negative\n",
            "49998  im going disagree previous comment side maltin...  negative\n",
            "49999  one expects star trek movie high art fan expec...  negative\n",
            "\n",
            "[50000 rows x 2 columns]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DyaSkfcvYGXk",
        "outputId": "96d5bc1f-9b63-429a-c760-29821b81ee62",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        }
      },
      "source": [
        "# Print Statistics of Data like avg length of sentence, proposition of data w.r.t class labels\n",
        "npos = 0\n",
        "sumlen = 0\n",
        "for i in range(len(dataset)):\n",
        "  if dataset.iloc[i][1] == 'positive':\n",
        "    npos += 1\n",
        "  sumlen += len(word_tokenize(dataset.iloc[i][0]))\n",
        "\n",
        "print(\"Average length of a review = {:.4f} words\\n\\nClass Distribution:\\nPositive reviews: {} | Negative reviews: {} | Proportion of positive reviews: {:.4f}%\".format(sumlen/len(dataset), npos, len(dataset) - npos, npos/len(dataset)*100))"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Average length of a review = 120.2169 words\n",
            "\n",
            "Class Distribution:\n",
            "Positive reviews: 25000 | Negative reviews: 25000 | Proportion of positive reviews: 50.0000%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_FkJ-e2pUwun"
      },
      "source": [
        "# Naive Bayes classifier"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eVq-mN28U_J4"
      },
      "source": [
        "# get reviews column from df\n",
        "reviews = dataset['review'].values\n",
        "\n",
        "# get labels column from df\n",
        "labels = dataset['sentiment'].values"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FjUsY4CqgZ1v"
      },
      "source": [
        "# Use label encoder to encode labels. Convert to 0/1\n",
        "encoder = LabelEncoder()\n",
        "encoded_labels = encoder.fit_transform(labels)\n",
        "dataset['encoded'] = encoded_labels\n",
        "encoder_mapping = dict(zip(encoder.classes_, encoder.transform(encoder.classes_)))\n",
        "labels = dataset['encoded']"
      ],
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wzG-C_EVWWET"
      },
      "source": [
        "# Split the data into train and test (80% - 20%). \n",
        "# Use stratify in train_test_split so that both train and test have similar ratio of positive and negative samples.\n",
        "train_sentences, test_sentences, train_labels, test_labels = train_test_split(reviews, labels, test_size = 0.2, stratify = labels)\n",
        "# train_sentences, test_sentences, train_labels, test_labels"
      ],
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bz1YdsSkiWCX"
      },
      "source": [
        "Here there are two approaches possible for building vocabulary for the naive Bayes.\n",
        "1. Take the whole data (train + test) to build the vocab. In this way while testing there is no word which will be out of vocabulary.\n",
        "2. Take the train data to build vocab. In this case, some words from the test set may not be in vocab and hence one needs to perform smoothing so that one the probability term is not zero.\n",
        " \n",
        "You are supposed to go by the 2nd approach.\n",
        " \n",
        "Also building vocab by taking all words in the train set is memory intensive, hence you are required to build vocab by choosing the top 2000 - 3000 frequent words in the training corpus.\n",
        "\n",
        "> $ P(x_i | w_j) = \\frac{ N_{x_i,w_j}\\, +\\, \\alpha }{ N_{w_j}\\, +\\, \\alpha*d} $\n",
        "\n",
        "\n",
        "$N_{x_i,w_j}$ : Number of times feature $x_i$ appears in samples of class $w_j$\n",
        "\n",
        "$N_{w_j}$ : Total count of features in class $w_j$\n",
        "\n",
        "$\\alpha$ : Parameter for additive smoothing. Here consider $\\alpha$ = 1\n",
        "\n",
        "$d$ : Dimentionality of the feature vector  $x = [x_1,x_2,...,x_d]$. In our case its the vocab size.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1cllNfGmUr77",
        "outputId": "ad2f1816-7b4a-4686-f596-f2222fceeb3e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "# Use Count vectorizer to get frequency of the words\n",
        "'''\n",
        "max_features parameter : If not None, build a vocabulary that only consider the top max_features ordered by term frequency across the corpus.\n",
        "vec = CountVectorizer(max_features = 3000)\n",
        "X = vec.fit_transform(Sentence_list)\n",
        "'''\n",
        "\n",
        "vec = CountVectorizer(max_features = 3000)\n",
        "X = vec.fit_transform(train_sentences)\n",
        "counts = X.sum(axis = 0).A1\n",
        "vocab = list(vec.get_feature_names())\n",
        "\n",
        "freq = Counter(dict(zip(vocab, counts)))\n",
        "\n",
        "print(\"The 100 most common words in the reviews are: \", freq.most_common(100), sep = '\\n')"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The 100 most common words in the reviews are: \n",
            "[('movie', 80720), ('film', 73550), ('one', 42981), ('like', 32039), ('time', 24037), ('good', 23217), ('character', 22282), ('story', 19782), ('even', 19653), ('get', 19466), ('would', 19061), ('make', 18957), ('see', 18933), ('really', 18342), ('scene', 16743), ('well', 15516), ('much', 15334), ('people', 14453), ('great', 14444), ('also', 14317), ('bad', 14299), ('first', 13773), ('show', 13771), ('dont', 13580), ('way', 13474), ('thing', 13111), ('made', 12325), ('think', 12085), ('could', 12062), ('life', 11849), ('go', 11558), ('know', 11403), ('watch', 11214), ('love', 10793), ('many', 10779), ('seen', 10619), ('plot', 10551), ('actor', 10545), ('two', 10402), ('say', 10364), ('never', 10288), ('end', 10226), ('look', 10128), ('acting', 10067), ('year', 9963), ('best', 9938), ('little', 9909), ('ever', 9451), ('better', 9050), ('man', 8945), ('take', 8909), ('come', 8821), ('still', 8682), ('work', 8534), ('find', 8124), ('part', 7978), ('something', 7945), ('want', 7853), ('give', 7756), ('lot', 7723), ('back', 7589), ('im', 7589), ('director', 7456), ('watching', 7257), ('performance', 7223), ('guy', 7218), ('real', 7216), ('doesnt', 7100), ('didnt', 7052), ('play', 6995), ('woman', 6975), ('funny', 6876), ('though', 6872), ('another', 6763), ('actually', 6716), ('role', 6628), ('nothing', 6588), ('going', 6473), ('new', 6394), ('every', 6333), ('old', 6309), ('girl', 6065), ('point', 6030), ('world', 6026), ('cant', 5992), ('cast', 5968), ('fact', 5904), ('day', 5900), ('thats', 5876), ('quite', 5811), ('feel', 5794), ('pretty', 5756), ('minute', 5737), ('comedy', 5711), ('got', 5687), ('thought', 5684), ('seems', 5662), ('around', 5621), ('young', 5586), ('horror', 5501)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xGUbhTMBjGR-"
      },
      "source": [
        "class Naive_Bayes:\n",
        "    def __init__(self, classes):\n",
        "      self.classes = classes\n",
        "\n",
        "    def smoothing(self, word, tclass):          \n",
        "      num = self.wcounts[tclass][word] + 1\n",
        "      den = self.n_items[tclass] + len(self.vocab)\n",
        "      return math.log(num / den)\n",
        "\n",
        "    def fit(self, X, y):\n",
        "        self.vocab = vocab\n",
        "        self.wcounts = {}\n",
        "        self.n_items = {}\n",
        "        self.log_p = {}\n",
        "        n = len(X)\n",
        "        grouped = self.group(X, y)\n",
        "        for c, data in grouped.items():\n",
        "          self.n_items[c] = len(data)\n",
        "          self.log_p[c] = math.log(self.n_items[c] / n) \n",
        "          self.wcounts[c] = defaultdict(lambda: 0)\n",
        "          for txt in data:\n",
        "            counts = Counter(nltk.word_tokenize(txt))\n",
        "            for word, count in counts.items():\n",
        "                self.wcounts[c][word] += count\n",
        "        return self\n",
        "\n",
        "    def predict(self, X):\n",
        "        result = []\n",
        "        for text in X:\n",
        "          scores = {c: self.log_p[c] for c in self.classes}\n",
        "          words = set(nltk.word_tokenize(text))\n",
        "          for word in words:\n",
        "              if word not in self.vocab: \n",
        "                continue\n",
        "              for c in self.classes:\n",
        "                log_wgc = self.smoothing(word, c)\n",
        "                scores[c] += log_wgc\n",
        "          result.append(max(scores, key = scores.get))\n",
        "        return result\n",
        "\n",
        "    def group(self, X, y):\n",
        "      data = {}\n",
        "      for c in self.classes:                          \n",
        "        data[c] = X[np.where(y == c)]\n",
        "      return data"
      ],
      "execution_count": 82,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GB40zLO9jWzh",
        "outputId": "844c72f6-6dfb-43ec-aa16-2fbc102cb35f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        }
      },
      "source": [
        "nb = Naive_Bayes(classes = np.unique(labels)).fit(train_sentences, train_labels)\n",
        "\n",
        "# Test the model on test set and report Accuracy\n",
        "predicted_labels = nb.predict(test_sentences)\n",
        "print(\"The accuracy of the Naive Bayes classifier is: \\n{:.4f}%\\n\\n\".format(accuracy_score(test_labels, predicted_labels) * 100))\n",
        "print(\"\\nThe classification report is as follows: \\n\\n\", classification_report(test_labels, predicted_labels))"
      ],
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The accuracy of the Naive Bayes classifier is: \n",
            "84.8300%\n",
            "\n",
            "\n",
            "\n",
            "The classification report is as follows: \n",
            "\n",
            "               precision    recall  f1-score   support\n",
            "\n",
            "           0       0.84      0.85      0.85      5000\n",
            "           1       0.85      0.84      0.85      5000\n",
            "\n",
            "    accuracy                           0.85     10000\n",
            "   macro avg       0.85      0.85      0.85     10000\n",
            "weighted avg       0.85      0.85      0.85     10000\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WlNql0acU7sa"
      },
      "source": [
        "# *LSTM* based Classifier\n",
        "\n",
        "Use the above train and test splits."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SkqnvbUOXoN0"
      },
      "source": [
        "# Hyperparameters of the model\n",
        "tokenizer = Tokenizer()\n",
        "tokenizer.fit_on_texts(train_sentences)\n",
        "word_index = tokenizer.word_index\n",
        "vocab_size = len(tokenizer.word_index) + 1\n",
        "oov_tok = '<OOK>'\n",
        "embedding_dim = 100\n",
        "max_length = 150\n",
        "padding_type='post'\n",
        "trunc_type='post'"
      ],
      "execution_count": 61,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UeycEg9nZAOF"
      },
      "source": [
        "# convert train dataset to sequence and pad sequences\n",
        "train_sequences = tokenizer.texts_to_sequences(train_sentences)\n",
        "train_padded = pad_sequences(train_sequences, padding='post', maxlen=max_length)\n",
        "\n",
        "# convert Test dataset to sequence and pad sequences\n",
        "test_sequences = tokenizer.texts_to_sequences(test_sentences)\n",
        "test_padded = pad_sequences(test_sequences, padding='post', maxlen=max_length)"
      ],
      "execution_count": 62,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PFld5wVHwMtr",
        "outputId": "6263eed9-26f0-4387-9032-cecfb120d107",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        }
      },
      "source": [
        "# model initialization\n",
        "model = keras.Sequential([\n",
        "    keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    keras.layers.Bidirectional(keras.layers.LSTM(64)),\n",
        "    keras.layers.Dense(24, activation='relu'),\n",
        "    keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "# compile model\n",
        "model.compile(loss='binary_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# model summary\n",
        "model.summary()"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (None, 150, 100)          13621300  \n",
            "_________________________________________________________________\n",
            "bidirectional (Bidirectional (None, 128)               84480     \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 24)                3096      \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 1)                 25        \n",
            "=================================================================\n",
            "Total params: 13,708,901\n",
            "Trainable params: 13,708,901\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HbS1RDOdwVZj",
        "outputId": "43152e17-2494-405c-a726-527218226913",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        }
      },
      "source": [
        "#training the model\n",
        "num_epochs = 5\n",
        "history = model.fit(train_padded, train_labels, \n",
        "                    epochs=num_epochs, verbose=1, \n",
        "                    validation_split=0.1)"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "1125/1125 [==============================] - 312s 278ms/step - loss: 0.3484 - accuracy: 0.8517 - val_loss: 0.2741 - val_accuracy: 0.8905\n",
            "Epoch 2/5\n",
            "1125/1125 [==============================] - 308s 274ms/step - loss: 0.1488 - accuracy: 0.9466 - val_loss: 0.3211 - val_accuracy: 0.8880\n",
            "Epoch 3/5\n",
            "1125/1125 [==============================] - 313s 278ms/step - loss: 0.0719 - accuracy: 0.9768 - val_loss: 0.4506 - val_accuracy: 0.8783\n",
            "Epoch 4/5\n",
            "1125/1125 [==============================] - 308s 273ms/step - loss: 0.0477 - accuracy: 0.9839 - val_loss: 0.4335 - val_accuracy: 0.8773\n",
            "Epoch 5/5\n",
            "1125/1125 [==============================] - 313s 278ms/step - loss: 0.0265 - accuracy: 0.9920 - val_loss: 0.5945 - val_accuracy: 0.8752\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TjEhWEr5Zq7M",
        "outputId": "8a203ae4-bcfb-47ff-b180-d77e4bfb45e4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 538
        }
      },
      "source": [
        "# Get probabilities\n",
        "prediction = model.predict(test_padded)\n",
        "print(\"Probabilities: \", prediction, sep='\\n')\n",
        "\n",
        "# Get labels based on probability 1 if p>= 0.5 else 0\n",
        "for p in prediction:\n",
        "  if p[0] >= 0.5:\n",
        "    p[0] = 1\n",
        "  else:\n",
        "    p[0] = 0\n",
        "\n",
        "prediction = prediction.astype('int32') \n",
        "print(\"\\nLabels:\", prediction, sep='\\n')\n",
        "\n",
        "# Calculate accuracy on Test data\n",
        "print(\"\\nAccuracy of the model: {:.4f}%\\n\".format(accuracy_score(test_labels, prediction) * 100))\n",
        "print(\"Classification report: \\n\", classification_report(test_labels, prediction, labels = [0, 1]), sep='\\n')\n"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Probabilities: \n",
            "[[5.7497329e-01]\n",
            " [1.8772483e-04]\n",
            " [9.3421090e-01]\n",
            " ...\n",
            " [2.2846460e-04]\n",
            " [1.4536679e-03]\n",
            " [5.0371885e-04]]\n",
            "\n",
            "Labels:\n",
            "[[1]\n",
            " [0]\n",
            " [1]\n",
            " ...\n",
            " [0]\n",
            " [0]\n",
            " [0]]\n",
            "\n",
            "Accuracy of the model: 87.4100%\n",
            "\n",
            "Classification report: \n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.89      0.85      0.87      5000\n",
            "           1       0.86      0.90      0.88      5000\n",
            "\n",
            "    accuracy                           0.87     10000\n",
            "   macro avg       0.87      0.87      0.87     10000\n",
            "weighted avg       0.87      0.87      0.87     10000\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TIICV-ySOYL0"
      },
      "source": [
        "## Get predictions for random examples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iXCGRLzawdqT",
        "outputId": "8c406940-f7b4-416b-fd53-a7df503bb24a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 168
        }
      },
      "source": [
        "# reviews on which we need to predict\n",
        "sentence = [\"The movie was very touching and heart whelming\", \n",
        "            \"I have never seen a terrible movie like this\", \n",
        "            \"the movie plot is terrible but it had good acting\"]\n",
        "\n",
        "# convert to a sequence\n",
        "sequences = tokenizer.texts_to_sequences(sentence)\n",
        "\n",
        "# pad the sequence\n",
        "padded = pad_sequences(sequences, padding='post', maxlen=max_length)\n",
        "\n",
        "# Get probabilities\n",
        "prediction = model.predict(padded)\n",
        "print(\"The probabilities are:\", prediction, sep='\\n')\n",
        "\n",
        "# Get labels based on probability 1 if p>= 0.5 else 0\n",
        "for p in prediction:\n",
        "    if p[0] >=0.5:\n",
        "        p[0] = 1\n",
        "    else:\n",
        "        p[0] = 0\n",
        "prediction = prediction.astype('int32') \n",
        "print(\"\\nLabels:\", prediction, sep='\\n')"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The probabilities are:\n",
            "[[0.94367266]\n",
            " [0.01868612]\n",
            " [0.00555304]]\n",
            "\n",
            "Labels:\n",
            "[[1]\n",
            " [0]\n",
            " [0]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELOQ96aIP5AW"
      },
      "source": [
        "**Using the Naive Bayes model, we get an accuracy of ~85%, while an accuracy of ~87% is obtained on using the LSTM model. Therefore, we can conclude that LSTM is a better model as compared to Naive Bayes for this task.**"
      ]
    }
  ]
}