Preprocessing_SQuAD.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Preprocessing_SQuAD.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kzafeiroudi/QuestRecommend/blob/master/Preprocessing_SQuAD.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "2SJrDBgZ1hA3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Preprocessing the SQuAD dataset"
      ]
    },
    {
      "metadata": {
        "id": "tT0f3WU51nmT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## The dataset\n",
        "\n",
        "After looking into the Quora Question Pairs dataset, it is now time to apply the deveoped process on the SQuAD dataset. The part of the SQuAD dataset that we are going to use here consists of:\n",
        "\n",
        "*  3111 unique questions\n",
        "*  30 unique article topics from Wikipedia\n",
        "\n",
        "Each instance of this part of the dataset has the following attributes:\n",
        "* sentID : A unique identifier for each question\n",
        "* Question: The full text of the question\n",
        "* Class: The article title of the Wikipedia page the question was derived from, used here as a nominal class"
      ]
    },
    {
      "metadata": {
        "id": "Uk81xvkJ5O6L",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Upload the dataset\n",
        "\n",
        "Upload the file `squad.csv` that will be used in this Python 3 notebook."
      ]
    },
    {
      "metadata": {
        "id": "pxHQmS7fC2jf",
        "colab_type": "code",
        "outputId": "fba62f1b-9b7b-4d6b-eeb1-a1803752bc17",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "\n",
        "# Choose from your own machine the file to upload - name should be \"squad.csv\"\n",
        "uploaded = files.upload()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-6fe2b7e7-cc2c-4c4d-bb83-d1eed2ff0813\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-6fe2b7e7-cc2c-4c4d-bb83-d1eed2ff0813\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving squad.csv to squad.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "q9VtpJqZ5RPG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Download a pre-trained English language model\n",
        "\n",
        "We will be using the large pre-trained statistical model for English, available by the **spaCy** free open-source library for NLP in Python. Find more [here](https://spacy.io/models/en#en_core_web_lg)."
      ]
    },
    {
      "metadata": {
        "id": "5zzfOAH1Dqcp",
        "colab_type": "code",
        "outputId": "ee527da5-f814-4b76-bc25-df93501c029c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 309
        }
      },
      "cell_type": "code",
      "source": [
        "!python -m spacy download en_core_web_lg\n",
        "\n",
        "# Load the model\n",
        "import spacy\n",
        "nlp = spacy.load('en_core_web_lg')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting en_core_web_lg==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.0.0/en_core_web_lg-2.0.0.tar.gz#egg=en_core_web_lg==2.0.0\n",
            "\u001b[?25l  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.0.0/en_core_web_lg-2.0.0.tar.gz (852.3MB)\n",
            "\u001b[K     |████████████████████████████████| 852.3MB 1.2MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: en-core-web-lg\n",
            "  Building wheel for en-core-web-lg (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-ppjihu0k/wheels/0d/bc/67/e6a9108ab86cd076703af19ad4e0f02f57381ac6583df16249\n",
            "Successfully built en-core-web-lg\n",
            "Installing collected packages: en-core-web-lg\n",
            "Successfully installed en-core-web-lg-2.0.0\n",
            "\n",
            "\u001b[93m    Linking successful\u001b[0m\n",
            "    /usr/local/lib/python3.6/dist-packages/en_core_web_lg -->\n",
            "    /usr/local/lib/python3.6/dist-packages/spacy/data/en_core_web_lg\n",
            "\n",
            "    You can now load the model via spacy.load('en_core_web_lg')\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "b5IEY7wm5Zih",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Importing Python libraries"
      ]
    },
    {
      "metadata": {
        "id": "e2bPUiQ35cpc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import csv\n",
        "import numpy as np\n",
        "import random\n",
        "from prettytable import PrettyTable"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4cRZLpKJ57aP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load the dataset"
      ]
    },
    {
      "metadata": {
        "id": "nw0AyuYfDrb_",
        "colab_type": "code",
        "outputId": "7454d8a3-9ba6-4e91-9cd3-869a1e060e32",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 646
        }
      },
      "cell_type": "code",
      "source": [
        "# Loading the SQuAD dataset\n",
        "data_file = 'squad.csv'\n",
        "new_data = []\n",
        "with open(data_file) as csvfile:\n",
        "    reader = csv.DictReader(csvfile)\n",
        "    for row in reader:\n",
        "        new_data.append(row)\n",
        "\n",
        "# Extracting the unique questions from the dataset, and all unique topics\n",
        "uniqueQ = []\n",
        "topics = {}\n",
        "for dd in new_data:\n",
        "    uniqueQ.append(dd['Question'])\n",
        "    if (dd['Class'] in topics):\n",
        "      topics[dd['Class']].append(dd['Question'])\n",
        "    else:\n",
        "      topics[dd['Class']] = [dd['Question']]\n",
        "\n",
        "# Print stats for the dataset\n",
        "print('Unique Questions: ', len(uniqueQ))\n",
        "print('Unique Topics: ', len(topics.keys()))\n",
        "print()\n",
        "\n",
        "# Print stats for each topic\n",
        "t = PrettyTable(['Topic', '# of Questions'])\n",
        "for tt in topics:\n",
        "  t.add_row([tt, len(topics[tt])])\n",
        "print(t)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Unique Questions:  3111\n",
            "Unique Topics:  30\n",
            "\n",
            "+---------------------------------+----------------+\n",
            "|              Topic              | # of Questions |\n",
            "+---------------------------------+----------------+\n",
            "|          Super_Bowl_50          |      106       |\n",
            "|              Warsaw             |      104       |\n",
            "|             Normans             |      101       |\n",
            "|           Nikola_Tesla          |      108       |\n",
            "| Computational_complexity_theory |      103       |\n",
            "|             Teacher             |      103       |\n",
            "|          Martin_Luther          |      102       |\n",
            "|       Southern_California       |      101       |\n",
            "|       Sky_(United_Kingdom)      |      103       |\n",
            "|       Victoria_(Australia)      |      104       |\n",
            "|             Huguenot            |      104       |\n",
            "|           Steam_engine          |      105       |\n",
            "|              Oxygen             |      105       |\n",
            "|         1973_oil_crisis         |      102       |\n",
            "|          Apollo_program         |      103       |\n",
            "|        European_Union_law       |      107       |\n",
            "|        Amazon_rainforest        |      108       |\n",
            "|            Ctenophora           |      105       |\n",
            "|        Fresno,_California       |      103       |\n",
            "|         Packet_switching        |      101       |\n",
            "|           Black_Death           |      104       |\n",
            "|             Geology             |      102       |\n",
            "|       Newcastle_upon_Tyne       |      104       |\n",
            "|    Victoria_and_Albert_Museum   |      103       |\n",
            "|  American_Broadcasting_Company  |      102       |\n",
            "|           Genghis_Khan          |      104       |\n",
            "|             Pharmacy            |      105       |\n",
            "|          Immune_system          |      105       |\n",
            "|        Civil_disobedience       |      106       |\n",
            "|           Construction          |       98       |\n",
            "+---------------------------------+----------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "KoOtI36QAAMd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Implementing a cosine similarity function\n",
        "\n",
        "`cosine(vA, vB)` calculates the cosine similarity between two vectors `vA` and `vB`."
      ]
    },
    {
      "metadata": {
        "id": "VtXwn27OABJE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def cosine(vA, vB):\n",
        "  cos = np.dot(vA, vB) / (np.sqrt(np.dot(vA, vA)) * np.sqrt(np.dot(vB, vB)))\n",
        "  return cos.astype('float64')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "th7x4i1gEsT8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Calculating the question vector representation\n",
        "\n",
        "We will use only the main verb arguments of the question to calculate the question embeddings, since the process we followed for the Quora Question Pairs dataset showed that question are less likely to be considered duplicates when they are not.\n",
        "\n",
        "The final result is saved in the file `squad_vectors.csv`, which will then be used to perform classification and clustering."
      ]
    },
    {
      "metadata": {
        "id": "1Lvk9Z-qD8sr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "squad = []\n",
        "for i in range(len(new_data)):\n",
        "  foo = nlp(new_data[i]['Question'])\n",
        "  calc = [np.zeros(300, dtype='float64')]\n",
        "  for nn in foo.noun_chunks:\n",
        "      calc.append(nn.vector)\n",
        "  vA = np.mean(calc, axis=0).astype('float64')\n",
        "  squad.append(vA)\n",
        "\n",
        "toCSV = []\n",
        "for i in range(len(new_data)):\n",
        "  boo = {}\n",
        "  boo['Class'] = new_data[i]['Class']\n",
        "  for j in range(300):\n",
        "    boo['vector_' + str(j+1)] = squad[i][j]\n",
        "  toCSV.append(boo)\n",
        "\n",
        "with open('squad_vectors.csv', 'w') as csvfile:\n",
        "  fieldnames = list(boo.keys())\n",
        "  writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n",
        "  writer.writeheader()\n",
        "  for i in range(len(toCSV)):\n",
        "    writer.writerow(toCSV[i])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7TpRIiDt8Sfv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Testing whether the dataset is large enough for the task\n",
        "\n",
        "In this section, we calculate the mean vector representation across all questions that belong in the same class, showcasing that these vectors do not coincide, and thus we have a heterogenous dataset good enough for the task.\n",
        "\n",
        "As we can see, by running this code, there is no occurence of *Same means* in the standard output."
      ]
    },
    {
      "metadata": {
        "id": "xLphPI4X8TER",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# cl: all different classes\n",
        "classes = list(topics.keys())\n",
        "\n",
        "# calc: to calculate the mean vector representation per class\n",
        "calc = {}\n",
        "for cc in classes:\n",
        "  calc[cc] = []\n",
        "\n",
        "for i in range(len(new_data)):\n",
        "  calc[new_data[i]['Class']].append(squad[i])\n",
        "\n",
        "# means: to store the mean vector representation of each class\n",
        "means = {}\n",
        "\n",
        "for cc in classes:\n",
        "  means[cc] = np.mean(calc[cc], axis=0)\n",
        "\n",
        "for i in range(len(classes)):\n",
        "  for j in range(i+1, len(classes)):\n",
        "    if (any(means[keys[i]] == means[keys[j]])):\n",
        "      print('Same means')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lRL0RkVLTLb5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Evaluating the results if we were to rely completely on the cosine similarity\n",
        "\n",
        "For this task, we will pick randomly 100 questions, we will calculate the vector representation on the full body of the question, and by calculating the cosine similarity between split1 (34 questions) and split2 (66 questions), we will show the questions that is suggestive to group together due to the process we have developed so far."
      ]
    },
    {
      "metadata": {
        "id": "92zQ7gJoP9Le",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Shuffling the sequence of questions\n",
        "new = list(range(len(new_data)))\n",
        "random.shuffle(new)\n",
        "\n",
        "# Picking randomly 340 for split1\n",
        "# and 660 for split2\n",
        "split1 = {}\n",
        "split2 = {}\n",
        "for i in new[:340]:\n",
        "  q = new_data[i]['Question']\n",
        "  split1[q] = {'vec': nlp(q), 'class' : new_data[i]['Class']}\n",
        "for i in new[340:1000]:\n",
        "  q = new_data[i]['Question']\n",
        "  split2[q] = {'vec': nlp(q), 'class' : new_data[i]['Class']}\n",
        "\n",
        "# Calculate the question vector on the full body of the question\n",
        "# and save the questions from split2 that are closely related to \n",
        "# questions from split1 (cosine similarity > 0.93 as per our first\n",
        "# classification task)\n",
        "rank = {}\n",
        "for tt in split1:\n",
        "  rank[tt] = []\n",
        "  vA = split1[tt]['vec'].vector\n",
        "  for tr in split2:\n",
        "    vB = split2[tr]['vec'].vector\n",
        "    cos = cosine(vA, vB)\n",
        "    if (cos > 0.93):\n",
        "      rank[tt].append(tr)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rMWlsJgATqyM",
        "colab_type": "code",
        "outputId": "10b85644-9ea9-4521-8437-08cecd048f8e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 5970
        }
      },
      "cell_type": "code",
      "source": [
        "for q in rank:\n",
        "  if(rank[q] != []):\n",
        "    print('Input Question:')\n",
        "    print('\\t-',q)\n",
        "    print('Related Questions:')\n",
        "    for rr in rank[q]:\n",
        "      print('\\t-', rr)\n",
        "    print()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input Question:\n",
            "\t- Which two treaties provided more formal institutions of the European Union?\n",
            "Related Questions:\n",
            "\t- Which treaty provides that the European Union law be applied to metropolitan territories of member states?\n",
            "\n",
            "Input Question:\n",
            "\t- In what types of organisms did the adaptive immune system first evolve?\n",
            "Related Questions:\n",
            "\t- What kind of immune system do bacteria have?\n",
            "\t- The adaptive immune system must distinguish between what types of molecules?\n",
            "\n",
            "Input Question:\n",
            "\t- Where did the water in the Amazon Basin flow towards when moving west?\n",
            "Related Questions:\n",
            "\t- Where did water to the east of the Amazon drainage basin flow towards?\n",
            "\t- Where did water on the eastern side of the amazon basin travel after the split?\n",
            "\n",
            "Input Question:\n",
            "\t- Rather than the fuel, what is oxygen to a fire?\n",
            "Related Questions:\n",
            "\t- What is a spark or heat to the progress of a fire?\n",
            "\n",
            "Input Question:\n",
            "\t- Phagocytes can be called to a specific location by what?\n",
            "Related Questions:\n",
            "\t- What entity is created if the three different institutions cannot come to a consensus at any stage?\n",
            "\n",
            "Input Question:\n",
            "\t- Approximately how many architectural drawings does the V&A hold in its collection?\n",
            "Related Questions:\n",
            "\t- Approximately how many architectural photographs does the V&A hold in its collection?\n",
            "\n",
            "Input Question:\n",
            "\t- Which NFL team won Super Bowl 50?\n",
            "Related Questions:\n",
            "\t- Which team won Super Bowl 50.\n",
            "\n",
            "Input Question:\n",
            "\t- What was the first point of the Reformation?\n",
            "Related Questions:\n",
            "\t- What was the principle shown in The Mark of Anarchy?\n",
            "\t- What device was one of the first to aid a controversy?\n",
            "\t- What was the name of the first Huguenot church in the New World?\n",
            "\n",
            "Input Question:\n",
            "\t- Name a text that might be used by a religious teacher to teach.\n",
            "Related Questions:\n",
            "\t- What is another general name for a religious teacher?\n",
            "\t- What is a name for the different ways to instruct students?\n",
            "\n",
            "Input Question:\n",
            "\t- What city in Victoria is called the sporting capital of Australia?\n",
            "Related Questions:\n",
            "\t- What city is the capital of Victoria?\n",
            "\n",
            "Input Question:\n",
            "\t- After its re-opening, which types of movies did the Tower Theatre show?\n",
            "Related Questions:\n",
            "\t- During what period did the Tower Theatre re-open?\n",
            "\n",
            "Input Question:\n",
            "\t- How much food does a ctenophora eat in a day?\n",
            "Related Questions:\n",
            "\t- How much can Ctenophores eat in one day?\n",
            "\n",
            "Input Question:\n",
            "\t- The success of the Britain Can Make It exhibition led to the planning of what exhibition in 1951?\n",
            "Related Questions:\n",
            "\t- What does the world's first Museum of Posters have one of the largest collections of in the world?\n",
            "\n",
            "Input Question:\n",
            "\t- What was the title given to the major project to redesign all the galleries in the museum?\n",
            "Related Questions:\n",
            "\t- What does the world's first Museum of Posters have one of the largest collections of in the world?\n",
            "\n",
            "Input Question:\n",
            "\t- What types of responsibilities might a pharmacy technician have?\n",
            "Related Questions:\n",
            "\t- What is one example of what a clinical pharmacist's duties entail?\n",
            "\n",
            "Input Question:\n",
            "\t- What did Tesla do in Tomingaj?\n",
            "Related Questions:\n",
            "\t- Where did Tesla teach in Gospic?\n",
            "\n",
            "Input Question:\n",
            "\t- What does the average temperatures exceed in the summer?\n",
            "Related Questions:\n",
            "\t- What is the high end of the temperature range in winter?\n",
            "\n",
            "Input Question:\n",
            "\t- Who is most likely to teach a child at home?\n",
            "Related Questions:\n",
            "\t- If someone is being taught at their place of residence, what is it called?\n",
            "\n",
            "Input Question:\n",
            "\t- What type of approaches for primary school are available that are different than the norm?\n",
            "Related Questions:\n",
            "\t- Time and space are both examples of what type of resource?\n",
            "\n",
            "Input Question:\n",
            "\t- What is the exact date of the V&A's official opening?\n",
            "Related Questions:\n",
            "\t- What is the name of the latest epoch?\n",
            "\n",
            "Input Question:\n",
            "\t- Who was Margaret's husband?\n",
            "Related Questions:\n",
            "\t- Who was Margaret's brother?\n",
            "\t- Who was Emma's brother?\n",
            "\n",
            "Input Question:\n",
            "\t- When was Sky Digital launched?\n",
            "Related Questions:\n",
            "\t- What satellite was used when Sky digital was launched?\n",
            "\n",
            "Input Question:\n",
            "\t- What fixed set of factors determine the actions of a deterministic Turing machine\n",
            "Related Questions:\n",
            "\t- What is the expression used to identify any given series of problems capable of being solved within time on a deterministic Turing machine?\n",
            "\n",
            "Input Question:\n",
            "\t- What do some community pharmacies do?\n",
            "Related Questions:\n",
            "\t- What practice do some internet pharmacies engage in?\n",
            "\n",
            "Input Question:\n",
            "\t- The traveling salesman problem is an example of what type of problem?\n",
            "Related Questions:\n",
            "\t- A function problem is an example of what?\n",
            "\t- What is one example of an instance that the quantitative answer to the traveling salesman problem fails to answer?\n",
            "\n",
            "Input Question:\n",
            "\t- Newcastle was one of the first cities in the world to have what innovation?\n",
            "Related Questions:\n",
            "\t- From what country did the first person in space come from?\n",
            "\t- What was the name of the first Huguenot church in the New World?\n",
            "\t- What does the world's first Museum of Posters have one of the largest collections of in the world?\n",
            "\n",
            "Input Question:\n",
            "\t- Who was given the esteemed status of MVP for Super Bowl 50?\n",
            "Related Questions:\n",
            "\t- Who was the Super Bowl 50 MVP?\n",
            "\n",
            "Input Question:\n",
            "\t- How much Saharan dust remains in the air over the Amazon each year?\n",
            "Related Questions:\n",
            "\t- How much dust is blown out of the Sahara each year?\n",
            "\t- How many tons of dust remains in the air?\n",
            "\t- How much Saharan dust is blown and falls upon the Caribbean Sea each year?\n",
            "\n",
            "Input Question:\n",
            "\t- How much windblown dust leaves the Sahara each year?\n",
            "Related Questions:\n",
            "\t- How much dust is blown out of the Sahara each year?\n",
            "\n",
            "Input Question:\n",
            "\t- What department store is thought to be the first in the world?\n",
            "Related Questions:\n",
            "\t- If someone is being taught at their place of residence, what is it called?\n",
            "\t- From what country did the first person in space come from?\n",
            "\t- Where can a tribute to the fall of Warsaw be found?\n",
            "\t- What device was one of the first to aid a controversy?\n",
            "\t- What was the name of the first Huguenot church in the New World?\n",
            "\t- What does the world's first Museum of Posters have one of the largest collections of in the world?\n",
            "\n",
            "Input Question:\n",
            "\t- What does man's justification depend on in faith?\n",
            "Related Questions:\n",
            "\t- What theology states that faith alone isn't enough to justify man?\n",
            "\n",
            "Input Question:\n",
            "\t- What is the name of the alphabet is most commonly used in a problem instance?\n",
            "Related Questions:\n",
            "\t- What is the general term used to describe the output to any given input in a problem instance?\n",
            "\t- What is the name for a teacher of just one person?\n",
            "\t- In a computational problem, what can be described as a string over an alphabet?\n",
            "\t- A function problem is an example of what?\n",
            "\n",
            "Input Question:\n",
            "\t- What kind of problems are one of the main topics studied in computational complexity theory?\n",
            "Related Questions:\n",
            "\t- What is the most commonplace model utilized in complexity theory?\n",
            "\n",
            "Input Question:\n",
            "\t- How much do ctenophore eat in a day?\n",
            "Related Questions:\n",
            "\t- How much can Ctenophores eat in one day?\n",
            "\n",
            "Input Question:\n",
            "\t- What was the Plos Pathogens paper about?\n",
            "Related Questions:\n",
            "\t- When did the Plos Pathogens paper come out?\n",
            "\n",
            "Input Question:\n",
            "\t- How many counties initially made up the definition of southern California?\n",
            "Related Questions:\n",
            "\t- The term \"southern\" California usually refers to how many of the southern-most counties of the state?\n",
            "\t- The two AAA clubs divided the state into a northern and southern California as opposed to what point of view?\n",
            "\n",
            "Input Question:\n",
            "\t- What is the name given to the district that is associated with the motion picture industry?\n",
            "Related Questions:\n",
            "\t- What device was one of the first to aid a controversy?\n",
            "\t- What is the name of the latest epoch?\n",
            "\t- What is the name of the law which imposed the speed limit?\n",
            "\t- What is one issue that adds to the complexity of a pharmacist's job?\n",
            "\n",
            "Input Question:\n",
            "\t- What are the agents detected by the immune system called?\n",
            "Related Questions:\n",
            "\t- What molecules are recognized as foreign by the immune system?\n",
            "\n",
            "Input Question:\n",
            "\t- Which part of the immune system protects the brain?\n",
            "Related Questions:\n",
            "\t- What separates the neuroimmune system and peripheral immune system in humans?\n",
            "\t- What is the subsystem that protects the human brain?\n",
            "\t- What field involves the study of the immune system?\n",
            "\n",
            "Input Question:\n",
            "\t- Which city is the most populous in California?\n",
            "Related Questions:\n",
            "\t- What is the eighth most populous city in the nation?\n",
            "\n",
            "Input Question:\n",
            "\t- In the determination of complexity classes, what are two examples of types of Turing machines?\n",
            "Related Questions:\n",
            "\t- Time and space are both examples of what type of resource?\n",
            "\n",
            "Input Question:\n",
            "\t- What was the average cost for a 30 second commercial during Super Bowl 50?\n",
            "Related Questions:\n",
            "\t- What was the average cost for a TV ad lasting 30 seconds during Super Bowl 50?\n",
            "\n",
            "Input Question:\n",
            "\t- What is involved in a review of prescribed medications?\n",
            "Related Questions:\n",
            "\t- What is one example of what a clinical pharmacist's duties entail?\n",
            "\n",
            "Input Question:\n",
            "\t- Outside of national recognition, what other kind of recognition do some of the cities get?\n",
            "Related Questions:\n",
            "\t- Cities that anchor the regions are often the hub for what kind of activity?\n",
            "\n",
            "Input Question:\n",
            "\t- What is the term for a task that generally lends itself to being solved by a computer?\n",
            "Related Questions:\n",
            "\t- What is the general term used to describe the output to any given input in a problem instance?\n",
            "\t- Is the output of a functional problem typically characterized by a simple or complex answer?\n",
            "\t- What device was one of the first to aid a controversy?\n",
            "\t- In a computational problem, what can be described as a string over an alphabet?\n",
            "\t- What is one issue that adds to the complexity of a pharmacist's job?\n",
            "\t- A function problem is an example of what?\n",
            "\t- What is one example of an instance that the quantitative answer to the traveling salesman problem fails to answer?\n",
            "\t- What field of computer science is primarily concerned with determining the likelihood of whether or not a problem can ultimately be solved using algorithms?\n",
            "\t- What is the expression used to identify any given series of problems capable of being solved within time on a deterministic Turing machine?\n",
            "\n",
            "Input Question:\n",
            "\t- What provides a solution to a list of integers provided as input that ned to be sorted?\n",
            "Related Questions:\n",
            "\t- What is the general term used to describe the output to any given input in a problem instance?\n",
            "\n",
            "Input Question:\n",
            "\t- What are the secondary sources of primary law?\n",
            "Related Questions:\n",
            "\t- What are the main sources of primary law?\n",
            "\n",
            "Input Question:\n",
            "\t- In lands attributed to what tribe are found remains of large settlements?\n",
            "Related Questions:\n",
            "\t- In which region tribe were large settlements discovered?\n",
            "\n",
            "Input Question:\n",
            "\t- Which phylum is more complex than sponges?\n",
            "Related Questions:\n",
            "\t- Ctenophora are less complex than which other phylum?\n",
            "\n",
            "Input Question:\n",
            "\t- Who was Robert's son?\n",
            "Related Questions:\n",
            "\t- Who was Margaret's brother?\n",
            "\t- Who was Emma's brother?\n",
            "\n",
            "Input Question:\n",
            "\t- Which performers joined the headliner during the Super Bowl 50 halftime show?\n",
            "Related Questions:\n",
            "\t- Which group headlined the Super Bowl 50 halftime show?\n",
            "\n",
            "Input Question:\n",
            "\t- What types of discipline did the teacher have access to?\n",
            "Related Questions:\n",
            "\t- What type of discipline do critics want to see?\n",
            "\t- What is a name for the different ways to instruct students?\n",
            "\n",
            "Input Question:\n",
            "\t- Assuming that T represents a polynomial in T(n), what is the term given to the corresponding algorithm?\n",
            "Related Questions:\n",
            "\t- What is the general term used to describe the output to any given input in a problem instance?\n",
            "\n",
            "Input Question:\n",
            "\t- What is the low end of the temperature range in summer?\n",
            "Related Questions:\n",
            "\t- What is the temperature in the highest portion of the mountain range in winter?\n",
            "\t- What is the high end of the temperature range in winter?\n",
            "\n",
            "Input Question:\n",
            "\t- What does the immune system protect against?\n",
            "Related Questions:\n",
            "\t- What is one part of the innate immune system that doesn't attack microbes directly?\n",
            "\t- The immune system protects organisms against what?\n",
            "\n",
            "Input Question:\n",
            "\t- Which article of the Treaty on European Union states that Commissioners should be completely independent and not take instructions from any Government?\n",
            "Related Questions:\n",
            "\t- What powers does the Court of Justice of the European Union have in regards to treaties?\n",
            "\t- Which articles state that powers stay with member states unless they've been conferred?\n",
            "\t- Which treaty provides that the European Union law be applied to metropolitan territories of member states?\n",
            "\t- Can the President of the Council vote on important matters related to the European Central Bank?\n",
            "\n",
            "Input Question:\n",
            "\t- In what year did Tesla die?\n",
            "Related Questions:\n",
            "\t- In what year did Tesla's father die?\n",
            "\n",
            "Input Question:\n",
            "\t- What was the advantage that the LOR method produced in the event of a spacecraft failure?\n",
            "Related Questions:\n",
            "\t- What device was one of the first to aid a controversy?\n",
            "\n",
            "Input Question:\n",
            "\t- Under what instances can individuals rely on primary law in the Court of Justice of European Union?\n",
            "Related Questions:\n",
            "\t- What are the three sources of European Union law?\n",
            "\t- What powers does the Court of Justice of the European Union have in regards to treaties?\n",
            "\t- What effect does European Union law have on laws of member states?\n",
            "\t- Who can enforce the European Union law when member states provide lesser rights?\n",
            "\n",
            "Input Question:\n",
            "\t- Who is most likely to be doing formal teaching?\n",
            "Related Questions:\n",
            "\t- Who might be responsible for student discipline?\n",
            "\t- Who would decide on the curricula that is consistent and standard?\n",
            "\n",
            "Input Question:\n",
            "\t- Which of his future generals joined forces with Temüjin around the time of his escape from the Tayichi'ud?\n",
            "Related Questions:\n",
            "\t- Which of his future general's fathers helped Temüjin escape the Tayichi'ud?\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "GgSxVyONZE3v",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Upload the context data"
      ]
    },
    {
      "metadata": {
        "id": "JesgA4zWZA3E",
        "colab_type": "code",
        "outputId": "75d41941-9098-4c43-fe09-b40676081a5a",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "cell_type": "code",
      "source": [
        "# Choose from your own machine the file to upload - name should be \"squad_contexts.csv\"\n",
        "uploaded = files.upload()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-f58f065f-ac96-4f6b-9ca2-ad9b07cf3f09\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-f58f065f-ac96-4f6b-9ca2-ad9b07cf3f09\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving squad_contexts.csv to squad_contexts.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "N22AA_UbZSv-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load the context data"
      ]
    },
    {
      "metadata": {
        "id": "n39eY03kZTCU",
        "colab_type": "code",
        "outputId": "42a4b96c-d2a3-4048-ed58-a1b236827483",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 629
        }
      },
      "cell_type": "code",
      "source": [
        "# Loading the SQuAD context dataset\n",
        "data_file = 'squad_contexts.csv'\n",
        "new_data = []\n",
        "with open(data_file) as csvfile:\n",
        "    reader = csv.DictReader(csvfile)\n",
        "    for row in reader:\n",
        "        new_data.append(row)\n",
        "\n",
        "# Extracting the unique classes/topics from the dataset, and all contexts\n",
        "contexts = {}\n",
        "for dd in new_data:\n",
        "    if (dd['Superclass'] in contexts):\n",
        "      contexts[dd['Superclass']].append(dd['Context'])\n",
        "    else:\n",
        "      contexts[dd['Superclass']] = [dd['Context']]\n",
        "\n",
        "# Print stats for the dataset\n",
        "print('Unique Topics: ', len(contexts))\n",
        "print()\n",
        "\n",
        "# Print stats for each topic\n",
        "t = PrettyTable(['Topic', '# of Contexts'])\n",
        "for tt in contexts:\n",
        "  t.add_row([tt, len(contexts[tt])])\n",
        "print(t)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Unique Topics:  30\n",
            "\n",
            "+---------------------------------+---------------+\n",
            "|              Topic              | # of Contexts |\n",
            "+---------------------------------+---------------+\n",
            "|          Super_Bowl_50          |       5       |\n",
            "|              Warsaw             |       22      |\n",
            "|             Normans             |       41      |\n",
            "|           Nikola_Tesla          |       13      |\n",
            "| Computational_complexity_theory |       26      |\n",
            "|             Teacher             |       23      |\n",
            "|          Martin_Luther          |       22      |\n",
            "|       Southern_California       |       22      |\n",
            "|       Sky_(United_Kingdom)      |       22      |\n",
            "|       Victoria_(Australia)      |       22      |\n",
            "|             Huguenot            |       22      |\n",
            "|           Steam_engine          |       24      |\n",
            "|              Oxygen             |       15      |\n",
            "|         1973_oil_crisis         |       24      |\n",
            "|          Apollo_program         |       25      |\n",
            "|        European_Union_law       |       12      |\n",
            "|        Amazon_rainforest        |       9       |\n",
            "|            Ctenophora           |       15      |\n",
            "|        Fresno,_California       |       22      |\n",
            "|         Packet_switching        |       23      |\n",
            "|           Black_Death           |       23      |\n",
            "|             Geology             |       23      |\n",
            "|       Newcastle_upon_Tyne       |       22      |\n",
            "|    Victoria_and_Albert_Museum   |       21      |\n",
            "|  American_Broadcasting_Company  |       23      |\n",
            "|           Genghis_Khan          |       23      |\n",
            "|             Pharmacy            |       22      |\n",
            "|          Immune_system          |       19      |\n",
            "|        Civil_disobedience       |       12      |\n",
            "|           Construction          |       22      |\n",
            "+---------------------------------+---------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "VCZ-zE4AaIU5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Calculating the context vector representation\n",
        "\n",
        "We will use only the main verb arguments of the context to calculate the context embeddings, as before.\n",
        "\n",
        "The final result is saved in the file `squad_contextsVectors.csv`, which will then be used to perform classification and clustering."
      ]
    },
    {
      "metadata": {
        "id": "m2jZYNofaHdR",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "squad_cont = []\n",
        "for i in range(len(new_data)):\n",
        "  foo = nlp(new_data[i]['Context'])\n",
        "  calc = [np.zeros(300, dtype='float64')]\n",
        "  for nn in foo.noun_chunks:\n",
        "      calc.append(nn.vector)\n",
        "  vA = np.mean(calc, axis=0).astype('float64')\n",
        "  squad_cont.append(vA)\n",
        "\n",
        "toCSV = []\n",
        "for i in range(len(new_data)):\n",
        "  boo = {}\n",
        "  boo['Class'] = new_data[i]['Superclass']\n",
        "  for j in range(300):\n",
        "    boo['vector_' + str(j+1)] = squad_cont[i][j]\n",
        "  toCSV.append(boo)\n",
        "\n",
        "with open('squad_contextsVectors.csv', 'w') as csvfile:\n",
        "  fieldnames = list(boo.keys())\n",
        "  writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n",
        "  writer.writeheader()\n",
        "  for i in range(len(toCSV)):\n",
        "    writer.writerow(toCSV[i])"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}