Preprocessing_Quora.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Preprocessing_Quora.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kzafeiroudi/QuestRecommend/blob/master/Preprocessing_Quora.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "14wquvXfznWM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Preprocessing the Quora Question Pairs dataset"
      ]
    },
    {
      "metadata": {
        "id": "QN3erKfPthqN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## The dataset\n",
        "\n",
        "For the purpose of this project, we first look into the Quora Question Pairs dataset that consists of:\n",
        "\n",
        "*  537933 unique questions\n",
        "*  404290 pairs of questions\n",
        "* 149263 pairs of questions marked as duplicates\n",
        "* 255027 pairs of questions marked as non duplicates\n",
        "\n",
        "Each instance of the dataset has the following attributes:\n",
        "* id : A unique identifier for each training set question pair\n",
        "* qid1: A unique identifier for the first question in the pair\n",
        "* qid2: A unique identifier for the second question in the pair\n",
        "* question1: The full text of the first question \n",
        "* question2: The full text of the second question\n",
        "* is_duplicate: The target variable, set to 1 if question1 and question2 have essentially the same meaning, and 0 otherwise"
      ]
    },
    {
      "metadata": {
        "id": "boq_CGJswRCs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Upload the dataset\n",
        "\n",
        "Upload the file *quora.csv* that will be used in this Python 3 notebook."
      ]
    },
    {
      "metadata": {
        "id": "diQhTZMN4q7k",
        "colab_type": "code",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "3ac719b3-fc61-45b3-e542-0c7a3cb335b7"
      },
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "\n",
        "# Choose from your own machine the file to upload - name should be \"quora.csv\"\n",
        "uploaded = files.upload()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-b239f5f6-a76b-4c73-a48b-0717db0a6238\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-b239f5f6-a76b-4c73-a48b-0717db0a6238\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving quora.csv to quora.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "anivhlqmzr8j",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Download a pre-trained English language model\n",
        "\n",
        "We will be using the large pre-trained statistical model for English, available by the **spaCy** free open-source library for NLP in Python. Find more [here](https://spacy.io/models/en#en_core_web_lg)."
      ]
    },
    {
      "metadata": {
        "id": "EcBv8yEG5abh",
        "colab_type": "code",
        "cellView": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "outputId": "9b639f04-52c7-4e64-d971-a48a5b122d8e"
      },
      "cell_type": "code",
      "source": [
        "!python -m spacy download en_core_web_lg\n",
        "\n",
        "# Load the model\n",
        "import spacy\n",
        "nlp = spacy.load('en_core_web_lg')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: en_core_web_lg==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.0.0/en_core_web_lg-2.0.0.tar.gz#egg=en_core_web_lg==2.0.0 in /usr/local/lib/python3.6/dist-packages (2.0.0)\n",
            "\n",
            "\u001b[93m    Linking successful\u001b[0m\n",
            "    /usr/local/lib/python3.6/dist-packages/en_core_web_lg -->\n",
            "    /usr/local/lib/python3.6/dist-packages/spacy/data/en_core_web_lg\n",
            "\n",
            "    You can now load the model via spacy.load('en_core_web_lg')\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "OMOhLm9q9ikQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Importing Python libraries"
      ]
    },
    {
      "metadata": {
        "id": "ZsdePxw49gjG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import csv\n",
        "import numpy as np\n",
        "from prettytable import PrettyTable"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_zCQspzi7f5n",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load the dataset"
      ]
    },
    {
      "metadata": {
        "id": "l1cBggXB5MYc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "cb1657fc-330e-4ebd-b6e4-da65b42559d1"
      },
      "cell_type": "code",
      "source": [
        "# Loading the Quora Question Pairs dataset\n",
        "data_file = 'quora.csv'\n",
        "new_data = []\n",
        "with open(data_file) as csvfile:\n",
        "    reader = csv.DictReader(csvfile)\n",
        "    for row in reader:\n",
        "        new_data.append(row)\n",
        "\n",
        "# Extracting the unique questions from the dataset,\n",
        "# all pairs, duplicate and non-duplicate pairs\n",
        "uniqueQ = {}\n",
        "pairs = {}\n",
        "pairs_count = 0\n",
        "duplicate = {}\n",
        "dup = 0\n",
        "non_duplicate = {}\n",
        "ndup = 0\n",
        "for dd in new_data:\n",
        "    uniqueQ[dd['qid1']] = dd['question1']\n",
        "    uniqueQ[dd['qid2']] = dd['question2']\n",
        "    pairs_count += 1\n",
        "    pairs[pairs_count] = (dd['question1'], dd['question2'])\n",
        "    if (dd['is_duplicate'] == '0'):\n",
        "      ndup += 1\n",
        "      non_duplicate[ndup] = (dd['question1'], dd['question2'])\n",
        "    else:\n",
        "      dup += 1\n",
        "      duplicate[dup] = (dd['question1'], dd['question2'])\n",
        "\n",
        "# Print stats for the dataset\n",
        "t = PrettyTable(['Pairs type', '# of Pairs'])\n",
        "t.add_row(['Unique Pairs', len(pairs)])\n",
        "t.add_row(['Duplicate Questions', len(duplicate)])\n",
        "t.add_row(['non-Duplicate Questions', len(non_duplicate)])\n",
        "print(t)\n"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-------------------------+------------+\n",
            "|        Pairs type       | # of Pairs |\n",
            "+-------------------------+------------+\n",
            "|       Unique Pairs      |   404290   |\n",
            "|   Duplicate Questions   |   149263   |\n",
            "| non-Duplicate Questions |   255027   |\n",
            "+-------------------------+------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "BZrw17JQAalx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Examples of duplicate Vs non-duplicate questions"
      ]
    },
    {
      "metadata": {
        "id": "C9s_ug71LD-i",
        "colab_type": "code",
        "outputId": "93a5fb56-dc04-4df1-ff96-5b588dac66ed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "cell_type": "code",
      "source": [
        "print('Duplicate questions:')\n",
        "print('\\t-' + duplicate[17][0])\n",
        "print('\\t-' + duplicate[17][1])\n",
        "print()\n",
        "print('non-Duplicate questions:')\n",
        "print('\\t-' + non_duplicate[3][0])\n",
        "print('\\t-' + non_duplicate[3][1])"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Duplicate questions:\n",
            "\t-Will a Blu Ray play on a regular DVD player? If so, how?\n",
            "\t-How can you play a Blu Ray DVD on a regular DVD player?\n",
            "\n",
            "non-Duplicate questions:\n",
            "\t-How can I increase the speed of my internet connection while using a VPN?\n",
            "\t-How can Internet speed be increased by hacking through DNS?\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "6KrHsieyA6eD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Implementing a cosine similarity function\n",
        "\n",
        "`cosine(vA, vB)` calculates the cosine similarity between two vectors `vA` and `vB`."
      ]
    },
    {
      "metadata": {
        "id": "ofm7NKimBM4u",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def cosine(vA, vB):\n",
        "  cos = np.dot(vA, vB) / (np.sqrt(np.dot(vA, vA)) * np.sqrt(np.dot(vB, vB)))\n",
        "  return cos.astype('float64')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9ZjqPzk5HyzF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Calculating the cosine similarity\n",
        "\n",
        "Here we are showing the differences in calculating the cosine similarity for a pair of duplicate/non-duplicate questions by taking into account the full text of the question when generating the vector representation, and when only the noun chunks extracted from a question are kept.\n",
        "\n",
        "For questions that are non-duplicate, we realize that using only the noun-chunks results in a lower similarity metric, which is preferable. We will test our hypothesis, that using only the main verb arguments of a question to calculate the question embeddings results in a more accurate representation that makes it easier to identify semantically closely related questions over non-related questions."
      ]
    },
    {
      "metadata": {
        "id": "Rkeusf9K7iPR",
        "colab_type": "code",
        "outputId": "af8caf27-6359-43f7-92a1-903f26794ff6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "(a, b) = duplicate[17]\n",
        "foo1 = nlp(a)\n",
        "foo2 = nlp(b)\n",
        "# Calculating the cosine similarity of the two questions (full text)\n",
        "cos = foo1.similarity(foo2)\n",
        "\n",
        "print('Duplicate questions:')\n",
        "print('Question1: ', foo1)\n",
        "print('Question2: ', foo2)\n",
        "print('Cosine similarity (full-text):', cos)\n",
        "\n",
        "# Calculating the cosine similarity (only noun chunks)\n",
        "calc1 = []\n",
        "for nn in foo1.noun_chunks:\n",
        "    calc1.append(nn.vector)\n",
        "calc2 = []\n",
        "for nn in foo2.noun_chunks:\n",
        "    calc2.append(nn.vector)\n",
        "# vA: question embedding for question1\n",
        "# vB: question embedding for question2\n",
        "vA = np.sum(calc1, axis=0)\n",
        "vB = np.sum(calc2, axis=0)\n",
        "cos = cosine(vA, vB)\n",
        "print('Cosine similarity (noun chunks):', cos)\n",
        "\n",
        "print()\n",
        "\n",
        "(a, b) = non_duplicate[3]\n",
        "foo1 = nlp(a)\n",
        "foo2 = nlp(b)\n",
        "# Calculating the cosine similarity of the two questions (full text)\n",
        "cos = foo1.similarity(foo2)\n",
        "\n",
        "print('non-Duplicate questions:')\n",
        "print('Question1: ', foo1)\n",
        "print('Question2: ', foo2)\n",
        "print('Cosine similarity (full-text):', cos)\n",
        "\n",
        "# Calculating the cosine similarity (only noun chunks)\n",
        "calc1 = []\n",
        "for nn in foo1.noun_chunks:\n",
        "    calc1.append(nn.vector)\n",
        "calc2 = []\n",
        "for nn in foo2.noun_chunks:\n",
        "    calc2.append(nn.vector)\n",
        "# vA: question embedding for question1\n",
        "# vB: question embedding for question2\n",
        "vA = np.sum(calc1, axis=0)\n",
        "vB = np.sum(calc2, axis=0)\n",
        "cos = cosine(vA, vB)\n",
        "print('Cosine similarity (noun chunks):', cos)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Duplicate questions:\n",
            "Question1:  Will a Blu Ray play on a regular DVD player? If so, how?\n",
            "Question2:  How can you play a Blu Ray DVD on a regular DVD player?\n",
            "Cosine similarity (full-text): 0.9717130630470544\n",
            "Cosine similarity (noun chunks): 0.914749264717102\n",
            "\n",
            "non-Duplicate questions:\n",
            "Question1:  How can I increase the speed of my internet connection while using a VPN?\n",
            "Question2:  How can Internet speed be increased by hacking through DNS?\n",
            "Cosine similarity (full-text): 0.9290680453645342\n",
            "Cosine similarity (noun chunks): 0.670642077922821\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "slWkNTuni9Ap",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Duplicate questions\n",
        "\n",
        "Here we are calculating the cosine similarity for about 5000 pairs of duplicate questions, where we:\n",
        "* Calculated the question vectors of each full text question.\n",
        "* Calculated the question vectors for the main noun chunks of each question.\n"
      ]
    },
    {
      "metadata": {
        "id": "6kfRlmFuJsQP",
        "colab_type": "code",
        "outputId": "c9944ad0-6810-43e3-f14f-c86bbf07f94b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        }
      },
      "cell_type": "code",
      "source": [
        "# sim_duplicate: stores the cosine similarity calculated on the full-text questions\n",
        "# sim_duplicate2: stores the cosine similarity calculated only on the noun chunks of the questions\n",
        "sim_duplicate = []\n",
        "sim_duplicate2 = []\n",
        "for i in range(1, 5000):\n",
        "  (a, b) = duplicate[i]\n",
        "  foo1 = nlp(a)\n",
        "  foo2 = nlp(b)\n",
        "  \n",
        "  calc1 = [np.zeros(300, dtype='float64')]\n",
        "  for nn in foo1.noun_chunks:\n",
        "      calc1.append(nn.vector)\n",
        "  calc2 = [np.zeros(300, dtype='float64')]\n",
        "  for nn in foo2.noun_chunks:\n",
        "      calc2.append(nn.vector)\n",
        "  vA = np.sum(calc1, axis=0).astype('float64')\n",
        "  vB = np.sum(calc2, axis=0).astype('float64')\n",
        "  cos = cosine(vA, vB)\n",
        "  try:\n",
        "    if not (np.isnan(cos)):\n",
        "      sim_duplicate.append(foo1.similarity(foo2))\n",
        "      sim_duplicate2.append(cos)\n",
        "  except:\n",
        "    continue\n",
        "\n",
        "# Print stats for each vector representation\n",
        "print('For pairs of questions that are duplicates, the cosine similarity stats:')\n",
        "t = PrettyTable(['', 'Full text question', 'Noun chunks'])\n",
        "t.add_row(['Min', min(sim_duplicate), min(sim_duplicate2)])\n",
        "t.add_row(['Max', max(sim_duplicate), max(sim_duplicate2)])\n",
        "t.add_row(['Avg', np.average(sim_duplicate), np.average(sim_duplicate2)])\n",
        "print(t)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in double_scalars\n",
            "  \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "For pairs of questions that are duplicates, the cosine similarity stats:\n",
            "+-----+--------------------+-----------------------+\n",
            "|     | Full text question |      Noun chunks      |\n",
            "+-----+--------------------+-----------------------+\n",
            "| Min | 0.6468996546040794 | -0.008805530308907383 |\n",
            "| Max | 1.0000000858950187 |   1.0000000000000004  |\n",
            "| Avg | 0.9465413936763243 |   0.8805787954723814  |\n",
            "+-----+--------------------+-----------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "LVQMshSokLn3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## non-Duplicate questions\n",
        "\n",
        "Here we are calculating the cosine similarity for about 5000 pairs of non-duplicate questions, where we:\n",
        "* Calculated the question vectors of each full text question.\n",
        "* Calculated the question vectors for the main noun chunks of each question.\n"
      ]
    },
    {
      "metadata": {
        "id": "Qo3lJFaRMSQt",
        "colab_type": "code",
        "outputId": "663d3ba0-5f7c-4a49-c71a-333ff33975ed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        }
      },
      "cell_type": "code",
      "source": [
        "# sim_nonduplicate: stores the cosine similarity calculated on the full-text questions\n",
        "# sim_nonduplicate2: stores the cosine similarity calculated only on the noun chunks of the questions\n",
        "sim_nonduplicate = []\n",
        "sim_nonduplicate2 = []\n",
        "for i in range(1, 5000):\n",
        "  (a, b) = non_duplicate[i]\n",
        "  foo1 = nlp(a)\n",
        "  foo2 = nlp(b)\n",
        "  \n",
        "  calc1 = [np.zeros(300, dtype='float64')]\n",
        "  for nn in foo1.noun_chunks:\n",
        "      calc1.append(nn.vector)\n",
        "  calc2 = [np.zeros(300, dtype='float64')]\n",
        "  for nn in foo2.noun_chunks:\n",
        "      calc2.append(nn.vector)\n",
        "  vA = np.sum(calc1, axis=0).astype('float64')\n",
        "  vB = np.sum(calc2, axis=0).astype('float64')\n",
        "  cos = cosine(vA, vB)\n",
        "  try:\n",
        "    if not (np.isnan(cos)):\n",
        "      sim_nonduplicate.append(foo1.similarity(foo2))\n",
        "      sim_nonduplicate2.append(cos)\n",
        "  except:\n",
        "    continue\n",
        "\n",
        "# Print stats for each vector representation\n",
        "print('For pairs of questions that are non duplicates, the cosine similarity stats:')\n",
        "t = PrettyTable(['', 'Full text question', 'Noun chunks'])\n",
        "t.add_row(['Min', min(sim_nonduplicate), min(sim_nonduplicate2)])\n",
        "t.add_row(['Max', max(sim_nonduplicate), max(sim_nonduplicate2)])\n",
        "t.add_row(['Avg', np.average(sim_nonduplicate), np.average(sim_nonduplicate2)])\n",
        "print(t)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in double_scalars\n",
            "  \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "For pairs of questions that are non duplicates, the cosine similarity stats:\n",
            "+-----+---------------------+---------------------+\n",
            "|     |  Full text question |     Noun chunks     |\n",
            "+-----+---------------------+---------------------+\n",
            "| Min | 0.29264479268934257 | -0.1132305785162487 |\n",
            "| Max |  1.0000001169458588 |  1.0000000000000002 |\n",
            "| Avg |  0.8947737652759219 |  0.7746351169036362 |\n",
            "+-----+---------------------+---------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "qg29JDVokhuN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Save the data\n",
        "\n",
        "Finally, we are creating two files:\n",
        "* `full_question.csv`\n",
        "* `noun_chunks.csv`\n",
        "\n",
        "Each file has a subset of roughly 10,000 pairs of questions, with 5,000 duplicate and 5,000 non-duplicate questions, so we can then perform classification and decide which is a better model to use to generate question embeddings for the purpose of finding semantically related questions."
      ]
    },
    {
      "metadata": {
        "id": "LiZ7jBGBcGe_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# toCSV1 stores the cosine similarity calculated using the vector representation of the full text of the question\n",
        "# toCSV2 stores the cosine similarity calculated using the vector representation of only the noun chunks of the question\n",
        "toCSV1 = []\n",
        "toCSV2 = []\n",
        "for i in sim_duplicate:\n",
        "  toCSV1.append({'Class':'Duplicate', 'Similarity_Metric':i})\n",
        "for i in sim_duplicate2:\n",
        "  toCSV2.append({'Class':'Duplicate', 'Similarity_Metric':i})\n",
        "for i in sim_nonduplicate:\n",
        "  toCSV1.append({'Class':'NonDuplicate', 'Similarity_Metric':i})\n",
        "for i in sim_nonduplicate2:\n",
        "  toCSV2.append({'Class':'NonDuplicate', 'Similarity_Metric':i})\n",
        "  \n",
        "with open('full_questions.csv', 'w') as csvfile:\n",
        "  fieldnames = ['Class', 'Similarity_Metric']\n",
        "  writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n",
        "  writer.writeheader()\n",
        "  for i in range(len(toCSV1)):\n",
        "    writer.writerow(toCSV1[i])\n",
        "\n",
        "with open('noun_chunks.csv', 'w') as csvfile:\n",
        "  fieldnames = ['Class', 'Similarity_Metric']\n",
        "  writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n",
        "  writer.writeheader()\n",
        "  for i in range(len(toCSV2)):\n",
        "    writer.writerow(toCSV2[i])"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}