TrainingOnQuora.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TrainingOnQuora.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kzafeiroudi/QuestRecommend/blob/master/TrainingOnQuora.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "lDeRMRlemnLi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Deciding the right vector representation\n",
        "\n",
        "In this section, we are going to train two simple tree classifiers, in order to identify the similarity metric that best divides question pairs that are duplicates from those that are not."
      ]
    },
    {
      "metadata": {
        "id": "A8DY8VAnnUpA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Upload the two datasets\n",
        "\n",
        "Upload the files `full_questions.csv` and `noun_chunks.csv` that will be used in this Python 3 notebook."
      ]
    },
    {
      "metadata": {
        "id": "H4TRxY7S2l_R",
        "colab_type": "code",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 125
        },
        "outputId": "8c4adf3f-edfa-437f-de9b-7aa9fde1a38f"
      },
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "\n",
        "# Choose from your own machine the files to upload - names should be \n",
        "# \"full_questions.csv\" and \"noun_chunks.csv\"\n",
        "uploaded = files.upload()\n",
        "uploaded = files.upload()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-1f5ac390-35c6-48b9-8863-0f2ad8c0de57\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-1f5ac390-35c6-48b9-8863-0f2ad8c0de57\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving full_questions.csv to full_questions (1).csv\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-fc9b5257-d5a7-4f2e-853f-e142cbf49c6d\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-fc9b5257-d5a7-4f2e-853f-e142cbf49c6d\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving noun_chunks.csv to noun_chunks.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "3BOYuEI0n65X",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Download all important packages\n",
        "\n",
        "We will be using the **Weka** machine learning software through a Python wrapper. The following commands are necessary to install the packages before we use the software."
      ]
    },
    {
      "metadata": {
        "id": "5W7AICQG21Z9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# pygraphviz\n",
        "!wget https://anaconda.org/anaconda/pygraphviz/1.3/download/linux-64/pygraphviz-1.3-py36h14c3975_1.tar.bz2\n",
        "!tar xvjf pygraphviz-1.3-py36h14c3975_1.tar.bz2\n",
        "!cp -r lib/python3.6/site-packages/* /usr/local/lib/python3.6/dist-packages/\n",
        "\n",
        "import pygraphviz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "nBFAf3DwpYDE",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "# Install deps from \n",
        "# https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md#-linux\n",
        "apt-get update\n",
        "apt-get install build-essential zlib1g-dev libsdl2-dev libjpeg-dev \\\n",
        "nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \\\n",
        "libopenal-dev timidity libwildmidi-dev unzip\n",
        "\n",
        "# Boost libraries\n",
        "apt-get install libboost-all-dev\n",
        "\n",
        "# Lua binding dependencies\n",
        "apt-get install liblua5.1-dev"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jDeoTh4M29E0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "532a52a7-d7a0-4591-9473-09e6b71d5788"
      },
      "cell_type": "code",
      "source": [
        "import os       #importing os to set environment variable\n",
        "def install_java():\n",
        "  !apt-get install -y openjdk-8-jdk-headless -qq > /dev/null      #install openjdk\n",
        "  os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"     #set environment variable\n",
        "  !java -version       #check java version\n",
        "install_java()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "openjdk version \"11.0.2\" 2019-01-15\n",
            "OpenJDK Runtime Environment (build 11.0.2+9-Ubuntu-3ubuntu118.04.3)\n",
            "OpenJDK 64-Bit Server VM (build 11.0.2+9-Ubuntu-3ubuntu118.04.3, mixed mode, sharing)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "n8ft58hg2-Ee",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!apt-get install libproj-dev proj-data proj-bin\n",
        "!apt-get install libgeos-dev\n",
        "!pip install cython\n",
        "!pip install python-weka-wrapper3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_PJcZNFDqDuS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Importing Python libraries"
      ]
    },
    {
      "metadata": {
        "id": "mrcWHtCRqFl-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "8c05f424-635d-40ca-e50e-b49c0e8d5235"
      },
      "cell_type": "code",
      "source": [
        "import weka.core.jvm as jvm\n",
        "jvm.start()\n",
        "\n",
        "from weka.core.converters import Loader\n",
        "from weka.core.classes import Random\n",
        "from weka.classifiers import Classifier, Evaluation"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "DEBUG:weka.core.jvm:Adding bundled jars\n",
            "DEBUG:weka.core.jvm:Classpath=['/usr/local/lib/python3.6/dist-packages/javabridge/jars/rhino-1.7R4.jar', '/usr/local/lib/python3.6/dist-packages/javabridge/jars/runnablequeue.jar', '/usr/local/lib/python3.6/dist-packages/javabridge/jars/cpython.jar', '/usr/local/lib/python3.6/dist-packages/weka/lib/weka.jar', '/usr/local/lib/python3.6/dist-packages/weka/lib/python-weka-wrapper.jar']\n",
            "DEBUG:weka.core.jvm:MaxHeapSize=default\n",
            "DEBUG:weka.core.jvm:Package support disabled\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "BHtKWNhjrc_r",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load the dataset `full_questions.csv`\n",
        "\n",
        "We will first train a tree classifier (J48) on the similarity metrics resulting after creating the question vector representations on the full text of the question."
      ]
    },
    {
      "metadata": {
        "id": "oQRaxwln3s4u",
        "colab_type": "code",
        "outputId": "85922283-ee12-454c-b335-eb3d10d7dae1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "loader = Loader(classname=\"weka.core.converters.CSVLoader\")\n",
        "data_file = 'full_questions.csv'\n",
        "data = loader.load_file(data_file)\n",
        "\n",
        "print('Sample size: ', data.num_instances)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sample size:  9888\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "RRnDv90QtDFI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Train the classifier\n",
        "\n",
        "We choose to classify on the nominal atrribute **Class**. We first split our dataset to train and test, with a 66% to the train split.\n",
        "\n",
        "For the J48 classifier, which generated a pruned C4.5 decision tree, we have chosen a confidence factor used for pruning of 0.25.\n",
        "\n",
        "The resulting decision tree can be seen in the standard output."
      ]
    },
    {
      "metadata": {
        "id": "8pR3isOn9GmD",
        "colab_type": "code",
        "outputId": "bec75868-f564-4055-d304-16617f60d3ba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "print('Classifying on: ', data.attribute(0))\n",
        "data.class_index = 0\n",
        "train, test = data.train_test_split(66.0, Random(1))\n",
        "\n",
        "\n",
        "cls = Classifier(classname=\"weka.classifiers.trees.J48\", options=[\"-C\", \"0.25\"])\n",
        "cls.build_classifier(train)\n",
        "\n",
        "print(cls)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Classifying on:  @attribute Class {Duplicate,NonDuplicate}\n",
            "J48 pruned tree\n",
            "------------------\n",
            "\n",
            "Similarity_Metric <= 0.927621: NonDuplicate (2799.0/814.0)\n",
            "Similarity_Metric > 0.927621: Duplicate (3727.0/1267.0)\n",
            "\n",
            "Number of Leaves  : \t2\n",
            "\n",
            "Size of the tree : \t3\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "fCTYZgGduRHw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Evaluating the classifier\n",
        "\n",
        "We evaluate the trained model against the test split."
      ]
    },
    {
      "metadata": {
        "id": "_olyJHR518oE",
        "colab_type": "code",
        "outputId": "5edf4876-8003-4ac6-88fd-8483b1140aed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "cell_type": "code",
      "source": [
        "evl = Evaluation(train)\n",
        "evl.test_model(cls, test)\n",
        "print(evl.summary())"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Correctly Classified Instances        2229               66.2998 %\n",
            "Incorrectly Classified Instances      1133               33.7002 %\n",
            "Kappa statistic                          0.3264\n",
            "Mean absolute error                      0.4397\n",
            "Root mean squared error                  0.4718\n",
            "Relative absolute error                 87.9477 %\n",
            "Root relative squared error             94.3515 %\n",
            "Total Number of Instances             3362     \n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "jYmInoEuvy0A",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Actual predictions on the test split\n",
        "\n",
        "We are printing the first 10 predictions on the test split."
      ]
    },
    {
      "metadata": {
        "id": "6zPATiVB0Kaw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "026cc680-df7e-4583-a9af-19fb75791924"
      },
      "cell_type": "code",
      "source": [
        "print(\"# - actual - predicted - error - distribution\")\n",
        "for index, inst in enumerate(itertools.islice(test, 10)):\n",
        "  pred = cls.classify_instance(inst)\n",
        "  dist = cls.distribution_for_instance(inst)\n",
        "  print(\n",
        "            \"%d - %s - %s - %s  - %s\" %\n",
        "            (index+1,\n",
        "             inst.get_string_value(inst.class_index),\n",
        "             inst.class_attribute.value(int(pred)),\n",
        "             \"yes\" if pred != inst.get_value(inst.class_index) else \"no\",\n",
        "             str(dist.tolist())))"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "# - actual - predicted - error - distribution\n",
            "1 - NonDuplicate - NonDuplicate - no  - [0.29081814933904965, 0.7091818506609503]\n",
            "2 - Duplicate - Duplicate - no  - [0.6600482962167964, 0.33995170378320366]\n",
            "3 - NonDuplicate - NonDuplicate - no  - [0.29081814933904965, 0.7091818506609503]\n",
            "4 - Duplicate - NonDuplicate - yes  - [0.29081814933904965, 0.7091818506609503]\n",
            "5 - NonDuplicate - NonDuplicate - no  - [0.29081814933904965, 0.7091818506609503]\n",
            "6 - NonDuplicate - NonDuplicate - no  - [0.29081814933904965, 0.7091818506609503]\n",
            "7 - Duplicate - NonDuplicate - yes  - [0.29081814933904965, 0.7091818506609503]\n",
            "8 - NonDuplicate - Duplicate - yes  - [0.6600482962167964, 0.33995170378320366]\n",
            "9 - Duplicate - Duplicate - no  - [0.6600482962167964, 0.33995170378320366]\n",
            "10 - NonDuplicate - NonDuplicate - no  - [0.29081814933904965, 0.7091818506609503]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "bJbefG03wJST",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load the dataset `noun_chunks.csv`\n",
        "\n",
        "We will now train a tree classifier (J48) on the similarity metrics resulting after creating the question vector representations on the main verb arguments extracted from the question."
      ]
    },
    {
      "metadata": {
        "id": "c_39HRMqBv0R",
        "colab_type": "code",
        "outputId": "308f6a37-7c13-46b7-96f5-7441017345b3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "data_file = 'noun_chunks.csv'\n",
        "data = loader.load_file(data_file)\n",
        "\n",
        "print('Sample size: ', data.num_instances)"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sample size:  9888\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "SH1AN6kUwjA5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Train the classifier\n",
        "\n",
        "We choose to classify on the nominal atrribute **Class**. We first split our dataset to train and test, with a 66% to the train split.\n",
        "\n",
        "For the J48 classifier, which generated a pruned C4.5 decision tree, we have chosen a confidence factor used for pruning of 0.25.\n",
        "\n",
        "The resulting decision tree can be seen in the standard output."
      ]
    },
    {
      "metadata": {
        "id": "Ah9WIhNR2mLv",
        "colab_type": "code",
        "outputId": "bada2b59-7d2c-4e6e-eabc-689532a65dac",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "print('Classifying on: ', data.attribute(0))\n",
        "data.class_index = 0\n",
        "train, test = data.train_test_split(66.0, Random(1))\n",
        "\n",
        "cls = Classifier(classname=\"weka.classifiers.trees.J48\", options=[\"-C\", \"0.25\"])\n",
        "cls.build_classifier(train)\n",
        "\n",
        "print(cls)"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Classifying on:  @attribute Class {Duplicate,NonDuplicate}\n",
            "J48 pruned tree\n",
            "------------------\n",
            "\n",
            "Similarity_Metric <= 0.883042: NonDuplicate (3483.0/1259.0)\n",
            "Similarity_Metric > 0.883042: Duplicate (3043.0/1028.0)\n",
            "\n",
            "Number of Leaves  : \t2\n",
            "\n",
            "Size of the tree : \t3\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "eNghK07_wvyD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Evaluating the classifier\n",
        "\n",
        "We evaluate the trained model against the test split."
      ]
    },
    {
      "metadata": {
        "id": "Y76NfwxI2qz7",
        "colab_type": "code",
        "outputId": "41eafb67-d9d6-49c8-b376-0c3ca85c0862",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "cell_type": "code",
      "source": [
        "evl = Evaluation(train)\n",
        "evl.test_model(cls, test)\n",
        "print(evl.summary())"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Correctly Classified Instances        2125               63.2064 %\n",
            "Incorrectly Classified Instances      1237               36.7936 %\n",
            "Kappa statistic                          0.264 \n",
            "Mean absolute error                      0.4603\n",
            "Root mean squared error                  0.4825\n",
            "Relative absolute error                 92.0636 %\n",
            "Root relative squared error             96.5002 %\n",
            "Total Number of Instances             3362     \n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "82RJ64Fyw1ow",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Actual predictions on the test split\n",
        "\n",
        "We are printing the first 10 predictions on the test split."
      ]
    },
    {
      "metadata": {
        "id": "YC_aLyOBw4ex",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "e3aea562-30b9-4396-dc7a-c6a39d09b384"
      },
      "cell_type": "code",
      "source": [
        "print(\"# - actual - predicted - error - distribution\")\n",
        "for index, inst in enumerate(itertools.islice(test, 10)):\n",
        "  pred = cls.classify_instance(inst)\n",
        "  dist = cls.distribution_for_instance(inst)\n",
        "  print(\n",
        "            \"%d - %s - %s - %s  - %s\" %\n",
        "            (index+1,\n",
        "             inst.get_string_value(inst.class_index),\n",
        "             inst.class_attribute.value(int(pred)),\n",
        "             \"yes\" if pred != inst.get_value(inst.class_index) else \"no\",\n",
        "             str(dist.tolist())))"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "# - actual - predicted - error - distribution\n",
            "1 - NonDuplicate - NonDuplicate - no  - [0.36146999712891187, 0.6385300028710882]\n",
            "2 - Duplicate - NonDuplicate - yes  - [0.36146999712891187, 0.6385300028710882]\n",
            "3 - NonDuplicate - NonDuplicate - no  - [0.36146999712891187, 0.6385300028710882]\n",
            "4 - Duplicate - NonDuplicate - yes  - [0.36146999712891187, 0.6385300028710882]\n",
            "5 - NonDuplicate - NonDuplicate - no  - [0.36146999712891187, 0.6385300028710882]\n",
            "6 - NonDuplicate - NonDuplicate - no  - [0.36146999712891187, 0.6385300028710882]\n",
            "7 - Duplicate - NonDuplicate - yes  - [0.36146999712891187, 0.6385300028710882]\n",
            "8 - NonDuplicate - Duplicate - yes  - [0.6621754847190273, 0.33782451528097274]\n",
            "9 - Duplicate - Duplicate - no  - [0.6621754847190273, 0.33782451528097274]\n",
            "10 - NonDuplicate - Duplicate - yes  - [0.6621754847190273, 0.33782451528097274]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "TPKpowT6xaqA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Discussion\n",
        "\n",
        "Both classifiers show an accuracy of 63-66%, which is not very high. It was expected that since both duplicate questions and non-duplicate questions show high correlation in the vocabulary used, it would be hard to separate them. The main purpose of traning a decision tree model was to find the value of the similarity metric that best separates the two classes, duplicates from non-duplicates.\n",
        "\n",
        "When for our question vector representation we used the full body of the text, we saw that the best separation arises for a cosine similarity of **0.93**, while when we only use the main verb arguments of the question to create our vector representation, we find that the cosine similarity best separating duplicate from non-duplicate questions is **0.88**. \n",
        "\n",
        "The main purpose of this exercise was to show that we can indeed achieve a decrease in the cosine similarity between questions that are not marked as duplicates."
      ]
    }
  ]
}