DocProductPresentationV6_0_2_0.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DocProductPresentationV6-0.2.0.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jonbaer/googlecolab/blob/master/DocProductPresentationV6_0_2_0.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qaw6CoC4qUb5",
        "colab_type": "text"
      },
      "source": [
        "![DoctorBert](https://snag.gy/oF84Gk.jpg)\n",
        "\n",
        "\n",
        "![DoctorBert](https://i.ytimg.com/vi/nPemP-Q0Xn8/hqdefault.jpg)\n",
        "\n",
        "This is a Colab Demo of our DocProduct Tensorflow 2.0 Hackathon Project\n",
        "\n",
        "Project details can be seen on our Github repo\n",
        "https://github.com/Santosh-Gupta/DocProduct\n",
        "\n",
        "and our Devpost page\n",
        "https://devpost.com/software/nlp-doctor\n",
        "\n",
        "In order to fit the computational limits of Google Colab, we had to reduce the amount of searchable Q&A datapoints from ~700,000 to ~200,000.\n",
        "We also converted our search embeddings from float32 to float16, although this does not seem to dimish the quality of search (*but that's a whole other product to explore*)\n",
        "\n",
        "Instructions:\n",
        "\n",
        "-Run the first cell to install TF2.0 and FAISS\n",
        "\n",
        "-Run the second cell to download the models and data\n",
        "\n",
        "-Run the third cell to load the weights and data\n",
        "\n",
        "-Input your question and search parameters into the fourth cell to play around with our model results\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jmMqTnR0kIBL",
        "colab_type": "code",
        "outputId": "acf7acbe-024a-4b2f-b546-2e5193060bb6",
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "#@title Install Faiss, TF 2.0, and our Github. Double Click to see code\n",
        "\n",
        "#To use CPU FAISS use\n",
        "!wget  https://anaconda.org/pytorch/faiss-cpu/1.2.1/download/linux-64/faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2\n",
        "#To use GPU FAISS use\n",
        "# !wget  https://anaconda.org/pytorch/faiss-gpu/1.2.1/download/linux-64/faiss-gpu-1.2.1-py36_cuda9.0.176_1.tar.bz2\n",
        "!tar xvjf faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2\n",
        "!cp -r lib/python3.6/site-packages/* /usr/local/lib/python3.6/dist-packages/\n",
        "!pip install mkl\n",
        "\n",
        "!pip install tensorflow-gpu==2.0.0-alpha0\n",
        "import tensorflow as tf\n",
        "!pip install https://github.com/re-search/DocProduct/archive/v0.2.0_dev.zip\n",
        "!pip install gpt2-estimator\n",
        "!pip install pyarrow\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-06-26 15:03:11--  https://anaconda.org/pytorch/faiss-cpu/1.2.1/download/linux-64/faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2\n",
            "Resolving anaconda.org (anaconda.org)... 104.17.92.24, 104.17.93.24, 2606:4700::6811:5d18, ...\n",
            "Connecting to anaconda.org (anaconda.org)|104.17.92.24|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 FOUND\n",
            "Location: https://binstar-cio-packages-prod.s3.amazonaws.com/5a15c9c5c376961204909d87/5aa7f0a65571b411e5c259be?response-content-disposition=attachment%3B%20filename%3D%22faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20190626T150311Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=AgoJb3JpZ2luX2VjEOL%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIHQWkI%2Fx0jfbv864Ozw3M3MuyKqXwM4U1hHqsDu1RC%2B4AiEAst6lxkdoKZbDP1hxuy6Ogw87Cvg%2FDkXasIB4wraX45oq2gMIKxAAGgw0NTU4NjQwOTgzNzgiDNtI61Y91cDZK04Dbiq3AwPlDta%2BdLOdmAncfYfMMD25X43yjG1WVPM%2FTGHv3TzyjeqH1x8s653i3LIVAGBocSLP%2FGSwKiO%2BzwWkD7Nq51saM0LPPbINpXOImUzkUM1nc%2BLS1YmE6vdM5nRJ7emNHd81zoZggRRm%2Fc2UI1KmWu%2BQOzg4XEZax9XNxiayf6klT67I6%2Bh4eNzsbye2x9uzrZbhhzf1lwph2qVp4QWpcP%2Bzw3QR3o35B4jgPFGXPO29R3wSl5%2BMDTrM93VSO38v%2FKwhoJPXCwA5p2EDNHCO1miaTPYbNXTHO0nHm9EiLB4MaNL%2F1fp6hbyOOgjEafvim8sGrNQE7QC0%2FpMz0buhX36z2K6CHyIU%2Fh0cOOfl6Lj5q5rDvXe5EnqVwOP2unny%2BWgptUtqcyT0Xi4AdR3PFG2anYzXRwpGY7PAoo1ySkjoreQVi%2BVG0HiaIoGiXmFJ9JAzZYe3ee3TeI0vBwDns6P18e0RPmTIyPHIuiSMuIsEQQlTLRoOhu35Atwms5VwypdJta09OB0KURHMMHWW9gyCA8nkpgvExnXGIbB2wnXDKFKdMXUxVDWX8fB0U6GVh9e%2FrLpaAm8w3YDN6AU6tAHHlo5LeoqqDQ2%2BtebPDD20slSTk0Hx4pU9YzIfdhrCa9bYpfx2DMS9uOXwgurNk27hu5D25n8pbMYf3tukrb1mRmhSVqB%2F2STyFMWCXKURh9onS5q%2BVc%2FoL2ebKFpZcnph5mWwfcUNDD1wm8MgfX%2F%2B9AT6rL74EO5uvy%2Ft1lWNyoynRZcjmkRaWOEEJ%2BQz9kzL2aDzfK15nOz7eXrt51ZP8PeSMBrwx2HIYTLIogsWv4netsk%3D&X-Amz-Credential=ASIAWUI46DZFFFQNR45N%2F20190626%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=df00a5fce3556129791beb2ebd29d613e08a1702774776741a6201436f56e3af [following]\n",
            "--2019-06-26 15:03:11--  https://binstar-cio-packages-prod.s3.amazonaws.com/5a15c9c5c376961204909d87/5aa7f0a65571b411e5c259be?response-content-disposition=attachment%3B%20filename%3D%22faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20190626T150311Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=AgoJb3JpZ2luX2VjEOL%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIHQWkI%2Fx0jfbv864Ozw3M3MuyKqXwM4U1hHqsDu1RC%2B4AiEAst6lxkdoKZbDP1hxuy6Ogw87Cvg%2FDkXasIB4wraX45oq2gMIKxAAGgw0NTU4NjQwOTgzNzgiDNtI61Y91cDZK04Dbiq3AwPlDta%2BdLOdmAncfYfMMD25X43yjG1WVPM%2FTGHv3TzyjeqH1x8s653i3LIVAGBocSLP%2FGSwKiO%2BzwWkD7Nq51saM0LPPbINpXOImUzkUM1nc%2BLS1YmE6vdM5nRJ7emNHd81zoZggRRm%2Fc2UI1KmWu%2BQOzg4XEZax9XNxiayf6klT67I6%2Bh4eNzsbye2x9uzrZbhhzf1lwph2qVp4QWpcP%2Bzw3QR3o35B4jgPFGXPO29R3wSl5%2BMDTrM93VSO38v%2FKwhoJPXCwA5p2EDNHCO1miaTPYbNXTHO0nHm9EiLB4MaNL%2F1fp6hbyOOgjEafvim8sGrNQE7QC0%2FpMz0buhX36z2K6CHyIU%2Fh0cOOfl6Lj5q5rDvXe5EnqVwOP2unny%2BWgptUtqcyT0Xi4AdR3PFG2anYzXRwpGY7PAoo1ySkjoreQVi%2BVG0HiaIoGiXmFJ9JAzZYe3ee3TeI0vBwDns6P18e0RPmTIyPHIuiSMuIsEQQlTLRoOhu35Atwms5VwypdJta09OB0KURHMMHWW9gyCA8nkpgvExnXGIbB2wnXDKFKdMXUxVDWX8fB0U6GVh9e%2FrLpaAm8w3YDN6AU6tAHHlo5LeoqqDQ2%2BtebPDD20slSTk0Hx4pU9YzIfdhrCa9bYpfx2DMS9uOXwgurNk27hu5D25n8pbMYf3tukrb1mRmhSVqB%2F2STyFMWCXKURh9onS5q%2BVc%2FoL2ebKFpZcnph5mWwfcUNDD1wm8MgfX%2F%2B9AT6rL74EO5uvy%2Ft1lWNyoynRZcjmkRaWOEEJ%2BQz9kzL2aDzfK15nOz7eXrt51ZP8PeSMBrwx2HIYTLIogsWv4netsk%3D&X-Amz-Credential=ASIAWUI46DZFFFQNR45N%2F20190626%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=df00a5fce3556129791beb2ebd29d613e08a1702774776741a6201436f56e3af\n",
            "Resolving binstar-cio-packages-prod.s3.amazonaws.com (binstar-cio-packages-prod.s3.amazonaws.com)... 52.216.171.99\n",
            "Connecting to binstar-cio-packages-prod.s3.amazonaws.com (binstar-cio-packages-prod.s3.amazonaws.com)|52.216.171.99|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 4106816 (3.9M) [application/x-tar]\n",
            "Saving to: ‘faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2’\n",
            "\n",
            "faiss-cpu-1.2.1-py3 100%[===================>]   3.92M  5.98MB/s    in 0.7s    \n",
            "\n",
            "2019-06-26 15:03:13 (5.98 MB/s) - ‘faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2’ saved [4106816/4106816]\n",
            "\n",
            "info/hash_input.json\n",
            "info/has_prefix\n",
            "info/index.json\n",
            "info/git\n",
            "info/files\n",
            "info/LICENSE.txt\n",
            "info/about.json\n",
            "info/paths.json\n",
            "lib/python3.6/site-packages/faiss-0.1-py3.6.egg-info/dependency_links.txt\n",
            "lib/python3.6/site-packages/faiss-0.1-py3.6.egg-info/not-zip-safe\n",
            "lib/python3.6/site-packages/faiss-0.1-py3.6.egg-info/requires.txt\n",
            "lib/python3.6/site-packages/faiss-0.1-py3.6.egg-info/top_level.txt\n",
            "lib/python3.6/site-packages/faiss-0.1-py3.6.egg-info/native_libs.txt\n",
            "info/test/run_test.py\n",
            "info/test/run_test.sh\n",
            "info/test/tests/run_tests.sh\n",
            "lib/python3.6/site-packages/faiss-0.1-py3.6.egg-info/SOURCES.txt\n",
            "info/recipe/conda_build_config.yaml\n",
            "info/recipe/build.sh\n",
            "info/test/tests/CMakeLists.txt\n",
            "info/test/tests/Makefile\n",
            "info/recipe/meta.yaml.template\n",
            "lib/python3.6/site-packages/faiss-0.1-py3.6.egg-info/PKG-INFO\n",
            "info/test/tests/test_factory.py\n",
            "info/test/tests/test_ivfpq_codec.cpp\n",
            "info/recipe/meta.yaml\n",
            "info/recipe/setup.py\n",
            "info/test/tests/test_blas.cpp\n",
            "info/recipe/makefile.inc\n",
            "info/test/tests/test_ivfpq_indexing.cpp\n",
            "info/test/tests/test_ondisk_ivf.cpp\n",
            "info/test/tests/test_build_blocks.py\n",
            "info/test/tests/test_merge.cpp\n",
            "info/test/tests/test_pairs_decoding.cpp\n",
            "info/test/tests/test_index_composite.py\n",
            "lib/python3.6/site-packages/faiss/__init__.py\n",
            "lib/python3.6/site-packages/faiss/__pycache__/__init__.cpython-36.pyc\n",
            "info/test/tests/test_index.py\n",
            "info/test/tests/test_blas\n",
            "lib/python3.6/site-packages/faiss/__pycache__/swigfaiss.cpython-36.pyc\n",
            "lib/python3.6/site-packages/faiss/swigfaiss.py\n",
            "lib/python3.6/site-packages/faiss/_swigfaiss.so\n",
            "Requirement already satisfied: mkl in /usr/local/lib/python3.6/dist-packages (2019.0)\n",
            "Requirement already satisfied: intel-openmp in /usr/local/lib/python3.6/dist-packages (from mkl) (2019.0)\n",
            "Collecting tensorflow-gpu==2.0.0-alpha0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1a/66/32cffad095253219d53f6b6c2a436637bbe45ac4e7be0244557210dc3918/tensorflow_gpu-2.0.0a0-cp36-cp36m-manylinux1_x86_64.whl (332.1MB)\n",
            "\u001b[K     |████████████████████████████████| 332.1MB 45kB/s \n",
            "\u001b[?25hRequirement already satisfied: google-pasta>=0.1.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (0.1.7)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (1.16.4)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (1.12.0)\n",
            "Collecting tf-estimator-nightly<1.14.0.dev2019030116,>=1.14.0.dev2019030115 (from tensorflow-gpu==2.0.0-alpha0)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/13/82/f16063b4eed210dc2ab057930ac1da4fbe1e91b7b051a6c8370b401e6ae7/tf_estimator_nightly-1.14.0.dev2019030115-py2.py3-none-any.whl (411kB)\n",
            "\u001b[K     |████████████████████████████████| 419kB 38.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (3.7.1)\n",
            "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (0.2.2)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (0.7.1)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (1.1.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (1.0.8)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (1.15.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (0.33.4)\n",
            "Collecting tb-nightly<1.14.0a20190302,>=1.14.0a20190301 (from tensorflow-gpu==2.0.0-alpha0)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a9/51/aa1d756644bf4624c03844115e4ac4058eff77acd786b26315f051a4b195/tb_nightly-1.14.0a20190301-py3-none-any.whl (3.0MB)\n",
            "\u001b[K     |████████████████████████████████| 3.0MB 33.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (1.1.0)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-alpha0) (0.8.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow-gpu==2.0.0-alpha0) (41.0.1)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tensorflow-gpu==2.0.0-alpha0) (2.8.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190302,>=1.14.0a20190301->tensorflow-gpu==2.0.0-alpha0) (3.1.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190302,>=1.14.0a20190301->tensorflow-gpu==2.0.0-alpha0) (0.15.4)\n",
            "Installing collected packages: tf-estimator-nightly, tb-nightly, tensorflow-gpu\n",
            "Successfully installed tb-nightly-1.14.0a20190301 tensorflow-gpu-2.0.0a0 tf-estimator-nightly-1.14.0.dev2019030115\n",
            "Collecting https://github.com/re-search/DocProduct/archive/v0.2.0_dev.zip\n",
            "\u001b[?25l  Downloading https://github.com/re-search/DocProduct/archive/v0.2.0_dev.zip\n",
            "\u001b[K     | 15.0MB 2.2MB/s\n",
            "\u001b[?25hCollecting pycurl (from docproduct==0.2.0)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ac/b3/0f3979633b7890bab6098d84c84467030b807a1e2b31f5d30103af5a71ca/pycurl-7.43.0.3.tar.gz (215kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 6.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyarrow in /usr/local/lib/python3.6/dist-packages (from docproduct==0.2.0) (0.13.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from docproduct==0.2.0) (1.16.4)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from docproduct==0.2.0) (3.0.3)\n",
            "Collecting tensorflow==2.0.0-alpha0 (from docproduct==0.2.0)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/29/39/f99185d39131b8333afcfe1dcdb0629c2ffc4ecfb0e4c14ca210d620e56c/tensorflow-2.0.0a0-cp36-cp36m-manylinux1_x86_64.whl (79.9MB)\n",
            "\u001b[K     |████████████████████████████████| 79.9MB 33.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: tensorflow-gpu==2.0.0-alpha0 in /usr/local/lib/python3.6/dist-packages (from docproduct==0.2.0) (2.0.0a0)\n",
            "Requirement already satisfied: Keras in /usr/local/lib/python3.6/dist-packages (from docproduct==0.2.0) (2.2.4)\n",
            "Collecting keras-pos-embd==0.9.0 (from docproduct==0.2.0)\n",
            "  Downloading https://files.pythonhosted.org/packages/56/5e/7b1e933104a25f2039b6788e392a650671e3bcbee6404ea29dcb92295614/keras-pos-embd-0.9.0.tar.gz\n",
            "Collecting keras-transformer==0.21.0 (from docproduct==0.2.0)\n",
            "  Downloading https://files.pythonhosted.org/packages/d3/3a/ad25f5c71adc6b8aa73f71b1367be873b4103125a614ba57c006d1a9b1ff/keras-transformer-0.21.0.tar.gz\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from docproduct==0.2.0) (4.28.1)\n",
            "Requirement already satisfied: faiss in /usr/local/lib/python3.6/dist-packages (from docproduct==0.2.0) (0.1)\n",
            "Requirement already satisfied: sklearn in /usr/local/lib/python3.6/dist-packages (from docproduct==0.2.0) (0.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from docproduct==0.2.0) (1.12.0)\n",
            "Collecting argparse (from docproduct==0.2.0)\n",
            "  Downloading https://files.pythonhosted.org/packages/f2/94/3af39d34be01a24a6e65433d19e107099374224905f1e0cc6bbe1fd22a2f/argparse-1.4.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->docproduct==0.2.0) (1.1.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->docproduct==0.2.0) (2.5.3)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->docproduct==0.2.0) (0.10.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->docproduct==0.2.0) (2.4.0)\n",
            "Requirement already satisfied: tf-estimator-nightly<1.14.0.dev2019030116,>=1.14.0.dev2019030115 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (1.14.0.dev2019030115)\n",
            "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (0.2.2)\n",
            "Requirement already satisfied: tb-nightly<1.14.0a20190302,>=1.14.0a20190301 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (1.14.0a20190301)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (1.1.0)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (0.7.1)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (3.7.1)\n",
            "Requirement already satisfied: google-pasta>=0.1.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (0.1.7)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (1.0.8)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (1.15.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (1.1.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (0.33.4)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.0.0-alpha0->docproduct==0.2.0) (0.8.0)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras->docproduct==0.2.0) (2.8.0)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras->docproduct==0.2.0) (3.13)\n",
            "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras->docproduct==0.2.0) (1.3.0)\n",
            "Collecting keras-multi-head==0.18.0 (from keras-transformer==0.21.0->docproduct==0.2.0)\n",
            "  Downloading https://files.pythonhosted.org/packages/d7/5d/8156def9ca75c55bb87819618e9a3e1f8e587c722570e2e93ad616b9269d/keras-multi-head-0.18.0.tar.gz\n",
            "Collecting keras-layer-normalization==0.11.0 (from keras-transformer==0.21.0->docproduct==0.2.0)\n",
            "  Downloading https://files.pythonhosted.org/packages/c6/b0/c786d5a5e79d985281a06da0a1f3f559cf425921464e6b07b9f1cb093a5a/keras-layer-normalization-0.11.0.tar.gz\n",
            "Collecting keras-position-wise-feed-forward==0.4.0 (from keras-transformer==0.21.0->docproduct==0.2.0)\n",
            "  Downloading https://files.pythonhosted.org/packages/91/21/4eefba0b6ea01de9c6e469970a39dbdbce14e5183a20274d9a181f55eaa8/keras-position-wise-feed-forward-0.4.0.tar.gz\n",
            "Collecting keras-embed-sim==0.3.0 (from keras-transformer==0.21.0->docproduct==0.2.0)\n",
            "  Downloading https://files.pythonhosted.org/packages/8e/16/b05954f9578ded225fd1bd56154ade949782c03b668a1fc424d5050e868a/keras-embed-sim-0.3.0.tar.gz\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sklearn->docproduct==0.2.0) (0.21.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib->docproduct==0.2.0) (41.0.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190302,>=1.14.0a20190301->tensorflow==2.0.0-alpha0->docproduct==0.2.0) (0.15.4)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190302,>=1.14.0a20190301->tensorflow==2.0.0-alpha0->docproduct==0.2.0) (3.1.1)\n",
            "Collecting keras-self-attention==0.39.0 (from keras-multi-head==0.18.0->keras-transformer==0.21.0->docproduct==0.2.0)\n",
            "  Downloading https://files.pythonhosted.org/packages/91/70/51150779d5bbd1488a30c62026b141073873faf81eac7a62c6460cb5efe0/keras-self-attention-0.39.0.tar.gz\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn->docproduct==0.2.0) (0.13.2)\n",
            "Building wheels for collected packages: docproduct, pycurl, keras-pos-embd, keras-transformer, keras-multi-head, keras-layer-normalization, keras-position-wise-feed-forward, keras-embed-sim, keras-self-attention\n",
            "  Building wheel for docproduct (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-gygb6b79/wheels/d6/ff/7e/7c265da92fd0d24132f65f8f7e28c63c8381f5478c32bd9135\n",
            "  Building wheel for pycurl (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/89/40/f5/7b4f2285aca871b5173887a6c69127210d92806c0d3a977e51\n",
            "  Building wheel for keras-pos-embd (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/fb/97/65/170068ed0a4bd2185d561afee6c93e23e87e8d735d61389590\n",
            "  Building wheel for keras-transformer (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/19/9f/ff/3b38f44f6db035cfd33cff4909edcc4864a6aeec80d9deaf23\n",
            "  Building wheel for keras-multi-head (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/9f/88/83/d7680876b48974c3c11fc334ed1d0a480ae218764062385bf3\n",
            "  Building wheel for keras-layer-normalization (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/86/dc/2e/3ac54a6b948bff68cb999d210c6ebf9e22df7a4a24cf114436\n",
            "  Building wheel for keras-position-wise-feed-forward (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/40/a1/13/3c913efde102d56ac584f61004a9fec6f8859b6feec6aa7aa7\n",
            "  Building wheel for keras-embed-sim (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/bf/f2/c6/0610efe9730c708b24ec29c25cebd38eb485acbc2eee7b5634\n",
            "  Building wheel for keras-self-attention (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/6c/d6/3e/cac34bf035198e38947006910f3ecb25613d6d9d76ea6d8ef2\n",
            "Successfully built docproduct pycurl keras-pos-embd keras-transformer keras-multi-head keras-layer-normalization keras-position-wise-feed-forward keras-embed-sim keras-self-attention\n",
            "Installing collected packages: pycurl, tensorflow, keras-pos-embd, keras-self-attention, keras-multi-head, keras-layer-normalization, keras-position-wise-feed-forward, keras-embed-sim, keras-transformer, argparse, docproduct\n",
            "  Found existing installation: tensorflow 1.14.0rc1\n",
            "    Uninstalling tensorflow-1.14.0rc1:\n",
            "      Successfully uninstalled tensorflow-1.14.0rc1\n",
            "Successfully installed argparse-1.4.0 docproduct-0.2.0 keras-embed-sim-0.3.0 keras-layer-normalization-0.11.0 keras-multi-head-0.18.0 keras-pos-embd-0.9.0 keras-position-wise-feed-forward-0.4.0 keras-self-attention-0.39.0 keras-transformer-0.21.0 pycurl-7.43.0.3 tensorflow-2.0.0a0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "argparse",
                  "tensorflow"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Collecting gpt2-estimator\n",
            "  Downloading https://files.pythonhosted.org/packages/b9/78/a83c8f020c7356bb592260fb92f13b5bf6405a31ffe7580f4847436d970a/gpt2_estimator-0.1.0-py3-none-any.whl\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from gpt2-estimator) (1.16.4)\n",
            "Requirement already satisfied: tensorflow-gpu==2.0.0a0 in /usr/local/lib/python3.6/dist-packages (from gpt2-estimator) (2.0.0a0)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.6/dist-packages (from gpt2-estimator) (3.7.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from gpt2-estimator) (0.24.2)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from gpt2-estimator) (2.21.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from gpt2-estimator) (4.28.1)\n",
            "Collecting regex (from gpt2-estimator)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6f/4e/1b178c38c9a1a184288f72065a65ca01f3154df43c6ad898624149b8b4e0/regex-2019.06.08.tar.gz (651kB)\n",
            "\u001b[K     |████████████████████████████████| 655kB 10.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: tf-estimator-nightly<1.14.0.dev2019030116,>=1.14.0.dev2019030115 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (1.14.0.dev2019030115)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (1.1.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (1.1.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (0.33.4)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (0.8.0)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (1.12.0)\n",
            "Requirement already satisfied: tb-nightly<1.14.0a20190302,>=1.14.0a20190301 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (1.14.0a20190301)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (0.7.1)\n",
            "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (0.2.2)\n",
            "Requirement already satisfied: google-pasta>=0.1.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (0.1.7)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (1.15.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0a0->gpt2-estimator) (1.0.8)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf->gpt2-estimator) (41.0.1)\n",
            "Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas->gpt2-estimator) (2.5.3)\n",
            "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas->gpt2-estimator) (2018.9)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->gpt2-estimator) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->gpt2-estimator) (2019.3.9)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->gpt2-estimator) (2.8)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->gpt2-estimator) (1.24.3)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190302,>=1.14.0a20190301->tensorflow-gpu==2.0.0a0->gpt2-estimator) (0.15.4)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190302,>=1.14.0a20190301->tensorflow-gpu==2.0.0a0->gpt2-estimator) (3.1.1)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tensorflow-gpu==2.0.0a0->gpt2-estimator) (2.8.0)\n",
            "Building wheels for collected packages: regex\n",
            "  Building wheel for regex (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/35/e4/80/abf3b33ba89cf65cd262af8a22a5a999cc28fbfabea6b38473\n",
            "Successfully built regex\n",
            "Installing collected packages: regex, gpt2-estimator\n",
            "Successfully installed gpt2-estimator-0.1.0 regex-2019.6.8\n",
            "Requirement already satisfied: pyarrow in /usr/local/lib/python3.6/dist-packages (0.13.0)\n",
            "Requirement already satisfied: numpy>=1.14 in /usr/local/lib/python3.6/dist-packages (from pyarrow) (1.16.4)\n",
            "Requirement already satisfied: six>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from pyarrow) (1.12.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2WSlwhdFC_0K",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Downaload all model checkpoints, and question/answer data. Double click to see this code\n",
        "\n",
        "def download_file_from_google_drive(id, destination):\n",
        "    URL = \"https://docs.google.com/uc?export=download\"\n",
        "\n",
        "    session = requests.Session()\n",
        "\n",
        "    response = session.get(URL, params = { 'id' : id }, stream = True)\n",
        "    token = get_confirm_token(response)\n",
        "\n",
        "    if token:\n",
        "        params = { 'id' : id, 'confirm' : token }\n",
        "        response = session.get(URL, params = params, stream = True)\n",
        "\n",
        "    save_response_content(response, destination)    \n",
        "\n",
        "def get_confirm_token(response):\n",
        "    for key, value in response.cookies.items():\n",
        "        if key.startswith('download_warning'):\n",
        "            return value\n",
        "\n",
        "    return None\n",
        "\n",
        "def save_response_content(response, destination):\n",
        "    CHUNK_SIZE = 32768\n",
        "\n",
        "    with open(destination, \"wb\") as f:\n",
        "        for chunk in response.iter_content(CHUNK_SIZE):\n",
        "            if chunk: # filter out keep-alive new chunks\n",
        "                f.write(chunk)\n",
        "                \n",
        "import os\n",
        "import requests\n",
        "\n",
        "import urllib.request\n",
        "\n",
        "# Download the file from `url` and save it locally under `file_name`:\n",
        "urllib.request.urlretrieve('https://github.com/naver/biobert-pretrained/releases/download/v1.0-pubmed-pmc/biobert_v1.0_pubmed_pmc.tar.gz', 'BioBert.tar.gz')\n",
        "\n",
        "if not os.path.exists('BioBertFolder'):\n",
        "    os.makedirs('BioBertFolder')\n",
        "    \n",
        "import tarfile\n",
        "tar = tarfile.open(\"BioBert.tar.gz\")\n",
        "tar.extractall(path='BioBertFolder/')\n",
        "tar.close()\n",
        "\n",
        "file_id = '1uCXv6mQkFfpw5txGnVCsl93Db7t5Z2mp'\n",
        "\n",
        "download_file_from_google_drive(file_id, 'Float16EmbeddingsExpanded5-27-19.pkl')\n",
        "\n",
        "file_id = 'https://onedrive.live.com/download?cid=9DEDF3C1E2D7E77F&resid=9DEDF3C1E2D7E77F%2132792&authkey=AEQ8GtkcDbe3K98'\n",
        "    \n",
        "urllib.request.urlretrieve( file_id, 'DataAndCheckpoint.zip')\n",
        "\n",
        "if not os.path.exists('newFolder'):\n",
        "    os.makedirs('newFolder')\n",
        "\n",
        "import zipfile\n",
        "zip_ref = zipfile.ZipFile('DataAndCheckpoint.zip', 'r')\n",
        "zip_ref.extractall('newFolder')\n",
        "zip_ref.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ztuTHeW4jFiH",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Load model weights and Q&A data. Double click to see code\n",
        "\n",
        "from docproduct.predictor import RetreiveQADoc\n",
        "\n",
        "pretrained_path = 'BioBertFolder/biobert_v1.0_pubmed_pmc/'\n",
        "# ffn_weight_file = None\n",
        "bert_ffn_weight_file = 'newFolder/models/bertffn_crossentropy/bertffn'\n",
        "embedding_file = 'Float16EmbeddingsExpanded5-27-19.pkl'\n",
        "\n",
        "doc = RetreiveQADoc(pretrained_path=pretrained_path,\n",
        "ffn_weight_file=None,\n",
        "bert_ffn_weight_file=bert_ffn_weight_file,\n",
        "embedding_file=embedding_file)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qZ110ZqNfBac",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Type in your question (512 word limit) and search search parameters\n",
        "\n",
        "question_text = \"I\\u2019ve had chronic migraines for four years and they just keep getting worse.  It\\u2019s to the point now that I have between 20 and 25 migraines a month. I\\u2019ve had a CT scan which came back normal, I\\u2019ve been on amitriptyline, topamax, and propranolol to try to prevent the migraines and I\\u2019ve used maxalt and sumatriptan to break migraines at onset. I haven\\u2019t had consistent relief with any medication, the 3 prophylactics only helped a few of my symptoms and the maxalt/ sumatriptan only succeeded in breaking my migraines about 25% of the time.  I just want to know if anyone can help me figure out what is wrong with me. I\\u2019ll list my symptoms below.  -Sharp shooting pain shooting down jawline from mandibular joint and across cheekbones. Usually on only one side of my face but it changes sides randomly.  -Extreme pain in eyes, sensitivity to light and movement. See flashes of light even when eyes are closed.  -sharp pain starting at base of head/top of neck that runs up head and causes pain in eyes.  -Sharp pain in ears usually accompanied by ringing  Dizziness and nausea, occasionally vomiting  Muscle spasms, usually in hands or face  All pain worsens with any change in position; standing, sitting, rolling over while lying down.  Usually I\\u2019ll know that I\\u2019m going to get a migraine because it starts with the shooting face pain or the eye pain. The 3 prophylactics helped the facial pain but didn\\u2019t reduce the number of migraines I had per month and they all gave me pretty bad side effects.  Side effects from medication  Topamax: felt drunk, nauseous, twitchy  Amitriptyline: extreme fatigue even into the next day when taken at night, confusion, balance issues, muscle spasms  Propranolol: nausea, hair loss, fatigue\"  #@param {type:\"string\"}\n",
        "\n",
        "search_similarity_by = 'answer'  #@param ['answer', \"question\"]\n",
        "\n",
        "number_results_toReturn=10 #@param {type:\"number\"}\n",
        "\n",
        "answer_only=True #@param [\"False\", \"True\"] {type:\"raw\"}\n",
        "\n",
        "returned_results = doc.predict( question_text ,\n",
        "                  search_by=search_similarity_by, topk=number_results_toReturn, answer_only=answer_only)\n",
        "\n",
        "print('')\n",
        "for jk in range(len(returned_results)):\n",
        "    print(\"Result \", jk+1)\n",
        "    print(returned_results[jk])\n",
        "    print('')\n"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}