ner_BERT_TF2.X.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bqbS2gLMhNog"
   },
   "source": [
    "# Named entity recognition with Bert TF 2.X\n",
    "\n",
    "Named Enity Recognition is one of the most common NLP problems. The goal is classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. What can you use it for? Here are a few ideas - social media, chatbot, customer support tickets, survey responses, and data mining!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MObExv0FhXmd"
   },
   "source": [
    "Let's start by importing some packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ULiI60JsTgbz"
   },
   "source": [
    "## Transformers\n",
    "Implementations of pre-trained BERT models already exist in TensorFlow due to its popularity. I leveraged the popular transformers library while building out this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0nqQ2Q1R9dhB"
   },
   "outputs": [],
   "source": [
    "!pip install transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wCwT-cxbTqwF"
   },
   "source": [
    "First, you install the transformers package by huggingface. This library contains some state-of-the-art pre-trained models for Natural Language Processing (NLP) like BERT, GPT, XLNet … etc.\n",
    "\n",
    "Now you have access to the pre-trained Bert models and the TensorFlow wrappers we will use here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1535,
     "status": "ok",
     "timestamp": 1595571355806,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "oNUyGjRQCZ3o"
   },
   "outputs": [],
   "source": [
    "files_path = \"./\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3353,
     "status": "ok",
     "timestamp": 1595571357991,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "tG83Y0AhC05p"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline\n",
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "import tensorflow as tf\n",
    "from tqdm import tqdm,trange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8d7tHjXbE6G4"
   },
   "source": [
    "# Data\n",
    "\n",
    "Let's take a look at data. [Annotated Corpus for Named Entity Recognition](https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus?select=ner_dataset.csv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 427
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8562,
     "status": "ok",
     "timestamp": 1595571364509,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "1S3bXmbXEm5g",
    "outputId": "b859ad73-f80c-41a0-b740-6be0f1c27023"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of sentences:  47959\n",
      "Number of words in the dataset:  35178\n",
      "Tags: ['I-gpe', 'B-per', 'I-geo', 'I-eve', 'O', 'B-tim', 'B-geo', 'I-tim', 'I-nat', 'B-art', 'I-art', 'B-nat', 'B-eve', 'I-org', 'B-org', 'B-gpe', 'I-per']\n",
      "Number of Labels:  17\n",
      "What the dataset looks like:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sentence #</th>\n",
       "      <th>Word</th>\n",
       "      <th>POS</th>\n",
       "      <th>Tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>Thousands</td>\n",
       "      <td>NNS</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>of</td>\n",
       "      <td>IN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>demonstrators</td>\n",
       "      <td>NNS</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>have</td>\n",
       "      <td>VBP</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>marched</td>\n",
       "      <td>VBN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>through</td>\n",
       "      <td>IN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>London</td>\n",
       "      <td>NNP</td>\n",
       "      <td>B-geo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>to</td>\n",
       "      <td>TO</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>protest</td>\n",
       "      <td>VB</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>the</td>\n",
       "      <td>DT</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Sentence #           Word  POS    Tag\n",
       "0  Sentence: 1      Thousands  NNS      O\n",
       "1  Sentence: 1             of   IN      O\n",
       "2  Sentence: 1  demonstrators  NNS      O\n",
       "3  Sentence: 1           have  VBP      O\n",
       "4  Sentence: 1        marched  VBN      O\n",
       "5  Sentence: 1        through   IN      O\n",
       "6  Sentence: 1         London  NNP  B-geo\n",
       "7  Sentence: 1             to   TO      O\n",
       "8  Sentence: 1        protest   VB      O\n",
       "9  Sentence: 1            the   DT      O"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv(f\"{files_path}ner_dataset.csv\", encoding=\"latin1\")\n",
    "data = data.fillna(method=\"ffill\")\n",
    "\n",
    "print(\"Number of sentences: \", len(data.groupby(['Sentence #'])))\n",
    "\n",
    "words = list(set(data[\"Word\"].values))\n",
    "n_words = len(words)\n",
    "print(\"Number of words in the dataset: \", n_words)\n",
    "\n",
    "tags = list(set(data[\"Tag\"].values))\n",
    "print(\"Tags:\", tags)\n",
    "n_tags = len(tags)\n",
    "print(\"Number of Labels: \", n_tags)\n",
    "\n",
    "print(\"What the dataset looks like:\")\n",
    "# Show the first 10 rows\n",
    "data.head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15795,
     "status": "ok",
     "timestamp": 1595571372128,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "HrIVsOi2Eu7o",
    "outputId": "e40cb6d2-a392-49a7-cc11-dace7a4e2154"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is what a sentence looks like:\n",
      "[('Thousands', 'NNS', 'O'), ('of', 'IN', 'O'), ('demonstrators', 'NNS', 'O'), ('have', 'VBP', 'O'), ('marched', 'VBN', 'O'), ('through', 'IN', 'O'), ('London', 'NNP', 'B-geo'), ('to', 'TO', 'O'), ('protest', 'VB', 'O'), ('the', 'DT', 'O'), ('war', 'NN', 'O'), ('in', 'IN', 'O'), ('Iraq', 'NNP', 'B-geo'), ('and', 'CC', 'O'), ('demand', 'VB', 'O'), ('the', 'DT', 'O'), ('withdrawal', 'NN', 'O'), ('of', 'IN', 'O'), ('British', 'JJ', 'B-gpe'), ('troops', 'NNS', 'O'), ('from', 'IN', 'O'), ('that', 'DT', 'O'), ('country', 'NN', 'O'), ('.', '.', 'O')]\n"
     ]
    }
   ],
   "source": [
    "class SentenceGetter(object):\n",
    "    \"\"\"Class to Get the sentence in this format:\n",
    "    [(Token_1, Part_of_Speech_1, Tag_1), ..., (Token_n, Part_of_Speech_1, Tag_1)]\"\"\"\n",
    "    def __init__(self, data):\n",
    "        \"\"\"Args:\n",
    "            data is the pandas.DataFrame which contains the above dataset\"\"\"\n",
    "        self.n_sent = 1\n",
    "        self.data = data\n",
    "        self.empty = False\n",
    "        agg_func = lambda s: [(w, p, t) for w, p, t in zip(s[\"Word\"].values.tolist(),\n",
    "                                                           s[\"POS\"].values.tolist(),\n",
    "                                                           s[\"Tag\"].values.tolist())]\n",
    "        self.grouped = self.data.groupby(\"Sentence #\").apply(agg_func)\n",
    "        self.sentences = [s for s in self.grouped]\n",
    "    \n",
    "    def get_next(self):\n",
    "        \"\"\"Return one sentence\"\"\"\n",
    "        try:\n",
    "            s = self.grouped[\"Sentence: {}\".format(self.n_sent)]\n",
    "            self.n_sent += 1\n",
    "            return s\n",
    "        except:\n",
    "            return None\n",
    "        \n",
    "getter = SentenceGetter(data)\n",
    "sent = getter.get_next()\n",
    "print('This is what a sentence looks like:')\n",
    "print(sent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 14898,
     "status": "ok",
     "timestamp": 1595571372129,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "k4VggbtJEvLJ",
    "outputId": "cb9f5c05-fb44-4a77-fb40-d9802d6b3009"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAcy0lEQVR4nO3debQmVXnv8e/PBhlEaYaWkIbYIEQv3EQgHcTIjQS4jCrGOOA12hK8naxlHJMomgGjcoNDRIhTUFAkiiIaaYeABMFoiECDiAyy7EATmiA0MojiBDz3j9pH3z726Xobz3vG72etd52qXbuqnjrV/T5n167alapCkqQNecR0ByBJmvlMFpKkXiYLSVIvk4UkqZfJQpLUy2QhSeplstCsleSAJGumOw5pPjBZaEZI8v2Bz0NJfjgw/8Lpjm+2SfLGJP803XFo7thkugOQAKpqq7HpJKuBl1bVv05fRJMvSYBU1UPTHYu0sWxZaEZLslmSdyX57/Z5V5LNJqj7iiTXJdmprfeOJP+V5PYk70+yRat3QJI1Sf4syR1JbktyzAZiuDjJ3yW5LMn3kpybZNuB5fsluSTJPUm+keSAceuekOTfgfuBXdez/dcluTXJfUluSHJQK39EkuOS/GeS7yY5e2y/SZYkqSTL2jHemeQv27LDgDcAz28ts2+08q2TnNaO99Ykb0myoC17SZKvtt/Z3UluSnL4QIzbJvlQOwd3J/nMwLKnJ7mqHf8lSX5ziFOrWcZkoZnuL4H9gL2AJwH7An81vlKSvwFeAjytqtYAJwK/3tbbDVgM/M3AKr8CbN3KjwXek2SbDcTxYuCPgB2BB4BT2n4XA58H3gJsC/w58KkkiwbWfRGwHHg0cPO4uJ8A/Cnw21X1aOBQYHVb/HLgWcDTgF8F7gbeMy6u/YEnAAcBf5Pkf1TVecD/Az5RVVtV1ZNa3Q+32HcD9gYOAV46sK0nAzcA2wNvA05rrSGAM4EtgT2BxwIntfj3Bk4H/hjYDvhHYMVECV2zWFX58TOjPnRflge36f8EjhhYdiiwuk0fANwKvBP4KrB1Kw/wA+DxA+s9BbhpYL0fApsMLL8D2G+CeC4GThyY3wP4CbAAeB1w5rj65wPLBtZ90waOdbe274OBTcctux44aGB+R+CndJePlwAF7DSw/DLg6Db9RuCfBpbtAPwY2GKg7AXARW36JcCqgWVbtu3/StvvQ8A264n/fcCbx5XdQJe0p/3fkp/J+9hnoZnuV1n3r/GbW9mYhXR/tT+/qu5tZYvovuyu+PkfxoTuy33Md6vqgYH5+4GtmNgt42LYlO4v8McBz03yjIHlmwIXTbDuOqpqVZJX0X2575nkfOA1VfXfbdv/nGSwj+NBui/+Md8Z8hge1+K6beB38ohxsf1sW1V1f6u3FV2L6a6qunuC7S5L8vKBskey7jnSHOBlKM10Y1+aY36tlY25G3g68KEkT21ld9K1HPasqoXts3UNdKI/DDuPi+GnbT+30LUsFg58HlVVJw7U3+DQzlX1saran+44C3hrW3QLcPi4bW9eVbcOEe/4fd5C17LYfmBbj6mqPYfY1i3AtkkWTrDshHExbllVZw2xXc0iJgvNdGcBf5VkUZLt6fod1rkltKouBl4IfDrJvtXdbfQB4KQkj4WubyHJob9EHH+YZI8kWwJvAs6pqgdbLM9IcmiSBUk2bx3oOw2z0SRPSHJgu8b/I7okN9aSeD9wQpLHtbqLkhw1ZLy3A0uSPAKgqm4Dvgj8fZLHtM7zxyd5Wt+G2rr/Arw3yTZJNk3yu23xB4A/SfLkdB6V5Mgkjx4yTs0SJgvNdG8BVgJXA98Ermxl66iqC+g6oD+bZB+6voRVwNeSfA/4V7qO4IfrTLoO4u8AmwOvaPu9BTiK7u6jtXR/af8Fw//f2oyuM/7Otu3HAq9vy04GVgBfTHIf8DW6TuhhfLL9/G6SK9v0i+kuEV1H1yI7h64/YhgvomtNfYuuj+VVAFW1Evi/wLvbNlfR9X9ojkmVLz+SNiTJxXSdxR+c7lik6WLLQpLUy2QhSerlZShJUi9bFpKkXnPyobztt9++lixZMt1hSNKscsUVV9xZVYvWt2ykySLd6KH30T11+kBVLW0DoX2CbriC1cDzquruNgbNycARdE+ivqSqrmzbWcbPxwN6S1WdsaH9LlmyhJUrV07+AUnSHJbk5omWTcVlqN+rqr2qammbPw64sKp2By5s8wCHA7u3z3K6MWdoyeV4uvvL9wWO7xnwTZI0yaajz+IoYKxlcAbdqJpj5R+pzteAhUl2pBs47oKqGhub5gLgsKkOWpLms1Eni6J7+vSKJMtb2Q5t+ADonlgdGxRtMesOaramlU1Uvo4ky5OsTLJy7dq1k3kMkjTvjbqDe/+qurWNz3NBkm8NLqyqSjIp9+5W1anAqQBLly71fmBJmkQjbVmMjY5ZVXcA/0zX53B7u7xE+3lHq34r647suVMrm6hckjRFRpYs2uiTjx6bpnsr1zV0A6Mta9WWAee26RXAi9vIlfsB97bLVecDh7TRLrdp2zl/VHFLkn7RKC9D7UD34pax/Xysqs5LcjlwdpJj6V4i87xW/wt0t82uort19hiAqroryZuBy1u9N1XVXSOMW5I0zpwc7mPp0qXlcxaStHGSXDHwmMM6HO5DktRrTg73ofVbctzn11u++sQjpzgSSbONLQtJUi+ThSSpl8lCktTLZCFJ6mWykCT18m4oTXiXFHinlKSOLQtJUi+ThSSpl8lCktTLZCFJ6mWykCT18m6oOWhDdzdJ0sNhy0KS1MtkIUnqZbKQJPUyWUiSetnBrQ3yhUmSwJaFJGkIJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqZLCRJvUwWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqNPFkkWZDk60k+1+Z3SXJpklVJPpHkka18sza/qi1fMrCN17fyG5IcOuqYJUnrmoqWxSuB6wfm3wqcVFW7AXcDx7byY4G7W/lJrR5J9gCOBvYEDgPem2TBFMQtSWpGmiyS7AQcCXywzQc4EDinVTkDeFabPqrN05Yf1OofBXy8qn5cVTcBq4B9Rxm3JGldo25ZvAt4LfBQm98OuKeqHmjza4DFbXoxcAtAW35vq/+z8vWs8zNJlidZmWTl2rVrJ/s4JGleG9k7uJM8Hbijqq5IcsCo9jOmqk4FTgVYunRpjXp/M8FE78eWpMk2smQBPBV4ZpIjgM2BxwAnAwuTbNJaDzsBt7b6twI7A2uSbAJsDXx3oHzM4DqSpCkwsstQVfX6qtqpqpbQdVB/qapeCFwEPKdVWwac26ZXtHna8i9VVbXyo9vdUrsAuwOXjSpuSdIvGmXLYiKvAz6e5C3A14HTWvlpwJlJVgF30SUYquraJGcD1wEPAC+rqgenPmxJmr+mJFlU1cXAxW36RtZzN1NV/Qh47gTrnwCcMLoIJUkb4hPckqReJgtJUq/p6LPQHDDRbburTzxyiiORNBVsWUiSepksJEm9TBaSpF4mC0lSL5OFJKmXyUKS1MtkIUnqZbKQJPUyWUiSepksJEm9TBaSpF4mC0lSLwcSnAV817ak6WbLQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVKv3mSR5PFJNmvTByR5RZKFow9NkjRTDPPyo08BS5PsBpwKnAt8DDhilIFpdproRU2rTzxyiiORNJmGuQz1UFU9APw+8A9V9RfAjqMNS5I0kwyTLH6a5AXAMuBzrWzTvpWSbJ7ksiTfSHJtkr9t5bskuTTJqiSfSPLIVr5Zm1/Vli8Z2NbrW/kNSQ7d2IOUJP1yhkkWxwBPAU6oqpuS7AKcOcR6PwYOrKonAXsBhyXZD3grcFJV7QbcDRzb6h8L3N3KT2r1SLIHcDSwJ3AY8N4kC4Y9QEnSL683WVTVdcDrgCvb/E1V9dYh1quq+n6b3bR9CjgQOKeVnwE8q00f1eZpyw9Kklb+8ar6cVXdBKwC9h3i2CRJk2SYu6GeAVwFnNfm90qyYpiNJ1mQ5CrgDuAC4D+Be1ofCMAaYHGbXgzcAtCW3wtsN1i+nnUG97U8ycokK9euXTtMeJKkIQ1zGeqNdH/J3wNQVVcBuw6z8ap6sKr2AnZq23jiwwtzqH2dWlVLq2rpokWLRrUbSZqXhurgrqp7x5U9tDE7qap7gIvo+j4WJhm7ZXcn4NY2fSuwM0BbvjXw3cHy9awjSZoCwySLa5P8H2BBkt2T/ANwSd9KSRaNPbyXZAvgfwPX0yWN57Rqy+ie2wBY0eZpy79UVdXKj253S+0C7A5cNtTRSZImxTDJ4uV0dyL9GDgL+B7wqiHW2xG4KMnVwOXABVX1ObrO8tckWUXXJ3Faq38asF0rfw1wHEBVXQucDVxH12/ysqp6cLjDkyRNht4nuKvqfuAv22doVXU1sPd6ym9kPXczVdWPgOdOsK0TgBM2Zv+SpMkzYbJI8lm6W13Xq6qeOZKIJEkzzoZaFu+YsigkSTPahMmiqr48Nt2G5HgiXUvjhqr6yRTENu9MNAifJE233j6LJEcC76d7oC7ALkn+uKr+ZdTBSZJmhmGGKP974PeqahV077cAPg+YLCRpnhjm1tn7xhJFcyNw34jikSTNQMO0LFYm+QLdsw5Fd3vr5UmeDVBVnx5hfJKkGWCYZLE5cDvwtDa/FtgCeAZd8jBZSNIcN8xDecdMRSCSpJlrmLuhdqEb8mPJYH0fypOk+WOYy1CfoRu36bNs5GizkqS5YZhk8aOqOmXkkUiSZqxhksXJSY4Hvkg38iwAVXXlyKKSJM0owySL3wBeRPfu7LHLUGPv0pYkzQPDJIvnArs6HpQkzV/DPMF9DbBw1IFIkmauYVoWC4FvJbmcdfssvHVWkuaJYZLF8SOPQpI0ow3zBPeX++pIfSZ6V8fqE4+c4kgkPRy9fRZJ9ktyeZLvJ/lJkgeTfG8qgpMkzQzDdHC/G3gB8G26AQRfCrxnlEFJkmaWYZIF7X0WC6rqwar6EHDYaMOSJM0kw3Rw39/ewX1VkrcBtzFkkpEkzQ3DfOm/qNX7U+AHwM7AH4wyKEnSzDLM3VA3t8kfJTkF2Hnca1YlSXPcMHdDXZzkMUm2Ba4EPpDknaMPTZI0UwxzGWrrqvoe8GzgI1X1ZODg0YYlSZpJhkkWmyTZEXge8LkRxyNJmoGGSRZvAs4HVlXV5Ul2pXvmQpI0TwzTwf1J4JMD8zfi3VCSNK8M85yFJtlE4yRJ0kzlw3WSpF4mC0lSr2Ges/irgenNht1wkp2TXJTkuiTXJnllK982yQVJvt1+btPKk+SUJKuSXJ1kn4FtLWv1v51k2cYdoiTplzVhskjyuiRPAZ4zUPwfG7HtB4A/q6o9gP2AlyXZAzgOuLCqdgcubPMAhwO7t89y4H0tjm3pXsD0ZGBf4PixBCNJmhoball8C3gusGuSryT5ALBdkicMs+Gquq2qrmzT9wHXA4uBo4AzWrUzgGe16aPoHvqrqvoasLA933EocEFV3VVVdwMX4Ki3kjSlNpQs7gHeAKwCDgBObuXHJblkY3aSZAmwN3ApsENV3dYWfQfYoU0vBm4ZWG1NK5uofPw+lidZmWTl2rVrNyY8SVKPDSWLQ4HPA48H3kl3GegHVXVMVf3OsDtIshXwKeBVbdiQn6mqAmqjo16Pqjq1qpZW1dJFixZNxiYlSc2EyaKq3lBVBwGrgTOBBcCiJF9N8tlhNp5kU7pE8dGq+nQrvr1dXqL9vKOV30o3/PmYnVrZROWSpCkyzK2z51fVyqo6FVhTVfsDx/StlCTAacD1VTU4Su0KYOyOpmXAuQPlL253Re0H3NsuV50PHJJkm9axfUgrkyRNkWGG+3jtwOxLWtmdQ2z7qXQvTvpmkqta2RuAE4GzkxwL3Ew3QCHAF4Aj6PpI7qclpKq6K8mbgctbvTdV1V1D7F+SNEk2ariPqvrGRtT9KpAJFh+0nvoFvGyCbZ0OnD7sviVJk8snuCVJvUwWkqReJgtJUi+ThSSpl++z0Iw00Ts/Vp945BRHIglsWUiShmCykCT18jKUppWvmJVmB1sWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqZLCRJvUwWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1cojyEXL4bUlzhS0LSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVIvk4UkqZfJQpLUa2TJIsnpSe5Ics1A2bZJLkjy7fZzm1aeJKckWZXk6iT7DKyzrNX/dpJlo4pXkjSxUbYsPgwcNq7sOODCqtoduLDNAxwO7N4+y4H3QZdcgOOBJwP7AsePJRhJ0tQZWbKoqn8D7hpXfBRwRps+A3jWQPlHqvM1YGGSHYFDgQuq6q6quhu4gF9MQJKkEZvqPosdquq2Nv0dYIc2vRi4ZaDemlY2UfkvSLI8ycokK9euXTu5UUvSPDdtHdxVVUBN4vZOraqlVbV00aJFk7VZSRJTnyxub5eXaD/vaOW3AjsP1NuplU1ULkmaQlOdLFYAY3c0LQPOHSh/cbsraj/g3na56nzgkCTbtI7tQ1qZJGkKjey1qknOAg4Atk+yhu6uphOBs5McC9wMPK9V/wJwBLAKuB84BqCq7kryZuDyVu9NVTW+01ySNGIjSxZV9YIJFh20nroFvGyC7ZwOnD6JoUmSNpJPcEuSepksJEm9TBaSpF4mC0lSL5OFJKmXyUKS1MtkIUnqNbLnLOaTJcd9frpDkKSRsmUhSeply0KzykStuNUnHjnFkUjziy0LSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb18zkJzgs9fSKNly0KS1MtkIUnqZbKQJPUyWUiSepksJEm9TBaSpF4mC0lSL5+z0Jzm8xfS5LBlIUnqZbKQJPUyWUiSetlnsREmuv6t2ce+DGnj2LKQJPUyWUiSenkZShrg5Slp/WxZSJJ6zZqWRZLDgJOBBcAHq+rEaQ5J88jDubnB1ojmklnRskiyAHgPcDiwB/CCJHtMb1SSNH/MlpbFvsCqqroRIMnHgaOA60axM2+R1WSYrH9HE7VQ7F/RVJotyWIxcMvA/BrgyYMVkiwHlrfZ7ye5YSP3sT1w58OOcHbxWGeRvHXoqtsDd25E/dls1p/XjTCVx/q4iRbMlmTRq6pOBU59uOsnWVlVSycxpBnLY52bPNa5aaYc66zoswBuBXYemN+plUmSpsBsSRaXA7sn2SXJI4GjgRXTHJMkzRuz4jJUVT2Q5E+B8+lunT29qq6d5N087EtYs5DHOjd5rHPTjDjWVNV0xyBJmuFmy2UoSdI0MllIknrN+2SR5LAkNyRZleS46Y5nMiXZOclFSa5Lcm2SV7bybZNckOTb7ec20x3rZEmyIMnXk3yuze+S5NJ2fj/RbpCY9ZIsTHJOkm8luT7JU+bqeU3y6vbv95okZyXZfC6d1ySnJ7kjyTUDZes9l+mc0o776iT7TFWc8zpZzINhRB4A/qyq9gD2A17Wju844MKq2h24sM3PFa8Erh+YfytwUlXtBtwNHDstUU2+k4HzquqJwJPojnnOndcki4FXAEur6n/S3eByNHPrvH4YOGxc2UTn8nBg9/ZZDrxvimKc38mCgWFEquonwNgwInNCVd1WVVe26fvovlAW0x3jGa3aGcCzpifCyZVkJ+BI4INtPsCBwDmtypw41iRbA78LnAZQVT+pqnuYo+eV7q7NLZJsAmwJ3MYcOq9V9W/AXeOKJzqXRwEfqc7XgIVJdpyKOOd7sljfMCKLpymWkUqyBNgbuBTYoapua4u+A+wwTWFNtncBrwUeavPbAfdU1QNtfq6c312AtcCH2iW3DyZ5FHPwvFbVrcA7gP+iSxL3AlcwN8/roInO5bR9Z833ZDEvJNkK+BTwqqr63uCy6u6dnvX3Tyd5OnBHVV0x3bFMgU2AfYD3VdXewA8Yd8lpDp3Xbej+mt4F+FXgUfziJZs5baacy/meLOb8MCJJNqVLFB+tqk+34tvHmq7t5x3TFd8keirwzCSr6S4nHkh3XX9hu3wBc+f8rgHWVNWlbf4cuuQxF8/rwcBNVbW2qn4KfJruXM/F8zpoonM5bd9Z8z1ZzOlhRNo1+9OA66vqnQOLVgDL2vQy4Nypjm2yVdXrq2qnqlpCdx6/VFUvBC4CntOqzZVj/Q5wS5IntKKD6Ibrn3Pnle7y035Jtmz/nseOdc6d13EmOpcrgBe3u6L2A+4duFw1UvP+Ce4kR9Bd6x4bRuSEaQ5p0iTZH/gK8E1+fh3/DXT9FmcDvwbcDDyvqsZ3sM1aSQ4A/ryqnp5kV7qWxrbA14E/rKofT2d8kyHJXnQd+Y8EbgSOofvjb86d1yR/Czyf7u6+rwMvpbtOPyfOa5KzgAPohiK/HTge+AzrOZctYb6b7lLc/cAxVbVySuKc78lCktRvvl+GkiQNwWQhSeplspAk9TJZSJJ6mSwkSb1MFpqRknx/BNvcIsmX2wCSI5NkdZLtR7mPtp+3t9FY3z6u/IAkvzPE+h9O8py+ekNs5x1JDvxlt6OZbVa8VlWaJH8EfLqqHpzuQCaSZJOBMY/6LAe2Xc/xHAB8H7hkMmPbgH8APgB8aYr2p2lgy0KzRpLHJzkvyRVJvpLkia38w22M/0uS3LiBv5ZfSHsStv31ffHAOyE+2h54WqdlkGRpkovb9BuTnNH2fXOSZyd5W5Jvtrg2HdjXa1v5ZUl2a+svSvKpJJe3z1MHtntmkn8Hzhx3zGktiGva9p7fylcAWwFXjJW18iXAnwCvTnJVkv+VZEmSL7X3H1yY5NfW87t9c/s9LkjyFy2+q9sDcbRtXJ/kA60188UkWwBU1c3Adkl+ZdhzqdnHZKHZ5FTg5VX1W8CfA+8dWLYjsD/wdODE8Su24Vx2rarVA8V7A6+ie5fJrnRjDvV5PN24U88E/gm4qKp+A/gh3fDoY+5t5e+mGyEAurGqTqqq3wb+gDaUerMHcHBVvWDc/p4N7EX3zoqDgbcn2bGqngn8sKr2qqpPjFVux/f+tp+9quordH/5n1FVvwl8FDhl3O/m7cAiuqfAD6J7V8K+bb+/leR3W9XdgfdU1Z7APe0YxlzJcL8/zVJehtKskG7k3N8BPtkaAACbDVT5TFU9BFyXZH1Dc29P9wU36LKqWtO2fxWwBPhqTyj/UlU/TfJNuiFizmvl32zrjzlr4OdJbfpgYI+B+B/TjgtgRVX9cD372x84q11quj3Jl4HfZuPGMHsKXdKBruXytoFlfw1cWlXLAZIcAhxCN4QGdK2X3enGaLqpqq5q5Vew7vHeQTcqrOYok4Vmi0fQvcNgrwmWD44LlPUs/yGw+QbWeZCf/394gJ+3ute7TlU9lOSn9fPxch5i3f9PtZ7pRwD7VdWPBjfYkscP1hPzVLicrvWwbRtHKsDfVdU/DlZql7fG/762GJjfnO53rDnKy1CaFdp7OG5K8lz42bX8J23E+ncDC5KM//Jfn9XAb7XpP9hAvQ15/sDP/2jTXwRePlahDQbY5yvA81tfwiK6N+Rd1rPOfcCjB+YvoRuJF7p+m68MLDuP7rLd55M8Gjgf+KOxFk+SxUkeO0Scvw5c01tLs5bJQjPVlknWDHxeQ/dFd2ySbwDXsvGvwP0i3WWdPn8LnJxkJd1f0A/HNkmupnsn+Ktb2SuApa3j+Dq6jug+/wxcDXyD7m6j17Yhyjfks8Dvj3Vw0yWoY1o8L2ox/UxVfZLubqYVdInkY8B/tEtt57Bu4vkFrWN/N2BKRj/V9HDUWc0bSfYBXl1VL5ruWOaSJL8P7FNVfz3dsWh0bFlo3qiqK4GLMuKH8uahTYC/n+4gNFq2LCRJvWxZSJJ6mSwkSb1MFpKkXiYLSVIvk4Ukqdf/B9dWWtejw9LFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get all the sentences\n",
    "sentences = getter.sentences\n",
    "\n",
    "# Plot sentence by lenght\n",
    "plt.hist([len(s) for s in sentences], bins=50)\n",
    "plt.title('Token per sentence')\n",
    "plt.xlabel('Len (number of token)')\n",
    "plt.ylabel('# samples')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PprAvnVuZTkE"
   },
   "source": [
    "# Data Preprocessing\n",
    "Before feeding the data into the model, we have to preprocess the text.\n",
    "\n",
    "* We will use the word2idx dictionary to convert each word to a corresponding integer ID and the tag2idx to do the same for the labels. Representing words as integers saves a lot of memory!\n",
    "* In order to feed the text into our Bi-LSTM-CRF, all texts should be the same length. We ensure this using the sequence.pad_sequences() method and MAX_LEN variable. All texts longer than MAX_LEN are truncated and shorter texts are padded to get them to the same length.\n",
    "The *Tokens per sentence* plot (see above) is useful for setting the MAX_LEN training hyperparameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 14428,
     "status": "ok",
     "timestamp": 1595571373493,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "c_EVfMnwXpK4",
    "outputId": "6465c32e-6b45-492d-ac00-5384506f8260"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((43163,), (4796,), (43163,), (4796,))"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag_list=data.Tag.unique()\n",
    "\n",
    "X = [[w[0] for w in s] for s in sentences]\n",
    "y = [[w[2] for w in s] for s in sentences]\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.1)\n",
    "np.array(X_tr).shape, np.array(X_te).shape, np.array(y_tr).shape, np.array(y_te).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KadItZmlPv2N"
   },
   "source": [
    "# Training Parameters\n",
    "We'll set the hyperparameters for training our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13286,
     "status": "ok",
     "timestamp": 1595571373494,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "Wj0wknTLPvlF"
   },
   "outputs": [],
   "source": [
    "MAX_LENGTH=128\n",
    "BERT_MODEL=\"bert-base-cased\"\n",
    "\n",
    "BATCH_SIZE=32\n",
    "\n",
    "pad_token=0,\n",
    "pad_token_segment_id=0,\n",
    "sequence_a_segment_id=0,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tNBkA5eRQGFd"
   },
   "source": [
    "## Tokenizer\n",
    "\n",
    "The Bert implementation comes with a pretrained tokenizer and a definied vocabulary. We load the one related to the smallest pre-trained model bert-base-cased. We use the cased variate since it is well suited for NER."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1345,
     "status": "ok",
     "timestamp": 1595572071527,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "GzTge7U0P58y"
   },
   "outputs": [],
   "source": [
    "from transformers import (\n",
    "    TF2_WEIGHTS_NAME,\n",
    "    BertConfig,\n",
    "    BertTokenizer,\n",
    "    TFBertForTokenClassification,\n",
    "    create_optimizer,BertForSequenceClassification)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15259,
     "status": "ok",
     "timestamp": 1595571376673,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "5ySJ_ZAaQB_J"
   },
   "outputs": [],
   "source": [
    "MODEL_CLASSES = {\"bert\": (BertConfig, TFBertForTokenClassification, BertTokenizer)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13655,
     "status": "ok",
     "timestamp": 1595571376674,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "_Wym8k0cQJ74"
   },
   "outputs": [],
   "source": [
    "label_map = {label: i for i, label in enumerate(tag_list)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12934,
     "status": "ok",
     "timestamp": 1595571376675,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "j199l7t6UR4e",
    "outputId": "937d5bd1-6eb9-4248-b785-6cf6ab176c5d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_labels = len(tag_list) + 1\n",
    "num_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12572,
     "status": "ok",
     "timestamp": 1595571376676,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "NnuS_svFQKqZ"
   },
   "outputs": [],
   "source": [
    "pad_token_label_id = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o1kSEXeSSutz"
   },
   "source": [
    "## Create a Model\n",
    "NER is the multi-class classification problem where the words are our input and tags are our labels. The transformers package provides a TFBertForTokenClassification class for token-level predictions.TFBertForTokenClassification is a fine-tuning model that wraps BertModel and adds token-level classifier on top of the BertModel. We load the pre-trained “bert-base-cased” model and provide the number of possible labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11822,
     "status": "ok",
     "timestamp": 1595571376676,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "1VRXEiUzQlBY"
   },
   "outputs": [],
   "source": [
    "config_class, model_class, tokenizer_class = MODEL_CLASSES['bert']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 66,
     "referenced_widgets": [
      "24d72d81b6ca40f9be3c321d43d938b8",
      "5437a20fcb57469f9d24f905ce4391e6",
      "838eb46b7a5a4a3d87d1b597c7241cd8",
      "89f2780b76a445a991ab1f65fd128bf3",
      "30899fed1e9449c3b68f1e322ece7fba",
      "9e82c1a619084d39a79cab3d2870e78f",
      "00ef1c5373c64b47a65ce6e250f2c758",
      "4faa2c1ab6a54c0fa8b7b47c171a7c6d"
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12890,
     "status": "ok",
     "timestamp": 1595571378253,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "lsR5zlxLQk5a",
    "outputId": "e958987d-70f6-4723-8801-1220eff2fb47"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24d72d81b6ca40f9be3c321d43d938b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=433.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "config = config_class.from_pretrained(BERT_MODEL,num_labels=num_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 66,
     "referenced_widgets": [
      "185aedd2207a4b19b1e28f3be4da71d6",
      "c5ea74ce0cd242fab58dc0f701b6482b",
      "2f9e40d67a744fc590a925f06323ebaf",
      "c204555630c543efb3e85b27acdeddb0",
      "7e4df53f97254e35a42d0b8e9494067d",
      "36305fe5cd2a4cee9e3d3616b910e20f",
      "dced0dddb3644323801164d79fb29226",
      "617363039cd84c42978f2e5643ecaf19"
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 14642,
     "status": "ok",
     "timestamp": 1595571380380,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "NyyCts8KQk2B",
    "outputId": "125ed0a3-51ae-4122-9e33-04af2df312cd"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "185aedd2207a4b19b1e28f3be4da71d6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=213450.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer = tokenizer_class.from_pretrained(BERT_MODEL,do_lower_case=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170,
     "referenced_widgets": [
      "f0f12bf68b144d92a52670c909befa81",
      "adf985195dc3451cbd2bab673d3fbf60",
      "7c5348197dbd4751baf3cde719c6cf40",
      "afaf60992aff4e329da4e1862439fb21",
      "46d9d6e55ce146d1a5fb0c6582e964be",
      "b80384fed0af4dd49b0da9bf39c53db1",
      "7182fd1f29f244be81c38f23827b9d60",
      "4c7484f716b24a698a744788f6e7ec88"
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 47408,
     "status": "ok",
     "timestamp": 1595571413529,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "SRgMPUFjQkyT",
    "outputId": "e389bcee-91ae-4c72-d1fd-c06be1622196"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f0f12bf68b144d92a52670c909befa81",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=526681800.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing TFBertForTokenClassification: ['nsp___cls', 'mlm___cls']\n",
      "- This IS expected if you are initializing TFBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
      "- This IS NOT expected if you are initializing TFBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of TFBertForTokenClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier', 'dropout_37']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = model_class.from_pretrained(\n",
    "                BERT_MODEL,\n",
    "                from_pt=bool(\".bin\" in BERT_MODEL),\n",
    "                config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 47041,
     "status": "ok",
     "timestamp": 1595571413530,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "iQ6iLRzyQkrk"
   },
   "outputs": [],
   "source": [
    " model.layers[-1].activation = tf.keras.activations.softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MhirjnX5Sd8a"
   },
   "source": [
    "### Prepare Input\n",
    "Before we can start fine-tuning the model, we have to prepare the data set for use with BERT. We need to set the text into 3 kinds of embeddings:\n",
    "\n",
    "#### Token embedding\n",
    "In order to make token embedding, we need to map the word token into the id.\n",
    "\n",
    "#### Attention Mask\n",
    "In order to make mask word embedding, we need to use 1 to indicate the real toke and 0 to indicate to pad token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 46271,
     "status": "ok",
     "timestamp": 1595571413531,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "coA1_1GsQkoP",
    "outputId": "ca4597c1-a484-4882-bb2d-b8fba8aa9c70"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.sequence import pad_sequences\n",
    "max_seq_length =128\n",
    "\n",
    "def convert_to_input(sentences,tags):\n",
    "    # Where “token_type_ids” are used to indicate whether this is the first sequence or the second sequence.\n",
    "    input_id_list,attention_mask_list,token_type_id_list=[],[],[]\n",
    "    label_id_list=[]\n",
    "  \n",
    "    for x,y in tqdm(zip(sentences,tags),total=len(tags)):\n",
    "  \n",
    "        tokens = []\n",
    "        label_ids = []\n",
    "\n",
    "        for word, label in zip(x, y):\n",
    "          word_tokens = tokenizer.tokenize(word)\n",
    "          tokens.extend(word_tokens)\n",
    "          # Use the real label id for the first token of the word, and padding ids for the remaining tokens\n",
    "          label_ids.extend([label_map[label]] + [pad_token_label_id] * (len(word_tokens) - 1))\n",
    "\n",
    "\n",
    "        special_tokens_count =  2\n",
    "        if len(tokens) > max_seq_length - special_tokens_count:\n",
    "          tokens = tokens[: (max_seq_length - special_tokens_count)]\n",
    "          label_ids = label_ids[: (max_seq_length - special_tokens_count)]\n",
    "\n",
    "        label_ids = [pad_token_label_id]+label_ids+[pad_token_label_id]\n",
    "        inputs = tokenizer.encode_plus(tokens,add_special_tokens=True, max_length=max_seq_length,truncation=True)\n",
    "\n",
    "        input_ids, token_type_ids = inputs[\"input_ids\"], inputs[\"token_type_ids\"]\n",
    "        attention_masks = [1] * len(input_ids)\n",
    "\n",
    "        attention_mask_list.append(attention_masks)\n",
    "        input_id_list.append(input_ids)\n",
    "        token_type_id_list.append(token_type_ids)\n",
    "\n",
    "        label_id_list.append(label_ids)\n",
    "\n",
    "    return input_id_list,token_type_id_list,attention_mask_list,label_id_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 97607,
     "status": "ok",
     "timestamp": 1595571465214,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "CRmsws9U_lPK",
    "outputId": "112233b0-22ac-4900-ccb1-b8326fcbbd42"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 43163/43163 [00:51<00:00, 838.00it/s]\n"
     ]
    }
   ],
   "source": [
    "input_ids_train,token_ids_train,attention_masks_train,label_ids_train=convert_to_input(X_tr,y_tr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 102664,
     "status": "ok",
     "timestamp": 1595571470743,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "sSzdTuqI_dpR",
    "outputId": "534110b2-e203-44f6-c1e3-5233805ea0d6"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4796/4796 [00:05<00:00, 861.69it/s]\n"
     ]
    }
   ],
   "source": [
    "input_ids_test,token_ids_test,attention_masks_test,label_ids_test=convert_to_input(X_te,y_te)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lXqSoCzRRjIX"
   },
   "source": [
    "Next, we cut and pad the token and label sequences to our desired length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 103046,
     "status": "ok",
     "timestamp": 1595571471898,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "fi2jjGn5_wWf"
   },
   "outputs": [],
   "source": [
    "input_ids_train = pad_sequences(input_ids_train,maxlen=max_seq_length,dtype=\"long\",truncating=\"post\",padding=\"post\")\n",
    "token_ids_train = pad_sequences(token_ids_train,maxlen=max_seq_length,dtype=\"long\",truncating=\"post\",padding=\"post\")\n",
    "attention_masks_train = pad_sequences(attention_masks_train,maxlen=max_seq_length,dtype=\"long\",truncating=\"post\",padding=\"post\")\n",
    "label_ids_train = pad_sequences(label_ids_train,maxlen=max_seq_length,dtype=\"long\",truncating=\"post\",padding=\"post\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 102587,
     "status": "ok",
     "timestamp": 1595571471899,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "HdI5url2_xb8",
    "outputId": "88f69a82-ffde-4897-8ab8-3423c626c7b9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((43163, 128), (43163, 128), (43163, 128), (43163, 128))"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(input_ids_train),np.shape(token_ids_train),np.shape(attention_masks_train),np.shape(label_ids_train),"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 102216,
     "status": "ok",
     "timestamp": 1595571471900,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "XgxZutfkAwH_"
   },
   "outputs": [],
   "source": [
    "input_ids_test = pad_sequences(input_ids_test,maxlen=max_seq_length,dtype=\"long\",truncating=\"post\",padding=\"post\")\n",
    "token_ids_test = pad_sequences(token_ids_test,maxlen=max_seq_length,dtype=\"long\",truncating=\"post\",padding=\"post\")\n",
    "attention_masks_test = pad_sequences(attention_masks_test,maxlen=max_seq_length,dtype=\"long\",truncating=\"post\",padding=\"post\")\n",
    "label_ids_test = pad_sequences(label_ids_test,maxlen=max_seq_length,dtype=\"long\",truncating=\"post\",padding=\"post\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 101770,
     "status": "ok",
     "timestamp": 1595571471901,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "YGJT6fkLQ-Bp",
    "outputId": "437272f6-af0c-4de4-d481-a9a0f4697d60"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4796, 128), (4796, 128), (4796, 128), (4796, 128))"
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(input_ids_test),np.shape(token_ids_test),np.shape(attention_masks_test),np.shape(label_ids_test),"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "snSlF9zmRW7E"
   },
   "source": [
    "The last step is to define tf.data.dataset. We shuffle the data at training time and at test time we just pass them sequentially."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 99260,
     "status": "ok",
     "timestamp": 1595571471902,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "Sn_OQlNcRAI8"
   },
   "outputs": [],
   "source": [
    "def example_to_features(input_ids,attention_masks,token_type_ids,y):\n",
    "    return {\"input_ids\": input_ids,\n",
    "          \"attention_mask\": attention_masks,\n",
    "          \"token_type_ids\": token_type_ids},y\n",
    "\n",
    "train_ds = tf.data.Dataset.from_tensor_slices((input_ids_train,attention_masks_train,token_ids_train,label_ids_train)).map(example_to_features).shuffle(1000).batch(32).repeat(5)\n",
    "\n",
    "\n",
    "test_ds=tf.data.Dataset.from_tensor_slices((input_ids_test,attention_masks_test,token_ids_test,label_ids_test)).map(example_to_features).batch(1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 98926,
     "status": "ok",
     "timestamp": 1595571472787,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "62pzyjzlRCxh",
    "outputId": "aa2925df-840b-410a-90bd-205ae3350152"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[  101,  3788,  2686,  1121,  1103,  1347,  1407,  2992,   117,\n",
      "         1134,  1145,  4102,  5586, 13298,   117,  1132,  1496,  1118,\n",
      "         1357,  1659,   117,  1170, 11344,  1132, 10456,   119,   102,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0]])>, 'attention_mask': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>, 'token_type_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>} tf.Tensor(\n",
      "[[ 0  0  0  0  0  7 12  0  0  0  0  0  0  0  0  0  0  0  7 12  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0]], shape=(1, 128), dtype=int64)\n",
      "{'input_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[  101,  1130,  1168,  9093,   117,  1103,   158,   119,   156,\n",
      "          119,  1764,  2536,  2967, 15692, 19091,  9029,  5286,  1222,\n",
      "          170,  1459,  7890,  1158,  1164,  1851,  6321, 17219,  1107,\n",
      "         1103,  2466,  3199,  1104,  2586,   118,  1760,  6824,   117,\n",
      "         1601,  1106,  1103,  8697,  3070,   119,   102,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0]])>, 'attention_mask': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>, 'token_type_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>} tf.Tensor(\n",
      "[[0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0\n",
      "  0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]], shape=(1, 128), dtype=int64)\n",
      "{'input_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[  101,  1828,   119, 14104, 17266,  1145,  4892,  1103,  1433,\n",
      "          112,   188,  1840,  1111, 14516, 17482, 11745,  2050,  9283,\n",
      "         1106, 14926,  3519,  7624,   119,   102,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0]])>, 'attention_mask': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>, 'token_type_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>} tf.Tensor(\n",
      "[[ 0  3  0 10  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0]], shape=(1, 128), dtype=int64)\n",
      "{'input_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[  101,  1109,  1210,  4421,  1811,  1441,   117, 12811,  8158,\n",
      "          141, 16090,  1116,   117, 21851,   156, 19456,  5567,   117,\n",
      "         1105,  8603,  1611,  2087,  2393,   118, 10136, 14721,  1127,\n",
      "         1145,  7590,  1104,  7078,  1106,  2393,   118, 22515,  2246,\n",
      "         3031,  1181,   117,  1126,  2369,  2475,  1106,  1129,  5128,\n",
      "         1106,  2393,   118,   154, 19954,  1161,  1105,  2917,  1118,\n",
      "         8158, 27389,  1830,  2393,   118,   163,  1813, 24750, 10073,\n",
      "          117,  1150,  1208,  3711,  1106,  1129,  1246,  1104,  2393,\n",
      "          118, 20205,  1107,  5008,   119,   102,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0]])>, 'attention_mask': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>, 'token_type_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>} tf.Tensor(\n",
      "[[ 0  0  0  0  0  0  0  3 10 10  0  0  0  3 10  0  0  0  0  3  0  0 10  0\n",
      "   0  0  0  0  0  0  0  0  5  0  0  0  0  0  0  0  0  0  0  0  0  0  5  0\n",
      "   0  0  0  0  0  0  3 10  0 10  0  0  0  0  0  0  0  0  0  0  0  0  0  5\n",
      "   0  0  0  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0  0  0  0]], shape=(1, 128), dtype=int64)\n",
      "{'input_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[  101,   138,  2880,  8382, 15465,  3659,  1103,  1260, 11379,\n",
      "         1880,  1104,  4149, 25775,  2047,   117,   170,  3519,  7041,\n",
      "         1121,  1756,  3625,   119,   102,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "            0,     0]])>, 'attention_mask': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>, 'token_type_ids': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=\n",
      "array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>} tf.Tensor(\n",
      "[[0 0 0 0 0 0 0 0 0 0 0 1 4 0 0 0 0 0 0 1 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]], shape=(1, 128), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "for x,y in test_ds.take(5):\n",
    "    print(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 252
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 97712,
     "status": "ok",
     "timestamp": 1595571472787,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "Xs_FkphAREij",
    "outputId": "ba61df02-3c52-4f89-c175-882039940b64"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"tf_bert_for_token_classification\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "bert (TFBertMainLayer)       multiple                  108310272 \n",
      "_________________________________________________________________\n",
      "dropout_37 (Dropout)         multiple                  0         \n",
      "_________________________________________________________________\n",
      "classifier (Dense)           multiple                  13842     \n",
      "=================================================================\n",
      "Total params: 108,324,114\n",
      "Trainable params: 108,324,114\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 96169,
     "status": "ok",
     "timestamp": 1595571472788,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "k5_k2RD5RHbx"
   },
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam()\n",
    "loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 95865,
     "status": "ok",
     "timestamp": 1595571472788,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "CTxxSyyURIAM"
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer, loss=loss, metrics=[metric])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Qn0z30JeRQVN"
   },
   "source": [
    "# Train Model\n",
    "\n",
    "Train the model for 3 epochs in mini-batches of 32 samples. This is 3 iterations over all samples in the train_ds and test_ds tensors. While training, monitor the model’s loss and accuracy on the samples from the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 5995985,
     "status": "ok",
     "timestamp": 1595526004434,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "UMWWVI6ORHyy",
    "outputId": "d181cc4a-51ab-45d3-a2ef-9503ae707b60"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6745/6745 [==============================] - 5993s 888ms/step - loss: 2.0077 - accuracy: 0.9738 - val_loss: 2.0077 - val_accuracy: 0.9738\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(train_ds, epochs=1, validation_data=test_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0PyOch_PtTLF"
   },
   "outputs": [],
   "source": [
    "# Load the TensorFlow model in PyTorch for inspection\n",
    "model.save_pretrained(f'{files_path}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fe4_3W_Yggum"
   },
   "source": [
    "# Training & Evaluate\n",
    "*Note:* Accuracy isn't the best metric to choose for evaluating this type of task because most of the time it will correctly predict **'O'** or **'PAD'** without identifing the important Tags, which are the ones we are interested in. So after training for some epochs, we can monitor the **precision**, **recall** and **f1-score** for each of the Tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11588,
     "status": "ok",
     "timestamp": 1595571592208,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "Y8lrgSdCeV2L"
   },
   "outputs": [],
   "source": [
    "model.load_weights(f'{files_path}tf_model.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZinaQ6P4g1WD"
   },
   "outputs": [],
   "source": [
    "# hist = pd.DataFrame(history.history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1186,
     "status": "ok",
     "timestamp": 1595575122673,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "v2IbbscXg1xM"
   },
   "outputs": [],
   "source": [
    "# plt.style.use(\"ggplot\")\n",
    "# plt.figure(figsize=(12, 12))\n",
    "# plt.plot(hist[\"accuracy\"])\n",
    "# plt.plot(hist[\"val_accuracy\"])\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 293145,
     "status": "ok",
     "timestamp": 1595571941883,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "pRdiGXcUxmqf",
    "outputId": "e0e7b923-4e6d-49f2-db09-baac3652577c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4796/4796 [==============================] - 292s 61ms/step - loss: 2.0076 - accuracy: 0.9739\n",
      "[2.0075690746307373, 0.9739007949829102]\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(test_ds)\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DUjTp5E_5ULF"
   },
   "source": [
    "# References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Kw4-xyLYtA3"
   },
   "source": [
    "[Named entity recognition with Bert\n",
    "](https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/)\n",
    "\n",
    "[Name Entity Recognition with BERT in TensorFlow](https://androidkt.com/name-entity-recognition-with-bert-in-tensorflow/)\n",
    "\n",
    "[Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.](https://github.com/huggingface/transformers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MJxcp60S5ZSI"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "authorship_tag": "ABX9TyOtTqULq74hv3lYbULmGLeR",
   "collapsed_sections": [],
   "mount_file_id": "1C2k0WHQ08FiriI_uN-USg4mZUXmSaypn",
   "name": "ner_BERT_TF2.X.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "00ef1c5373c64b47a65ce6e250f2c758": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "185aedd2207a4b19b1e28f3be4da71d6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2f9e40d67a744fc590a925f06323ebaf",
       "IPY_MODEL_c204555630c543efb3e85b27acdeddb0"
      ],
      "layout": "IPY_MODEL_c5ea74ce0cd242fab58dc0f701b6482b"
     }
    },
    "24d72d81b6ca40f9be3c321d43d938b8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_838eb46b7a5a4a3d87d1b597c7241cd8",
       "IPY_MODEL_89f2780b76a445a991ab1f65fd128bf3"
      ],
      "layout": "IPY_MODEL_5437a20fcb57469f9d24f905ce4391e6"
     }
    },
    "2f9e40d67a744fc590a925f06323ebaf": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_36305fe5cd2a4cee9e3d3616b910e20f",
      "max": 213450,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_7e4df53f97254e35a42d0b8e9494067d",
      "value": 213450
     }
    },
    "30899fed1e9449c3b68f1e322ece7fba": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "36305fe5cd2a4cee9e3d3616b910e20f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "46d9d6e55ce146d1a5fb0c6582e964be": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "4c7484f716b24a698a744788f6e7ec88": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4faa2c1ab6a54c0fa8b7b47c171a7c6d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5437a20fcb57469f9d24f905ce4391e6": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "617363039cd84c42978f2e5643ecaf19": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7182fd1f29f244be81c38f23827b9d60": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7c5348197dbd4751baf3cde719c6cf40": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b80384fed0af4dd49b0da9bf39c53db1",
      "max": 526681800,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_46d9d6e55ce146d1a5fb0c6582e964be",
      "value": 526681800
     }
    },
    "7e4df53f97254e35a42d0b8e9494067d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "838eb46b7a5a4a3d87d1b597c7241cd8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9e82c1a619084d39a79cab3d2870e78f",
      "max": 433,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_30899fed1e9449c3b68f1e322ece7fba",
      "value": 433
     }
    },
    "89f2780b76a445a991ab1f65fd128bf3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4faa2c1ab6a54c0fa8b7b47c171a7c6d",
      "placeholder": "​",
      "style": "IPY_MODEL_00ef1c5373c64b47a65ce6e250f2c758",
      "value": " 433/433 [00:00&lt;00:00, 514B/s]"
     }
    },
    "9e82c1a619084d39a79cab3d2870e78f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "adf985195dc3451cbd2bab673d3fbf60": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "afaf60992aff4e329da4e1862439fb21": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4c7484f716b24a698a744788f6e7ec88",
      "placeholder": "​",
      "style": "IPY_MODEL_7182fd1f29f244be81c38f23827b9d60",
      "value": " 527M/527M [00:21&lt;00:00, 24.1MB/s]"
     }
    },
    "b80384fed0af4dd49b0da9bf39c53db1": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c204555630c543efb3e85b27acdeddb0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_617363039cd84c42978f2e5643ecaf19",
      "placeholder": "​",
      "style": "IPY_MODEL_dced0dddb3644323801164d79fb29226",
      "value": " 213k/213k [00:00&lt;00:00, 357kB/s]"
     }
    },
    "c5ea74ce0cd242fab58dc0f701b6482b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dced0dddb3644323801164d79fb29226": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f0f12bf68b144d92a52670c909befa81": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7c5348197dbd4751baf3cde719c6cf40",
       "IPY_MODEL_afaf60992aff4e329da4e1862439fb21"
      ],
      "layout": "IPY_MODEL_adf985195dc3451cbd2bab673d3fbf60"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}