8. Embeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "8. Embeddings.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DataScienceUB/DeepLearningMaster2019/blob/master/8.%20Embeddings.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "If6j5zRpoWts",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Embeddings\n",
        "\n",
        "Word games:\n",
        "\n",
        "+ [Talk to books](https://books.google.com/talktobooks/)\n",
        "\n",
        "+ [Semantris](https://research.google.com/semantris)\n",
        "\n",
        "Deep Learning algorithms require the input to be represented as (sequences of) fixed-length feature vectors. \n",
        "\n",
        "+ Words in documents and other categorical features such as user/product ids in recommeders, names of places, visited URLs, etc. are usually represented by using a one-of-K scheme (**one-hot encoding**). \n",
        "\n",
        "+ Phrases are represented by bag-of-words or bag-of-ngrams features, loosing the ordering of words and ignoring semantics. \n",
        "\n",
        "Are these good representations for deep learning?\n",
        "\n",
        "Let's see how to represent **words**, but this line of reasoning can be extended to other items.\n",
        "\n",
        "+ There are an estimated 13 million tokens for the English language. \n",
        "\n",
        "+ One possible strategy is to encode word tokens each into some vector that represents a point in some sort of *word space* that *represents* language semantics. \n",
        "\n",
        "+ The most intuitive reason is that perhaps there actually exists some $N$-dimensional space (such that $N << 13$ million) that is sufficient to encode all semantics of our language. \n",
        "\n",
        "+ Each dimension would encode some meaning that we transfer using speech."
      ]
    },
    {
      "metadata": {
        "id": "xmiMc411oWtv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### One-hot encoding\n",
        "\n",
        "If we represent every word as an $\\mathbb{R}^{|V|\\times 1}$ vector with all $0$s and one $1$ at the index of that word in the sorted english language, word vectors in this type of encoding would appear as the following:\n",
        "\n",
        "<centering>\n",
        "$$w^{aardvark} = \\left[ \\begin{array}{c} 1 \\\\ 0 \\\\ 0 \\\\ \\vdots \\\\ 0 \\end{array} \\right], w^{a} = \\left[ \\begin{array}{c} 0 \\\\ 1 \\\\ 0 \\\\ \\vdots \\\\ 0 \\end{array} \\right] , w^{at} = \\left[ \\begin{array}{c} 0 \\\\ 0 \\\\ 1 \\\\ \\vdots \\\\ 0 \\end{array} \\right] , \\cdots,  w^{zebra} = \\left[ \\begin{array}{c} 0 \\\\ 0 \\\\ 0 \\\\ \\vdots \\\\ 1 \\end{array} \\right] $$\n",
        "</centering>\n",
        "\n",
        "\n",
        "We represent each word as a completely independent entity:\n",
        "\n",
        "$$(w^{hotel})^Tw^{motel} = (w^{hotel})^Tw^{cat} = 0$$\n",
        "\n",
        "What other alternatives are there?"
      ]
    },
    {
      "metadata": {
        "id": "7HyeTq90oWtw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Semantics from word-document matrix\n",
        "\n",
        "As our first attempt, we make the bold conjecture that words that are related will often appear in the same documents (or phrases, paragraphs, etc.). \n",
        "\n",
        "For instance, \"banks\", \"bonds\", \"stocks\", \"money\", etc. are probably likely to appear together. But \"banks\", \"octopus\", \"banana\", and \"hockey\" would probably not consistently appear together. \n",
        "\n",
        "We use this fact to build a word-document matrix, $X$ in the following manner: \n",
        "\n",
        "+ Loop over billions of documents and for each time word $i$ appears in document $j$, we add one to entry $X_{ij}$. \n",
        "\n",
        "This is obviously a very large matrix ($\\mathbb{R}^{|V|\\times M}$) and it scales with the number of documents ($M$). \n",
        "\n",
        "So perhaps we can try something better, such as building a window based co-occurrence matrix.\n",
        "\n",
        "In this method we count the number of times each word appears inside a window of a particular size around the word of interest (this is a sparse ($\\mathbb{R}^{|V|\\times |V|}$) matrix). We calculate this count for all the words in corpus. \n",
        "\n",
        "Let our corpus contain just three sentences and the window size be 1:\n",
        "\n",
        "+ ``I enjoy flying``.\n",
        "+ ``I like NLP``.\n",
        "+ ``I like deep learning``.\n",
        "\n",
        "The resulting counts matrix will then be:\n",
        "\n",
        "$$X=\\left[ \\begin{array}{cccccccc}\n",
        "        & I & like & enjoy & deep & learning & NLP & flying & . \\\\\n",
        "     I & 0 & 2 & 1 & 0 & 0 & 0 & 0 & 0\\\\\n",
        "    like & 2 & 0 & 0 & 1 & 0 & 1 & 0 & 0\\\\\n",
        "    enjoy & 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 \\\\\n",
        "    deep & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 0 \\\\\n",
        "    learning & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 1\\\\\n",
        "    NLP & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1\\\\\n",
        "    flying & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 1\\\\\n",
        "    . & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 \\\\\n",
        "  \\end{array} \\right]$$\n",
        "\n",
        "Once the $|V|\\times|V|$ co-occurrence matrix $X$ has been generated, we can apply SVD on $X$ to get $X = USV^T$ and select the first $k$ columns of $U$ to get a $k$-dimensional word vectors. \n",
        "\n",
        "$\\frac{\\sum_{i = 1}^{k}\\sigma_i}{\\sum_{i = 1}^{|V|}\\sigma_i}$ indicates the amount of variance captured by the first $k$ dimensions.\n",
        "\n",
        "These vectors encode some kind of semantics but they have some problems:\n",
        "\n",
        "+ The dimensions of the matrix can change very often (new words are added very frequently and corpus changes in size).\n",
        "+ SVD based methods do not scale well for big matrices and it is hard to incorporate new words or documents. \n",
        "+ The matrix is extremely sparse since most words do not co-occur.\n",
        "+ The matrix is very high dimensional in general ($\\approx 10^6 \\times 10^6$)\n",
        "+ Quadratic cost to train (i.e. to perform SVD)\n",
        "+ Requires the incorporation of some hacks on $X$ to account for the drastic imbalance in word frequency\n",
        "\n",
        "Some solutions to exist to resolve some of the issues discussed above:\n",
        "+ Ignore function words such as \"the\", \"he\", \"has\", etc.\n",
        "+ Apply a ramp window -- i.e. weight the co-occurrence count based on distance between the words in the document. \n",
        "+ Use Pearson correlation and set negative counts to 0 instead of using just raw count.\n",
        "\n",
        "But a NN method can solve many of these issues in a far more elegant manner....\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Mco2vL8LoWtw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## ``word2vec``\n",
        "\n",
        "Instead of computing and storing global information about some huge dataset (which might be billions of sentences), we can try to create a model that will be able to learn one iteration at a time and eventually be able **to encode the probability of a word given its context** (or, alternatively, the probability of the context given a word). \n",
        "\n",
        "> The **context of a word** is the set of $m$ surrounding words. For instance, the $m = 2$ context of the word ``fox`` in the sentence ``The quick brown fox jumped over the lazy dog`` is \\{``quick``, ``brown``, ``jumped``, ``over``\\}.\n",
        "\n",
        "The idea is to design a model whose parameters are the word vectors. Then, train the model on a certain objective related to representing the probability model. \n",
        "\n",
        "At every iteration we run our model, evaluate the errors, and follow an update rule that has some notion of penalizing the model parameters that caused the error. Thus, we learn our word vectors. \n",
        "\n",
        "Mikolov presented a simple, probabilistic model in 2013 that is known as ``word2vec``. In fact, ``word2vec`` includes 2 algorithms (**CBOW** and **skip-gram**) and 2 training methods (negative sampling and hierarchical softmax).\n",
        "\n",
        "> This model relies on a very important hypothesis in linguistics, *distributional semantics*. The basic idea of distributional semantics can be summed up in the so-called distributional hypothesis: linguistic items with similar distributions have similar meanings.\n",
        "\n",
        "> The distributional hypothesis can be applied to other data than words: items in shopping baskets, neural activations in wet neural networks, etc."
      ]
    },
    {
      "metadata": {
        "id": "Ukv5D1sJoWtx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "First, we need to create such a model that will assign a probability to a sequence of tokens. Let us start with an example: ``The cat jumped over the puddle``. \n",
        "\n",
        "A **good language model will give this sentence a high probability because this is a completely valid sentence**, syntactically and semantically. Similarly, the sentence ``stock boil fish is toy`` should have a very low probability because it makes no sense. \n",
        "\n",
        "Mathematically, we can call this probability on any given sequence of $n$ words:\n",
        "\n",
        "$$P(w_{1}, w_{2}, \\cdots, w_{n})$$\n",
        "\n",
        "### Language Models\n",
        "\n",
        "We know that \n",
        "\n",
        "$$P(w_{1}, w_{2}, \\cdots, w_{n}) = P(w_{1}) P(w_{2} | w_{1}) \\dots P(w_{n} | w_{1}, w_{2}, \\cdots, w_{n-1})$$\n",
        "\n",
        "but we alse know that we cannot compute this terms from a corpus by **counting**. All we can do if to approximate it.\n",
        "\n",
        "We can take the **unary language model** approach and break apart this probability by assuming the word occurrences are completely independent:\n",
        "\n",
        "$$P(w_{1}, w_{2}, \\cdots, w_{n}) \\approx \\prod_{i=1}^n P(w_{i})$$\n"
      ]
    },
    {
      "metadata": {
        "id": "PLsBYvWhIt67",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "However, we know the next word is highly contingent upon the previous sequence of words. So perhaps we let the probability of the sequence depend on the pairwise probability of a word in the sequence and the word next to it. We call this the **bigram model** and represent it as:\n",
        "\n",
        "$$P(w_{1}, w_{2}, \\cdots, w_{n}) \\approx \\prod_{i=2}^n P(w_{i} | w_{i-1})$$\n",
        "\n",
        "Again this is certainly a bit naive since we are only concerning ourselves with pairs of neighboring words rather than evaluating a whole sentence, but as we will see, this representation gets us pretty far along. Note in the Word-Word Matrix with a context of size 1, we basically can learn these pairwise probabilities. But again, this would require computing and storing global information about a massive dataset."
      ]
    },
    {
      "metadata": {
        "id": "TzRNwdZ0oWtx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Skip-gram model\n",
        "\n",
        "The skip-gram approach is to create a model such that given the center word ``jumped``, the model will be able to predict or generate the surrounding words ``The``, ``cat``, ``over``, ``the``, ``puddle``. Here we call the word ``jumped`` the context. \n",
        "\n",
        "How can we learn this model? Well, we need to create an **objective function**. \n",
        "\n",
        "Let's suppose we have a text composed of $T$ words. In this case, for each position $t = 1, … , T$, our task is to predict context words within a window of fixed size $m$, given\tcenter word $w_t$:\n",
        "\n",
        "$$\n",
        "L = \\prod^T_{t=1} \\prod_{-m \\leq j \\leq m ; j \\neq 0} P(w_{t+j} | w_t)\n",
        "$$\n",
        "\n",
        "The objective function is the average negative log likelihood:\n",
        "\n",
        "$$\n",
        "J =  - \\frac{1}{T} \\sum^T_{t=1} \\sum_{-m \\leq j \\leq m ; j \\neq 0} \\log P(w_{t+j} | w_t)\n",
        "$$\n",
        "\n",
        "How to calculate $P(w_{t+j} | w_t)$?\n",
        "\n",
        "**We will use a model that uses two vectors per word**. \n",
        "\n",
        "We create two matrices, $\\mathcal{V} \\in \\mathbb{R}^{n\\times|V|}$ and $\\mathcal{U} \\in \\mathbb{R}^{|V|\\times n}$, where $n$ is an arbitrary size which defines the size of our embedding space. \n",
        "\n",
        "$\\mathcal{V}$ is the input word matrix such that the $i$-th column of $\\mathcal{V}$ is the $n$-dimensional **embedded vector** for word $w_{i}$ when it is an input to this model. We denote this $n\\times1$ vector as $v_{i}$. \n",
        "\n",
        "Similarly, $\\mathcal{U}$ is the output word matrix. The $j$-th row of $\\mathcal{U}$ is an $n$-dimensional embedded vector for word $w_{j}$ when it is an output of the model. We denote this row of $\\mathcal{U}$ as $u_{j}$. \n",
        "\n",
        "Then, for a center word $c$ and a context word $o$, we assume the following model:\n",
        "\n",
        "$$\n",
        "P(o | c) = \\frac{\\exp(u_o^T v_c)}{\\sum_{w \\in V} \\exp(u_w^T v_c)}\n",
        "$$\n",
        "\n",
        "where $u_i = \\mathcal{U} w_i$, $v_i = \\mathcal{V} w_i$, and $w_i$ is the one hot encoding of a word $i$. Thus, our objective function is:\n",
        "\n",
        "$$\n",
        "J = - \\frac{1}{T} \\sum^T_{c=1} \\sum_{-m \\leq j \\leq m ; j \\neq 0} \\log \\frac{\\exp(u_{c+j}^T v_c)}{\\sum_{w \\in V} \\exp(u_w^T v_c)} = \\frac{1}{T} \\sum^T_{c=1} \\left(\\sum_{-m \\leq j \\leq m ; j \\neq 0} - (u_{c+j}^T v_c) + 2m \\log \\sum_{w \\in V} \\exp(u_w^T v_c) \\right)\n",
        "$$\n",
        "\n",
        "It is important to note that the second term involves a large number of vector products!\n",
        "\n",
        "The model works in 6 steps:\n",
        "\n",
        "+ We generate our one hot vector for the input context, $w_c$.\n",
        "+ We get our embedded word vector from the context $v_c = \\mathcal{V} w_c$.\n",
        "+ We generate $2m$ score vectors $u_o^T v_c $, where  $ u_o = \\mathcal{U} w_o$.\n",
        "+ Turn each of these scores into probabilities (which involves a large number of vector products). \n",
        "+ Our objective is to match these $2m$ probability vectors to the one hot vectors of the actual input.\n",
        "\n",
        "This is the graphical representation of our model ($ W $ encodes $ \\mathcal{V}$ and $ W'$ encodes $ \\mathcal{U}$). The left part involves only one vector multiplications, but the right one is much heavier!\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/fword2vec-sg.png?raw=1\" alt=\"\" style=\"width: 400px;\"/> \n",
        "</center>\n",
        "\n",
        "The computational complexity of this algorithm computed in a straightforward fashion is the size of our vocabulary, $O(V)$. This is because of the term $\\sum_{w \\in V} \\exp(u_w^T v_c)$. This denominator computes the similarity of all possible contexts $u_w$ and the target word $v_c$. \n",
        "\n",
        "### Negative sampling\n",
        "\n",
        "Loss functions $ J $ is expensive to compute because of the softmax normalization, where we sum over all $ |V| $ scores! A simple idea is we could instead just approximate it.\n",
        "\n",
        "While negative sampling is based on the Skip-Gram model, it is in fact optimizing a different objective. \n",
        "\n",
        "Consider a pair $(w, c)$ of word and context. Did this pair come from the training data? Let's denote by $P(D = 1|w, c)$ the probability that (w, c) came from the corpus data. Correspondingly, $P(D = 0|w, c)$ will be the probability that $(w, c)$ did not come from the corpus data. \n",
        "\n",
        "First, let's model $P(D = 1|w, c)$ with the sigmoid function:\n",
        "\n",
        "$$ P(D = 1|w, c, \\theta) = \\sigma (v_c^T v_w) = \\frac{1}{1+ e^{(-v_c^Tv_w)}}$$\n",
        "Now, we build a new objective function that tries to maximize the probability of a word and context being in the corpus data if it indeed is, and maximize the probability of a word and context not being in the corpus data if it indeed is not. We take a simple maximum likelihood approach of these two probabilities. (Here we take $\\theta$ to be the parameters of the model, and in our case it is $\\mathcal{V}$ and $\\mathcal{U}$.)\n",
        "\\begin{align*}\n",
        "\\theta &= \\mbox{argmax}_{\\theta} \\prod_{(w,c) \\in D} P(D = 1|w, c, \\theta) \\prod_{(w,c) \\in \\tilde{D}} P(D = 0|w, c, \\theta) \\\\\n",
        "&= \\mbox{argmax}_{\\theta} \\prod_{(w,c) \\in D} P(D = 1|w, c, \\theta) \\prod_{(w,c) \\in \\tilde{D}} (1 - P(D = 1|w, c, \\theta))\\\\\n",
        "&= \\mbox{argmax}_{\\theta} \\sum_{(w,c) \\in D} \\log P(D = 1|w, c, \\theta) + \\sum_{(w,c) \\in \\tilde{D}} \\log(1 - P(D = 1|w, c, \\theta))\\\\\n",
        "&= \\mbox{argmax}_{\\theta} \\sum_{(w,c) \\in D} \\log \\frac{1}{1 + \\exp(-u_w^Tv_c)} + \\sum_{(w,c) \\in \\tilde{D}} \\log(1 - \\frac{1}{1 + \\exp(-u_w^Tv_c)} )\\\\\n",
        "&= \\mbox{argmax}_{\\theta} \\sum_{(w,c) \\in D} \\log \\frac{1}{1 + \\exp(-u_w^Tv_c)} + \\sum_{(w,c) \\in \\tilde{D}} \\log(\\frac{1}{1 + \\exp(u_w^Tv_c)} )\\\\\n",
        "\\end{align*}\n",
        "Note that maximizing the likelihood is the same as minimizing the negative log likelihood\n",
        "\n",
        "$$\n",
        "J = - \\sum_{(w,c) \\in D} \\log \\frac{1}{1 + \\exp(-u_w^Tv_c)} - \\sum_{(w,c) \\in \\tilde{D}} \\log(\\frac{1}{1 + \\exp(u_w^Tv_c)} )\n",
        "$$\n",
        "\n",
        "Note that $\\tilde{D}$ is a \"false\" or \"negative\" corpus.\n",
        "\n",
        "For skip-gram, our new objective function for observing the context word $ c-m+j$ given the center word $ c $ would be\n",
        "\n",
        "\n",
        "$$ - \\log \\sigma (u_{c-m+j}^{T}\\cdot v_{c}) -\\sum_{k = 1}^K \\log \\sigma (- \\tilde{u}_{k}^{T}\\cdot v_{c}) $$\n",
        "\n",
        "\n",
        "In the above formulation, $\\{\\tilde{u}_{k} | k = 1\\dots K\\}$ are sampled from $P_n(w)$, the unigram distribution. Let's discuss what $P_n(w)$ should be. While there is much discussion of what makes the best approximation, what seems to work best is the Unigram Model raised to the power of 3/4. Why 3/4? Here's an example that might help gain some intuition:\n",
        "\n",
        "+ ``is``: $0.9^{3/4} = 0.92$\n",
        "+ ``Constitution``: $0.09^{3/4} = 0.16$\n",
        "+ ``bombastic``: $0.01^{3/4} = 0.032$\n",
        "\n",
        "``Bombastic`` is now 3x more likely to be sampled while ``is`` only went up marginally.\n"
      ]
    },
    {
      "metadata": {
        "id": "k774UXsooWty",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Continuous Bag of Words Model (CBOW)\n",
        "\n",
        "The CBOW approach is to treat {``The``, ``cat``, ``over``, ``the``, ``puddle``} as a context and from these words, be able to predict or generate the center word ``jumped``:\n",
        "\n",
        "+ We generate our one hot vectors for the input context ($w_j, \\forall j$).\n",
        "+ We get our word vectors for the input context ($v_j = \\mathcal{V} w_j, \\forall j$).\n",
        "+ Average these vectors to get a unique vector ($\\hat v = \\frac{1}{2m} \\sum_j v_j$).\n",
        "+ Get the score vector ($z_i = \\mathcal{U} \\hat v$).\n",
        "+ Turn the score into probabilities.\n",
        "+ Our objective is to match this probability vector to the one hot vector of the actual word.\n",
        "\n",
        "Our objective function is:\n",
        "\n",
        "$$\n",
        "J =  \\frac{1}{T} \\sum^T_{c=1} \\left(- (u_{c}^T \\hat v) +  \\log \\sum_{w \\in V} \\exp(u_{w}^T \\hat v) \\right)\n",
        "$$\n",
        "\n",
        "and the graphical representation of the model:\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/word2vec-cbow.png?raw=1\" alt=\"\" style=\"width: 400px;\"/> \n",
        "</center>"
      ]
    },
    {
      "metadata": {
        "id": "1VT7lOWnoWty",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## A ``word2vec`` implementation in Keras\n",
        "\n",
        "A word embedding layer is usually regarded as a mapping from a discrete set of objects (words) to a real valued vector, i.e. \n",
        "\n",
        "$$k\\in\\{1..|V|\\} \\rightarrow \\mathbb{R}^{n}$$\n",
        "\n",
        "Thus, we can represent the *Embedding layer* as $|V|\\times n$ matrix, or just a table/dictionary.\n",
        "\n",
        "$$\n",
        "\\begin{matrix}\n",
        "word_1: \\\\\n",
        "word_2:\\\\\n",
        "\\vdots\\\\\n",
        "word_{|V|}: \\\\\n",
        "\\end{matrix}\n",
        "\\left[\n",
        "\\begin{matrix}\n",
        "x_{1,1}&x_{1,2}& \\dots &x_{1,n}\\\\\n",
        "x_{2,1}&x_{2,2}& \\dots &x_{2,n}\\\\\n",
        "\\vdots&&\\\\\n",
        "x_{{|V|},1}&x_{{|V|},2}& \\dots &x_{{|V|},n}\\\\\n",
        "\\end{matrix}\n",
        "\\right]\n",
        "$$\n",
        "\n",
        "In this sense, the basic operation that an embedding layer has to accomplish is that given a certain word it returns the assigned code. And the goal in learning is to learn the values in the matrix.\n",
        "\n",
        "\n",
        "To train our data set using negative sampling and the skip-gram method, we need to create data samples for both valid context words and for negative samples. \n",
        "\n",
        "This involves scanning through the data set and picking target words, then randomly selecting context words from within the window of words around the target word (i.e. if the target word is “on” from “the cat sat on the mat”, with a window size of 2 the words “cat”, “sat”, “the”, “mat” could all be randomly selected as valid context words).  \n",
        "\n",
        "It also involves randomly selecting negative samples outside of the selected target word context. \n",
        "\n",
        "Finally, we also need to set a label of 1 or 0, depending on whether the supplied context word is a true context word or a negative sample.  \n",
        "\n",
        "Thankfully, Keras has a function (``skipgrams``) which does all that for us."
      ]
    },
    {
      "metadata": {
        "id": "XnViLO1voWtz",
        "colab_type": "code",
        "outputId": "b03fc9b2-be61-4c4e-9ca2-7455856709db",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        }
      },
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.layers import Input, Dense, Reshape, dot\n",
        "from tensorflow.keras.layers import Embedding\n",
        "from tensorflow.keras.preprocessing.sequence import skipgrams\n",
        "from tensorflow.keras.preprocessing import sequence\n",
        "\n",
        "import urllib.request\n",
        "import collections\n",
        "import os\n",
        "import zipfile\n",
        "\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "def maybe_download(filename, url, expected_bytes):\n",
        "    \"\"\"Download a file if not present, and make sure it's the right size.\"\"\"\n",
        "    if not os.path.exists(filename):\n",
        "        filename, _ = urllib.request.urlretrieve(url + filename, filename)\n",
        "    statinfo = os.stat(filename)\n",
        "    if statinfo.st_size == expected_bytes:\n",
        "        print('Found and verified', filename)\n",
        "    else:\n",
        "        print(statinfo.st_size)\n",
        "        raise Exception(\n",
        "            'Failed to verify ' + filename + '. Can you get to it with a browser?')\n",
        "    return filename\n",
        "\n",
        "\n",
        "# Read the data into a list of strings.\n",
        "def read_data(filename):\n",
        "    \"\"\"Extract the first file enclosed in a zip file as a list of words.\"\"\"\n",
        "    with zipfile.ZipFile(filename) as f:\n",
        "        data = tf.compat.as_str(f.read(f.namelist()[0])).split()\n",
        "    return data\n",
        "\n",
        "\n",
        "def build_dataset(words, n_words):\n",
        "    \"\"\"Process raw inputs into a dataset.\"\"\"\n",
        "    count = [['UNK', -1]]\n",
        "    count.extend(collections.Counter(words).most_common(n_words - 1))\n",
        "    dictionary = dict()\n",
        "    for word, _ in count:\n",
        "        dictionary[word] = len(dictionary)\n",
        "    data = list()\n",
        "    unk_count = 0\n",
        "    for word in words:\n",
        "        if word in dictionary:\n",
        "            index = dictionary[word]\n",
        "        else:\n",
        "            index = 0  # dictionary['UNK']\n",
        "            unk_count += 1\n",
        "        data.append(index)\n",
        "    count[0][1] = unk_count\n",
        "    reversed_dictionary = dict(zip(dictionary.values(), dictionary.keys()))\n",
        "    return data, count, dictionary, reversed_dictionary\n",
        "\n",
        "def collect_data(vocabulary_size=10000):\n",
        "    url = 'http://mattmahoney.net/dc/'\n",
        "    filename = maybe_download('text8.zip', url, 31344016)\n",
        "    vocabulary = read_data(filename)\n",
        "    print('First words of the dataset:',vocabulary[:7])\n",
        "    data, count, dictionary, reverse_dictionary = build_dataset(vocabulary,\n",
        "                                                                vocabulary_size)\n",
        "    del vocabulary  # Hint to reduce memory.\n",
        "    return data, count, dictionary, reverse_dictionary\n",
        "\n",
        "vocab_size = 10000\n",
        "data, count, dictionary, reverse_dictionary = collect_data(vocabulary_size=vocab_size)\n",
        "print('First words representation:',data[:7])\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found and verified text8.zip\n",
            "First words of the dataset: ['anarchism', 'originated', 'as', 'a', 'term', 'of', 'abuse']\n",
            "First words representation: [5234, 3081, 12, 6, 195, 2, 3134]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "-oA_-AWu57ud",
        "colab_type": "code",
        "outputId": "ef145c63-6f34-4c99-d23f-427512d2905c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "window_size = 3\n",
        "vector_dim = 300\n",
        "epochs = 200000\n",
        "\n",
        "valid_size = 16     # Random set of words to evaluate similarity on.\n",
        "valid_window = 100  \n",
        "valid_examples = np.random.choice(valid_window, valid_size, replace=False)\n",
        "\n",
        "# Generates a word rank-based probabilistic sampling table.\n",
        "sampling_table = sequence.make_sampling_table(vocab_size)\n",
        "\n",
        "# Generates skipgram word pairs.\n",
        "# This function transforms a sequence of word indexes \n",
        "# (list of integers) into tuples of words of the form:\n",
        "# (word, word in the same window), with label 1 (positive samples).\n",
        "# (word, random word from the vocabulary), with label 0 (negative samples).\n",
        "couples, labels = skipgrams(data, vocab_size, window_size=window_size, sampling_table=sampling_table)\n",
        "\n",
        "word_target, word_context = zip(*couples)\n",
        "word_target = np.array(word_target, dtype=\"int32\")\n",
        "word_context = np.array(word_context, dtype=\"int32\")\n",
        "\n",
        "print(couples[:10], labels[:10])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[1207, 139], [7, 267], [876, 4029], [1991, 7486], [406, 418], [4071, 8], [9821, 1130], [3187, 2], [780, 5], [470, 6]] [1, 1, 0, 0, 0, 1, 1, 1, 1, 1]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Wy2tYyEL4ft9",
        "colab_type": "code",
        "outputId": "c9bf64a0-2830-4fe2-9475-835381c4a908",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 9675
        }
      },
      "cell_type": "code",
      "source": [
        "# create some input variables\n",
        "input_target = Input((1,))\n",
        "input_context = Input((1,))\n",
        "\n",
        "embedding = Embedding(vocab_size, vector_dim, input_length=1, name='embedding')\n",
        "target = embedding(input_target)\n",
        "target = Reshape((vector_dim, 1))(target)\n",
        "\n",
        "context = embedding(input_context)\n",
        "context = Reshape((vector_dim, 1))(context)\n",
        "\n",
        "# setup a cosine similarity operation which will be output in a secondary model\n",
        "similarity = dot([target, context], normalize=True, axes=0)\n",
        "\n",
        "# now perform the dot product operation to get a similarity measure\n",
        "dot_product = dot([target, context], axes=1)\n",
        "dot_product = Reshape((1,))(dot_product)\n",
        "\n",
        "# add the sigmoid output layer\n",
        "output = Dense(1, activation='sigmoid')(dot_product)\n",
        "\n",
        "# create the primary training model\n",
        "model = Model(inputs=[input_target, input_context], outputs=output)\n",
        "model.compile(loss='binary_crossentropy', optimizer='rmsprop')\n",
        "\n",
        "# create a secondary validation model to run our similarity checks during training\n",
        "validation_model = Model(inputs=[input_target, input_context], outputs=similarity)\n",
        "\n",
        "class SimilarityCallback:\n",
        "    def run_sim(self):\n",
        "        for i in range(valid_size):\n",
        "            valid_word = reverse_dictionary[valid_examples[i]]\n",
        "            top_k = 8  # number of nearest neighbors\n",
        "            sim = self._get_sim(valid_examples[i])\n",
        "            nearest = (-sim).argsort()[1:top_k + 1]\n",
        "            log_str = 'Nearest to %s:' % valid_word\n",
        "            for k in range(top_k):\n",
        "                close_word = reverse_dictionary[nearest[k]]\n",
        "                log_str = '%s %s,' % (log_str, close_word)\n",
        "            print(log_str)\n",
        "\n",
        "    @staticmethod\n",
        "    def _get_sim(valid_word_idx):\n",
        "        sim = np.zeros((vocab_size,))\n",
        "        in_arr1 = np.zeros((1,))\n",
        "        in_arr2 = np.zeros((1,))\n",
        "        in_arr1[0,] = valid_word_idx\n",
        "        for i in range(vocab_size):\n",
        "            in_arr2[0,] = i\n",
        "            out = validation_model.predict_on_batch([in_arr1, in_arr2])\n",
        "            sim[i] = out\n",
        "        return sim\n",
        "      \n",
        "sim_cb = SimilarityCallback()\n",
        "\n",
        "arr_1 = np.zeros((1,))\n",
        "arr_2 = np.zeros((1,))\n",
        "arr_3 = np.zeros((1,))\n",
        "for cnt in range(epochs):\n",
        "    idx = np.random.randint(0, len(labels)-1)\n",
        "    arr_1[0,] = word_target[idx]\n",
        "    arr_2[0,] = word_context[idx]\n",
        "    arr_3[0,] = labels[idx]\n",
        "    loss = model.train_on_batch([arr_1, arr_2], arr_3)\n",
        "    if cnt % 100 == 0:\n",
        "        print(\"Iteration {}, loss={}\".format(cnt, loss))\n",
        "    if cnt % 10000 == 0:\n",
        "        sim_cb.run_sim()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "Iteration 0, loss=0.6924847364425659\n",
            "Nearest to the: exotic, accidental, website, gettysburg, outbreak, emotions, monkey, strait,\n",
            "Nearest to has: bias, yale, joke, surveillance, occasions, acknowledge, exponential, tips,\n",
            "Nearest to three: court, batman, sixty, failure, leadership, accused, montgomery, riemann,\n",
            "Nearest to if: chapters, sky, jargon, nominee, residence, singers, baseball, bullet,\n",
            "Nearest to their: came, queen, cuisine, jacob, sixth, believers, infant, sean,\n",
            "Nearest to with: naming, practically, metal, avoided, guild, storyline, fingers, her,\n",
            "Nearest to d: explicit, register, guiana, scale, zeus, ptolemy, honduras, census,\n",
            "Nearest to are: outstanding, bright, dog, luthor, anime, because, layers, exhibits,\n",
            "Nearest to will: peter, co, moderate, living, stake, morphology, journalism, walking,\n",
            "Nearest to during: marvel, interesting, gdp, agave, do, centred, mountains, authority,\n",
            "Nearest to states: influenced, mrna, variant, alone, thinking, fifth, men, contributors,\n",
            "Nearest to for: corresponds, submit, hop, hussein, habit, constants, beckham, uses,\n",
            "Nearest to have: beatles, parties, what, day, compilation, edges, allied, density,\n",
            "Nearest to of: wider, drops, carroll, ferry, usaf, rounds, reached, druze,\n",
            "Nearest to by: mongolia, protect, hunting, techniques, fungi, marriages, navigation, shifted,\n",
            "Nearest to up: polk, wave, frankish, anti, exploring, postal, publishes, predominantly,\n",
            "Iteration 100, loss=0.6833710074424744\n",
            "Iteration 200, loss=0.6806989312171936\n",
            "Iteration 300, loss=0.6918450593948364\n",
            "Iteration 400, loss=0.6879776120185852\n",
            "Iteration 500, loss=0.6965792179107666\n",
            "Iteration 600, loss=0.7006227970123291\n",
            "Iteration 700, loss=0.6955129504203796\n",
            "Iteration 800, loss=0.6978708505630493\n",
            "Iteration 900, loss=0.6931575536727905\n",
            "Iteration 1000, loss=0.689212441444397\n",
            "Iteration 1100, loss=0.6856670379638672\n",
            "Iteration 1200, loss=0.6873395442962646\n",
            "Iteration 1300, loss=0.7051771283149719\n",
            "Iteration 1400, loss=0.7074903249740601\n",
            "Iteration 1500, loss=0.7098552584648132\n",
            "Iteration 1600, loss=0.712811291217804\n",
            "Iteration 1700, loss=0.7150670886039734\n",
            "Iteration 1800, loss=0.7077709436416626\n",
            "Iteration 1900, loss=0.7209975719451904\n",
            "Iteration 2000, loss=0.6620565056800842\n",
            "Iteration 2100, loss=0.6743097901344299\n",
            "Iteration 2200, loss=0.67574143409729\n",
            "Iteration 2300, loss=0.7137861847877502\n",
            "Iteration 2400, loss=0.6780238151550293\n",
            "Iteration 2500, loss=0.6644510626792908\n",
            "Iteration 2600, loss=0.7138492465019226\n",
            "Iteration 2700, loss=0.6671689748764038\n",
            "Iteration 2800, loss=0.6767339706420898\n",
            "Iteration 2900, loss=0.6705089807510376\n",
            "Iteration 3000, loss=0.716980516910553\n",
            "Iteration 3100, loss=0.6728495359420776\n",
            "Iteration 3200, loss=0.6720362305641174\n",
            "Iteration 3300, loss=0.7078726291656494\n",
            "Iteration 3400, loss=0.7070281505584717\n",
            "Iteration 3500, loss=0.7063913941383362\n",
            "Iteration 3600, loss=0.6978500485420227\n",
            "Iteration 3700, loss=0.7223390340805054\n",
            "Iteration 3800, loss=0.709594190120697\n",
            "Iteration 3900, loss=0.6938915252685547\n",
            "Iteration 4000, loss=0.6856236457824707\n",
            "Iteration 4100, loss=0.7005009055137634\n",
            "Iteration 4200, loss=0.6958315372467041\n",
            "Iteration 4300, loss=0.697701096534729\n",
            "Iteration 4400, loss=0.6950245499610901\n",
            "Iteration 4500, loss=0.6894142031669617\n",
            "Iteration 4600, loss=0.6993498802185059\n",
            "Iteration 4700, loss=0.701834499835968\n",
            "Iteration 4800, loss=0.696371853351593\n",
            "Iteration 4900, loss=0.6931904554367065\n",
            "Iteration 5000, loss=0.7066123485565186\n",
            "Iteration 5100, loss=0.6855080723762512\n",
            "Iteration 5200, loss=0.6890085339546204\n",
            "Iteration 5300, loss=0.6969581246376038\n",
            "Iteration 5400, loss=0.6895936727523804\n",
            "Iteration 5500, loss=0.6970634460449219\n",
            "Iteration 5600, loss=0.701576828956604\n",
            "Iteration 5700, loss=0.6918135285377502\n",
            "Iteration 5800, loss=0.6969614028930664\n",
            "Iteration 5900, loss=0.690156877040863\n",
            "Iteration 6000, loss=0.7061783075332642\n",
            "Iteration 6100, loss=0.7038400769233704\n",
            "Iteration 6200, loss=0.68365079164505\n",
            "Iteration 6300, loss=0.6814502477645874\n",
            "Iteration 6400, loss=0.6897324323654175\n",
            "Iteration 6500, loss=0.6985477209091187\n",
            "Iteration 6600, loss=0.6896430850028992\n",
            "Iteration 6700, loss=0.6976961493492126\n",
            "Iteration 6800, loss=0.690611720085144\n",
            "Iteration 6900, loss=0.691172182559967\n",
            "Iteration 7000, loss=0.7002450227737427\n",
            "Iteration 7100, loss=0.690337598323822\n",
            "Iteration 7200, loss=0.6834379434585571\n",
            "Iteration 7300, loss=0.6971532106399536\n",
            "Iteration 7400, loss=0.6938209533691406\n",
            "Iteration 7500, loss=0.6852649450302124\n",
            "Iteration 7600, loss=0.6912152171134949\n",
            "Iteration 7700, loss=0.6897150874137878\n",
            "Iteration 7800, loss=0.6797724962234497\n",
            "Iteration 7900, loss=0.7013716697692871\n",
            "Iteration 8000, loss=0.6895738244056702\n",
            "Iteration 8100, loss=0.6996318697929382\n",
            "Iteration 8200, loss=0.7033630013465881\n",
            "Iteration 8300, loss=0.692715048789978\n",
            "Iteration 8400, loss=0.7131690979003906\n",
            "Iteration 8500, loss=0.6840319037437439\n",
            "Iteration 8600, loss=0.6727493405342102\n",
            "Iteration 8700, loss=0.6912104487419128\n",
            "Iteration 8800, loss=0.6810553073883057\n",
            "Iteration 8900, loss=0.6878522634506226\n",
            "Iteration 9000, loss=0.6828323602676392\n",
            "Iteration 9100, loss=0.7232803106307983\n",
            "Iteration 9200, loss=0.6906563639640808\n",
            "Iteration 9300, loss=0.6897001266479492\n",
            "Iteration 9400, loss=0.6819965243339539\n",
            "Iteration 9500, loss=0.6975233554840088\n",
            "Iteration 9600, loss=0.6759443879127502\n",
            "Iteration 9700, loss=0.6775051951408386\n",
            "Iteration 9800, loss=0.687252402305603\n",
            "Iteration 9900, loss=0.7105591297149658\n",
            "Iteration 10000, loss=0.686861515045166\n",
            "Nearest to the: coverage, heavily, monkey, execution, labor, found, jeremy, southeastern,\n",
            "Nearest to has: tips, additionally, int, bias, emerged, semantics, mercy, fy,\n",
            "Nearest to three: more, band, deities, notice, author, received, followed, bahrain,\n",
            "Nearest to if: fail, arrested, emperor, public, criticism, nominee, wrestler, embedded,\n",
            "Nearest to their: basel, position, autonomy, cuisine, composer, journalism, frankenstein, believers,\n",
            "Nearest to with: based, jonah, england, discovered, naming, because, moisture, preserved,\n",
            "Nearest to d: eugenics, explicit, madison, deng, led, gang, amendment, malta,\n",
            "Nearest to are: because, older, arab, surrounded, proceeds, logical, praised, maize,\n",
            "Nearest to will: stake, witness, peter, references, strait, car, winters, hull,\n",
            "Nearest to during: agave, latest, grand, medieval, product, lex, capoeira, applications,\n",
            "Nearest to states: variant, ought, contributors, hindus, journey, dependency, apparatus, soviet,\n",
            "Nearest to for: suffix, accompanying, noble, kasparov, surname, services, sometimes, major,\n",
            "Nearest to have: viii, cinema, beatles, indo, embassy, prevention, specially, terms,\n",
            "Nearest to of: obligations, han, imaging, battalion, depth, ceremony, sizes, successor,\n",
            "Nearest to by: regulated, broad, fighters, record, touch, navigation, democracy, citing,\n",
            "Nearest to up: wave, polk, exploring, abba, postal, elder, genetically, archaic,\n",
            "Iteration 10100, loss=0.6912906765937805\n",
            "Iteration 10200, loss=0.6964226961135864\n",
            "Iteration 10300, loss=0.7002304792404175\n",
            "Iteration 10400, loss=0.706841230392456\n",
            "Iteration 10500, loss=0.7043596506118774\n",
            "Iteration 10600, loss=0.7015297412872314\n",
            "Iteration 10700, loss=0.69344162940979\n",
            "Iteration 10800, loss=0.6829563975334167\n",
            "Iteration 10900, loss=0.6916696429252625\n",
            "Iteration 11000, loss=0.6985136270523071\n",
            "Iteration 11100, loss=0.6733413934707642\n",
            "Iteration 11200, loss=0.7030799388885498\n",
            "Iteration 11300, loss=0.6832107901573181\n",
            "Iteration 11400, loss=0.677725076675415\n",
            "Iteration 11500, loss=0.690872311592102\n",
            "Iteration 11600, loss=0.7029371857643127\n",
            "Iteration 11700, loss=0.7002249360084534\n",
            "Iteration 11800, loss=0.6818138957023621\n",
            "Iteration 11900, loss=0.6776388883590698\n",
            "Iteration 12000, loss=0.7087478041648865\n",
            "Iteration 12100, loss=0.7233911752700806\n",
            "Iteration 12200, loss=0.6700525879859924\n",
            "Iteration 12300, loss=0.6759885549545288\n",
            "Iteration 12400, loss=0.6706156134605408\n",
            "Iteration 12500, loss=0.7190060019493103\n",
            "Iteration 12600, loss=0.6868448257446289\n",
            "Iteration 12700, loss=0.6954402923583984\n",
            "Iteration 12800, loss=0.6932697296142578\n",
            "Iteration 12900, loss=0.6947895884513855\n",
            "Iteration 13000, loss=0.6711024641990662\n",
            "Iteration 13100, loss=0.679045557975769\n",
            "Iteration 13200, loss=0.6784355640411377\n",
            "Iteration 13300, loss=0.7082040905952454\n",
            "Iteration 13400, loss=0.6999104022979736\n",
            "Iteration 13500, loss=0.6930408477783203\n",
            "Iteration 13600, loss=0.695401132106781\n",
            "Iteration 13700, loss=0.6822315454483032\n",
            "Iteration 13800, loss=0.7145271301269531\n",
            "Iteration 13900, loss=0.7033823728561401\n",
            "Iteration 14000, loss=0.6798856258392334\n",
            "Iteration 14100, loss=0.703245997428894\n",
            "Iteration 14200, loss=0.7004646062850952\n",
            "Iteration 14300, loss=0.6781977415084839\n",
            "Iteration 14400, loss=0.7049912214279175\n",
            "Iteration 14500, loss=0.7065854072570801\n",
            "Iteration 14600, loss=0.6999896168708801\n",
            "Iteration 14700, loss=0.7013773322105408\n",
            "Iteration 14800, loss=0.7113033533096313\n",
            "Iteration 14900, loss=0.7039580345153809\n",
            "Iteration 15000, loss=0.6808428168296814\n",
            "Iteration 15100, loss=0.6725041270256042\n",
            "Iteration 15200, loss=0.6824322938919067\n",
            "Iteration 15300, loss=0.6897695064544678\n",
            "Iteration 15400, loss=0.6961054801940918\n",
            "Iteration 15500, loss=0.6770743131637573\n",
            "Iteration 15600, loss=0.6977472901344299\n",
            "Iteration 15700, loss=0.6983062624931335\n",
            "Iteration 15800, loss=0.6997784376144409\n",
            "Iteration 15900, loss=0.7258049249649048\n",
            "Iteration 16000, loss=0.6924550533294678\n",
            "Iteration 16100, loss=0.6925626397132874\n",
            "Iteration 16200, loss=0.7032467126846313\n",
            "Iteration 16300, loss=0.7003669738769531\n",
            "Iteration 16400, loss=0.6939108967781067\n",
            "Iteration 16500, loss=0.6958789825439453\n",
            "Iteration 16600, loss=0.6937708854675293\n",
            "Iteration 16700, loss=0.6883823871612549\n",
            "Iteration 16800, loss=0.7061365842819214\n",
            "Iteration 16900, loss=0.7185702919960022\n",
            "Iteration 17000, loss=0.7142750024795532\n",
            "Iteration 17100, loss=0.6784815192222595\n",
            "Iteration 17200, loss=0.7016957402229309\n",
            "Iteration 17300, loss=0.6632252931594849\n",
            "Iteration 17400, loss=0.6615115404129028\n",
            "Iteration 17500, loss=0.6624191999435425\n",
            "Iteration 17600, loss=0.6745197772979736\n",
            "Iteration 17700, loss=0.7264634966850281\n",
            "Iteration 17800, loss=0.6754295229911804\n",
            "Iteration 17900, loss=0.6899930238723755\n",
            "Iteration 18000, loss=0.69704669713974\n",
            "Iteration 18100, loss=0.7054838538169861\n",
            "Iteration 18200, loss=0.720944881439209\n",
            "Iteration 18300, loss=0.6734867095947266\n",
            "Iteration 18400, loss=0.6866251826286316\n",
            "Iteration 18500, loss=0.666837215423584\n",
            "Iteration 18600, loss=0.6647413372993469\n",
            "Iteration 18700, loss=0.7198557257652283\n",
            "Iteration 18800, loss=0.6665199995040894\n",
            "Iteration 18900, loss=0.6553764343261719\n",
            "Iteration 19000, loss=0.7060015201568604\n",
            "Iteration 19100, loss=0.6993573307991028\n",
            "Iteration 19200, loss=0.6676745414733887\n",
            "Iteration 19300, loss=0.7062401175498962\n",
            "Iteration 19400, loss=0.7027454376220703\n",
            "Iteration 19500, loss=0.6781211495399475\n",
            "Iteration 19600, loss=0.6932406425476074\n",
            "Iteration 19700, loss=0.6928150057792664\n",
            "Iteration 19800, loss=0.6993774175643921\n",
            "Iteration 19900, loss=0.6839975118637085\n",
            "Iteration 20000, loss=0.7302452921867371\n",
            "Nearest to the: light, monument, southeastern, refusing, should, ireland, others, canterbury,\n",
            "Nearest to has: fights, wins, complications, which, networks, tips, friday, printing,\n",
            "Nearest to three: five, deities, level, citizen, construction, followed, organizational, leadership,\n",
            "Nearest to if: institutional, prolonged, indonesian, fired, criticism, reserves, history, stops,\n",
            "Nearest to their: adobe, exploitation, algorithms, journalism, came, xiii, autonomy, ambient,\n",
            "Nearest to with: contact, his, categories, synonymous, macedonian, mod, commonly, bolshevik,\n",
            "Nearest to d: fell, led, scale, recommendation, sterling, importance, january, considerably,\n",
            "Nearest to are: there, always, conscience, script, listed, levy, math, pushed,\n",
            "Nearest to will: harvard, sets, references, peter, jump, strait, reality, mind,\n",
            "Nearest to during: his, affinity, century, campaigns, agave, places, interesting, extreme,\n",
            "Nearest to states: hindus, contributors, united, apparatus, integers, skin, phrase, annexed,\n",
            "Nearest to for: windows, votes, florida, cpus, beneficial, distinct, transition, optics,\n",
            "Nearest to have: viii, technologies, beatles, venue, judges, contrast, kant, algol,\n",
            "Nearest to of: sicily, minded, immune, obligations, mathematics, chapter, provides, viewing,\n",
            "Nearest to by: designed, dropping, testing, martyr, speaker, mirror, payment, fighters,\n",
            "Nearest to up: wave, anti, exploring, genetically, gradual, angular, polk, jointly,\n",
            "Iteration 20100, loss=0.7151730060577393\n",
            "Iteration 20200, loss=0.696704626083374\n",
            "Iteration 20300, loss=0.6854209899902344\n",
            "Iteration 20400, loss=0.7063788175582886\n",
            "Iteration 20500, loss=0.6945846080780029\n",
            "Iteration 20600, loss=0.6835086345672607\n",
            "Iteration 20700, loss=0.690499484539032\n",
            "Iteration 20800, loss=0.7201815843582153\n",
            "Iteration 20900, loss=0.6466553211212158\n",
            "Iteration 21000, loss=0.7012763619422913\n",
            "Iteration 21100, loss=0.6904008388519287\n",
            "Iteration 21200, loss=0.6901326775550842\n",
            "Iteration 21300, loss=0.6573610901832581\n",
            "Iteration 21400, loss=0.6981637477874756\n",
            "Iteration 21500, loss=0.6942213773727417\n",
            "Iteration 21600, loss=0.7201545834541321\n",
            "Iteration 21700, loss=0.6672528982162476\n",
            "Iteration 21800, loss=0.660103976726532\n",
            "Iteration 21900, loss=0.7171911597251892\n",
            "Iteration 22000, loss=0.6697226762771606\n",
            "Iteration 22100, loss=0.6613172292709351\n",
            "Iteration 22200, loss=0.6947604417800903\n",
            "Iteration 22300, loss=0.6774910688400269\n",
            "Iteration 22400, loss=0.6999570727348328\n",
            "Iteration 22500, loss=0.6696962118148804\n",
            "Iteration 22600, loss=0.6719700694084167\n",
            "Iteration 22700, loss=0.6875044107437134\n",
            "Iteration 22800, loss=0.6609956622123718\n",
            "Iteration 22900, loss=0.7244068384170532\n",
            "Iteration 23000, loss=0.7268833518028259\n",
            "Iteration 23100, loss=0.7070605754852295\n",
            "Iteration 23200, loss=0.6986402869224548\n",
            "Iteration 23300, loss=0.7012649774551392\n",
            "Iteration 23400, loss=0.7002937197685242\n",
            "Iteration 23500, loss=0.6860553622245789\n",
            "Iteration 23600, loss=0.7115781307220459\n",
            "Iteration 23700, loss=0.6631603837013245\n",
            "Iteration 23800, loss=0.661288321018219\n",
            "Iteration 23900, loss=0.6677227020263672\n",
            "Iteration 24000, loss=0.6782281994819641\n",
            "Iteration 24100, loss=0.6678350567817688\n",
            "Iteration 24200, loss=0.7085492610931396\n",
            "Iteration 24300, loss=0.6822343468666077\n",
            "Iteration 24400, loss=0.7160108089447021\n",
            "Iteration 24500, loss=0.6667947173118591\n",
            "Iteration 24600, loss=0.7128536105155945\n",
            "Iteration 24700, loss=0.7192366719245911\n",
            "Iteration 24800, loss=0.6652548909187317\n",
            "Iteration 24900, loss=0.7200998067855835\n",
            "Iteration 25000, loss=0.6919861435890198\n",
            "Iteration 25100, loss=0.7182996869087219\n",
            "Iteration 25200, loss=0.6929192543029785\n",
            "Iteration 25300, loss=0.7112899422645569\n",
            "Iteration 25400, loss=0.7100747227668762\n",
            "Iteration 25500, loss=0.7216777801513672\n",
            "Iteration 25600, loss=0.6838999390602112\n",
            "Iteration 25700, loss=0.6945463418960571\n",
            "Iteration 25800, loss=0.6822761297225952\n",
            "Iteration 25900, loss=0.7027475237846375\n",
            "Iteration 26000, loss=0.6835027933120728\n",
            "Iteration 26100, loss=0.6861354112625122\n",
            "Iteration 26200, loss=0.6967182159423828\n",
            "Iteration 26300, loss=0.7161040902137756\n",
            "Iteration 26400, loss=0.7148347496986389\n",
            "Iteration 26500, loss=0.7086364030838013\n",
            "Iteration 26600, loss=0.6913193464279175\n",
            "Iteration 26700, loss=0.6967599987983704\n",
            "Iteration 26800, loss=0.6701978445053101\n",
            "Iteration 26900, loss=0.6872744560241699\n",
            "Iteration 27000, loss=0.7336145639419556\n",
            "Iteration 27100, loss=0.7171816229820251\n",
            "Iteration 27200, loss=0.690426230430603\n",
            "Iteration 27300, loss=0.6897093653678894\n",
            "Iteration 27400, loss=0.6943480968475342\n",
            "Iteration 27500, loss=0.6590710878372192\n",
            "Iteration 27600, loss=0.7099723815917969\n",
            "Iteration 27700, loss=0.6808221340179443\n",
            "Iteration 27800, loss=0.6726719737052917\n",
            "Iteration 27900, loss=0.6782954931259155\n",
            "Iteration 28000, loss=0.7310482859611511\n",
            "Iteration 28100, loss=0.6972809433937073\n",
            "Iteration 28200, loss=0.6956346035003662\n",
            "Iteration 28300, loss=0.7061941623687744\n",
            "Iteration 28400, loss=0.7176558971405029\n",
            "Iteration 28500, loss=0.6473872065544128\n",
            "Iteration 28600, loss=0.7093850374221802\n",
            "Iteration 28700, loss=0.6738762259483337\n",
            "Iteration 28800, loss=0.6169957518577576\n",
            "Iteration 28900, loss=0.6769801378250122\n",
            "Iteration 29000, loss=0.691576361656189\n",
            "Iteration 29100, loss=0.7216103076934814\n",
            "Iteration 29200, loss=0.6970075964927673\n",
            "Iteration 29300, loss=0.67978435754776\n",
            "Iteration 29400, loss=0.6760698556900024\n",
            "Iteration 29500, loss=0.668563723564148\n",
            "Iteration 29600, loss=0.6719030737876892\n",
            "Iteration 29700, loss=0.745377242565155\n",
            "Iteration 29800, loss=0.7021445631980896\n",
            "Iteration 29900, loss=0.7229529023170471\n",
            "Iteration 30000, loss=0.6836389899253845\n",
            "Nearest to the: of, action, code, light, by, location, in, comprise,\n",
            "Nearest to has: wins, boy, say, inputs, is, complications, which, robinson,\n",
            "Nearest to three: all, four, o, one, five, definite, inaugurated, zero,\n",
            "Nearest to if: shi, institutional, exceptions, prolonged, indonesian, fail, exploring, apache,\n",
            "Nearest to their: missed, temporary, journalism, view, them, orbitals, tell, cocoa,\n",
            "Nearest to with: contact, commonly, themselves, his, synonymous, categories, morris, proceed,\n",
            "Nearest to d: drafted, fell, drum, diagram, center, formal, east, madison,\n",
            "Nearest to are: conventions, off, there, trivial, muslim, contributions, these, bright,\n",
            "Nearest to will: harvard, stake, interstate, fields, cinema, co, to, manufactured,\n",
            "Nearest to during: his, par, century, agave, sand, died, legacy, circus,\n",
            "Nearest to states: united, contributors, wrestling, apparatus, integers, dependency, stated, confederate,\n",
            "Nearest to for: instance, windows, except, cpus, pleasant, private, solution, ark,\n",
            "Nearest to have: kant, paid, annotated, technologies, judges, ll, child, confusion,\n",
            "Nearest to of: the, ideas, sicily, combination, qur, brief, justice, tactics,\n",
            "Nearest to by: the, designed, inspired, techniques, mirror, occupy, rouge, excellent,\n",
            "Nearest to up: granite, substantial, polk, lucy, usually, year, wave, villages,\n",
            "Iteration 30100, loss=0.6957594752311707\n",
            "Iteration 30200, loss=0.6778017282485962\n",
            "Iteration 30300, loss=0.6868853569030762\n",
            "Iteration 30400, loss=0.6399481296539307\n",
            "Iteration 30500, loss=0.6826813817024231\n",
            "Iteration 30600, loss=0.6688591837882996\n",
            "Iteration 30700, loss=0.7187936902046204\n",
            "Iteration 30800, loss=0.6832696795463562\n",
            "Iteration 30900, loss=0.7207279205322266\n",
            "Iteration 31000, loss=0.6771658658981323\n",
            "Iteration 31100, loss=0.7293110489845276\n",
            "Iteration 31200, loss=0.7009219527244568\n",
            "Iteration 31300, loss=0.6917925477027893\n",
            "Iteration 31400, loss=0.6588281393051147\n",
            "Iteration 31500, loss=0.6507372260093689\n",
            "Iteration 31600, loss=0.7261417508125305\n",
            "Iteration 31700, loss=0.6492571234703064\n",
            "Iteration 31800, loss=0.679068922996521\n",
            "Iteration 31900, loss=0.630605161190033\n",
            "Iteration 32000, loss=0.7202935218811035\n",
            "Iteration 32100, loss=0.6737169623374939\n",
            "Iteration 32200, loss=0.6373607516288757\n",
            "Iteration 32300, loss=0.6747208833694458\n",
            "Iteration 32400, loss=0.6428685784339905\n",
            "Iteration 32500, loss=0.7393499612808228\n",
            "Iteration 32600, loss=0.6143278479576111\n",
            "Iteration 32700, loss=0.651159405708313\n",
            "Iteration 32800, loss=0.6520199775695801\n",
            "Iteration 32900, loss=0.6564723253250122\n",
            "Iteration 33000, loss=0.7493182420730591\n",
            "Iteration 33100, loss=0.6726227402687073\n",
            "Iteration 33200, loss=0.73833829164505\n",
            "Iteration 33300, loss=0.6735588312149048\n",
            "Iteration 33400, loss=0.6504902839660645\n",
            "Iteration 33500, loss=0.6707026362419128\n",
            "Iteration 33600, loss=0.7547575235366821\n",
            "Iteration 33700, loss=0.6821660399436951\n",
            "Iteration 33800, loss=0.8603982329368591\n",
            "Iteration 33900, loss=0.6889999508857727\n",
            "Iteration 34000, loss=0.6815826892852783\n",
            "Iteration 34100, loss=0.6955466270446777\n",
            "Iteration 34200, loss=0.6922047138214111\n",
            "Iteration 34300, loss=0.7116609215736389\n",
            "Iteration 34400, loss=0.7169146537780762\n",
            "Iteration 34500, loss=0.6491500735282898\n",
            "Iteration 34600, loss=0.6523797512054443\n",
            "Iteration 34700, loss=0.6240466237068176\n",
            "Iteration 34800, loss=0.6801379919052124\n",
            "Iteration 34900, loss=0.6313124895095825\n",
            "Iteration 35000, loss=0.7331181764602661\n",
            "Iteration 35100, loss=0.7057686448097229\n",
            "Iteration 35200, loss=0.7460929155349731\n",
            "Iteration 35300, loss=0.7282137870788574\n",
            "Iteration 35400, loss=0.7386757731437683\n",
            "Iteration 35500, loss=0.6793607473373413\n",
            "Iteration 35600, loss=0.7392839193344116\n",
            "Iteration 35700, loss=0.6938198804855347\n",
            "Iteration 35800, loss=0.25566959381103516\n",
            "Iteration 35900, loss=0.6601082682609558\n",
            "Iteration 36000, loss=0.753139853477478\n",
            "Iteration 36100, loss=0.7288905382156372\n",
            "Iteration 36200, loss=0.6937403082847595\n",
            "Iteration 36300, loss=0.7410568594932556\n",
            "Iteration 36400, loss=0.6979401111602783\n",
            "Iteration 36500, loss=0.739355206489563\n",
            "Iteration 36600, loss=0.6669212579727173\n",
            "Iteration 36700, loss=0.7195786833763123\n",
            "Iteration 36800, loss=0.6725819706916809\n",
            "Iteration 36900, loss=0.6787410378456116\n",
            "Iteration 37000, loss=0.8273199796676636\n",
            "Iteration 37100, loss=0.6843239665031433\n",
            "Iteration 37200, loss=0.7138371467590332\n",
            "Iteration 37300, loss=0.5951225757598877\n",
            "Iteration 37400, loss=0.6710744500160217\n",
            "Iteration 37500, loss=0.6582775712013245\n",
            "Iteration 37600, loss=0.6741175055503845\n",
            "Iteration 37700, loss=0.6873902082443237\n",
            "Iteration 37800, loss=0.6793870329856873\n",
            "Iteration 37900, loss=0.6382907032966614\n",
            "Iteration 38000, loss=0.6498401761054993\n",
            "Iteration 38100, loss=0.6448284387588501\n",
            "Iteration 38200, loss=0.6637110114097595\n",
            "Iteration 38300, loss=0.6988369822502136\n",
            "Iteration 38400, loss=0.7199987769126892\n",
            "Iteration 38500, loss=0.688991367816925\n",
            "Iteration 38600, loss=0.6606878042221069\n",
            "Iteration 38700, loss=0.688022792339325\n",
            "Iteration 38800, loss=0.7547260522842407\n",
            "Iteration 38900, loss=0.6143144369125366\n",
            "Iteration 39000, loss=0.6621866822242737\n",
            "Iteration 39100, loss=0.6678462028503418\n",
            "Iteration 39200, loss=0.7186973094940186\n",
            "Iteration 39300, loss=0.723068356513977\n",
            "Iteration 39400, loss=0.5873510837554932\n",
            "Iteration 39500, loss=0.7557520270347595\n",
            "Iteration 39600, loss=0.7006196975708008\n",
            "Iteration 39700, loss=0.7164231538772583\n",
            "Iteration 39800, loss=0.6962356567382812\n",
            "Iteration 39900, loss=0.7124812602996826\n",
            "Iteration 40000, loss=0.7526856660842896\n",
            "Nearest to the: of, in, front, southeastern, action, that, son, japan,\n",
            "Nearest to has: wins, involved, contested, spencer, apartment, probe, seized, complications,\n",
            "Nearest to three: one, five, four, six, km, nine, all, zero,\n",
            "Nearest to if: institutional, prolonged, apache, shi, beverages, exceptions, amount, then,\n",
            "Nearest to their: view, trouble, notation, jamaica, suddenly, ontario, jay, bah,\n",
            "Nearest to with: contact, his, mixture, commonly, the, of, orchestra, terrorists,\n",
            "Nearest to d: donald, drum, playoff, bernard, hold, written, zero, consist,\n",
            "Nearest to are: muslim, there, of, conventions, not, creatures, kinds, air,\n",
            "Nearest to will: believe, have, fields, am, manufactured, monitor, participating, stake,\n",
            "Nearest to during: ex, his, affinity, game, grand, to, agave, circus,\n",
            "Nearest to states: united, in, fill, contributors, soviet, targets, apparatus, photographs,\n",
            "Nearest to for: instance, except, collect, represented, the, homepage, functional, windows,\n",
            "Nearest to have: annotated, meaning, will, judges, technologies, giuseppe, venue, changed,\n",
            "Nearest to of: the, it, in, sicily, earth, that, combination, died,\n",
            "Nearest to by: government, the, licence, drake, inspired, except, of, founder,\n",
            "Nearest to up: granite, gradual, promoted, participate, transition, wave, finns, flight,\n",
            "Iteration 40100, loss=0.7016277313232422\n",
            "Iteration 40200, loss=0.7101706266403198\n",
            "Iteration 40300, loss=0.6878336071968079\n",
            "Iteration 40400, loss=0.7171863913536072\n",
            "Iteration 40500, loss=0.7062207460403442\n",
            "Iteration 40600, loss=0.703620433807373\n",
            "Iteration 40700, loss=0.6807808876037598\n",
            "Iteration 40800, loss=0.641755998134613\n",
            "Iteration 40900, loss=0.6783998608589172\n",
            "Iteration 41000, loss=0.6904529929161072\n",
            "Iteration 41100, loss=0.6974070072174072\n",
            "Iteration 41200, loss=0.6776502728462219\n",
            "Iteration 41300, loss=0.6701646447181702\n",
            "Iteration 41400, loss=0.6982722282409668\n",
            "Iteration 41500, loss=0.7087511420249939\n",
            "Iteration 41600, loss=0.7178574800491333\n",
            "Iteration 41700, loss=0.7169519066810608\n",
            "Iteration 41800, loss=0.7156929969787598\n",
            "Iteration 41900, loss=0.6999479532241821\n",
            "Iteration 42000, loss=0.6603257656097412\n",
            "Iteration 42100, loss=0.805125892162323\n",
            "Iteration 42200, loss=0.6949938535690308\n",
            "Iteration 42300, loss=0.6787785291671753\n",
            "Iteration 42400, loss=0.6962705850601196\n",
            "Iteration 42500, loss=0.7226024270057678\n",
            "Iteration 42600, loss=0.29052734375\n",
            "Iteration 42700, loss=0.692847728729248\n",
            "Iteration 42800, loss=0.7089745402336121\n",
            "Iteration 42900, loss=0.5790544748306274\n",
            "Iteration 43000, loss=0.7288342118263245\n",
            "Iteration 43100, loss=0.6516852378845215\n",
            "Iteration 43200, loss=0.6841632127761841\n",
            "Iteration 43300, loss=0.7205244898796082\n",
            "Iteration 43400, loss=0.6082156300544739\n",
            "Iteration 43500, loss=0.7095439434051514\n",
            "Iteration 43600, loss=0.6161795854568481\n",
            "Iteration 43700, loss=0.6467768549919128\n",
            "Iteration 43800, loss=0.650288999080658\n",
            "Iteration 43900, loss=0.6623265147209167\n",
            "Iteration 44000, loss=0.6782047152519226\n",
            "Iteration 44100, loss=0.7272722125053406\n",
            "Iteration 44200, loss=0.6998750567436218\n",
            "Iteration 44300, loss=0.6863601803779602\n",
            "Iteration 44400, loss=0.6446477174758911\n",
            "Iteration 44500, loss=0.6612274050712585\n",
            "Iteration 44600, loss=0.6620894074440002\n",
            "Iteration 44700, loss=0.7181470990180969\n",
            "Iteration 44800, loss=0.6040917038917542\n",
            "Iteration 44900, loss=0.4998635947704315\n",
            "Iteration 45000, loss=0.5556743144989014\n",
            "Iteration 45100, loss=0.6517425775527954\n",
            "Iteration 45200, loss=0.6888195872306824\n",
            "Iteration 45300, loss=0.6972845196723938\n",
            "Iteration 45400, loss=0.6488813757896423\n",
            "Iteration 45500, loss=0.6321799755096436\n",
            "Iteration 45600, loss=0.673970639705658\n",
            "Iteration 45700, loss=0.7262200713157654\n",
            "Iteration 45800, loss=0.7557867765426636\n",
            "Iteration 45900, loss=0.6747227311134338\n",
            "Iteration 46000, loss=0.7136977314949036\n",
            "Iteration 46100, loss=0.6802753210067749\n",
            "Iteration 46200, loss=0.7188102006912231\n",
            "Iteration 46300, loss=0.7410799860954285\n",
            "Iteration 46400, loss=0.6910861730575562\n",
            "Iteration 46500, loss=0.6594362854957581\n",
            "Iteration 46600, loss=0.6475749611854553\n",
            "Iteration 46700, loss=0.6083500385284424\n",
            "Iteration 46800, loss=0.6928054690361023\n",
            "Iteration 46900, loss=0.6731308102607727\n",
            "Iteration 47000, loss=0.7160362005233765\n",
            "Iteration 47100, loss=0.7069646716117859\n",
            "Iteration 47200, loss=0.7136169075965881\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "1YEHjTsCoWt7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## ``par2vec``\n",
        "\n",
        "What about a vector representation for phrases/paragraphs/documents?\n",
        "\n",
        "The ``par2vec`` approach for learning paragraph vectors is inspired by the methods for learning the word vectors. The inspiration is that the word vectors are asked to contribute to a prediction task about the next word in the sentence.\n",
        "\n",
        "We will consider a *paragraph* vector. The paragraph vectors are also\n",
        "asked to contribute to the prediction task of the next word\n",
        "given many contexts sampled from the paragraph.\n",
        "\n",
        "In ``par2vec`` framework, every paragraph is mapped to a unique vector, represented by a column in matrix D and every word is also mapped to a unique vector, represented by a column in matrix W. The paragraph vector and word vectors are averaged or concatenated to predict the next word in a context.\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/par2vec.png?raw=1\" alt=\"\" style=\"width: 500px;\"/> \n",
        "(Source: https://cs.stanford.edu/~quocle/paragraph_vector.pdf)\n",
        "</center>\n",
        "\n",
        "The paragraph token can be thought of as another word. It acts as a memory that remembers what is missing from the\n",
        "current context – or the topic of the paragraph. For this reason, we often call this model the Distributed Memory\n",
        "Model of Paragraph Vectors (PV-DM).\n",
        "\n",
        "The contexts are fixed-length and sampled from a sliding window over the paragraph. \n",
        "\n",
        "The paragraph vector is shared across all contexts generated from the same paragraph but not across paragraphs. \n",
        "\n",
        "The word vector matrix W, however, is shared across paragraphs. "
      ]
    },
    {
      "metadata": {
        "id": "mIMHfFKFoWt8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "At prediction time, one needs to perform an inference step to compute the paragraph vector for a new paragraph. This\n",
        "is also obtained by gradient descent. In this step, the parameters for the rest of the model, the word vectors and\n",
        "the softmax weights, are fixed."
      ]
    },
    {
      "metadata": {
        "id": "89gB95WdoWt8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Using GloVe pre-trained word embeddings for classification. 1-D Convolutions.\n",
        "\n",
        "GloVe (https://nlp.stanford.edu/pubs/glove.pdf) consists of a weighted least squares model that trains on global word-word co-occurrence counts and thus makes efficient use of statistics. The model produces a word vector space with meaningful sub-structure. It shows state-of-the-art performance on the word analogy task, and outperforms other current methods on several word similarity tasks.\n",
        "\n",
        "GloVe consists of a weighted least squares model that trains on global word-word co-occurrence counts and thus makes efficient use of statistics. The model produces a word vector space with meaningful sub-structure. It shows state-of-the-art performance on the word analogy task, and outperforms other current methods on several word similarity tasks.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "o3YVnEQKoWt9",
        "colab_type": "code",
        "outputId": "aaab88b1-b86e-4c3c-b37c-612b999b89d3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 253
        }
      },
      "cell_type": "code",
      "source": [
        "'''This script loads pre-trained word embeddings (GloVe embeddings)\n",
        "into a frozen Keras Embedding layer, and uses it to\n",
        "train a text classification model on the 20 Newsgroup dataset\n",
        "(classication of newsgroup messages into 20 different categories).\n",
        "GloVe embedding data can be found at:\n",
        "http://nlp.stanford.edu/data/glove.6B.zip (822MB)\n",
        "(source page: http://nlp.stanford.edu/projects/glove/)\n",
        "20 Newsgroup data can be found at:\n",
        "http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/news20.html\n",
        "'''\n",
        "\n",
        "from __future__ import print_function\n",
        "\n",
        "import os\n",
        "import sys\n",
        "import numpy as np\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from keras.layers import Dense, Input, Flatten\n",
        "from keras.layers import Conv1D, MaxPooling1D, Embedding\n",
        "from keras.models import Model\n",
        "\n",
        "def to_categorical(y, num_classes=None):\n",
        "    \"\"\"Converts a class vector (integers) to binary class matrix.\n",
        "    E.g. for use with categorical_crossentropy.\n",
        "    # Arguments\n",
        "        y: class vector to be converted into a matrix\n",
        "            (integers from 0 to num_classes).\n",
        "        num_classes: total number of classes.\n",
        "    # Returns\n",
        "        A binary matrix representation of the input.\n",
        "    \"\"\"\n",
        "    y = np.array(y, dtype='int').ravel()\n",
        "    if not num_classes:\n",
        "        num_classes = np.max(y) + 1\n",
        "    n = y.shape[0]\n",
        "    categorical = np.zeros((n, num_classes))\n",
        "    categorical[np.arange(n), y] = 1\n",
        "    return categorical\n",
        "\n",
        "BASE_DIR = ''\n",
        "GLOVE_DIR = BASE_DIR + '/glove.6B/'\n",
        "TEXT_DATA_DIR = BASE_DIR + '/20_newsgroup/'\n",
        "MAX_SEQUENCE_LENGTH = 1000\n",
        "MAX_NB_WORDS = 20000\n",
        "EMBEDDING_DIM = 100\n",
        "VALIDATION_SPLIT = 0.2\n",
        "\n",
        "# first, build index mapping words in the embeddings set\n",
        "# to their embedding vector\n",
        "\n",
        "print('Indexing word vectors.')\n",
        "\n",
        "embeddings_index = {}\n",
        "f = open(os.path.join(GLOVE_DIR, 'glove.6B.100d.txt'))\n",
        "for line in f:\n",
        "    values = line.split()\n",
        "    word = values[0]\n",
        "    coefs = np.asarray(values[1:], dtype='float32')\n",
        "    embeddings_index[word] = coefs\n",
        "f.close()\n",
        "\n",
        "print('Found %s word vectors.' % len(embeddings_index))\n",
        "\n",
        "# second, prepare text samples and their labels\n",
        "print('Processing text dataset')\n",
        "\n",
        "texts = []  # list of text samples\n",
        "labels_index = {}  # dictionary mapping label name to numeric id\n",
        "labels = []  # list of label ids\n",
        "for name in sorted(os.listdir(TEXT_DATA_DIR)):\n",
        "    path = os.path.join(TEXT_DATA_DIR, name)\n",
        "    if os.path.isdir(path):\n",
        "        label_id = len(labels_index)\n",
        "        labels_index[name] = label_id\n",
        "        for fname in sorted(os.listdir(path)):\n",
        "            if fname.isdigit():\n",
        "                fpath = os.path.join(path, fname)\n",
        "                if sys.version_info < (3,):\n",
        "                    f = open(fpath)\n",
        "                else:\n",
        "                    f = open(fpath, encoding='latin-1')\n",
        "                t = f.read()\n",
        "                i = t.find('\\n\\n')  # skip header\n",
        "                if 0 < i:\n",
        "                    t = t[i:]\n",
        "                texts.append(t)\n",
        "                f.close()\n",
        "                labels.append(label_id)\n",
        "\n",
        "print('Found %s texts.' % len(texts))\n",
        "\n",
        "# finally, vectorize the text samples into a 2D integer tensor\n",
        "tokenizer = Tokenizer(num_words=MAX_NB_WORDS)\n",
        "tokenizer.fit_on_texts(texts)\n",
        "sequences = tokenizer.texts_to_sequences(texts)\n",
        "\n",
        "word_index = tokenizer.word_index\n",
        "print('Found %s unique tokens.' % len(word_index))\n",
        "\n",
        "data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)\n",
        "\n",
        "labels = to_categorical(np.asarray(labels))\n",
        "print('Shape of data tensor:', data.shape)\n",
        "print('Shape of label tensor:', labels.shape)\n",
        "\n",
        "# split the data into a training set and a validation set\n",
        "indices = np.arange(data.shape[0])\n",
        "np.random.shuffle(indices)\n",
        "data = data[indices]\n",
        "labels = labels[indices]\n",
        "num_validation_samples = int(VALIDATION_SPLIT * data.shape[0])\n",
        "\n",
        "x_train = data[:-num_validation_samples]\n",
        "y_train = labels[:-num_validation_samples]\n",
        "x_val = data[-num_validation_samples:]\n",
        "y_val = labels[-num_validation_samples:]\n",
        "\n",
        "print('Preparing embedding matrix.')\n",
        "\n",
        "# prepare embedding matrix\n",
        "num_words = min(MAX_NB_WORDS, len(word_index))\n",
        "embedding_matrix = np.zeros((num_words, EMBEDDING_DIM))\n",
        "for word, i in word_index.items():\n",
        "    if i >= MAX_NB_WORDS:\n",
        "        continue\n",
        "    embedding_vector = embeddings_index.get(word)\n",
        "    if embedding_vector is not None:\n",
        "        # words not found in embedding index will be all-zeros.\n",
        "        embedding_matrix[i] = embedding_vector\n",
        "\n",
        "# load pre-trained word embeddings into an Embedding layer\n",
        "# note that we set trainable = False so as to keep the embeddings fixed\n",
        "embedding_layer = Embedding(num_words,\n",
        "                            EMBEDDING_DIM,\n",
        "                            weights=[embedding_matrix],\n",
        "                            input_length=MAX_SEQUENCE_LENGTH,\n",
        "                            trainable=False)\n",
        "\n",
        "print('Training model.')\n",
        "\n",
        "# train a 1D convnet with global maxpooling\n",
        "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
        "embedded_sequences = embedding_layer(sequence_input)\n",
        "x = Conv1D(128, 5, activation='relu')(embedded_sequences)\n",
        "x = MaxPooling1D(5)(x)\n",
        "x = Conv1D(128, 5, activation='relu')(x)\n",
        "x = MaxPooling1D(5)(x)\n",
        "x = Conv1D(128, 5, activation='relu')(x)\n",
        "x = MaxPooling1D(35)(x)\n",
        "x = Flatten()(x)\n",
        "x = Dense(128, activation='relu')(x)\n",
        "preds = Dense(len(labels_index), activation='softmax')(x)\n",
        "\n",
        "model = Model(sequence_input, preds)\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='rmsprop',\n",
        "              metrics=['acc'])\n",
        "\n",
        "model.fit(x_train, y_train,\n",
        "          batch_size=128,\n",
        "          epochs=10,\n",
        "          validation_data=(x_val, y_val))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Indexing word vectors.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "FileNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-2-6b4e05c14df7>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     53\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m \u001b[0membeddings_index\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mGLOVE_DIR\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'glove.6B.100d.txt'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     56\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m     \u001b[0mvalues\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '/glove.6B/glove.6B.100d.txt'"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "lTyFm8wwQh83",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Are word embeddings still useful?\n",
        "\n",
        "The word embeding models we have seen have several limitations:\n",
        "\n",
        "+ **Word2Vec** and **Glove** handle whole words, and can't easily handle words they haven't seen before. \n",
        "+ Words can be ambigous, but we are assigning only one embedding. Embeddings don't depend on the context.\n",
        "\n",
        "**FastText** (based on Word2Vec) is word-fragment (character) based and can usually handle unseen words, although it still generates one vector per word. \n",
        "\n",
        "Lately, several new \"ontext-aware\" models have been proposed.\n",
        "\n",
        "**ELMo** and **BERT**  incorporate context, handling polysemy and nuance much better (e.g. sentences like \"Time flies like an arrow. Fruit flies like bananas\") . This in general improves performance notably on downstream tasks.\n",
        "\n",
        "For natural language tasks, **ELMo** and **BERT** represent the best option at this time. For other kinds of tasks (for example, item-embedding for recommendert systems), **Word2Vec** is still an alternative. \n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "n4i0LQ-ETCcL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Bibliography\n",
        "\n",
        "On word embeddings (Part I, II and III): http://ruder.io/word-embeddings-1/"
      ]
    }
  ]
}