week02_classification/seminar_tf2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Large scale text analysis with deep learning (3 points)\n",
    "\n",
    "Today we're gonna apply the newly learned tools for the task of predicting job salary.\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/kaggle-competitions/kaggle/3342/media/salary%20prediction%20engine%20v2.png\" width=400px>\n",
    "\n",
    "_Special thanks to [Oleg Vasilev](https://github.com/Omrigan/) for the core assignment idea._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### About the challenge\n",
    "For starters, let's download and unpack the data from [here](https://www.dropbox.com/s/5msc5ix7ndyba10/Train_rev1.csv.tar.gz?dl=0). \n",
    "\n",
    "You can also get it from [yadisk url](https://yadi.sk/d/vVEOWPFY3NruT7) the competition [page](https://www.kaggle.com/c/job-salary-prediction/data) (pick `Train_rev1.*`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\n",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0\n",
      "100  119M  100  119M    0     0  12.6M      0  0:00:09  0:00:09 --:--:-- 27.4M\n",
      "Train_rev1.csv\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(244768, 12)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!curl -L https://www.dropbox.com/s/5msc5ix7ndyba10/Train_rev1.csv.tar.gz?dl=1 -o Train_rev1.csv.tar.gz\n",
    "!tar -xvzf ./Train_rev1.csv.tar.gz\n",
    "data = pd.read_csv(\"./Train_rev1.csv\", index_col=None)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One problem with salary prediction is that it's oddly distributed: there are many people who are paid standard salaries and a few that get tons o money. The distribution is fat-tailed on the right side, which is inconvenient for MSE minimization.\n",
    "\n",
    "There are several techniques to combat this: using a different loss function, predicting log-target instead of raw target or even replacing targets with their percentiles among all salaries in the training set. We gonna use logarithm for now.\n",
    "\n",
    "_You can read more [in the official description](https://www.kaggle.com/c/job-salary-prediction#description)._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['Log1pSalary'] = np.log1p(data['SalaryNormalized']).astype('float32')\n",
    "\n",
    "plt.figure(figsize=[8, 4])\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.hist(data[\"SalaryNormalized\"], bins=20);\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.hist(data['Log1pSalary'], bins=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our task is to predict one number, __Log1pSalary__.\n",
    "\n",
    "To do so, our model can access a number of features:\n",
    "* Free text: __`Title`__ and  __`FullDescription`__\n",
    "* Categorical: __`Category`__, __`Company`__, __`LocationNormalized`__, __`ContractType`__, and __`ContractTime`__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_columns = [\"Title\", \"FullDescription\"]\n",
    "categorical_columns = [\"Category\", \"Company\", \"LocationNormalized\", \"ContractType\", \"ContractTime\"]\n",
    "target_column = \"Log1pSalary\"\n",
    "\n",
    "data[categorical_columns] = data[categorical_columns].fillna('NaN') # cast missing values to string \"NaN\"\n",
    "\n",
    "data.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing text data\n",
    "\n",
    "Just like last week, applying NLP to a problem begins from tokenization: splitting raw text into sequences of tokens (words, punctuation, etc).\n",
    "\n",
    "__Your task__ is to lowercase and tokenize all texts under `Title` and `FullDescription` columns. Store the tokenized data as a __space-separated__ string of tokens for performance reasons.\n",
    "\n",
    "It's okay to use nltk tokenizers. Assertions were designed for WordPunctTokenizer, slight deviations are okay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Raw text:\")\n",
    "print(data[\"FullDescription\"][2::100000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "tokenizer = nltk.tokenize.WordPunctTokenizer()\n",
    "\n",
    "# see task above\n",
    "<YOUR CODE HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can assume that our text is a space-separated list of tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Tokenized:\")\n",
    "print(data[\"FullDescription\"][2::100000])\n",
    "assert data[\"FullDescription\"][2][:50] == 'mathematical modeller / simulation analyst / opera'\n",
    "assert data[\"Title\"][54321] == 'international digital account manager ( german )'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all words are equally useful. Some of them are typos or rare words that are only present a few times. \n",
    "\n",
    "Let's count how many times is each word present in the data so that we can build a \"white list\" of known words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count how many times does each token occur in both \"Title\" and \"FullDescription\" in total\n",
    "# build a dictionary { token -> it's count }\n",
    "token_counts = <YOUR CODE>\n",
    "\n",
    "# hint: you may or may not want to use collections.Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Total unique tokens :\", len(token_counts))\n",
    "print('\\n'.join(map(str, token_counts.most_common(n=5))))\n",
    "print('...')\n",
    "print('\\n'.join(map(str, token_counts.most_common()[-3:])))\n",
    "\n",
    "assert token_counts.most_common(1)[0][1] in  range(2600000, 2700000)\n",
    "assert len(token_counts) in range(200000, 210000)\n",
    "print('Correct!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's see how many words are there for each count\n",
    "plt.hist(list(token_counts.values()), range=[0, 10**4], bins=50, log=True)\n",
    "plt.xlabel(\"Word counts\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now filter tokens a list of all tokens that occur at least 10 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_count = 10\n",
    "\n",
    "# tokens from token_counts keys that had at least min_count occurrences throughout the dataset\n",
    "tokens = <YOUR CODE HERE>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a special tokens for unknown and empty words\n",
    "UNK, PAD = \"UNK\", \"PAD\"\n",
    "tokens = [UNK, PAD] + sorted(tokens)\n",
    "print(\"Vocabulary size:\", len(tokens))\n",
    "\n",
    "assert type(tokens) == list\n",
    "assert len(tokens) in range(30000, 40000)\n",
    "assert 'me' in tokens\n",
    "assert UNK in tokens\n",
    "print(\"Correct!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build an inverse token index: a dictionary from token(string) to it's index in `tokens` (int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_to_id = <your code here>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert isinstance(token_to_id, dict)\n",
    "assert len(token_to_id) == len(tokens)\n",
    "for tok in tokens:\n",
    "    assert tokens[token_to_id[tok]] == tok\n",
    "\n",
    "print(\"Correct!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, let's use the vocabulary you've built to map text lines into neural network-digestible matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "UNK_IX, PAD_IX = map(token_to_id.get, [UNK, PAD])\n",
    "\n",
    "def as_matrix(sequences, max_len=None):\n",
    "    \"\"\" Convert a list of tokens into a matrix with padding \"\"\"\n",
    "    if isinstance(sequences[0], str):\n",
    "        sequences = list(map(str.split, sequences))\n",
    "        \n",
    "    max_len = min(max(map(len, sequences)), max_len or float('inf'))\n",
    "    \n",
    "    matrix = np.full((len(sequences), max_len), np.int32(PAD_IX))\n",
    "    for i,seq in enumerate(sequences):\n",
    "        row_ix = [token_to_id.get(word, UNK_IX) for word in seq[:max_len]]\n",
    "        matrix[i, :len(row_ix)] = row_ix\n",
    "    \n",
    "    return matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Lines:\")\n",
    "print('\\n'.join(data[\"Title\"][::100000].values), end='\\n\\n')\n",
    "print(\"Matrix:\")\n",
    "print(as_matrix(data[\"Title\"][::100000]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's  encode the categirical data we have.\n",
    "\n",
    "As usual, we shall use one-hot encoding for simplicity. Kudos if you implement more advanced encodings: tf-idf, pseudo-time-series, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "\n",
    "# we only consider top-1k most frequent companies to minimize memory usage\n",
    "top_companies, top_counts = zip(*Counter(data['Company']).most_common(1000))\n",
    "recognized_companies = set(top_companies)\n",
    "data[\"Company\"] = data[\"Company\"].apply(lambda comp: comp if comp in recognized_companies else \"Other\")\n",
    "\n",
    "categorical_vectorizer = DictVectorizer(dtype=np.float32, sparse=False)\n",
    "categorical_vectorizer.fit(data[categorical_columns].apply(dict, axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The deep learning part\n",
    "\n",
    "Once we've learned to tokenize the data, let's design a machine learning experiment.\n",
    "\n",
    "As before, we won't focus too much on validation, opting for a simple train-test split.\n",
    "\n",
    "__To be completely rigorous,__ we've comitted a small crime here: we used the whole data for tokenization and vocabulary building. A more strict way would be to do that part on training set only. You may want to do that and measure the magnitude of changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_val = train_test_split(data, test_size=0.2, random_state=42)\n",
    "data_train.index = range(len(data_train))\n",
    "data_val.index = range(len(data_val))\n",
    "\n",
    "print(\"Train size = \", len(data_train))\n",
    "print(\"Validation size = \", len(data_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_batch(data, max_len=None, word_dropout=0):\n",
    "    \"\"\"\n",
    "    Creates a keras-friendly dict from the batch data.\n",
    "    :param word_dropout: replaces token index with UNK_IX with this probability\n",
    "    :returns: a dict with {'title' : int64[batch, title_max_len]\n",
    "    \"\"\"\n",
    "    batch = {}\n",
    "    batch[\"Title\"] = as_matrix(data[\"Title\"].values, max_len)\n",
    "    batch[\"FullDescription\"] = as_matrix(data[\"FullDescription\"].values, max_len)\n",
    "    batch['Categorical'] = categorical_vectorizer.transform(data[categorical_columns].apply(dict, axis=1))\n",
    "    \n",
    "    if word_dropout != 0:\n",
    "        batch[\"FullDescription\"] = apply_word_dropout(batch[\"FullDescription\"], 1. - word_dropout)\n",
    "    \n",
    "    if target_column in data.columns:\n",
    "        batch[target_column] = data[target_column].values\n",
    "    \n",
    "    return batch\n",
    "\n",
    "def apply_word_dropout(matrix, keep_prop, replace_with=UNK_IX, pad_ix=PAD_IX,):\n",
    "    dropout_mask = np.random.choice(2, np.shape(matrix), p=[keep_prop, 1 - keep_prop])\n",
    "    dropout_mask &= matrix != pad_ix\n",
    "    return np.choose(dropout_mask, [matrix, np.full_like(matrix, replace_with)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "make_batch(data_train[:3], max_len=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Architecture\n",
    "\n",
    "Our basic model consists of three branches:\n",
    "* Title encoder\n",
    "* Description encoder\n",
    "* Categorical features encoder\n",
    "\n",
    "We will then feed all 3 branches into one common network that predicts salary.\n",
    "\n",
    "<img src=\"https://github.com/yandexdataschool/nlp_course/raw/master/resources/w2_conv_arch.png\" width=600px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This clearly doesn't fit into keras' __Sequential__ interface. To build such a network, one will have to use __[Keras Functional API](https://keras.io/models/model/)__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "import keras.layers as L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(n_tokens=len(tokens), n_cat_features=len(categorical_vectorizer.vocabulary_), hid_size=64):\n",
    "    \"\"\" Build a model that maps three data sources to a single linear output: predicted log1p(salary) \"\"\"\n",
    "    \n",
    "    l_title = L.Input(shape=[None], name=\"Title\")\n",
    "    l_descr = L.Input(shape=[None], name=\"FullDescription\")\n",
    "    l_categ = L.Input(shape=[n_cat_features], name=\"Categorical\")\n",
    "    \n",
    "    # Build your monster!\n",
    "    \n",
    "    # <YOUR CODE>\n",
    "    \n",
    "    output_layer = <...>\n",
    "    # end of your code\n",
    "    \n",
    "    model = keras.models.Model(inputs=[l_title, l_descr, l_categ], outputs=[output_layer])\n",
    "    model.compile('adam', 'mean_squared_error', metrics=['mean_absolute_error'])\n",
    "    return model\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = build_model()\n",
    "model.summary()\n",
    "\n",
    "dummy_pred = model.predict(make_batch(data_train[:100]))\n",
    "dummy_loss = model.train_on_batch(make_batch(data_train[:100]), data_train['Log1pSalary'][:100])[0]\n",
    "assert dummy_pred.shape == (100, 1)\n",
    "assert len(np.unique(dummy_pred)) > 20, \"model returns suspiciously few unique outputs. Check your initialization\"\n",
    "assert np.ndim(dummy_loss) == 0 and 0. <= dummy_loss <= 250., \"make sure you minimize MSE\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training and evaluation\n",
    "\n",
    "As usual, we gonna feed our monster with random minibatches of data. \n",
    "\n",
    "As we train, we want to monitor not only loss function, which is computed in log-space, but also the actual error measured in dollars."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def iterate_minibatches(data, batch_size=256, shuffle=True, cycle=False, **kwargs):\n",
    "    \"\"\" iterates minibatches of data in random order \"\"\"\n",
    "    while True:\n",
    "        indices = np.arange(len(data))\n",
    "        if shuffle:\n",
    "            indices = np.random.permutation(indices)\n",
    "\n",
    "        for start in range(0, len(indices), batch_size):\n",
    "            batch = make_batch(data.iloc[indices[start : start + batch_size]], **kwargs)\n",
    "            target = batch.pop(target_column)\n",
    "            yield batch, target\n",
    "        \n",
    "        if not cycle: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model training\n",
    "\n",
    "We can now fit our model the usual minibatch way. The interesting part is that we train on an infinite stream of minibatches, produced by `iterate_minibatches` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 256\n",
    "epochs = 10            # definitely too small\n",
    "steps_per_epoch = 100  # for full pass over data: (len(data_train) - 1) // batch_size + 1\n",
    "\n",
    "model = build_model()\n",
    "\n",
    "model.fit_generator(iterate_minibatches(data_train, batch_size, cycle=True, word_dropout=0.05), \n",
    "                    epochs=epochs, steps_per_epoch=steps_per_epoch,\n",
    "                    \n",
    "                    validation_data=iterate_minibatches(data_val, batch_size, cycle=True),\n",
    "                    validation_steps=data_val.shape[0] // batch_size\n",
    "                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_metrics(model, data, batch_size=batch_size, name=\"\", **kw):\n",
    "    squared_error = abs_error = num_samples = 0.0\n",
    "    for batch_x, batch_y in iterate_minibatches(data, batch_size=batch_size, shuffle=False, **kw):\n",
    "        batch_pred = model.predict(batch_x)[:, 0]\n",
    "        squared_error += np.sum(np.square(batch_pred - batch_y))\n",
    "        abs_error += np.sum(np.abs(batch_pred - batch_y))\n",
    "        num_samples += len(batch_y)\n",
    "    print(\"%s results:\" % (name or \"\"))\n",
    "    print(\"Mean square error: %.5f\" % (squared_error / num_samples))\n",
    "    print(\"Mean absolute error: %.5f\" % (abs_error / num_samples))\n",
    "    return squared_error, abs_error\n",
    "    \n",
    "print_metrics(model, data_train, name='Train')\n",
    "print_metrics(model, data_val, name='Val');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus part: explaining model predictions\n",
    "\n",
    "It's usually a good idea to understand how your model works before you let it make actual decisions. It's simple for linear models: just see which words learned positive or negative weights. However, its much harder for neural networks that learn complex nonlinear dependencies.\n",
    "\n",
    "There are, however, some ways to look inside the black box:\n",
    "* Seeing how model responds to input perturbations\n",
    "* Finding inputs that maximize/minimize activation of some chosen neurons (_read more [on distill.pub](https://distill.pub/2018/building-blocks/)_)\n",
    "* Building local linear approximations to your neural network: [article](https://arxiv.org/abs/1602.04938), [eli5 library](https://github.com/TeamHG-Memex/eli5/tree/master/eli5/formatters)\n",
    "\n",
    "Today we gonna try the first method just because it's the simplest one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def explain(model, sample, col_name='Title'):\n",
    "    \"\"\" Computes the effect each word had on model predictions \"\"\"\n",
    "    sample = dict(sample)\n",
    "    sample_col_tokens = [tokens[token_to_id.get(tok, 0)] for tok in sample[col_name].split()]\n",
    "    data_drop_one_token = pd.DataFrame([sample] * (len(sample_col_tokens) + 1))\n",
    "\n",
    "    for drop_i in range(len(sample_col_tokens)):\n",
    "        data_drop_one_token.loc[drop_i, col_name] = ' '.join(UNK if i == drop_i else tok\n",
    "                                                   for i, tok in enumerate(sample_col_tokens)) \n",
    "\n",
    "    *predictions_drop_one_token, baseline_pred = model.predict(make_batch(data_drop_one_token))[:, 0]\n",
    "    diffs = baseline_pred - predictions_drop_one_token\n",
    "    return list(zip(sample_col_tokens, diffs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import HTML, display_html\n",
    "\n",
    "def draw_html(tokens_and_weights, cmap=plt.get_cmap(\"bwr\"), display=True,\n",
    "              token_template=\"\"\"<span style=\"background-color: {color_hex}\">{token}</span>\"\"\",\n",
    "              font_style=\"font-size:14px;\"\n",
    "             ):\n",
    "    \n",
    "    def get_color_hex(weight):\n",
    "        rgba = cmap(1. / (1 + np.exp(weight)), bytes=True)\n",
    "        return '#%02X%02X%02X' % rgba[:3]\n",
    "    \n",
    "    tokens_html = [\n",
    "        token_template.format(token=token, color_hex=get_color_hex(weight))\n",
    "        for token, weight in tokens_and_weights\n",
    "    ]\n",
    "    \n",
    "    \n",
    "    raw_html = \"\"\"<p style=\"{}\">{}</p>\"\"\".format(font_style, ' '.join(tokens_html))\n",
    "    if display:\n",
    "        display_html(HTML(raw_html))\n",
    "        \n",
    "    return raw_html\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 36605\n",
    "tokens_and_weights = explain(model, data.loc[i], \"Title\")\n",
    "draw_html([(tok, weight * 5) for tok, weight in tokens_and_weights], font_style='font-size:20px;');\n",
    "\n",
    "tokens_and_weights = explain(model, data.loc[i], \"FullDescription\")\n",
    "draw_html([(tok, weight * 10) for tok, weight in tokens_and_weights]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 12077\n",
    "tokens_and_weights = explain(model, data.loc[i], \"Title\")\n",
    "draw_html([(tok, weight * 5) for tok, weight in tokens_and_weights], font_style='font-size:20px;');\n",
    "\n",
    "tokens_and_weights = explain(model, data.loc[i], \"FullDescription\")\n",
    "draw_html([(tok, weight * 10) for tok, weight in tokens_and_weights]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p style=\"font-size:20px;\"><span style=\"background-color: #FFD3D3\">cleaning</span> <span style=\"background-color: #FFBABA\">operative</span></p>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<p style=\"font-size:14px;\"><span style=\"background-color: #FFFAFA\">12</span> <span style=\"background-color: #FFF8F8\">.</span> <span style=\"background-color: #FEFEFF\">5</span> <span style=\"background-color: #FFC2C2\">hours</span> <span style=\"background-color: #FFE6E6\">per</span> <span style=\"background-color: #FFFCFC\">week</span> <span style=\"background-color: #E3E3FF\">monday</span> <span style=\"background-color: #FEFEFF\">friday</span> <span style=\"background-color: #FCFCFF\">9am</span> <span style=\"background-color: #FFECEC\">11</span> <span style=\"background-color: #FAFAFF\">.</span> <span style=\"background-color: #FCFCFF\">30am</span> <span style=\"background-color: #FFFEFE\">duties</span> <span style=\"background-color: #FCFCFF\">to</span> <span style=\"background-color: #FFFCFC\">include</span> <span style=\"background-color: #FFF2F2\">sweeping</span> <span style=\"background-color: #FFE8E8\">,</span> <span style=\"background-color: #F3F3FF\">mopping</span> <span style=\"background-color: #FFF4F4\">,</span> <span style=\"background-color: #ECECFF\">vacuuming</span> <span style=\"background-color: #FFF8F8\">,</span> <span style=\"background-color: #FFF4F4\">buffing</span> <span style=\"background-color: #FFFEFE\">,</span> <span style=\"background-color: #FFFCFC\">cleaning</span> <span style=\"background-color: #FFF0F0\">staff</span> <span style=\"background-color: #FFFAFA\">toilets</span> <span style=\"background-color: #FFFAFA\">and</span> <span style=\"background-color: #FFFAFA\">rest</span> <span style=\"background-color: #F3F3FF\">room</span> <span style=\"background-color: #EEEEFF\">.</span> <span style=\"background-color: #DCDCFF\">must</span> <span style=\"background-color: #F8F8FF\">be</span> <span style=\"background-color: #FFF8F8\">able</span> <span style=\"background-color: #FFFEFE\">to</span> <span style=\"background-color: #F2F2FF\">read</span> <span style=\"background-color: #FFF0F0\">as</span> <span style=\"background-color: #FFFAFA\">they</span> <span style=\"background-color: #FFFEFE\">will</span> <span style=\"background-color: #FFEAEA\">be</span> <span style=\"background-color: #FFF0F0\">using</span> <span style=\"background-color: #FFFEFE\">UNK</span> <span style=\"background-color: #FCFCFF\">which</span> <span style=\"background-color: #F0F0FF\">need</span> <span style=\"background-color: #FFFEFE\">UNK</span> <span style=\"background-color: #FFF0F0\">as</span> <span style=\"background-color: #ECECFF\">per</span> <span style=\"background-color: #FFEAEA\">instructions</span> <span style=\"background-color: #FFEEEE\">on</span> <span style=\"background-color: #FFF6F6\">the</span> <span style=\"background-color: #FFF6F6\">containers</span> <span style=\"background-color: #FEFEFF\">.</span> <span style=\"background-color: #FFF8F8\">sucessfull</span> <span style=\"background-color: #FEFEFF\">applicants</span> <span style=\"background-color: #F0F0FF\">will</span> <span style=\"background-color: #FFEEEE\">be</span> <span style=\"background-color: #FFFAFA\">trained</span> <span style=\"background-color: #F6F6FF\">on</span> <span style=\"background-color: #FFEAEA\">all</span> <span style=\"background-color: #F0F0FF\">electrical</span> <span style=\"background-color: #FFF2F2\">appliances</span> <span style=\"background-color: #FFFAFA\">and</span> <span style=\"background-color: #FFFEFE\">UNK</span> <span style=\"background-color: #FEFEFF\">of</span> <span style=\"background-color: #FFC8C8\">cleaning</span> <span style=\"background-color: #FFFAFA\">materials</span> <span style=\"background-color: #FFFEFE\">.</span></p>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "i = np.random.randint(len(data))\n",
    "print(\"Index:\", i)\n",
    "print(\"Salary (gbp):\", np.expm1(model.predict(make_batch(data.iloc[i: i+1]))[0, 0]))\n",
    "\n",
    "tokens_and_weights = explain(model, data.loc[i], \"Title\")\n",
    "draw_html([(tok, weight * 5) for tok, weight in tokens_and_weights], font_style='font-size:20px;');\n",
    "\n",
    "tokens_and_weights = explain(model, data.loc[i], \"FullDescription\")\n",
    "draw_html([(tok, weight * 10) for tok, weight in tokens_and_weights]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Terrible start-up idea #1962:__ make a tool that automaticaly rephrases your job description (or CV) to meet salary expectations :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}