0. NLP Preprocessing Pipeline for QA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP Preprocessing Pipeline for QA \n",
    "\n",
    "This notebook serves following purposes:\n",
    "1. I initially starting building a Question Answering system from scratch. This involved writing a lot of NLP preprocessing code. This notebook has all those functions. These steps are common to many NLP tasks and hence can be useful to many people out there. I do not use any higher level library for creating batches, vocabularies, datasets, dataloaders etc. My preprocessing pipeline only uses spacy for tokenization.\n",
    "2. I have also tried the same preprocessing steps using torchtext. And using torchtext leads to faster convergence in case of BiDAF. I am not sure as to why this happens or what under-the-hood optimizations torchtext uses that I am missing out on. I am still working on it and will update the repo with new results.\n",
    "4. The preprocessing part of this notebook goes into the script preprocess.py."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "\n",
    "Following are the major steps involved in the pipeline of QA. Some of these are common to all NLP tasks too.\n",
    "1. Load the dataset files. 2 JSON files.\n",
    "2. Parse the JSON files to get the data in the desired format.\n",
    "3. Filter large examples: Based on sequence lengths, drop some of the outliers.\n",
    "4. Gather text to build vocabularies. Vocabularies in NLP are just a mapping from words to integers. The integers then represent that word and are subsequently converted into word embeddings. \n",
    "5. For each example in the dataset, map the word to its corresponding id. This process is also known as numericalization of text.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import re, os, string, typing, gc, json\n",
    "import spacy\n",
    "from collections import Counter\n",
    "nlp = spacy.load('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_json(path):\n",
    "    '''\n",
    "    Loads the JSON file of the Squad dataset.\n",
    "    Returns the json object of the dataset.\n",
    "    '''\n",
    "    with open(path, 'r', encoding='utf-8') as f:\n",
    "        data = json.load(f)\n",
    "        \n",
    "    print(\"Length of data: \", len(data['data']))\n",
    "    print(\"Data Keys: \", data['data'][0].keys())\n",
    "    print(\"Title: \", data['data'][0]['title'])\n",
    "    \n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset json files has the following structure:\n",
    "\n",
    "<img src=\"images/squadjson.PNG\" width=\"500\" height=\"400\"/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_data(data:dict)->list:\n",
    "    '''\n",
    "    Parses the JSON file of Squad dataset by looping through the\n",
    "    keys and values and returns a list of dictionaries with\n",
    "    context, query and label triplets being the keys of each dict.\n",
    "    '''\n",
    "    data = data['data']\n",
    "    qa_list = []\n",
    "\n",
    "    for paragraphs in data:\n",
    "\n",
    "        for para in paragraphs['paragraphs']:\n",
    "            context = para['context']\n",
    "\n",
    "            for qa in para['qas']:\n",
    "                \n",
    "                id = qa['id']\n",
    "                question = qa['question']\n",
    "                \n",
    "                for ans in qa['answers']:\n",
    "                    answer = ans['text']\n",
    "                    ans_start = ans['answer_start']\n",
    "                    ans_end = ans_start + len(answer)\n",
    "                    \n",
    "                    qa_dict = {}\n",
    "                    qa_dict['id'] = id\n",
    "                    qa_dict['context'] = context\n",
    "                    qa_dict['question'] = question\n",
    "                    qa_dict['label'] = [ans_start, ans_end]\n",
    "\n",
    "                    qa_dict['answer'] = answer\n",
    "                    qa_list.append(qa_dict)    \n",
    "\n",
    "    \n",
    "    return qa_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_large_examples(df):\n",
    "    '''\n",
    "    Returns ids of examples where context lengths, query lengths and answer lengths are\n",
    "    above a particular threshold. These ids can then be dropped from the dataframe. \n",
    "    This is explicitly mentioned in QANet but can be done for other models as well.\n",
    "    '''\n",
    "    \n",
    "    ctx_lens = []\n",
    "    query_lens = []\n",
    "    ans_lens = []\n",
    "    for index, row in df.iterrows():\n",
    "        ctx_tokens = [w.text for w in nlp(row.context, disable=['parser','ner','tagger'])]\n",
    "        if len(ctx_tokens)>400:\n",
    "            ctx_lens.append(row.name)\n",
    "\n",
    "        query_tokens = [w.text for w in nlp(row.question, disable=['parser','tagger','ner'])]\n",
    "        if len(query_tokens)>50:\n",
    "            query_lens.append(row.name)\n",
    "\n",
    "        ans_tokens = [w.text for w in nlp(row.answer, disable=['parser','tagger','ner'])]\n",
    "        if len(ans_tokens)>30:\n",
    "            ans_lens.append(row.name)\n",
    "\n",
    "        assert row.name == index\n",
    "    \n",
    "    return set(ans_lens + ctx_lens + query_lens)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gather_text_for_vocab(dfs:list):\n",
    "    '''\n",
    "    Gathers text from contexts and questions to build a vocabulary.\n",
    "    \n",
    "    :param dfs: list of dataframes of SQUAD dataset.\n",
    "    :returns: list of contexts and questions\n",
    "    '''\n",
    "    \n",
    "    text = []\n",
    "    total = 0\n",
    "    for df in dfs:\n",
    "        unique_contexts = list(df.context.unique())\n",
    "        unique_questions = list(df.question.unique())\n",
    "        total += df.context.nunique() + df.question.nunique()\n",
    "        text.extend(unique_contexts + unique_questions)\n",
    "    \n",
    "    assert len(text) == total\n",
    "    \n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_word_vocab(vocab_text):\n",
    "    '''\n",
    "    Builds a word-level vocabulary from the given text.\n",
    "    \n",
    "    :param list vocab_text: list of contexts and questions\n",
    "    :returns \n",
    "        dict word2idx: word to index mapping of words\n",
    "        dict idx2word: integer to word mapping\n",
    "        list word_vocab: list of words sorted by frequency\n",
    "    '''\n",
    "    \n",
    "    \n",
    "    words = []\n",
    "    for sent in vocab_text:\n",
    "        for word in nlp(sent, disable=['parser','tagger','ner']):\n",
    "            words.append(word.text)\n",
    "\n",
    "    word_counter = Counter(words)\n",
    "    word_vocab = sorted(word_counter, key=word_counter.get, reverse=True)\n",
    "    print(f\"raw-vocab: {len(word_vocab)}\")\n",
    "    #word_vocab = list(set(word_vocab).intersection(set(glove_words)))\n",
    "    print(f\"glove-vocab: {len(word_vocab)}\")\n",
    "    word_vocab.insert(0, '<unk>')\n",
    "    word_vocab.insert(1, '<pad>')\n",
    "    print(f\"vocab-length: {len(word_vocab)}\")\n",
    "    word2idx = {word:idx for idx, word in enumerate(word_vocab)}\n",
    "    print(f\"word2idx-length: {len(word2idx)}\")\n",
    "    idx2word = {v:k for k,v in word2idx.items()}\n",
    "    \n",
    "    \n",
    "    return word2idx, idx2word, word_vocab\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_char_vocab(vocab_text):\n",
    "    '''\n",
    "    Builds a character-level vocabulary from the given text.\n",
    "    \n",
    "    :param list vocab_text: list of contexts and questions\n",
    "    :returns \n",
    "        dict char2idx: character to index mapping of words\n",
    "        list char_vocab: list of characters sorted by frequency\n",
    "    '''\n",
    "    \n",
    "    chars = []\n",
    "    for sent in vocab_text:\n",
    "        for ch in sent:\n",
    "            chars.append(ch)\n",
    "\n",
    "    char_counter = Counter(chars)\n",
    "    char_vocab = sorted(char_counter, key=char_counter.get, reverse=True)\n",
    "    print(f\"raw-char-vocab: {len(char_vocab)}\")\n",
    "    high_freq_char = [char for char, count in char_counter.items() if count>=20]\n",
    "    char_vocab = list(set(char_vocab).intersection(set(high_freq_char)))\n",
    "    print(f\"char-vocab-intersect: {len(char_vocab)}\")\n",
    "    char_vocab.insert(0,'<unk>')\n",
    "    char_vocab.insert(1,'<pad>')\n",
    "    char2idx = {char:idx for idx, char in enumerate(char_vocab)}\n",
    "    print(f\"char2idx-length: {len(char2idx)}\")\n",
    "    \n",
    "    return char2idx, char_vocab\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def context_to_ids(text, word2idx):\n",
    "    '''\n",
    "    Converts context text to their respective ids by mapping each word\n",
    "    using word2idx. Input text is tokenized using spacy tokenizer first.\n",
    "    \n",
    "    :param str text: context text to be converted\n",
    "    :returns list context_ids: list of mapped ids\n",
    "    \n",
    "    :raises assertion error: sanity check\n",
    "    \n",
    "    '''\n",
    "\n",
    "    context_tokens = [w.text for w in nlp(text, disable=['parser','tagger','ner'])]\n",
    "    context_ids = [word2idx[word] for word in context_tokens]\n",
    "    \n",
    "    assert len(context_ids) == len(context_tokens)\n",
    "    return context_ids\n",
    "    \n",
    "def question_to_ids(text, word2idx):\n",
    "    '''\n",
    "    Converts question text to their respective ids by mapping each word\n",
    "    using word2idx. Input text is tokenized using spacy tokenizer first.\n",
    "    \n",
    "    :param str text: question text to be converted\n",
    "    :returns list context_ids: list of mapped ids\n",
    "    \n",
    "    :raises assertion error: sanity check\n",
    "    \n",
    "    '''\n",
    "\n",
    "    question_tokens = [w.text for w in nlp(text, disable=['parser','tagger','ner'])]\n",
    "    question_ids = [word2idx[word] for word in question_tokens]\n",
    "    \n",
    "    assert len(question_ids) == len(question_tokens)\n",
    "    return question_ids\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Purpose behind this test was to ensure that the label for each example was correct.\n",
    "Two ways to test this.\n",
    "* First is to calculate spans of the context and check if start and end indices from the label are present in the calculated spans. Index value of the example which fails the start and end tests are appended in separate lists.\n",
    "* Second is to get the start and end indexes of the answer in the context_ids list. Get the ids corresponding to those positions, convert them to string using `word2idx` and compare them with the start and end tokens from the given answer. Examples which fail this test have their position added to a list.\n",
    "* The reason why some examples fail here is largely due to the absence of a `' '` or a space before and after the answer in the context. There are some spans that the tokenizer fails to capture or is simply a case where the example is not cleaned. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_indices(df, idx2word):\n",
    "    '''\n",
    "    Performs the tests mentioned above. This method also gets the start and end of the answers\n",
    "    with respect to the context_ids for each example.\n",
    "    \n",
    "    :param dataframe df: SQUAD df\n",
    "    :returns\n",
    "        list start_value_error: example idx where the start idx is not found in the start spans\n",
    "                                of the text\n",
    "        list end_value_error: example idx where the end idx is not found in the end spans\n",
    "                              of the text\n",
    "        list assert_error: examples that fail assertion errors. A majority are due to the above errors\n",
    "        \n",
    "    '''\n",
    "\n",
    "    start_value_error = []\n",
    "    end_value_error = []\n",
    "    assert_error = []\n",
    "    for index, row in df.iterrows():\n",
    "\n",
    "        answer_tokens = [w.text for w in nlp(row['answer'], disable=['parser','tagger','ner'])]\n",
    "\n",
    "        start_token = answer_tokens[0]\n",
    "        end_token = answer_tokens[-1]\n",
    "        \n",
    "        context_span  = [(word.idx, word.idx + len(word.text)) \n",
    "                         for word in nlp(row['context'], disable=['parser','tagger','ner'])]\n",
    "\n",
    "        starts, ends = zip(*context_span)\n",
    "\n",
    "        answer_start, answer_end = row['label']\n",
    "\n",
    "        try:\n",
    "            start_idx = starts.index(answer_start)\n",
    "        except:\n",
    "            start_value_error.append(index)\n",
    "        try:\n",
    "            end_idx  = ends.index(answer_end)\n",
    "        except:\n",
    "            end_value_error.append(index)\n",
    "\n",
    "        try:\n",
    "            assert idx2word[row['context_ids'][start_idx]] == answer_tokens[0]\n",
    "            assert idx2word[row['context_ids'][end_idx]] == answer_tokens[-1]\n",
    "        except:\n",
    "            assert_error.append(index)\n",
    "\n",
    "\n",
    "    return start_value_error, end_value_error, assert_error\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_error_indices(df, idx2word):\n",
    "    '''\n",
    "    Gets error indices from the method above and returns a \n",
    "    set of those indices.\n",
    "    '''\n",
    "    \n",
    "    start_value_error, end_value_error, assert_error = test_indices(df)\n",
    "    err_idx = start_value_error + end_value_error + assert_error\n",
    "    err_idx = set(err_idx)\n",
    "    print(f\"Error indices: {len(err_idx)}\")\n",
    "    \n",
    "    return err_idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def index_answer(row, idx2word):\n",
    "    '''\n",
    "    Takes in a row of the dataframe or one training example and\n",
    "    returns a tuple of start and end positions of answer by calculating \n",
    "    spans.\n",
    "    '''\n",
    "    \n",
    "    context_span = [(word.idx, word.idx + len(word.text)) for word in nlp(row.context, disable=['parser','tagger','ner'])]\n",
    "    starts, ends = zip(*context_span)\n",
    "    \n",
    "    answer_start, answer_end = row.label\n",
    "    start_idx = starts.index(answer_start)\n",
    " \n",
    "    end_idx  = ends.index(answer_end)\n",
    "    \n",
    "    ans_toks = [w.text for w in nlp(row.answer,disable=['parser','tagger','ner'])]\n",
    "    ans_start = ans_toks[0]\n",
    "    ans_end = ans_toks[-1]\n",
    "    assert idx2word[row.context_ids[start_idx]] == ans_start\n",
    "    assert idx2word[row.context_ids[end_idx]] == ans_end\n",
    "    \n",
    "    return [start_idx, end_idx]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}