Question Answering with SQuAD

The goal of this project was for me to get familiar with the Question Answering task, a very active topic in NLP research.

To this end, I implemented a Bidirectional Attention Flow neural network as a baseline, improving Chris Chute's model implementation, adding word-character inputs as described in the original paper.

I found this project very useful from a learning perspective so I highly recommend you to dig into the code and work on improving this baseline.

You can reproduce the work following the Set-Up section, and potentially (recommended!) train the model on a single GPU setting the cuda variable in config.py to True.

Model Architecture

Source: BiDAF paper

Code Organization

├── config.py          <- Configuration file with data directories and hyperparamters to train the model
├── data_loader.py     <- Define an iterator who collects batches of data to train the model
├── eval.py            <- Evaluate the model on a new pair of (context, question)
├── layers.py          <- Define the various layers to be used by the main BiDAF model
├── make_dataset.py    <- Download the SquAD dataset and pre-process the data for training
├── model.py.          <- Define the BiDAF model architecture
├── requirements.txt   <- Required Python libraries to build the project
├── test.py            <- Test the performance of a trained model on the DEV dataset
├── train.py           <- Train a model using the TRAIN dataset only
├── utils.py           <- Group a bunch of useful functions to process the data

Results

Exact-Match and F1 Score on Validation set after training:

EM	F1
0.64	0.75

Set-Up

• Clone the repository
• Create a directory for your experiments, logs and model weights: mkdir output
• Download GloVE word vectors: https://nlp.stanford.edu/projects/glove/
• Modify the config.py file to set up the paths where your GloVE, SquAD and models will be located
• Create a Python virtual environment, source to it: mkvirualenv qa-env ; workon qa-env if you use virtualenvwrapper
• Install the dependencies: pip install -r requirements.txt ; python -m spacy download en
• Run python make_dataset.py to download SquAD dataset and pre-process the data
• Run python train.py to train the model with hyper-parameters found in config.py
• Run python test.py to test the model EM and F1 scores on Dev examples
• Play with eval.py to answer your own questions! :)

Next Steps

• set up a variable to choose between training the model with word only VS word + characters
• collect the moving average of the weights during training and use them during testing
• add the ability to train the model on multiple GPUs, and offer half-precision training to speed-up the training
• improve this baseline using pre-training encoding such as BERT, and/or set-up a multi-task learning pipeline to jointly learn to answer questions together with another closely related NLP task.

Resources

• SQuAD dataset: https://arxiv.org/abs/1606.05250
• Bidirectional Attention Flow for Machine Comprehension" by Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, Hannaneh Hajishirzi : https://arxiv.org/abs/1611.01603
• Authors' TensorFlow implementation: https://allenai.github.io/bi-att-flow/
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova : https://arxiv.org/abs/1810.04805
• BiDAF baseline model: https://github.com/chrischute/squad
• PyTorch pretrained BERT: https://github.com/huggingface/pytorch-pretrained-BERT
• GloVE: https://nlp.stanford.edu/projects/glove/