PyTorch Transformers

This repo contains tutorials covering transformer models using PyTorch 1.4 and TorchText 0.5 using Python 3.6.

If you find any mistakes with the code, please do not hesitate to submit an issue. I welcome any feedback, positive or negative!

Getting Started

To install PyTorch, see installation instructions on the PyTorch website.

To install TorchText:

pip install torchtext

We'll also make use of spaCy to tokenize our data. To install spaCy, follow the instructions here making sure to install both the English and German models with:

python -m spacy download en
python -m spacy download de

Tutorials

• 1 - Multi Head Attention
• 2 - Multi Head Attention using(einsum Notation)
• 3 - Multi Query Attention
• 4 - Multi Query Attention using (einsum Notation)

References

Here are some things I looked at while making these tutorials. Some of it may be out of date.

• https://github.com/bentrevett/pytorch-seq2seq