Simple Transformers

Implementing transformers papers very simplified because until you code it you don't really understand it.

Features

Models

• Implement GPT
• Implement BERT
• Implement T5

Training

• Implement p-tuning
• Implement prompt tuning
• Implement LoRA
• Implement DoRA
• Implement DPO
• Implement Efficient Training of Language Models to Fill in the Middle

Inference

• Implement top-k
• Implement temperature

Supervised Fine Tuning

• Implement addition of new tokens

Metrics

• Implement hallucination metrics

Optimizations

• Flash attention