Skip to content

Latest commit

 

History

History
111 lines (104 loc) · 4.8 KB

README.md

File metadata and controls

111 lines (104 loc) · 4.8 KB
Raw

File Structure:

.
├── Assignment_2.pdf
├── corpus
│   ├── dev.en
│   ├── dev.fr
│   ├── processed_dev.en
│   ├── processed_dev.fr
│   ├── processed_test.en
│   ├── processed_test.fr
│   ├── processed_train.en
│   ├── processed_train.fr
│   ├── test.en
│   ├── test.fr
│   ├── train.en
│   └── train.fr
├── metrics
│   └── testbleu.txt
├── plots.ipynb
├── README.md
├── Report.pdf
└── src
    ├── decoder.py
    ├── encoder.py
    ├── __init__.py
    ├── test.py
    ├── train.py
    └── utils.py

Instructions to run the code:

  • All scripts can be run by staying in the current (root) directory.

Processing the corpus for the first time:

  • Run the following command to process the corpus for the first time:
python -m src.utils
  • This will create the processed files in the corpus directory with processed_ prefix.

Training the model:

  • Run the following command to train the model:
python -m src.train
  • This will train the model and save the model in the weights directory.
  • To change the hyperparameters, you can pass them as arguments to the above command. For example:
python -m src.train --epochs 10 --batch_size 64
  • To see all the hyperparameters, you can run:
python -m src.train --help

Testing the model:

  • Run the following command to test the model:
python -m src.test
  • This will test the model and save the BLEU score for the test set in the metrics directory.

Loading the model:

  • The file weights/transformer.pt is a state dict containing various parameters of the model and other information computed during the training process.
  • A list of these parameters is as follows:
    • args: Hyperparameters of the training run.
    • model_state_dict: The state dict of the model.
    • optimizer_state_dict: The state dict of the optimizer.
    • en_seq_len: The sequence length of the English train dataset.
    • fr_seq_len: The sequence length of the French train dataset.
    • word2idx_en: The word to index mapping of the English train dataset.
    • word2idx_fr: The word to index mapping of the French train dataset.
    • idx2word_en: The index to word mapping of the English train dataset.
    • idx2word_fr: The index to word mapping of the French train dataset.
    • train_losses: The training losses.
    • dev_losses: The validation losses.
    • dev_bleu: The BLEU score on the validation set.
    • dev_rouge: The ROUGE score on the validation set.
  • To load the model, you can use the following code:
model = Transformer(args) # Pass the necessary args
model.load_state_dict(torch.load('weights/transformer.pt')['model_state_dict'])

Implementation Assumptions:

  • Used a sequence length of 77 for faster training during hyperparameter tuning.
  • If a sequence length of 77 is used, the model will assume that apart from the <BOS> and <EOS> tokens, the maximum length of the sentence is 77.
  • The model will pad the sentences to a length of 77 if the sentence is shorter than 77.
  • The model will truncate the sentences to a length of 77 if the sentence is longer than 77.
  • The model will not learn the padding token.
  • The position embeddings are not learned.
  • The model uses the Adam optimizer with a learning rate of 0.0001.
  • The model uses the CrossEntropyLoss loss function.

Pretrained Model weights:

  • The main model with 6 blocks, 8 heads, 512-dim embeddings and 0.1 has been trained for 10 epochs on the maximum sequence length of the training data.
    • The model can be downloaded from here
  • Some more models saved during the hyperparameter tuning process can be downloaded from the following links:
    • 6 blocks, 8 heads, 512-dim embeddings and 0.1 dropout rate trained for 10 epochs with a sequence length of 77:
    • 6 blocks, 8 heads, 640-dim embeddings and 0.2 dropout rate trained for 10 epochs with a sequence length of 77:
    • 8 blocks, 10 heads, 640-dim embeddings and 0.1 dropout rate trained for 10 epochs with a sequence length of 77: