Virtual Adversarial Training for NMT (Transformer model)

Implementation of "Effective Adversarial Regularization for Neural Machine Translation", ACL 2019

References

Motoki Sato, Jun Suzuki, Shun Kiyono. "Effective Adversarial Regularization for Neural Machine Translation", ACL 2019 paper bib

How to use

Requirements

• Python3.6+
• Chainer 6.x+
• Cupy 6.x+

# install chainer and cupy
$ pip install cupy
$ pip install chainer
$ pip install logzero

Please see how to install chainer: https://docs.chainer.org/en/stable/install.html

Train (iwslt2016-de-en)

$ python3 -u chainer_transformer.py --mode train --gpus 0 --dataset iwslt2016-de-en --seed 1212 --epoch 40 --out model_transformer_de-en

Train with VAT (iwslt2016-de-en)

$ python3 -u chainer_transformer.py --mode train --gpus 0 --dataset iwslt2016-de-en --seed 1212 --epoch 40 --out model_transformer_de-en_vat_enc --use-vat 1 --eps 1.0 --perturbation-target 0

perturbation types

perturbation-target	(enc, dec, enc-dec)
0	enc
1	dec
0 1	enc-dec (both)

VAT, Adv, VAT-Adv

use-vat	(vat, adv, vat-adv)
0	non (baseline)
1	vat
2	adv
3	vat-adv (both)

Eval

$ python3 -u chainer_transformer.py --mode test --gpus 0 --dataset iwslt2016-de-en --batchsize 600 --model model_transformer_de-en/model_epoch_40.npz --beam 20 --max-length 60 --datatype eval1

License

MIT License. Please see the LICENSE file for details.

Authors

We thank Takeru Miyato (@takerum), who gave us valuable comments about AdvT/VAT.

The codebase of the transformer is developed by Shun Kiyono (@butsugiri)

Contact

Please give me comments or questions: @aonotas