Use BiLSTM_attention, BERT, RoBERTa, XLNet and ALBERT models to classify the SST-2 data set based on pytorch.
These codes are recommended to run in Google Colab, where you may use free GPU resources.
The BiLSTM_attention model can let us know which words in a sentence do contributions to the sentiment of this sentence. The code is avalibale in "bilstm_attention.ipynb", where two types of self-attention mechanism have been achieved. You can run it in Google Colab for practice. The visualization result is shown below:
For specific BERT models, you can find them from https://huggingface.co/models and then do modify in "models.py".
| Model | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|
| BERT (base-uncased) | 91.8 | 91.8 | 91.8 | 91.8 |
| RoBERTa (base-uncased) | 93.4 | 93.5 | 93.4 | 93.3 |
| XLNet (base-uncased) | 92.5 | 92.5 | 92.5 | 92.5 |
| ALBERT (base-v2-uncased) | 91.4 | 91.4 | 91.4 | 91.4 |
| Model | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|
| BERT (large-uncased) | 93.1 | 93.2 | 93.1 | 93.1 |
| RoBERTa (large-uncased) | 94.9 | 95.0 | 95.0 | 94.9 |
| XLNet (large-uncased) | 94.6 | 94.7 | 94.6 | 94.6 |
| ALBERT (large-v2-uncased) | 92.2 | 92.3 | 92.2 | 92.2 |
| ALBERT (xlarge-v2-uncased) | 93.8 | 93.8 | 93.9 | 93.8 |
| ALBERT (xxlarge-v2-uncased) | 95.9 | 95.9 | 95.9 | 95.9 |
| Model | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|
| BERT (base-uncased) + textattack | 92.4 | 92.8 | 92.4 | 92.4 |
| RoBERTa (base-uncased) + textattack | 94.3 | 94.3 | 94.3 | 94.3 |
| XLNet (base-uncased) + textattack | 93.7 | 93.8 | 93.7 | 93.7 |
| ALBERT (base-uncased) + textattack | 92.0 | 92.0 | 92.0 | 92.0 |
Please refer to MIT License Copyright (c) 2020 YJiangcm