Skip to content

Latest commit

 

History

History
211 lines (169 loc) · 10.1 KB

README.md

File metadata and controls

211 lines (169 loc) · 10.1 KB
Raw

SARN: Inference

Quantifiers and monotonicity (and opposite adjectives) in reasoning tasks

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Usage

source .venv/bin/activate
# Finetuning
python -m sarn.train --output-dir models/bart-mq --log-dir logs/bart-mq facebook/bart-large-mnli data/training.csv
# Inference of two sequences (forwards)
python -m sarn.classify --model models/bart-mq "All dogs jumped over the fence." "Some small dogs jumped over the fence."
# ROC curve (SVG and PDF diagram)
python -m sarn.roc microsoft/deberta-large-mnli data/evaluation.csv
# Model accuracy on dataset
python -m sarn.accuracy models/deberta-mq data/evaluation.csv
# Dataset statistics
python -m sarn.stats data/training.csv
# Language Interpretability Tool
python -m sarn.lit \
  --models "facebook/bart-large-mnli" \
           "microsoft/deberta-large-mnli" \
           "./models/bart-mq" \
           "./models/deberta-mq" \
           "./models/bart-adj" \
           "./models/deberta-adj" \
  --datasets "./data/evaluation.csv" "./data/evaluation-adj.csv" \
  --cache_dir=cache_dir

As model, any valid Huggingface model (local or remote) can be specified that has been finetuned for sequence classification, e.g., facebook/bart-large-mnli, microsoft/deberta-large-mnli or a local path like models/bart-mq.

Tips

Download Models from CoLi Servers

# export COLI_USER=<your name>
scp -r ${COLI_USER:?}@last.cl.uni-heidelberg.de:/mnt/semproj/sem_proj20/proj1/models .

Check if labels do not have typos in datasets

for i in ./data/*.csv; do
  python3 -m sarn.validate_datasets "$i"
done

Datasets

See data/README.md for more information.

  • data/training.csv: Training dataset for quantifiers and monotonicity in reasoning tasks
  • data/evaluation.csv: Evaluation dataset for quantifiers and monotonicity in reasoning tasks
  • data/training.csv: Training dataset for quantifiers and monotonicity in reasoning tasks with opposite adjectives
  • data/evaluation.csv: Evaluation dataset for quantifiers and monotonicity in reasoning tasks with opposite adjectives

Creating the datasets

source .venv/bin/activate

# data/training.csv
wget https://github.com/verypluming/MED/raw/master/MED.tsv
python -m sarn.convert.med
wget https://github.com/verypluming/HELP/raw/master/output_en/pmb_train_v1.0.tsv
python -m sarn.convert.help
cat data/med.csv data/help.csv > data/training.csv

# data/evaluation.csv
wget https://nlp.stanford.edu/~wcmac/downloads/fracas.xml
python -m sarn.convert.fracas
wget -O diagnostic-full.tsv https://www.dropbox.com/s/ju7d95ifb072q9f/diagnostic-full.tsv?dl=1
python -m sarn.convert.superglue
cat data/fracas.csv data/superglue.csv > data/evaluation.csv

# data/training-adj.csv
wget https://github.com/verypluming/MED/raw/master/MED.tsv
python -m sarn.convert.med_adjectives
# manual step here (you may modify sentences where it makes sense):
# - label data/med_adjectives_1.csv by hand (third column)
# - label data/med_adjectives_2.csv by hand (third column)
# - remove fourth column in both files
wget https://nlp.stanford.edu/~wcmac/downloads/fracas.xml
python -m sarn.convert.fracas_adjectives
cat data/med_adjectives_1.csv data/med_adjectives_2.csv data/fracas_adjectives.csv > data/training-adj.csv
python -m sarn.validate_datasets data/training-adj.csv

# data/evaluation-adj.csv
python -m sarn.convert.evaluation_adjectives
# manual step here (you may modify sentences where it makes sense):
# - label data/evaluation-adj.csv by hand (third column)
# - remove fourth column
python -m sarn.validate_datasets data/evaluation-adj.csv

Dataset statistics

Character length

Dataset avg median min max
data/training.csv
Premises 48.26 41 5 478
Hypotheses 48.93 42 5 478
data/evaluation.csv
Premises 79.84 58 26 206
Hypotheses 61.57 50 26 186
data/training-adj.csv
Premises 48.93 44 14 212
Hypotheses 50.78 46 18 210
data/evaluation-adj.csv
Premises 100.19 83 25 189
Hypotheses 86.62 69 35 189

Word length

Dataset avg median min max
data/training.csv
Premises 9.98 9 2 83
Hypotheses 10.10 9 2 83
data/evaluation.csv
Premises 13.03 10 5 34
Hypotheses 10.14 9 5 30
data/training-adj.csv
Premises 8.89 8 3 29
Hypotheses 9.18 9 3 29
data/evaluation-adj.csv
Premises 15.44 12 5 31
Hypotheses 13.47 11 5 31

Labels

Dataset total contradiction neutral entailment
data/training.csv 41'273 0 (0.00%) 20'699 (50.15%) 20'574 (49.85%)
data/evaluation.csv 118 15 (12.71%) 52 (44.07%) 51 (43.22%)
data/training-adj.csv 1'206 420 (34.83%) 749 (62.11%) 37 (3.07%)
data/evaluation-adj.csv 144 47 (32.64%) 84 (58.33%) 13 (9.03%)

Models

  • facebook/bart-large-mnli: Pretrained model of BART finetuned on MultiNLI
  • microsoft/deberta-large-mnli: Pretrained model of DeBERTa finetuned on MultiNLI
  • models/bart-mq: finetuned version of facebook/bart-large-mnli on data/training.csv
  • models/deberta-mq: finetuned version of microsoft/deberta-large-mnli on data/training.csv
  • models/bart-adj: finetuned version of models/bart-mq on data/training-adj.csv
  • models/deberta-adj: finetuned version of models/deberta-mq on data/training-adj.csv

Creating the models

source .venv/bin/activate

# facebook/bart-large-mnli and microsoft/deberta-large-mnli will automatically
# be downloaded from huggingface.co when used

# models/bart-mq
python -m sarn.train --output-dir "models/bart-mq" --log-dir "logs/bart-mq" "facebook/bart-large-mnli" "data/training.csv"

# models/deberta-mq
python -m sarn.train --output-dir "models/deberta-mq" --log-dir "logs/deberta-mq" "microsoft/deberta-large-mnli" "data/training.csv"

# models/bart-adj
python -m sarn.train --output-dir "models/bart-adj" --log-dir "logs/bart-adj" "facebook/bart-large-mnli" "data/training-adj.csv"

# models/deberta-adj
python -m sarn.train --output-dir "models/deberta-adj" --log-dir "logs/deberta-adj" "microsoft/deberta-large-mnli" "data/training-adj.csv"

Model statistics

Accuracy

Model data/evaluation.csv data/evaluation-adj.csv
facebook/bart-large-mnli 65.25% 40.97%
microsoft/deberta-large-mnli 71.19% 47.22%
models/bart-mq 57.63% 34.72%
models/deberta-mq 61.86% 34.72%
models/bart-adj 45.76% 58.33%
models/deberta-adj 42.37% 57.64%

ROC curves

An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. [...] AUC stands for "Area under the ROC Curve." That is, AUC measures the entire two-dimensional area underneath the entire ROC curve (think integral calculus) from (0,0) to (1,1). [...] AUC provides an aggregate measure of performance across all possible classification thresholds. One way of interpreting AUC is as the probability that the model ranks a random positive example more highly than a random negative example.

— Google Machine Learning Crash Course

BART DeBERTa