PragmEval : A Pragmatics-Centered Evaluation Framework for Natural Language Understanding

This is a data/code release accompanying this paper:

• Title: "A Pragmatics-Centered Evaluation Framework for Natural Language Understanding"
• Authors: Damien Sileo, Tim Van de Cruys, Camille Pradel and Philippe Muller
• Accepted at LREC2022
• https://arxiv.org/abs/1907.08672

Contents

PragmEval is a compilation of 11 evaluation datasets with a focus on discourse, that can be used for evaluation of English Natural Language Understanding, or as auxiliary training tasks for NLP models.

While the idea of meaning as use permeates NLP, it's not clear that current evaluations fully account for that aspect. Previous evaluation frameworks have no clear way to evaluate how models deal with implicatures or different kinds of speech acts, and are arguably focussing on semantics (Natural Language Inference of Semantic Similarity) rather than use. We propose a discourse-centered evaluation with a focus on meaning as use.

dataset	categories	example	class	#train
PDTB	discourse relation	it was censorship / it was outrageous	conjunction	13k
STAC	discourse relation	what ? / i literally lost	question-answer-pair	11k
GUM	discourse relation	do not drink / if underage in your country	condition	2k
Emergent	stance	a meteorite landed in nicaragua. / small meteorite hits managua	for	2k
SarcasmV2	presence of sarcasm	don't quit your day job / [...] i was going to sell this joke. [...]	sarcasm	9k
SwitchBoard	speech act	well , a little different , actually ,	hedge	19k
MRDA	speect act	yeah that 's that 's that 's what i meant .	acknowledge-answer	14k
Verifiability	verifiability	I've been a physician for 20 years.	verifiable-experiential	6k
Persuasion	C/E/P/S/S/R	Co-operation is essential for team work / lions hunt in a team	low specificity	0.6k
Squinky	I/I/F	boo ya.	uninformative, high implicature, unformal,	4k
EmoBank	V/A/D	I wanted to be there..	low valence, high arousal, low dominance	5k

Instructions

Recommended usage

from datasets import load_dataset
dataset = load_dataset('pragmeval','gum')

Evaluate your model with this script: https://colab.research.google.com/drive/1sg--LF4z7XR1wxAOfp0-3d4J6kQ9nj_A?usp=sharing

Building from sources:

git clone https://github.com/synapse-developpement/PragmEval.git

The preprocessed datasets are available in the pragmeval folder in tsv format.

Run the bash get_data.bash in data to download dataset from the sources

Run the notebook Make PragmEval 1.0 after having specified pragmeval_base_path in the third cell to perform preprocessing and exports.

Citation

@inproceedings{sileo-etal-2022-pragmatics,
    title = "A Pragmatics-Centered Evaluation Framework for Natural Language Understanding",
    author = "Sileo, Damien  and
      Muller, Philippe  and
      Van de Cruys, Tim  and
      Pradel, Camille",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.255",
    pages = "2382--2394",
}

Contact

For further information, you can contact:

damien dot sileo at gmail dot com