Skip to content

Commit

Permalink
Author middle names in 2023.genbench-1.4 & 2021.blackboxnlp-1.38 (#3951)
Browse files Browse the repository at this point in the history
  • Loading branch information
nschneid authored Oct 14, 2024
1 parent 7bdac50 commit b7590a6
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
2 changes: 1 addition & 1 deletion data/xml/2021.blackboxnlp.xml
Original file line number Diff line number Diff line change
Expand Up @@ -514,7 +514,7 @@
</paper>
<paper id="38">
<title>The Acceptability Delta Criterion: Testing Knowledge of Language using the Gradience of Sentence Acceptability</title>
<author><first>Héctor</first><last>Vázquez Martínez</last></author>
<author><first>Héctor Javier</first><last>Vázquez Martínez</last></author>
<pages>479–495</pages>
<abstract>Any test that promises to assess Human Knowledge of Language (KoL) for any statistically-based Language Model (LM) must meet three requirements: (1) comprehensive coverage of linguistic phenomena; (2) replicable and statistically-vetted human judgement data; and (3) test the LM’s ability to track the gradience of sentence acceptability. To this end, we propose here the LI-Adger dataset: a comprehensive collection of 519 sentence types (4177 sentences) spanning the field of current generative linguistics, accompanied by attested and replicable human acceptability judgements (Sprouse &amp; Almeida, 2012; Sprouse et al. 2013; Sprouse &amp; Almeida, 2017). Finally, we posit the Acceptability Delta Criterion (ADC), an evaluation metric that tests how well a LM can track changes in human acceptability judgements across minimal pairs instead of testing whether the LM assigned a greater likelihood to the expert-labeled acceptable sequence of a minimal pair (S_1 &gt; S_2). We benchmark six different BERT (Devlin et al. 2018) models and a baseline trigram model with the ADC. Although the best performing BERT model scores 94%, and the trigram scores 75% classification accuracy under the traditional metric, performance drops precipitously to 38% for BERT and 30% for the trigram model under the ADC. Adopting the ADC reveals how much harder it is for LMs to track the gradience of acceptability across minimal pairs. With this work, we propose and provide the three necessary requirements for a comprehensive linguistic analysis and test of the apparently Human KoL exhibited by LMs that we believe is currently missing in the field of Computational Linguistics.</abstract>
<url hash="db5a7389">2021.blackboxnlp-1.38</url>
Expand Down
4 changes: 2 additions & 2 deletions data/xml/2023.genbench.xml
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@
</paper>
<paper id="4">
<title>Evaluating Neural Language Models as Cognitive Models of Language Acquisition</title>
<author><first>Héctor</first><last>Vázquez Martínez</last><affiliation>University of Pennsylvania</affiliation></author>
<author><first>Annika</first><last>Lea Heuser</last><affiliation>University of Pennsylvania</affiliation></author>
<author><first>Héctor Javier</first><last>Vázquez Martínez</last><affiliation>University of Pennsylvania</affiliation></author>
<author><first>Annika</first><last>Heuser</last><affiliation>University of Pennsylvania</affiliation></author>
<author><first>Charles</first><last>Yang</last><affiliation>University of Pennsylvania</affiliation></author>
<author><first>Jordan</first><last>Kodner</last><affiliation>Stony Brook University</affiliation></author>
<pages>48-64</pages>
Expand Down

0 comments on commit b7590a6

Please sign in to comment.