This is the site for "It's evaluation's world, we just live in it.", a fully-online Ph.D. course taught by Asad Sayeed at the University of Gothenburg via the Centre for Linguistic Theory and Studies in Probability (CLASP). The point of the course, in a nutshell, is to take a somewhat wry look at the rather haphazard, insufficiently critiqued concept of evaluation that pervades langauge science and technology.
Ph.D. students from the computational linguistics program at the department of Philosophy, Linguistics, and Theory of Science as well as from other departments and universities that will recognize the credits (7.5 ECTS), are welcome to take the course, and interested people from other institutions are also welcome to attend and present by contacting Asad directly (by Twitter DM or email, figure it out yourself) to obtain the meeting link. Discussion is intended to be highly informal and can cover a paper, a topic area, a novel idea in evaluation, and so on. Credit is given for discussion leadership as well as a relevant small research project.
This page is a work in progress.
- 2020 November 12 15:30 CET - Introductory session and presentation by Asad
- 2020 November 19 - EMNLP conference, holiday
- 2020 November 26 - Swedish Language Technology Conference
- 2020 December 3 15:30 CET - Discussion (led by Asad) of: ** Dror et al. (2018). The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. LREC.
- 2020 December 10 15:30 CET - Discussion on comparing vector embeddings (led by Jean-Philippe Bernardy, possible time change)
- 2020 December 17 15:30 CET - Discussion (led by Vlad Maraev) of Passoneau and Carpenter (2014). The Benefits of a Model of Annotation. TACL.
- 2020 January 21 15:30 CET - Discussion (led by Axel Almquist) of Horel and Giesecke (2019). Significance Tests for Neural Networks. Journal of Machine Learning Research.
- 2020 January 28 15:30 CET - Discussion (led by Nikolai Ilinykh) of Sellam et al. (2020). BLEURT: Learning Robust Metrics for Text Generation. ACL.
- 2020 February 4 15:30 CET - Discussion (led by Adam Ek) of Rethmeier et al. (2020). TX-Ray: Quantifying and Explaining Model-Knowledge Transfer in(Un-)Supervised NLP. UAI.
- 2020 February 11 15:30 CET - Discussion (led by Bill Noble) of Dubossarsky et al. (2017). Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models. EMNLP.
- 2020 February 18 15:30 CET - project proposal presentations; LMER overview given by Asad Sayeed.
- 2020 February 25 15:30 CET - Discussion (led by Vidya Somashekarappa) of Fischer et al. (2018). RT-GENE: Real-Time Eye Gaze Estimationin Natural Environments. ECCV.
There is one, write to Asad to join it.
- Significance Tests for Neural Networks(suggested by Axel)