Tanja Samardžić, University of Geneva, Lecture notes, Autumn 2024

Introduction to NLP

(Traitement automatique du langage naturel - TALN)

These notes should be used as a guide for acquiring the most important notions and terminology in contemporary Natural Language Processing. Most of the notions mentioned in the notes are explained in the listed sources. A few visualisations are included in the notes for a better overview and intuitive understanding. This course includes a practical part too, which is managed on Moodle.

 

Online textbooks:

• Dan Jurafsky and James H. Martin: Speech and Language Processing (3rd ed. draft)
• Jacob Eisenstein: Natural Language Processing
• Yoav Goldberg: A Primer on Neural Network Models for Natural Language Processing
• Tom Mitchell: Machine Learning - an old classic

Blogs and other learning resources:

• Graham Neubig, YouTube channel CMU Advanced NLP
• Benoît Sagot, Apprendre les langues aux machines, cours du Collège de France
• YouTube channel Crash Course Linguistics
• Lena Voita's NLP course
• Hugging Face Course
• Jay Alammar's blog
• Serrano Academy
• YouTube channel 3blue1brown
• YouTube channel Crash Course Linguistics

Topics

1. NLP tasks, data sets, benchmarks

2. Large language models (LLMs), Artificial Intelligence (AI) and Natural language processing (NLP), history of NLP

3. Evaluation, data splits

4. History of language modelling

5. Representing the meaning of words with word2vec

6. Language modelling with Transformers NNs

7. Attention in language modelling

8. Subword tokenization

9. Transfer learning: performing tasks with pre-trained models

10. Cross-lingual transfer and multilingual NLP

11. History of NN architectures in NLP: CNNs, LSTMs

12. What is knowledge about language?