Skip to content

Latest commit

 

History

History

domain_adaptation

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 

Lecture slides - by Boris Kovarsky

More materials

  • Russian videos (2021): lecture and practice
  • Materials '2020 : lecture, seminar
  • An overview of proxy-label approaches for semi-supervised learning - blog post
  • A Little Review of Domain Adaptation in 2017 - blog post
  • Awesome Transfer Learning - list of papers repo
  • Awesome Domain Adaptation - list of papers repo

Practice

Optional assignment: ./homework.ipynb

Videos

  • The Future of Natural Language Processing by Thomas Wolf [youtube]
  • Tutorial on Domain Adaptation by Hal Daumé III [youtube]
  • Domain Adaptation with Structural Correspondence Learning by John Blitzer [youtube]
  • Learning with Scarce and Biased Data in Natural Language Processing by Barbara Plank [youtube]

Articles

General articles

  • Neural Unsupervised Domain Adaptation in NLP---A Survey by Alan Ramponi, Barbara Plank, 2021 [article]

Theory

  • Learning Bounds for Domain Adaptation Blitzer et al. 2007 [article]
  • A theory of learning from different domains Shai Ben-David, John Blitzer 2010 [article]
  • Analysis of Representations for Domain Adaptation Ben-Daivid et al. 2006 [article]

In-domain vs out-domain generalization

  • Learning and Evaluating General Linguistic Intelligence Yogatama et al. 2019 [article]
  • BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance R. Thomas McCoy, Junghyun Min, Tal Linzen 2019 [article]

Data-centric methods for unsupervised domain adaptation

Unsupervised pre-training

  • Don't Stop Pretraining: Adapt Language Models to Domains and Tasks Gururangan et al. 2020 [article]
  • Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling Xiaochuang Han, Jacob Eisenstein 2020 [article]
  • UDALM: Unsupervised Domain Adaptation through Language Modeling Constantinos Karouzos et al. 2021 [article]

Instance weighting/Data selection

  • Instance Weighting for Domain Adaptation in NLP Jiang, Zhai 2007 [pdf]
  • Instance Weighting for Neural Machine Translation Domain Adaptation Wang et al. 2017 [pdf]
  • Cost Weighting for Neural Machine Translation Domain Adaptation Chen et al. 2017 [pdf]
  • Learning to select data for transfer learning with Bayesian Optimization by Ruder, Plank 2017 [article]

Proxy-label methods

  • Asymmetric Tri-training for Unsupervised Domain Adaptation Saito et al. 2017 [arxiv]
  • Strong Baselines for Neural Semi-supervised Learning under Domain Shift Sebastian Ruder, Barbara Plank 2018 [arxiv]

Back-translation

  • Improving Neural Machine Translation Models with Monolingual Data Sennrich et al. 2016 [arxiv]
  • Iterative Back-Translation for Neural Machine Translation Hoang et al. 2018 [pdf]

Model-centric methods for unsupervised domain adaptation

Deep Feature Alignment/Adversarial methods

  • Unsupervised Domain Adaptation by Backpropagation Yaroslav Ganin, Victor Lempitsky 2014 [arxiv]
  • Learning Transferable Features with Deep Adaptation Networks Long et al. 2015 [arxiv]
  • Adversarial and Domain-Aware BERT for Cross-Domain Sentiment Analysis Du et al. 2020 [article]
  • Domain Adaptation with Structural Correspondence Learning Blitzer et al. 2007 [article]

Pivot

Batch Normalization-like:
  • Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe, Christian Szegedy, 2015 [arxiv]
  • Revisiting Batch Normalization For Practical Domain Adaptation Li et al. 2016 [arxiv]
  • AutoDIAL: Automatic DomaIn Alignment Layers Carlucci et al. 2017 [arxiv]

Knowledge Distillation

  • Distilling the Knowledge in a Neural Network Hinton et al. 2015 [arxiv]
  • Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data Dakwale, Monz 2017 [pdf]