[Paper Review] Unsupervised Domain Adaptation of Contextualized Embeddings: A Case Study in Early Modern English.

Xiaochuang Han, Jacob Eisenstein|arXiv (Cornell University)|Apr 4, 2019

Topic Modeling15 citations

TL;DR

This paper proposes domain-adaptive fine-tuning to adapt contextualized embeddings like BERT to low-resource, out-of-domain texts—specifically Early Modern English and Twitter—by pretraining them on unlabeled target-domain text via masked language modeling. The method significantly improves sequence labeling performance, especially on out-of-vocabulary words, outperforming standard BERT baselines without requiring labeled data in the target domain.

ABSTRACT

Contextualized word embeddings such as ELMo and BERT provide a foundation for strong performance across a wide range of natural language processing tasks by pretraining on large corpora of unlabeled text. However, the applicability of this approach is unknown when the target domain varies substantially from the pretraining corpus. We are specifically interested in the scenario in which labeled data is available in only a canonical source domain such as newstext, and the target domain is distinct from both the labeled and pretraining texts. To address this scenario, we propose domain-adaptive fine-tuning, in which the contextualized embeddings are adapted by masked language modeling on text from the target domain. We test this approach on sequence labeling in two challenging domains: Early Modern English and Twitter. Both domains differ substantially from existing pretraining corpora, and domain-adaptive fine-tuning yields substantial improvements over strong BERT baselines, with particularly impressive results on out-of-vocabulary words. We conclude that domain-adaptive fine-tuning offers a simple and effective approach for the unsupervised adaptation of sequence labeling to difficult new domains.

Motivation & Objective

• To address the challenge of applying pretrained contextualized embeddings to domains that differ substantially from standard pretraining corpora.
• To explore unsupervised domain adaptation when labeled data is only available in a source domain like newstext and the target domain is distinct and low-resource.
• To improve sequence labeling performance in challenging domains such as Early Modern English and Twitter, where standard BERT models underperform due to domain shift.
• To evaluate whether domain-adaptive fine-tuning on unlabeled target-domain text can effectively adapt contextualized embeddings without requiring labeled data in the target domain.

Proposed method

• Propose domain-adaptive fine-tuning, where BERT-style contextualized embeddings are further pre-trained on unlabeled text from the target domain using masked language modeling.
• Use the same architecture as BERT but continue pretraining on domain-specific text, such as Early Modern English manuscripts or Twitter posts.
• Train the model on a large-scale, unlabeled target-domain corpus to adapt the contextualized representations to domain-specific linguistic patterns.
• Apply the adapted embeddings to sequence labeling tasks, such as named entity recognition or part-of-speech tagging, without fine-tuning on labeled target data.
• Evaluate performance on downstream sequence labeling tasks using only labeled source-domain data for initial BERT pretraining and no labeled target data.
• Compare the domain-adapted model against standard BERT baselines to isolate the impact of domain-adaptive fine-tuning.

Experimental results

Research questions

• RQ1 Can domain-adaptive fine-tuning improve sequence labeling performance in low-resource domains such as Early Modern English and Twitter?
• RQ2 Does unsupervised adaptation of contextualized embeddings via masked language modeling on target-domain text lead to better generalization than standard BERT fine-tuning?
• RQ3 How does domain-adaptive fine-tuning affect performance on out-of-vocabulary words in challenging domains?
• RQ4 To what extent does domain shift between pretraining and target domains limit the performance of standard BERT models?
• RQ5 Is domain-adaptive fine-tuning effective when no labeled data is available in the target domain?

Key findings

• Domain-adaptive fine-tuning yields substantial improvements over strong BERT baselines on sequence labeling tasks in both Early Modern English and Twitter.
• The method shows particularly strong gains on out-of-vocabulary words, indicating better generalization to rare or unseen forms.
• Performance gains are consistent across both domains, demonstrating the method's robustness to diverse linguistic shifts.
• The approach achieves these improvements without requiring any labeled data from the target domain, making it suitable for low-resource settings.
• The results confirm that adapting contextualized embeddings via masked language modeling on target-domain text is an effective and simple strategy for unsupervised domain adaptation.
• The study establishes that domain-adaptive fine-tuning is a viable and effective alternative to supervised fine-tuning when labeled data is unavailable in the target domain.