[Paper Review] Self-training Improves Pre-training for Natural Language Understanding

Jingfei Du, Édouard Grave+6 more|arXiv (Cornell University)|Oct 5, 2020

Topic ModelingComputer Science43 references46 citations

TL;DR

The paper proposes SentAugment, a data augmentation and self-training approach that retrieves task-relevant in-domain sentences from a large web bank to improve RoBERTa-based NLU models, showing gains over strong baselines across multiple tasks. It also demonstrates benefits for knowledge distillation and few-shot learning.

ABSTRACT

Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.

Motivation & Objective

• Motivate semi-supervised learning for natural language understanding by leveraging unlabeled data beyond in-domain sources.
• Introduce SentAugment to retrieve task-relevant, in-domain sentences from a web-scale sentence bank.
• Demonstrate that self-training with SentAugment complements strong pre-training baselines and improves downstream task performance.
• Explore extensions to knowledge distillation and few-shot learning using open-domain data.

Proposed method

• Construct a large external bank of unlabeled web sentences (CC-100M/1B/5B) for retrieval.
• Train a SentAugment Sentence Encoder (SASE) with a multilingual masked LM objective and triplet loss to produce paraphrase-oriented embeddings.
• For each downstream task, build task embeddings (all-average, label-average, per-sentence) to query the sentence bank for similar sentences.
• Filter retrieved sentences by teacher-model confidence to obtain a manageable, in-domain candidate set.
• Use a RoBERTa-Large teacher fine-tuned on the task to synthetically label retrieved sentences, then train a RoBERTa-Large student with KL-divergence on the synthetic data.
• Optionally apply knowledge distillation with a smaller student model, and evaluate in few-shot settings with augmented data.

Experimental results

Research questions

• RQ1 Can self-training be complementary to pre-training in natural language understanding when using open-domain, in-domain augmented data?
• RQ2 Does SentAugment enable effective domain adaptation without in-domain unlabeled data, and how does it impact standard, few-shot, and distillation tasks?
• RQ3 What are the key factors (retrieval strategy, embedding quality, labeling paradigm) that drive gains from SentAugment-based self-training?
• RQ4 How does increasing the scale of the external sentence bank affect performance and computational cost?

Key findings

• Self-training with SentAugment improves RoBERTa-Large by an average of 1.2% accuracy across six benchmarks over the strong baseline.
• Few-shot learning gains average 3.5% with SentAugment (from 72.0% to 75.5%).
• Knowledge distillation with SentAugment yields up to 85.4% average accuracy, approaching the teacher with significantly fewer parameters.
• Using task-specific, label-average retrieval outperforms all-average retrieval for most tasks (83.1% avg).
• Sentence embeddings based on para-embeddings (SASE) outperform word-average baselines, with SASE achieving 83.1% avg on evaluated tasks.
• Increasing bank size from 50M to 1B lines improves performance; gains saturate beyond 1B up to 5B, with potential benefits for rare domains.
• Using continuous posterior probabilities as synthetic labels (logits) yields better self-training results than discrete labels.