[Paper Review] Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

Iulia Turc, Ming‐Wei Chang+2 more|arXiv (Cornell University)|Aug 23, 2019

Topic Modeling36 references428 citations

TL;DR

The paper shows that pre-training compact Transformer models, followed by distillation from a large teacher and optional fine-tuning, yields competitive or superior performance compared to more elaborate compression methods, across various model sizes and data conditions.

ABSTRACT

Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly, the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

Motivation & Objective

• Demonstrate that pre-training compact models is beneficial for end-task performance under memory and latency constraints.
• Show that combining pre-training with distillation (and optional fine-tuning) is competitive with or superior to existing compression methods.
• Analyze how model size and unlabeled data influence the gains from pre-training and distillation.
• Investigate the interaction between LM pre-training and task-specific distillation when applied sequentially on the same data.
• Make available a suite of pre-trained miniature BERT models to accelerate future research.

Proposed method

• Apply a three-step training process: MLM pre-training on a large unlabeled LM corpus, distillation from a high-capacity teacher using soft labels on unlabeled transfer data, and optional fine-tuning on labeled data.
• Compare Pre-trained Distillation (PD) with baselines: basic training, standard distillation, and pre-training plus fine-tuning (PF).
• Vary 24 compact model sizes (4M to 110M parameters) and assess performance under different unlabeled data sizes and domain similarity.
• Evaluate on GLUE-style tasks and several datasets (MNLI, RTE, SST-2, Book Reviews) to study robustness to transfer data size and domain shift.
• Analyze the compound effect of pre-training and distillation, and compare with concurrent model compression work.

Experimental results

Research questions

• RQ1 Does pre-training Transformer layers on compact models improve end-task performance relative to standard distillation or PF baselines?
• RQ2 How do model size and the size/domain of unlabeled data affect the gains from pre-training and distillation?
• RQ3 Is there a compounded benefit when applying LM pre-training and distillation sequentially on the same data?
• RQ4 How robust is Pre-trained Distillation to transfer set size and domain shift between labeled and unlabeled data?

Key findings

• Pre-training plus distillation (PD) consistently outperforms baselines across multiple tasks and model sizes.
• Depth is more valuable than width for pre-trained compact models; pre-training enables better utilization of depth.
• PD can match or exceed teacher performance with far smaller models and less transfer data than vanilla distillation.
• PF is competitive when the transfer set is not much larger than the labeled set, but PD remains superior overall and especially robust to transfer data variations.
• PD is more robust to domain shift between labeled and transfer data than standard distillation, and chaining pre-training with distillation yields additive gains.