[Paper Review] Pre-training Text Representations as Meta Learning

Shangwen Lv, Yuechen Wang+11 more|arXiv (Cornell University)|Apr 12, 2020

Topic Modeling29 references7 citations

TL;DR

This paper proposes pre-training text representations as model-agnostic meta-learning (MAML), directly optimizing for downstream task performance instead of proxy objectives like language modeling. By introducing a sequence of meta-train steps during pre-training, the method learns better initializations, outperforming BERT across diverse NLP tasks and showing improved convergence from the first fine-tuning epoch.

ABSTRACT

Pre-training text representations has recently been shown to significantly improve the state-of-the-art in many natural language processing tasks. The central goal of pre-training is to learn text representations that are useful for subsequent tasks. However, existing approaches are optimized by minimizing a proxy objective, such as the negative log likelihood of language modeling. In this work, we introduce a learning algorithm which directly optimizes model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps. The standard multi-task learning objective adopted in BERT is a special case of our learning algorithm where the depth of meta-train is zero. We study the problem in two settings: unsupervised pre-training and supervised pre-training with different pre-training objects to verify the generality of our approach.Experimental results show that our algorithm brings improvements and learns better initializations for a variety of downstream tasks.

Motivation & Objective

• To address the mismatch between pre-training proxy objectives (e.g., language modeling) and the true goal of learning useful representations for downstream tasks.
• To investigate whether pre-training can be reinterpreted as meta-learning to improve model initialization for downstream fine-tuning.
• To evaluate the effectiveness of the proposed meta-learning-based pre-training across unsupervised and supervised settings with diverse pre-training objectives.
• To demonstrate that the method can learn better representations from scratch, even with lightweight models like ELMo.
• To show empirically that the meta-learned initialization leads to faster and better convergence during fine-tuning.

Proposed method

• Reformulates pre-training as model-agnostic meta-learning (MAML), where the model is optimized to adapt quickly to downstream tasks after few gradient steps.
• Introduces a sequence of meta-train steps during pre-training, with the model updated on task-specific gradients before a meta-update step.
• Uses a standard multi-task learning objective in BERT as a special case when the number of meta-train steps is zero.
• Applies the meta-learning algorithm to both unsupervised pre-training (e.g., masked language modeling) and supervised pre-training (e.g., sentence pair matching).
• Employs a two-phase process: meta-train steps on downstream task gradients, followed by a meta-update to improve initialization.
• Validates the method on both BERT and ELMo architectures, including training from random initialization.

Experimental results

Research questions

• RQ1 Can pre-training text representations be effectively framed as meta-learning to improve downstream task performance?
• RQ2 How does the number of meta-train steps during pre-training affect the quality of the learned initialization?
• RQ3 Does the proposed meta-learning-based pre-training outperform standard proxy-objective pre-training (e.g., BERT) on diverse downstream tasks?
• RQ4 Can the method learn better representations from scratch, even with lightweight models like ELMo?
• RQ5 Does the meta-learned model achieve better performance at early fine-tuning epochs, indicating a superior initialization?

Key findings

• The proposed meta-learning-based pre-training method outperforms BERT on all evaluated downstream tasks, including SST-2, MNLI, SNLI, CLOTH, and QDC.
• On the QDC dataset, the model with k=10 meta-train steps achieved 86.05% accuracy, surpassing BERT’s 85.44% at convergence.
• The best performance was consistently achieved at k=5 or k=10 meta-train steps, with results dropping at k=20, suggesting an optimal meta-training depth.
• At the first fine-tuning epoch, models with k≥1 achieved higher accuracy than BERT on SST-2, SST-5, and QDC, confirming a better initialization.
• When pre-training ELMo from random initialization using the meta-learning method, the model achieved 88.3% accuracy on SNLI, outperforming the official ELMo (88.0%).
• Using the official ELMo as initialization and applying meta-learning further improved performance to 88.5% on SNLI, showing the method enhances even strong starting points.