[Paper Review] Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

Jason Phang, Thibault Févry+1 more|arXiv (Cornell University)|Nov 2, 2018

Topic Modeling29 references258 citations

TL;DR

STILTs adds a supplementary training phase on labeled intermediate tasks between unsupervised pretraining and target-task fine-tuning, improving GLUE results for BERT, GPT, and ELMo, especially in data-scarce settings.

ABSTRACT

Pretraining sentence encoders with language modeling and related unsupervised tasks has recently been shown to be very effective for language understanding tasks. By supplementing language model-style pretraining with further training on data-rich supervised tasks, such as natural language inference, we obtain additional performance improvements on the GLUE benchmark. Applying supplementary training on BERT (Devlin et al., 2018), we attain a GLUE score of 81.8---the state of the art (as of 02/24/2019) and a 1.4 point improvement over BERT. We also observe reduced variance across random restarts in this setting. Our approach yields similar improvements when applied to ELMo (Peters et al., 2018a) and Radford et al. (2018)'s model. In addition, the benefits of supplementary training are particularly pronounced in data-constrained regimes, as we show in experiments with artificially limited training data.

Motivation & Objective

• Motivate improving robustness and effectiveness of transfer learning for sentence encoders by adding a second-stage supervised pretraining on data-rich tasks.
• Evaluate whether STILTs improves target-task performance on GLUE across three architectures (BERT, GPT, ELMo).
• Identify which intermediate tasks (MNLI, SNLI, QQP, fake-sentence-detection) yield gains and under what data conditions.
• Assess stability and variance across random restarts when using STILTs.

Proposed method

• Pretrain encoders on unlabeled data (language modeling or equivalent).
• Insert a supplementary training phase on an intermediate labeled-data task before target-task fine-tuning (STILTs).
• Fine-tune end-to-end on GLUE tasks; compare with and without STILTs across BERT, GPT, ELMo.
• Use four intermediate tasks: MNLI, SNLI, QQP, and a fake-sentence-detection task based on BooksCorpus.
• Evaluate under full data, and under constrained data regimes (1k and 5k target-task examples).
• Provide results on GLUE development and test sets; analyze stability across random restarts.

Experimental results

Research questions

• RQ1 Does STILTs improve GLUE task performance across BERT, GPT, and ELMo?
• RQ2 Which intermediate tasks most consistently yield gains, and in which target tasks or data regimes?
• RQ3 How does STILTs affect training stability and the number of degenerate runs in small-data settings?
• RQ4 Is the benefit of STILTs larger for sentence-pair tasks than for single-sentence tasks?

Key findings

• STILTs yields significant gains on most GLUE tasks across BERT, GPT, and ELMo, with the largest improvements in data-scarce settings.
• BERT on STILTs achieves state-of-the-art GLUE performance on the full benchmark at the time of writing.
• STILTs reduces variance across random restarts and decreases degenerate runs, especially for the 24-layer BERT model on limited data.
• Sentence-pair tasks tend to benefit more from supplementary training than single-sentence tasks (CoLA and SST show mixed results).
• Improvements are larger for GPT and ELMo than for BERT in several settings, potentially due to architectural differences and pretraining objectives.
• Among the intermediate tasks, MNLI and SNLI often provide robust gains, while the fake-sentence-detection task benefits GPT more broadly; some intermediate tasks can be detrimental for certain target tasks.