[Paper Review] Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?

Yada Pruksachatkun, Jason Phang+7 more|arXiv (Cornell University)|May 1, 2020

Topic Modeling53 references52 citations

TL;DR

The paper conducts a large-scale study of RoBERTa with 110 intermediate–target task pairs and 25 probing tasks to understand when and why intermediate-task training helps natural language understanding, finding that high-level reasoning tasks are generally beneficial and that forgetting pretraining can limit transfer.

ABSTRACT

While pretrained models such as BERT have shown large gains across natural language understanding tasks, their performance can be improved by further training the model on a data-rich intermediate task, before fine-tuning it on a target task. However, it is still poorly understood when and why intermediate-task training is beneficial for a given target task. To investigate this, we perform a large-scale study on the pretrained RoBERTa model with 110 intermediate-target task combinations. We further evaluate all trained models with 25 probing tasks meant to reveal the specific skills that drive transfer. We observe that intermediate tasks requiring high-level inference and reasoning abilities tend to work best. We also observe that target task performance is strongly correlated with higher-level abilities such as coreference resolution. However, we fail to observe more granular correlations between probing and target task performance, highlighting the need for further work on broad-coverage probing benchmarks. We also observe evidence that the forgetting of knowledge learned during pretraining may limit our analysis, highlighting the need for further work on transfer learning methods in these settings.

Motivation & Objective

• Investigate which intermediate tasks most benefit a wide range of target NLU tasks.
• Identify the linguistic skills learned during intermediate-task training that transfer to targets.
• Examine how probing task performance correlates with target-task improvements to explain transfer.
• Assess whether dataset size of intermediate tasks explains transfer differences.
• Explore potential limitations such as catastrophic forgetting during transfer learning.

Proposed method

• Fine-tune RoBERTa on each of 11 intermediate tasks individually.
• Fine-tune the intermediate-task trained models on 10 target tasks and 25 probing tasks separately.
• Evaluate transfer by comparing target-task performance to baselines without intermediate training.
• Use 3 random restarts to obtain 1260 observations across tasks and baselines.
• Apply a hyperparameter sweep for learning rate and dropout, then fix best parameters per task.
• Utilize RoBERTa-Large and standard fine-tuning procedures consistent with prior work.

Experimental results

Research questions

• RQ1 Which intermediate tasks broadly improve performance across diverse target tasks?
• RQ2 What linguistic skills do intermediate tasks teach that aid target tasks, as revealed by probing tasks?
• RQ3 How do probing-task performances relate to target-task improvements, and can they explain transfer benefits?
• RQ4 Does intermediate-task dataset size or forgetting of pretraining constrain transfer effects?

Key findings

• Tasks requiring high-level inference and commonsense reasoning tend to be good intermediate tasks.
• MNLI and CosmosQA-like tasks show positive transfer across many targets; SocialIQA often yields negative transfer.
• Low-level input-preservation skills show little correlation with target-task performance, while higher-level abilities tied to MLM-like tasks correlate more.
• Probing correlations indicate that semantic and coreference-related probes correlate with target performance, whereas many SentEval probes do not.
• Catastrophic forgetting of pretraining may limit transfer; integrating MLM objectives during intermediate training could help mitigate forgetting.
• Degenerate runs are less likely with intermediate-task training, but highly negative transfer can increase degeneracy in some cases.