[Paper Review] Revisiting Few-sample BERT Fine-tuning

Tianyi Zhang, Felix Wu+3 more|arXiv (Cornell University)|Jun 10, 2020

Advanced Neural Network ApplicationsComputer Science57 references55 citations

TL;DR

The paper analyzes instability in few-sample BERT fine-tuning, identifies biased gradient estimation from biased Adam, detrimental high-layer initialization, and fixed iteration counts, and proposes remedies like debiased Adam and layer re-initialization, reassessing prior stability methods.

ABSTRACT

This paper is a study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios. We identify several factors that cause this instability: the common use of a non-standard optimization method with biased gradient estimation; the limited applicability of significant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations. We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instability of the process. In light of these observations, we re-visit recently proposed methods to improve few-sample fine-tuning with BERT and re-evaluate their effectiveness. Generally, we observe the impact of these methods diminishes significantly with our modified process.

Motivation & Objective

• Understand causes of instability in few-sample BERT fine-tuning on small datasets.
• Evaluate how optimization choices, initialization, and iteration counts affect stability and performance.
• Propose practical remedies to reduce degenerate runs and improve convergence.
• Re-evaluate existing stabilization methods under corrected optimization settings.

Proposed method

• Compare standard BERT fine-tuning with debiased Adam versus biased BERT Adam.
• Investigate re-initialization of top pre-trained layers (pooler and Transformer blocks) during fine-tuning.
• Evaluate impact of training longer than the commonly used three epochs.
• Assess how debiasing and re-init interact with existing stabilization methods (e.g., Mixout, weight decay, intermediate-task transfer).
• Analyze layer-wise parameter changes (L2 distance from initialization) during fine-tuning.

Experimental results

Research questions

• RQ1 What causes instability in few-sample BERT fine-tuning on small datasets?
• RQ2 Can debiasing in Adam reduce degenerate runs and variance across seeds?
• RQ3 Does re-initializing top BERT layers improve convergence and performance?
• RQ4 How does training longer than three epochs affect stability and results?
• RQ5 Do existing stabilization methods retain usefulness under unbiased optimization?

Key findings

• Debiased Adam significantly reduces variability and degenerate runs across small datasets.
• Re-initializing top pre-trained layers speeds up convergence and often improves mean performance.
• Longer training beyond three epochs improves stability and performance on several datasets.
• Top-layer overspecialization to pre-training can hinder fine-tuning, and re-init mitigates this by providing a better initialization.
• With debiased optimization, the relative gains of several previously proposed stabilization methods diminish.