[論文レビュー] Revisiting Few-sample BERT Fine-tuning

Tianyi Zhang, Felix Wu+3 more|arXiv (Cornell University)|Jun 10, 2020

Advanced Neural Network Applications参考文献 57被引用数 55

ひとこと要約

本論文は、少数サンプルでのBERTファインチューニングにおける不安定性を分析し、biased Adamによる偏りのある勾配推定、上位レイヤーの初期化の有害性、固定された反復回数を特定し、debiased Adamやレイヤー再初期化のような対策を提案し、既存の安定化手法を再評価する。

ABSTRACT

This paper is a study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios. We identify several factors that cause this instability: the common use of a non-standard optimization method with biased gradient estimation; the limited applicability of significant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations. We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instability of the process. In light of these observations, we re-visit recently proposed methods to improve few-sample fine-tuning with BERT and re-evaluate their effectiveness. Generally, we observe the impact of these methods diminishes significantly with our modified process.

研究の動機と目的

• Understand causes of instability in few-sample BERT fine-tuning on small datasets.
• Evaluate how optimization choices, initialization, and iteration counts affect stability and performance.
• Propose practical remedies to reduce degenerate runs and improve convergence.
• Re-evaluate existing stabilization methods under corrected optimization settings.

提案手法

• Compare standard BERT fine-tuning with debiased Adam versus biased BERT Adam.
• Investigate re-initialization of top pre-trained layers (pooler and Transformer blocks) during fine-tuning.
• Evaluate impact of training longer than the commonly used three epochs.
• Assess how debiasing and re-init interact with existing stabilization methods (e.g., Mixout, weight decay, intermediate-task transfer).
• Analyze layer-wise parameter changes (L2 distance from initialization) during fine-tuning.

実験結果

リサーチクエスチョン

• RQ1 What causes instability in few-sample BERT fine-tuning on small datasets?
• RQ2 Can debiasing in Adam reduce degenerate runs and variance across seeds?
• RQ3 Does re-initializing top BERT layers improve convergence and performance?
• RQ4 How does training longer than three epochs affect stability and results?
• RQ5 Do existing stabilization methods retain usefulness under unbiased optimization?

主な発見

• Debiased Adam significantly reduces variability and degenerate runs across small datasets.
• Re-initializing top pre-trained layers speeds up convergence and often improves mean performance.
• Longer training beyond three epochs improves stability and performance on several datasets.
• Top-layer overspecialization to pre-training can hinder fine-tuning, and re-init mitigates this by providing a better initialization.
• With debiased optimization, the relative gains of several previously proposed stabilization methods diminish.