[Paper Review] To Tune or Not To Tune? How About the Best of Both Worlds?

Ran Wang, Haibo Su+3 more|arXiv (Cornell University)|Jul 9, 2019

Topic Modeling36 references15 citations

TL;DR

This paper proposes a two-stage adaptation method for pre-trained language models: first training a task-specific head with frozen BERT parameters, then jointly fine-tuning both the head and BERT. The method achieves consistent accuracy gains—4.7% on semantic similarity, 0.99% on sequence labeling, and 0.72% on text classification—demonstrating that stack-and-finetune outperforms standard fine-tune-only strategies.

ABSTRACT

The introduction of pre-trained language models has revolutionized natural language research communities. However, researchers still know relatively little regarding their theoretical and empirical properties. In this regard, Peters et al. perform several experiments which demonstrate that it is better to adapt BERT with a light-weight task-specific head, rather than building a complex one on top of the pre-trained language model, and freeze the parameters in the said language model. However, there is another option to adopt. In this paper, we propose a new adaptation method which we first train the task model with the BERT parameters frozen and then fine-tune the entire model together. Our experimental results show that our model adaptation method can achieve 4.7% accuracy improvement in semantic similarity task, 0.99% accuracy improvement in sequence labeling task and 0.72% accuracy improvement in the text classification task.

Motivation & Objective

• To investigate whether a two-stage training strategy—pre-training a task head with frozen BERT, then jointly fine-tuning both—yields better performance than standard fine-tune-only methods.
• To evaluate the impact of architectural design on top of pre-trained language models, especially in comparison to simple head addition or word embedding-based strategies.
• To explore the role of overfitting mitigation and gradient flow in deep transfer learning with large pre-trained models.
• To determine whether stacking complex neural networks on top of BERT can improve performance beyond simple fine-tuning, particularly when tailored to BERT's representation structure.

Proposed method

• First, train a task-specific neural network head on top of frozen BERT parameters until convergence, monitoring for overfitting via training vs. validation accuracy gap.
• Then, jointly fine-tune both the BERT encoder and the task-specific head together for improved parameter alignment.
• Use a two-phase training schedule: initial head training with frozen BERT, followed by end-to-end fine-tuning with all parameters updated.
• Design and evaluate multiple network architectures on top of BERT, including modified BIMPM and a Sim-Transformer variant that replaces bi-LSTM with a transformer in the matching layer.
• Apply early stopping during the first phase based on validation performance to prevent overfitting before joint fine-tuning.
• Compare results against standard fine-tune-only baselines and BERT-only fixed representations across three NLP tasks: semantic similarity, sequence labeling, and text classification.

Experimental results

Research questions

• RQ1 Does a two-stage training strategy—pre-train head with frozen BERT, then jointly fine-tune—yield better performance than standard fine-tune-only methods?
• RQ2 Can architectural modifications on top of BERT, such as replacing bi-LSTMs with transformers, lead to performance gains when combined with the proposed two-stage training?
• RQ3 How does the performance of stack-and-finetune compare to fine-tune-only across different NLP tasks with varying data scales?
• RQ4 What training strategies are most effective in preventing overfitting when fine-tuning large pre-trained models with high parameter counts?
• RQ5 To what extent does the depth and structure of pre-trained models influence the design and effectiveness of top-level neural networks?

Key findings

• The proposed stack-and-finetune method achieved a 4.7% accuracy improvement on the semantic similarity task compared to the fine-tune-only baseline.
• On the sequence labeling task, the method improved accuracy by 0.99% over the fine-tune-only approach.
• In the text classification task, the method yielded a 0.72% accuracy gain compared to fine-tuning only.
• The Sim-Transformer model, which replaces the bi-LSTM in BIMPM with a transformer, outperformed BIMPM after joint fine-tuning, despite lower performance when BERT was frozen.
• Removing the first bi-LSTM layer in BIMPM had minimal impact on performance, indicating that BERT’s representations are already highly expressive.
• The two-stage training strategy effectively reduced overfitting and improved generalization, especially when combined with careful monitoring of training and validation accuracy gaps.