[Paper Review] How to Fine-Tune BERT for Text Classification?

Chi Sun, Xipeng Qiu+2 more|arXiv (Cornell University)|May 14, 2019

Topic ModelingComputer Science39 references93 citations

TL;DR

The paper conducts exhaustive experiments to analyze and optimize BERT fine-tuning for text classification, proposing a general three-step fine-tuning pipeline and showing state-of-the-art results on eight datasets.

ABSTRACT

Language model pre-training has proven to be useful in learning universal language representations. As a state-of-the-art language model pre-training model, BERT (Bidirectional Encoder Representations from Transformers) has achieved amazing results in many language understanding tasks. In this paper, we conduct exhaustive experiments to investigate different fine-tuning methods of BERT on text classification task and provide a general solution for BERT fine-tuning. Finally, the proposed solution obtains new state-of-the-art results on eight widely-studied text classification datasets.

Motivation & Objective

• Investigate how to maximize BERT's performance for text classification.
• Propose a general fine-tuning solution consisting of pre-training, task-specific fine-tuning, and optional multi-task training.
• Study effects of long-text handling, layer selection, learning rates, and catastrophic forgetting on fine-tuning.
• Evaluate the impact of within-task and in-domain pre-training on multiple datasets.

Proposed method

• Three-step general fine-tuning pipeline: (1) further pre-train BERT on task-specific or in-domain data; (2) optionally apply multi-task learning if related tasks are available; (3) fine-tune for the target task.
• Experiment with long-text preprocessing (truncation and hierarchical methods) to fit 512-token limits.
• Layer-wise learning rates where lower layers have smaller updates to mitigate catastrophic forgetting.
• Explore within-task, in-domain, and cross-domain further pre-training to boost performance.
• Compare single-task fine-tuning with multi-task fine-tuning when multiple datasets are available.

Experimental results

Research questions

• RQ1 How should BERT be fine-tuned to maximize text classification performance across diverse datasets?
• RQ2 Does further pre-training (within-task, in-domain, cross-domain) improve BERT's downstream classification results?
• RQ3 What are effective strategies for handling long texts, layer selection, and layer-wise learning rates to avoid catastrophic forgetting?
• RQ4 Can multi-task fine-tuning provide gains over single-task fine-tuning in practice?
• RQ5 How does BERT-base compare to BERT-large with task-specific pre-training for text classification tasks?

Key findings

• The last layer of BERT generally yields the best features for text classification.
• Layer-wise decreasing learning rates help mitigate catastrophic forgetting and improve fine-tuning stability.
• Within-task and in-domain further pre-training significantly boost performance across several datasets; cross-domain pre-training provides limited additional gains.
• Multi-task fine-tuning can improve results but its benefits are smaller when substantial task-specific pre-training is already used.
• BERT can achieve state-of-the-art results on eight datasets; task-specific larger models with targeted pre-training outperform several baselines.
• Few-shot scenarios benefit notably from BERT fine-tuning, with further pre-training providing substantial gains at very small data sizes.