[Paper Review] Parameter-Efficient Transfer Learning for NLP

Neil Houlsby, Andrei Giurgiu+6 more|arXiv (Cornell University)|Feb 2, 2019

Topic Modeling48 references144 citations

TL;DR

This paper introduces adapter modules for Transformer models to enable parameter-efficient transfer learning for NLP tasks, achieving near full fine-tuning performance with only a small fraction of task-specific parameters.

ABSTRACT

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.

Motivation & Objective

• Motivate parameter-efficient transfer learning for many NLP tasks arriving sequentially.
• Propose adapter modules inserted into pre-trained Transformers to enable compact, extensible task adaptation.
• Demonstrate that adapters achieve near state-of-the-art performance on GLUE and other NLP tasks with a small parameter footprint.

Proposed method

• Introduce a bottleneck adapter module inserted after each Transformer sub-layer; adapters are small, trainable layers with near-identity initialization.
• Freeze original model parameters; train only adapter parameters and task-specific layer norms and classifier head.
• Add adapters in a two-layer bottleneck configuration to control parameter growth (size m, with input dimension d).
• Attach adapters after the projection back to the input size but before the residual connection, per Transformer layer, and train per-task layer norm parameters.
• Demonstrate performance close to full fine-tuning on GLUE and SQuAD while using a small fraction of trainable parameters (as low as 0.5-8% of original model per layer).
• Compare adapter-tuning to full fine-tuning and top-layer fine-tuning across diverse datasets, showing the parameter-efficiency advantage.

Experimental results

Research questions

• RQ1 Can adapter modules enable sequential, task-by-task learning without re-training the entire model?
• RQ2 How does parameter efficiency of adapter-tuning compare to full fine-tuning across multiple NLP tasks?
• RQ3 What is the impact of adapter size on performance vs. parameter count?
• RQ4 Do adapters generalize beyond classification to tasks like SQuAD?
• RQ5 Which layers contribute most to task adaptation when using adapters?

Key findings

• Adapter-based tuning achieves performance within 0.4% of full fine-tuning on GLUE while using only about 3% of the training parameters per task.
• Across GLUE and 17 additional tasks, adapters yield near state-of-the-art results with a substantial reduction in task-specific parameters (1.3x total for GLUE; 1.19x across 17 tasks).
• Adapter size provides a clear trade-off; even very small adapters (0.5-2% of parameters) deliver strong performance, and higher layers tend to be more task-specific.
• Layer-normalization-only training is far less effective than adapters, underscoring the benefit of modifying deeper representations.
• SQuAD results show adapters achieve competitive performance with far fewer trainable parameters (e.g., 2% adapters give near-best F1; 0.1% adapters still yield strong results).
• Ablation suggests adapters collectively enable performance gains, with higher layers contributing more to task adaptation.