[Paper Review] Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

Rabeeh Karimi Mahabadi, James Henderson+1 more|arXiv (Cornell University)|Jun 8, 2021

Topic Modeling64 references82 citations

TL;DR

Compacter introduces low-rank, hypercomplex adapter layers for fine-tuning large language models, achieving comparable or better task performance than full fine-tuning while training only a tiny fraction of parameters (~0.047%).

ABSTRACT

Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-inefficient, unstable in low-resource settings, and wasteful as it requires storing a separate copy of the model for each task. Recent work has developed parameter-efficient fine-tuning methods, but these approaches either still require a relatively large number of parameters or underperform standard fine-tuning. In this work, we propose Compacter, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work. Compacter accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers. Specifically, Compacter inserts task-specific weight matrices into a pretrained model's weights, which are computed efficiently as a sum of Kronecker products between shared "slow" weights and "fast" rank-one matrices defined per Compacter layer. By only training 0.047% of a pretrained model's parameters, Compacter performs on par with standard fine-tuning on GLUE and outperforms standard fine-tuning on SuperGLUE and low-resource settings. Our code is publicly available at~\url{https://github.com/rabeehk/compacter}.

Motivation & Objective

• Motivate memory-efficient, parameter-efficient fine-tuning of large-scale pretrained language models (PLMs).
• Develop adapters that reduce trainable parameters while maintaining or improving task performance on NLP benchmarks.
• Leverage Kronecker/low-rank decompositions and hypercomplex multiplication to create compact adapter layers.
• Empirically evaluate on GLUE and SuperGLUE against strong baselines and analyze efficiency trade-offs.

Proposed method

• Insert task-specific weight matrices into pretrained model weights via a sum of Kronecker products between shared slow weights and fast rank-one matrices per Compacter layer.
• Use low-rank parameterization of the fast components to reduce parameters to O(k+d) compared to adapters' O(kd).
• Share the A_i matrices across all layers while making B_i specific to each layer, and further factor B_i into s_i t_i^T with rank r (typically r=1).
• Replace down-projection and up-projection in adapters with LPHM (low-rank parameterized hypercomplex multiplication) layers.
• Maintain a fixed pretrained model during training and update layer norms and adapters (as in standard adapters).
• Optionally evaluate Compacter ++ by removing the Compacter layer after self-attention in each block for further parameter reductions.

Experimental results

Research questions

• RQ1 Can Compacter achieve parity with full fine-tuning while training orders of magnitude fewer parameters?
• RQ2 How do LPHM-based adapters compare to standard adapters and other parameter-efficient fine-tuning methods in GLUE/SuperGLUE?
• RQ3 What are the memory, training time, and accuracy trade-offs of Compacter across different n and rank configurations?
• RQ4 Is sharing A_i across layers and using low-rank B_i sufficient to retain performance across tasks and resource settings?

Key findings

• Compacter achieves comparable or better performance than full fine-tuning on GLUE and SuperGLUE while training only 0.073% of parameters (Compacter) and 0.047% (Compacter ++).
• Compacter outperforms several parameter-efficient baselines (including adapters, Adapter-LowRank, and prompt-tuning variants) on GLUE/SuperGLUE benchmarks.
• LPHM-based layers reduce parameter count to O(k+d) versus O(kd) for standard adapters, with shared A_i across layers and rank-one B_i factors.
• In low-resource settings, Compacter and Compacter ++ can outperform standard fine-tuning in terms of accuracy with far fewer trainable parameters.
• Compacter ++ can achieve near-parity with full fine-tuning while using the smallest memory footprint among the evaluated methods in many settings.