[Paper Review] Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing

Wei Dong, Dawei Yan+2 more|arXiv (Cornell University)|Oct 10, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Adapter Re-Composing (ARC), a parameter-efficient method for fine-tuning large Vision Transformers by sharing down- and up-projection matrices across layers and learning low-dimensional re-composing coefficients to generate layer-adaptive adapters. The approach reduces adaptation parameters while maintaining strong transfer learning performance across 24 image classification tasks.

ABSTRACT

The advent of high-capacity pre-trained models has revolutionized problem-solving in computer vision, shifting the focus from training task-specific models to adapting pre-trained models. Consequently, effectively adapting large pre-trained models to downstream tasks in an efficient manner has become a prominent research area. Existing solutions primarily concentrate on designing lightweight adapters and their interaction with pre-trained models, with the goal of minimizing the number of parameters requiring updates. In this study, we propose a novel Adapter Re-Composing (ARC) strategy that addresses efficient pre-trained model adaptation from a fresh perspective. Our approach considers the reusability of adaptation parameters and introduces a parameter-sharing scheme. Specifically, we leverage symmetric down-/up-projections to construct bottleneck operations, which are shared across layers. By learning low-dimensional re-scaling coefficients, we can effectively re-compose layer-adaptive adapters. This parameter-sharing strategy in adapter design allows us to significantly reduce the number of new parameters while maintaining satisfactory performance, thereby offering a promising approach to compress the adaptation cost. We conduct experiments on 24 downstream image classification tasks using various Vision Transformer variants to evaluate our method. The results demonstrate that our approach achieves compelling transfer learning performance with a reduced parameter count. Our code is available at \href{https://github.com/DavidYanAnDe/ARC}{https://github.com/DavidYanAnDe/ARC}.

Motivation & Objective

• Address the high cost of adapting large pre-trained Vision Transformers by reducing the number of trainable parameters in fine-tuning.
• Overcome the limitation of existing adapter methods that scale linearly with the number of layers by introducing parameter reusability.
• Explore whether low-rank adaptation matrices can be effectively re-composed using shared basis projections across layers.
• Develop a scalable, lightweight adapter design that maintains performance without increasing inference cost.

Proposed method

• Introduce a low-rank adapter structure using symmetric down-projection and up-projection matrices to form a bottleneck operation.
• Share the down- and up-projection matrices across all layers, reducing the number of unique parameters to be learned.
• Learn layer-specific re-composing coefficients (low-dimensional vectors) to dynamically adjust the adapter's effective projection for each layer.
• Position adapters before both Multi-Head Attention (MHA) and Feed-Forward Network (FFN) modules to enable comprehensive feature adaptation.
• Use a parameter-sharing scheme that ensures the number of adaptation parameters does not grow linearly with the number of layers.
• Maintain inference efficiency by placing adapters in a way that avoids additional computation during inference.

Experimental results

Research questions

• RQ1 Can adaptation parameters be effectively reused across layers in Vision Transformers to reduce the number of trainable parameters?
• RQ2 Does sharing down- and up-projection matrices across layers while learning only re-composing coefficients lead to better parameter efficiency without performance degradation?
• RQ3 How does the choice of bottleneck dimensionality affect the trade-off between model performance and parameter count?
• RQ4 What is the optimal placement of adapters (before or after MHA/FFN) for maximizing transfer learning performance?
• RQ5 How does the number of inserted adapter layers influence the final performance, and does sequential or parallel insertion yield better results?

Key findings

• A bottleneck dimensionality of 50 achieves the best balance between performance and parameter efficiency, with performance degrading at 10 dimensions and worsening at higher dimensions.
• Placing adapters after MHA and/or FFN modules leads to performance degradation compared to placing them before both modules.
• Using both MHA and FFN adapters yields better performance than using only one type, indicating the benefit of multi-module adaptation.
• The parameter-sharing design significantly reduces the number of trainable parameters without performance loss, and non-sharing or asymmetric designs increase parameters without improving results.
• Inserting adapters in the first six layers yields better performance than inserting them in the last six layers, suggesting early layers benefit more from adaptation.
• The parameter-sharing scheme ensures that the number of adaptation parameters does not scale linearly with the number of layers, enabling better scalability for deeper models.