[論文レビュー] AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

Shoufa Chen, Chongjian Ge+5 more|arXiv (Cornell University)|May 26, 2022

Advanced Neural Network Applications被引用数 260

ひとこと要約

AdaptFormerは、軽量なボトルネックモジュール（AdaptMLP）を導入し、事前学習済みVision Transformersを効率的に微調整します。画像と動画タスク全体で、新規パラメータが2%未満で、完全微調整と同等またはそれ以上の結果を達成します。

ABSTRACT

Pretraining Vision Transformers (ViTs) has achieved great success in visual recognition. A following scenario is to adapt a ViT to various image and video recognition tasks. The adaptation is challenging because of heavy computation and memory storage. Each model needs an independent and complete finetuning process to adapt to different tasks, which limits its transferability to different visual domains. To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently. It possesses several benefits more appealing than prior arts. Firstly, AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100\% fully fine-tuned models on action recognition benchmarks. Secondly, it can be plug-and-play in different Transformers and scalable to many visual tasks. Thirdly, extensive experiments on five image and video datasets show that AdaptFormer largely improves ViTs in the target domains. For example, when updating just 1.5% extra parameters, it achieves about 10% and 19% relative improvement compared to the fully fine-tuned models on Something-Something~v2 and HMDB51, respectively. Code is available at https://github.com/ShoufaChen/AdaptFormer.

研究の動機と目的

• Motivate universal adaptation of pre-trained ViTs to diverse image and video tasks without full fine-tuning.
• Develop a parameter-efficient adapter (AdaptMLP) inserted parallel to the ViT MLP block.
• Demonstrate scalability and robustness of AdaptFormer across multiple datasets and modalities.

提案手法

• Replace the ViT MLP block with AdaptMLP, a parallel bottleneck module with a down- and up-projection (W_down, W_up) and ReLU, fused via a residual with a learnable scale s.
• Fine-tune only the added AdaptMLP parameters while freezing the pre-trained backbone.
• Infer using the combined frozen backbone plus adapted parameters, allowing a single model to adapt to multiple tasks by loading task-specific AdaptMLP weights.
• Initialize W_down with Kaiming Normal and W_up/biases with zeros to ensure stable start resembling the original function.
• Compare AdaptFormer to linear probing, full fine-tuning, and Visual Prompt Tuning (VPT) on image datasets (CIFAR-100, SVHN, Food-101) and video datasets (Something-Something V2, HMDB51).
• Show that AdaptFormer with varying middle dimension d_hat provides strong performance with <2% tunable parameters.

実験結果

リサーチクエスチョン

• RQ1 Can a lightweight adapter module inserted into ViT encoders enable efficient fine-tuning for diverse downstream visual tasks?
• RQ2 How does AdaptFormer perform relative to full fine-tuning and other parameter-efficient methods across image and video benchmarks?
• RQ3 What are the effects of adapter size and fusion strategy on transfer performance and stability?
• RQ4 Is AdaptFormer robust when scaling the number of tunable parameters and across modality shifts (image vs. video)?

主な発見

• AdaptFormer consistently outperforms linear probing and Visual Prompt Tuning across image and video datasets.
• With only about 0.1–1.6% tunable parameters per model (depending on configuration), AdaptFormer matches or exceeds full fine-tuning in several tasks, and reaches a ~5% to ~10% relative advantage on some video benchmarks.
• On self-supervised ViT-B/16, AdaptFormer-64 surpasses VPT and closes in on full fine-tuning, achieving higher top-1 accuracy on SSv2 while using far fewer tunable parameters.
• AdaptFormer demonstrates strong scalability and robustness as the middle dimension (d_hat) and number of AdaptMLP layers increase, with parallel insertion outperforming sequential.
• Across multiple benchmarks, AdaptFormer achieves substantial gains vs. full fine-tuning in terms of parameter efficiency and often accuracy, e.g., on SSv2 and HMDB51, while maintaining small parameter growth.