[Paper Review] Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation

Wangbo Zhao, Jiasheng Tang+6 more|arXiv (Cornell University)|Mar 18, 2024

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes Dynamic Tuning (DyT), a novel method for Vision Transformer (ViT) adaptation that enhances both parameter and inference efficiency. DyT introduces a learnable token dispatcher to dynamically skip redundant computation by identifying and bypassing less informative tokens during inference, while using lightweight adapters and a mixture-of-experts design to maintain performance. On VTAB-1K, DyT achieves comparable or better accuracy than full fine-tuning with only 43.52 GFLOPs (71%-85% of baseline FLOPs), demonstrating significant efficiency gains without sacrificing accuracy.

ABSTRACT

Existing parameter-efficient fine-tuning (PEFT) methods have achieved significant success on vision transformers (ViTs) adaptation by improving parameter efficiency. However, the exploration of enhancing inference efficiency during adaptation remains underexplored. This limits the broader application of pre-trained ViT models, especially when the model is computationally extensive. In this paper, we propose Dynamic Tuning (DyT), a novel approach to improve both parameter and inference efficiency for ViT adaptation. Specifically, besides using the lightweight adapter modules, we propose a token dispatcher to distinguish informative tokens from less important ones, allowing the latter to dynamically skip the original block, thereby reducing the redundant computation during inference. Additionally, we explore multiple design variants to find the best practice of DyT. Finally, inspired by the mixture-of-experts (MoE) mechanism, we introduce an enhanced adapter to further boost the adaptation performance. We validate DyT across various tasks, including image/video recognition and semantic segmentation. For instance, DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.

Motivation & Objective

• To address the gap in inference efficiency in existing parameter-efficient fine-tuning (PEFT) methods for Vision Transformers (ViTs), which often do not reduce computation during inference despite improving parameter efficiency.
• To develop a unified approach that simultaneously enhances both parameter and inference efficiency during ViT adaptation, especially for computationally heavy models.
• To enable efficient adaptation across diverse vision tasks, including image recognition, video action recognition, and semantic segmentation, without compromising performance.
• To explore dynamic token pruning mechanisms that are compatible with dense prediction tasks and do not reduce the total number of tokens.
• To investigate the impact of dynamic token skipping on model performance and identify optimal design variants and hyperparameters.

Proposed method

• Introduces a learnable token dispatcher for each ViT block that determines whether each token should be activated or skipped during inference, based on its informativeness.
• Only activated tokens pass through the original transformer block and a lightweight adapter; inactive tokens are processed solely by the adapter, reducing computation.
• Employs a Gumbel-Softmax reparameterization to enable differentiable training of the token dispatcher, allowing end-to-end optimization.
• Uses a bottleneck adapter with a mixture-of-experts (MoE) mechanism to enhance feature representation with negligible additional FLOPs.
• Employs a temperature scheduling strategy for the Gumbel-Softmax to improve training stability and performance, with temperature decaying from 5 to 0.1.
• Designs four model variants to evaluate the impact of different activation rates (r) on performance and efficiency.

Experimental results

Research questions

• RQ1 Can dynamic token skipping during inference reduce computational cost in ViT adaptation without sacrificing accuracy?
• RQ2 How does the performance of DyT compare to full fine-tuning and existing PEFT methods in terms of both accuracy and FLOPs across diverse vision benchmarks?
• RQ3 What is the optimal activation rate and temperature setting for the token dispatcher to balance efficiency and performance?
• RQ4 Can the proposed dynamic tuning mechanism generalize across different vision tasks, including recognition, video understanding, and dense prediction?
• RQ5 How does the MoE-based adapter improve performance compared to a standard adapter under the same parameter and FLOP constraints?

Key findings

• On the VTAB-1K benchmark, DyT achieves an average accuracy of 77.0% with only 43.52 GFLOPs, outperforming full fine-tuning (75.7%) while reducing FLOPs by 29.8%.
• For ViT-B/16 on VTAB-1K, DyT achieves comparable or better performance than existing PEFT methods while consuming only 71%-85% of their FLOPs.
• On the K400 video action recognition benchmark, DyT reduces FLOPs by 37 GFLOPs compared to baseline methods, with improved generalization.
• On the ADE20K semantic segmentation benchmark, DyT outperforms full fine-tuning with a 21 GFLOP reduction, demonstrating strong performance in dense prediction tasks.
• The token dispatcher successfully learns to activate informative tokens—especially in deeper layers—verifying its ability to identify salient visual features.
• Temperature scheduling from 5 to 0.1 improves performance on Food-101 and K400, indicating that adaptive temperature tuning enhances training dynamics.