[Paper Review] VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding

Xin Yi, Junlong Du+3 more|arXiv (Cornell University)|Dec 14, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes VMT-Adapter, a parameter-efficient transfer learning method for multi-task dense scene understanding that achieves near-$O(1)$ training and inference cost regardless of task count by sharing projections across tasks and using lightweight task-specific knowledge extraction modules. It achieves a 3.96% relative improvement over single-task full fine-tuning while using only ~1% of the pre-trained model's trainable parameters, with VMT-Adapter-Lite reducing parameters to 0.36%.

ABSTRACT

Large-scale pre-trained models have achieved remarkable success in various computer vision tasks. A standard approach to leverage these models is to fine-tune all model parameters for downstream tasks, which poses challenges in terms of computational and storage costs. Recently, inspired by Natural Language Processing (NLP), parameter-efficient transfer learning has been successfully applied to vision tasks. However, most existing techniques primarily focus on single-task adaptation, and despite limited research on multi-task adaptation, these methods often exhibit suboptimal training and inference efficiency. In this paper, we first propose an once-for-all Vision Multi-Task Adapter (VMT-Adapter), which strikes approximately O(1) training and inference efficiency w.r.t task number. Concretely, VMT-Adapter shares the knowledge from multiple tasks to enhance cross-task interaction while preserves task-specific knowledge via independent knowledge extraction modules. Notably, since task-specific modules require few parameters, VMT-Adapter can handle an arbitrary number of tasks with a negligible increase of trainable parameters. We also propose VMT-Adapter-Lite, which further reduces the trainable parameters by learning shared parameters between down- and up-projections. Extensive experiments on four dense scene understanding tasks demonstrate the superiority of VMT-Adapter(-Lite), achieving a 3.96%(1.34%) relative improvement compared to single-task full fine-tuning, while utilizing merely ~1% (0.36%) trainable parameters of the pre-trained model.

Motivation & Objective

• To address the high computational and storage costs of full fine-tuning large pre-trained vision models for multiple dense prediction tasks.
• To overcome the inefficiency of existing parameter-efficient methods that scale linearly with task count in training and inference.
• To enable effective cross-task knowledge transfer while preserving task-specific representations in a unified, scalable framework.
• To reduce trainable parameters significantly without sacrificing performance, especially in multi-task settings.
• To develop a method that is flexible across different backbone architectures and decoder structures.

Proposed method

• VMT-Adapter introduces shared projections that are independent of task count, enabling $O(1)$ training and inference cost scaling with respect to the number of tasks.
• It employs task-specific knowledge extraction modules with minimal parameters to preserve individual task representations.
• The method integrates into any transformer intermediate layer and unifies learning of task-generic and task-specific features.
• VMT-Adapter-Lite further reduces parameters by sharing learnable weights between down- and up-projection matrices in the adapter module.
• The adapter is applied at multiple layers in the transformer encoder and decoder, enabling multi-scale feature adaptation.
• The framework is evaluated on four dense scene understanding tasks: semantic segmentation, surface normal estimation, saliency detection, and depth estimation.

Experimental results

Research questions

• RQ1 Can a parameter-efficient method achieve high performance across multiple dense scene understanding tasks while maintaining low training and inference cost?
• RQ2 How can cross-task knowledge be effectively shared without compromising task-specific representation learning?
• RQ3 Can the number of trainable parameters be reduced to less than 1% of the full model while outperforming full fine-tuning?
• RQ4 Does parameter sharing between down- and up-projections in adapters significantly reduce model size with minimal performance drop?
• RQ5 How does the method generalize across different backbone architectures and decoder designs?

Key findings

• VMT-Adapter achieves a 3.96% relative improvement over single-task full fine-tuning across four dense scene understanding tasks, using only 1.13M trainable parameters (~1% of the full model).
• VMT-Adapter-Lite reduces trainable parameters to 0.40M (0.36% of the full model) while still achieving a 1.34% relative improvement over full fine-tuning.
• The method outperforms strong baselines such as Polyhistor (+2.34% improvement) and Hyperformer (+2.64%), despite using significantly fewer parameters.
• VMT-Adapter generalizes well across different backbones: it achieves +7.10% average improvement on SwinTransformer-Base, outperforming full fine-tuning.
• The method maintains strong performance across different decoders, including HRNet-V2’s high-resolution decoder, with +2.41% relative improvement.
• Ablation studies show that performance improves with higher down-projection ratio (ρ) up to ρ=8, and setting m=3 in VMT-Adapter-Lite offers the best trade-off between parameters and accuracy.