[Paper Review] Task Residual for Tuning Vision-Language Models

Changyuan Yu, Zhihe Lu+3 more|arXiv (Cornell University)|Nov 18, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Task Residual Tuning (TaskRes), a simple yet effective method for efficient fine-tuning of vision-language models (VLMs) by adding task-specific, prior-independent parameters (the 'task residual') to the frozen pre-trained text-based classifier. TaskRes significantly outperforms existing methods like prompt tuning and adapter-style tuning across 11 benchmark datasets, achieving state-of-the-art performance with minimal implementation effort—requiring only one line of code change.

ABSTRACT

Large-scale vision-language models (VLMs) pre-trained on billion-level data have learned general visual representations and broad visual concepts. In principle, the well-learned knowledge structure of the VLMs should be inherited appropriately when being transferred to downstream tasks with limited data. However, most existing efficient transfer learning (ETL) approaches for VLMs either damage or are excessively biased towards the prior knowledge, e.g., prompt tuning (PT) discards the pre-trained text-based classifier and builds a new one while adapter-style tuning (AT) fully relies on the pre-trained features. To address this, we propose a new efficient tuning approach for VLMs named Task Residual Tuning (TaskRes), which performs directly on the text-based classifier and explicitly decouples the prior knowledge of the pre-trained models and new knowledge regarding a target task. Specifically, TaskRes keeps the original classifier weights from the VLMs frozen and obtains a new classifier for the target task by tuning a set of prior-independent parameters as a residual to the original one, which enables reliable prior knowledge preservation and flexible task-specific knowledge exploration. The proposed TaskRes is simple yet effective, which significantly outperforms previous ETL methods (e.g., PT and AT) on 11 benchmark datasets while requiring minimal effort for the implementation. Our code is available at https://github.com/geekyutao/TaskRes.

Motivation & Objective

• To address the limitations of existing efficient transfer learning (ETL) methods for vision-language models, which either damage prior knowledge or overly rely on it.
• To explicitly decouple prior knowledge from task-specific knowledge during fine-tuning to preserve strong pre-trained representations while enabling flexible adaptation.
• To develop a simple, parameter-efficient method that maintains high performance even with limited downstream data.
• To investigate the relationship between learned task residual magnitude and transfer difficulty across diverse downstream tasks.

Proposed method

• TaskRes keeps the pre-trained text-based classifier weights frozen and introduces a set of learnable, prior-independent parameters as a residual to the original classifier.
• The residual is added directly to the classifier weights via an additive update: W_new = W_base + α × W_residual, where α is a scaling factor.
• The method operates solely on the classifier head, avoiding re-computation of text embeddings or architectural modifications.
• The task residual is trained end-to-end with the rest of the model, while the base classifier remains fixed.
• The approach is parameter-efficient and requires only a single line of code change to implement.
• The method is compatible with various vision encoders (e.g., ResNet, ViT) and works across multiple few-shot settings.

Experimental results

Research questions

• RQ1 How does TaskRes compare to prompt tuning and adapter-style tuning in preserving prior knowledge while adapting to downstream tasks?
• RQ2 What is the relationship between the magnitude of the learned task residual and the difficulty of transferring a pre-trained VLM to a new task?
• RQ3 Can a simple, additive residual mechanism significantly improve few-shot transfer performance without architectural changes?
• RQ4 Does the task residual adapt automatically to the complexity of the downstream task, as indicated by performance gains?
• RQ5 What is the impact of different visual backbones (e.g., ResNet, ViT) on the effectiveness of TaskRes?

Key findings

• TaskRes achieves state-of-the-art performance on 11 diverse benchmark datasets across 1-shot to 16-shot settings, outperforming Zero-shot CLIP, CoOp, CLIP-Adapter, and Tip-Adapter-F.
• On ImageNet with 1-shot learning, TaskRes achieves 64.75% accuracy, surpassing Zero-shot CLIP (58.18%) and CLIP-Adapter (63.59%).
• The magnitude of the learned task residual correlates strongly with relative transfer difficulty, increasing significantly on harder tasks such as EuroSAT (1-shot gain: 23.71%) and DTD (1-shot gain: 7.85%).
• TaskRes shows consistent improvements across different visual backbones, including ResNet-50, ResNet-101, ViT-B/32, and ViT-B/16, with gains up to 2.5% over baseline methods.
• The method achieves these gains with minimal implementation effort—requiring only one line of code to integrate into existing models.
• Despite strong overall performance, TaskRes exhibits negative transfer on two datasets (OxfordPets and Food101), suggesting limitations in high-difficulty, high-precision zero-shot scenarios.