[Paper Review] TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter

Binjie Zhang, Yixiao Ge+3 more|arXiv (Cornell University)|Jun 22, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper proposes TaCA, a task-agnostic compatible adapter that enables seamless hot-plugging upgrades of visual foundation models without retraining downstream modules. By aligning the latent representations of a new foundation model with those of an older one through parameter-efficient tuning, TaCA achieves performance gains across video-text retrieval, video classification, and visual question answering, with improvements up to +5.1% on MSR-VTT and +2.2% on Kinetics-400.

ABSTRACT

Visual foundation models like CLIP excel in learning feature representations from extensive datasets through self-supervised methods, demonstrating remarkable transfer learning and generalization capabilities. A growing number of applications based on visual foundation models are emerging, including innovative solutions such as BLIP-2. These applications employ pre-trained CLIP models as upstream feature extractors and train various downstream modules to accomplish diverse tasks. In situations involving system upgrades that require updating the upstream foundation model, it becomes essential to re-train all downstream modules to adapt to the new foundation model, which is inflexible and inefficient. In this paper, we introduce a parameter-efficient and task-agnostic adapter, dubbed TaCA, that facilitates compatibility across distinct foundation models while ensuring enhanced performance for the new models. TaCA allows downstream applications to seamlessly integrate better-performing foundation models without necessitating retraining. We conduct extensive experimental validation of TaCA using different scales of models with up to one billion parameters on various tasks such as video-text retrieval, video recognition, and visual question answering. The results consistently demonstrate the emergent ability of TaCA on hot-plugging upgrades for visual foundation models. Codes and models will be available at https://github.com/TencentARC/TaCA.

Motivation & Objective

• Address the inefficiency and high cost of retraining downstream modules when upgrading visual foundation models.
• Enable seamless integration of improved foundation models into existing systems without modifying downstream components.
• Develop a task-agnostic adaptation mechanism that ensures compatibility between new and old foundation models in the latent space.
• Achieve enhanced performance on downstream tasks by upgrading to stronger foundation models while preserving compatibility with existing downstream modules.
• Demonstrate the feasibility and effectiveness of hot-plugging upgrades in real-world multimodal systems using modular frameworks like Flamingo and BLIP-2.

Proposed method

• Train a lightweight, parameter-efficient adapter (TaCA) on top of a new visual foundation model to align its latent representations with those of the original model.
• Use a dimension-alignment projector to project the new model's features into the same space as the old model’s representations.
• Optimize TaCA using a distillation loss that minimizes the L2 distance between the new and old visual features, derived solely from the old model.
• Combine the distillation loss with a contrastive loss using cross-attention between the new visual features and the old text features to preserve cross-modal alignment.
• Freeze the backbone parameters of both the old and new models, and only fine-tune the adapter and projector components during training.
• Apply the method in modular frameworks such as Flamingo and BLIP-2, replacing the visual encoder with the TaCA-enhanced model while keeping all downstream modules unchanged.

Experimental results

Research questions

• RQ1 Can a parameter-efficient adapter be designed to enable hot-plugging upgrades of visual foundation models without retraining downstream modules?
• RQ2 To what extent can a new foundation model improve downstream performance when integrated via a task-agnostic adapter?
• RQ3 How does the placement of TaCA adapters within the model architecture affect performance and compatibility?
• RQ4 Does the proposed method maintain or enhance cross-modal alignment (e.g., image-text) when upgrading to a stronger foundation model?
• RQ5 Can TaCA generalize across diverse downstream tasks such as video-text retrieval, video classification, and visual question answering?

Key findings

• Replacing ViT-B/16 with TaCA-ViT-H/14 improved text-to-video retrieval performance on MSR-VTT by +5.1% in R@1.
• The same upgrade achieved a +2.2% improvement in top-1 accuracy on Kinetics-400 for video classification.
• On the VQAv2 benchmark, TaCA-ViT-H/14 outperformed OpenFlamingo (ViT-L/14) by +0.6% in zero-shot visual question answering.
• Inserting TaCA adapters into all layers (1–24) of the ViT-H/14 model yielded the best performance, with a +2.6% gain over the baseline on MSR-VTT.
• Placing adapters in shallow layers (1–12) led to better performance than deep layers (13–24), indicating that early-layer adaptation is more effective for representation alignment.
• The method achieved consistent performance gains across multiple frameworks, including Flamingo and BLIP-2, demonstrating broad compatibility and scalability.