[Paper Review] Learning Domain Invariant Prompt for Vision-Language Models

Cairong Zhao, Yubin Wang+5 more|arXiv (Cornell University)|Dec 8, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes MetaPrompt, a meta-learning-based prompt tuning method that generates domain-invariant prompts for vision-language models like CLIP, enabling strong generalization to unseen domains. By using a dual-modality prompt network with asymmetric contrastive loss and episodic training, MetaPrompt achieves state-of-the-art performance on 11 base-to-new generalization and 4 domain generalization benchmarks, significantly outperforming existing methods.

ABSTRACT

Prompt learning is one of the most effective and trending ways to adapt powerful vision-language foundation models like CLIP to downstream datasets by tuning learnable prompt vectors with very few samples. However, although prompt learning achieves excellent performance over in-domain data, it still faces the major challenge of generalizing to unseen classes and domains. Some existing prompt learning methods tackle this issue by adaptively generating different prompts for different tokens or domains but neglecting the ability of learned prompts to generalize to unseen domains. In this paper, we propose a novel prompt learning paradigm that directly generates \emph{domain invariant} prompt that can be generalized to unseen domains, called MetaPrompt. Specifically, a dual-modality prompt tuning network is proposed to generate prompts for input from both image and text modalities. With a novel asymmetric contrastive loss, the representation from the original pre-trained vision-language model acts as supervision to enhance the generalization ability of the learned prompt. More importantly, we propose a meta-learning-based prompt tuning algorithm that explicitly constrains the task-specific prompt tuned for one domain or class to also achieve good performance in another domain or class. Extensive experiments on 11 datasets for base-to-new generalization and 4 datasets for domain generalization demonstrate that our method consistently and significantly outperforms existing methods.

Motivation & Objective

• To address the poor generalization of prompt-tuned vision-language models to unseen domains and classes.
• To overcome the limitation of existing prompt learning methods that fail to explicitly enforce generalization to out-of-domain data.
• To develop a prompt tuning paradigm that generates prompts invariant to domain shifts while preserving performance on in-domain data.
• To leverage meta-learning and episodic training to improve generalization guarantees in few-shot settings.
• To design a dual-modality prompt network that learns image and text prompt vectors jointly with enhanced invariance.

Proposed method

• Proposes a dual-modality prompt tuning network that learns separate continuous prompt vectors for image and text inputs.
• Introduces an asymmetric contrastive loss that uses the pre-trained model’s representation as supervision to enhance prompt generalization.
• Employs a batch-wise episodic training strategy where each episode samples a subset of domains or classes for prompt tuning.
• Applies meta-learning to optimize prompts such that performance on one domain generalizes well to other unseen domains.
• Uses a theoretical analysis based on [43] to justify the generalization bound of O(1/√N) for episodic prompt tuning.
• Implements prompt tuning across multiple layers (2–12) of the vision-language model to study depth effects.

Experimental results

Research questions

• RQ1 Can meta-learning-based prompt tuning improve generalization to unseen domains and classes in vision-language models?
• RQ2 How does the proposed episodic training strategy affect prompt generalization compared to standard fine-tuning?
• RQ3 What is the optimal prompt length and layer configuration for achieving domain-invariant prompt performance?
• RQ4 Does a dual-modality prompt network with asymmetric contrastive loss improve generalization over single-modality or symmetric approaches?
• RQ5 How does MetaPrompt compare to input-conditional prompt learning methods in zero-shot and few-shot generalization settings?

Key findings

• MetaPrompt achieves state-of-the-art performance on 11 base-to-new generalization datasets, with a 12-layer prompt configuration yielding 83.65% accuracy on base classes and 75.48% on new classes.
• On 4 domain generalization datasets, MetaPrompt achieves an average accuracy of 81.20% with 10 layers and 8 prompt vectors, outperforming prior methods.
• The episodic training strategy improves performance by over 3% on FGVCAircraft (base-to-new) and VLCS (domain generalization), demonstrating strong robustness to distribution shift.
• Ablation studies show that 2 prompt vectors per modality yield the best balance between base and new class performance, while 4 vectors are optimal for domain generalization.
• Prompt tuning across all 12 layers yields the best overall performance, with accuracy increasing steadily with depth in both generalization settings.
• The asymmetric contrastive loss significantly improves generalization by leveraging the pre-trained model’s representation as supervision, reducing overfitting to in-domain data.