[Paper Review] Amortized Prompt: Lightweight Fine-Tuning for CLIP in Domain Generalization

Xin Zhang, Yusuke Iwasawa+2 more|arXiv (Cornell University)|Nov 25, 2021

Domain Adaptation and Few-Shot Learning40 references4 citations

TL;DR

This paper proposes Amortized Prompt (AP), a lightweight fine-tuning method for CLIP in domain generalization that generates domain-specific prompts at inference time. By leveraging prompt tuning without updating CLIP’s parameters, AP boosts accuracy from 71.3% to 79.3% on standard DG benchmarks, demonstrating strong performance gains with minimal computation.

ABSTRACT

Domain generalization (DG) is a difficult transfer learning problem aiming to learn a generalizable model to unseen domains. Recent massive pre-trained models such as CLIP and GPT-3, i.e. foundation models (FMs), have been shown to be robust to many distribution shifts and therefore should lead to substantial improvements in DG. In this work, we study generic ways to adopt CLIP for DG problems in image classification, where we evaluate on naive zero-shot learning and full DG learning settings. For the latter, we propose AP (Amortized Prompt), as a novel approach for domain inference in the form of prompt generation. Using several standard datasets on domain generalization benchmark, namely PACS, VLCS, OfficeHome, and TerraIncognita, CLIP provides comparable performance without fine-tuning any parameters, suggesting the applicability and importance of FM in DG. In addition, we show that combining domain prompt inference with CLIP enables AP to outperform strong baselines and the naive CLIP baselines by a large margin, raising accuracy from 71.3\% to 79.3\%. We hope the simplicity and success of our approach emphasizes the importance of and leads to wider more adoption and analysis of foundation models in the field of domain generalization.

Motivation & Objective

• To investigate the potential of foundation models like CLIP in domain generalization (DG) without full fine-tuning.
• To address the challenge of adapting CLIP to unseen domains in image classification under distribution shifts.
• To develop a parameter-efficient method for domain-invariant representation learning using prompt engineering.
• To improve upon naive zero-shot CLIP performance in DG settings through domain-specific prompt generation.

Proposed method

• Proposes Amortized Prompt (AP), a method that generates domain-specific prompts at inference time without updating CLIP’s parameters.
• Uses a prompt encoder to predict optimal text prompts for each domain during inference, enabling domain generalization.
• Applies prompt tuning in a lightweight, parameter-efficient manner, avoiding fine-tuning of CLIP’s vision or text encoders.
• Trains a prompt generator on source domains to generalize to unseen domains during inference.
• Employs a differentiable prompt generation mechanism that is amortized across domains, reducing per-inference cost.
• Combines the generated prompts with CLIP’s zero-shot classification head for end-to-end inference on unseen domains.

Experimental results

Research questions

• RQ1 Can foundation models like CLIP achieve strong performance in domain generalization without fine-tuning?
• RQ2 How effective is prompt-based adaptation in improving CLIP’s generalization across unseen domains?
• RQ3 Can a lightweight, parameter-efficient prompt generation method outperform strong baselines in DG?
• RQ4 What is the impact of domain-specific prompt tuning on zero-shot CLIP performance in DG benchmarks?

Key findings

• CLIP achieves competitive performance (71.3%) on domain generalization benchmarks without any parameter updates, indicating strong zero-shot robustness.
• The proposed Amortized Prompt (AP) method improves accuracy to 79.3% on standard DG benchmarks, significantly outperforming naive CLIP and strong baselines.
• AP achieves this gain with minimal computational overhead, as it does not fine-tune CLIP’s parameters.
• The method generalizes effectively to unseen domains, demonstrating strong domain generalization capability.
• The success of AP highlights the potential of foundation models in DG when combined with efficient prompt engineering.
• The results suggest that prompt-based adaptation is a promising, efficient alternative to full fine-tuning in domain generalization.