[Paper Review] Exploring Visual Prompts for Adapting Large-Scale Models

Hyojin Bahng, Ali Jahanian+2 more|arXiv (Cornell University)|Mar 31, 2022

CCD and CMOS Imaging SensorsEngineering106 citations

TL;DR

The paper studies learning a single visual prompt (pixel-space perturbation) to adapt frozen vision and vision-language models, finding CLIP particularly receptive and robust to distribution shifts, with performance competitive to linear probes.

ABSTRACT

We investigate the efficacy of visual prompting to adapt large-scale models in vision. Following the recent approach from prompt tuning and adversarial reprogramming, we learn a single image perturbation such that a frozen model prompted with this perturbation performs a new task. Through comprehensive experiments, we demonstrate that visual prompting is particularly effective for CLIP and robust to distribution shift, achieving performance competitive with standard linear probes. We further analyze properties of the downstream dataset, prompt design, and output transformation in regard to adaptation performance. The surprising effectiveness of visual prompting provides a new perspective on adapting pre-trained models in vision. Code is available at http://hjbahng.github.io/visual_prompting .

Motivation & Objective

• Investigate whether pixel-space visual prompts can adapt frozen pre-trained vision models to new tasks.
• Evaluate prompt-based adaptation across multiple models and 15 datasets.
• Analyze how dataset properties, prompt design, and output mapping influence performance.
• Compare visual prompting to fine-tuning, linear probing, and text prompting across tasks and shifts.

Proposed method

• Learn a single task-specific visual prompt v_phi by backpropagation with frozen model F.
• Use input transformation to create prompted images x + v_phi and optimize phi via cross-entropy loss.
• Apply two output mapping schemes: hard-coded label mapping for vision models and discrete text prompts for CLIP.
• Adopt a padding template with prompt size p = 30 as default across experiments.
• Evaluate on CLIP and three vision models (Instagram-pretrained ResNeXt, BiT-M, RN50) across 12 datasets and distribution shifts (WILDS).
• Compare against fine-tuning, linear probes, and text prompting as baselines.

Experimental results

Research questions

• RQ1 Can a fixed, learnable visual prompt in pixel space enable a frozen model to perform new downstream tasks?
• RQ2 How does visual prompting compare to linear probes and fine-tuning across diverse datasets and pre-trained models?
• RQ3 What dataset properties and prompt designs influence adaptation effectiveness?
• RQ4 Is CLIP particularly amenable to visual prompting compared to standard vision models?
• RQ5 How robust is visual prompting to distribution shifts?

Key findings

• Visual prompting with CLIP achieves competitive performance to linear probes, outperforming text prompts by ~24% on average.
• Across 12 datasets, CLIP with a visual prompt (VP+TP) outperforms CLIP text prompting (TP) on most tasks (e.g., EuroSAT, SVHN, CLEVR).
• Visual prompting reduces average out-of-distribution gap versus linear probe and fine-tuning in WILDS benchmarks (averages: 4.5% and 3.5% respectively).
• Prompt design matters: padding with p=30 generally yields best results; even a single-pixel prompt can improve CLIP by ~3% on EuroSAT.
• For vision models without CLIP, prompting lags behind linear probes, highlighting CLIP’s unique suitability for visual prompts.
• Hard-coded output mapping for vision models can be sensitive to label semantics; CLIP output via text prompts can leverage semantic alignment.