[Paper Review] Retrieval-Enhanced Visual Prompt Learning for Few-shot Classification

Jintao Rong, Hao Chen+4 more|arXiv (Cornell University)|Jun 4, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes RePrompt, a retrieval-augmented visual prompt learning framework that enhances few-shot image classification by caching and retrieving relevant training embeddings to guide prompt tuning. By integrating retrieval into multiple stages of prompt learning—input, intermediate layers, and inference—RePrompt achieves state-of-the-art performance across 11 vision datasets and 4 domain generalization benchmarks, especially excelling in low-shot and high-domain-gap scenarios.

ABSTRACT

The Contrastive Language-Image Pretraining (CLIP) model has been widely used in various downstream vision tasks. The few-shot learning paradigm has been widely adopted to augment its capacity for these tasks. However, current paradigms may struggle with fine-grained classification, such as satellite image recognition, due to widening domain gaps. To address this limitation, we propose retrieval-enhanced visual prompt learning (RePrompt), which introduces retrieval mechanisms to cache and reuse the knowledge of downstream tasks. RePrompt constructs a retrieval database from either training examples or external data if available, and uses a retrieval mechanism to enhance multiple stages of a simple prompt learning baseline, thus narrowing the domain gap. During inference, our enhanced model can reference similar samples brought by retrieval to make more accurate predictions. A detailed analysis reveals that retrieval helps to improve the distribution of late features, thus, improving generalization for downstream tasks. Reprompt attains state-of-the-art performance on a wide range of vision datasets, including 11 image datasets, 3 video datasets, 1 multi-view dataset, and 4 domain generalization benchmarks.

Motivation & Objective

• Address the poor generalization of prompt learning in few-shot and domain-shifted settings, especially when intra-class visual variance is high.
• Overcome the limitations of fixed or learnable prompt templates in adapting frozen vision-language models like CLIP to low-resource downstream tasks.
• Improve zero-shot and few-shot performance by incorporating external knowledge from retrieved similar examples during prompt learning.
• Enable effective adaptation to unseen domains through retrieval-based knowledge infusion, even with only 1–16 shots per class.
• Design a unified framework that integrates retrieval into prompt learning at input, intermediate, and output stages, with end-to-end training guidance.

Proposed method

• Construct a retrieval database using frozen image encoder features (keys) and corresponding labels or features (values) from the few-shot training set.
• Use maximum inner product search (MIPS) to retrieve top-K most similar training samples based on query image features.
• Integrate retrieved knowledge into the visual prompt learning process via two strategies: (1) inserting retrieval-enhanced visual prompts into multiple layers of the image encoder, and (2) applying a parametric k-NN adapter to refine classification logits.
• Apply a differentiable, prior-distribution-based loss guided by retrieval results to train the model end-to-end, balancing retrieval influence and model confidence.
• Use the CLIP ViT-B/16 image encoder as the retrieval encoder ($e_R$), which outperforms other encoders like Timm’s ViT-B/16 in retrieval quality.
• Dynamically adjust the depth of retrieval prompt insertion (J) and the temperature hyperparameter ($\gamma$) to control retrieval influence and avoid noise.

Experimental results

Research questions

• RQ1 Can retrieval of similar training examples improve few-shot visual prompt learning when domain gaps increase?
• RQ2 How does integrating retrieval into different stages of prompt learning (input, middle, output) affect model performance?
• RQ3 What is the optimal depth and temperature ($\gamma$) for retrieval-enhanced prompt insertion to balance knowledge gain and noise?
• RQ4 Does retrieval-based prompt learning generalize better than standard prompt learning on out-of-distribution and fine-grained datasets?
• RQ5 How does intra-class visual variance affect the effectiveness of retrieval-augmented prompt learning?

Key findings

• RePrompt achieves state-of-the-art performance on 11 few-shot vision datasets, including challenging benchmarks like FGVC Aircraft, ImageNet, and Oxford Pets, with gains up to 94.58% accuracy on Oxford Pets under 16-shot.
• On ImageNet, RePrompt achieves 74.57% accuracy with retrieval-enhanced prompts inserted into the first 3 layers, outperforming baseline CoOP and VLPT.
• On domain generalization benchmarks (ImageNet-Sketch, ImageNet-A, ImageNet-R, ImageNetV2), RePrompt demonstrates superior robustness and generalization, confirming its effectiveness in unseen domains.
• The optimal insertion depth (J) varies by dataset: 5 layers for FGVC Aircraft, 3 for ImageNet, and 6 for Oxford Pets, indicating that moderate retrieval integration is most effective.
• Lower $\gamma$ values (stronger retrieval influence) improve performance, showing that retrieval augmentation is consistently beneficial when properly tuned.
• Using CLIP’s ViT-B/16 as the retrieval encoder ($e_R$) yields better results than Timm’s supervised ViT-B/16, especially for hard samples like 'velvet fabric'.