[Paper Review] SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification

Peng Fang, Xiaoshan Yang+3 more|arXiv (Cornell University)|Nov 28, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

SgVA-CLIP proposes a semantic-guided visual adaptation framework that enhances vision-language models like CLIP for few-shot image classification by learning discriminative visual features through implicit knowledge distillation, vision-specific contrastive loss, and cross-modal contrastive loss. It achieves state-of-the-art performance, improving accuracy by up to 4.88% over baseline CLIP on miniImageNet and 1.25% on ResNet101.

ABSTRACT

Although significant progress has been made in few-shot learning, most of existing few-shot image classification methods require supervised pre-training on a large amount of samples of base classes, which limits their generalization ability in real world application. Recently, large-scale Vision-Language Pre-trained models (VLPs) have been gaining increasing attention in few-shot learning because they can provide a new paradigm for transferable visual representation learning with easily available text on the Web. However, the VLPs may neglect detailed visual information that is difficult to describe by language sentences, but important for learning an effective classifier to distinguish different images. To address the above problem, we propose a new framework, named Semantic-guided Visual Adapting (SgVA), which can effectively extend vision-language pre-trained models to produce discriminative adapted visual features by comprehensively using an implicit knowledge distillation, a vision-specific contrastive loss, and a cross-modal contrastive loss. The implicit knowledge distillation is designed to transfer the fine-grained cross-modal knowledge to guide the updating of the vision adapter. State-of-the-art results on 13 datasets demonstrate that the adapted visual features can well complement the cross-modal features to improve few-shot image classification.

Motivation & Objective

• Address the limitation of vision-language pre-trained models (VLPs) in few-shot learning, which often overlook fine-grained visual details difficult to describe in text.
• Improve few-shot image classification by enhancing visual representations beyond cross-modal alignment, especially when labeled data is scarce.
• Enable effective adaptation of pre-trained VLPs like CLIP without fine-tuning the vision encoder, preserving generalization while learning task-specific features.
• Comprehensively integrate unimodal visual representation learning with cross-modal alignment to improve classification robustness and accuracy.

Proposed method

• Introduce a visual adapter layer that learns task-specific visual features while keeping the CLIP vision backbone frozen.
• Apply implicit knowledge distillation to transfer fine-grained cross-modal knowledge from the pre-trained CLIP to the visual adapter.
• Optimize the visual adapter using a vision-specific contrastive loss to enhance discriminative power of adapted visual features.
• Simultaneously optimize with a cross-modal contrastive loss to maintain alignment between image and text representations.
• Use a learnable prompt to further improve text embedding quality, enabling better zero-shot transfer to novel classes.
• Train the visual adapter and learnable prompt end-to-end with the frozen CLIP encoder, minimizing parameter updates while maximizing performance gain.

Experimental results

Research questions

• RQ1 Can visual representations adapted through knowledge distillation and contrastive learning improve few-shot image classification beyond standard cross-modal alignment?
• RQ2 How does the integration of unimodal visual feature learning with cross-modal alignment affect performance on few-shot benchmarks?
• RQ3 What is the optimal configuration of the visual adapter (e.g., hidden dimension, temperature) for maximizing few-shot accuracy?
• RQ4 Does SgVA-CLIP generalize across different vision backbones and few-shot settings?
• RQ5 To what extent can the model improve performance without fine-tuning the vision encoder, relying only on a small number of external parameters?

Key findings

• SgVA-CLIP achieves 97.95% accuracy on miniImageNet under the 5-way 1-shot setting, a 4.88% improvement over the CLIP baseline.
• On tieredImageNet, SgVA-CLIP reaches 95.73% accuracy in the 5-way 1-shot setting, outperforming the baseline by 6.01 percentage points.
• The combination of visual adapter and learnable prompt yields the highest performance, with a 2.74% average accuracy gain over CoOp on ViT-B/16.
• The optimal temperature hyperparameter for knowledge distillation is τ₂ = 5, yielding the best performance on both ImageNet and SUN397.
• A visual adapter with 4096 hidden dimensions achieves the best trade-off between representation capacity and redundancy, outperforming smaller and larger configurations.
• Visualization confirms that adapted visual features are more discriminative than pre-trained features, especially in separating visually similar classes like birds with distinct beaks.