[Paper Review] Collaboration of Pre-trained Models Makes Better Few-shot Learner

Renrui Zhang, Bohao Li+5 more|arXiv (Cornell University)|Sep 25, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes CoMo, a few-shot learning framework that collaborates pre-trained models—CLIP (language-contrastive), DINO (vision-contrastive), and DALL-E (language-generative)—to improve few-shot classification. By generating synthetic images via DALL-E and adaptively ensembling predictions through a learnable Multi-Knowledge Adapter, CoMo achieves state-of-the-art performance across 11 datasets without additional annotated data.

ABSTRACT

Few-shot classification requires deep neural networks to learn generalized representations only from limited training images, which is challenging but significant in low-data regimes. Recently, CLIP-based methods have shown promising few-shot performance benefited from the contrastive language-image pre-training. Based on this point, we question if the large-scale pre-training can alleviate the few-shot data deficiency and also assist the representation learning by the pre-learned knowledge. In this paper, we propose CoMo, a Collaboration of pre-trained Models that incorporates diverse prior knowledge from various pre-training paradigms for better few-shot learning. Our CoMo includes: CLIP's language-contrastive knowledge, DINO's vision-contrastive knowledge, and DALL-E's language-generative knowledge. Specifically, CoMo works in two aspects: few-shot data expansion and diverse knowledge ensemble. For one, we generate synthetic images via zero-shot DALL-E to enrich the few-shot training data without any manpower. For the other, we introduce a learnable Multi-Knowledge Adapter (MK-Adapter) to adaptively blend the predictions from CLIP and DINO. By such collaboration, CoMo can fully unleash the potential of different pre-training methods and unify them to perform state-of-the-art for few-shot classification. We conduct extensive experiments on 11 datasets to demonstrate the superiority and generalization ability of our approach.

Motivation & Objective

• To investigate whether combining diverse pre-training paradigms can alleviate data scarcity in few-shot learning.
• To improve few-shot classification performance by leveraging complementary knowledge from CLIP, DINO, and DALL-E.
• To develop a method that enhances representation learning using only limited annotated data and pre-trained models.
• To design a learnable adapter that adaptively fuses predictions from multiple pre-trained models.

Proposed method

• CoMo generates synthetic images using zero-shot DALL-E conditioned on class-specific text prompts to expand few-shot training data.
• It employs a Multi-Knowledge Adapter (MK-Adapter) with dual-key cache to produce and adaptively combine logits from CLIP and DINO.
• The MK-Adapter reweights predictions using distribution similarity between the pre-trained models’ outputs and CLIP’s zero-shot logits.
• The final prediction is an ensemble of CLIP’s zero-shot logits, DINO’s vision-contrastive logits, and the MK-Adapter’s fused output.
• Only the lightweight MK-Adapter is fine-tuned on the expanded few-shot data, preserving the frozen weights of the pre-trained models.
• The framework is trained end-to-end with a cross-entropy loss on the expanded data, enabling effective knowledge distillation.

Experimental results

Research questions

• RQ1 Can combining pre-trained models with distinct pre-training objectives improve few-shot classification beyond single-model baselines?
• RQ2 Does zero-shot image generation from DALL-E effectively expand few-shot training data without manual annotation?
• RQ3 Can adaptive fusion of diverse pre-trained model predictions lead to better generalization than fixed ensemble strategies?
• RQ4 How does the collaboration of CLIP, DINO, and DALL-E compare to existing few-shot learning methods across diverse datasets?

Key findings

• CoMo achieves state-of-the-art performance on 11 few-shot classification datasets, outperforming existing methods including CLIP-Adapter and Tip-Adapter-F.
• On 16-shot ImageNet, CoMo achieves 74.48% top-1 accuracy with ViT-B/16, surpassing the second-best method (Tip-Adapter-F) by 0.79 percentage points.
• The optimal number of synthetic images per class is 4, as increasing the number beyond this point degrades performance due to inclusion of low-quality samples.
• Using CLIP’s zero-shot logits as the distribution baseline for adaptive ensemble yields the best performance, outperforming average or max pooling strategies.
• t-SNE visualization shows that CoMo produces more separable and contrastive class clusters than Tip-Adapter-F, reducing class aliasing.
• Learning curves demonstrate that CoMo converges faster and achieves higher accuracy than single-model baselines when trained over 20 epochs.