[Paper Review] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

Renrui Zhang, Rongyao Fang+6 more|arXiv (Cornell University)|Nov 6, 2021

Multimodal Machine Learning ApplicationsComputer Science66 references128 citations

TL;DR

Tip-Adapter constructs a training-free, non-parametric two-layer MLP adapter from a few-shot cache to augment CLIP, achieving competitive few-shot performance and fast convergence compared to training-based adapters.

ABSTRACT

Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations by using large-scale contrastive image-text pairs. It shows impressive performance on zero-shot knowledge transfer to downstream tasks. To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification. However, such a process still needs extra training and computational resources. In this paper, we propose extbf{T}raining-Free CL extbf{IP}- extbf{Adapter} ( extbf{Tip-Adapter}), which not only inherits CLIP's training-free advantage but also performs comparably or even better than CLIP-Adapter. Tip-Adapter does not require any back propagation for training the adapter, but creates the weights by a key-value cache model constructed from the few-shot training set. In this non-parametric manner, Tip-Adapter acquires well-performed adapter weights without any training, which is both efficient and effective. Moreover, the performance of Tip-Adapter can be further boosted by fine-tuning such properly initialized adapter for only a few epochs with super-fast convergence speed. We conduct extensive experiments of few-shot classification on ImageNet and other 10 datasets to demonstrate the superiority of proposed Tip-Adapter. The code will be released at \url{https://github.com/gaopengcuhk/Tip-Adapter}.

Motivation & Objective

• Motivate improving CLIP’s few-shot capabilities without full adapter finetuning or prompt design.
• Propose a training-free, cache-based adapter to fuse few-shot knowledge with pre-trained CLIP features.
• Demonstrate competitive few-shot classification performance across diverse datasets and backbones.
• Show that fine-tuning initialized by the cache further boosts performance with fast convergence.

Proposed method

• Append a two-layer MLP adapter to CLIP with a residual connection.
• Construct a key-value cache from the K-shot training set, where keys are CLIP visual features and values are one-hot labels.
• Set adapter weights W1 and W2 directly from the cache (W1 = F_train, W2 = L_train^T) to make the adapter training-free.
• Compute test-time logits as a combination of cached-propagated predictions and pre-trained CLIP predictions, balanced by a residual ratio alpha.
• Optionally unfreeze W1 and fine-tune for a few epochs (e.g., 20) to further improve performance with fast convergence.
• Use a new activation phi(x) = exp(-beta(1 - x)) to modulate affinities in the cache retrieval.

Experimental results

Research questions

• RQ1 Can a training-free, cache-based adapter match or surpass the performance of SGD-finetuned CLIP-Adapter in few-shot classification?
• RQ2 How does integrating a few-shot cache with CLIP affect zero-shot and few-shot transfer across diverse datasets and backbones?
• RQ3 Does a small amount of fine-tuning from a cache-initialized state yield faster convergence and higher accuracy?

Key findings

• Tip-Adapter achieves competitive few-shot performance compared to CLIP-Adapter without any training.
• Tip-Adapter-F (with a few epochs of fine-tuning) surpasses all compared methods across multiple datasets and backbones.
• The cache-based initialization enables fast convergence, requiring substantially fewer epochs than CLIP-Adapter (e.g., 20 vs 200).
• Performance gains from the cache increase with more shots, but the gains diminish as the cache size is fixed (16 in experiments).
• A residual ratio alpha balances adaptation and prior CLIP knowledge, with an optimal value around alpha ≈ 1.0 in their ablations.