[Paper Review] Finetune like you pretrain: Improved finetuning of zero-shot vision models

Sachin Goyal, Ananya Kumar+3 more|arXiv (Cornell University)|Dec 1, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Finetune Like You Pretrain (FLYP), a simple yet highly effective method for fine-tuning zero-shot vision models like CLIP by using the same contrastive loss during fine-tuning as was used during pretraining. By treating class labels as text prompts and optimizing the contrastive loss between image embeddings and prompt embeddings, FLYP achieves state-of-the-art performance across 16 benchmarks, outperforming prior methods by up to 4.6% in few-shot learning and 2.7% on OOD generalization.

ABSTRACT

Finetuning image-text models such as CLIP achieves state-of-the-art accuracies on a variety of benchmarks. However, recent works like WiseFT (Wortsman et al., 2021) and LP-FT (Kumar et al., 2022) have shown that even subtle differences in the finetuning process can lead to surprisingly large differences in the final performance, both for in-distribution (ID) and out-of-distribution (OOD) data. In this work, we show that a natural and simple approach of mimicking contrastive pretraining consistently outperforms alternative finetuning approaches. Specifically, we cast downstream class labels as text prompts and continue optimizing the contrastive loss between image embeddings and class-descriptive prompt embeddings (contrastive finetuning). Our method consistently outperforms baselines across 7 distribution shifts, 6 transfer learning, and 3 few-shot learning benchmarks. On WILDS-iWILDCam, our proposed approach FLYP outperforms the top of the leaderboard by $2.3\%$ ID and $2.7\%$ OOD, giving the highest reported accuracy. Averaged across 7 OOD datasets (2 WILDS and 5 ImageNet associated shifts), FLYP gives gains of $4.2\%$ OOD over standard finetuning and outperforms the current state of the art (LP-FT) by more than $1\%$ both ID and OOD. Similarly, on 3 few-shot learning benchmarks, our approach gives gains up to $4.6\%$ over standard finetuning and $4.4\%$ over the state of the art. In total, these benchmarks establish contrastive finetuning as a simple, intuitive, and state-of-the-art approach for supervised finetuning of image-text models like CLIP. Code is available at https://github.com/locuslab/FLYP.

Motivation & Objective

• To address the performance gap between in-distribution and out-of-distribution generalization in fine-tuned vision-language models.
• To investigate whether aligning the fine-tuning objective with the pretraining objective improves robustness and accuracy.
• To evaluate whether a minimal, intuitive approach—using contrastive loss during fine-tuning—can outperform complex, multi-stage or ensembling-based baselines.
• To establish a new standard baseline for supervised fine-tuning of zero-shot vision models by demonstrating consistent superiority of contrastive fine-tuning.
• To explore the effectiveness of this approach in few-shot and distribution shift settings where labeled data is scarce.

Proposed method

• Represent each class label as a natural language prompt (e.g., 'a photo of a {class}' reverberates the pretraining prompt style).
• Use the image encoder to embed input images and the text encoder to embed the class-descriptive prompts.
• Optimize the contrastive loss between image embeddings and their corresponding prompt embeddings, mirroring the pretraining objective.
• Perform end-to-end fine-tuning using only the contrastive loss, without any cross-entropy head or additional regularization.
• Train the model on a downstream dataset using standard optimization (e.g., AdamW) with a learning rate schedule.
• Apply the same loss function during fine-tuning as used during pretraining, ensuring consistency in the optimization objective.

Figure 1 : Finetune Like You Pretrain (FLYP ): Given a downstream classification dataset, standard finetuning approaches revolve around using the cross-entropy loss. In this work, we show that simply using the same loss as the pretraining i.e. contrastive loss, with “task supervision” coming from th

Experimental results

Research questions

• RQ1 Does using the same contrastive loss during fine-tuning as used in pretraining lead to better performance than standard cross-entropy fine-tuning?
• RQ2 Can a simple, unified objective (contrastive loss) outperform complex, multi-stage fine-tuning methods like LP-FT or ensembling strategies?
• RQ3 How does FLYP perform on out-of-distribution generalization benchmarks compared to state-of-the-art methods?
• RQ4 Does the contrastive fine-tuning approach scale effectively to few-shot learning scenarios with limited labeled data?
• RQ5 Why does a naive contrastive fine-tuning approach outperform more sophisticated baselines that include regularization or weight ensembling?

Key findings

• On WILDS-iWildCam, FLYP outperforms the top of the leaderboard (ModelSoups, 70+ ensembled models) by 2.3% in-distribution and 2.7% out-of-distribution accuracy using CLIP ViT-L/14@336px.
• On 7 OOD benchmarks (2 WILDS and 5 ImageNet shifts), FLYP achieves 4.2% higher OOD accuracy than standard fine-tuning and over 1% higher than LP-FT on both ID and OOD.
• In few-shot learning, FLYP improves accuracy by up to 4.6% over standard fine-tuning and 4.4% over the SOTA (LP-FT) on binary classification tasks like Rendered-SST2 and PatchCamelyon.
• FLYP consistently dominates the ID-OOD frontier curve across benchmarks, indicating better trade-offs between in-distribution and robustness performance.
• Adding cross-entropy loss to FLYP degrades performance, showing that the contrastive objective alone is sufficient and optimal.
• The method is simple, intuitive, and requires no architectural modifications, hyperparameter tuning, or ensembling, yet achieves SOTA results across diverse settings.

(a) OpenAI CLIP ViT-B/16 finetuned on ImageNet