[Paper Review] Diverse Imagenet Models Transfer Better

Niv Nayman, Avram Golbert+3 more|arXiv (Cornell University)|Apr 19, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes that feature diversity, alongside ImageNet accuracy, is a critical factor for transferability in vision models. By introducing Controlled Label Injection (CLI), the authors generate models with both high ImageNet accuracy and high feature diversity, resulting in a calibrated score (CIS) that better predicts transfer performance than accuracy alone across CNNs and Vision Transformers on diverse downstream tasks.

ABSTRACT

A commonly accepted hypothesis is that models with higher accuracy on Imagenet perform better on other downstream tasks, leading to much research dedicated to optimizing Imagenet accuracy. Recently this hypothesis has been challenged by evidence showing that self-supervised models transfer better than their supervised counterparts, despite their inferior Imagenet accuracy. This calls for identifying the additional factors, on top of Imagenet accuracy, that make models transferable. In this work we show that high diversity of the features learnt by the model promotes transferability jointly with Imagenet accuracy. Encouraged by the recent transferability results of self-supervised models, we propose a method that combines self-supervised and supervised pretraining to generate models with both high diversity and high accuracy, and as a result high transferability. We demonstrate our results on several architectures and multiple downstream tasks, including both single-label and multi-label classification.

Motivation & Objective

• To identify factors beyond ImageNet accuracy that improve model transferability.
• To validate that feature diversity is a key predictor of transfer performance, especially for self-supervised models.
• To develop a method that enables controlled trade-offs between accuracy and diversity in pre-trained models.
• To demonstrate that the product of ImageNet accuracy and feature diversity (CIS) correlates more strongly with transferability than accuracy alone.
• To show that CLI-generated models outperform standard fine-tuning and hybrid pretraining approaches on multiple downstream tasks.

Proposed method

• Introduce the Calibrated Imagenet Score (CIS) as a product of ImageNet accuracy and feature diversity: CIS = ImageNet Accuracy × Feature Diversity.
• Propose Controlled Label Injection (CLI), a method to inject label information into self-supervised pre-trained models in a controlled manner to adjust feature diversity and accuracy.
• Use two formulations of feature diversity: (1) inter-class feature variance and (2) intra-class feature variance, to measure diversity across features.
• Train models using CLI with varying control cycle values (T), allowing gradual injection of supervised signals while preserving self-supervised invariances.
• Evaluate models on 14 single-label and multi-label classification datasets, including MS-COCO, to measure transferability via linear probing and fine-tuning.
• Compare CLI-generated models against standard supervised, self-supervised, and hybrid pretraining baselines to isolate the effect of diversity and accuracy.

Experimental results

Research questions

• RQ1 Does feature diversity contribute independently to transferability beyond ImageNet accuracy?
• RQ2 Can a hybrid pretraining strategy that balances supervised and self-supervised signals improve transfer performance?
• RQ3 Is there an optimal trade-off between ImageNet accuracy and feature diversity that maximizes transferability?
• RQ4 How does the Calibrated Imagenet Score (CIS) compare to other metrics like CKA or class separation in predicting transfer performance?
• RQ5 Can Controlled Label Injection (CLI) generate models with both high accuracy and high diversity, leading to better downstream performance?

Key findings

• The Calibrated Imagenet Score (CIS), which combines ImageNet accuracy and feature diversity, correlates significantly better with transferability than ImageNet accuracy alone.
• Models generated via CLI achieve higher transfer performance than standard supervised or self-supervised models, even when ImageNet accuracy is similar.
• On the MS-COCO multi-label classification benchmark, CLI-generated models outperform baseline models by up to 3.5% in top-1 accuracy.
• For ViT models pre-trained with MAE, CLI maintains high feature diversity while increasing ImageNet accuracy, leading to improved CIS and transferability.
• On the DINO pre-trained ViT, CLI increases both accuracy and diversity simultaneously, with models at T=2 and T=1 showing identical accuracy but differing transferability due to diversity differences.
• The CIS metric successfully identifies high-performing models across architectures (ResNets and ViTs), pretraining methods (MoCo-v2, SimCLR, SwAV, DINO, MAE), and downstream tasks.