[Paper Review] Revisiting the Transferability of Supervised Pretraining: an MLP Perspective

Yizhou Wang, Shixiang Tang+5 more|arXiv (Cornell University)|Dec 1, 2021

Domain Adaptation and Few-Shot Learning56 references4 citations

TL;DR

This paper proposes enhancing supervised pretraining with a multilayer perceptron (MLP) projector before the classifier, revealing that this simple addition significantly improves transferability by preserving intra-class variation, reducing feature distribution distance between pretraining and evaluation datasets, and decreasing feature redundancy. The method achieves state-of-the-art performance on concept generalization (+7.2% top-1 accuracy), linear evaluation across 12 domains (+5.8% accuracy), and COCO object detection (+0.8% AP), making supervised pretraining comparable or superior to unsupervised methods.

ABSTRACT

The pretrain-finetune paradigm is a classical pipeline in visual learning. Recent progress on unsupervised pretraining methods shows superior transfer performance to their supervised counterparts. This paper revisits this phenomenon and sheds new light on understanding the transferability gap between unsupervised and supervised pretraining from a multilayer perceptron (MLP) perspective. While previous works focus on the effectiveness of MLP on unsupervised image classification where pretraining and evaluation are conducted on the same dataset, we reveal that the MLP projector is also the key factor to better transferability of unsupervised pretraining methods than supervised pretraining methods. Based on this observation, we attempt to close the transferability gap between supervised and unsupervised pretraining by adding an MLP projector before the classifier in supervised pretraining. Our analysis indicates that the MLP projector can help retain intra-class variation of visual features, decrease the feature distribution distance between pretraining and evaluation datasets, and reduce feature redundancy. Extensive experiments on public benchmarks demonstrate that the added MLP projector significantly boosts the transferability of supervised pretraining, e.g. +7.2% top-1 accuracy on the concept generalization task, +5.8% top-1 accuracy for linear evaluation on 12-domain classification tasks, and +0.8% AP on COCO object detection task, making supervised pretraining comparable or even better than unsupervised pretraining.

Motivation & Objective

• To investigate why unsupervised pretraining often outperforms supervised pretraining in transfer learning despite using richer semantic labels.
• To identify the missing factor in supervised pretraining that limits its transferability, especially when there is a large semantic gap between pretraining and downstream datasets.
• To evaluate whether adding an MLP projector to supervised pretraining can close the performance gap with unsupervised methods.
• To provide empirical and theoretical justification for the role of the MLP projector in improving feature quality for transfer learning.

Proposed method

• Introduce an MLP projector before the classifier in standard supervised pretraining (SL), forming SL-MLP, to enhance feature representation for transfer.
• Use the MLP projector to preserve intra-class variation in visual features during pretraining, preventing overfitting to class-specific patterns.
• Apply the MLP projector to reduce the distribution distance between features from the pretraining and evaluation datasets, improving domain generalization.
• Train the backbone and classifier jointly with cross-entropy loss, while the MLP projector is trained end-to-end to refine features before classification.
• Conduct extensive ablation studies on hidden units and output dimensions of the MLP to assess sensitivity and robustness.
• Evaluate SL-MLP on multiple benchmarks: concept generalization, 12-domain linear evaluation, and COCO object detection, comparing against SL and unsupervised methods like BYOL and SimCLR.

Experimental results

Research questions

• RQ1 Why does unsupervised pretraining often generalize better than supervised pretraining in transfer learning, despite using richer annotations?
• RQ2 What role does the MLP projector play in the superior transferability of unsupervised pretraining methods like BYOL and SimCLR?
• RQ3 Can adding an MLP projector to standard supervised pretraining close the performance gap with unsupervised methods?
• RQ4 How does the MLP projector affect intra-class variation, feature distribution distance, and redundancy in pretraining features?
• RQ5 Is the performance gain from the MLP projector robust across different architectures, datasets, and downstream tasks?

Key findings

• Adding an MLP projector to supervised pretraining (SL-MLP) improves top-1 accuracy by +7.2% on the concept generalization task compared to standard SL.
• On 12-domain linear evaluation, SL-MLP achieves an average +5.8% accuracy gain over SL, outperforming even BYOL in some settings.
• SL-MLP improves COCO object detection performance by +0.8% AP, demonstrating strong generalization to dense prediction tasks.
• The MLP projector preserves intra-class variation in features, which helps maintain discriminative structure across diverse downstream tasks.
• The feature distribution distance between pretraining and evaluation datasets is significantly reduced by the MLP projector, enhancing domain generalization.
• The performance gain from the MLP projector is robust to hyperparameter choices such as hidden units and output dimensions, indicating its stability and practicality.