[Paper Review] ConvNets vs. Transformers: Whose Visual Representations are More Transferable?

Hong-Yu Zhou, Chixiang Lu+2 more|arXiv (Cornell University)|Aug 11, 2021

Domain Adaptation and Few-Shot Learning27 references4 citations

TL;DR

This paper compares the transferability of visual representations learned by ConvNets and vision transformers using a controlled evaluation framework where models are pre-trained on ImageNet with similar top-1 accuracy. Results show that vision transformers consistently outperform ConvNets across 13 out of 15 downstream tasks, especially in domain-shift scenarios and multi-task learning, demonstrating superior generalization and robustness.

ABSTRACT

Vision transformers have attracted much attention from computer vision researchers as they are not restricted to the spatial inductive bias of ConvNets. However, although Transformer-based backbones have achieved much progress on ImageNet classification, it is still unclear whether the learned representations are as transferable as or even more transferable than ConvNets' features. To address this point, we systematically investigate the transfer learning ability of ConvNets and vision transformers in 15 single-task and multi-task performance evaluations. Given the strong correlation between the performance of pre-trained models and transfer learning, we include 2 residual ConvNets (i.e., R-101x3 and R-152x4) and 3 Transformer-based visual backbones (i.e., ViT-B, ViT-L and Swin-B), which have close error rates on ImageNet, that indicate similar transfer learning performance on downstream datasets. We observe consistent advantages of Transformer-based backbones on 13 downstream tasks (out of 15), including but not limited to fine-grained classification, scene recognition (classification, segmentation and depth estimation), open-domain classification, face recognition, etc. More specifically, we find that two ViT models heavily rely on whole network fine-tuning to achieve performance gains while Swin Transformer does not have such a requirement. Moreover, vision transformers behave more robustly in multi-task learning, i.e., bringing more improvements when managing mutually beneficial tasks and reducing performance losses when tackling irrelevant tasks. We hope our discoveries can facilitate the exploration and exploitation of vision transformers in the future.

Motivation & Objective

• To investigate whether vision transformers learn more transferable visual representations than ConvNets under controlled pre-training conditions.
• To evaluate transfer learning performance across diverse downstream tasks with varying domain gaps from ImageNet.
• To compare the robustness of ConvNets and vision transformers in multi-task learning settings, particularly when tasks are mutually beneficial or unrelated.
• To analyze the fine-tuning requirements of vision transformers versus ConvNets, especially regarding full-network vs. linear head fine-tuning.
• To identify scenarios where ConvNets still maintain advantages despite the overall superiority of vision transformers in transferability.

Proposed method

• Select pre-trained models with comparable ImageNet top-1 error rates: two ResNet variants (R-101×3, R-152×4) and three vision transformers (ViT-B, ViT-L, Swin-B).
• Apply the same pre-training and fine-tuning protocol across all models to ensure fair comparison.
• Evaluate on 15 downstream tasks: 10 single-task (e.g., fine-grained classification, scene recognition, face recognition) and 5 multi-task (e.g., combining scene, in-domain, and open-domain classification).
• Use both full-network fine-tuning and linear evaluation protocols to assess representation quality.
• Implement multi-task learning with shared backbones and task-specific heads, including segmentation and classification jointly.
• Apply standard data augmentation (crop, flip), cross-entropy loss, and learning rate scheduling for consistent training across all models.

Figure 1: Overview of our investigation procedure. We ask pre-trained ConvNet and Transformer models to have close top-1 error rates on ImageNet classification. The pre-trained weights are then transferred to 15 downstream tasks (i.e., 10 single-task and 5 multi-task duties), to evaluate the transfe

Experimental results

Research questions

• RQ1 Do vision transformers learn visual representations that are more transferable than those from ConvNets when pre-trained models have similar ImageNet accuracy?
• RQ2 How do vision transformers and ConvNets compare in transfer learning performance on tasks with large domain gaps from ImageNet?
• RQ3 What is the impact of multi-task learning on the performance of ConvNets versus vision transformers, especially when tasks are unrelated?
• RQ4 Does the fine-tuning strategy (full network vs. linear head) significantly affect the performance gain of vision transformers compared to ConvNets?
• RQ5 In which downstream tasks do ConvNets still maintain an advantage despite the overall transferability edge of vision transformers?

Key findings

• Vision transformers outperform ConvNets on 13 out of 15 downstream tasks, including fine-grained classification, scene recognition, depth estimation, and face recognition.
• Swin-B achieves the highest performance rank on both Indoor67 and Caltech101 in multi-task learning, demonstrating strong robustness and generalization.
• Transformer-based models suffer significantly smaller performance drops on Caltech101 (nearly 5% less) compared to ConvNets when combining with WikiArt in multi-task learning.
• ViT models rely heavily on full-network fine-tuning to achieve performance gains, whereas Swin-B performs well even with linear head fine-tuning.
• Vision transformers reduce performance loss more effectively than ConvNets when combining unrelated tasks, such as scene segmentation with open-domain classification.
• ConvNets maintain slight advantages in in-domain classification and unsupervised image retrieval, though these are outweighed by the broader transferability of vision transformers.