[论文解读] ConvNets vs. Transformers: Whose Visual Representations are More Transferable?

Hong-Yu Zhou, Chixiang Lu+2 more|arXiv (Cornell University)|Aug 11, 2021

Domain Adaptation and Few-Shot Learning参考文献 27被引用 4

一句话总结

该论文通过一个受控的评估框架，比较了卷积神经网络（ConvNets）和视觉变换器（vision transformers）在ImageNet上以相似的top-1准确率进行预训练后所学习的视觉表征的可迁移性。结果表明，视觉变换器在15项下游任务中的13项上均持续优于卷积神经网络，尤其在领域偏移场景和多任务学习中表现更优，展现出更强的泛化能力和鲁棒性。

ABSTRACT

Vision transformers have attracted much attention from computer vision researchers as they are not restricted to the spatial inductive bias of ConvNets. However, although Transformer-based backbones have achieved much progress on ImageNet classification, it is still unclear whether the learned representations are as transferable as or even more transferable than ConvNets' features. To address this point, we systematically investigate the transfer learning ability of ConvNets and vision transformers in 15 single-task and multi-task performance evaluations. Given the strong correlation between the performance of pre-trained models and transfer learning, we include 2 residual ConvNets (i.e., R-101x3 and R-152x4) and 3 Transformer-based visual backbones (i.e., ViT-B, ViT-L and Swin-B), which have close error rates on ImageNet, that indicate similar transfer learning performance on downstream datasets. We observe consistent advantages of Transformer-based backbones on 13 downstream tasks (out of 15), including but not limited to fine-grained classification, scene recognition (classification, segmentation and depth estimation), open-domain classification, face recognition, etc. More specifically, we find that two ViT models heavily rely on whole network fine-tuning to achieve performance gains while Swin Transformer does not have such a requirement. Moreover, vision transformers behave more robustly in multi-task learning, i.e., bringing more improvements when managing mutually beneficial tasks and reducing performance losses when tackling irrelevant tasks. We hope our discoveries can facilitate the exploration and exploitation of vision transformers in the future.

研究动机与目标

• 探究在受控的预训练条件下，视觉变换器是否能学习到比卷积神经网络更具可迁移性的视觉表征。
• 评估在来自ImageNet的领域差距各异的多样化下游任务中，迁移学习的性能表现。
• 在多任务学习设置中，比较卷积神经网络与视觉变换器的鲁棒性，特别是当任务之间互惠或无关时。
• 分析视觉变换器与卷积神经网络在微调需求上的差异，尤其是全网络微调与线性头微调的对比。
• 识别尽管视觉变换器整体可迁移性更优，卷积神经网络仍保持优势的具体场景。

提出的方法

• 选择在ImageNet上具有相近top-1错误率的预训练模型：两种ResNet变体（R-101×3，R-152×4）和三种视觉变换器（ViT-B，ViT-L，Swin-B）。
• 在所有模型上应用相同的预训练与微调协议，以确保公平比较。
• 在15项下游任务上进行评估：10项单任务（如细粒度分类、场景识别、人脸识别）和5项多任务（如结合场景、域内和开放域分类）。
• 采用全网络微调和线性评估协议，以衡量表征质量。
• 实施多任务学习，使用共享主干网络和任务特定头，包括分割与分类任务的联合训练。
• 统一应用标准数据增强（裁剪、翻转）、交叉熵损失和学习率调度，以确保所有模型的训练一致性。

Figure 1: Overview of our investigation procedure. We ask pre-trained ConvNet and Transformer models to have close top-1 error rates on ImageNet classification. The pre-trained weights are then transferred to 15 downstream tasks (i.e., 10 single-task and 5 multi-task duties), to evaluate the transfe

实验结果

研究问题

• RQ1 当预训练模型在ImageNet上的准确率相近时，视觉变换器学习到的视觉表征是否比卷积神经网络更具可迁移性？
• RQ2 在与ImageNet存在较大领域差距的任务上，视觉变换器与卷积神经网络的迁移学习性能如何比较？
• RQ3 多任务学习对卷积神经网络与视觉变换器性能的影响如何，特别是在任务互惠或无关的情况下？
• RQ4 微调策略（全网络微调 vs. 线性头微调）是否显著影响视觉变换器相比卷积神经网络的性能增益？
• RQ5 在哪些下游任务中，尽管视觉变换器整体可迁移性更优，卷积神经网络仍保持优势？

主要发现

• 视觉变换器在15项下游任务中的13项上优于卷积神经网络，包括细粒度分类、场景识别、深度估计和人脸识别。
• 在多任务学习中，Swin-B在Indoor67和Caltech101上均取得最高性能排名，展现出强大的鲁棒性和泛化能力。
• 在多任务学习中联合WikiArt时，与卷积神经网络相比，基于Transformer的模型在Caltech101上的性能下降显著更小（几乎低5%）。
• ViT模型在实现性能增益方面严重依赖全网络微调，而Swin-B即使采用线性头微调也能表现良好。
• 在组合无关任务（如场景分割与开放域分类）时，视觉变换器比卷积神经网络更能有效减少性能损失。
• 尽管视觉变换器整体可迁移性更优，卷积神经网络在域内分类和无监督图像检索任务中仍保持微弱优势，但这些优势被视觉变换器更广泛的可迁移性所超越。