[论文解读] Contrastive Masked Autoencoders are Stronger Vision Learners

Zhicheng Huang, Xiaojie Jin+6 more|arXiv (Cornell University)|Jul 27, 2022

Domain Adaptation and Few-Shot Learning参考文献 62被引用 8

一句话总结

本文提出对比掩码自编码器（CMAE），一种统一掩码图像建模（MIM）与对比学习（CL）的自监督视觉预训练框架，采用孪生编码器-解码器架构，配备动量编码器与特征解码器。CMAE在ImageNet-1k上达到85.3%的top-1准确率，在ADE20k上达到52.5%的mIoU，分别超越先前方法0.7%与1.8%。

ABSTRACT

Masked image modeling (MIM) has achieved promising results on various vision tasks. However, the limited discriminability of learned representation manifests there is still plenty to go for making a stronger vision learner. Towards this goal, we propose Contrastive Masked Autoencoders (CMAE), a new self-supervised pre-training method for learning more comprehensive and capable vision representations. By elaboratively unifying contrastive learning (CL) and masked image model (MIM) through novel designs, CMAE leverages their respective advantages and learns representations with both strong instance discriminability and local perceptibility. Specifically, CMAE consists of two branches where the online branch is an asymmetric encoder-decoder and the momentum branch is a momentum updated encoder. During training, the online encoder reconstructs original images from latent representations of masked images to learn holistic features. The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart. To make CL compatible with MIM, CMAE introduces two new components, i.e. pixel shifting for generating plausible positive views and feature decoder for complementing features of contrastive pairs. Thanks to these novel designs, CMAE effectively improves the representation quality and transfer performance over its MIM counterpart. CMAE achieves the state-of-the-art performance on highly competitive benchmarks of image classification, semantic segmentation and object detection. Notably, CMAE-Base achieves $85.3\%$ top-1 accuracy on ImageNet and $52.5\%$ mIoU on ADE20k, surpassing previous best results by $0.7\%$ and $1.8\%$ respectively. The source code is publicly accessible at \url{https://github.com/ZhichengHuang/CMAE}.

研究动机与目标

• 为解决掩码图像建模（MIM）表征中判别能力有限的问题，该问题限制了下游视觉任务的性能。
• 探索如何有效整合对比学习（CL）与MIM，以提升特征质量与实例判别能力。
• 设计一个统一框架，协调MIM（局部可感知性）与CL（实例判别能力）的优势，同时不损害任一特性。
• 通过更优的预训练，提升在图像分类、语义分割与目标检测任务上的迁移性能。

提出的方法

• CMAE采用孪生架构，其中在线非对称编码器-解码器分支用于掩码图像重建，动量更新的编码器分支用于对比学习监督。
• 在在线分支中引入一种新型特征解码器，用于重建掩码标记特征，从而从不完整表征中实现有意义的对比学习。
• 应用像素偏移增强方法，生成合理的正样本视图用于对比学习，确保空间一致性并兼容MIM。
• 动量编码器处理完整图像，生成语义完整特征，以指导在线编码器并增强特征判别能力。
• 框架联合优化掩码重建损失与对比损失，对齐不同视图间的特征，提升表征质量。
• 该方法在ViT架构（小、基、大）上均具可扩展性，且在各模型尺寸下均持续优于MAE。

实验结果

研究问题

• RQ1 对比学习能否有效整合到掩码图像建模中，以提升表征质量？
• RQ2 在输入增强与特征对齐方面存在差异的情况下，如何使对比学习与MIM的掩码重建目标兼容？
• RQ3 结合MIM与CL是否能提升在视觉基准上的下游迁移性能？
• RQ4 为确保对比学习增强而非干扰MIM目标，需要何种架构与增强设计？

主要发现

• CMAE-Base在ImageNet-1k上达到85.3%的top-1准确率，超越此前SOTA结果0.7%。
• 在ADE20k语义分割基准上，CMAE-Base实现52.5%的mIoU，较先前最佳结果提升1.8%。
• 在COCO目标检测任务中，CMAE-Base达到45.7%的AP，展现出对密集预测任务的强泛化能力。
• 在部分微调设置下，CMAE在仅微调一个Transformer块时，性能较MoCo-V3高出3.8%，表明其特征质量更高。
• 线性探测结果较MAE提升5.9%，证实在各类评估协议下表征质量均得到增强。
• 特征分析表明，CMAE学习到更紧凑的类内聚类与更大、更均匀的类间距离，表明其判别能力更优。