[论文解读] Revealing the Dark Secrets of Masked Image Modeling

Zhenda Xie, Zigang Geng+4 more|arXiv (Cornell University)|May 26, 2022

Advanced Neural Network Applications被引用 4

一句话总结

本文研究了掩码图像建模（MIM）作为一种自监督预训练方法，揭示了MIM在视觉变换器的所有层中引入了强烈的局部性归纳偏置，保持了较高的注意力头多样性，并在几何和运动任务上实现了卓越性能。在不进行架构修改的情况下，标准的MIM预训练SwinV2-L在姿态估计（COCO上78.9 AP）、深度估计（NYUv2上0.287 RMSE）和视频跟踪（LaSOT上70.7 SUC）任务上均取得了最先进结果，优于监督模型在细粒度和几何任务上的表现。

ABSTRACT

Masked image modeling (MIM) as pre-training is shown to be effective for numerous vision downstream tasks, but how and where MIM works remain unclear. In this paper, we compare MIM with the long-dominant supervised pre-trained models from two perspectives, the visualizations and the experiments, to uncover their key representational differences. From the visualizations, we find that MIM brings locality inductive bias to all layers of the trained models, but supervised models tend to focus locally at lower layers but more globally at higher layers. That may be the reason why MIM helps Vision Transformers that have a very large receptive field to optimize. Using MIM, the model can maintain a large diversity on attention heads in all layers. But for supervised models, the diversity on attention heads almost disappears from the last three layers and less diversity harms the fine-tuning performance. From the experiments, we find that MIM models can perform significantly better on geometric and motion tasks with weak semantics or fine-grained classification tasks, than their supervised counterparts. Without bells and whistles, a standard MIM pre-trained SwinV2-L could achieve state-of-the-art performance on pose estimation (78.9 AP on COCO test-dev and 78.0 AP on CrowdPose), depth estimation (0.287 RMSE on NYUv2 and 1.966 RMSE on KITTI), and video object tracking (70.7 SUC on LaSOT). For the semantic understanding datasets where the categories are sufficiently covered by the supervised pre-training, MIM models can still achieve highly competitive transfer performance. With a deeper understanding of MIM, we hope that our work can inspire new and solid research in this direction.

研究动机与目标

• 理解MIM在多样化视觉任务中表现优异的关键机制。
• 从表示归纳偏置和特征层次结构的角度，比较MIM与监督预训练的异同。
• 研究MIM与监督模型在语义理解、几何和运动任务之间的可迁移性差异。
• 揭示为何MIM在语义信息较弱或细粒度内容的任务中表现更优。
• 为MIM与监督模型中的注意力头多样性及层间表示相似性提供深入见解。

提出的方法

• 通过可视化平均注意力距离，分析MIM与监督模型中的局部性归纳偏置。
• 使用注意力图之间KL散度度量注意力头多样性。
• 计算中心化核对齐（CKA）以比较不同层之间的表示相似性。
• 在多样化基准上评估微调性能：COCO、NYUv2、KITTI、LaSOT、ADE-20K以及细粒度数据集。
• 在MIM与监督预训练中采用完全相同的训练设置，以确保公平比较。
• 以SwinV2-L作为骨干网络，并为姿态、深度和跟踪任务使用标准头设计。

Figure 1: The averaged attention distance in different attention heads (dots) w.r.t the layer number on supervised model (a), contrastive learning model (b), and SimMIM model (c) with ViT-B as the backbone architecture.

实验结果

研究问题

• RQ1 MIM相较于监督预训练在表示方面存在哪些关键差异？
• RQ2 MIM如何影响网络各层中的注意力局部性与头多样性？
• RQ3 尽管语义监督较弱，为何MIM在几何和运动任务上仍优于监督模型？
• RQ4 MIM与监督模型在层间相似性与可迁移性方面，其表示结构有何不同？
• RQ5 注意力头多样性在多大程度上与下游微调性能相关？

主要发现

• MIM预训练在所有层中均引入了强烈的局部性归纳偏置，而监督模型则随着深度增加，注意力从局部聚焦逐渐转向全局聚焦。
• MIM在所有层中均保持了较高的注意力头多样性，而监督模型在最后三层中多样性显著下降，影响了微调性能。
• MIM模型在姿态估计任务上实现了最先进性能（COCO测试集开发集上78.9 AP），且无需任何架构修改。
• 在深度估计任务中，MIM在NYUv2上达到0.287 RMSE，在KITTI上达到1.966 RMSE，优于监督基线模型。
• 在视频目标跟踪任务中，MIM在LaSOT上达到70.7 SUC，展现出对长时跟踪任务的强大可迁移性。
• 在细粒度和低语义内容任务（如iNaturalist-18和COG）上，MIM显著优于监督模型，归因于更优的表示迁移能力。

Figure 2: (a) The error rate of fine-tuning on ImageNet-1K (blue circle $\circ$ ) and averaged attention distance (red diamond $\diamond$ ) w.r.t AvgDist (averaged distance of masked pixels to the nearest visible pixels) with Swin-B as the backbone. Points ( $\diamond$ or $\circ$ ) denote the SimMIM