[Paper Review] Revealing the Dark Secrets of Masked Image Modeling

Zhenda Xie, Zigang Geng+4 more|arXiv (Cornell University)|May 26, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper investigates masked image modeling (MIM) as a self-supervised pre-training method, revealing that MIM introduces strong locality inductive bias across all layers of Vision Transformers, preserves high attention head diversity, and enables superior performance on geometric and motion tasks. Without architectural modifications, a standard MIM-pretrained SwinV2-L achieves state-of-the-art results on pose estimation (78.9 AP on COCO), depth estimation (0.287 RMSE on NYUv2), and video tracking (70.7 SUC on LaSOT), outperforming supervised models on fine-grained and geometric tasks.

ABSTRACT

Masked image modeling (MIM) as pre-training is shown to be effective for numerous vision downstream tasks, but how and where MIM works remain unclear. In this paper, we compare MIM with the long-dominant supervised pre-trained models from two perspectives, the visualizations and the experiments, to uncover their key representational differences. From the visualizations, we find that MIM brings locality inductive bias to all layers of the trained models, but supervised models tend to focus locally at lower layers but more globally at higher layers. That may be the reason why MIM helps Vision Transformers that have a very large receptive field to optimize. Using MIM, the model can maintain a large diversity on attention heads in all layers. But for supervised models, the diversity on attention heads almost disappears from the last three layers and less diversity harms the fine-tuning performance. From the experiments, we find that MIM models can perform significantly better on geometric and motion tasks with weak semantics or fine-grained classification tasks, than their supervised counterparts. Without bells and whistles, a standard MIM pre-trained SwinV2-L could achieve state-of-the-art performance on pose estimation (78.9 AP on COCO test-dev and 78.0 AP on CrowdPose), depth estimation (0.287 RMSE on NYUv2 and 1.966 RMSE on KITTI), and video object tracking (70.7 SUC on LaSOT). For the semantic understanding datasets where the categories are sufficiently covered by the supervised pre-training, MIM models can still achieve highly competitive transfer performance. With a deeper understanding of MIM, we hope that our work can inspire new and solid research in this direction.

Motivation & Objective

• To understand the key mechanisms behind MIM's strong performance across diverse vision tasks.
• To compare MIM and supervised pre-training in terms of representational inductive biases and feature hierarchy structures.
• To investigate the transferability of MIM vs. supervised models across semantic understanding, geometric, and motion tasks.
• To uncover why MIM excels on tasks with weak semantics or fine-grained content.
• To provide insights into attention head diversity and layer-wise representation similarity in MIM and supervised models.

Proposed method

• Visualizing averaged attention distances to analyze locality inductive bias in MIM and supervised models.
• Measuring attention head diversity using KL-divergence between attention maps across heads.
• Computing Centered Kernel Alignment (CKA) to compare representation similarity across layers.
• Evaluating fine-tuning performance on diverse benchmarks: COCO, NYUv2, KITTI, LaSOT, ADE-20K, and fine-grained datasets.
• Training models with identical settings across MIM and supervised pre-training for fair comparison.
• Using SwinV2-L as the backbone and standard head designs for pose, depth, and tracking tasks.

Figure 1: The averaged attention distance in different attention heads (dots) w.r.t the layer number on supervised model (a), contrastive learning model (b), and SimMIM model (c) with ViT-B as the backbone architecture.

Experimental results

Research questions

• RQ1 What representational differences underlie the success of MIM compared to supervised pre-training?
• RQ2 How does MIM influence attention locality and head diversity across network layers?
• RQ3 Why does MIM outperform supervised models on geometric and motion tasks despite weaker semantic supervision?
• RQ4 How do representation structures in MIM and supervised models differ in terms of layer-wise similarity and transferability?
• RQ5 To what extent does attention head diversity correlate with downstream fine-tuning performance?

Key findings

• MIM pre-training introduces strong locality inductive bias across all layers, while supervised models shift from local to global focus as depth increases.
• MIM maintains high attention head diversity across all layers, whereas supervised models lose diversity in the last three layers, harming fine-tuning performance.
• MIM models achieve state-of-the-art performance on pose estimation (78.9 AP on COCO test-dev) without any architectural modifications.
• On depth estimation, MIM achieves 0.287 RMSE on NYUv2 and 1.966 RMSE on KITTI, outperforming supervised baselines.
• For video object tracking, MIM achieves 70.7 SUC on LaSOT, demonstrating strong transferability to long-term tracking.
• On fine-grained and low-semantic-content tasks like iNaturalist-18 and COG, MIM significantly outperforms supervised models due to better representation transfer.

Figure 2: (a) The error rate of fine-tuning on ImageNet-1K (blue circle $\circ$ ) and averaged attention distance (red diamond $\diamond$ ) w.r.t AvgDist (averaged distance of masked pixels to the nearest visible pixels) with Swin-B as the backbone. Points ( $\diamond$ or $\circ$ ) denote the SimMIM