[Paper Review] RevColV2: Exploring Disentangled Representations in Masked Image Modeling

Qi Han, Yuxuan Cai+1 more|arXiv (Cornell University)|Sep 2, 2023

Advanced Neural Network Applications6 citations

TL;DR

RevColV2 proposes a unified autoencoder architecture that maintains both encoder and decoder during pre-training and fine-tuning to learn disentangled low-level and semantic representations via reversible multi-column design in masked image modeling (MIM). By preserving the full autoencoder and enabling progressive disentanglement through symmetric bottom-up and top-down reversible columns, RevColV2 achieves state-of-the-art performance on image classification, semantic segmentation, and object detection without task-specific adapters.

ABSTRACT

Masked image modeling (MIM) has become a prevalent pre-training setup for vision foundation models and attains promising performance. Despite its success, existing MIM methods discard the decoder network during downstream applications, resulting in inconsistent representations between pre-training and fine-tuning and can hamper downstream task performance. In this paper, we propose a new architecture, RevColV2, which tackles this issue by keeping the entire autoencoder architecture during both pre-training and fine-tuning. The main body of RevColV2 contains bottom-up columns and top-down columns, between which information is reversibly propagated and gradually disentangled. Such design enables our architecture with the nice property: maintaining disentangled low-level and semantic information at the end of the network in MIM pre-training. Our experimental results suggest that a foundation model with decoupled features can achieve competitive performance across multiple downstream vision tasks such as image classification, semantic segmentation and object detection. For example, after intermediate fine-tuning on ImageNet-22K dataset, RevColV2-L attains 88.4% top-1 accuracy on ImageNet-1K classification and 58.6 mIoU on ADE20K semantic segmentation. With extra teacher and large scale dataset, RevColv2-L achieves 62.1 box AP on COCO detection and 60.4 mIoU on ADE20K semantic segmentation. Code and models are released at https://github.com/megvii-research/RevCol

Motivation & Objective

• To address the inconsistency between pre-training and fine-tuning in masked image modeling (MIM), where decoders are typically discarded, leading to information loss and entangled features.
• To maintain a unified autoencoder architecture across pre-training and fine-tuning to preserve learned representations and improve transferability.
• To explore how reversible multi-column architectures can enable disentangled learning of low-level and semantic features during MIM pre-training.
• To demonstrate that disentangled representations improve downstream performance across diverse vision tasks, including classification, segmentation, and detection.

Proposed method

• Design a symmetric autoencoder with bottom-up reversible column encoders and top-down reversible column decoders to enable lossless, bidirectional feature propagation.
• Introduce reversible connections between bottom-up and top-down columns to ensure gradient flow and support disentangled feature learning.
• Use a reconstruction loss at the end of the top-down decoder to guide low-level features to the bottom and semantic features to the top through hierarchical information flow.
• Maintain the full autoencoder during fine-tuning, allowing the decoder to contribute to downstream task features without retraining or adapter insertion.
• Construct models of varying scale (e.g., RevColV2-L) with consistent FLOPs for fair comparison across architectures and training schedules.
• Leverage strong distillation signals (e.g., CLIP) and large-scale datasets (e.g., ImageNet-22K) to further boost performance.

Experimental results

Research questions

• RQ1 Can a unified autoencoder architecture that retains the decoder during fine-tuning improve downstream performance compared to encoder-only MIM methods?
• RQ2 Does the reversible multi-column design in RevColV2 effectively disentangle low-level and semantic features during MIM pre-training?
• RQ3 How does the disentangled representation learning in RevColV2 affect transferability across diverse vision tasks such as classification, segmentation, and detection?
• RQ4 What is the impact of training schedule length on the final performance of RevColV2 models?

Key findings

• After intermediate fine-tuning on ImageNet-22K, RevColV2-L achieves 88.4% top-1 accuracy on ImageNet-1K image classification.
• With additional distillation and large-scale pre-training, RevColV2-L attains 62.1 box AP on COCO object detection and 60.4 mIoU on ADE20K semantic segmentation.
• The ablation study confirms that the reversible multi-column decoder outperforms standard MAE, UNet-style, and non-reversible column decoders, achieving 84.1% top-1 accuracy on ImageNet-1K.
• Longer pre-training schedules (1600 epochs) yield significant gains—+1.1% accuracy for the large model compared to 300 epochs—indicating that full optimization potential is unlocked with extended training.
• The unified architecture with disentangled features consistently improves performance across multiple tasks, demonstrating the benefit of preserving both encoder and decoder in downstream adaptation.