[Paper Review] MetaFormer Is Actually What You Need for Vision

Weihao Yu, Mi Luo+6 more|arXiv (Cornell University)|Nov 22, 2021

Advanced Neural Network ApplicationsComputer Science54 references28 citations

TL;DR

The paper abstracts Transformers into a general MetaFormer architecture and shows that even a simple pooling-based token mixer (PoolFormer) yields competitive vision performance, highlighting MetaFormer as the key to strong vision models.

ABSTRACT

Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in Transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the Transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in Transformers with an embarrassingly simple spatial pooling operator to conduct only basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned Vision Transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 50%/62% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of "MetaFormer", a general architecture abstracted from Transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent Transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design. Code is available at https://github.com/sail-sg/poolformer.

Motivation & Objective

• Propose a general architecture for Vision Models that decouples the token mixer from the overall design (MetaFormer).
• Investigate whether a simple, non-learnable pooling operator can serve as an effective token mixer within MetaFormer.
• Empirically compare PoolFormer against ViT/MLP-like baselines across image classification, object detection, instance segmentation, and semantic segmentation.

Proposed method

• Define MetaFormer as a Transformer-like architecture with an unspecified token mixer and standard residual/MLP components.
• Instantiate MetaFormer with a pooling-based token mixer (PoolFormer) to assess the role of the architecture itself.
• Use a 4-stage hierarchical PoolFormer design with varying embedding dimensions and block counts (S12, S24, S36, M36, M48) and a default pooling size of 3.
• Train on ImageNet-1K with data augmentation, AdamW optimization, and modified Layer Normalization (MLN) for channel-first data.
• Evaluate on ImageNet-1K, COCO (RetinaNet, Mask R-CNN), and ADE20K (semantic segmentation) to assess generalization across vision tasks.
• Conduct ablations varying token mixers, normalization, activation, and stage configurations to isolate MetaFormer effects.

Experimental results

Research questions

• RQ1 Is MetaFormer’s general architecture the primary driver of performance across vision tasks, independent of the specific token mixer?
• RQ2 Can a pooling-based token mixer (PoolFormer) achieve competitive performance compared to attention-based Transformers and MLP-like models?
• RQ3 How does PoolFormer perform across classification, detection/segmentation, and semantic segmentation benchmarks relative to state-of-the-art baselines?

Key findings

• PoolFormer with a simple pooling token mixer achieves competitive ImageNet-1K top-1 accuracy, surpassing DeiT-B and ResMLP-B24 baselines on MACs/parameters.
• PoolFormer-S24 reaches 80.3% top-1 with 21M parameters and 3.4G MACs, outperforming ViT/MLP-like baselines under similar budgets.
• PoolFormer-M36 reaches 82.1% top-1 with 56.1M parameters and 8.8G MACs, outperforming several stronger baselines under fewer resources.
• PoolFormer-M48 reaches 82.5% top-1 with 73.4M parameters and 11.6G MACs, continuing the competitive trend.
• Ablations show that even identity mapping as token mixer yields 74.3% top-1, supporting the argument that MetaFormer architecture largely drives performance.