[Paper Review] MetaFormer Baselines for Vision

Weihao Yu, Chenyang Si+6 more|arXiv (Cornell University)|Oct 24, 2022

Advanced Neural Network Applications101 references33 citations

TL;DR

The paper studies MetaFormer with basic token mixers, showing a solid lower bound, universal compatibility with mixers, and state-of-the-art results such as CAFormer achieving 85.5% top-1 on ImageNet-1K, along with StarReLU improvements.

ABSTRACT

MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, without focusing on token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and summarize our observations as follows: (1) MetaFormer ensures solid lower bound of performance. By merely adopting identity mapping as the token mixer, the MetaFormer model, termed IdentityFormer, achieves >80% accuracy on ImageNet-1K. (2) MetaFormer works well with arbitrary token mixers. When specifying the token mixer as even a random matrix to mix tokens, the resulting model RandFormer yields an accuracy of >81%, outperforming IdentityFormer. Rest assured of MetaFormer's results when new token mixers are adopted. (3) MetaFormer effortlessly offers state-of-the-art results. With just conventional token mixers dated back five years ago, the models instantiated from MetaFormer already beat state of the art. (a) ConvFormer outperforms ConvNeXt. Taking the common depthwise separable convolutions as the token mixer, the model termed ConvFormer, which can be regarded as pure CNNs, outperforms the strong CNN model ConvNeXt. (b) CAFormer sets new record on ImageNet-1K. By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1K: it achieves an accuracy of 85.5% at 224x224 resolution, under normal supervised training without external data or distillation. In our expedition to probe MetaFormer, we also find that a new activation, StarReLU, reduces 71% FLOPs of activation compared with GELU yet achieves better performance. We expect StarReLU to find great potential in MetaFormer-like models alongside other neural networks.

Motivation & Objective

• Investigate the capacity of MetaFormer by using basic or common token mixers.
• Assess whether MetaFormer provides a solid lower bound on performance across mixers.
• Examine the universality of MetaFormer with arbitrary token mixers.
• Demonstrate state-of-the-art performance using conventional token mixers via ConvFormer and CAFormer.
• Introduce activation and training techniques (StarReLU, scaling) to improve MetaFormer efficiency and accuracy.

Proposed method

• Formalize IdentityFormer and RandFormer as basic-token-mixer instantiations within the MetaFormer framework.
• Construct ConvFormer and CAFormer by using separable convolutions and vanilla self-attention as token mixers in different stages.
• Adopt a 4-stage hierarchical architecture with stage-specific block configurations.
• Introduce StarReLU as a low-cost activation to reduce activation FLOPs and improve resilience to distribution shift.
• Apply ResScale-style scaling in residual branches and disable biases to stabilize training.
• Benchmark on ImageNet-1K with standard augmentation/training settings and perform pre-training on ImageNet-21K for scalability tests.

Experimental results

Research questions

• RQ1 Does MetaFormer yield a solid lower bound on performance when using minimal or random token mixers?
• RQ2 Can MetaFormer maintain competitive performance with arbitrary or older token mixers (e.g., random, pooled, separable conv, attention)?
• RQ3 Do CNN-like or hybrid MetaFormer variants (ConvFormer, CAFormer) achieve state-of-the-art results on ImageNet-1K under conventional training?
• RQ4 What is the impact of StarReLU and scaling techniques on MetaFormer efficiency and accuracy?

Key findings

• IdentityFormer achieves 80.4% top-1 accuracy on ImageNet-1K with 73M parameters and 11.5G MACs.
• RandFormer reaches 81.4% top-1 accuracy, outperforming IdentityFormer.
• ConvFormer (separable convolutions) can outperform ConvNeXt.
• CAFormer achieves a new record of 85.5% top-1 accuracy on ImageNet-1K at 224^2 with normal supervised training, without external data or distillation.
• StarReLU reduces activation FLOPs by ~71% versus GELU and, with learnable scale/bias, can improve accuracy.
• Using legacy token mixers in MetaFormer yields strong baselines and demonstrates universal compatibility with diverse mixers.