[Paper Review] A ConvNet for the 2020s

Zhuang Liu, Hanzi Mao+4 more|arXiv (Cornell University)|Jan 10, 2022

Advanced Neural Network Applications169 citations

TL;DR

The paper develops ConvNeXt, a pure ConvNet that modernizes ResNet design ideas toward Transformer-like performance, achieving competitive ImageNet accuracy and strong results on COCO and ADE20K without attention mechanisms.

ABSTRACT

The "Roaring 20s" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model. A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers (e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive biases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually "modernize" a standard ResNet toward the design of a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.

Motivation & Objective

• Revisit ConvNet design with Transformer-inspired modernization to close the gap with Vision Transformers.
• Identify architectural and training decisions that contribute to performance gains beyond data augmentations.
• Demonstrate ConvNet competitiveness on downstream tasks such as object detection and semantic segmentation.

Proposed method

• Start from a ResNet baseline trained with Transformer-like training tricks (AdamW, Mixup, CutMix, RandAugment, etc.).
• Gradually modernize macro design (stage compute ratio, stem, patchify stem) toward Transformer-like structure.
• Adopt ResNeXt-style grouped/depthwise convolutions to increase width with controlled FLOPs.
• Incorporate inverted bottlenecks to reduce overall FLOPs while increasing representational capacity.
• Explore large-kernel depthwise convolutions (7x7) and repositioning depthwise conv within blocks to emulate Transformer block characteristics.
• Apply micro-design tweaks (activation and normalization choices, downsampling strategies, LN vs BN) to maximize ConvNet performance.
• Evaluate on ImageNet-1K/22K, COCO (Mask R-CNN, Cascade Mask R-CNN), and ADE20K to demonstrate transferability and scalability.

Experimental results

Research questions

• RQ1 How far can a pure ConvNet be modernized to rival hierarchical Vision Transformers in accuracy and scalability?
• RQ2 Which architectural and training choices (macro, micro, and training tricks) most impact ConvNet performance compared to Transformers?
• RQ3 Can ConvNet backbones trained with Transformer-style data and tricks outperform Swin Transformers on downstream tasks like detection and segmentation?
• RQ4 Do large-scale pre-training regimes (ImageNet-22K) erase inductive-bias advantages of Transformers relative to ConvNets?

Key findings

• ConvNeXt variants achieve ImageNet-1K top-1 accuracies around 82–87% depending on size and resolution.
• ConvNeXt outperforms Swin Transformer at comparable FLOPs across several configurations (e.g., ConvNeXt-B vs Swin-B at 224^2: 83.8% vs 83.5%).
• ConvNeXt-B at 384^2 reaches 85.1% top-1 accuracy, with higher throughput than Swin-B under similar compute.
• ConvNeXt-XL achieves 87.8% top-1 after ImageNet-22K pre-training, demonstrating strong scalability.
• On downstream tasks, ConvNeXt backbones match or exceed Swin Transformers in COCO detection/segmentation and ADE20K segmentation, often with better throughput.
• Isotropic ConvNeXt blocks trained in ViT-like settings perform on par with ViT variants, showing competitive block design without non-convolutional attention.