[Paper Review] A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP

Yucheng Zhao, Guangting Wang+4 more|arXiv (Cornell University)|Aug 30, 2021

Advanced Neural Network Applications42 references57 citations

TL;DR

This paper presents a unified SPACH framework to fairly compare CNN, Transformer, and MLP architectures for image classification, finds that all three can perform competitively, highlights the benefits of multi-stage design and local modeling, and proposes hybrid convolution-Transformer models achieving strong ImageNet-1K results.

ABSTRACT

Convolutional neural networks (CNN) are the dominant deep neural network (DNN) architecture for computer vision. Recently, Transformer and multi-layer perceptron (MLP)-based models, such as Vision Transformer and MLP-Mixer, started to lead new trends as they showed promising results in the ImageNet classification task. In this paper, we conduct empirical studies on these DNN structures and try to understand their respective pros and cons. To ensure a fair comparison, we first develop a unified framework called SPACH which adopts separate modules for spatial and channel processing. Our experiments under the SPACH framework reveal that all structures can achieve competitive performance at a moderate scale. However, they demonstrate distinctive behaviors when the network size scales up. Based on our findings, we propose two hybrid models using convolution and Transformer modules. The resulting Hybrid-MS-S+ model achieves 83.9% top-1 accuracy with 63M parameters and 12.3G FLOPS. It is already on par with the SOTA models with sophisticated designs. The code and models are publicly available at https://github.com/microsoft/SPACH.

Motivation & Objective

• Clarify how CNN, Transformer, and MLP architectures perform under a fair, unified framework for vision tasks.
• Isolate design choices (multi-stage, local modeling) that affect performance across architectures?
• Explore whether hybrid models combining convolution and Transformer blocks can surpass single-structure models on ImageNet-1K.

Proposed method

• Introduce SPACH, a plug-and-play framework with spatial mixing and channel mixing blocks to compare CNN, Transformer, and MLP under the same settings.
• Implement spatial mixing with convolution, self-attention, or MLP to form three variants within SPACH.
• Evaluate single-stage and multi-stage SPACH variants on ImageNet-1K with consistent training pipelines (AdamW, 300 epochs, data augmentation).
• Analyze the impact of local modeling by inserting a lightweight 3x3 depth-wise convolution as a bypass in MLP and Transformer blocks.
• Propose Hybrid-MS-S and Hybrid-MS-XS models by replacing selected layers with Transformer blocks in a convolution-based backbone; enhance with deep patch embedding.

Experimental results

Research questions

• RQ1 Do CNN, Transformer, and MLP architectures offer comparable accuracy–latency (or FLOPs) trade-offs under a unified SPACH framework?
• RQ2 How do multi-stage design and local modeling affect performance across the three architecture families?
• RQ3 Can hybrid models that combine convolution and Transformer components yield state-of-the-art-like performance on ImageNet-1K without complex tricks?

Key findings

• All three architectures can achieve competitive ImageNet-1K accuracy at moderate scale within SPACH.
• Multi-stage designs consistently outperform single-stage counterparts for CNN, Transformer, and MLP across sizes.
• Local modeling via a lightweight 3x3 depth-wise convolution significantly boosts Transformer and MLP performance with minimal parameter/FLOP cost.
• MLP models suffer from over-fitting at larger sizes, but multi-stage design and weight sharing alleviate this, bringing performance closer to Transformer/CNN levels.
• Convolution and Transformer are complementary: convolution provides better generalization, while Transformer offers higher capacity; hybrid Conv-Transformer models can achieve strong results without elaborate tricks.
• Hybrid-MS-S+ (63M params, 12.3G FLOPs) achieves 83.9% top-1 on ImageNet-1K, surpassing several SOTA baselines with fewer FLOPs; Hybrid-MS-XS+ reaches 82.8% with 28M params.