[Paper Review] CycleMLP: A MLP-like Architecture for Dense Prediction

Shoufa Chen, Enze Xie+4 more|arXiv (Cornell University)|Jul 21, 2021

Advanced Neural Network ApplicationsComputer Science79 references99 citations

TL;DR

CycleMLP introduces Cycle Fully-Connected Layers (Cycle FC) to create a hierarchical, MLP-like backbone with linear complexity to image size, enabling effective dense prediction and competitive performance with CNNs and Transformers across detection, segmentation, and classification tasks.

ABSTRACT

This paper presents a simple MLP-like architecture, CycleMLP, which is a versatile backbone for visual recognition and dense predictions. As compared to modern MLP architectures, e.g., MLP-Mixer, ResMLP, and gMLP, whose architectures are correlated to image size and thus are infeasible in object detection and segmentation, CycleMLP has two advantages compared to modern approaches. (1) It can cope with various image sizes. (2) It achieves linear computational complexity to image size by using local windows. In contrast, previous MLPs have $O(N^2)$ computations due to fully spatial connections. We build a family of models which surpass existing MLPs and even state-of-the-art Transformer-based models, e.g., Swin Transformer, while using fewer parameters and FLOPs. We expand the MLP-like models' applicability, making them a versatile backbone for dense prediction tasks. CycleMLP achieves competitive results on object detection, instance segmentation, and semantic segmentation. In particular, CycleMLP-Tiny outperforms Swin-Tiny by 1.3% mIoU on ADE20K dataset with fewer FLOPs. Moreover, CycleMLP also shows excellent zero-shot robustness on ImageNet-C dataset. Code is available at https://github.com/ShoufaChen/CycleMLP.

Motivation & Objective

• Motivate and enable MLP-like backbones for dense prediction tasks (detection, segmentation) that handle variable input scales.
• Address limitations of prior MLP models: non-hierarchical blocks, fixed input scale, and quadratic cost.
• Propose Cycle FC to enlarge receptive field while preserving input-scale flexibility and linear complexity.
• Build a family of CycleMLP models with hierarchical architecture for recognition and dense prediction.
• Demonstrate competitive performance against CNNs, Transformers, and state-of-the-art MLPs on standard benchmarks.

Proposed method

• Introduce Cycle Fully-Connected Layer (Cycle FC) that samples along the channel dimension to create a receptive field with offsets delta_i(c), delta_j(c) and stepsizes SH, SW.
• Cycle FC has linear parameter and FLOP complexity with image size, enabling arbitrary input resolutions.
• Construct CycleMLP blocks with three parallel Cycle FC branches (1x7, 7x1, 1x1) plus a channel-MLP, with LayerNorm and residual connections.
• Build a four-stage hierarchical backbone with progressive token reduction and channel expansion to support dense prediction tasks.
• Create two model zoos (PVT-style CycleMLP-B1..B5 and Swin-style CycleMLP-T/S/B) by adjusting stride, channel dimensions, depth, and expansion ratios.
• Demonstrate resolution adaptability and compare Cycle FC to Channel FC and MHSA, highlighting sparsity and receptive field advantages.

Experimental results

Research questions

• RQ1 Can Cycle FC provide a scalable, input-scale-agnostic operator for dense prediction tasks without self-attention?
• RQ2 Does a hierarchical CycleMLP backbone with Cycle FC blocks outperform existing MLP-like models and be competitive with CNN/Transformer backbones on detection, segmentation, and classification?
• RQ3 How does CycleMLP perform under varying input resolutions and multi-scale training strategies typical of dense prediction workloads?

Key findings

• CycleMLP-B2 achieves 81.6% top-1 accuracy on ImageNet-1K with 27M parameters and 3.9G FLOPs, outperforming several MLP-like models.
• CycleMLP-B3/B4/B5 variants achieve competitive or superior results to Transformer backbones on ImageNet-1K without extra data, with favorable FLOPs and parameter counts.
• CycleMLP-based backbones improve object detection and instance segmentation metrics on COCO compared to ResNet and PVT baselines under similar budgets.
• In semantic segmentation on ADE20K, CycleMLP-B2 and CycleMLP-B3 yield higher mIoU than ResNet/PVT baselines and approach or surpass Swin in some settings, with larger effective receptive fields than Swin as visualized.
• CycleMLP demonstrates strong robustness on ImageNet-C across corruption types, outperforming several Transformers and other MLP-based models in mean corruption error.
• Ablation studies confirm the necessity of the three parallel Cycle FC branches for best performance, and Cycle FC outperforms Channel FC and Spatial FC in multiple tasks.