[Paper Review] ResNeSt: Split-Attention Networks

Hang Zhang, Chongruo Wu+10 more|arXiv (Cornell University)|Apr 19, 2020

Domain Adaptation and Few-Shot Learning455 citations

TL;DR

ResNeSt introduces Split-Attention blocks that apply channel-wise attention within multi-path groups, achieving superior accuracy–latency trade-offs and strong transfer learning results across vision tasks.

ABSTRACT

It is well known that featuremap attention and multi-path representation are important for visual recognition. In this paper, we present a modularized architecture, which applies the channel-wise attention on different network branches to leverage their success in capturing cross-feature interactions and learning diverse representations. Our design results in a simple and unified computation block, which can be parameterized using only a few variables. Our model, named ResNeSt, outperforms EfficientNet in accuracy and latency trade-off on image classification. In addition, ResNeSt has achieved superior transfer learning results on several public benchmarks serving as the backbone, and has been adopted by the winning entries of COCO-LVIS challenge. The source code for complete system and pretrained models are publicly available.

Motivation & Objective

• Motivation to improve feature representation in CNNs by capturing cross-channel interactions within multi-path architectures.
• Develop a unified Split-Attention block that is modular, efficient, and interchangeable with standard residual blocks.
• Demonstrate improved accuracy and latency trade-offs on ImageNet compared to state-of-the-art CNNs.
• Showcase transfer learning improvements when ResNeSt backbones are used in detection, segmentation, and similar tasks.

Proposed method

• Define a Split-Attention block that splits feature maps into G groups (cardinality K, radix R, total groups G=KR).
• Within each cardinal group, apply R transformations, fuse splits via a learned channel-wise attention mechanism, and concatenate to form the block output.
• Use a shortcut connection to form a ResNeSt block and stack blocks in a ResNet-like architecture.
• Provide radix-major and cardinality-major implementations to enable efficient computation with standard CNN operators."
• Train with large mini-batch distributed training, cosine learning rate schedule, label smoothing, AutoAugment, Mixup, larger crop sizes, and DropBlock regularization.
• Compare performance with state-of-the-art models on ImageNet and evaluate transfer learning on MS-COCO (detection/segmentation) and ADE20K (semantic segmentation).

Experimental results

Research questions

• RQ1 Can Split-Attention within multi-path groups improve cross-feature interactions and representation diversity beyond SE-Net and SK-Net?
• RQ2 Do ResNeSt variants offer better speed-accuracy trade-offs on large-scale datasets like ImageNet compared to NAS-optimized models?
• RQ3 Do ResNeSt backbones provide consistent gains in downstream tasks such as object detection, instance segmentation, and semantic segmentation?

Key findings

• ResNeSt achieves better accuracy-latency trade-offs than state-of-the-art CNNs on ImageNet, e.g., ResNeSt-269 with lower latency than EfficientNet-B7 at comparable accuracy.
• Split-Attention blocks provide improvements in transfer learning across detection, instance segmentation, and semantic segmentation when used as backbones.
• Backbones with ResNeSt-50/101 outperform ResNet-50/101 in detection and segmentation tasks with fewer parameters in several cases.
• Mixup, AutoAugment, and larger input crops contribute to accuracy gains in combination with Split-Attention.
• Radix and cardinality settings show that increasing radix from 0 to 4 improves accuracy but also latency/memory, with a practical balance found at 2s1x64d in experiments.
• ResNeSt backbones are effective replacements for ResNet in common detection/segmentation frameworks, achieving notable gains over ResNet baselines.