[Paper Review] More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity

Shiwei Liu, Tianlong Chen+8 more|University of Twente Research Information|Jul 7, 2022

Advanced Neural Network ApplicationsComputer Science86 citations

TL;DR

The paper introduces Sparse Large Kernel Network (SLaK), a pure CNN that uses decomposed large kernels (51×51) with sparse groups and dynamic sparsity to scale beyond 31×31, achieving competitive or superior results to state-of-the-art Transformers on ImageNet and downstream vision tasks.

ABSTRACT

Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local-window attention mechanism, showing appealing performance and efficiency. While one of them, i.e. RepLKNet, impressively manages to scale the kernel size to 31x31 with improved performance, the performance starts to saturate as the kernel size continues growing, compared to the scaling trend of advanced ViTs such as Swin Transformer. In this paper, we explore the possibility of training extreme convolutions larger than 31x31 and test whether the performance gap can be eliminated by strategically enlarging convolutions. This study ends up with a recipe for applying extremely large kernels from the perspective of sparsity, which can smoothly scale up kernels to 61x61 with better performance. Built on this recipe, we propose Sparse Large Kernel Network (SLaK), a pure CNN architecture equipped with sparse factorized 51x51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architectures like ConvNeXt and RepLKNet, on ImageNet classification as well as a wide range of downstream tasks including semantic segmentation on ADE20K, object detection on PASCAL VOC 2007, and object detection/segmentation on MS COCO.

Motivation & Objective

• Motivate scaling ConvNets with extremely large kernels beyond 31×31 to capture long-range dependencies while preserving locality.
• Develop a practical training recipe leveraging kernel decomposition and sparsity to enable large kernels.
• Build and evaluate SLaK, a pure CNN with 51×51 kernels, against state-of-the-art CNNs and Transformers across ImageNet and downstream tasks.

Proposed method

• Decompose a large M×M kernel into two parallel rectangular kernels of sizes M×N and N×M (with N < M) to balance long-range dependencies with local detail.
• Replace dense kernels with sparse groups and apply dynamic sparsity during training to prune and grow connections, enabling scalable width Expansion.
• Use a 5×5 parallel branch after decomposition and apply BatchNorm before summing outputs to stabilize training.
• Inherit ConvNeXt-style architecture while increasing kernel sizes per stage to [51, 49, 47, 13], and sparsify with 40% sparsity and width expansion (1.3×).
• Train models with extended epochs (300 for ImageNet-1K) following established data augmentation and regularization protocols, comparing to ConvNeXt and RepLKNet baselines.

Experimental results

Research questions

• RQ1 Can extremely large kernels (beyond 31×31) be trained effectively in CNNs without performance collapse?
• RQ2 Does kernel decomposition plus sparse groups enable scalable, high-performance large-kernel CNNs compared to Transformer-based architectures?
• RQ3 What is the impact of sparse training dynamics on accuracy, FLOPs, and parameter efficiency for very large kernels?
• RQ4 Do large kernels provide advantages on high-resolution inputs and downstream vision tasks (segmentation, detection) beyond ImageNet?
• RQ5 How does SLaK compare to state-of-the-art CNNs and Transformers across diverse vision benchmarks?

Key findings

• A decomposition strategy (M×N and N×M with N=5) enables scaling to 61×61 kernels with improved performance over naive large kernels.
• Dynamic sparsity plus width expansion (40% sparsity, 1.3× width) yields notable accuracy gains and reduces FLOPs, achieving 81.6% top-1 on 51×51 kernels for ConvNeXt backbone in 120-epoch experiments.
• On ImageNet-1K, SLaK-T achieves 82.5% and SLaK-B achieves 84.0% (224×224) with comparable or better efficiency than ConvNeXt and RepLKNet baselines; at 384×384, SLaK-B reaches 85.5%.
• On ADE20K, larger kernels yield higher mIoU, with SLaK-T at 51×51 achieving 46.2 vs. ConvNeXt-T at 31×31 (RepLKNet) 45.0, and 300-epoch training surpassing ConvNeXt-S.
• On PASCAL VOC 2007 object detection, SLaK-T with 51×51 achieves 82.7 mAP vs. 80.6 for ConvNeXt-T-7×7 and 81.3 for RepLKNet 31×31.
• On MS COCO, SLaK-T (51×51) achieves 48.4 APbox and 51.3 APbox with longer fine-tuning, outperforming several dense-kernel baselines while using comparable or lower FLOPs.