[Paper Review] Selective Kernel Networks

Xiang Li, Wenhai Wang+2 more|arXiv (Cornell University)|Mar 15, 2019

Advanced Neural Network ApplicationsComputer Science54 references95 citations

TL;DR

Selective Kernel Networks (SKNets) introduce a dynamic, attention-guided mechanism to adaptively select kernel sizes in CNNs, achieving improved object recognition with similar complexity to prior architectures on ImageNet and CIFAR.

ABSTRACT

In standard Convolutional Neural Networks (CNNs), the receptive fields of artificial neurons in each layer are designed to share the same size. It is well-known in the neuroscience community that the receptive field size of visual cortical neurons are modulated by the stimulus, which has been rarely considered in constructing CNNs. We propose a dynamic selection mechanism in CNNs that allows each neuron to adaptively adjust its receptive field size based on multiple scales of input information. A building block called Selective Kernel (SK) unit is designed, in which multiple branches with different kernel sizes are fused using softmax attention that is guided by the information in these branches. Different attentions on these branches yield different sizes of the effective receptive fields of neurons in the fusion layer. Multiple SK units are stacked to a deep network termed Selective Kernel Networks (SKNets). On the ImageNet and CIFAR benchmarks, we empirically show that SKNet outperforms the existing state-of-the-art architectures with lower model complexity. Detailed analyses show that the neurons in SKNet can capture target objects with different scales, which verifies the capability of neurons for adaptively adjusting their receptive field sizes according to the input. The code and models are available at https://github.com/implus/SKNet.

Motivation & Objective

• Motivate improving CNN multi-scale feature extraction by dynamically adapting receptive field sizes in response to input stimuli.
• Propose a lightweight Selective Kernel (SK) convolution that combines multiple kernels through attention-guided fusion.
• Show that SKNet achieves superior accuracy with comparable or lower complexity than prior state-of-the-art models on ImageNet and CIFAR.
• Demonstrate through ablations how different kernel configurations and attention mechanisms affect performance.
• Provide analysis of how the selective mechanism adapts kernel sizes across layers and categories.

Proposed method

• Introduce Selective Kernel (SK) convolution with three operations: Split (generate multiple kernel paths with different sizes), Fuse (aggregate branch information and produce channel-wise statistics), Select (use soft-attention to weight branches).
• Replace traditional large-kernel blocks in a ResNeXt-like backbone with SK convolutions to obtain SK units.
• Use a reduction ratio r to control the bottleneck in the fuse stage and a cap L for the reduced dimension d= max(C/r, L).
• Apply grouped/depthwise/dilated convolutions within SK branches to control cost while enabling multi-scale information gathering.
• Stack SK units into SKNet architectures (e.g., SKNet-50, SKNet-101) with configurable paths M, group G, and reduction r.
• Evaluate on ImageNet, CIFAR-10/100, and lightweight models to show performance and parameter-efficiency benefits.

Experimental results

Research questions

• RQ1 Does adaptively selecting kernel sizes within a single layer improve recognition accuracy compared to fixed multi-branch or single-branch convolutions?
• RQ2 How does the SK attention mechanism allocate attention across kernel sizes as target object scale changes and across network depth?
• RQ3 Can SK convolutions provide accuracy gains with similar or lower parameter count and FLOPs compared to ResNeXt/SENet backbones?
• RQ4 What are the optimal SK hyperparameters (M, G, r) for different architectures and datasets?
• RQ5 Do SK nets maintain or improve performance on compact models and smaller datasets like CIFAR?

Key findings

• SKNet-50 improves top-1 accuracy over ResNeXt-50 with comparable complexity, demonstrating the benefit of adaptive kernel selection.
• SKNet architectures achieve state-of-the-art performance versus other attention-based CNNs at similar budgets on ImageNet.
• Ablation studies show using multiple kernels with SK attention yields lower error than simple summation of branches and that increasing the number of paths generally helps but with diminishing returns beyond M=2 or M=3.
• The softer attention mechanism across branches enables adaptive receptive field sizes that respond to input scale, particularly in lower/middle layers.
• SK convolutions boost small-model performance (e.g., ShuffleNetV2 variants) and are effective on compact architectures.
• CIFAR-10/100 results show SKNet-29 achieving competitive or better accuracy with fewer parameters compared to ResNeXt/SENet baselines.