[Paper Review] Dynamic Convolution: Attention over Convolution Kernels

Yinpeng Chen, Xiyang Dai+4 more|arXiv (Cornell University)|Dec 7, 2019

Advanced Neural Network ApplicationsComputer Science43 references69 citations

TL;DR

The paper introduces Dynamic Convolution (DY-CNNs), which aggregates multiple small kernels per layer via input-dependent attention to boost performance with minimal extra FLOPs, achieving notable gains on ImageNet and COCO across MobileNet and ResNet backbones.

ABSTRACT

Light-weight convolutional neural networks (CNNs) suffer performance degradation as their low computational budgets constrain both the depth (number of convolution layers) and the width (number of channels) of CNNs, resulting in limited representation capability. To address this issue, we present Dynamic Convolution, a new design that increases model complexity without increasing the network depth or width. Instead of using a single convolution kernel per layer, dynamic convolution aggregates multiple parallel convolution kernels dynamically based upon their attentions, which are input dependent. Assembling multiple kernels is not only computationally efficient due to the small kernel size, but also has more representation power since these kernels are aggregated in a non-linear way via attention. By simply using dynamic convolution for the state-of-the-art architecture MobileNetV3-Small, the top-1 accuracy of ImageNet classification is boosted by 2.9% with only 4% additional FLOPs and 2.9 AP gain is achieved on COCO keypoint detection.

Motivation & Objective

• Motivate the need for higher representation power in ultra-efficient CNNs under tight compute budgets.
• Propose dynamic convolution as a drop-in replacement that aggregates multiple small kernels via attention.
• Demonstrate training strategies to effectively learn dynamic kernels and attentions.
• Evaluate DY-CNNs on ImageNet and COCO to quantify accuracy and efficiency gains across architectures.

Proposed method

• Define dynamic perceptron as a sum of K linear functions with input-dependent attention that sums to one.
• Instantiate dynamic perceptron as dynamic convolution with K parallel kernels of identical size, aggregated by attention\\pi_k(x).
• Compute kernel attentions using a squeeze-and-excitation (SE) style block with softmax (temperature-controlled) to ensure convex combination.
• Share output channels across kernels to keep depth/width unchanged while increasing model capacity.
• Provide guidance for training DY-CNNs, including high softmax temperature in early training and temperature annealing.
• Demonstrate that dynamic convolution can replace any static convolution (1x1, 3x3, depthwise, etc.) in existing architectures.

Experimental results

Research questions

• RQ1 How can multiple small convolution kernels be combined dynamically to increase expressiveness without increasing depth/width?
• RQ2 What are effective training strategies to jointly optimize kernels and attention in dynamic convolution?
• RQ3 What are the performance and compute trade-offs when applying dynamic convolution to lightweight architectures like MobileNet variants and shallow ResNets?
• RQ4 Does dynamic convolution provide complementary benefits to SE and NAS-derived architectures?

Key findings

• Dynamic convolution consistently improves top-1 accuracy across MobileNetV2, MobileNetV3, and ResNet backbones with about 4% extra FLOPs.
• On ImageNet, DY-MobileNetV2/0.5 achieves 69.9% top-1 (4.5 points gain) vs static 65.4% at similar FLOPs; DY-MobileNetV3-Small reaches 70.3% top-1 (2.9 points gain).
• Compared to CondConv with similar settings, DY-CNN uses fewer kernels per layer and achieves higher accuracy with lower parameter count and FLOPs in many configurations.
• In COCO keypoint detection, DY-CNNs yield notable AP improvements when applied to backbone and head, e.g., DY-ResNet-18 and DY-MobileNetV2 show gains over their static counterparts.
• Two training insights are crucial: (1) constrain attention to sum to one to ease learning; (2) use near-uniform attention in early epochs (high temperature in softmax) to stabilize training and then anneal.