[Paper Review] Omni-Dimensional Dynamic Convolution

Chao Li, Aojun Zhou+1 more|arXiv (Cornell University)|Sep 16, 2022

Advanced Neural Network Applications173 citations

TL;DR

ODConv introduces multi-dimensional attentions over convolutional kernels to create input-dependent dynamic convolutions, achieving notable accuracy gains with fewer extra parameters and serving as a drop-in replacement for regular convolutions. It improves ImageNet and MS-COCO results across backbones.

ABSTRACT

Learning a single static convolutional kernel in each convolutional layer is the common training paradigm of modern Convolutional Neural Networks (CNNs). Instead, recent research in dynamic convolution shows that learning a linear combination of $n$ convolutional kernels weighted with their input-dependent attentions can significantly improve the accuracy of light-weight CNNs, while maintaining efficient inference. However, we observe that existing works endow convolutional kernels with the dynamic property through one dimension (regarding the convolutional kernel number) of the kernel space, but the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each convolutional kernel) are overlooked. Inspired by this, we present Omni-dimensional Dynamic Convolution (ODConv), a more generalized yet elegant dynamic convolution design, to advance this line of research. ODConv leverages a novel multi-dimensional attention mechanism with a parallel strategy to learn complementary attentions for convolutional kernels along all four dimensions of the kernel space at any convolutional layer. As a drop-in replacement of regular convolutions, ODConv can be plugged into many CNN architectures. Extensive experiments on the ImageNet and MS-COCO datasets show that ODConv brings solid accuracy boosts for various prevailing CNN backbones including both light-weight and large ones, e.g., 3.77%~5.71%|1.86%~3.72% absolute top-1 improvements to MobivleNetV2|ResNet family on the ImageNet dataset. Intriguingly, thanks to its improved feature learning ability, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, substantially reducing extra parameters. Furthermore, ODConv is also superior to other attention modules for modulating the output features or the convolutional weights.

Motivation & Objective

• Motivate dynamic convolution beyond a single attention dimension by exploiting four kernel-space dimensions (spatial, input-channel, output-channel, kernel-index).
• Propose ODConv as a generalized dynamic convolution with parallel multi-dimensional attentions.
• Show that ODConv achieves accuracy gains with fewer parameters across lightweight and large CNN backbones on ImageNet and MS-COCO.
• Demonstrate that ODConv can be a drop-in replacement for standard convolutions and transfer gains to detection tasks.

Proposed method

• Define ODConv as a multi-kernel dynamic convolution with four attentions (location-wise, channel-wise, filter-wise, kernel-wise) applied in parallel to kernel W_i.
• Compute four attentions (α_s, α_c, α_f, α_w) via a multi-head SE-type module conditioned on input x.
• Use location-wise, channel-wise, filter-wise, and kernel-wise multiplications across the kernel space to form the composite dynamic kernel.
• Allow ODConv with a single kernel to compete with multi-kernel dynamic methods, reducing parameter blow-up.
• Compare ODConv with CondConv and DyConv, and benchmark against SE/CBAM/ECA and other weight-modulation methods on ImageNet and MS-COCO.

Experimental results

Research questions

• RQ1 Can ODConv’s multi-dimensional attention over the kernel space provide greater representational power than existing one-dimensional dynamic convolutions?
• RQ2 What is the impact of using ODConv as a drop-in replacement across different backbones and scales (lightweight vs large) on ImageNet and MS-COCO?
• RQ3 Does ODConv with fewer kernels or smaller reductions maintain or improve performance compared to existing dynamic convolution approaches?
• RQ4 How do ODConv variants transfer to downstream tasks like object detection when used in pre-trained backbones?

Key findings

• On MobileNetV2 backbones, ODConv with 1x achieves higher top-1 accuracy than CondConv with 8x and is competitive with DyConv 4x while using fewer parameters.
• ODConv with 4x consistently yields the best results across MobileNetV2 and ResNet backbones, e.g., MobileNetV2 1.0x improves to 75.42% top-1 and 92.18% top-5.
• On ResNet18, ODConv 1x improves top-1 by 2.85%; ODConv 4x by 3.72% over the baseline; ODConv outperforms CondConv, DyConv, and DCD in many settings, especially at larger backbones.
• ODConv 4x yields strong gains on ResNet101 (top-1 78.98%), and Table 3 shows improvements over SE/CBAM/ECA baselines.
• In MS-COCO object detection, ODConv improves Faster R-CNN/Mask R-CNN with ResNet50 and MobileNetV2 backbones, often outperforming CondConv and DyConv in AP metrics.