[Paper Review] Deformable Convolutional Networks

Jifeng Dai, Haozhi Qi+5 more|arXiv (Cornell University)|Mar 17, 2017

Advanced Neural Network ApplicationsComputer Science41 references434 citations

TL;DR

Introduces deformable convolution and deformable RoI pooling to enable dense, input-aware spatial transformations in CNNs, leading to improved performance on detection and segmentation tasks without heavy overhead.

ABSTRACT

Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in its building modules. In this work, we introduce two new modules to enhance the transformation modeling capacity of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the effectiveness of our approach on sophisticated vision tasks of object detection and semantic segmentation. The code would be released.

Motivation & Objective

• Motivate how fixed geometric structures in CNNs limit modeling of geometric transformations.
• Propose deformable convolution to learn dense sampling offsets from data.
• Propose deformable RoI pooling to adapt pooling regions to object shapes.
• Show that deformable modules can replace plain counterparts and be trained end-to-end.

Proposed method

• Deformable convolution augments the regular sampling grid with learnable 2D offsets per location.
• Offsets are produced by a parallel convolutional layer and learned end-to-end via backpropagation through bilinear interpolation.
• Deformable RoI pooling adds learnable offsets to RoI pooling bins and uses bilinear interpolation for fractional locations.
• Deformable PS RoI pooling extends deformable RoI pooling with class-specific score maps and fully convolutional offset learning.
• Experiments integrate deformable modules with ResNet-101 and Aligned-Inception-ResNet backbones across segmentation and detection pipelines.

Experimental results

Research questions

• RQ1 Can learnable spatial offsets enable CNNs to model large or non-r rigid geometric transformations without manual hand-crafted modules?
• RQ2 Do deformable convolution and deformable RoI pooling improve performance for segmentation and object detection tasks across standard benchmarks?
• RQ3 Is end-to-end training with bilinear interpolation sufficient for learning meaningful offsets in dense prediction settings?

Key findings

• Deformable modules enable adaptive receptive fields that correlate with object size and shape.
• Adding deformable RoI pooling improves localization, particularly for non-rigid objects.
• Joint use of deformable convolution and deformable RoI pooling yields notable gains over plain CNNs in segmentation and detection benchmarks.
• Offsets are learned end-to-end and are typically small, with zero-initialization ensuring minimal disruption at start of training.
• Deformable ConvNets introduce only modest additional parameters and computation yet deliver substantial accuracy improvements.