[Paper Review] Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation

Hang Gao, Xizhou Zhu+2 more|arXiv (Cornell University)|Oct 7, 2019

Advanced Neural Network Applications37 references35 citations

TL;DR

Introduces Deformable Kernels (DKs), a drop-in convolution operator that directly adapts the effective receptive field (ERF) by sampling in kernel space, enabling better handling of object deformation without changing the data sampling grid.

ABSTRACT

Convolutional networks are not aware of an object's geometric variations, which leads to inefficient utilization of model and data capacity. To overcome this issue, recent works on deformation modeling seek to spatially reconfigure the data towards a common arrangement such that semantic recognition suffers less from deformation. This is typically done by augmenting static operators with learned free-form sampling grids in the image space, dynamically tuned to the data and task for adapting the receptive field. Yet adapting the receptive field does not quite reach the actual goal -- what really matters to the network is the "effective" receptive field (ERF), which reflects how much each pixel contributes. It is thus natural to design other approaches to adapt the ERF directly during runtime. In this work, we instantiate one possible solution as Deformable Kernels (DKs), a family of novel and generic convolutional operators for handling object deformations by directly adapting the ERF while leaving the receptive field untouched. At the heart of our method is the ability to resample the original kernel space towards recovering the deformation of objects. This approach is justified with theoretical insights that the ERF is strictly determined by data sampling locations and kernel values. We implement DKs as generic drop-in replacements of rigid kernels and conduct a series of empirical studies whose results conform with our theories. Over several tasks and standard base models, our approach compares favorably against prior works that adapt during runtime. In addition, further experiments suggest a working mechanism orthogonal and complementary to previous works.

Motivation & Objective

• Motivate improving how networks cope with object deformation by targeting the ERF rather than the theoretical receptive field.
• Introduce Deformable Kernels (DKs) that resample kernel space to adapt ERF during inference.
• Provide two DK variants (global and local) and analyze their effects and compatibility with prior methods.
• Demonstrate DKs as drop-in replacements that yield improvements on image classification and object detection tasks.

Proposed method

• Define ERF and analyze its dependence on data sampling locations and kernel values.
• Formulate Deformable Kernels (DKs) as resampling of the original kernel space via learned offsets: O_j = sum_k I_{j+k} W_{k+Δk}.
• Show that ERF under DKs is given by ERF_DK(i;j) = sum_k∈K R^{(n)}(i; j+k, k+Δk).
• Implement two DK variants: global DK (one offset grid per image) and local DK (offsets per output location), using bilinear interpolation for sub-pixel kernel sampling.
• Discuss connections and compatibility with Deformable Convolutions and Dynamic/Conditional Convolutions.
• Evaluate DKs on ImageNet (classification) and COCO (object detection) with ResNet-50-DW and MobileNet-V2 backbones.

Experimental results

Research questions

• RQ1 Can direct adaptation of the ERF via learned kernel-space sampling improve robustness to object deformation without altering the data sampling grid?
• RQ2 How do global vs. local DK variants compare in terms of accuracy and efficiency across backbones and tasks?
• RQ3 Are DKs complementary to existing deformation or dynamic inference methods (e.g., Deformable Convolutions, Conditional Convolutions) when combined?
• RQ4 What is the impact of kernel scope size on performance and optimization?
• RQ5 Do DKs learn ERFs that align with object scales rather than semantics, and how does this affect their complementary behavior with other methods?

Key findings

• DKs improve performance over rigid kernels, with gains influenced by kernel scope size; for example, local DKs with a 4x4 scope on ResNet-50-DW yield a top-1 accuracy of 78.1% compared to 76.3% without DKs.
• Increasing scope beyond 4x4 yields diminishing returns, with a maximum observed gain around +1.4 top-1 points for ImageNet in the presented ablations.
• On object detection, local DKs provide consistent mAP gains over baselines, e.g., ResNet-50-DW with local DKs reaches 37.8 mAP (vs 36.6 baseline) and 38.4 mAP when combining with global DKs.
• DKs tend to learn ERFs that are more localized to object regions and are more scale-aware, suggesting complementary dynamics to semantics-focused methods like Conditional Convolutions.
• Combining DKs with Deformable Convolutions and Conditional Convolutions yields additive improvements, indicating orthogonal and compatible working mechanisms.
• Across tasks, DKs offer competitive or superior performance relative to strong baselines while using fewer parameters in certain configurations (e.g., comparing against SCC) and showing strong compatibility when fused with prior approaches.