[Paper Review] Dilated convolution with learnable spacings

Ismail Khalfaoui-Hassani, Thomas Pellegrini+1 more|arXiv (Cornell University)|Dec 7, 2021

Advanced Neural Network Applications46 references18 citations

TL;DR

This paper proposes Dilated Convolution with Learnable Spacings (DCLS), a differentiable method that learns the positions of non-zero kernel elements in dilated convolutions via interpolation, enabling larger receptive fields without increasing parameters. DCLS improves ImageNet-1K accuracy and downstream task performance in ConvNeXt with minimal throughput cost, outperforming standard and fixed-dilation convolutions.

ABSTRACT

Recent works indicate that convolutional neural networks (CNN) need large receptive fields (RF) to compete with visual transformers and their attention mechanism. In CNNs, RFs can simply be enlarged by increasing the convolution kernel sizes. Yet the number of trainable parameters, which scales quadratically with the kernel's size in the 2D case, rapidly becomes prohibitive, and the training is notoriously difficult. This paper presents a new method to increase the RF size without increasing the number of parameters. The dilated convolution (DC) has already been proposed for the same purpose. DC can be seen as a convolution with a kernel that contains only a few non-zero elements placed on a regular grid. Here we present a new version of the DC in which the spacings between the non-zero elements, or equivalently their positions, are no longer fixed but learnable via backpropagation thanks to an interpolation technique. We call this method "Dilated Convolution with Learnable Spacings" (DCLS) and generalize it to the n-dimensional convolution case. However, our main focus here will be on the 2D case. We first tried our approach on ResNet50: we drop-in replaced the standard convolutions with DCLS ones, which increased the accuracy of ImageNet1k classification at iso-parameters, but at the expense of the throughput. Next, we used the recent ConvNeXt state-of-the-art convolutional architecture and drop-in replaced the depthwise convolutions with DCLS ones. This not only increased the accuracy of ImageNet1k classification but also of typical downstream and robustness tasks, again at iso-parameters but this time with negligible cost on throughput, as ConvNeXt uses separable convolutions. Conversely, classic DC led to poor performance with both ResNet50 and ConvNeXt. The code of the method is available at: https://github.com/K-H-Ismail/Dilated-Convolution-with-Learnable-Spacings-PyTorch.

Motivation & Objective

• To address the limitation of fixed, regular grid spacing in standard dilated convolutions, which restricts receptive field optimization.
• To enable gradient-based learning of non-zero kernel element positions in dilated convolutions, overcoming the non-differentiability of discrete positions.
• To improve performance in image classification and robustness tasks without increasing model parameters or computational cost.
• To evaluate DCLS as a drop-in replacement in modern CNNs like ConvNeXt, especially with depthwise separable convolutions.
• To explore the scalability and effectiveness of DCLS on large-scale vision benchmarks like ImageNet-1k.

Proposed method

• DCLS uses bilinear interpolation to make the discrete positions of non-zero kernel elements differentiable, enabling end-to-end backpropagation.
• The method parameterizes kernel positions as continuous offsets, which are optimized during training while maintaining sparsity.
• It generalizes to 1D, 2D, 3D, and mixed-dimensional (e.g., 2-1D, 3-2D) convolutions, with focus on 2D for computer vision.
• Kernel weights and positions are jointly optimized, with position sharing across channels and spatial locations to reduce overfitting.
• The method is compatible with efficient implementations like depthwise implicit GEMM, minimizing throughput overhead.
• The approach avoids input-dependent mechanisms like deformable convolutions, maintaining input-independence and stability.

Experimental results

Research questions

• RQ1 Can learning the positions of non-zero elements in dilated convolutions improve receptive field efficiency and model accuracy?
• RQ2 Does DCLS outperform standard and fixed-dilation convolutions in image classification and downstream tasks at the same parameter count?
• RQ3 Can DCLS be effectively used as a drop-in replacement in modern CNNs like ConvNeXt without significant throughput degradation?
• RQ4 How does position sharing and interpolation affect the training stability and performance of DCLS?
• RQ5 Is DCLS effective on large-scale benchmarks like ImageNet-1k and robustness benchmarks?

Key findings

• DCLS improved ImageNet-1k top-1 accuracy in ConvNeXt when replacing depthwise separable convolutions, with negligible throughput cost.
• In ResNet50, DCLS increased accuracy at iso-parameters but incurred a throughput penalty, unlike in ConvNeXt.
• Standard dilated convolution failed to improve performance in both ResNet50 and ConvNeXt, indicating its rigidity limits performance.
• DCLS outperformed input-dependent methods like deformable convolution v2 in ConvNeXt, which caused training instability and high latency.
• The method showed consistent gains across downstream and robustness tasks, suggesting broader applicability beyond classification.
• Position sharing within stages was critical for stable and effective training of DCLS.