[Paper Review] FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes

David W. Romero, Robert-Jan Bruintjes+4 more|arXiv (Cornell University)|Oct 15, 2021

Advanced Neural Network Applications61 references26 citations

TL;DR

FlexConv introduces learnable, high-bandwidth continuous convolution kernels with a differentiable kernel size, enabling dynamic, resolution-robust CNNs (FlexNets) that achieve state-of-the-art results on sequential data and competitive image performance.

ABSTRACT

When designing Convolutional Neural Networks (CNNs), one must select the size\break of the convolutional kernels before training. Recent works show CNNs benefit from different kernel sizes at different layers, but exploring all possible combinations is unfeasible in practice. A more efficient approach is to learn the kernel size during training. However, existing works that learn the kernel size have a limited bandwidth. These approaches scale kernels by dilation, and thus the detail they can describe is limited. In this work, we propose FlexConv, a novel convolutional operation with which high bandwidth convolutional kernels of learnable kernel size can be learned at a fixed parameter cost. FlexNets model long-term dependencies without the use of pooling, achieve state-of-the-art performance on several sequential datasets, outperform recent works with learned kernel sizes, and are competitive with much deeper ResNets on image benchmark datasets. Additionally, FlexNets can be deployed at higher resolutions than those seen during training. To avoid aliasing, we propose a novel kernel parameterization with which the frequency of the kernels can be analytically controlled. Our novel kernel parameterization shows higher descriptive power and faster convergence speed than existing parameterizations. This leads to important improvements in classification accuracy.

Motivation & Objective

• Motivate learning kernel sizes during training instead of fixing them a priori.
• Develop a continuous kernel parameterization that supports high bandwidth kernels at fixed parameter cost.
• Propose MAGNets to control frequency content and prevent aliasing, enabling high-resolution deployment.
• Demonstrate that FlexNets outperform prior learned-kernel methods on sequential tasks and are competitive with deeper CNNs on image benchmarks.

Proposed method

• Define FlexConv as the product of a continuous kernel MLP ψ with an anisotropic Gaussian mask to learn kernel size during training.
• Introduce MAGNets (Multiplicative Anisotropic Gabor Networks) as the kernel parameterization basis to control spectral properties.
• Regularize the MAGNet-based representations to mitigate aliasing when deploying at higher resolutions (alias-free MAGNets).
• Provide analytic expressions for the maximum frequency of MAGNets and incorporate them into an aliasing regularization loss.
• Enable deployment at higher resolutions via sampling density adjustments and Cropping/efficient convolution acceleration (e.g., using Fourier-based speedups).
• Conduct extensive experiments on sequential datasets (FlexTCNs/FlexNets) and image datasets (CIFAR-10) comparing against fixed-kernel, dilated, and learned-kernel baselines.

Experimental results

Research questions

• RQ1 Can a convolutional kernel be learned with high bandwidth while its effective size is differentiable and trainable?
• RQ2 Does a continuous, MAGNet-based kernel parameterization allow alias-free deployment at higher resolutions while maintaining or improving accuracy?
• RQ3 Do FlexConvs and FlexNets outperform prior learned-kernel methods on sequential data and compete with pooling-based CNNs on image benchmarks?
• RQ4 Does learning kernel size offer advantages in convergence speed and computational efficiency compared to fixed or dilated kernels?

Key findings

• FlexConvs enable learning high-bandwidth kernels with a fixed parameter count by parameterizing the kernel as a neural network multiplied by a Gaussian mask.
• MAGNets provide analytic control over frequency content, enabling alias-free deployment to higher resolutions with minimal accuracy loss.
• FlexNets achieve state-of-the-art results on several sequential datasets and competitive performance on CIFAR-10 compared with pooling-based architectures and learned-kernel methods.
• Training at low resolutions with alias-free MAGNets can reduce compute time while preserving performance when deployed at higher resolutions.
• Alias regularization focused on MAGNet frequencies improves cross-resolution generalization and stability during upsampling to higher resolutions.