[Paper Review] Scale Steerable Filters for Locally Scale-Invariant Convolutional Neural Networks

Rohan Ghosh, Anupam K. Gupta|arXiv (Cornell University)|Jun 10, 2019

Optical measurement and interference techniquesComputer Science11 references35 citations

TL;DR

The paper introduces log-radial harmonic scale-steerable filters to build a locally scale-invariant CNN (SS-CNN) and demonstrates improved performance on MNIST-Scale and FMNIST-Scale compared to baselines.

ABSTRACT

Augmenting transformation knowledge onto a convolutional neural network's weights has often yielded significant improvements in performance. For rotational transformation augmentation, an important element to recent approaches has been the use of a steerable basis i.e. the circular harmonics. Here, we propose a scale-steerable filter basis for the locally scale-invariant CNN, denoted as log-radial harmonics. By replacing the kernels in the locally scale-invariant CNN \cite{lsi_cnn} with scale-steered kernels, significant improvements in performance can be observed on the MNIST-Scale and FMNIST-Scale datasets. Training with a scale-steerable basis results in filters which show meaningful structure, and feature maps demonstrate which demonstrate visibly higher spatial-structure preservation of input. Furthermore, the proposed scale-steerable CNN shows on-par generalization to global affine transformation estimation methods such as Spatial Transformers, in response to test-time data distortions.

Motivation & Objective

• Motivate the need for transformation-aware CNN weights beyond rotations and translations.
• Propose a scale-steerable basis (log-radial harmonics) to enable exact scale steering of filters.
• Integrate the scale-steered basis into a locally scale-invariant CNN (SS-CNN) and train end-to-end.
• Demonstrate improved accuracy and structure preservation on scale-varied datasets (MNIST-Scale, FMNIST-Scale) and with local scale variations.

Proposed method

• Define log-radial harmonic basis filters that are steerable in scale and orientation.
• Represent each kernel as the real part of a linear combination of the basis filters with complex coefficients.
• Derive the scale-steering relation W^s(as)=s^{m-2} e^{-i k log s} (sum over basis) and take the real part for real weights.
• Construct SS-CNN by convolving with scale-steered filters and performing max-pooling over scales to obtain scale-invariant representations.
• Compare against baselines (LocScaleInv-CNN, LocScaleEq-CNN, Spatial Transformer) on MNIST-Scale and FMNIST-Scale, using similar parameter budgets.

Experimental results

Research questions

• RQ1 Can log-radial harmonic scale-steerable filters provide exact scale steering for CNNs?
• RQ2 Do SS-CNNs improve robustness and accuracy on datasets with global and local scale variations compared to existing scale-invariant approaches?
• RQ3 How do scale-steered filters impact filter structure and intermediate feature maps?
• RQ4 How does SS-CNN perform relative to Spatial Transformer Networks in handling test-time distortions?
• RQ5 What design choices (centrality, scale sampling) influence transformation sensitivity and performance?

Key findings

• SS-CNN achieves competitive accuracy to Spatial Transformer Networks on MNIST-Scale and outperforms several scale-equivariant baselines.
• Scale-steerable filters yield more structured, central, and interpretable first-layer filters, with better preservation of spatial structure in feature maps.
• On elastic distortions, SS-CNN remains robust and often outperforms baselines, demonstrating strong generalization under distortions.
• For local scale variations, SS-CNN shows superior performance, especially with limited training data, compared to a Spatial Transformer baseline.
• Visualizations indicate higher spatial-structure preservation in SS-CNN activations compared to non-scale-steered counterparts.