[Paper Review] Scale-Equivariant Steerable Networks

Ivan Sosnovik, Michał Szmaja+1 more|arXiv (Cornell University)|Oct 14, 2019

Geometric and Algebraic Topology33 references43 citations

TL;DR

The paper develops a theory of scale-equivariant neural networks using steerable filters, enabling scale Convolution on the scale-translation group and achieving state-of-the-art results on MNIST-scale and STL-10 without interpolations.

ABSTRACT

The effectiveness of Convolutional Neural Networks (CNNs) has been substantially attributed to their built-in property of translation equivariance. However, CNNs do not have embedded mechanisms to handle other types of transformations. In this work, we pay attention to scale changes, which regularly appear in various tasks due to the changing distances between the objects and the camera. First, we introduce the general theory for building scale-equivariant convolutional networks with steerable filters. We develop scale-convolution and generalize other common blocks to be scale-equivariant. We demonstrate the computational efficiency and numerical stability of the proposed method. We compare the proposed models to the previously developed methods for scale equivariance and local scale invariance. We demonstrate state-of-the-art results on MNIST-scale dataset and on STL-10 dataset in the supervised learning setting.

Motivation & Objective

• Motivate and formalize how to handle scale transformations in CNNs beyond translation equivariance.
• Introduce steerable filter parametrization to enable scale-equivariant convolutions without tensor resizing.
• Derive efficient algorithms for scale Convolution on the scale-translation group.
• Demonstrate computational efficiency and numerical stability of SESN.
• Empirically compare SESN to existing scale-equivariant and locally scale-invariant methods on MNIST-scale and STL-10.

Proposed method

• Define scale transformations and steerable filters with a parameterization that makes filter scaling explicit (Equation 3).
• Construct scale-convolution on the scale-translation group H using group-equivariant convolution (Equation 5–7).
• Ensure nonlinearities commute with scaling to preserve equivariance (Equation 8).
• Provide pooling strategies to obtain scale-invariant representations (max scale projection and spatial/global pooling).
• Present practical implementation with a steerable filter basis to avoid tensor resizing (basis ; use Hermite polynomials with Gaussian envelope).
• Offer efficient computational schemes for Conv T->H and Conv H->H including interscale interaction options (Equations 9–10).

Experimental results

Research questions

• RQ1 How to build CNN layers that are equivariant to discrete scale factors in addition to translations?
• RQ2 Can steerable filter parametrizations enable fast, interpolation-free scale-convolution across multiple scales?
• RQ3 What are effective pooling strategies to obtain scale invariance without losing semantic information?
• RQ4 How does SESN compare to existing scale-equivariant or locally scale-invariant architectures in terms of accuracy and efficiency?

Key findings

• SESN achieves scale and translation equivariance with arbitrary discrete scale factors using steerable filters.
• The proposed scale-convolution framework enables fast, interpolation-free computations with comparable speed to standard CNNs.
• SESN outperforms prior scale-equivariant and local scale-invariant methods on MNIST-scale, with both scalar and vector variants showing strong gains.
• On STL-10, SESN variants attain state-of-the-art supervised accuracy, surpassing several baselines.
• Using interscale interaction can improve performance, but may introduce higher equivariance error; careful design (e.g., scale projections) yields robust gains.
• SESN demonstrates low equivariance error under realistic discretization and deep architectures, validating its practical viability.