[Paper Review] Scale equivariance in CNNs with vector fields

Diego Marcos, Benjamin Kellenberger+2 more|arXiv (Cornell University)|Jul 31, 2018

Generative Adversarial Networks and Image Synthesis11 references43 citations

TL;DR

The paper introduces scale-equivariant CNNs by applying filters at multiple scales and representing activations as a 2D vector field (magnitude and scale), achieving improved MNIST-scale classification and superior scale-factor regression.

ABSTRACT

We study the effect of injecting local scale equivariance into Convolutional Neural Networks. This is done by applying each convolutional filter at multiple scales. The output is a vector field encoding for the maximally activating scale and the scale itself, which is further processed by the following convolutional layers. This allows all the intermediate representations to be locally scale equivariant. We show that this improves the performance of the model by over $20\%$ in the scale equivariant task of regressing the scaling factor applied to randomly scaled MNIST digits. Furthermore, we find it also useful for scale invariant tasks, such as the actual classification of randomly scaled digits. This highlights the usefulness of allowing for a compact representation that can also learn relationships between different local scales by keeping internal scale equivariance.

Motivation & Objective

• Motivate incorporating local scale information into CNNs for tasks where object size is informative.
• Propose a scale-equivariant convolutional framework that preserves scale information across layers.
• Evaluate the approach on MNIST-scale for both classification and scale-factor regression.
• Demonstrate that keeping scale information improves performance and enables learning scale relationships.

Proposed method

• Convolve inputs with filters at multiple scales and pool across scales to produce per-location max activations.
• Encode the maximally activating scale as the angle of a 2D vector, with the magnitude being the activation strength.
• Process the resulting vector field with vector-field convolutions that consider both magnitude and scale.
• Use a scale range (angle) of 120 degrees to parameterize scale interactions.
• Train end-to-end; for classification, use magnitudes for the FC layer while scale angles contribute to scale regression.
• Keep a vector-field representation (magnitude and angle) rather than a scalar output to enable local scale equivariance.

Experimental results

Research questions

• RQ1 Can local scale equivariance improve performance on scale-variant image classification tasks?
• RQ2 Does preserving scale information through vector-field representations improve scale-factor regression performance?
• RQ3 How does scale equivariance compare to scale invariance for MNIST-scale classification and regression tasks?
• RQ4 What is the impact of the vector-field angle range and multi-scale pooling on learning scale relationships?

Key findings

• Scale-equivariant CNN achieves 2.44% error on MNIST-scale classification, outperforming several baselines including scale-invariant and standard CNN variants.
• Scale-equivariant model reduces scale-factor RMSE to 0.206, outperforming standard (0.254) and scale-invariant (0.256) models.
• Using vector-field representations (magnitude + scale) yields improvements over magnitude-only (scale-invariant) representations on classification.
• The approach enables learning interactions between different local scales, even for tasks that are scale-invariant by nature.
• The method achieves similar or better performance with fewer learnable filters compared to a standard CNN.