[Paper Review] Deep Scale-spaces: Equivariance Over Scale

Daniel E. Worrall, Max Welling|arXiv (Cornell University)|May 28, 2019

Computer Graphics and Visualization TechniquesComputer Science40 references43 citations

TL;DR

The paper introduces deep scale-spaces, a semigroup-equivariant extension of CNNs that achieves scale-equivariant cross-correlations, enabling multiscale information fusion within standard architectures. It demonstrates improved results on Patch Camelyon and Cityscapes and analyzes equivariance properties.

ABSTRACT

We introduce deep scale-spaces (DSS), a generalization of convolutional neural networks, exploiting the scale symmetry structure of conventional image recognition tasks. Put plainly, the class of an image is invariant to the scale at which it is viewed. We construct scale equivariant cross-correlations based on a principled extension of convolutions, grounded in the theory of scale-spaces and semigroups. As a very basic operation, these cross-correlations can be used in almost any modern deep learning architecture in a plug-and-play manner. We demonstrate our networks on the Patch Camelyon and Cityscapes datasets, to prove their utility and perform introspective studies to further understand their properties.

Motivation & Objective

• Motivate the need for explicit scale handling in CNNs due to inherent scale symmetry in vision tasks.
• Generalize convolutions to semigroup-equivariant correlations to handle noninvertible scale transformations.
• Embed scale-space theory into a plug-and-play CNN operation to fuse multiscale information.
• Evaluate scale-equivariant networks on medical (Patch Camelyon) and urban scene (Cityscapes) datasets.
• Analyze the quality of equivariance and boundary effects in finite-scale implementations.

Proposed method

• Define a semigroup correlation that transforms the signal with a scale-space action and computes inner products with filters across the semigroup domain.
• Specialize the semigroup correlation to the scale-space action that combines Gaussian blurs and anisotropic downsampling.
• Lift inputs to the scale-space domain so that scale transformations act simply as shifts in the semigroup index.
• Use a scale-space CNN where dilations correspond to moving up/down scales in the lifted representation and perform dilated convolutions across scales.
• Interleave kernels with different scale-dimensions to mitigate boundary effects and maintain expressiveness.

Experimental results

Research questions

• RQ1 How can scale transformations be modeled in CNNs when they are noninvertible (semigroups) rather than invertible groups?
• RQ2 Can a semigroup-equivariant correlation produce scale-equivariant representations suitable for deep learning architectures?
• RQ3 Do scale-equivariant networks improve performance on real-world vision tasks with multiscale variation (e.g., PCam, Cityscapes) compared to non-equivariant baselines?
• RQ4 What are the practical effects and limitations (e.g., boundary effects, computational overhead) of implementing deep scale-spaces in CNNs?

Key findings

• Scale-equivariant cross-correlations built from semigroup theory enable multiscale information fusion within CNNs.
• Experiments show scale-equivariant models outperform non-equivariant baselines on Patch Camelyon and Cityscapes tasks.
• Equivariance quality remains high (average error typically < 0.01) until boundary effects from scale-space truncation become significant.
• Interleaving scale-dimension 1 and higher-dimension correlations improves performance and helps mitigate boundary issues.
• On Patch Camelyon, S-DenseNet achieves 88.1% accuracy versus 87.0% baseline; Veeling et al. reach 89.8%.
• On Cityscapes, S-ResNet with multiscale interaction reaches 63.53% mAP and 64.78% with no interaction, outperforming matched-parameter baselines (45.66% and 49.99% respectively).