[Paper Review] Equivariant Transformer Networks

Kai Sheng Tai, Peter Bailis+1 more|arXiv (Cornell University)|Jan 25, 2019

Domain Adaptation and Few-Shot Learning32 references33 citations

TL;DR

Equivariant Transformer (ET) layers incorporate domain invariances via canonical coordinates to produce image-to-image mappings that are self-consistent to predefined transformation groups, improving robustness and sample efficiency with minimal parameter overhead; ETs outperform standard ST layers on projective distortions and can reduce network size while maintaining accuracy.

ABSTRACT

How can prior knowledge on the transformation invariances of a domain be incorporated into the architecture of a neural network? We propose Equivariant Transformers (ETs), a family of differentiable image-to-image mappings that improve the robustness of models towards pre-defined continuous transformation groups. Through the use of specially-derived canonical coordinate systems, ETs incorporate functions that are equivariant by construction with respect to these transformations. We show empirically that ETs can be flexibly composed to improve model robustness towards more complicated transformation groups in several parameters. On a real-world image classification task, ETs improve the sample efficiency of ResNet classifiers, achieving relative improvements in error rate of up to 15% in the limited data regime while increasing model parameter count by less than 1%.

Motivation & Objective

• Leverage prior knowledge of domain transformation invariances to improve neural network robustness.
• Develop a differentiable layer family that enforces invariance to predefined continuous transformation groups.
• Enable composition of multiple transformation invariances through stacking ET layers.
• Demonstrate practical gains in sample efficiency and accuracy on real-world and synthetic image tasks.

Proposed method

• Introduce Equivariant Transformer (ET) layers that use canonical coordinates for a given transformation group G to achieve self-consistency.
• Define an ET layer L_{G,ρ}(φ) = T^{-1}_{f_{ρ}(φ)} φ where f_{ρ} is a self-consistent pose predictor in canonical coordinates ρ.
• Construct canonical coordinates ρ for Lie groups by solving first-order PDEs via the method of characteristics.
• Stack ET layers to handle compositions of multiple transformation groups when a single ρ cannot capture all parameters.
• Implement lazy resampling to avoid repeated interpolation when stacking ET layers, and apply bilinear interpolation for coordinate changes.

Experimental results

Research questions

• RQ1 How can canonical coordinates for Lie groups be used to enforce invariance to continuous transformations in neural networks?
• RQ2 Can Equivariant Transformer layers improve robustness and sample efficiency compared to Spatial Transformers on transformation-heavy tasks?
• RQ3 How effective is stacking ET layers for multi-parameter transformation groups in practice?
• RQ4 What are the practical considerations for implementing ET layers with minimal interpolation artifacts?

Key findings

• ET layers yield local invariance by construction, collapsing transformation orbits to a canonical representation.
• On Projective MNIST, ETs with log-polar coordinates and transformations outperform standard STs and Cartesian CNN baselines across varying numbers of transformed samples.
• ETs achieve up to 15% relative error rate improvement over baselines in limited data regimes on SVHN-like tasks.
• A ResNet-10 with ET layers can exceed the accuracy of a larger ResNet-34 without ETs, with reduced memory and compute costs.
• Stacking ET layers provides practical benefits for multi-parameter transformation groups, improving robustness beyond single-parameter canonical coordinates.