[Paper Review] Geometric Deep Learning and Equivariant Neural Networks

Jan E. Gerken, Jimmy Aronsson+5 more|arXiv (Cornell University)|May 28, 2021

Topological and Geometric Data Analysis5 citations

TL;DR

This paper presents a rigorous mathematical framework for geometric deep learning using group and gauge equivariant neural networks on manifolds, leveraging principal bundles and induced representations. It establishes that equivariant layers are intertwiners between representations, enabling state-of-the-art performance in tasks like semantic segmentation and object detection on spherical and curved data via Fourier analysis on SO(3).

ABSTRACT

We survey the mathematical foundations of geometric deep learning, focusing on group equivariant and gauge equivariant neural networks. We develop gauge equivariant convolutional neural networks on arbitrary manifolds $\mathcal{M}$ using principal bundles with structure group $K$ and equivariant maps between sections of associated vector bundles. We also discuss group equivariant neural networks for homogeneous spaces $\mathcal{M}=G/K$, which are instead equivariant with respect to the global symmetry $G$ on $\mathcal{M}$. Group equivariant layers can be interpreted as intertwiners between induced representations of $G$, and we show their relation to gauge equivariant convolutional layers. We analyze several applications of this formalism, including semantic segmentation and object detection networks. We also discuss the case of spherical networks in great detail, corresponding to the case $\mathcal{M}=S^2=\mathrm{SO}(3)/\mathrm{SO}(2)$. Here we emphasize the use of Fourier analysis involving Wigner matrices, spherical harmonics and Clebsch-Gordan coefficients for $G=\mathrm{SO}(3)$, illustrating the power of representation theory for deep learning.

Motivation & Objective

• To develop a unified mathematical foundation for geometric deep learning using group and gauge equivariance.
• To formalize equivariant convolutional layers on arbitrary manifolds using principal bundles and sections of associated vector bundles.
• To connect group equivariant layers to intertwiners between induced representations of symmetry groups.
• To apply the formalism to real-world tasks such as semantic segmentation and object detection on non-Euclidean data.
• To provide a detailed treatment of spherical networks using SO(3) representation theory, including Wigner matrices and Clebsch–Gordan coefficients.

Proposed method

• Formalizes gauge equivariant convolutions using principal bundles with structure group K and equivariant maps between sections of associated vector bundles.
• Derives the general form of gauge equivariant convolutional layers via the theory of induced representations and intertwiners.
• Applies the framework to homogeneous spaces M = G/K, where equivariance is with respect to the global symmetry group G.
• Uses Fourier analysis on SO(3) via spherical harmonics, Wigner matrices, and Clebsch–Gordan coefficients to construct spherical convolutions.
• Constructs SE(3)-equivariant networks by projecting regular representations of SE(3) onto 2D image planes, enabling equivariance under 3D rigid transformations.
• Demonstrates that group equivariant layers are constrained by kernel invariance under group actions, leading to a systematic design of equivariant layers.

Experimental results

Research questions

• RQ1 How can group and gauge equivariant neural networks be systematically constructed on arbitrary manifolds using differential geometry and fiber bundle theory?
• RQ2 What is the precise mathematical relationship between equivariant layers and intertwiners between induced representations of a symmetry group?
• RQ3 How can spherical convolutions be derived and implemented using representation theory of SO(3), including Wigner matrices and Clebsch–Gordan coefficients?
• RQ4 To what extent does equivariance improve performance in semantic segmentation and object detection on non-Euclidean data such as fisheye images or spherical signals?
• RQ5 What are the theoretical and practical challenges in extending equivariance to non-transitive group actions and non-linear manifolds like those in pinhole camera models?

Key findings

• The paper establishes that group equivariant convolutional layers are mathematically equivalent to intertwiners between induced representations of the symmetry group G.
• Gauge equivariant networks on manifolds are constructed using sections of associated vector bundles over principal bundles, with equivariance ensured via structure group K.
• For spherical data (M = S² = SO(3)/SO(2)), the formalism enables efficient computation using Fourier transforms based on spherical harmonics and Wigner matrices.
• The use of Clebsch–Gordan coefficients allows for the decomposition of feature maps into irreducible representations, enabling structured and efficient layer design.
• The framework successfully enables SE(3)-equivariant object detection by projecting 3D rigid transformations onto 2D image planes, with output features transforming under the regular representation of SE(3).
• The authors demonstrate that equivariance improves per-sample efficiency and reduces reliance on data augmentation, with theoretical justification for linear models and empirical support in non-linear settings.