[Paper Review] A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups

Marc Finzi, Max Welling+1 more|arXiv (Cornell University)|Apr 19, 2021

Topic Modeling40 references25 citations

TL;DR

The paper develops a general algorithm to build equivariant multilayer perceptrons (EMLPs) for any matrix group, including groups like O(5), O(1,3), Sp(n), and the Rubik’s cube group, with an efficient Krylov-based method and a bilinear layer.

ABSTRACT

Symmetries and equivariance are fundamental to the generalization of neural networks on domains such as images, graphs, and point clouds. Existing work has primarily focused on a small number of groups, such as the translation, rotation, and permutation groups. In this work we provide a completely general algorithm for solving for the equivariant layers of matrix groups. In addition to recovering solutions from other works as special cases, we construct multilayer perceptrons equivariant to multiple groups that have never been tackled before, including $\mathrm{O}(1,3)$, $\mathrm{O}(5)$, $\mathrm{Sp}(n)$, and the Rubik's cube group. Our approach outperforms non-equivariant baselines, with applications to particle physics and dynamical systems. We release our software library to enable researchers to construct equivariant layers for arbitrary matrix groups.

Motivation & Objective

• Motivate and formalize the need for neural networks that respect arbitrary group symmetries across diverse domains.
• Provide a complete, general algorithm to characterize and construct equivariant linear layers between representations of any matrix group.
• Enable practical deployment by offering efficient computation of equivariant bases and scalable network architectures.
• Demonstrate the approach on previously infeasible groups (e.g., O(5), O(1,3), Sp(n), Rubik’s cube group) and release a usable software library.

Proposed method

• Formulate the equivariance constraint for maps between representations: rho2(g) W rho1(g)^{-1} = W.
• Reduce the infinite set of constraints to a finite set of O(M+D) conditions, where M is the number of discrete generators and D is the Lie group dimension.
• Construct the equivariant basis by solving the homogeneous system via SVD, yielding v in the nullspace C v = 0 and expressing equivariant maps as v = Q β.
• Introduce a fast Krylov-based nullspace procedure to compute the basis efficiently without forming large representation matrices.
• Exploit block structure and Kronecker products to accelerate computations, achieving runtime comparable to standard MLPs.
• Assemble Equivariant MLPs (EMLP) using equivariant linear layers, gated nonlinearities, and a cheap bilinear layer to enable contractions between different tensor representations.

Experimental results

Research questions

• RQ1 Can we derive a complete and practical algorithm for constructing equivariant layers for arbitrary matrix groups and representations?
• RQ2 How can the equivariant basis be computed efficiently for large or continuous groups without enumerating the entire group?
• RQ3 Do EMLPs outperform non-equivariant baselines on tasks with symmetries from groups like O(5), O(1,3), Sp(n), and the Rubik’s cube group?
• RQ4 Is it feasible to build full EMLPs for new groups by specifying group generators and representations?

Key findings

• The method reduces equivariance constraints to O(M+D) finite conditions and provides a polynomial-time solver.
• A fast Krylov-based algorithm computes the equivariant basis efficiently, enabling high-dimensional representations to be handled in practice.
• EMLPs constructed with the proposed method outperform non-equivariant baselines in synthetic and dynamical-system tasks.
• The framework successfully handles previously infeasible groups such as O(5), O(1,3), Sp(n), and the Rubik’s cube group using the same underlying architecture.
• Empirical results demonstrate data efficiency and competitive runtime comparable to standard MLPs.