[Paper Review] Equivariance Through Parameter-Sharing

Siamak Ravanbakhsh, Jeff Schneider+1 more|arXiv (Cornell University)|Feb 27, 2017

Gaussian Processes and Bayesian Inference23 references106 citations

TL;DR

The paper shows that neural network layer equivariance to a discrete group action can be achieved by designing parameter-sharing via a colored bipartite graph whose automorphism group matches the desired symmetry.

ABSTRACT

We propose to study equivariance in deep neural networks through parameter symmetries. In particular, given a group $\mathcal{G}$ that acts discretely on the input and output of a standard neural network layer $ϕ_{W}: \Re^{M} o \Re^{N}$, we show that $ϕ_{W}$ is equivariant with respect to $\mathcal{G}$-action iff $\mathcal{G}$ explains the symmetries of the network parameters $W$. Inspired by this observation, we then propose two parameter-sharing schemes to induce the desirable symmetry on $W$. Our procedures for tying the parameters achieve $\mathcal{G}$-equivariance and, under some conditions on the action of $\mathcal{G}$, they guarantee sensitivity to all other permutation groups outside $\mathcal{G}$.

Motivation & Objective

• Motivate encoding domain symmetries in neural networks via parameter-sharing rather than data augmentation alone.
• Formalize the link between network parameter symmetries and equivariance to group actions.
• Propose two schemes (dense and sparse) for parameter-sharing that induce G-equivariance.
• Provide conditions under which unique G-equivariance is guaranteed in neural layers.

Proposed method

• Represent a neural layer as a colored multi-edged bipartite graph Omega where edges with the same color share parameters.
• Show that the layer phi(x; w, Omega) is uniquely Aut(Omega)-equivariant when edge colors (parameters) are distinct, establishing a link between graph automorphisms and equivariance.
• Provide a dense design where Omega ties edges by G_N,M-orbits, guaranteeing G_N,M-equivariance.
• Introduce a sparse design using orbits and symmetric generating sets A to achieve Aut(Omega) containing G_N,M and, under semi-regular actions, equality Aut(Omega) = G_N,M.
• Extend the scheme to multiple inputs/outputs and discuss composition of layers for deep networks.

Experimental results

Research questions

• RQ1 Can discrete group actions on input and output indices be exactly captured by parameter-sharing structures in neural layers?
• RQ2 What are the sufficient conditions on the shared-parameter graphs to guarantee unique G-equivariance?
• RQ3 How can dense and sparse parameter-sharing designs be constructed to realize a given group action?
• RQ4 How do these designs extend to multi-channel layers and deep architectures?

Key findings

• A colored bipartite graph Omega can encode parameter-sharing such that the neural layer is uniquely equivariant to the automorphism group Aut(Omega).
• Corollary: any subgroup H of Aut(Omega) yields H-equivariance, enabling controllable symmetry guarantees.
• Dense design links G_N,M actions to edge-orbit colors, ensuring equivariance for the full group, though not always unique.
• Sparse design using orbits and symmetric generating sets yields Aut(Omega) containing G_N,M and, under semi-regularity, equality, enabling unique equivariance with potentially fewer parameters.
• The framework subsumes special cases like group convolution, permutation-equivariant layers, and set-based architectures as instances of the parameter-sharing approach.