[Paper Review] Probabilistic symmetries and invariant neural networks

Benjamin Bloem-Reddy, Yee Whye Teh|arXiv (Cornell University)|Jan 18, 2019

Neural Networks and Applications109 references52 citations

TL;DR

The paper develops a probabilistic framework linking functional and probabilistic symmetry, yielding exact representations for invariant and equivariant neural networks under compact groups, and provides a general program to construct symmetric models for sequences, arrays, and graphs.

ABSTRACT

Treating neural network inputs and outputs as random variables, we characterize the structure of neural networks that can be used to model data that are invariant or equivariant under the action of a compact group. Much recent research has been devoted to encoding invariance under symmetry transformations into neural network architectures, in an effort to improve the performance of deep neural networks in data-scarce, non-i.i.d., or unsupervised settings. By considering group invariance from the perspective of probabilistic symmetry, we establish a link between functional and probabilistic symmetry, and obtain generative functional representations of probability distributions that are invariant or equivariant under the action of a compact group. Our representations completely characterize the structure of neural networks that can be used to model such distributions and yield a general program for constructing invariant stochastic or deterministic neural networks. We demonstrate that examples from the recent literature are special cases, and develop the details of the general program for exchangeable sequences and arrays.

Motivation & Objective

• Motivate and formalize neural network architectures that respect symmetry (invariance or equivariance) under group actions.
• Link functional symmetry (deterministic mappings) with probabilistic symmetry (conditional distributions).
• Provide functional representations for invariant/equivariant conditional distributions to guide network design.
• Develop a general construction program for symmetric stochastic or deterministic networks applicable to sequences, arrays, and graphs.

Proposed method

• Define and relate functional symmetry (invariance/equivariance of functions) to probabilistic symmetry (invariance/equivariance of conditional distributions).
• Introduce noise-outsourced functional representations Y = f(η, X) that realize invariant or equivariant conditionals with η independent of X.
• Characterize invariant conditional distributions via maximal invariants M(X) and representations (X, Y) a.s. = (X, f(η, M(X))).
• Characterize equivariant conditionals via representations (X, Y) a.s. = (X, f(η, X)) with f satisfying g·Y = f(η, g·X) for all g in G.
• Specialize to exchangeable sequences/arrays/graphs and provide canonical forms (e.g., empirical measures, canonical CX, representative equivariants).
• Discuss practical considerations for constructing symmetric networks and the role of stochasticity and function class choice.

Experimental results

Research questions

• RQ1 What are the necessary and sufficient probabilistic conditions for Y|X to be invariant or equivariant under a group action G?
• RQ2 How can invariant/equivariant conditional distributions be represented functionally, including noise-outsourced forms?
• RQ3 How do maximal invariants and sufficiency concepts enable practical neural network architectures that respect symmetry?
• RQ4 How can the framework be specialized to exchangeable sequences, arrays, and graphs to yield concrete network designs?
• RQ5 What guidelines arise for choosing function classes and incorporating stochasticity in symmetric architectures?

Key findings

• Invariant conditional distributions given exchangeable inputs admit a noise-outside functional representation Y = f(η, MX) with η independent of X.
• Equivariant conditionals under exchangeable inputs admit representations that preserve permutation structure through functions of η and MX, with appropriate symmetry constraints on f.
• For general compact groups, invariant and equivariant conditionals can be represented via maximal invariants M(X) and representative equivariants, enabling systematic neural network construction.
• The empirical measure and canonical forms play a central role as maximal invariants that capture all relevant information under permutations.
• The framework extends to exchangeable matrices, graphs, and higher-dimensional arrays, providing analogous canonical representations (CANON, CX) and broadcasted features to achieve symmetry.
• The approach yields a unifying, probabilistic perspective that encompasses existing invariant architectures as special cases and offers a general program for symmetric network design.