[Paper Review] Universal approximations of permutation invariant/equivariant functions by deep neural networks

Akiyoshi Sannai, Yuuki Takai+1 more|arXiv (Cornell University)|Mar 5, 2019

Neural Networks and Applications24 references49 citations

TL;DR

The paper proves invariant/equivariant universal approximation theorems for finite group actions and constructs deep neural networks that approximate G-invariant/equivariant functions with exponentially fewer parameters using representation theory.

ABSTRACT

In this paper, we develop a theory about the relationship between $G$-invariant/equivariant functions and deep neural networks for finite group $G$. Especially, for a given $G$-invariant/equivariant function, we construct its universal approximator by deep neural network whose layers equip $G$-actions and each affine transformations are $G$-equivariant/invariant. Due to representation theory, we can show that this approximator has exponentially fewer free parameters than usual models.

Motivation & Objective

• Motivate the study of neural networks that respect permutation invariance/equivariance.
• Establish invariant/equivariant universal approximation theorems for finite groups.
• Quantify parameter efficiency of invariant/equivariant architectures using representation theory.

Proposed method

• Define G-invariant and G-equivariant networks under finite group actions via embeddings into symmetric groups.
• Leverage Kolmogorov–Arnold representations to construct networks approximating invariant functions.
• Prove that G-equivariant maps can be represented using Stabilizer subgroups and invariant components.
• Show that the number of free parameters in invariant/equivariant networks is exponentially smaller than in standard networks.

Experimental results

Research questions

• RQ1 Can every continuous G-equivariant function be uniformly approximated by a G-equivariant neural network with ReLU activations?
• RQ2 How can stabilizer subgroups be used to reduce G-equivariant approximation to invariant approximation?
• RQ3 What is the parameter-count comparison between invariant/equivariant networks and usual networks for finite groups?
• RQ4 How does the construction specialize to the symmetric group S_n and relate to existing invariant/equivariant architectures?

Key findings

• An invariant/equivariant universal approximation theorem is established for any finite group G.
• For G = S_n, the equivariant approximator aligns closely with permutation-equivariant models like Zaheer et al. (2017).
• The number of free parameters in invariant/equivariant models is exponentially smaller than in usual models under a union-of-permutations action.
• A constructive method reduces G-equivariant function approximation to stabilizer-invariant components.
• The approach uses representation theory to bound widths/depths and to justify parameter efficiency.