[Paper Review] Low Dimensional Invariant Embeddings for Universal Geometric Learning

Nadav Dym, Steven J. Gortler|arXiv (Cornell University)|May 5, 2022

Neural Networks and Applications4 citations

TL;DR

This paper proposes a method to construct low-dimensional separating invariants for geometric deep learning by randomly selecting $2D+1$ semi-algebraic invariants from a continuous family, achieving universal approximation with far fewer features than traditional methods. The key contribution is proving that only $2D+1$ invariants are sufficient for full orbit separation in $D$-dimensional domains under group actions, drastically reducing computational cost while maintaining theoretical universality.

ABSTRACT

This paper studies separating invariants: mappings on $D$ dimensional domains which are invariant to an appropriate group action, and which separate orbits. The motivation for this study comes from the usefulness of separating invariants in proving universality of equivariant neural network architectures. We observe that in several cases the cardinality of separating invariants proposed in the machine learning literature is much larger than the dimension $D$. As a result, the theoretical universal constructions based on these separating invariants is unrealistically large. Our goal in this paper is to resolve this issue. We show that when a continuous family of semi-algebraic separating invariants is available, separation can be obtained by randomly selecting $2D+1 $ of these invariants. We apply this methodology to obtain an efficient scheme for computing separating invariants for several classical group actions which have been studied in the invariant learning literature. Examples include matrix multiplication actions on point clouds by permutations, rotations, and various other linear groups. Often the requirement of invariant separation is relaxed and only generic separation is required. In this case, we show that only $D+1$ invariants are required. More importantly, generic invariants are often significantly easier to compute, as we illustrate by discussing generic and full separation for weighted graphs. Finally we outline an approach for proving that separating invariants can be constructed also when the random parameters have finite precision.

Motivation & Objective

• To resolve the issue of excessively large numbers of separating invariants in existing universal equivariant network constructions.
• To show that only $2D+1$ invariants are sufficient to separate orbits under continuous group actions on $D$-dimensional domains.
• To provide an efficient, computationally feasible method for constructing separating invariants for classical group actions such as permutations, rotations, and linear groups.
• To explore the trade-off between full separation and generic separation, showing that $D+1$ invariants suffice for generic separation, which is often easier to compute.
• To lay the groundwork for proving universality with finite-precision random parameters in invariant learning.

Proposed method

• Leverage a continuous family of semi-algebraic separating invariants to construct a random embedding using $2D+1$ randomly selected invariants.
• Apply the embedding $\phi(x) = (\phi_1(x), \dots, \phi_m(x))$ with $m = 2D+1$, where each $\phi_i$ is a semi-algebraic invariant function.
• Use the theoretical result that such a random selection ensures orbit separation almost surely, based on transversality and dimension arguments.
• Demonstrate the method on group actions including permutations, rotations, and linear groups acting on point clouds.
• For generic separation, use only $D+1$ invariants, which are easier to compute and sufficient when the singular set is negligible.
• Validate the approach empirically using synthetic point clouds with low intrinsic dimension $D$, showing high test accuracy with $m = 2D+1$.

Experimental results

Research questions

• RQ1 Can the number of separating invariants required for universal geometric learning be reduced below the size of polynomial generator sets?
• RQ2 Is it possible to achieve full orbit separation using only $2D+1$ invariants drawn from a continuous family of semi-algebraic invariants?
• RQ3 How does generic separation with $D+1$ invariants compare in practice to full separation with $2D+1$ invariants in terms of accuracy and computational cost?
• RQ4 What is the stability of the resulting invariant embeddings under perturbations of the group action or data distribution?
• RQ5 Can the theoretical framework be extended to finite-precision random parameters in real-world implementations?

Key findings

• For $D=1$, an embedding with $m=3$ invariants achieved 98% test accuracy on a binary classification task, demonstrating effective separation despite high extrinsic dimension.
• The diagonal entries in the experimental tables confirm that $m = 2D+1$ embeddings achieve high accuracy, validating the theoretical guarantee of injectivity.
• Using $m > 2D+1$ invariants further improves accuracy, consistent with over-parameterization benefits in deep learning.
• Generic separation with $D+1$ invariants is often computationally feasible and sufficient when the singular set has low measure in the data distribution.
• The method reduces the number of required invariants from over 180 million (e.g., power sum polynomials) to just 2048 for a $3\times1024$ point cloud, a reduction by orders of magnitude.
• The approach enables provably universal invariant learning with significantly reduced computational complexity compared to classical invariant theory constructions.