[Paper Review] Universal Invariant and Equivariant Graph Neural Networks

Nicolas Keriven, Gabriel Peyré|arXiv (Cornell University)|May 13, 2019

Advanced Graph Neural Networks46 references81 citations

TL;DR

The paper proves universality for a class of one-hidden-layer invariant and equivariant Graph Neural Networks (GNNs) using Stone-Weierstrass-type arguments, and extends results to functions on graphs of varying size.

ABSTRACT

Graph Neural Networks (GNN) come in many flavors, but should always be either invariant (permutation of the nodes of the input graph does not affect the output) or equivariant (permutation of the input permutes the output). In this paper, we consider a specific class of invariant and equivariant networks, for which we prove new universality theorems. More precisely, we consider networks with a single hidden layer, obtained by summing channels formed by applying an equivariant linear operator, a pointwise non-linearity and either an invariant or equivariant linear operator. Recently, Maron et al. (2019) showed that by allowing higher-order tensorization inside the network, universal invariant GNNs can be obtained. As a first contribution, we propose an alternative proof of this result, which relies on the Stone-Weierstrass theorem for algebra of real-valued functions. Our main contribution is then an extension of this result to the equivariant case, which appears in many practical applications but has been less studied from a theoretical point of view. The proof relies on a new generalized Stone-Weierstrass theorem for algebra of equivariant functions, which is of independent interest. Finally, unlike many previous settings that consider a fixed number of nodes, our results show that a GNN defined by a single set of parameters can approximate uniformly well a function defined on graphs of varying size.

Motivation & Objective

• Extend universal approximation results to invariant and equivariant GNNs with a single hidden layer.
• Provide alternative proofs leveraging Stone-Weierstrass theory for invariants and a new equivariant Stone-Weierstrass theorem.
• Show uniform approximation on graphs of different sizes using a single parameter set.
• Characterize the parameter count and tensorization needed for universality in invariant/equivariant settings.

Proposed method

• Consider one-layer GNNs of the form f(G)=sum_s H_s[ρ(F_s[G]+B_s)]+b with F_s invariant/equivariant and H_s invariant or equivariant.
• Use Kronecker-augmented networks to form a closed subalgebra of continuous invariant functions.
• Apply Stone-Weierstrass theorem to prove dense approximation of continuous invariant functions on the graph set.
• Develop a generalized Stone-Weierstrass theorem for equivariant functions to handle vector-valued outputs over graphs.
• Prove universality for graphs of varying size with a single parameter set, and analyze tensorization orders k_s.

Experimental results

Research questions

• RQ1 Can a single set of parameters in a shallow invariant GNN approximate any continuous invariant function on graphs of varying sizes?
• RQ2 Can the universality be extended to equivariant GNNs with vector-valued outputs while using permutation symmetry?
• RQ3 What are the minimal tensorization orders required inside the network to achieve universality in invariant and equivariant cases?
• RQ4 How does the proof adapt Stone-Weierstrass theory to the equivariant, high-order-graph setting?
• RQ5 What are the limitations when outputs are graph- or higher-order tensors rather than scalars?

Key findings

• Invariant GNNs with a single parameter set are dense in the space of continuous invariant functions on graph collections, including varying sizes.
• Equivariant GNNs with a single parameter set are dense in the space of continuous equivariant functions, via a generalized Stone-Weierstrass theorem.
• The approach uses Kronecker-augmented networks to form a closed algebra of functions and separate points to apply density results.
• Higher-order tensorization within the network is crucial for practical universality, as increasing tensorization order reduces approximation error in experiments.
• The results demonstrate uniform approximation across graphs with different numbers of nodes (up to a max n_max) using the same parameters.