[Paper Review] Invariant and Equivariant Graph Networks

Haggai Maron, Heli Ben-Hamu+2 more|arXiv (Cornell University)|Dec 24, 2018

Advanced Graph Neural Networks113 citations

TL;DR

This paper provides a complete characterization of permutation invariant and equivariant linear layers for (hyper-)graph data, giving Bell-number-based basis functions and demonstrating universal approximation power relative to message-passing networks."

ABSTRACT

Invariant and equivariant networks have been successfully used for learning images, sets, point clouds, and graphs. A basic challenge in developing such networks is finding the maximal collection of invariant and equivariant linear layers. Although this question is answered for the first three examples (for popular transformations, at-least), a full characterization of invariant and equivariant linear layers for graphs is not known. In this paper we provide a characterization of all permutation invariant and equivariant linear layers for (hyper-)graph data, and show that their dimension, in case of edge-value graph data, is 2 and 15, respectively. More generally, for graph data defined on k-tuples of nodes, the dimension is the k-th and 2k-th Bell numbers. Orthogonal bases for the layers are computed, including generalization to multi-graph data. The constant number of basis elements and their characteristics allow successfully applying the networks to different size graphs. From the theoretical point of view, our results generalize and unify recent advancement in equivariant deep learning. In particular, we show that our model is capable of approximating any message passing neural network Applying these new linear layers in a simple deep neural network framework is shown to achieve comparable results to state-of-the-art and to have better expressivity than previous invariant and equivariant bases.

Motivation & Objective

• Motivate learning on graphs/hyper-graphs with symmetry considerations (permutation invariance/equivariance).
• Characterize the full space of invariant and equivariant linear layers for tensor-encoded graph data.
• Provide computable orthogonal bases and show basis maximality and size-independence (n).
• Demonstrate that the model can approximate any message-passing neural network and perform competitively on graph tasks.

Proposed method

• Derive fixed-point equations for permutation invariance (L vec(A) fixed under P^{⊗k}) and equivariance (L vec(A) fixed under P^{⊗2k}).
• Characterize solution spaces using equality-pattern partitions of index sets, leading to Bell-number-sized bases B^{γ}.
• Construct explicit invariant/equivariant basis tensors B^{γ} and C^{λ} (and their multi-set/generalized forms) to span the solution spaces.
• Extend to biased layers and feature-bearing layers, deriving explicit parameter counts and basis forms (Theorem 2).
• Generalize to multi-set node partitions and mixed-order tensors, yielding dimensions ∏ b(k_i) for invariants and ∏ b(k_i+l_i) for equivariants.
• Argue universality by showing the model can approximate any message-passing neural network.

Experimental results

Research questions

• RQ1 What is the maximal dimensionality of permutation invariant/equivariant linear layers for graph/hyper-graph data?
• RQ2 Can we construct an orthogonal basis that spans all invariant/equivariant linear layers independent of graph size n?
• RQ3 Do invariant/equivariant layers subsume or match the expressivity of existing graph networks, including message-passing models?
• RQ4 How do biases and node features integrate into the invariant/equivariant linear layer framework?
• RQ5 Can the proposed bases generalize to multi-node sets and mixed-order tensors while preserving size-independence?

Key findings

• The space of invariant linear layers R^{n^k} -> R has dimension b(k) (Bell number of k).
• The space of equivariant linear layers R^{n^k} -> R^{n^k} has dimension b(2k).
• For edge-valued graphs (k=2), invariant layers have dimension 2 and equivariant layers have dimension 15, independent of n.
• The paper provides explicit orthogonal bases B^{γ} and C^{λ} to span these spaces (and their biased variants).
• The framework extends to multi-graph/multi-set scenarios and mixed-order maps with dimensions ∏ b(k_i) and ∏ b(k_i+l_i).
• Appendix results indicate the model can approximate any message-passing neural network, establishing a universal approximation capability within this invariant/equivariant linear-layer paradigm.