[Paper Review] Covariant Compositional Networks For Learning Graphs

Risi Kondor, Hy Truong Son+3 more|arXiv (Cornell University)|Jan 7, 2018

Advanced Graph Neural Networks39 references88 citations

TL;DR

The paper introduces Covariant Compositional Networks (CCNs) for graphs, promising higher expressive power by enforcing covariance under permutations via tensor representations, improving over conventional invariant message passing neural networks.

ABSTRACT

Most existing neural networks for learning graphs address permutation invariance by conceiving of the network as a message passing scheme, where each node sums the feature vectors coming from its neighbors. We argue that this imposes a limitation on their representation power, and instead propose a new general architecture for representing objects consisting of a hierarchy of parts, which we call Covariant Compositional Networks (CCNs). Here, covariance means that the activation of each neuron must transform in a specific way under permutations, similarly to steerability in CNNs. We achieve covariance by making each activation transform according to a tensor representation of the permutation group, and derive the corresponding tensor aggregation rules that each neuron must implement. Experiments show that CCNs can outperform competing methods on standard graph learning benchmarks.

Motivation & Objective

• Motivate and formalize the limitation of invariant message passing neural networks (MPNNs) for graphs.
• Propose covariant compositional networks that respect permutation covariance through tensor representations of the permutation group.
• Develop a general comp-net framework and show how standard graph neural models are special cases.
• Derive tensor aggregation rules that preserve covariance across graph hierarchies.
• Demonstrate empirical improvements over scalar MPNNs on standard graph benchmarks.

Proposed method

• Define compositional networks (comp-nets) as DAGs with parts corresponding to atoms and receptive fields for hierarchical representation.
• Introduce permutation invariance and covariance principles to ensure representations do not depend on vertex labeling.
• Show that standard MPNNs are special cases of comp-nets and discuss their limitations in expressive power.
• Introduce covariant comp-nets where activations transform according to tensor representations of the permutation group (first, second, third order covariants, etc.).
• Derive aggregation rules using tensor contractions (e.g., Pπ for first order, Pπ ⊗ Pπ for second order) to maintain covariance.
• Associate layers with receptive fields and define how f_i (or F_i) transforms under permutations of its receptive field as per defined representations.

Experimental results

Research questions

• RQ1 How do current graph learning methods limit representation power due to invariant design under vertex permutations?
• RQ2 Can graph representations be enhanced by enforcing covariance rather than mere invariance under permutations?
• RQ3 What is the appropriate tensor-based framework to implement covariant activations in comp-nets for graphs?
• RQ4 How do covariant comp-nets relate to and generalize existing message passing neural networks?
• RQ5 What are the practical implications and potential performance gains of CCNs on standard graph benchmarks?

Key findings

• CCNs provide a principled way to incorporate permutation covariance in graph representations via tensor activations.
• Standard MPNNs are recovered as a special case within the comp-net framework.
• Covariant activations can be organized as first, second, and higher-order tensor representations under the permutation group.
• Aggregation rules correspond to contractions of higher-order tensors with permutation representations.
• Experiments show CCNs can outperform scalar MPNNs on standard graph learning benchmarks.