[Paper Review] Efficient Graph Generation with Graph Recurrent Attention Networks

Renjie Liao, Yujia Li+7 more|arXiv (Cornell University)|Oct 2, 2019

Advanced Graph Neural NetworksComputer Science173 citations

TL;DR

GRAN generates graphs in blocks with graph neural network based attention to improve conditioning, using a mixture Bernoulli output and canonical orderings to achieve efficient and scalable graph generation up to 5K nodes.

ABSTRACT

We propose a new family of efficient and expressive deep generative models of graphs, called Graph Recurrent Attention Networks (GRANs). Our model generates graphs one block of nodes and associated edges at a time. The block size and sampling stride allow us to trade off sample quality for efficiency. Compared to previous RNN-based graph generative models, our framework better captures the auto-regressive conditioning between the already-generated and to-be-generated parts of the graph using Graph Neural Networks (GNNs) with attention. This not only reduces the dependency on node ordering but also bypasses the long-term bottleneck caused by the sequential nature of RNNs. Moreover, we parameterize the output distribution per block using a mixture of Bernoulli, which captures the correlations among generated edges within the block. Finally, we propose to handle node orderings in generation by marginalizing over a family of canonical orderings. On standard benchmarks, we achieve state-of-the-art time efficiency and sample quality compared to previous models. Additionally, we show our model is capable of generating large graphs of up to 5K nodes with good quality. To the best of our knowledge, GRAN is the first deep graph generative model that can scale to this size. Our code is released at: https://github.com/lrjconan/GRAN.

Motivation & Objective

• Develop a scalable, auto-regressive graph generative model that leverages graph structure during generation.
• Reduce dependence on node ordering and long-term sequential bottlenecks of prior RNN-based methods.
• Introduce a block-wise generation mechanism with attention to trade off generation speed and sample quality.
• Capture edge correlation within a block via a mixture of Bernoulli outputs.
• Handle node orderings by marginalizing over a family of canonical orderings with a variational perspective.

Proposed method

• Generate graphs by producing one block of B rows of the lower-triangular adjacency matrix L^π at a time, resulting in O(N) generation steps.
• Use a Graph Neural Network with attentive messages to condition current block generation on the already-generated subgraph.
• Represent each block with multi-hop GNN message passing (R rounds) to compute node representations h_i^R used to predict edges.
• Model the output distribution for the block as a mixture of Bernoulli distributions, with mixture weights and edge probabilities computed from pairwise node representations.
• Marginalize over a family of canonical orderings Q (e.g., DFS, BFS, k-core) to approximate p(G) via a tractable lower bound with a variational interpretation.
• Provide block size B, stride S to control the speed-quality trade-off, with strided sampling allowing overlap between blocks during generation.

Experimental results

Research questions

• RQ1 Can an auto-regressive graph generator leverage graph structure in the conditioning to reduce long-range dependencies and improve scalability?
• RQ2 Does generating graphs in blocks with attention-based GNN conditioning and a mixture output improve both sample quality and generation speed compared to prior GraphRNN-like models?
• RQ3 How does marginalizing over a family of canonical node orderings affect likelihood estimation and generation performance on large graphs?
• RQ4 What is the impact of block size and stride on efficiency-quality trade-offs in GRAN?
• RQ5 Can GRAN scale to large graphs (up to thousands of nodes) while maintaining competitive graph statistics?

Key findings

• Outperforms prior models on standard benchmarks in sample quality and efficiency, achieving state-of-the-art results among scalable auto-regressive graph generators.
• Block-wise generation reduces the sequential bottleneck from O(N^2) steps to O(N) steps, enabling generation of graphs up to thousands of nodes.
• Attentive GNN conditioning improves dependence modeling between generated and existing parts of the graph and reduces ordering sensitivity.
• Mixtures of Bernoulli outputs capture edge correlations within a block, improving modeling of complex edge patterns.
• Marginalizing over a chosen family of canonical orderings yields better log-likelihood approximations and helps alleviate permutation issues without enumerating all permutations.
• Strided sampling allows a tunable trade-off between generation speed and sample quality, with larger strides yielding faster generation at some cost to quality.