[Paper Review] Scalable Deep Generative Modeling for Sparse Graphs

Hanjun Dai, Azade Nazi+3 more|arXiv (Cornell University)|Jun 28, 2020

Advanced Graph Neural Networks17 citations

TL;DR

This paper proposes BiGG, a scalable autoregressive generative model for sparse graphs that reduces time complexity from Ω(n²) to O((n+m)log n) by exploiting graph sparsity. It achieves state-of-the-art generation quality on large graphs (up to 100k nodes) with efficient parallel training requiring only O(log n) synchronization steps.

ABSTRACT

Learning graph generative models is a challenging task for deep learning and has wide applicability to a range of domains like chemistry, biology and social science. However current deep neural methods suffer from limited scalability: for a graph with $n$ nodes and $m$ edges, existing deep neural methods require $Ω(n^2)$ complexity by building up the adjacency matrix. On the other hand, many real world graphs are actually sparse in the sense that $m\ll n^2$. Based on this, we develop a novel autoregressive model, named BiGG, that utilizes this sparsity to avoid generating the full adjacency matrix, and importantly reduces the graph generation time complexity to $O((n + m)\log n)$. Furthermore, during training this autoregressive model can be parallelized with $O(\log n)$ synchronization stages, which makes it much more efficient than other autoregressive models that require $Ω(n)$. Experiments on several benchmarks show that the proposed approach not only scales to orders of magnitude larger graphs than previously possible with deep autoregressive graph generative models, but also yields better graph generation quality.

Motivation & Objective

• To address the scalability bottleneck in deep generative models for graphs, which typically require Ω(n²) complexity due to full adjacency matrix construction.
• To enable training and generation of large sparse graphs (e.g., 100k nodes) that current autoregressive models cannot handle due to quadratic time and memory costs.
• To reduce training time and memory usage by leveraging sparsity and designing a parallelizable autoregressive structure with sublinear synchronization.
• To maintain or improve generation quality on benchmark datasets despite significant complexity reduction.

Proposed method

• BiGG uses a binary tree data structure to generate edges in O(log n) time per edge, inspired by R-MAT, avoiding explicit full adjacency matrix computation.
• It employs a tree-structured autoregressive model to generate edges per node, with node ordering (e.g., BFS/DFS) minimizing tree width and improving efficiency.
• The model defines an autoregressive sequence over nodes, where each node’s edge set is generated conditionally on previously processed nodes.
• Training is parallelized via a hierarchical context embedding scheme that requires only O(log n) sequential steps, enabling efficient synchronization.
• Memory usage is reduced to O(√(m log n)) during training and O(log n) during inference, enabling single-GPU training on graphs with millions of nodes.
• The model is trained using a likelihood objective with differentiable sampling, allowing end-to-end optimization despite discrete edge generation.

Experimental results

Research questions

• RQ1 Can we design a deep generative model for graphs that scales efficiently to large sparse graphs without incurring Ω(n²) complexity?
• RQ2 How can we maintain high-quality graph generation while reducing time and memory costs in autoregressive graph models?
• RQ3 Can we parallelize the training of autoregressive graph models to reduce synchronization overhead compared to existing methods?
• RQ4 Does exploiting graph sparsity significantly improve scalability without sacrificing generation quality?

Key findings

• BiGG achieves O((n+m)log n) time complexity for graph generation, representing a substantial improvement over the Ω(n²) complexity of prior deep autoregressive models.
• The model successfully generates graphs with up to 100,000 nodes on a single GPU, a scale orders of magnitude larger than previously feasible with deep autoregressive models.
• On synthetic and real-world datasets (proteins, 3D meshes, SAT instances), BiGG matches or exceeds the generation quality of SOTA models like GRAN, with MMD scores as low as 2.54e-2 on 100k-node grid graphs.
• Training synchronization is reduced to O(log n) steps, enabling significantly faster training than GraphRNN (Ω(n²)) and GRAN (O(n)), while maintaining high-quality samples.
• Memory cost is reduced to O(√(m log n)) during training, allowing single-GPU training on large graphs where GRAN fails due to RAM limitations.
• BiGG maintains strong performance across varying edge densities, with spectral MMD scores close to the ground truth (Erdős–Rényi model) even at 1% edge density.