[Paper Review] Autoregressive Diffusion Model for Graph Generation

Lingkai Kong, Jiaming Cui+4 more|arXiv (Cornell University)|Jul 17, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes GraphArm, the first autoregressive diffusion model for graph generation that operates directly in discrete graph space via a node-absorbing diffusion process. By learning a data-dependent node ordering and using autoregressive denoising, it achieves state-of-the-art generation quality with significantly faster sampling than prior diffusion-based methods, while enabling constraint-aware generation.

ABSTRACT

Diffusion-based graph generative models have recently obtained promising results for graph generation. However, existing diffusion-based graph generative models are mostly one-shot generative models that apply Gaussian diffusion in the dequantized adjacency matrix space. Such a strategy can suffer from difficulty in model training, slow sampling speed, and incapability of incorporating constraints. We propose an \emph{autoregressive diffusion} model for graph generation. Unlike existing methods, we define a node-absorbing diffusion process that operates directly in the discrete graph space. For forward diffusion, we design a \emph{diffusion ordering network}, which learns a data-dependent node absorbing ordering from graph topology. For reverse generation, we design a \emph{denoising network} that uses the reverse node ordering to efficiently reconstruct the graph by predicting the node type of the new node and its edges with previously denoised nodes at a time. Based on the permutation invariance of graph, we show that the two networks can be jointly trained by optimizing a simple lower bound of data likelihood. Our experiments on six diverse generic graph datasets and two molecule datasets show that our model achieves better or comparable generation performance with previous state-of-the-art, and meanwhile enjoys fast generation speed.

Motivation & Objective

• To address the limitations of existing one-shot diffusion-based graph generative models, including slow sampling, poor constraint handling, and reliance on continuous dequantization.
• To develop a diffusion process that operates directly in the discrete graph space, avoiding distributional distortion from dequantization.
• To enable efficient, autoregressive graph generation by learning a data-dependent node absorption ordering that captures structural regularities.
• To jointly train a diffusion ordering network and a denoising network using a variational lower bound on likelihood, leveraging permutation invariance of graphs.
• To demonstrate improved generation speed and performance on diverse graph datasets, including molecules and social networks.

Proposed method

• Proposes a node-absorbing autoregressive diffusion process that sequentially removes one node (and its edges) at a time, operating directly in the discrete graph space.
• Introduces a diffusion ordering network that learns a data-dependent node ordering based on graph topology, improving likelihood approximation and training stability.
• Designs a reverse denoising network that reconstructs the graph autoregressively by predicting the type and connections of each new node using previously generated nodes.
• Uses a variational lower bound of the data likelihood to jointly train the diffusion ordering and denoising networks via reinforcement learning and gradient descent.
• Models edge predictions using a mixture of multinomial distributions to capture dependencies among edges during autoregressive generation.
• Leverages permutation invariance of graphs to simplify training and ensure the reverse process uses the exact reverse of the learned ordering.

Figure 1: The autoregressive graph diffusion process. In the forward pass, the nodes are autoregressively decayed into the absorbing states, dictated by an ordering generated by the diffusion ordering network $q_{\phi}(\sigma|G_{0})$ . In the reverse pass, the generator network $p_{\theta}(G_{t}|G_{

Experimental results

Research questions

• RQ1 Can a diffusion-based graph generative model operate directly in the discrete graph space without dequantization, improving training stability and fidelity?
• RQ2 Does learning a data-dependent node ordering in the diffusion process lead to better generation quality and faster convergence compared to random ordering?
• RQ3 Can autoregressive diffusion in the discrete graph space enable efficient, constraint-aware graph generation that is not feasible with one-shot diffusion models?
• RQ4 How does the joint training of a diffusion ordering network and denoising network via a variational lower bound affect model performance and training efficiency?
• RQ5 To what extent can the learned node ordering generalize across graphs with similar structural properties?

Key findings

• GraphArm achieves state-of-the-art or competitive performance on six generic graph datasets and two molecular datasets, outperforming prior diffusion-based models in generation quality.
• On the Caveman dataset, GraphArm achieves 100% validity with a degree score of 0.176, clustering score of 0.157, and orbit score of 0.144, outperforming OM and GRAN.
• On the Community-small dataset, GraphArm reduces cross-cluster node transitions to 2.6 on average, significantly lower than OA-ARDM’s 5.6, indicating better structural consistency in generation.
• The learned diffusion ordering in GraphArm leads to more coherent generation, first generating one community and then another, whereas OA-ARDM with random ordering fails to capture such structure.
• GraphArm’s sampling speed is substantially faster than prior one-shot diffusion models due to the autoregressive nature, with the number of steps equal to the number of nodes, not a long sequence of noise steps.
• Ablation studies confirm that the learned diffusion ordering is critical: OA-ARDM with random ordering shows significantly degraded performance, especially in structural coherence and constraint satisfaction.

Figure 2: The generation procedure at step $t$ with the denoising network $p_{\theta}(G_{t}|G_{t+1})$ . The denoising network predicts the node type of node $v_{\sigma_{t}}$ and its edges with all previously denoised nodes.