[Paper Review] Learning Deep Generative Models of Graphs

Yujia Li, Oriol Vinyals+3 more|arXiv (Cornell University)|Mar 8, 2018

Advanced Graph Neural NetworksComputer Science33 references447 citations

TL;DR

This paper introduces a graph-net based generative model that learns distributions over arbitrary graphs by simulating a sequential graph-building process, and demonstrates strong performance on synthetic graphs and molecular graphs, including conditional generation.

ABSTRACT

Graphs are fundamental data structures which concisely capture the relational structure in many important real-world domains, such as knowledge graphs, physical and social interactions, language, and chemistry. Here we introduce a powerful new approach for learning generative models over graphs, which can capture both their structure and attributes. Our approach uses graph neural networks to express probabilistic dependencies among a graph's nodes and edges, and can, in principle, learn distributions over any arbitrary graph. In a series of experiments our results show that once trained, our models can generate good quality samples of both synthetic graphs as well as real molecular graphs, both unconditionally and conditioned on data. Compared to baselines that do not use graph-structured representations, our models often perform far better. We also explore key challenges of learning generative models of graphs, such as how to handle symmetries and ordering of elements during the graph generation process, and offer possible solutions. Our work is the first and most general approach for learning generative models over arbitrary graphs, and opens new directions for moving away from restrictions of vector- and sequence-like knowledge representations, toward more expressive and flexible relational data structures.

Motivation & Objective

• Motivate the need for expressive generative models over graphs beyond traditional random graph or grammar-based approaches.
• Propose a graph neural network–driven generative process that adds nodes and edges sequentially to build graphs.
• Show that graph-structured generation outperforms LSTM baselines on synthetic graph tasks and molecular graph generation.
• Explore conditioning and ordering issues in graph generation and demonstrate conditional generation capabilities on molecular graphs.

Proposed method

• Define a sequential graph-generating process where at each step a new node is added, followed by potential edges to connect the new node to the existing graph.
• Parameterize decision modules using graph nets to compute probabilities for adding nodes, adding edges, and selecting connection targets.
• Use a propagation-based graph representation (multi-round message passing) to compute node/graph embeddings that feed the decision modules.
• Train by maximizing the joint likelihood p(G, π) over graphs and their generation orderings using approximations (importance sampling) for marginal likelihoods.
• Allow conditional generation by injecting conditioning vectors into initialization or output modules to steer graph construction.

Experimental results

Research questions

• RQ1 Can a graph neural network–based generative model learn distributions over arbitrary graphs including those with cycles and complex topology?
• RQ2 How does the proposed model perform on synthetic graph generation tasks (cycles, trees, Barabási–Albert graphs) compared to baselines?
• RQ3 Is the model capable of generating valid, novel molecular graphs and how does it compare to SMILES/LSTM and Grammar-based approaches?
• RQ4 What is the impact of node/edge ordering on learning and generation, and can conditioning improve conditional graph generation (e.g., molecular properties)?

Key findings

• The graph generative model achieves higher asymptotic log-likelihood performance than an LSTM baseline on cycles, trees, and Barabási–Albert graphs.
• The graph model yields higher percentages of valid samples for cycles and trees and better alignment of degree distributions for Barabási–Albert graphs than baselines.
• In molecule generation on ChEMBL data, the graph model with graph-based generation achieves higher valid and novel samples than LSTMs trained on SMILES or graph-generated sequences, and competitive marginal likelihood on small molecules.
• Conditional graph generation shows that the graph model produces more valid and novel samples across conditioning scenarios, particularly in interpolation and extrapolation cases.