[Paper Review] Constrained Generation of Semantically Valid Graphs via Regularizing Variational Autoencoders

Tengfei Ma, Jie Chen+1 more|arXiv (Cornell University)|Sep 7, 2018

Bioinformatics and Genomic NetworksBiochemistry, Genetics and Molecular Biology82 citations

TL;DR

The paper introduces a regularized VAE framework that imposes graph validity constraints via marginalization of constraint penalties, enabling higher rates of semantically valid graph generation for molecules and node-compatible graphs.

ABSTRACT

Deep generative models have achieved remarkable success in various data domains, including images, time series, and natural languages. There remain, however, substantial challenges for combinatorial structures, including graphs. One of the key challenges lies in the difficulty of ensuring semantic validity in context. For examples, in molecular graphs, the number of bonding-electron pairs must not exceed the valence of an atom; whereas in protein interaction networks, two proteins may be connected only when they belong to the same or correlated gene ontology terms. These constraints are not easy to be incorporated into a generative model. In this work, we propose a regularization framework for variational autoencoders as a step toward semantic validity. We focus on the matrix representation of graphs and formulate penalty terms that regularize the output distribution of the decoder to encourage the satisfaction of validity constraints. Experimental results confirm a much higher likelihood of sampling valid graphs in our approach, compared with others reported in the literature.

Motivation & Objective

• Motivate and address the challenge of generating semantically valid graphs with deep generative models.
• Propose a regularization framework for VAEs that enforces graph validity constraints through marginalization over latent variables.
• Demonstrate improved validity of generated graphs on molecular datasets and synthetic node-compatible graphs.
• Show how the regularization affects latent space and reconstruction performance.

Proposed method

• Represent graphs with a node-label matrix F and edge-label tensor E in a matrix form.
• Model p_theta(G|z) with a VAE and regularize the decoder output by marginalizing constraint penalties over z (g_i(theta,z)≤0 for all z).
• Define tilde-hatted constraint terms (hats) by integrating squared constraints over the latent prior p_theta(z) to form the regularization.
• Use a ramp function g_i(theta,z)_+ to ensure only violations (g_i>0) are penalized, and approximate the integral with Monte Carlo sampling.
• Train with -ELBO plus mu times the square-root of the marginalized constraint penalties (Eq. 7 and Eq. 8).
• Form concrete constraints for ghosts/valence (Eq. 9), connectivity (Eq. 10), and node compatibility (Eq. 11).

Experimental results

Research questions

• RQ1 Can a regularized VAE framework enforce semantic validity of graph generations across different domains (molecules and node-compatible graphs)?
• RQ2 How do constraint marginals and Ramp-based penalties affect validity, novelty, and reconstruction metrics compared to standard VAEs?
• RQ3 What is the impact of regularization on the latent space structure and denoising/reconstruction capability?
• RQ4 Do the proposed constraints generalize to non-molecular graphs via ghost nodes, valence, connectivity, and compatibility constraints?

Key findings

• Regularization substantially increases the percentage of valid graphs generated from the prior across QM9, ZINC, and node-compatible datasets.
• On QM9, validity rises from 83.2% to 96.6% with regularization (ELBO drops slightly from -17.3 to -18.5).
• On ZINC, validity rises from 29.6% to 34.9% with regularization (ELBO changes from -46.5 to -47.0).
• On node-compatible graphs, validity rises from 40.2% to 98.4% with regularization (ELBO changes from -42.5 to -51.2).
• Regularized VAEs outperform baselines in validity (QM9: 96.6% vs GVAE 60.2% and CVAE 10.3%; ZINC: 34.9% vs GVAE 7.2% and CVAE 0.7%).
• Regularization enables meaningful denoising/reconstruction and smooth latent-space transitions (latent space visualizations and interpolations shown).
• Regularization significantly improves the probability of reconstructing valid graphs under perturbations (standard VAE: 11.2% vs regularized: 93.8%).