[Paper Review] Syntax-Directed Variational Autoencoder for Structured Data

Hanjun Dai, Yingtao Tian+3 more|arXiv (Cornell University)|Feb 24, 2018

Topic Modeling22 references201 citations

TL;DR

The paper introduces SD-VAE, which enforces both syntactic and semantic constraints in structured data generation by using stochastic lazy attributes, improving reconstruction, prior validity, and optimization for programs and molecules.

ABSTRACT

Deep generative models have been enjoying success in modeling continuous data. However it remains challenging to capture the representations for discrete structures with formal grammars and semantics, e.g., computer programs and molecular structures. How to generate both syntactically and semantically correct data still remains largely an open problem. Inspired by the theory of compiler where the syntax and semantics check is done via syntax-directed translation (SDT), we propose a novel syntax-directed variational autoencoder (SD-VAE) by introducing stochastic lazy attributes. This approach converts the offline SDT check into on-the-fly generated guidance for constraining the decoder. Comparing to the state-of-the-art methods, our approach enforces constraints on the output space so that the output will be not only syntactically valid, but also semantically reasonable. We evaluate the proposed model with applications in programming language and molecules, including reconstruction and program/molecule optimization. The results demonstrate the effectiveness in incorporating syntactic and semantic constraints in discrete generative models, which is significantly better than current state-of-the-art approaches.

Motivation & Objective

• Motivate and address the challenge of generating syntactically valid and semantically meaningful structured data.
• Incorporate semantic constraints into the decoder using syntax-directed (attribute) grammars.
• Convert offline semantic checking into online guidance during generation via stochastic lazy attributes.
• Demonstrate improved reconstruction, prior validity, and optimization performance on programs and molecules.

Proposed method

• Introduce a stochastic syntax-directed decoder that samples production rules conditioned on context and stochastic attributes.
• Extend attribute grammars with stochastic lazy attributes to enable on-the-fly synthesis of synthesized attributes.
• Use inherited attributes to propagate semantic constraints during tree generation and lazy linking to enforce semantic consistency.
• Employ a structure-based encoder that maps CFG-derived parse representations to a latent space.
• Train with variational autoencoder objective (ELBO) and a likelihood pθ(x|z) computed via syntax-directed decoding.

Experimental results

Research questions

• RQ1 How can semantic constraints be integrated into neural generative decoders for structured data (beyond CFG)?,
• RQ2 Can online semantic guidance via stochastic lazy attributes improve syntactic and semantic validity without sacrificing efficiency? ,
• RQ3 Do SD-VAE representations yield better reconstruction, prior validity, and optimization performance compared with CVAE and GVAE on programs and molecules?

Key findings

• SD-VAE achieves near-perfect reconstruction for programs and very high prior validity on both programs and SMILES tokens.
• Prior validity under the SD-VAE decoder is effectively perfect in evaluation, significantly outperforming baselines.
• SD-VAE enables more effective Bayesian optimization, finding programs and molecules with better target properties than CVAE and GVAE.
• Latent representations learned by SD-VAE are more discriminative (lower RMSE, higher log-likelihood) than baselines.
• The method preserves diversity of generated molecules while narrowing the output space to semantically valid structures.
• Visualizations illustrate smoother and more coherent latent space interpolations compared with prior methods.