[Paper Review] Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation

Jiaxuan You, Bowen Liu+3 more|arXiv (Cornell University)|Jun 6, 2018

Machine Learning in Materials ScienceMaterials Science34 references445 citations

TL;DR

GCPN introduces a graph-convolutional policy network trained with reinforcement learning and adversarial losses to generate molecular graphs that optimize target properties while obeying chemical rules.

ABSTRACT

Generating novel graph structures that optimize given objectives while obeying some given underlying rules is fundamental for chemistry, biology and social science research. This is especially important in the task of molecular graph generation, whose goal is to discover novel molecules with desired properties such as drug-likeness and synthetic accessibility, while obeying physical laws such as chemical valency. However, designing models to find molecules that optimize desired properties while incorporating highly complex and non-differentiable rules remains to be a challenging task. Here we propose Graph Convolutional Policy Network (GCPN), a general graph convolutional network based model for goal-directed graph generation through reinforcement learning. The model is trained to optimize domain-specific rewards and adversarial loss through policy gradient, and acts in an environment that incorporates domain-specific rules. Experimental results show that GCPN can achieve 61% improvement on chemical property optimization over state-of-the-art baselines while resembling known molecules, and achieve 184% improvement on the constrained property optimization task.

Motivation & Objective

• Motivate the need to navigate large, discrete chemical spaces to design molecules with desired properties.
• Propose a graph-based generation framework that can incorporate hard chemical constraints via environment dynamics.
• Leverage reinforcement learning and adversarial training to directly optimize domain-specific properties while staying realistic.

Proposed method

• Represent molecules as graphs and generate them iteratively through actions adding nodes/bonds.
• Formulate molecule generation as a Markov Decision Process with a chemistry-aware environment.
• Use a Graph Convolutional Network to compute node embeddings over an extended graph and predict actions.
• Apply PPO policy gradient to optimize a reward combining domain-specific properties and adversarial loss from a discriminator.
• Incorporate expert pretraining to bootstrap learning and improve stability.

Experimental results

Research questions

• RQ1 Can a graph-based RL agent learn to generate molecules that optimize domain-specific properties under chemical validity constraints?
• RQ2 Does adversarial training with a molecule discriminator improve the realism and validity of generated molecules while optimizing properties?
• RQ3 How does the graph-based approach compare to state-of-the-art baselines on property optimization, targeting, and constrained optimization tasks?

Key findings

• GCPN achieves 61% improvement in chemical property optimization over the best baseline in penalized logP and 184% in constrained property optimization on average.
• GCPN yields near-perfect validity (100%) and strong resemblance to realistic molecules across tasks.
• In property targeting, GCPN significantly outperforms baselines in success rates and diversity.
• GCPN outperforms JT-VAE and ORGAN across reported metrics on the ZINC dataset for multiple objectives.
• The graph-based representation enables step-wise valency checks that ensure 100% validity in reported results.