[Paper Review] A novel molecule generative model of VAE combined with Transformer for unseen structure generation

Yasuhiro Yoshikai, Tadahaya Mizuno+2 more|arXiv (Cornell University)|Feb 19, 2024

Process Optimization and Integration4 citations

TL;DR

This paper proposes a novel generative model that combines Variational Autoencoders (VAE) with a Transformer architecture to generate diverse, unseen molecular structures. By optimizing structural and parameter compatibility between VAE and Transformer, the model achieves superior performance in generating novel molecules while enabling accurate property prediction from a compact ~32-dimensional latent space.

ABSTRACT

Recently, molecule generation using deep learning has been actively investigated in drug discovery. In this field, Transformer and VAE are widely used as powerful models, but they are rarely used in combination due to structural and performance mismatch of them. This study proposes a model that combines these two models through structural and parameter optimization in handling diverse molecules. The proposed model shows comparable performance to existing models in generating molecules, and showed by far superior performance in generating molecules with unseen structures. Another advantage of this VAE model is that it generates molecules from latent representation, and therefore properties of molecules can be easily predicted or conditioned with it, and indeed, we show that the latent representation of the model successfully predicts molecular properties. Ablation study suggested the advantage of VAE over other generative models like language model in generating novel molecules. It also indicated that the latent representation can be shortened to ~32 dimensional variables without loss of reconstruction, suggesting the possibility of a much smaller molecular descriptor or model than existing ones. This study is expected to provide a virtual chemical library containing a wide variety of compounds for virtual screening and to enable efficient screening.

Motivation & Objective

• To address the challenge of combining VAE and Transformer architectures in molecular generation due to structural and performance mismatches.
• To develop a unified generative model capable of producing diverse, unseen molecular structures.
• To enable efficient molecular property prediction via a low-dimensional latent representation.
• To explore the potential of VAE-based models for generating novel compounds compared to autoregressive language models.

Proposed method

• The model integrates a VAE with a Transformer decoder, optimizing both architecture and parameters for compatibility in molecular generation.
• A latent space of ~32 dimensions is learned, enabling compact molecular representation without reconstruction loss.
• The VAE encodes molecular SMILES into a latent vector, which the Transformer decodes into new molecular structures.
• The model is trained end-to-end on a diverse molecular dataset to maximize reconstruction and novelty.
• Ablation studies compare the VAE-Transformer model against standalone models to evaluate performance and generalization.
• Property prediction is performed directly on the latent representation, demonstrating its utility for conditional generation.

Experimental results

Research questions

• RQ1 Can a VAE-Transformer hybrid model generate novel molecular structures more effectively than existing models?
• RQ2 How does the performance of the VAE-Transformer model compare to autoregressive language models in generating unseen molecules?
• RQ3 Can a low-dimensional latent space (~32D) preserve molecular reconstruction fidelity and enable accurate property prediction?
• RQ4 What is the contribution of the VAE component in enabling conditional generation and property prediction?
• RQ5 To what extent does structural and parameter optimization improve the synergy between VAE and Transformer in molecular generation?

Key findings

• The proposed VAE-Transformer model achieves comparable performance to existing models in standard molecular generation benchmarks.
• The model demonstrates superior performance in generating molecules with unseen structural motifs, indicating strong generalization.
• The latent representation enables accurate molecular property prediction, supporting conditional generation applications.
• The latent space can be compressed to ~32 dimensions without significant reconstruction loss, suggesting a highly efficient molecular descriptor.
• Ablation studies confirm that VAE-based generation outperforms autoregressive language models in novelty and structural diversity.
• The model’s latent space is highly structured and predictive, enabling downstream tasks like property prediction directly from latent vectors.