[Paper Review] GP-MoLFormer: A Foundation Model For Molecular Generation

Jerret Ross, Brian Belgodere+5 more|arXiv (Cornell University)|Apr 4, 2024

Nanomaterials for catalytic reactions9 citations

TL;DR

GP-MoLFormer is an autoregressive SMILES generator with 46.8M parameters trained on 0.65–1.1B canonicalized SMILES, analyzing memorization and data bias, and applying pair-tuning for scaffold decoration and property-driven optimization across three molecular-generation tasks.

ABSTRACT

Transformer-based models trained on large and general purpose datasets consisting of molecular strings have recently emerged as a powerful tool for successfully modeling various structure-property relations. Inspired by this success, we extend the paradigm of training chemical language transformers on large-scale chemical datasets to generative tasks in this work. Specifically, we propose GP-MoLFormer, an autoregressive molecular string generator that is trained on more than 1.1B (billion) chemical SMILES. GP-MoLFormer uses a 46.8M parameter transformer decoder model with linear attention and rotary positional encodings as the base architecture. GP-MoLFormer's utility is evaluated and compared with that of existing baselines on three different tasks: de novo generation, scaffold-constrained molecular decoration, and unconstrained property-guided optimization. While the first two are handled with no additional training, we propose a parameter-efficient fine-tuning method for the last task, which uses property-ordered molecular pairs as input. We call this new approach pair-tuning. Our results show GP-MoLFormer performs better or comparable with baselines across all three tasks, demonstrating its general utility for a variety of molecular generation tasks. We further report strong memorization of training data in GP-MoLFormer generations, which has so far remained unexplored for chemical language models. Our analyses reveal that training data memorization and novelty in generations are impacted by the quality and scale of the training data; duplication bias in training data can enhance memorization at the cost of lowering novelty. We further establish a scaling law relating inference compute and novelty in generations.

Motivation & Objective

• Understand how scale and training data bias affect memorization and generation in a large chemical language model.
• Demonstrate de novo molecular generation quality and diversity at scale.
• Evaluate scaffold-constrained decoration and unconstrained property-guided optimization using a parameter-efficient tuning method.
• Provide practical insights on how data deduplication influences novelty and memorization in chemical language models.

Proposed method

• Decoder-only transformer with 12 layers, 12 heads, hidden size 768, and linear attention using generalized random Fourier features.
• Rotary positional embeddings are used to model SMILES token dependencies.
• Autoregressive causal language modeling objective predicting the next token given prior context.
• Pre-training on 0.65–1.1B canonicalized SMILES from public databases.
• Evaluation of memorization vs. novelty across varying training data quality and generation pool sizes.
• Pair-tuning: a prompt-tuning method that learns enhancement tokens to condition generation toward property-optimizing molecules without full model fine-tuning.

Experimental results

Research questions

• RQ1 How does training data size and deduplication affect memorization and novelty in a large generative chemical language model?
• RQ2 Can GP-MoLFormer generate novel, valid, and diverse molecules at billion-scale generational pools?
• RQ3 Is GP-MoLFormer competitive with baselines on de novo generation, scaffold-constrained decoration, and unconstrained property-guided optimization?
• RQ4 Can pair-tuning enable efficient property optimization without full model fine-tuning?

Key findings

• GP-MoLFormer can generate novel, valid, and unique SMILES even when generating up to 10 billion molecules; 99% remain valid across pools.
• Novelty is around 32% for raw training data and increases modestly (to about 7–8% higher) when training data are de-duplicated (Clean).
• At 10M generations, novelty with Raw is 0.322, while Clean yields 0.322 with slight increases in novelty; memorization is evident with high exact-match rates to training data, up to 60%.
• Training data deduplication reduces memorization bias and increases novelty by reducing overrepresentation of certain molecules in the data manifold.
• GP-MoLFormer matches or surpasses baselines on de novo generation, scaffold-constrained decoration, and unconstrained property optimization.
• Pair-tuning enables competitive or superior results for penalized logP, QED, and DRD2 activity optimization without full fine-tuning; demonstrations include tabled comparisons against multiple baselines.