[Paper Review] Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning

Gang Liu, Michael Sun+3 more|arXiv (Cornell University)|Oct 5, 2024

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

Llamole is the first multimodal large language model (MLLM) that enables interleaved text and graph generation for inverse molecular design with retrosynthetic planning. By integrating a base LLM with Graph Diffusion Transformers and GNNs, and using A* search with LLM-based heuristics, Llamole achieves 35% success in retrosynthetic planning—up from 5.5% for prior models—while improving molecular design performance by up to 80.9% across 12 metrics.

ABSTRACT

While large language models (LLMs) have integrated images, adapting them to graphs remains challenging, limiting their applications in materials and drug design. This difficulty stems from the need for coherent autoregressive generation across texts and graphs. To address this, we introduce Llamole, the first multimodal LLM capable of interleaved text and graph generation, enabling molecular inverse design with retrosynthetic planning. Llamole integrates a base LLM with the Graph Diffusion Transformer and Graph Neural Networks for multi-conditional molecular generation and reaction inference within texts, while the LLM, with enhanced molecular understanding, flexibly controls activation among the different graph modules. Additionally, Llamole integrates A* search with LLM-based cost functions for efficient retrosynthetic planning. We create benchmarking datasets and conduct extensive experiments to evaluate Llamole against in-context learning and supervised fine-tuning. Llamole significantly outperforms 14 adapted LLMs across 12 metrics for controllable molecular design and retrosynthetic planning.

Motivation & Objective

• To address the challenge of generating molecules with specific properties and synthesizability using large language models (LLMs), which traditionally struggle with graph-structured molecular data.
• To develop a multimodal LLM capable of interleaved text and molecular graph generation, enabling controllable inverse molecular design.
• To integrate A* search with LLM-based cost functions for efficient retrosynthetic pathway planning.
• To create a benchmark dataset with realistic, human-like molecular design instructions for evaluating multimodal generation.
• To overcome limitations of text-only LLMs and graph-based methods by combining their strengths through a unified autoregressive framework.

Proposed method

• Llamole integrates a base LLM with two pre-trained graph modules: a Graph Diffusion Transformer (Graph DiT) for multi-conditional molecular generation and a GNN for reaction template prediction.
• The model uses trigger-query-prediction mechanism with special tokens to activate graph modules during autoregressive generation, enabling interleaved text and graph generation.
• After generating a molecule or reaction, the base LLM resumes text generation with a graph encoder that encodes the previously generated molecular structure.
• For retrosynthetic planning, Llamole employs A* search with LLM-computed heuristics to navigate the large reaction space efficiently.
• The model frames generation as a multi-class prediction task over word vocabulary, atom/bond types, and reaction templates.
• Fine-tuning is performed on a curated dataset of multimodal molecular design instructions, including human conversation-style prompts and retrosynthetic pathways.

Figure 1: Comparison of Controllability: Results are averaged from the best numbers from Table 1 .

Experimental results

Research questions

• RQ1 Can a multimodal LLM achieve coherent interleaved generation of text and molecular graphs for inverse molecular design?
• RQ2 How does integrating graph models with LLMs improve controllability and quality in molecular generation compared to text-only LLMs?
• RQ3 To what extent can LLMs with graph integration outperform both pure LLMs and graph-based baselines in retrosynthetic planning?
• RQ4 Can A* search guided by LLM-based heuristics efficiently explore large reaction spaces for multi-step retrosynthesis?
• RQ5 How effective is the proposed framework in generating molecules that satisfy complex, multi-property requirements?

Key findings

• Llamole achieves a 35% success rate in retrosynthetic planning, a significant improvement from the 5.5% success rate of the best baseline model.
• Llamole improves performance across 12 metrics for controllable molecular design by up to 80.9% compared to 14 adapted LLMs.
• In text generation for reaction conditions, Llamole achieves the highest ROUGE-L score (0.268), indicating strong structural coherence, despite lower BLEU-4 scores due to the diversity of valid reaction conditions.
• Case studies show Llamole generates chemically valid and synthesizable molecules that meet complex requirements, while ICL and SFT baselines produce hallucinated or invalid structures.
• For polymer design, Llamole successfully generates valid monomer structures with desired properties and a two-step retrosynthetic pathway, whereas baselines fail to generate valid polymers or follow instructions.
• The model demonstrates robustness across diverse molecular types, including small molecules and polymers, with improved instruction adherence and chemical validity.

Figure 2: Three LLM-based methods for molecular design. The question outlines requirements for properties, structures, and synthesis, addressed as follows: (a) In-Context Learning and (b) Supervised Fine-Tuning use text-only data for demonstrations and instruction tuning, respectively. (c) The propo