[Paper Review] DrugLLM: Open Large Language Model for Few-shot Molecule Generation

Xianggen Liu, Yan Guo+5 more|arXiv (Cornell University)|May 7, 2024

Computational Drug Discovery Methods4 citations

TL;DR

DrugLLM is a 7-billion-parameter large language model fine-tuned for few-shot small-molecule generation using Group-based Molecular Representation (GMR), which encodes molecules as structured sequences of functional groups and positions. It achieves state-of-the-art performance in generating novel molecules with desired properties from only a few modification examples, outperforming existing deep learning and LLM baselines in both few-shot and zero-shot settings.

ABSTRACT

Large Language Models (LLMs) have made great strides in areas such as language processing and computer vision. Despite the emergence of diverse techniques to improve few-shot learning capacity, current LLMs fall short in handling the languages in biology and chemistry. For example, they are struggling to capture the relationship between molecule structure and pharmacochemical properties. Consequently, the few-shot learning capacity of small-molecule drug modification remains impeded. In this work, we introduced DrugLLM, a LLM tailored for drug design. During the training process, we employed Group-based Molecular Representation (GMR) to represent molecules, arranging them in sequences that reflect modifications aimed at enhancing specific molecular properties. DrugLLM learns how to modify molecules in drug discovery by predicting the next molecule based on past modifications. Extensive computational experiments demonstrate that DrugLLM can generate new molecules with expected properties based on limited examples, presenting a powerful few-shot molecule generation capacity.

Motivation & Objective

• To address the challenge of few-shot molecule generation in drug discovery, where data scarcity limits traditional deep learning models.
• To improve the ability of large language models to understand and generate molecules with specific pharmacological properties.
• To develop a method that captures structure-property relationships in molecules using a novel molecular representation (GMR).
• To enable effective few-shot and zero-shot optimization of molecules by learning from limited modification examples.
• To bridge the gap between natural language modeling and chemical language by adapting LLMs to molecular generation tasks.

Proposed method

• DrugLLM uses a modified LLaMA architecture with 32 layers, 32 attention heads, and 4096 hidden dimensions, fine-tuned on a curated dataset of molecule modification sequences.
• Molecules are represented via Group-based Molecular Representation (GMR), which decomposes SMILES strings into structural fragments and their positional indices, improving interpretability and reducing token ambiguity.
• The training objective is autoregressive next-token prediction over sequences of optimization descriptions and molecule modifications, enabling the model to learn transformation patterns.
• The model is fine-tuned on a dataset of molecule modification cases, where each input sequence contains a description of the desired property change and a few example modifications.
• During inference, DrugLLM generates new molecules by predicting the next molecule in a sequence given a few example modifications (few-shot) or a task description (zero-shot).
• The decoding process reconstructs molecules from GMR-encoded fragments using connection and positional information, ensuring structural correctness and reversibility.

Experimental results

Research questions

• RQ1 Can a large language model be effectively fine-tuned to perform few-shot molecule generation with high accuracy and chemical validity?
• RQ2 How does the Group-based Molecular Representation (GMR) improve the modeling of molecular structure and modification patterns compared to standard SMILES?
• RQ3 To what extent can DrugLLM generalize to novel molecular optimization tasks with only a few examples?
• RQ4 How does DrugLLM compare to existing deep generative models (e.g., JTVAE, VJTNN, MoLeR) and general-purpose LLMs (e.g., GPT-4, ChatGLM) in few-shot molecule generation?
• RQ5 Can DrugLLM perform zero-shot molecule optimization without any fine-tuned examples, relying solely on task descriptions?

Key findings

• DrugLLM achieves state-of-the-art performance in few-shot molecule generation, significantly outperforming baseline models like JTVAE, VJTNN, and MoLeR in generating molecules with target properties.
• The model demonstrates strong generalization in zero-shot settings, generating chemically valid and property-optimized molecules based solely on task descriptions and a query molecule.
• GMR-based representation enables more accurate and interpretable molecular encoding, reducing token ambiguity and improving structural fidelity during generation.
• DrugLLM outperforms general-purpose LLMs like GPT-4 and ChatGLM in molecule generation tasks, even with carefully engineered prompts, due to its domain-specific fine-tuning and representation.
• The model generates molecules with higher property scores (e.g., logP, QED, SA score) compared to baselines, indicating better optimization of desired pharmaceutical properties.
• Extensive ablation studies confirm that both the GMR representation and the few-shot fine-tuning strategy are critical for the model’s performance.