[Paper Review] MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction

Yuyan Liu, Sirui Ding+3 more|arXiv (Cornell University)|Jun 18, 2024

Machine Learning in Materials Science10 citations

TL;DR

MolecularGPT is an instruction-tuned open LLM designed for few-shot molecular property prediction, using structure-aware few-shot instructions and a hybrid zero-shot/few-shot prompt set to generalize across unseen MPP tasks. It achieves competitive zero-shot and superior few-shot performance on multiple benchmarks, outperforming several baselines.

ABSTRACT

Molecular property prediction (MPP) is a fundamental and crucial task in drug discovery. However, prior methods are limited by the requirement for a large number of labeled molecules and their restricted ability to generalize for unseen and new tasks, both of which are essential for real-world applications. To address these challenges, we present MolecularGPT for few-shot MPP. From a perspective on instruction tuning, we fine-tune large language models (LLMs) based on curated molecular instructions spanning over 1000 property prediction tasks. This enables building a versatile and specialized LLM that can be adapted to novel MPP tasks without any fine-tuning through zero- and few-shot in-context learning (ICL). MolecularGPT exhibits competitive in-context reasoning capabilities across 10 downstream evaluation datasets, setting new benchmarks for few-shot molecular prediction tasks. More importantly, with just two-shot examples, MolecularGPT can outperform standard supervised graph neural network methods on 4 out of 7 datasets. It also excels state-of-the-art LLM baselines by up to 15.7% increase on classification accuracy and decrease of 17.9 on regression metrics (e.g., RMSE) under zero-shot. This study demonstrates the potential of LLMs as effective few-shot molecular property predictors. The code is available at https://github.com/NYUSHCS/MolecularGPT.

Motivation & Objective

• Bridge the gap between traditional supervised MPP and open LLMs by fine-tuning an LLM with molecular instructions to generalize to unseen MPP tasks.
• Leverage SMILES-based instruction tuning and structure-aware few-shot demonstrations to incorporate molecular graph information.
• Explore a hybrid instruction set to balance zero-shot and few-shot in-context learning capabilities in MPP.
• Evaluate on diverse datasets to establish few-shot benchmarks and analyze instruction design impacts.

Proposed method

• Use SMILES to convert molecular graphs into a unified string representation for instruction construction.
• Introduce structure-aware few-shot instructions by retrieving top-K similar molecules via MACCS/ Tanimoto similarity and using them as demonstrations in prompts.
• Develop a hybrid instruction tuning regime combining zero-shot and few-shot templates to preserve zero-shot reasoning while enabling strong few-shot ICL.
• Fine-tune a pre-trained LLM (LLaMA2-7b-chat) with QLoRA on the constructed instruction set across over 1000 MPP tasks.
• Evaluate using ROC-AUC for classification and RMSE for regression across 10 downstream datasets, comparing to GNN baselines and larger LLM baselines.

Experimental results

Research questions

• RQ1 RQ1: Can MolecularGPT effectively and robustly handle new property prediction tasks through zero- and few-shot ICL?
• RQ2 RQ2: What is the optimal design for in-context instruction set to improve MolecularGPT’s generalization and ICL abilities during tuning?
• RQ3 RQ3: How do the number, order, and diversity of in-context examples affect MolecularGPT’s performance?

Key findings

• MolecularGPT shows competitive zero-shot performance compared with other language models and improves upon GIMLET and LLaMA-based baselines in several datasets.
• With two-shot demonstrations, MolecularGPT outperforms standard supervised GNN methods on 4 of 7 datasets and matches or exceeds finetuned Graphormer-p on BBBP under two-shot conditions.
• MolecularGPT achieves up to 16.6% average improvement in ROC-AUC across classification tasks and substantial RMSE reductions in regression under zero-shot relative to strong LLM baselines.
• A hybrid instruction set that blends zero-shot and few-shot prompts consistently improves both zero-shot and few-shot performance across tasks.
• Larger instruction sets and mixed-shot tuning enhance performance, with retrieved demonstrations most effective when ranked by descending similarity to the query.