[Paper Review] Can Large Language Models Empower Molecular Property Prediction?

Qian Chen, Huayi Tang+3 more|arXiv (Cornell University)|Jul 14, 2023

Machine Learning in Materials Science13 citations

TL;DR

The paper investigates using large language models (LLMs) for molecular property prediction via zero/few-shot in-context classification and a new CaR representation built from LLM-generated explanations of SMILES, showing CaR often matches or surpasses baselines on benchmark datasets.

ABSTRACT

Molecular property prediction has gained significant attention due to its transformative potential in multiple scientific disciplines. Conventionally, a molecule graph can be represented either as a graph-structured data or a SMILES text. Recently, the rapid development of Large Language Models (LLMs) has revolutionized the field of NLP. Although it is natural to utilize LLMs to assist in understanding molecules represented by SMILES, the exploration of how LLMs will impact molecular property prediction is still in its early stage. In this work, we advance towards this objective through two perspectives: zero/few-shot molecular classification, and using the new explanations generated by LLMs as representations of molecules. To be specific, we first prompt LLMs to do in-context molecular classification and evaluate their performance. After that, we employ LLMs to generate semantically enriched explanations for the original SMILES and then leverage that to fine-tune a small-scale LM model for multiple downstream tasks. The experimental results highlight the superiority of text explanations as molecular representations across multiple benchmark datasets, and confirm the immense potential of LLMs in molecular property prediction tasks. Codes are available at \url{https://github.com/ChnQ/LLM4Mol}.

Motivation & Objective

• Motivate molecular property prediction as a domain for NLP-era models and evaluate LLM capabilities on SMILES-based tasks.
• Investigate zero/few-shot molecular classification using in-context learning with ChatGPT.
• Propose CaR (Captions as New Representation) to generate semantic explanations of SMILES for downstream modeling.
• Fine-tune a small-scale LM on LLM-generated explanations to perform classification and regression tasks.
• Compare CaR against graph-based and SMILES-based baselines across multiple datasets.

Proposed method

• Prompt ChatGPT to perform in-context molecular classification to assess zero/few-shot capabilities.
• Generate semantically enriched explanations for SMILES strings using ChatGPT and use these explanations as molecular representations (CaR).
• Fine-tune a small pre-trained LM (e.g., RoBERTa) on downstream tasks using CaR representations.
• Evaluate performance across nine datasets spanning classification and regression tasks.
• Conduct convergence analyses and ablations by replacing the small LM with other pre-trained models (DeBERTa, adaptive-lm-molecules) and training from scratch.

Experimental results

Research questions

• RQ1 Can ChatGPT perform zero/few-shot molecular classification effectively compared to traditional methods?
• RQ2 Do textual explanations generated by an LLM provide meaningful representations for molecular property prediction?
• RQ3 Does CaR improve performance over traditional GNNs and SMILES-based methods across diverse datasets?
• RQ4 How does CaR behave under different data splits (random vs scaffold) and different small LM backbones?
• RQ5 What is the impact of using pre-trained vs. scratch LMs in the CaR pipeline?

Key findings

• CaR achieves superior results on almost all datasets under random split for both classification and regression.
• CaR shows a 53% improvement over traditional methods on the PTC dataset under random split.
• Under scaffold splitting, CaR attains competitive results on several datasets, with notable gains in some cases and slight deficits in Lipophilicity compared to GNNs.
• Convergence analyses indicate CaR loss decreases rapidly initially and then fluctuates toward convergence, while ROC-AUC shows inverse trends.
• Replacing the small LM with other pre-trained LMs (DeBERTa, adaptive-LM Molecules) yields similar or better performance than training from scratch, validating CaR effectiveness.
• Zero/few-shot ChatGPT performance varies by dataset and prompt, improves with more shots, but prompt design critically affects results.