[Paper Review] Benchmarking Large Language Models for Molecule Prediction Tasks

Zhiqiang Zhong, Kuangyu Zhou+1 more|arXiv (Cornell University)|Mar 8, 2024

Machine Learning in Materials Science6 citations

TL;DR

The paper evaluates how well large language models (LLMs) perform on molecule prediction tasks, compares them to ML models designed for molecular data, and explores prompts and hybrid approaches to leverage LLMs as knowledge augmenters rather than direct predictors.

ABSTRACT

Large Language Models (LLMs) stand at the forefront of a number of Natural Language Processing (NLP) tasks. Despite the widespread adoption of LLMs in NLP, much of their potential in broader fields remains largely unexplored, and significant limitations persist in their design and implementation. Notably, LLMs struggle with structured data, such as graphs, and often falter when tasked with answering domain-specific questions requiring deep expertise, such as those in biology and chemistry. In this paper, we explore a fundamental question: Can LLMs effectively handle molecule prediction tasks? Rather than pursuing top-tier performance, our goal is to assess how LLMs can contribute to diverse molecule tasks. We identify several classification and regression prediction tasks across six standard molecule datasets. Subsequently, we carefully design a set of prompts to query LLMs on these tasks and compare their performance with existing Machine Learning (ML) models, which include text-based models and those specifically designed for analysing the geometric structure of molecules. Our investigation reveals several key insights: Firstly, LLMs generally lag behind ML models in achieving competitive performance on molecule tasks, particularly when compared to models adept at capturing the geometric structure of molecules, highlighting the constrained ability of LLMs to comprehend graph data. Secondly, LLMs show promise in enhancing the performance of ML models when used collaboratively. Lastly, we engage in a discourse regarding the challenges and promising avenues to harness LLMs for molecule prediction tasks. The code and models are available at https://github.com/zhiqiangzhongddu/LLMaMol.

Motivation & Objective

• Assess whether LLMs can effectively handle molecule prediction tasks across multiple benchmark datasets.
• Compare LLM-based approaches with traditional ML models including graph-aware methods.
• Investigate prompt designs (zero-shot and few-shot) and the potential of integrating LLM outputs with ML models.
• Examine whether LLMs benefit from incorporating molecular geometry information via text-encoded descriptions.

Proposed method

• Represent molecules using SMILES strings and text-descriptions encoding atom features and graph structure.
• Design and evaluate prompts (IF, IP, IE, IFD, IPD, IED, FS) to elicit predictions from fixed-parameter LLMs in a black-box setting.
• Run LLM-based pipelines (LLM_Solo and LLM_Duo) alongside LM-based and GNN-based baselines.
• Utilize standard ML models (GCN, GIN, DeBERTa-based LM) for comparison on six benchmark datasets.
• Analyze predictor performance and response consistency to assess the utility and limitations of LLMs for molecular tasks.

Experimental results

Research questions

• RQ1 Can LLMs achieve competitive performance on molecule graph property prediction tasks across six benchmark datasets?
• RQ2 Do LLMs benefit from prompt engineering and few-shot examples in predicting molecular properties?
• RQ3 Does encoding graph structure into text descriptions help or hinder LLM performance on molecular tasks?
• RQ4 Can LLMs augment traditional ML models to improve prediction accuracy in molecule tasks?
• RQ5 What are the limitations and promising directions for using LLMs in molecule prediction tasks?

Key findings

• LLMs generally lag behind ML models, especially those that capture molecular geometry, in molecule prediction tasks.
• GPT-family models (GPT-3.5, GPT-4) outperform other LLMs in this domain, but GPT-4 is substantially more costly and slower than GPT-3.5.
• Adding text descriptions of molecular geometry to prompts does not reliably improve performance and can reduce response consistency.
• Models that incorporate geometric structure outperform text-only LLMs, highlighting the importance of geometry in accurate predictions.
• LLMs can meaningfully augment ML models to improve performance in several scenarios, suggesting LLMs are better used as knowledge augmenters rather than sole predictors.
• A practical takeaway is to consider using LLMs as domain-knowledge enhancers for ML pipelines rather than as direct molecule property predictors.