[論文レビュー] A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?

Agustinus Kristiadi, Felix Strieth‐Kalthoff+4 more|arXiv (Cornell University)|Feb 7, 2024

Computational Drug Discovery Methods被引用数 8

ひとこと要約

本論文はLLMsが分子のベイズ最適化を支援するかを評価し、chemistry-specific pretraining or finetuning は principled Bayesian surrogates と共に使用した場合に性能を向上させることを示している。

ABSTRACT

Automation is one of the cornerstones of contemporary material discovery. Bayesian optimization (BO) is an essential part of such workflows, enabling scientists to leverage prior domain knowledge into efficient exploration of a large molecular space. While such prior knowledge can take many forms, there has been significant fanfare around the ancillary scientific knowledge encapsulated in large language models (LLMs). However, existing work thus far has only explored LLMs for heuristic materials searches. Indeed, recent work obtains the uncertainty estimate -- an integral part of BO -- from point-estimated, non-Bayesian LLMs. In this work, we study the question of whether LLMs are actually useful to accelerate principled Bayesian optimization in the molecular space. We take a sober, dispassionate stance in answering this question. This is done by carefully (i) viewing LLMs as fixed feature extractors for standard but principled BO surrogate models and by (ii) leveraging parameter-efficient finetuning methods and Bayesian neural networks to obtain the posterior of the LLM surrogate. Our extensive experiments with real-world chemistry problems show that LLMs can be useful for BO over molecules, but only if they have been pretrained or finetuned with domain-specific data.

研究の動機と目的

• Motivate automated material discovery and the need for efficient exploration in large molecular spaces.
• Investigate whether LLMs provide useful priors for Bayesian optimization in chemistry.
• Compare fixed-feature LLM surrogates against PEFT-based Bayesian surrogates.
• Provide an accessible software library for principled BO on discrete molecular spaces.

提案手法

• Treat LLMs as fixed feature extractors, using their embeddings with standard Bayesian surrogates (GPs or Laplace-approximated NNs).
• Assess chemistry-specific vs general-purpose LLMs for BO performance across multiple molecular tasks.
• Apply parameter-efficient fine-tuning (PEFT) methods (e.g., LoRA) and use Laplace approximation to obtain posterior predictive distributions.
• Formulate Bayesian PEFT by conditioning on pretrained weights and performing Bayesian inference on PEFT parameters.
• Evaluate single- and multi-objective BO with various prompts and prompting strategies across real-world chemistry datasets.

実験結果

リサーチクエスチョン

• RQ1 Do chemistry-specific LLMs provide better features for BO over molecules compared to general-purpose LLMs?
• RQ2 Can PEFT coupled with Bayesian inference yield competitive or superior BO performance relative to fixed-feature surrogates?
• RQ3 How do prompting choices affect BO performance when using LLM-derived features?
• RQ4 Is in-context learning-based BO (ICL) competitive with principled Bayesian surrogates using LLM features?

主な発見

• Domain-specific LLM features (e.g., T5-Chem, MolFormer) generally outperform general-purpose LLMs and traditional fingerprints for BO over molecules.
• Finetuning LLMs with PEFT (e.g., LoRA) plus Laplace inference improves BO performance in most tasks, sometimes substantially.
• Chemistry-focused LLMs with simple prompts (e.g., SMILES) yield strong BO performance without extensive prompt engineering.
• In-context learning-based BO (BO-LIFT) is less cost-effective and sometimes less effective than principled Bayesian surrogates with chemistry-specific LLMs.
• Prompt choice affects performance, with chemistry-specific models often preferring prompts aligned with pretraining (e.g., SMILES).
• PEFT-based Bayesian surrogates offer a principled, cost-efficient alternative to full fine-tuning or ICL for BO over discrete molecular spaces.