[Paper Review] Bayesian Optimization of Catalysis With In-Context Learning

Mayk Caldas Ramos, Shane S. Michtavy+2 more|arXiv (Cornell University)|Apr 11, 2023

Machine Learning in Materials Science30 citations

TL;DR

The paper demonstrates regression with uncertainty via in-context learning (ICL) using frozen LLMs to enable Bayesian optimization for catalyst design and synthesis-conditioned predictions without training. It compares prompting strategies and model families across solubility and C2-yield tasks, and shows BO feasibility with ICL vs baselines.

ABSTRACT

Large language models (LLMs) can perform accurate classification with zero or few examples through in-context learning. We extend this capability to regression with uncertainty estimation using frozen LLMs (e.g., GPT-3.5, Gemini), enabling Bayesian optimization (BO) in natural language without explicit model training or feature engineering. We apply this to materials discovery by representing experimental catalyst synthesis and testing procedures as natural language prompts. A key challenge in materials discovery is the need to characterize suboptimal candidates, which slows progress. While BO is effective for navigating large design spaces, standard surrogate models like Gaussian processes assume smoothness and continuity, an assumption that fails in highly non-linear domains such as heterogeneous catalysis. Our task-agnostic BO workflow overcomes this by operating directly in language space, producing interpretable and actionable predictions without requiring structural or electronic descriptors. On benchmarks like aqueous solubility and oxidative coupling of methane (OCM), BO-ICL matches or outperforms Gaussian processes. In live experiments on the reverse water-gas shift (RWGS) reaction, BO-ICL identifies near-optimal multi-metallic catalysts within six iterations from a pool of 3,700 candidates. Our method redefines materials representation and accelerates discovery, with broad applications across catalysis, materials science, and AI. Code: https://github.com/ur-whitelab/BO-ICL.

Motivation & Objective

• Demonstrate that in-context learning with frozen LLMs can produce predictions with uncertainty suitable for Bayesian optimization in catalysis and related materials design.
• Show how natural-language synthesis procedures can represent catalysts and conditions to predict properties.
• Evaluate performance across solubility and catalyst-yield datasets and compare against baselines and fine-tuning.

Proposed method

• Use LIFT to convert catalytic synthesis procedures into natural-language prompts for decoder-only LLMs (GPT-3, GPT-3.5, GPT-4).
• Derive regression with uncertainty from token-level probabilities via two prompting strategies: multi (five options) and topk (k completions).
• Quantify uncertainty to implement Bayesian optimization with acquisition functions (EI, UCB) in an ask-tell loop.
• Compare ICL with baselines (LIFT fine-tuning, KRR, GPR, KNN) using ESOL solubility and C2 yield datasets.
• Apply context selection via max marginal relevance (MMR) to extend ICL beyond the model context window.
• Evaluate newer model performance (GPT-4) and assess calibration with uncertainty recalibration steps.

Figure 1: Our approach uses a Language-Interfaced Fine-Tuning (LIFT) framework with a Generative Pre-trained Transformer (GPT) to generate tokens that represent the reaction conditions that include a synthesis procedure. The catalyst synthesis and testing data is converted to an embedding vector and

Experimental results

Research questions

• RQ1 Can in-context learning with frozen LLMs provide predictive uncertainty sufficient for Bayesian optimization in catalysis problems?
• RQ2 How do prompting strategies (multi vs topk) and context-sample selection affect ICL accuracy and BO performance?
• RQ3 How does ICL compare to traditional baselines (KRR, GPR, KNN) and fine-tuning in predicting solubility and catalytic yields?
• RQ4 What is the impact of model size and freshness (GPT-4 vs Curie) on BO outcomes in this domain?
• RQ5 Is inverse design feasible using ICL prompts to steer experimental synthesis procedures toward desired properties?

Key findings

• ICL can yield competitive or superior performance to baselines in solubility prediction, especially after uncertainty recalibration.
• Topk prompts generally offer better data-efficiency than multi prompts, requiring fewer examples to reach comparable MAE and correlation.
• GPT-4 and newer models improve performance over earlier LLMs, though chat models may lack logprobs, affecting uncertainty uso.
• BO with ICL is feasible in low-data regimes and can identify high-performance solubility values, though C2-yield presents higher complexity where GPR with embeddings can outperform ICL in some settings.
• Recalibrating uncertainties improves calibration and can enable ICL to outperform baselines after recalibration on solubility tasks.
• A GPR baseline using text embeddings can be strong, and fine-tuning remains superior on the more complex C2 dataset, while ICL avoids training costs.

Figure 2: Dependence of the six models considered in this work as a function of the number of training points $N$ from where the model could select examples to create the context (for ICL models) or to train (for baseline models). In these experiments, our ICL models have a fixed example selector si