[Paper Review] ChemCrow: Augmenting large-language models with chemistry tools

Andres M Bran, Sam Cox+4 more|arXiv (Cornell University)|Apr 11, 2023

Machine Learning in Materials ScienceMaterials Science131 citations

TL;DR

ChemCrow augments LLMs with 18 chemistry tools to autonomously plan and execute syntheses and support discovery tasks, improving chemical reasoning beyond plain LLMs.

ABSTRACT

Over the last decades, excellent computational chemistry tools have been developed. Integrating them into a single platform with enhanced accessibility could help reaching their full potential by overcoming steep learning curves. Recently, large-language models (LLMs) have shown strong performance in tasks across domains, but struggle with chemistry-related problems. Moreover, these models lack access to external knowledge sources, limiting their usefulness in scientific applications. In this study, we introduce ChemCrow, an LLM chemistry agent designed to accomplish tasks across organic synthesis, drug discovery, and materials design. By integrating 18 expert-designed tools, ChemCrow augments the LLM performance in chemistry, and new capabilities emerge. Our agent autonomously planned and executed the syntheses of an insect repellent, three organocatalysts, and guided the discovery of a novel chromophore. Our evaluation, including both LLM and expert assessments, demonstrates ChemCrow's effectiveness in automating a diverse set of chemical tasks. Surprisingly, we find that GPT-4 as an evaluator cannot distinguish between clearly wrong GPT-4 completions and Chemcrow's performance. Our work not only aids expert chemists and lowers barriers for non-experts, but also fosters scientific advancement by bridging the gap between experimental and computational chemistry.

Motivation & Objective

• Motivate bridging LLMs with domain-specific tools to overcome chemistry reasoning limitations.
• Demonstrate autonomous planning and execution of chemical syntheses using an LLM-agent framework.
• Showcase human–AI collaboration for discovery tasks like chromophore design.
• Evaluate ChemCrow against a pure LLM (GPT-4) using expert chemist assessments.
• Illustrate safety and risk-mitigation strategies for LLM-driven chemistry.

Proposed method

• Prompt an LLM (GPT-4) with a described toolset and an explicit Thought–Action input loop (as in ReAct/MRKL) to decide tool usage and inputs.
• Integrate 18 domain-specific chemistry tools (web/literature search, molecule/reaction tools, safety checks) via LangChain.
• Enable autonomous execution on cloud-connected platforms (e.g., IBM RoboRXN) for synthesis and validation.
• Use iterative tool queries and observations to refine actions until tasks are completed.
• Assess performance with expert chemists and an evaluator LLM (EvaluatorGPT) alongside a GPT-4 baseline.
• Highlight safety guidelines and risk-mitigation strategies to prevent unsafe recommendations.

Experimental results

Research questions

• RQ1 Can an LLM-powered chemistry agent autonomously plan and execute multi-step syntheses in a lab setting?
• RQ2 Does integrating domain-specific tools improve chemical factuality, reasoning quality, and task completion compared with a tool-less LLM?
• RQ3 How does ChemCrow perform on discovery tasks (e.g., novel chromophore design) involving human–AI collaboration?
• RQ4 What safety, ethical, and IP considerations emerge with LLM-driven chemistry, and how can they be mitigated?

Key findings

• ChemCrow autonomously planned and executed syntheses of DEET (insect repellent) and three thiourea organocatalysts using a RoboRXN platform.
• A novel chromophore with an absorption maximum near 336 nm was discovered through human–AI collaboration and subsequently synthesized and characterized.
• ChemCrow outperformed a tool-less GPT-4 in chemical factuality, reasoning, and completeness across increasingly complex tasks, as rated by expert chemists.
• GPT-4 alone showed strengths in memorization-friendly tasks (e.g., known molecules like paracetamol) and in fluent prose, but struggled with novel chemical reasoning.
• The reported random forest model for guiding chromophore screening achieved a root mean squared error of 37 nm in absorption prediction.
• The study highlights the need for robust evaluation methods, tool quality, and safety/IP considerations in LLM-powered chemistry engines.