[Paper Review] CACTUS: Chemistry Agent Connecting Tool-Usage to Science

Andrew McNaughton, Gautham Krishna Sankar Ramalaxmi+4 more|arXiv (Cornell University)|May 2, 2024

Various Chemistry Research Topics5 citations

TL;DR

CACTUS is an open-source LLM-based agent that integrates cheminformatics tools to enable domain-specific reasoning and molecular discovery, demonstrating improved accuracy over baseline models across a chemistry question benchmark.

ABSTRACT

Large language models (LLMs) have shown remarkable potential in various domains, but they often lack the ability to access and reason over domain-specific knowledge and tools. In this paper, we introduced CACTUS (Chemistry Agent Connecting Tool-Usage to Science), an LLM-based agent that integrates cheminformatics tools to enable advanced reasoning and problem-solving in chemistry and molecular discovery. We evaluate the performance of CACTUS using a diverse set of open-source LLMs, including Gemma-7b, Falcon-7b, MPT-7b, Llama2-7b, and Mistral-7b, on a benchmark of thousands of chemistry questions. Our results demonstrate that CACTUS significantly outperforms baseline LLMs, with the Gemma-7b and Mistral-7b models achieving the highest accuracy regardless of the prompting strategy used. Moreover, we explore the impact of domain-specific prompting and hardware configurations on model performance, highlighting the importance of prompt engineering and the potential for deploying smaller models on consumer-grade hardware without significant loss in accuracy. By combining the cognitive capabilities of open-source LLMs with domain-specific tools, CACTUS can assist researchers in tasks such as molecular property prediction, similarity searching, and drug-likeness assessment. Furthermore, CACTUS represents a significant milestone in the field of cheminformatics, offering an adaptable tool for researchers engaged in chemistry and molecular discovery. By integrating the strengths of open-source LLMs with domain-specific tools, CACTUS has the potential to accelerate scientific advancement and unlock new frontiers in the exploration of novel, effective, and safe therapeutic candidates, catalysts, and materials. Moreover, CACTUS's ability to integrate with automated experimentation platforms and make data-driven decisions in real time opens up new possibilities for autonomous discovery.

Motivation & Objective

• Motivate integrating domain-specific tools with LLMs to enhance chemistry reasoning and molecular discovery.
• Develop an extensible agent that selects and sequences cheminformatics tools for chemical questions.
• Evaluate performance of open-source 7B models with domain-specific prompting and hardware variations.
• Demonstrate potential for running effective models on consumer-grade hardware to democratize access.
• Outline future directions for integrating physics-based models and improving explainability.

Proposed method

• Use LangChain to implement a MRKL-style agent with tools, LLMChain, and a zero-shot agent in a ReAct framework.
• Wrap RDKit, SciPy, and databases (PubChem, ChEMBL, ZINC) into domain-specific tools for SMILES-based analysis.
• Define promoters: Minimal vs Domain Prompt to assess impact on domain understanding and accuracy.
• Benchmark 1000 cheminformatics questions across 10 tools with qualitative, quantitative, and full sets.
• Evaluate multiple 7B-open-source models (Gemma-7b, Falcon-7b, MPT-7b, Llama2-7b, Mistral-7b, Phi2, OLMo-1b) on GPUs (A100, V100, RTX 2080 Ti).
• Investigate hardware, prompt strategies, and model performance to identify deployment trade-offs.

Experimental results

Research questions

• RQ1 How can an LLM-powered agent optimally select and compose cheminformatics tools to answer chemistry questions?
• RQ2 What is the impact of domain-specific prompting on the accuracy and speed of open-source 7B models in cheminformatics tasks?
• RQ3 Can smaller models on consumer-grade hardware achieve competitive accuracy for molecular property prediction tasks?
• RQ4 What are the practical challenges and resolutions in deploying tool-augmented LLMs for chemistry workflows?
• RQ5 What are the prospects for integrating advanced physics-based and symbolic reasoning components into CACTUS for autonomous discovery?

Key findings

• CACTUS outperforms baseline LLMs on a 1000-question cheminformatics benchmark across 10 tools.
• Gemma-7b and Mistral-7b achieve the highest accuracy regardless of prompting strategy.
• Domain prompts improve qualitative question accuracy more than minimal prompts, with variable effects across models.
• Smaller models like Phi2 and OLMo-1b can be effective on consumer-grade hardware with domain prompts.
• Domain prompting often yields faster inference and higher accuracy for several models, though some exceptions exist.
• Open-source models can be deployed with reasonable performance on consumer hardware when paired with optimized deployment (e.g., vLLM) and tailored prompts.