[Paper Review] LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

Yuan Chiang, Hsieh, Elvis+2 more|arXiv (Cornell University)|Jan 30, 2024

Machine Learning in Materials Science21 citations

TL;DR

LLaMP is a multimodal retrieval-augmented generation framework that uses hierarchical ReAct agents to ground LLMs in high-fidelity materials data from Materials Project, reducing hallucinations without fine-tuning.

ABSTRACT

Reducing hallucination of Large Language Models (LLMs) is imperative for use in the sciences, where reliability and reproducibility are crucial. However, LLMs inherently lack long-term memory, making it a nontrivial, ad hoc, and inevitably biased task to fine-tune them on domain-specific literature and data. Here we introduce LLaMP, a multimodal retrieval-augmented generation (RAG) framework of hierarchical reasoning-and-acting (ReAct) agents that can dynamically and recursively interact with computational and experimental data on Materials Project (MP) and run atomistic simulations via high-throughput workflow interface. Without fine-tuning, LLaMP demonstrates strong tool usage ability to comprehend and integrate various modalities of materials science concepts, fetch relevant data stores on the fly, process higher-order data (such as crystal structure and elastic tensor), and streamline complex tasks in computational materials and chemistry. We propose a simple metric combining uncertainty and confidence estimates to evaluate the self-consistency of responses by LLaMP and vanilla LLMs. Our benchmark shows that LLaMP effectively mitigates the intrinsic bias in LLMs, counteracting the errors on bulk moduli, electronic bandgaps, and formation energies that seem to derive from mixed data sources. We also demonstrate LLaMP's capability to edit crystal structures and run annealing molecular dynamics simulations using pre-trained machine-learning force fields. The framework offers an intuitive and nearly hallucination-free approach to exploring and scaling materials informatics, and establishes a pathway for knowledge distillation and fine-tuning other language models. Code and live demo are available at https://github.com/chiang-yuan/llamp

Motivation & Objective

• Motivate the need for reliable, memory-enabled LLMs in science for reproducibility and up-to-date data access.
• Propose a retrieval-augmented, memory-grounded framework (LLaMP) using multimodal data sources for materials informatics.
• Demonstrate hierarchy-based ReAct agent orchestration to fetch, process, and synthesize material data without fine-tuning.
• Evaluate LLaMP against intrinsic GPT-3.5 knowledge on materials properties and crystal structures to quantify hallucination reduction.

Proposed method

• Implement a multimodal retrieval-augmented generation (RAG) framework that connects LLMs with Materials Project, arXiv, and Wikipedia via API calls.
• Use hierarchical, multi-agent ReAct planning where a top-level agent coordinates specialized bottom-level agents with toolkits and data stores.
• Incorporate API schemas and memory buffers to ground reasoning and actions in high-fidelity data.
• Demonstrate higher-order data handling (tensors, crystal structures) and multimodal inferences through planned task decomposition.
• Compare LLaMP outputs to GPT-3.5 intrinsic knowledge to quantify reductions in MAPE for formation energies and bandgaps.

Experimental results

Research questions

• RQ1 Can LLaMP reduce hallucinations in materials knowledge tasks compared to intrinsic GPT-3.5 knowledge?
• RQ2 How effectively can hierarchical ReAct agents retrieve and integrate multimodal materials data (e.g., tensors, crystal structures) for complex queries?
• RQ3 To what extent can LLaMP ground synthesis procedures and crystal-generation tasks to high-fidelity MP data?
• RQ4 What is the impact of RAG on correcting errors in scalar properties (e.g., formation energy, bandgaps) and tensor properties?

Key findings

• GPT-3.5 exhibits large MAPE on formation energies and partial or incorrect elastic tensors without retrieval support.
• LLaMP with RAG aligns outputs with MP ground truth across several properties, reducing hallucinations and enabling correct tensor and structure retrieval.
• MAPE for formation energies drops from 1103.54% (GPT-3.5 intrinsic) to near MP-ground-truth values with LLaMP.
• LLaMP maintains correct lattice parameters and crystal descriptions when generating or editing crystal structures, outperforming vanilla GPT-3.5.
• LLaMP can extract synthesis procedures grounded in MP data, avoiding hallucinated steps and irrelevant precursors.