[Paper Review] Are LLMs Ready for Real-World Materials Discovery?

Santiago Miret, N. M. Anoop Krishnan|arXiv (Cornell University)|Feb 7, 2024

Machine Learning in Materials ScienceMaterials Science20 citations

TL;DR

A position paper outlining current failures of LLMs in materials science, proposing MatSci-LLMs grounded in domain knowledge with a multi-modal, data-rich roadmap for real-world materials discovery.

ABSTRACT

Large Language Models (LLMs) create exciting possibilities for powerful language processing tools to accelerate research in materials science. While LLMs have great potential to accelerate materials understanding and discovery, they currently fall short in being practical materials science tools. In this position paper, we show relevant failure cases of LLMs in materials science that reveal current limitations of LLMs related to comprehending and reasoning over complex, interconnected materials science knowledge. Given those shortcomings, we outline a framework for developing Materials Science LLMs (MatSci-LLMs) that are grounded in materials science knowledge and hypothesis generation followed by hypothesis testing. The path to attaining performant MatSci-LLMs rests in large part on building high-quality, multi-modal datasets sourced from scientific literature where various information extraction challenges persist. As such, we describe key materials science information extraction challenges which need to be overcome in order to build large-scale, multi-modal datasets that capture valuable materials science knowledge. Finally, we outline a roadmap for applying future MatSci-LLMs for real-world materials discovery via: 1. Automated Knowledge Base Generation; 2. Automated In-Silico Material Design; and 3. MatSci-LLM Integrated Self-Driving Materials Laboratories.

Motivation & Objective

• Define the need for materials-science-grounded LLMs (MatSci-LLMs) and identify their current limitations.
• Propose core requirements for MatSci-LLMs including grounded domain knowledge and interpretable hypothesis generation.
• Highlight data and dataset challenges for multi-modal materials science information extraction.
• Outline a roadmap for automated knowledge base generation, in-silico design, and self-driving laboratories in materials science.

Proposed method

• Review failure cases of LLMs in materials science to identify gaps in reasoning and grounding.
• Articulate requirements for MatSci-LLMs based on domain knowledge and augmentation of researchers.
• Discuss multi-modal data extraction challenges across text, tables, figures, CIF files, and other formats.
• Describe data collection, annotation, and dataset-building challenges for material science literature.
• Propose a practical roadmap and potential interfaces for MatSci-LLMs in knowledge base generation, in-silico design, and autonomous experimentation.

Figure 1: Overview of MatSci-LLM requirements related to knowledge acquisition and science acceleration. MatSci-LLMs require knowledge contained across multiple documents along multiple data modalities. Pertinent materials science knowledge includes understanding materials structure, properties and

Experimental results

Research questions

• RQ1 What are the key limitations of current LLMs when applied to materials science knowledge and reasoning?
• RQ2 What requirements must MatSci-LLMs meet to effectively assist materials scientists in hypothesis generation and experimentation?
• RQ3 What data modalities and extraction challenges must be addressed to build large-scale MatSci-LLMs?
• RQ4 How can MatSci-LLMs be integrated into end-to-end workflows including automated knowledge bases, in-silico design, and self-driving laboratories?
• RQ5 What is a feasible roadmap for deploying MatSci-LLMs in real-world materials discovery?

Key findings

• LLMs struggle with domain-specific reasoning, numerical grounding, and correct interpretation of materials structures and notations.
• Current material science data are highly multi-modal and context-dependent, with critical information in tables, figures, CIF files, and allied literature.
• There are substantial data-access and annotation challenges, including paywalls, old literature, and non-machine-readable formats that hinder training of MatSci-LLMs.
• Grounding LLMs in domain-specific language and notations is essential but nontrivial due to the field’s diversity and lack of standard notations.
• A successful MatSci-LLM requires integrated workflows linking knowledge bases, in-silico design, and autonomous laboratory execution.

Figure 2: Roadmap of a Mat-Sci LLM based materials discovery cycle. The cycle starts with materials query from a researcher that specifies desired properties or an application. The MatSci-LLM then draws from external and internal knowledge bases to generate a materials design hypothesis which is eva