[Paper Review] Foundational Large Language Models for Materials Research

Vaibhav Mishra, Sourabh Singh+8 more|arXiv (Cornell University)|Dec 12, 2024

Machine Learning in Materials Science13 citations

TL;DR

LLaMat is a domain-adapted family of language models for materials science that uses continued pretraining and instruction/task finetuning to outperform general-purpose LLMs on MatSci tasks, with two variants (LLaMat-Chat and LLaMat-CIF) for NLP and crystal structure generation, respectively; an observed adaptation rigidity highlights limits of overtrained models.

ABSTRACT

Materials discovery and development are critical for addressing global challenges. Yet, the exponential growth in materials science literature comprising vast amounts of textual data has created significant bottlenecks in knowledge extraction, synthesis, and scientific reasoning. Large Language Models (LLMs) offer unprecedented opportunities to accelerate materials research through automated analysis and prediction. Still, their effective deployment requires domain-specific adaptation for understanding and solving domain-relevant tasks. Here, we present LLaMat, a family of foundational models for materials science developed through continued pretraining of LLaMA models on an extensive corpus of materials literature and crystallographic data. Through systematic evaluation, we demonstrate that LLaMat excels in materials-specific NLP and structured information extraction while maintaining general linguistic capabilities. The specialized LLaMat-CIF variant demonstrates unprecedented capabilities in crystal structure generation, predicting stable crystals with high coverage across the periodic table. Intriguingly, despite LLaMA-3's superior performance in comparison to LLaMA-2, we observe that LLaMat-2 demonstrates unexpectedly enhanced domain-specific performance across diverse materials science tasks, including structured information extraction from text and tables, more particularly in crystal structure generation, a potential adaptation rigidity in overtrained LLMs. Altogether, the present work demonstrates the effectiveness of domain adaptation towards developing practically deployable LLM copilots for materials research. Beyond materials science, our findings reveal important considerations for domain adaptation of LLMs, such as model selection, training methodology, and domain-specific performance, which may influence the development of specialized scientific AI systems.

Motivation & Objective

• Address bottlenecks in mining vast materials literature by developing domain-adapted foundational LLMs.
• Create LLaMat variants for materials text processing and crystal structure generation.
• Evaluate LLaMat against commercial LLMs across MatSci NLP, SIE, and crystal generation tasks.
• Analyze how pretraining and finetuning strategies affect domain-specific performance and general language capabilities.

Proposed method

• Three-stage development: continued pretraining on a materials-focused corpus (R2CID) with a 3% RedPajama subset to preserve English skills.
• Two instruction-finetuning pathways yield LLaMat-Chat (general and MatSci-specific tasks with downstream QA capabilities) and LLaMat-CIF (crystallographic file-focused tasks).
• Parameter-efficient finetuning (PEFT) enables crystal generation from CIF data using LLaMat-CIF.
• Systematic evaluation across MatSci NLP, MatSIE, and crystal generation benchmarks with comparisons to closed-source LLMs.

Experimental results

Research questions

• RQ1 How can domain-adaptive pretraining and instruction fine-tuning improve MatSci natural language processing and information extraction?
• RQ2 Can domain-adapted LLMs generate valid and stable crystal structures from CIF data, and how do they compare to existing methods?
• RQ3 What are the trade-offs between model size, pretraining data scale, and domain adaptation efficacy in MatSci tasks?
• RQ4 Does overtraining (adaptation rigidity) limit domain adaptability of larger base models like LLaMA-3 compared to LLaMA-2 in materials applications?

Key findings

• LLaMat-Chat variants outperform base LLaMA and closed-source models on MatSci NLP and SIE tasks.
• LLaMat-2-CIF achieves high composition validity (0.995) and stability (49.49% of generated structures stable) in crystal generation, with strong coverage (0.986 recall, 0.996 precision).
• LLaMat-3-CIF generates more complex structures but with lower structural validity and efficiency, indicating adaptation rigidity in highly pretrained models.
• Across tasks, domain-adapted LLaMat models consistently surpass commercial LLMs (GPT, Claude, Gemini) in MatSci-related analyses.
• Adaptation rigidity suggests smaller, well-targeted domain-adapted models (LLaMat-2) can outperform larger successors (LLaMat-3) on many MatSci tasks.