[Paper Review] LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction

Andre Niyongabo Rubungo, Kangming Li+2 more|arXiv (Cornell University)|Oct 31, 2024

Machine Learning in Materials Science6 citations

TL;DR

LLM4Mat-Bench is a large-scale benchmark evaluating how well various LLMs can predict properties of crystalline materials using composition, CIF, or text descriptions, highlighting the superiority of task-specific models over general-purpose LLMs for materials property prediction.

ABSTRACT

Large language models (LLMs) are increasingly being used in materials science. However, little attention has been given to benchmarking and standardized evaluation for LLM-based materials property prediction, which hinders progress. We present LLM4Mat-Bench, the largest benchmark to date for evaluating the performance of LLMs in predicting the properties of crystalline materials. LLM4Mat-Bench contains about 1.9M crystal structures in total, collected from 10 publicly available materials data sources, and 45 distinct properties. LLM4Mat-Bench features different input modalities: crystal composition, CIF, and crystal text description, with 4.7M, 615.5M, and 3.1B tokens in total for each modality, respectively. We use LLM4Mat-Bench to fine-tune models with different sizes, including LLM-Prop and MatBERT, and provide zero-shot and few-shot prompts to evaluate the property prediction capabilities of LLM-chat-like models, including Llama, Gemma, and Mistral. The results highlight the challenges of general-purpose LLMs in materials science and the need for task-specific predictive models and task-specific instruction-tuned LLMs in materials property prediction.

Motivation & Objective

• Motivate the need for standardized benchmarks to evaluate LLMs in materials property prediction.
• Create a comprehensive, diverse benchmark (LLM4Mat-Bench) with multiple data sources, modalities, and properties.
• Evaluate a range of models from task-specific predictors to general-purpose LLMs to identify strengths and limitations.

Proposed method

• Assemble ~1.9M crystal structures from 10 data sources into 1,978,985 composition–structure–description pairs after removing duplicates.
• Generate crystal structure descriptions deterministically with Robocrystallographer to create a text-based input modality devoid of data contamination.
• Evaluate three material representations (Composition, CIF, Description) across multiple model families including LLM-Prop, MatBERT, Llama, Gemma, Mistral, and CGCNN as a baseline.
• Fine-tune small, task-specific models (LLM-Prop, MatBERT) and compare against zero-shot and few-shot prompting of larger chat-like LLMs.
• Use fixed train/validation/test splits and standard metrics (MAD:MAE for regression, AUC for classification) to enable reproducible comparisons.

Experimental results

Research questions

• RQ1 Can LLMs be effectively used for materials property prediction across diverse data sources and input modalities?
• RQ2 Do task-specific, smaller LLMs outperform general-purpose conversational LLMs for materials property prediction?
• RQ3 Which input representation (Composition, CIF, Description) yields the best predictive performance for LLM-based models?
• RQ4 How do prompt-based zero-shot and few-shot evaluations of chat-like LLMs compare to fine-tuned predictive models in this domain?

Key findings

• Task-specific, smaller predictive LLMs (LLM-Prop and MatBERT) outperform general-purpose chat-like LLMs across regression and classification tasks.
• Description-based inputs generally yield better performance for LLM-based property predictors than CIF or composition inputs.
• More advanced, larger generative LLMs show limited improvements and often produce invalid outputs or hallucinations for material properties.
• Energetic properties are predicted more accurately than other property types across datasets.
• Fine-tuning on MP data can be effective, but gains vary across datasets and properties; general LLMs require task-specific tuning to excel.