[Paper Review] Toward Accurate Interpretable Predictions of Materials Properties within Transformer Language Models

Vadim Korolev, Pavel Protsenko|arXiv (Cornell University)|Mar 21, 2023

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

This paper proposes a language-model-based framework for accurate and interpretable materials property prediction by representing materials as natural language sequences (e.g., chemical composition, symmetry, geometry) and fine-tuning a pretrained BERT-like model (MatBERT) on these texts. The approach achieves state-of-the-art performance on five materials properties, outperforming crystal graph networks on four, while providing faithful, human-readable explanations via local interpretability techniques such as SHAP, LIME, and integrated gradients.

ABSTRACT

Property prediction accuracy has long been a key parameter of machine learning in materials informatics. Accordingly, advanced models showing state-of-the-art performance turn into highly parameterized black boxes missing interpretability. Here, we present an elegant way to make their reasoning transparent. Human-readable text-based descriptions automatically generated within a suite of open-source tools are proposed as materials representation. Transformer language models pretrained on 2 million peer-reviewed articles take as input well-known terms, e.g., chemical composition, crystal symmetry, and site geometry. Our approach outperforms crystal graph networks by classifying four out of five analyzed properties if one considers all available reference data. Moreover, fine-tuned text-based models show high accuracy in the ultra-small data limit. Explanations of their internal machinery are produced using local interpretability techniques and are faithful and consistent with domain expert rationales. This language-centric framework makes accurate property predictions accessible to people without artificial-intelligence expertise.

Motivation & Objective

• To address the trade-off between model accuracy and interpretability in materials property prediction by replacing complex, black-box representations with human-readable text.
• To enable accurate property prediction using minimal labeled data by leveraging pre-trained language models fine-tuned on materials-specific text.
• To provide faithful, local interpretability of model decisions using multiple XAI techniques (SHAP, LIME, IG, SM) applied at the token level.
• To validate that model explanations align with domain expert rationales through quantitative faithfulness and plausibility metrics.
• To make advanced AI-driven materials prediction accessible to non-AI experts through a text-centric, transparent framework.

Proposed method

• Materials are represented as natural language sequences composed of standard terms (e.g., 'NaCl', 'perovskite', 'cubic symmetry', 'octahedral coordination').
• A fine-tuned MatBERT model—pretrained on 2 million materials science articles—is used as the core architecture for property prediction tasks.
• Three variants are trained: random BERT with original tokenizer, random BERT with MatBERT tokenizer, and the pre-trained MatBERT with MatBERT tokenizer.
• Local interpretability is achieved using four techniques: saliency maps (SM), integrated gradients (IG), LIME, and Shapley values (Owen values) to assign token-level importance scores.
• Faithfulness is evaluated using comprehensiveness and sufficiency via area over the perturbation curve (AOPC), computed over 10%–100% subsets of top-ranked tokens.
• Plausibility is assessed by comparing model-generated rationales with expert-annotated rationales using F1 score and AUPRC, with k=10 as the average rationale length.

Experimental results

Research questions

• RQ1 Can a text-based representation of materials, derived from natural language descriptions, achieve state-of-the-art accuracy in property prediction compared to graph neural networks?
• RQ2 How well do local interpretability methods (SHAP, LIME, IG, SM) produce explanations that align with domain expert rationales in materials science?
• RQ3 To what extent can fine-tuned language models generalize to low-data regimes in materials property prediction?
• RQ4 How do faithfulness metrics (comprehensiveness and sufficiency) vary across different explanation techniques in the context of materials property prediction?
• RQ5 Can a language-centric framework make accurate, interpretable AI predictions accessible to non-experts in machine learning?

Key findings

• The fine-tuned MatBERT model outperforms crystal graph networks on four out of five tested materials properties when using the full reference dataset.
• The model achieves high accuracy even in the ultra-small data regime, demonstrating strong generalization capability with minimal labeled examples.
• Explanations generated by SHAP, LIME, IG, and SM are faithful and consistent with domain expert rationales, as confirmed by high F1 scores (mean F1 ≈ 0.85) and AUPRC values.
• Comprehensiveness and sufficiency metrics (AOPC) show strong faithfulness, with AOPC values exceeding 0.85 for most explanation methods across all tasks.
• The approach enables transparent, human-readable model reasoning without requiring AI expertise, making advanced property prediction accessible to materials scientists.
• The trained MatBERT model for band gap classification is publicly available at Hugging Face, and the expert-annotated dataset is shared via Zenodo.