[Paper Review] Less can be more for predicting properties with large language models

Nawaf Alampara, Santiago Miret+1 more|arXiv (Cornell University)|Jun 25, 2024

Machine Learning in Materials Science13 citations

TL;DR

The paper introduces mattext, a benchmark and framework to study text-based material representations for property prediction, showing that adding geometric information and scaling pretraining often does not improve LLM-based material property predictions. It finds locality as a strong inductive bias and reveals limitations in current text-only approaches to leverage crystal geometry.

ABSTRACT

Predicting properties from coordinate-category data -- sets of vectors paired with categorical information -- is fundamental to computational science. In materials science, this challenge manifests as predicting properties like formation energies or elastic moduli from crystal structures comprising atomic positions (vectors) and element types (categorical information). While large language models (LLMs) have increasingly been applied to such tasks, with researchers encoding structural data as text, optimal strategies for achieving reliable predictions remain elusive. Here, we report fundamental limitations in LLM's ability to learn from coordinate information in coordinate-category data. Through systematic experiments using synthetic datasets with tunable coordinate and category contributions, combined with a comprehensive benchmarking framework (MatText) spanning multiple representations and model scales, we find that LLMs consistently fail to capture coordinate information while excelling at category patterns. This geometric blindness persists regardless of model size (up to 70B parameters), dataset scale (up to 2M structures), or text representation strategy. Our findings suggest immediate practical implications: for materials property prediction tasks dominated by structural effects, specialized geometric architectures consistently outperform LLMs by significant margins, as evidenced by a clear "GNN-LM wall" in performance benchmarks. Based on our analysis, we provide concrete guidelines for architecture selection in scientific machine learning, while highlighting the critical importance of understanding model inductive biases when tackling scientific prediction problems.

Motivation & Objective

• Motivate the need for a standardized, text-based representation framework for materials to enable systematic benchmarking of language models.
• Develop mattext representations and a software suite to convert crystal structures into text with controllable inductive biases.
• Create a comprehensive, scalable dataset and benchmark covering multiple representations to evaluate material property prediction with LLMs.
• Analyze how different representations and data/model scales affect predictive performance, emphasizing locality and geometric information.
• Provide insights into the limitations of current text-based modeling approaches for materials science and propose directions for better representations and benchmarks.

Proposed method

• Define nine text-based representations of solid-state materials, including five novel ones, each embedding different inductive biases.
• Compile pretraining and finetuning data from NOMAD, aligning with MatBench conventions and providing standardized splits (30k, 100k, 300k, 2M; test set 20k).
• Implement mattext as an object-oriented framework with tokenizers, decoders, and robustness utilities to translate crystal structures into text.
• Pretrain and fine-tune encoder-only and decoder-only transformer models (e.g., BERT, Llama-3) on mattext representations for property prediction tasks.
• Evaluate models across representations, data scales, and architectures to assess reliance on compositional vs geometric information.
• Use a physics-inspired hypothetical potential to isolate contributions from composition and geometry and probe model sensitivity to geometric information.

Figure 1: Overview of the \mattext framework. For \mattext , we compile pretraining and fine-tuning data and implement text representations of crystal structures with different inductive biases in a standardized, object-oriented framework. We use the data and text representations to build a property

Experimental results

Research questions

• RQ1 How do different text-based representations of crystal structures affect material property prediction with language models?
• RQ2 Does increasing pretraining data or model size improve downstream material-property prediction when using text representations?
• RQ3 To what extent can LLMs leverage geometric/positional information encoded in text representations for solids?
• RQ4 Is locality a stronger inductive bias than explicit geometry for predicting material properties?
• RQ5 Can a framework like mattext reveal fundamental shortcomings of text-based approaches and guide better representations?

Key findings

• Current LLMs consistently struggle to capture geometric information necessary for materials modeling across representations.
• Scaling pretraining data or model size shows limited or mixed benefits for downstream material-property tasks.
• Representations that emphasize local environments (locality bias) can perform comparably to geometry-rich representations, suggesting models rely more on local information.
• Some representations lacking explicit geometry (e.g., SLICES) can rival more information-rich ones, indicating that added geometric detail is not reliably leveraged by current models.
• Fine-tuning decoder-only LLMs (e.g., Llama-3) often does not make positional information useful, and explicit positional data can even hurt performance in some datasets.
• The mattext framework exposes weaknesses in text-based material modeling and points toward locality-focused, coarse-grained representations as promising directions.

Figure 2: \mattext -BERT scaling cross-validation scores. We pretrained BERT models with increasing pretraining dataset sizes (indicated in color) and fine-tuned them for material property prediction tasks using different material text representations. The error bars indicate the standard deviation