[Paper Review] LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions

Andre Niyongabo Rubungo, Craig B. Arnold+2 more|arXiv (Cornell University)|Oct 21, 2023

Machine Learning in Materials Science29 citations

TL;DR

LLM-Prop finetunes a T5 encoder to predict crystal properties from text descriptions, achieving state-of-the-art results over GNN baselines on band gap, direct/indirect classification, and unit cell volume using the TextEdge dataset.

ABSTRACT

The prediction of crystal properties plays a crucial role in the crystal design process. Current methods for predicting crystal properties focus on modeling crystal structures using graph neural networks (GNNs). Although GNNs are powerful, accurately modeling the complex interactions between atoms and molecules within a crystal remains a challenge. Surprisingly, predicting crystal properties from crystal text descriptions is understudied, despite the rich information and expressiveness that text data offer. One of the main reasons is the lack of publicly available data for this task. In this paper, we develop and make public a benchmark dataset (called TextEdge) that contains text descriptions of crystal structures with their properties. We then propose LLM-Prop, a method that leverages the general-purpose learning capabilities of large language models (LLMs) to predict the physical and electronic properties of crystals from their text descriptions. LLM-Prop outperforms the current state-of-the-art GNN-based crystal property predictor by about 4% in predicting band gap, 3% in classifying whether the band gap is direct or indirect, and 66% in predicting unit cell volume. LLM-Prop also outperforms a finetuned MatBERT, a domain-specific pre-trained BERT model, despite having 3 times fewer parameters. Our empirical results may highlight the current inability of GNNs to capture information pertaining to space group symmetry and Wyckoff sites for accurate crystal property prediction.

Motivation & Objective

• Motivate rapid crystal property prediction to accelerate crystal design using textual descriptions of crystal structures.
• Provide a public benchmark (TextEdge) with crystal text descriptions and properties to foster NLP for materials science.
• Demonstrate that finetuning a pretrained encoder (T5) on text descriptions can outperform graph neural networks in crystal property prediction.
• Show that text-based representations can capture space-group and Wyckoff information often missed by GNNs.

Proposed method

• Finetune the T5 encoder (discarding the decoder) for regression and classification using a linear output layer.
• Process crystal text descriptions derived from Robocrystallographer, with a 32k vocabulary trained on the benchmark data.
• Preprocess inputs by removing stopwords and replacing bond distances with a [NUM] token and bond angles with an [ANG] token; prepend a [CLS] token for pooling.
• Train using normalized targets (z-score, min-max, or log normalization) and denormalize predictions for evaluation.
• Use the CLS token embedding as input to the final linear predictor, enabling longer input sequences (up to 888 tokens) for richer crystal descriptions.
• Compare against strong baselines including CGCNN, MEGNet, ALIGNN (GNNs), and MatBERT (text baseline) on band gap, direct/indirect, and volume tasks.

Figure 1 : LLM-Prop architecture. On the left most of the figure, we show how we get the description from the crystal structure using Robocrystallographer. The middle part shows the comparison between our approach (text-based) and baselines (structure-based). The yellow colored information is relate

Experimental results

Research questions

• RQ1 Can crystal properties be accurately predicted from text descriptions alone, without explicit graph representations?
• RQ2 How does a pretrained encoder-decoder model (T5) perform when its decoder is removed and the encoder is finetuned for regression/classification in this domain?
• RQ3 What preprocessing and input-length strategies best enable LLMs to capture crystal symmetry, space-group, and Wyckoff-site information from text?
• RQ4 How does LLM-Prop compare to domain-specialized LLMs like MatBERT and to GNN baselines on multiple crystal-property tasks?
• RQ5 What data efficiency can LLM-Prop achieve relative to graph-based models for crystal property prediction?

Key findings

• LLM-Prop consistently outperforms state-of-the-art GNN baselines (e.g., ALIGNN) on band gap and volume predictions; band gap MAE improves by about 4–8% and volume MAE by roughly 66–67%.
• For band-gap direct/indirect classification, LLM-Prop surpasses the best GNN baseline by ~5% AUC on validation and matches/test gains with different setups.
• When using 512 tokens, LLM-Prop still outperforms GNN baselines, and with 888 tokens it achieves the best performance while using about half the parameters of encoder-decoder setups.
• LLM-Prop often surpasses MatBERT and demonstrates strong transfer learning between tasks (band gap to volume and vice versa).
• Text-based representations can capture space-group and Wyckoff-site information that pose challenges for GNN-based crystal predictors, suggesting a complementary direction for future models.
• Data efficiency: LLM-Prop attains state-of-the-art results with around 90k training samples for band gap and volume, and as few as ~30k for volume prediction, outperforming GNN baselines that use substantially more data.

Figure 2 : Training data size needed by LLM-Prop to achieve state-of-the-art (SOTA) results on predicting different properties. For instance, LLM-Prop achieves SOTA results on prediciting band gap and volume with just about 90k training data ( $log_{10}90k\approx 4.95$ on the figure ) that correspon