[Paper Review] CTGNN: Crystal Transformer Graph Neural Network for Crystal Material Property Prediction

Zijian Du, Luozhijie Jin+5 more|arXiv (Cornell University)|May 19, 2024

Welding Techniques and Residual Stresses4 citations

TL;DR

This paper proposes CTGNN, a novel Crystal Transformer Graph Neural Network that integrates graph neural networks with dual-Transformer architectures to model both local atomic interactions and long-range inter-atomic relationships in crystal structures. By incorporating angular encoding and leveraging attention mechanisms, CTGNN achieves state-of-the-art performance in predicting formation energy and bandgap, with a 51.85% lower MAE than CGCNN on perovskite materials.

ABSTRACT

The combination of deep learning algorithm and materials science has made significant progress in predicting novel materials and understanding various behaviours of materials. Here, we introduced a new model called as the Crystal Transformer Graph Neural Network (CTGNN), which combines the advantages of Transformer model and graph neural networks to address the complexity of structure-properties relation of material data. Compared to the state-of-the-art models, CTGNN incorporates the graph network structure for capturing local atomic interactions and the dual-Transformer structures to model intra-crystal and inter-atomic relationships comprehensively. The benchmark carried on by the proposed CTGNN indicates that CTGNN significantly outperforms existing models like CGCNN and MEGNET in the prediction of formation energy and bandgap properties. Our work highlights the potential of CTGNN to enhance the performance of properties prediction and accelerates the discovery of new materials, particularly for perovskite materials.

Motivation & Objective

• To address the limitations of existing graph neural networks and Transformers in capturing complex crystal structure-property relationships.
• To improve prediction accuracy for key materials properties such as formation energy and bandgap by integrating local atomic geometry with global structural attention.
• To develop a model that preserves the inductive bias of GNNs while enhancing long-range dependency modeling through dual-Transformer mechanisms.
• To validate the effectiveness of angular encoding and dual-attention mechanisms in improving predictive performance on benchmark crystal datasets.

Proposed method

• CTGNN combines a graph neural network backbone with two distinct Transformer encoders: one for intra-crystal atomic interactions and another for inter-atomic relationships within atomic neighborhoods.
• An angular encoder kernel is introduced to encode bond angles, enriching the geometric representation beyond simple interatomic distances used in prior models.
• The model employs multi-head self-attention mechanisms, where queries, keys, and values are derived from atomic features, enabling dynamic attention weighting across atomic neighborhoods.
• The dual-Transformer design allows separate modeling of local chemical environments and broader structural correlations, enhancing representation learning at multiple scales.
• The model is trained end-to-end using mean absolute error (MAE) and coefficient of determination (R²) as optimization objectives on crystal structure-property prediction tasks.
• Ablation studies are conducted by removing components such as angular encoding and inter-atomic Transformer to evaluate their individual contributions to performance.

Figure 1: Our CTGNN framework. Transformer Layer denotes as Transformer encoder for atom features and neighbor features. After the Transformer Layers, CGCNN convolution layers are used. And finally pooling and predicting layers are used to get the prediction. Topological features include angular and

Experimental results

Research questions

• RQ1 Can a hybrid GNN-Transformer architecture outperform existing state-of-the-art models in predicting crystal material properties like formation energy and bandgap?
• RQ2 How does the inclusion of angular encoding improve the geometric representation of crystal structures compared to distance-only encoding?
• RQ3 What is the relative contribution of the intra-crystal and inter-atomic Transformer components to overall model performance?
• RQ4 Does the dual-Transformer design enable better generalization across diverse crystal systems, especially perovskites?

Key findings

• On the perovskite dataset, CTGNN achieved a mean absolute error (MAE) of 0.013 eV/atom for formation energy prediction, representing a 51.85% improvement over CGCNN (0.027 eV/atom) and a 59.38% improvement over MEGNet (0.032 eV/atom).
• For bandgap prediction on the same dataset, CTGNN achieved an MAE of 0.156 eV, which is 45.26% lower than CGCNN (0.285 eV) and 47.30% lower than MEGNet (0.296 eV).
• The R² score for formation energy prediction on the perovskite dataset reached 0.996, outperforming CGCNN (0.988) and MEGNet (0.982), indicating superior predictive fidelity.
• Ablation studies showed that removing angular encoding increased MAE to 0.188 eV (from 0.156 eV), and removing the inter-atomic Transformer raised it to 0.190 eV, confirming their critical role in performance.
• The full CTGNN model achieved an MAE of 0.469 eV on the JARVIS-DFT dataset for bandgap prediction, outperforming CGCNN (0.531 eV) and MEGNet (0.493 eV), with a 11.67% and 4.87% improvement respectively.
• The performance drop when removing the dual-Transformer architecture was substantial, with MAE rising to 0.285 eV, highlighting the importance of the dual-attention mechanism in capturing multi-scale relationships.

Figure 2: Plots of predicted formation energy and bandgap versus target for CGCNN, MEGNet, and CTGNN models, respectively. the upper and right part are the target and prediction data distribution.