[Paper Review] Periodic Graph Transformers for Crystal Material Property Prediction

Keqiang Yan, Yi Liu+2 more|arXiv (Cornell University)|Sep 23, 2022

Machine Learning in Materials Science43 citations

TL;DR

Matformer, a periodic graph transformer, achieves periodic invariance and explicit periodic-pattern encoding to predict crystal properties, outperforming baselines on Materials Project and Jarvis benchmarks. It also shows efficiency gains over prior methods.

ABSTRACT

We consider representation learning on periodic graphs encoding crystal materials. Different from regular graphs, periodic graphs consist of a minimum unit cell repeating itself on a regular lattice in 3D space. How to effectively encode these periodic structures poses unique challenges not present in regular graph representation learning. In addition to being E(3) invariant, periodic graph representations need to be periodic invariant. That is, the learned representations should be invariant to shifts of cell boundaries as they are artificially imposed. Furthermore, the periodic repeating patterns need to be captured explicitly as lattices of different sizes and orientations may correspond to different materials. In this work, we propose a transformer architecture, known as Matformer, for periodic graph representation learning. Our Matformer is designed to be invariant to periodicity and can capture repeating patterns explicitly. In particular, Matformer encodes periodic patterns by efficient use of geometric distances between the same atoms in neighboring cells. Experimental results on multiple common benchmark datasets show that our Matformer outperforms baseline methods consistently. In addition, our results demonstrate the importance of periodic invariance and explicit repeating pattern encoding for crystal representation learning.

Motivation & Objective

• Motivate accurate crystal material property prediction by leveraging the periodic nature of crystal structures.
• Develop a graph transformer that is invariant to unit cell periodic shifts and scalable unit cells.
• Explicitly encode periodic lattice patterns to capture size and orientation effects on properties.

Proposed method

• Introduce Matformer with two periodic-invariant graph constructions: multi-edge and fully-connected crystal graphs.
• Encode periodic patterns by adding six self-connecting edges that represent lattice vector lengths and their pairwise sums.
• Use edge-wise attention in a transformer-based message passing scheme tailored to multi-edge crystal graphs.
• Employ a training setup and baselines on Materials Project and Jarvis datasets to evaluate performance.
• Compare with GNN and graph transformer baselines and perform ablations on periodic invariance and pattern encoding.

Experimental results

Research questions

• RQ1 How can periodic invariance be guaranteed in crystal graph representations for property prediction?
• RQ2 What is the impact of explicitly encoding periodic patterns on predictive performance for crystal properties?
• RQ3 Do periodic-invariant graph constructions plus pattern encoding improve accuracy and efficiency over existing methods?

Key findings

• Matformer achieves best test MAE on the Materials Project dataset across formation energy, band gap, bulk moduli, and shear moduli.
• On Jarvis tasks, Matformer outperforms baselines on formation energy, bandgap (OPT), total energy, Ehull, and bandgap (MBJ).
• Matformer is more efficient than ALIGNN in training and inference time and uses a smaller model footprint.
• A EwT analysis shows Matformer provides higher energy predictions within thresholds than ALIGNN, especially on larger datasets like Materials Project.
• Ablation studies underscore the importance of periodic invariance and explicit periodic-pattern encoding for crystal representations.