[Paper Review] Matbench Discovery -- A framework to evaluate machine learning crystal stability predictions

Janosh Riebesell, Rhys E. A. Goodall+8 more|arXiv (Cornell University)|Aug 28, 2023

Machine Learning in Materials Science29 citations

TL;DR

Matbench Discovery evaluates ML models for predicting crystal thermodynamic stability in a high-throughput discovery workflow, finding universal interatomic potentials (UIP) lead in accuracy and discovery acceleration, with an emphasis on stability classification over traditional regression metrics.

ABSTRACT

The rapid adoption of machine learning (ML) in domain sciences necessitates best practices and standardized benchmarking for performance evaluation. We present Matbench Discovery, an evaluation framework for ML energy models, applied as pre-filters for high-throughput searches of stable inorganic crystals. This framework addresses the disconnect between thermodynamic stability and formation energy, as well as retrospective vs. prospective benchmarking in materials discovery. We release a Python package to support model submissions and maintain an online leaderboard, offering insights into performance trade-offs. To identify the best-performing ML methodologies for materials discovery, we benchmarked various approaches, including random forests, graph neural networks (GNNs), one-shot predictors, iterative Bayesian optimizers, and universal interatomic potentials (UIP). Our initial results rank models by test set F1 scores for thermodynamic stability prediction: EquiformerV2 + DeNS > Orb > SevenNet > MACE > CHGNet > M3GNet > ALIGNN > MEGNet > CGCNN > CGCNN+P > Wrenformer > BOWSR > Voronoi fingerprint random forest. UIPs emerge as the top performers, achieving F1 scores of 0.57-0.82 and discovery acceleration factors (DAF) of up to 6x on the first 10k stable predictions compared to random selection. We also identify a misalignment between regression metrics and task-relevant classification metrics. Accurate regressors can yield high false-positive rates near the decision boundary at 0 eV/atom above the convex hull. Our results demonstrate UIPs' ability to optimize computational budget allocation for expanding materials databases. However, their limitations remain underexplored in traditional benchmarks. We advocate for task-based evaluation frameworks, as implemented here, to address these limitations and advance ML-guided materials discovery.

Motivation & Objective

• Simulate ML-based pre-filters for DFT in high-throughput discovery of stable inorganic crystals.
• Align evaluation with real-world discovery challenges by predicting stability from unrelaxed structures and measuring impact on discovery effort.
• Compare diverse ML methodologies (UIP, GNN, one-shot predictors, Bayesian optimizers, and fingerprints) under a standardized framework.
• Highlight limitations of regression metrics for discovery tasks and emphasize task-relevant classification metrics.

Proposed method

• Define a discovery-mimicking benchmark where test-time inputs are unrelaxed structures and targets are convex-hull distances from MP data.
• Use Materials Project v2022.10.28 as training data and WBM as the test set; allow unrelaxed inputs and relaxation-based targets to reflect real workflows.
• Evaluate 10 models including MACE, CHGNet, M3GNet, ALIGNN, MEGNet, CGCNN, CGCNN+P, Wrenformer, BOWSR, and Voronoi RF on stability (classification) and energy (regression) metrics.
• Employ F1-score as the primary stability metric, alongside DAF (discovery acceleration factor), TPR, TNR, MAE, RMSE, and R^2 to assess performance.
• Highlight the role of forces and stresses in UIPs by comparing force-free models to UIPs trained with relaxation data.
• Provide a Python package and online leaderboard to facilitate submissions and performance tracking.

Figure 1 : CHGNet achieves the highest cumulative precision and recall during the initial phase of the discovery campaign simulated in this figure. After the first $\mathord{\sim}$ 25k most stable predictions, MACE overtakes CHGNet in both precision and recall and maintains the lead to the end of th

Experimental results

Research questions

• RQ1 Which ML methodology yields the best balance of accuracy and robustness for ML-guided materials discovery?
• RQ2 How do regression metrics relate to task-relevant classification performance in stability prediction?
• RQ3 What is the benefit of universal interatomic potentials (UIP) that leverage relaxation data (forces/stresses) for discovery efficiency?
• RQ4 How does model performance vary with out-of-distribution tests, such as different substitution batches in the WBM dataset?

Key findings

• UIPs (MACE, CHGNet, M3GNet) outperform other models on stability classification and show large advantages in discovery acceleration factors.
• Top three models achieve F1 scores up to 0.6 for stability classification and DAF up to 5x on the first 10k most stable predictions.
• Force-enabled UIPs substantially outperform force-free models in classification, though some force-free models may match regression metrics but lag in discovery utility.
• Regression metrics (R^2, RMSE) can mislead about global predictive power for discovery; accurate energy predictions near the stability boundary can still yield many false positives/negatives.
• CHGNet delivers the best early-campaign precision/recall, while MACE leads in later-stage discovery performance in the full test set.
• There is a notable disconnect between standard regression scores and practical discovery guidance; metrics should emphasize classification performance and early-detection capability.

Figure 2 : Universal potentials are more reliable classifiers because they exit the red triangle earliest. These lines show the rolling MAE on the WBM test set as the energy to the convex hull of the MP training set is varied. Lower is better. Inside the large red ’triangle of peril’, models are mos