[Paper Review] Models and code from: Performance Assessment of Universal Machine Learning Interatomic Potentials: Challenges and Directions for Materials' Surfaces

Bruno Focassio, Luis Paulo Mezzina Freitas+1 more|arXiv (Cornell University)|Mar 7, 2024

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

This study evaluates the zero-shot generalization performance of universal machine learning interatomic potentials (UIPs) — MACE, CHGNet, and M3GNet — on surface energy prediction, revealing significant accuracy shortcomings due to out-of-domain deviations from their bulk-dominated training data. Fine-tuning these foundational models on surface-specific data dramatically improves performance, demonstrating that while UIPs are poor out-of-the-box for surfaces, they serve as highly efficient starting points for specialized, high-accuracy models.

ABSTRACT

All trained models, training data, and test data are available as files with instructions on their use in a compressed .zip file.

Motivation & Objective

• To assess the zero-shot generalization capability of universal machine learning interatomic potentials (UIPs) on materials surfaces, a domain not well-represented in their training data.
• To determine whether fine-tuning a universal UIP is more efficient and accurate than training a specialized MLIP from scratch.
• To identify the root causes of prediction errors in UIPs when applied to surfaces, particularly relating to energy scale and chemical composition.
• To advocate for the creation of a more comprehensive, universal training dataset that includes surfaces, interfaces, and diverse materials beyond bulk solids.

Proposed method

• The study uses the Materials Project surface dataset, comprising 1,497 surface structures from 73 elements, to evaluate UIPs on surface energy prediction.
• Three open-source universal models — MACE, CHGNet, and M3GNet — are evaluated using their pre-trained weights on the surface dataset.
• Fine-tuning is performed on MACE using a 90:10 train-validation split of a curated surface dataset, with hyperparameters optimized for energy and force loss.
• Specialized models are trained from scratch using NequIP and MTP architectures on the same surface data for direct performance comparison.
• Error analysis correlates prediction inaccuracies with the total energy of surface simulations and the chemical environment, identifying out-of-domain deviations.
• A comparison is made between UIPs, fine-tuned UIPs, and specialized MLIPs using parity plots and error distributions across elements.

Figure 1: MP surfaces dataset exploration. (a) Periodic table heatmap for the number of surfaces of each element. (b) Boxplot and violin plot for surface energy of each element within the dataset. The horizontal lines in the middle of the boxes mark the medians. The boxes are plotted from the first

Experimental results

Research questions

• RQ1 Can universal machine learning interatomic potentials (UIPs) accurately predict surface energies without fine-tuning, given their training on bulk materials?
• RQ2 What is the relationship between prediction error in UIPs and the energy scale or chemical environment of surface systems?
• RQ3 Is fine-tuning a universal UIP more efficient and accurate than training a specialized MLIP from scratch for surface energy prediction?
• RQ4 How do the prediction errors of UIPs correlate with the out-of-domain distance from their original training data?
• RQ5 What role does dataset composition play in the generalization performance of universal MLIPs, and how can it be improved?

Key findings

• All open-source universal MLIPs (MACE, CHGNet, M3GNet) exhibit significant prediction errors in surface energy calculations when used out-of-the-box, indicating poor zero-shot generalization to surfaces.
• Prediction errors are strongly correlated with the total energy of the surface system, suggesting that the models struggle with systems far from the energy range of their bulk training data.
• Fine-tuning a universal UIP like MACE on a small, surface-specific dataset reduces errors substantially, achieving performance comparable to specialized models trained from scratch.
• The study identifies that current universal MLIPs are not truly universal for surfaces due to insufficient coverage of surface and interface configurations in their training data.
• The results highlight that while universal models are excellent starting points for fine-tuning, they are not yet ready for direct application to complex materials surfaces without adaptation.
• The authors advocate for the creation of a comprehensive, universal training dataset that includes molecules, surfaces, interfaces, and defects to enable true generalization across the materials space.

Figure 2: Performance assessment of the universal interatomic potentials over the surfaces dataset. (a) Parity plot for the total energy per atom of the bulk systems that gave origin to the surfaces of the dataset. (b) Parity plot for the total energy per atom of the surfaces within the dataset. (c)