[Paper Review] NOMAD 2018 Kaggle Competition: Solving Materials Science Challenges Through Crowd Sourcing

Christopher Sutton, Luca M. Ghiringhelli+8 more|arXiv (Cornell University)|Nov 30, 2018

Machine Learning in Materials ScienceMaterials Science84 references20 citations

TL;DR

This paper presents the NOMAD 2018 Kaggle competition, a benchmarking initiative that evaluated machine learning models for predicting two critical materials properties—electronic band gap and crystalline formation energy—in 3,000 (AlₓGa_yIn_z)₂O₃ compounds. The top solutions employed novel crystal graph representations, ensemble descriptors with gradient boosting, and smooth overlap of atomic positions with neural networks, revealing that representation choice significantly influences model performance more than the regression algorithm used.

ABSTRACT

Machine learning (ML) is increasingly used in the field of materials science, where statistical estimates of computed properties are employed to rapidly examine the chemical space for new compounds. However, a systematic comparison of several ML models for this domain has been hindered by the scarcity of appropriate datasets of materials properties, as well as the lack of thorough benchmarking studies. To address this, a public data-analytics competition was organized by the Novel Materials Discovery (NOMAD) Centre of Excellence and hosted by the on-line platform Kaggle using a dataset of $3\,000$ (Al$_x$ Ga$_y$ In$_z$)$_2$ O$_3$ compounds (with $x+y+z = 1$). The aim of this challenge was to identify the best ML model for the prediction of two key physical properties that are relevant for optoelectronic applications: the electronic band gap energy and the crystalline formation energy. In this contribution, we present a summary of the top three ML approaches of the competition including the 1st place solution based on a crystal graph representation that is new for ML of the properties of materials. The 2nd place model combined many candidate descriptors from a set of compositional, atomic environment-based, and average structural properties with the light gradient-boosting machine regression model. The 3rd place model employed the smooth overlap of atomic positions representation with a neural network. To gain insight into whether the representation or the regression model determines the overall model performance, nine ML models obtained by combining the representations and regression models of the top three approaches were compared by looking at the correlations among prediction errors. At fixed representation, the largest correlation is observed in predictions made with kernel ridge regression and neural network, reflecting a similar performance on the same test set samples.

Motivation & Objective

• To address the scarcity of standardized datasets and benchmarking studies in materials science machine learning.
• To evaluate and compare diverse machine learning models for predicting key optoelectronic properties in complex oxide materials.
• To identify the most effective feature representations and regression models for materials property prediction using a public, collaborative competition framework.
• To provide insights into the relative impact of representation versus model architecture on predictive performance.

Proposed method

• A public data science competition was hosted on Kaggle using a curated dataset of 3,000 (AlₓGa_yIn_z)₂O₃ compounds with x + y + z = 1.
• The competition focused on predicting two physical properties: electronic band gap energy and crystalline formation energy.
• The first-place solution used a novel crystal graph neural network representation to encode atomic connectivity and symmetry in the material structure.
• The second-place model combined compositional, atomic environment-based, and average structural descriptors with a light gradient-boosting machine (LGBM) regression model.
• The third-place approach applied the smooth overlap of atomic positions (SOAP) representation combined with a deep neural network for regression.
• A comparative analysis was conducted across nine hybrid models to assess the influence of representation and regression model on prediction error correlations.

Experimental results

Research questions

• RQ1 Which machine learning representation—crystal graph, SOAP, or composite descriptors—yields the most accurate predictions for materials properties in complex oxides?
• RQ2 How do different regression models (e.g., kernel ridge, neural networks, gradient boosting) perform when applied to the same set of feature representations?
• RQ3 To what extent does the choice of feature representation determine model performance compared to the choice of regression algorithm?
• RQ4 What is the correlation structure of prediction errors across diverse model combinations, and what does this reveal about model generalization and error patterns?

Key findings

• The first-place solution, based on a novel crystal graph representation, achieved the highest predictive accuracy for both band gap and formation energy.
• The second-place model, which combined multiple descriptor types with light gradient-boosting regression, demonstrated strong generalization and robustness across the test set.
• The third-place model using the smooth overlap of atomic positions (SOAP) with a neural network showed competitive performance, particularly in capturing local structural effects.
• At a fixed representation, kernel ridge regression and neural networks exhibited the highest correlation in prediction errors, indicating similar performance patterns on the same test samples.
• The study revealed that representation choice has a stronger influence on model performance than the choice of regression model, as evidenced by error correlation patterns across hybrid models.
• The competition highlighted the importance of structural and compositional feature engineering in improving machine learning predictions for materials science applications.