[Paper Review] Importance of feature engineering and database selection in a machine learning model: A case study on carbon crystal structures

Franz M. Rohrhofer, Santanu Saha+4 more|arXiv (Cornell University)|Jan 30, 2021

Machine Learning in Materials Science4 citations

TL;DR

This study investigates how feature engineering and database selection impact machine learning performance in predicting total energy of carbon crystal structures. Using kernel ridge regression with simple, interpretable features (SGD, RDF, ADF), the authors show model performance strongly depends on feature choice and database composition, with optimal results achieved through systematic hyperparameter tuning and database subset selection, highlighting non-transferable performance across structural types.

ABSTRACT

Drive towards improved performance of machine learning models has led to the creation of complex features representing a database of condensed matter systems. The complex features, however, do not offer an intuitive explanation on which physical attributes do improve the performance. The effect of the database on the performance of the trained model is often neglected. In this work we seek to understand in depth the effect that the choice of features and the properties of the database have on a machine learning application. In our experiments, we consider the complex phase space of carbon as a test case, for which we use a set of simple, human understandable and cheaply computable features for the aim of predicting the total energy of the crystal structure. Our study shows that (i) the performance of the machine learning model varies depending on the set of features and the database, (ii) is not transferable to every structure in the phase space and (iii) depends on how well structures are represented in the database.

Motivation & Objective

• To investigate the influence of feature engineering and database composition on machine learning model performance in predicting total energy of carbon crystal structures.
• To evaluate the impact of different features—SGD, RDF, and ADF—on model generalization and accuracy.
• To determine whether model performance is transferable across different structural types in the carbon phase space.
• To identify optimal hyperparameters and feature combinations through systematic grid search and cross-validation.
• To provide practical guidelines for constructing robust, interpretable ML models in condensed matter systems.

Proposed method

• The study employs kernel ridge regression (KRR) as the machine learning model for predicting total energy of carbon crystal structures.
• Features include simple, physically interpretable descriptors: short-range order (SGD), radial distribution function (RDF), and angular distribution function (ADF).
• A grid search over hyperparameters (regularization σ and kernel coefficient γ) was conducted using 5-fold cross-validation to optimize model performance.
• The MHM+RS database of 11,500 carbon structures was used, with additional subset selection into Cluster 1 (loose, sp²-dominated) and Cluster 2 (compact, sp³-dominated) for testing generalization.
• Feature uniqueness was validated via Euclidean distance between normalized feature vectors; non-zero distances confirmed uniqueness.
• Model training and prediction were implemented in Python using Scikit-learn, with feature extraction taking ~15 minutes on a single processor.

Experimental results

Research questions

• RQ1 How do different feature sets (SGD, RDF, ADF) affect the predictive performance of a machine learning model for carbon crystal total energy?
• RQ2 To what extent does the composition and representation of the training database influence model generalization and accuracy?
• RQ3 Is the performance of a trained model transferable across distinct structural types in the carbon phase space?
• RQ4 What is the optimal combination of hyperparameters (σ and γ) for the KRR model across different feature and database configurations?
• RQ5 How do feature distributions and structural clustering (e.g., Cluster 1 vs. Cluster 2) correlate with model performance?

Key findings

• Model performance varies significantly depending on the choice of features and database composition, with no single combination universally optimal.
• The optimal bin size for the ADF feature was found to be Δθ = 15°, with smoothing σ_ADF = 15°, yielding 12 bins over the 0°–180° range.
• The MHM+RS database, when combined with SGD+RDF+ADF features, achieved the lowest mean squared error in 5-fold cross-validation.
• Performance was not transferable across structural types: models trained on Cluster 1 structures performed poorly on Cluster 2 and vice versa.
• Feature uniqueness was confirmed, as all pairwise Euclidean distances between normalized feature vectors were non-zero, ensuring distinct representation.
• The study demonstrates that database representation quality—especially coverage of structural types—directly impacts model generalization and predictive accuracy.