[Paper Review] Recent advances in the SISSO method and their implementation in the SISSO++ code

Thomas A. R. Purcell, Matthias Scheffler+1 more|arXiv (Cornell University)|May 2, 2023

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

This paper presents major algorithmic and implementation advances in the SISSO++ code, introducing binary expression trees for grammatically consistent feature representation, parametric SISSO for non-linear parameter optimization, and multi-residual learning to improve model expressivity and accuracy. These enhancements significantly expand the range of physically meaningful, explainable AI models discoverable in materials science applications.

ABSTRACT

Accurate and explainable artificial-intelligence (AI) models are promising tools for the acceleration of the discovery of new materials, ore new applications for existing materials. Recently, symbolic regression has become an increasingly popular tool for explainable AI because it yields models that are relatively simple analytical descriptions of target properties. Due to its deterministic nature, the sure-independence screening and sparsifying operator (SISSO) method is a particularly promising approach for this application. Here we describe the new advancements of the SISSO algorithm, as implemented into SISSO++, a C++ code with Python bindings. We introduce a new representation of the mathematical expressions found by SISSO. This is a first step towards introducing ``grammar'' rules into the feature creation step. Importantly, by introducing a controlled non-linear optimization to the feature creation step we expand the range of possible descriptors found by the methodology. Finally, we introduce refinements to the solver algorithms for both regression and classification, that drastically increase the reliability and efficiency of SISSO. For all of these improvements to the basic SISSO algorithm, we not only illustrate their potential impact, but also fully detail how they operate both mathematically and computationally.

Motivation & Objective

• To enhance the expressivity and reliability of the SISSO symbolic regression method for materials property prediction.
• To enable grammatically correct and physically consistent mathematical expressions through binary expression tree representation.
• To extend SISSO's capability to discover complex, non-linear models via parametric feature optimization.
• To improve descriptor identification through multi-residual learning and efficient solver algorithms.
• To provide a modular, open-source C++ codebase with Python bindings for broader adoption in materials informatics.

Proposed method

• Representing features as binary expression trees enables recursive manipulation, unit tracking, and domain/range control for physical consistency.
• Introducing parametric SISSO allows non-linear parameters to be optimized during feature creation using a loss function based on compressed sensing metrics.
• Implementing multiple residuals in the descriptor identification step enables learning from progressively refined approximations of the target property.
• Using linear programming for classification problems improves solver efficiency and reliability.
• Applying sure-independence screening (SIS) with projection scores to select relevant features from a large pool of generated expressions.
• Performing ℓ₀-regularized optimization on a subspace 𝒮 to identify the optimal n-dimensional descriptor for regression or classification.

Figure 1: A demonstration of the new representation of the features in the SISSO++ code. The feature is stored as the root of the tree (represented by the thick border), the primary features are the leaves, and the rung corresponds to the height of the tree, i.e. the longest path between each leaf a

Experimental results

Research questions

• RQ1 How can symbolic regression in SISSO be made more expressive and capable of discovering complex, non-linear models?
• RQ2 Can grammatical constraints be systematically introduced into symbolic regression to ensure physical consistency and numerical stability?
• RQ3 How does multi-residual learning improve the accuracy and convergence of SISSO models?
• RQ4 What impact does parametric optimization of features have on model performance and interpretability?
• RQ5 How can the computational efficiency and reliability of SISSO solvers be enhanced for real-world materials science applications?

Key findings

• The binary expression tree representation enables full control over units, domains, and ranges of generated features, reducing numerical errors and ensuring physical consistency.
• Parametric SISSO allows the discovery of non-linear models by optimizing parameters during feature creation, demonstrated in predicting thermal conductivity.
• Multi-residual learning reduces the required SIS subspace size by over 90% compared to single-residual approaches, enabling faster convergence to accurate models.
• With 50 residuals, SISSO achieved near-perfect recovery of a synthetic two-dimensional model with noise, while single-residual learning required over 400 features to achieve similar performance.
• The linear programming implementation for classification problems improves solver stability and efficiency, especially in high-dimensional descriptor spaces.
• The SISSO++ code enables reproducible, modular, and extensible use of symbolic regression in materials discovery, with all data and scripts openly available.

Figure 3: A comparison of the expressions found non-parametric (a, c, and e) and parametric SISSO (b, d, and f) for a Lorentzian (a, b, e, and f) and sin (c, d) function. Blue dots represent the training data, and the red line represents the expressions found by SISSO. The parameterization scheme ei