[Paper Review] Learning to extrapolate using continued fractions: Predicting the critical temperature of superconductor materials

Pablo Moscato, Mohammad Nazmul Haque+3 more|arXiv (Cornell University)|Nov 27, 2020

Machine Learning in Materials Science4 citations

TL;DR

This paper proposes a novel multivariate regression method, Spln-CFR, which combines continued fractions with additive spline models to predict the critical temperature of superconductors. It outperforms 10 established ML methods in out-of-domain generalization, particularly in ranking high-Tc materials, with Spln-CFR's top predictions averaging 103.64 K compared to XGBoost’s 91.31 K, indicating superior extrapolation capability for materials design.

ABSTRACT

In the field of Artificial Intelligence (AI) and Machine Learning (ML), the approximation of unknown target functions $y=f(\mathbf{x})$ using limited instances $S={(\mathbf{x^{(i)}},y^{(i)})}$, where $\mathbf{x^{(i)}} \in D$ and $D$ represents the domain of interest, is a common objective. We refer to $S$ as the training set and aim to identify a low-complexity mathematical model that can effectively approximate this target function for new instances $\mathbf{x}$. Consequently, the model's generalization ability is evaluated on a separate set $T=\{\mathbf{x^{(j)}}\} \subset D$, where $T eq S$, frequently with $T \cap S = \emptyset$, to assess its performance beyond the training set. However, certain applications require accurate approximation not only within the original domain $D$ but also in an extended domain $D'$ that encompasses $D$. This becomes particularly relevant in scenarios involving the design of new structures, where minimizing errors in approximations is crucial. For example, when developing new materials through data-driven approaches, the AI/ML system can provide valuable insights to guide the design process by serving as a surrogate function. Consequently, the learned model can be employed to facilitate the design of new laboratory experiments. In this paper, we propose a method for multivariate regression based on iterative fitting of a continued fraction, incorporating additive spline models. We compare the performance of our method with established techniques, including AdaBoost, Kernel Ridge, Linear Regression, Lasso Lars, Linear Support Vector Regression, Multi-Layer Perceptrons, Random Forests, Stochastic Gradient Descent, and XGBoost. To evaluate these methods, we focus on an important problem in the field: predicting the critical temperature of superconductors based on physical-chemical characteristics.

Motivation & Objective

• To address the challenge of accurate extrapolation in machine learning models for materials science, particularly for predicting critical temperatures (Tc) of superconductors beyond the training domain.
• To develop a surrogate model that can guide experimental design by identifying promising new superconducting materials with high Tc.
• To improve generalization performance on unseen, high-Tc materials by leveraging structural similarities in molecular descriptors through a continued fraction framework.
• To evaluate the method’s performance against state-of-the-art ML techniques in both in-domain and out-of-domain settings.

Proposed method

• The method uses a continued fraction representation to model the target function f(x), where the coefficients are optimized via a memetic algorithm to minimize mean squared error.
• It integrates additive spline models to enhance flexibility and local fitting in high-dimensional input spaces of chemical-physical descriptors.
• The model is trained using a dataset of superconducting materials with known Tc values and molecular descriptors, with optimization focused on minimizing training error while preserving generalization.
• The continued fraction structure enables implicit nonlinear transformation of input features, allowing better extrapolation than standard linear or kernel-based models.
• The method is compared against 10 established ML models: AdaBoost, Kernel Ridge, Linear Regression, Lasso Lars, Linear SVR, MLP, Random Forest, SGD, XGBoost, and the baseline CFR.
• Out-of-domain generalization is evaluated by testing on materials with Tc values higher than those in the training set, simulating real-world discovery scenarios.

(a) depth 3

Experimental results

Research questions

• RQ1 Can a continued fraction-based regression model generalize better than standard ML models in predicting critical temperatures of superconductors beyond the training domain?
• RQ2 How does the inclusion of additive spline components affect the model’s ability to extrapolate to high-Tc materials?
• RQ3 To what extent does the structural similarity of high-Tc materials in the training set enable the model to generalize to even higher Tc values?
• RQ4 Does the Spln-CFR method provide more reliable top-priority material predictions for experimental validation than conventional regression techniques?

Key findings

• Spln-CFR achieved the best generalization performance among all 11 methods tested, with a mean squared error (MSE) that outperformed all others in 192 out of 354 datasets.
• The average predicted critical temperature for the top 20 materials by Spln-CFR was 103.64 K, closely matching the actual average of 103.64 K, indicating strong predictive accuracy.
• XGBoost had a lower MSE (nearly three times smaller than Spln-CFR), but its top 20 predictions averaged only 91.31 K, significantly underestimating high-Tc materials.
• Linear Regression and Multi-Layer Perceptron both produced top predictions below 100 K, while actual Tc values for the top materials were in the 98–132.6 K range, highlighting their poor extrapolation.
• The results suggest that Spln-CFR’s superior performance in out-of-domain prediction may stem from exploiting structural similarities in high-Tc materials present in the training set.
• The study indicates that future improvements could come from incorporating domain-specific descriptors and using modified loss functions to better prioritize high-Tc material discovery.

(b) depth 5