[Paper Review] Variable selection with genetic algorithms using repeated cross-validation of PLS regression models as fitness measure

David Kepplinger, Peter Filzmoser+1 more|ArXiv.org|Nov 17, 2017

Spectroscopy and Chemometric Analyses3 citations

TL;DR

This paper proposes a novel fitness evaluation framework for genetic algorithms in variable selection, using repeated cross-validation of PLS regression models to estimate internal and external prediction performance. By optimizing for prediction accuracy via cross-validated PLS models, the method identifies variable subsets with superior external prediction power compared to traditional fitness measures, particularly outperforming standard R²-based criteria in real-world chemometric datasets.

ABSTRACT

Genetic algorithms are a widely used method in chemometrics for extracting variable subsets with high prediction power. Most fitness measures used by these genetic algorithms are based on the ordinary least-squares fit of the resulting model to the entire data or a subset thereof. Due to multicollinearity, partial least squares regression is often more appropriate, but rarely considered in genetic algorithms due to the additional cost for estimating the optimal number of components. We introduce two novel fitness measures for genetic algorithms, explicitly designed to estimate the internal prediction performance of partial least squares regression models built from the variable subsets. Both measures estimate the optimal number of components using cross-validation and subsequently estimate the prediction performance by predicting the response of observations not included in model-fitting. This is repeated multiple times to estimate the measures' variations due to different random splits. Moreover, one measure was optimized for speed and more accurate estimation of the prediction performance for observations not included during variable selection. This leads to variable subsets with high internal and external prediction power. Results on high-dimensional chemical-analytical data show that the variable subsets acquired by this approach have competitive internal prediction power and superior external prediction power compared to variable subsets extracted with other fitness measures.

Motivation & Objective

• To address the limitation of existing genetic algorithm fitness measures in chemometrics, which often rely on ordinary least squares and fail to account for multicollinearity.
• To develop a fitness evaluation method that accurately estimates the internal and external prediction performance of PLS regression models built from variable subsets.
• To enhance variable selection in high-dimensional chemical-analytical data by incorporating cross-validation to estimate optimal PLS components and prediction error.
• To improve the predictive robustness of QSPR models by favoring variable subsets that generalize well to unseen data.

Proposed method

• The method uses repeated double cross-validation (rdCV) and a simplified variant, simple repeated cross-validation (srCV), as fitness measures for genetic algorithms.
• For each variable subset, the optimal number of PLS components is estimated via cross-validation to minimize prediction error on unseen data.
• Prediction performance is evaluated by predicting responses of observations not included in model fitting, repeated across multiple random data splits to estimate variability.
• The srCV variant improves computational efficiency by relaxing constraints on test set size, enabling faster yet accurate estimation of external prediction power.
• The fitness measure is designed to prioritize models with high internal and external prediction accuracy, not just model fit.
• The approach is implemented in the R package gaselect, available on CRAN, for broad accessibility in chemometric applications.

Experimental results

Research questions

• RQ1 Can cross-validated PLS regression models provide a more reliable fitness measure for genetic algorithms in variable selection than traditional R²-based criteria?
• RQ2 How does the use of repeated cross-validation for estimating PLS components affect the prediction performance of selected variable subsets?
• RQ3 Does a simplified cross-validation approach (srCV) maintain high accuracy while significantly reducing computation time compared to repeated double cross-validation?
• RQ4 To what extent do variable subsets selected using the proposed fitness measures outperform those selected with standard methods in terms of external prediction power?

Key findings

• The srCV fitness measure produced variable subsets with significantly superior external prediction performance, achieving a total RMSEP of 0.503 on the KOC dataset, outperforming the best subset reported in Gramatica et al. (2011), which had an RMSEP of 0.532.
• The variable subsets selected using the proposed cross-validation-based fitness measures showed higher external prediction power (RMSEP_ext. validation) than those selected using R² or BIC-based criteria.
• The srCV method reduced computation time by over 500% compared to repeated double cross-validation while maintaining high accuracy in performance estimation.
• Models selected with the new fitness measures required fewer PLS components on average, indicating simpler and more robust models.
• All validation criteria overestimated model performance, as evidenced by lower RMSEP on training data than on validation data, confirming the need for proper external validation.
• The proposed method, particularly srCV, enables the identification of variable subsets with high generalization ability, making it suitable for high-dimensional chemometric data with multicollinearity.