[Paper Review] Comparing methods addressing multi-collinearity when developing prediction models

Artuur Leeuwenberg, Maarten van Smeden+6 more|arXiv (Cornell University)|Jan 5, 2021

Artificial Intelligence in Healthcare and Education4 citations

TL;DR

This study compares methods to address multicollinearity in clinical prediction model development, evaluating penalization (Lasso, Ridge, Elastic Net), dimensionality reduction (PCA), and constrained optimization (dropout, PCM). While multicollinearity did not affect predictive performance, it severely undermined coefficient stability—especially under Lasso, which showed high variability in predictor selection due to collinearity-induced instability.

ABSTRACT

Clinical prediction models are developed widely across medical disciplines. When predictors in such models are highly collinear, unexpected or spurious predictor-outcome associations may occur, thereby potentially reducing face-validity and explainability of the prediction model. Collinearity can be dealt with by exclusion of collinear predictors, but when there is no a priori motivation (besides collinearity) to include or exclude specific predictors, such an approach is arbitrary and possibly inappropriate. We compare different methods to address collinearity, including shrinkage, dimensionality reduction, and constrained optimization. The effectiveness of these methods is illustrated via simulations. In the conducted simulations, no effect of collinearity was observed on predictive outcomes. However, a negative effect of collinearity on the stability of predictor selection was found, affecting all compared methods, but in particular methods that perform strong predictor selection (e.g., Lasso).}

Motivation & Objective

• To evaluate the impact of multicollinearity on predictive performance and coefficient stability in clinical prediction models.
• To compare the effectiveness of multiple methods—penalization (Lasso, Ridge, Elastic Net), dimensionality reduction (PCA), and constrained optimization (dropout, PCM)—in addressing multicollinearity.
• To assess whether multicollinearity affects model calibration, discrimination, or predictor selection stability.
• To determine whether strong predictor selection methods like Lasso are unduly influenced by collinearity in high-dimensional, correlated predictor settings.
• To provide evidence-based guidance on method selection for robust model development under multicollinearity.

Proposed method

• Simulated data based on real radiotherapy complication data (NTCP models) with 1000 patients and 20 predictors, including high pairwise correlations (r > 0.8) among dose-volume parameters.
• Applied five methods: standard logistic regression (LR), Lasso, Ridge, Elastic Net, Principal Component Analysis (PCA), and a novel method using prior knowledge (PCM) to constrain coefficients.
• Used 1000 bootstrap resamples per simulation to assess stability of coefficient estimates and predictor selection across methods.
• Evaluated performance using C-statistic (discrimination), Brier score (calibration), and coefficient stability (variance of estimated coefficients across bootstrap samples).
• Applied dropout regularization to reduce co-adaptation of coefficients in neural network-based models, though results were limited to logistic regression and penalized methods.
• Used cross-validation to tune hyperparameters (e.g., lambda in Lasso, alpha in Elastic Net) and ensure fair comparison across methods.

Experimental results

Research questions

• RQ1 Does multicollinearity significantly affect the predictive performance (discrimination and calibration) of clinical prediction models?
• RQ2 How does multicollinearity impact the stability of coefficient estimates and predictor selection across different modeling methods?
• RQ3 Which method—penalization, dimensionality reduction, or constrained optimization—best preserves model stability and predictive accuracy under high multicollinearity?
• RQ4 Does the Lasso method suffer from increased instability in predictor selection when predictors are highly collinear, despite its shrinkage properties?
• RQ5 Can prior clinical knowledge (via PCM) improve model stability and reduce the impact of multicollinearity compared to data-driven methods?

Key findings

• Multicollinearity did not significantly affect predictive performance: C-statistic and Brier score were comparable across all methods, including standard logistic regression.
• Despite stable predictive performance, multicollinearity severely reduced coefficient stability, especially for methods relying on strong predictor selection like Lasso, which showed high variance in selected predictors across bootstrap samples.
• Lasso exhibited the highest instability in predictor selection, with large variability in which predictors were retained across bootstrap resamples, even when predictors were highly correlated.
• Ridge and Elastic Net showed better stability due to their grouping effect on collinear predictors, reducing variance in coefficient estimates.
• PCA-based models showed moderate stability but often led to less interpretable models due to transformed components with unclear clinical meaning.
• The PCM method, which incorporated prior knowledge to constrain coefficients, improved stability and reduced variance in coefficient estimates compared to purely data-driven methods, particularly in high-collinearity settings.