[Paper Review] Imputation for prediction: beware of diminishing returns

Marine Le Morvan, Gaël Varoquaux|arXiv (Cornell University)|Jul 29, 2024

Forecasting Techniques and ApplicationsDecision Sciences3 citations

TL;DR

This study investigates whether improved imputation techniques significantly enhance predictive performance across 20 datasets under MCAR conditions. It finds that while imputation accuracy matters, gains in prediction are marginal—especially with flexible models or when using missingness indicators—challenging the common assumption that better imputation always improves prediction.

ABSTRACT

Missing values are prevalent across various fields, posing challenges for training and deploying predictive models. In this context, imputation is a common practice, driven by the hope that accurate imputations will enhance predictions. However, recent theoretical and empirical studies indicate that simple constant imputation can be consistent and competitive. This empirical study aims at clarifying if and when investing in advanced imputation methods yields significantly better predictions. Relating imputation and predictive accuracies across combinations of imputation and predictive models on 19 datasets, we show that imputation accuracy matters less i) when using expressive models, ii) when incorporating missingness indicators as complementary inputs, iii) matters much more for generated linear outcomes than for real-data outcomes. Interestingly, we also show that the use of the missingness indicator is beneficial to the prediction performance, even in MCAR scenarios. Overall, on real-data with powerful models, improving imputation only has a minor effect on prediction performance. Thus, investing in better imputations for improved predictions often offers limited benefits.

Motivation & Objective

• To empirically assess whether enhanced imputation accuracy leads to measurable improvements in downstream predictive performance.
• To evaluate the impact of imputation quality under Missing Completely At Random (MCAR) conditions, where imputation methods are theoretically valid.
• To investigate how model flexibility, missingness indicators, and outcome linearity modulate the relationship between imputation accuracy and prediction performance.
• To clarify whether advanced imputation methods provide meaningful benefits in real-world predictive modeling scenarios.
• To provide actionable insights for practitioners on when to invest in complex imputation versus focusing on modeling choices.

Proposed method

• The study evaluates 20 real-world numerical datasets under MCAR missingness with varying missing rates (20%, 50%).
• It compares multiple imputation methods—including mean imputation, missforest, iterativeBR, and advanced methods like VAE and GAN-based imputers—on their accuracy.
• Prediction models include linear models, XGBoost, and neural networks, with and without a missingness indicator (a binary feature marking missing values).
• Imputation and prediction pipelines are systematically combined across all model and imputation pairs, with hyperparameter tuning via cross-validation.
• Prediction performance is measured using RMSE for regression and AUC for classification, while imputation accuracy is assessed via RMSE or MAE.
• The analysis isolates the contribution of imputation quality by comparing prediction gains across imputation methods, controlling for model choice and missingness indicators.

Experimental results

Research questions

• RQ1 Does improving imputation accuracy lead to significant gains in predictive performance across diverse datasets and models?
• RQ2 How does the use of a missingness indicator affect the benefit of better imputations in MCAR scenarios?
• RQ3 Is the relationship between imputation accuracy and prediction performance stronger for linear outcomes than for real-world data?
• RQ4 How do model expressivity (e.g., XGBoost vs. linear models) modulate the impact of imputation quality on prediction?
• RQ5 Are gains in imputation accuracy consistently translated into gains in prediction performance, or is the correlation weak?

Key findings

• Improving imputation accuracy yields only marginal gains in predictive performance when using powerful, flexible models like XGBoost or neural networks.
• The inclusion of a missingness indicator significantly boosts prediction performance—even in MCAR settings where missingness is uninformative—enabling better utilization of advanced imputations.
• For linearly generated outcomes, better imputations lead to better predictions, but this relationship breaks down in real-world data with non-linear relationships.
• Even with 50% missing rate, missforest-based predictors outperform iterativeBR-based ones despite comparable imputation accuracy, indicating that imputation quality alone is not a reliable proxy for prediction gain.
• The benefit of advanced imputation diminishes when models can natively handle missing values or when the missingness indicator is used, suggesting that modeling choices often matter more than imputation sophistication.
• In MCAR settings, where imputation is theoretically valid, the gains from better imputation are still small, implying even smaller gains in real-world MNAR scenarios.