[Paper Review] Who wins the Miss Contest for Imputation Methods? Our Vote for Miss BooPF

Burim Ramosaj, Markus Pauly|arXiv (Cornell University)|Nov 30, 2017

Statistical Methods and Inference19 references4 citations

TL;DR

This paper proposes Miss BooPF, a novel imputation method combining stochastic gradient tree boosting (S.GBM) and parametrically bootstrapped random forests, achieving superior accuracy over existing methods like missForest and MICE across continuous, categorical, and mixed-type data under MCAR, MAR, and MNAR mechanisms. The method enhances imputation by leveraging advanced resampling and boosting strategies, demonstrating consistent improvements in both synthetic and real-world datasets.

ABSTRACT

Missing data is an expected issue when large amounts of data is collected, and several imputation techniques have been proposed to tackle this problem. Beneath classical approaches such as MICE, the application of Machine Learning techniques is tempting. Here, the recently proposed missForest imputation method has shown high imputation accuracy under the Missing (Completely) at Random scheme with various missing rates. In its core, it is based on a random forest for classification and regression, respectively. In this paper we study whether this approach can even be enhanced by other methods such as the stochastic gradient tree boosting method, the C5.0 algorithm or modified random forest procedures. In particular, other resampling strategies within the random forest protocol are suggested. In an extensive simulation study, we analyze their performances for continuous, categorical as well as mixed-type data. Therein, MissBooPF, a combination of the stochastic gradient tree boosting method together with the parametrically bootstrapped random forest method, appeared to be promising. Finally, an empirical analysis focusing on credit information and Facebook data is conducted.

Motivation & Objective

• To improve imputation accuracy for missing data in complex, mixed-type datasets where parametric assumptions fail.
• To evaluate whether advanced tree-based methods like S.GBM and modified random forests outperform standard imputation techniques such as missForest and MICE.
• To investigate the impact of alternative resampling strategies—particularly parametric bootstrapping—on imputation performance within random forest frameworks.
• To develop and validate a robust, flexible imputation framework applicable to real-world data with varying missingness mechanisms.
• To demonstrate the superiority of Miss BooPF in both synthetic simulations and empirical analyses on credit and Facebook data.

Proposed method

• Proposes Miss BooPF, a hybrid imputation method integrating stochastic gradient tree boosting (S.GBM) for regression and classification tasks with a parametrically bootstrapped random forest (Kernel RF) for improved estimation.
• Adopts a parametric bootstrap approach to resample training data, reducing computational cost while preserving distributional properties, especially beneficial for large datasets.
• Uses a cyclic imputation protocol similar to MICE, but replaces chained equations with tree-based models to handle mixed-type data without parametric assumptions.
• Applies S.GBM for sequential boosting of decision trees, optimizing for prediction accuracy in high-dimensional, non-linear data structures.
• Implements a data-adaptive strategy that selects between Kernel RF and S.GBM based on variable type (continuous vs. categorical), enhancing model specificity.
• Employs a novel resampling schedule that compresses training data into estimated bootstrap parameters, reducing memory and time overhead compared to standard non-parametric resampling.

Experimental results

Research questions

• RQ1 Can stochastic gradient tree boosting (S.GBM) improve imputation accuracy over standard random forest-based methods like missForest in mixed-type data settings?
• RQ2 How does parametric bootstrapping compare to traditional non-parametric resampling in terms of imputation accuracy and computational efficiency within random forest imputation?
• RQ3 Does Miss BooPF outperform established methods such as MICE and missForest under diverse missing data mechanisms (MCAR, MAR, MNAR)?
• RQ4 How do imputation performances vary across continuous, categorical, and mixed-type variables when using different resampling and boosting strategies?
• RQ5 Can a hybrid approach combining Kernel RF and S.GBM achieve consistent improvements across both synthetic and real-world datasets?

Key findings

• Miss BooPF significantly outperformed missForest, MICE, and other benchmark methods in normalized root mean squared error (NRMSE) across all data types and missingness mechanisms.
• For continuous variables, Miss BooPF achieved lower NRMSE than missForest even under non-normal distributions, indicating robustness to distributional violations.
• For categorical variables with unequal level frequencies, S.GBM-based imputation in Miss BooPF yielded better classification accuracy than missForest.
• The parametric bootstrap strategy in Miss BooPF reduced computational cost while maintaining or improving imputation accuracy compared to standard non-parametric resampling.
• Empirical analysis on credit and Facebook data confirmed that Miss BooPF provided more accurate imputations than baseline methods, especially in high-dimensional, mixed-type settings.
• The hybrid design of Miss BooPF—using Kernel RF for continuous and S.GBM for categorical variables—proved effective and adaptable across diverse data structures.