[Paper Review] Probabilistic Missing Value Imputation for Mixed Categorical and Ordered Data

Yuxuan Zhao, Alex Townsend+1 more|arXiv (Cornell University)|Oct 13, 2022

Data Management and Algorithms4 citations

TL;DR

This paper proposes the extended Gaussian copula (EGC), a hyperparameter-free probabilistic model for mixed categorical and ordered data that uses a latent Gaussian distribution with argmax-based modeling of categorical variables. It achieves state-of-the-art imputation accuracy for both categorical and ordered variables, supports both single and multiple imputation, and outperforms existing methods in accuracy and efficiency on synthetic and real-world datasets.

ABSTRACT

Many real-world datasets contain missing entries and mixed data types including categorical and ordered (e.g. continuous and ordinal) variables. Imputing the missing entries is necessary, since many data analysis pipelines require complete data, but this is challenging especially for mixed data. This paper proposes a probabilistic imputation method using an extended Gaussian copula model that supports both single and multiple imputation. The method models mixed categorical and ordered data using a latent Gaussian distribution. The unordered characteristics of categorical variables is explicitly modeled using the argmax operator. The method makes no assumptions on the data marginals nor does it require tuning any hyperparameters. Experimental results on synthetic and real datasets show that imputation with the extended Gaussian copula outperforms the current state-of-the-art for both categorical and ordered variables in mixed data.

Motivation & Objective

• To address the challenge of imputing missing values in mixed-type datasets containing both categorical and ordered variables.
• To develop a probabilistic imputation method that explicitly models unordered categorical variables without arbitrary encoding or preprocessing.
• To enable both single and multiple imputation with uncertainty quantification for downstream analysis.
• To create a model that makes no assumptions about marginal distributions and requires no hyperparameters.
• To improve robustness and efficiency over existing methods like MICE, missForest, and LRMC, especially in high-dimensional or long-skinny data regimes.

Proposed method

• The extended Gaussian copula models each categorical variable as the argmax of a latent Gaussian vector, preserving the unordered nature of categories.
• Ordered variables (continuous and ordinal) are modeled as latent Gaussian scalars, with dependence captured through a latent correlation matrix.
• The method uses a latent Gaussian copula framework with explicit marginal modeling, enabling joint distribution estimation without assuming parametric forms for marginals.
• A robust, scalable parameter fitting algorithm is proposed, supporting mini-batch training, parallelization, and low-rank structure for efficiency.
• Multiple imputation is enabled by sampling from the full conditional distribution, providing category probabilities and confidence intervals.
• The model is identifiable and can match arbitrary categorical marginal distributions, unlike prior approaches with redundant parameters.

Experimental results

Research questions

• RQ1 Can a probabilistic imputation model effectively handle mixed categorical and ordered data without requiring data preprocessing or hyperparameter tuning?
• RQ2 How well can a latent Gaussian model with argmax-based categorical representation capture complex dependence structures in mixed data?
• RQ3 Does the extended Gaussian copula outperform state-of-the-art methods like MICE, missForest, and softImpute in both imputation accuracy and computational efficiency?
• RQ4 Can the model accurately estimate categorical marginal distributions and propagate uncertainty through multiple imputations?
• RQ5 How does the method perform under different missing data mechanisms and data scales?

Key findings

• The extended Gaussian copula (EGC) achieves state-of-the-art imputation accuracy on both categorical and ordered variables across synthetic and real-world datasets.
• On synthetic datasets with n=1000, EGC achieved a categorical error of 0.64(0.01) and continuous error of 1.81(0.04), outperforming missForest and softImpute.
• For n=10000 and n=20000, EGC maintained superior accuracy with 0.64(0.01) and 0.64(0.01) categorical error, respectively, while being significantly faster than missForest.
• The model demonstrated robust categorical marginal fitting, with estimated probabilities closely matching empirical distributions across 70 datasets and varying numbers of categories.
• Runtime for EGC was 107(4) seconds at n=10000, compared to 1006(70) seconds for missForest, showing a fourfold speedup despite higher accuracy.
• The method consistently outperformed softImpute on categorical data but showed poor performance on ordered variables, highlighting the need for joint modeling of data types.