[Paper Review] Impact of Missing Values in Machine Learning: A Comprehensive Analysis

Abu Fuad Ahmad, Md Shohel Sayeed+2 more|arXiv (Cornell University)|Oct 10, 2024

Stock Market Forecasting Methods4 citations

TL;DR

This paper provides a comprehensive analysis of how missing values impact machine learning workflows, examining their types, causes, and consequences on model performance, bias, and computational load. It evaluates imputation and removal strategies, highlights challenges in cross-validation and model evaluation, and offers practical insights through real-world case studies, advocating for ethical and transparent handling of missing data to ensure robust and reliable ML outcomes.

ABSTRACT

Machine learning (ML) has become a ubiquitous tool across various domains of data mining and big data analysis. The efficacy of ML models depends heavily on high-quality datasets, which are often complicated by the presence of missing values. Consequently, the performance and generalization of ML models are at risk in the face of such datasets. This paper aims to examine the nuanced impact of missing values on ML workflows, including their types, causes, and consequences. Our analysis focuses on the challenges posed by missing values, including biased inferences, reduced predictive power, and increased computational burdens. The paper further explores strategies for handling missing values, including imputation techniques and removal strategies, and investigates how missing values affect model evaluation metrics and introduces complexities in cross-validation and model selection. The study employs case studies and real-world examples to illustrate the practical implications of addressing missing values. Finally, the discussion extends to future research directions, emphasizing the need for handling missing values ethically and transparently. The primary goal of this paper is to provide insights into the pervasive impact of missing values on ML models and guide practitioners toward effective strategies for achieving robust and reliable model outcomes.

Motivation & Objective

• To analyze the nuanced effects of missing values on machine learning model performance and generalization.
• To identify the types, causes, and consequences of missing data in ML workflows.
• To evaluate imputation and removal strategies for handling missing values.
• To investigate how missing values affect model evaluation metrics and cross-validation procedures.
• To promote ethical and transparent handling of missing data in ML applications.

Proposed method

• Systematic review of missing data types (MCAR, MAR, MNAR) and their impact on ML pipelines.
• Evaluation of common imputation techniques, including mean/mode imputation, KNN imputation, and model-based imputation.
• Comparison of removal strategies (listwise and pairwise deletion) in terms of data loss and model bias.
• Integration of case studies using real-world datasets to demonstrate practical implications.
• Analysis of how missing values distort cross-validation and model selection processes.
• Discussion of ethical considerations in data handling, emphasizing transparency and reproducibility.

Experimental results

Research questions

• RQ1 How do different types of missing data (MCAR, MAR, MNAR) affect the predictive performance of machine learning models?
• RQ2 What are the relative trade-offs between imputation and deletion strategies in terms of bias, variance, and computational cost?
• RQ3 To what extent do missing values compromise the reliability of model evaluation metrics such as accuracy, AUC, and F1-score?
• RQ4 How do missing values introduce complexities in cross-validation and model selection?
• RQ5 What ethical and transparent practices should be adopted when handling missing data in machine learning?

Key findings

• Missing values significantly reduce model predictive power and increase the risk of biased inferences, especially under MAR and MNAR mechanisms.
• Imputation techniques, particularly model-based and KNN imputation, outperform simple mean/mode imputation in preserving model performance.
• Removal strategies lead to substantial data loss and can distort model generalization, especially when missingness is not MCAR.
• Missing values introduce systematic bias in cross-validation, leading to overly optimistic or pessimistic performance estimates.
• The study demonstrates through case studies that improper handling of missing data can invalidate model interpretability and fairness.
• Ethical data handling practices, including documentation of missingness patterns and transparent reporting, are essential for reproducible and trustworthy ML research.