[Paper Review] Impacts of Dirty Data: and Experimental Evaluation

Zhixin Qi, Hongzhi Wang+2 more|arXiv (Cornell University)|Mar 16, 2018

Machine Learning and Data Classification44 references11 citations

TL;DR

This paper conducts an experimental evaluation of how missing, inconsistent, and conflicting data affect classification and clustering algorithms, introducing novel metrics—sensibility and keeping point—to quantify robustness. It provides data-driven guidelines for algorithm selection and targeted data cleaning based on error rates and task requirements, reducing unnecessary cleaning costs while maintaining accuracy.

ABSTRACT

Data quality issues have attracted widespread attention due to the negative impacts of dirty data on data mining and machine learning results. The relationship between data quality and the accuracy of results could be applied on the selection of the appropriate algorithm with the consideration of data quality and the determination of the data share to clean. However, rare research has focused on exploring such relationship. Motivated by this, this paper conducts an experimental comparison for the effects of missing, inconsistent and conflicting data on classification and clustering algorithms. Based on the experimental findings, we provide guidelines for algorithm selection and data cleaning.

Motivation & Objective

• To investigate the specific impacts of missing, inconsistent, and conflicting data on classification and clustering algorithms.
• To develop new evaluation metrics that capture the fluctuation and robustness of algorithm performance under dirty data conditions.
• To provide actionable guidelines for users to select appropriate algorithms and clean data selectively based on data quality and task requirements.
• To reduce data cleaning costs by identifying the minimal data quality threshold (keeping point) required for acceptable performance.

Proposed method

• The authors selected twelve classical classification and clustering algorithms for experimental evaluation.
• They generated synthetic datasets with controlled levels of missing, inconsistent, and conflicting data across varying data sizes and error rates.
• Two novel metrics were introduced: sensibility (measuring sensitivity to data quality issues) and keeping point (the maximum error rate an algorithm can tolerate while maintaining acceptable performance).
• Performance was evaluated using standard metrics (Precision, Recall, F-measure) alongside the new metrics to assess stability and robustness.
• Experiments were conducted across multiple datasets and error types to analyze trends in algorithm behavior.
• Guidelines were derived from empirical results, linking algorithm choice and cleaning thresholds to error types and data quality levels.

Experimental results

Research questions

• RQ1 How do different types of dirty data—missing, inconsistent, and conflicting—affect the performance of classification and clustering algorithms?
• RQ2 What metrics can effectively quantify the sensitivity and robustness of machine learning algorithms to data quality issues?
• RQ3 Which algorithms are most tolerant to specific types of dirty data, and how does data size influence this tolerance?
• RQ4 What is the maximum allowable error rate (keeping point) for each algorithm to maintain acceptable performance?
• RQ5 How can users optimize data cleaning by focusing only on the most critical data quality issues?

Key findings

• Logistic Regression is the most robust classification algorithm under missing data, showing the highest keeping point and lowest sensibility.
• LVQ is the most conflict-tolerant clustering algorithm, with the highest keeping point for conflicting data, while CURE is the least tolerant due to sensitivity to conflicting values.
• DBSCAN is the most stable clustering algorithm for large datasets, maintaining consistent performance even as data size increases.
• The F-measure of most algorithms becomes unstable when data size increases, except DBSCAN, which remains robust under high data volume and dirty data conditions.
• Algorithms with F-measure > 80% on clean data show significant performance fluctuation under dirty data, especially as error rates rise.
• The keeping point metric effectively identifies the threshold error rate beyond which algorithm performance becomes unacceptable, enabling targeted data cleaning.