[Paper Review] Data Smells: Categories, Causes and Consequences, and Detection of Suspicious Data in AI-based Systems

Harald Foidl, Michael Felderer+1 more|arXiv (Cornell University)|Mar 19, 2022

Software Engineering Research4 citations

TL;DR

This paper introduces 'data smells'—latent, context-independent data quality issues resembling code smells—that signal potential problems in AI-based systems. It proposes a catalogue of 36 data smells across three categories (Believability, Understandability, Consistency), outlines detection methods using rule-based and ML-based approaches, and demonstrates through an evaluation on 246 real-world datasets that data smells are detectable and manageable in practice, enabling proactive data quality assurance and reducing technical debt.

ABSTRACT

High data quality is fundamental for today's AI-based systems. However, although data quality has been an object of research for decades, there is a clear lack of research on potential data quality issues (e.g., ambiguous, extraneous values). These kinds of issues are latent in nature and thus often not obvious. Nevertheless, they can be associated with an increased risk of future problems in AI-based systems (e.g., technical debt, data-induced faults). As a counterpart to code smells in software engineering, we refer to such issues as Data Smells. This article conceptualizes data smells and elaborates on their causes, consequences, detection, and use in the context of AI-based systems. In addition, a catalogue of 36 data smells divided into three categories (i.e., Believability Smells, Understandability Smells, Consistency Smells) is presented. Moreover, the article outlines tool support for detecting data smells and presents the result of an initial smell detection on more than 240 real-world datasets.

Motivation & Objective

• To address the lack of research on latent data quality issues in AI-based systems that are not obvious but can lead to future faults.
• To conceptualize data smells by analogy to code smells, defining their characteristics, causes, and consequences in the context of AI systems.
• To develop a comprehensive catalogue of 36 data smells categorized into Believability, Understandability, and Consistency Smells.
• To propose and evaluate detection techniques—rule-based and ML-based—that are decoupled from traditional data validation for early identification of problematic data.
• To demonstrate the practical applicability of data smell detection through an experimental evaluation on 246 real-world datasets from Kaggle.

Proposed method

• Proposes a conceptual framework for data smells based on analogy to code smells, emphasizing their role as early indicators of latent data quality problems.
• Categorizes 36 data smells into three types: Believability Smells (e.g., implausible values), Understandability Smells (e.g., ambiguous labels), and Consistency Smells (e.g., conflicting data representations).
• Develops a rule-based detection approach using predefined heuristics to identify suspicious data patterns without requiring domain-specific constraints.
• Designs a machine learning-based detection method trained on synthetic and real data to identify complex, non-obvious data smell patterns.
• Implements both detection approaches in tool support to enable integration into data engineering pipelines and validate on 246 real-world datasets from Kaggle.
• Uses data smell density as a metric to prioritize data streams in high-volume environments, guiding targeted validation efforts.

Experimental results

Research questions

• RQ1 What are the defining characteristics, causes, and consequences of data smells in AI-based systems?
• RQ2 How can data smells be systematically categorized and distinguished from actual data errors?
• RQ3 What are effective, context-independent detection techniques for data smells that support early identification in the AI development lifecycle?
• RQ4 How feasible is the detection of data smells in real-world datasets, and what is the scale of detected issues?
• RQ5 In what ways can data smell detection improve data quality assurance and reduce technical debt in AI systems?

Key findings

• The experimental evaluation on 246 real-world datasets from Kaggle revealed that data smells are detectable and occur at a manageable scale, indicating practical feasibility.
• The rule-based detection approach successfully identified suspicious data patterns without requiring domain-specific constraints, enabling broad applicability.
• The ML-based detection method demonstrated effectiveness in uncovering complex, non-obvious data smell patterns that may evade traditional validation.
• Data smell density emerged as a useful metric for prioritizing data validation efforts in systems with high data volume and multiple data streams.
• Data smell detection can uncover subtle data handling flaws that bypass traditional quality assurance, reducing the risk of model degradation and data-induced bugs.
• The application of data smell detection in production systems can identify previously undetected data issues, enabling proactive root cause analysis and validation rule refinement.