[Paper Review] Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective

Steven Euijong Whang, Yuji Roh+2 more|arXiv (Cornell University)|Dec 13, 2021

Data Quality and Management40 citations

TL;DR

This survey reviews data collection, validation, cleaning, sanitization, robust training, and fairness techniques for deep learning from a data-centric AI perspective, emphasizing data as a first-class citizen and the role of data quality in model performance.

ABSTRACT

Data-centric AI is at the center of a fundamental shift in software engineering where machine learning becomes the new software, powered by big data and computing infrastructure. Here software engineering needs to be re-thought where data becomes a first-class citizen on par with code. One striking observation is that a significant portion of the machine learning process is spent on data preparation. Without good data, even the best machine learning algorithms cannot perform well. As a result, data-centric AI practices are now becoming mainstream. Unfortunately, many datasets in the real world are small, dirty, biased, and even poisoned. In this survey, we study the research landscape for data collection and data quality primarily for deep learning applications. Data collection is important because there is lesser need for feature engineering for recent deep learning approaches, but instead more need for large amounts of data. For data quality, we study data validation, cleaning, and integration techniques. Even if the data cannot be fully cleaned, we can still cope with imperfect data during model training using robust model training techniques. In addition, while bias and fairness have been less studied in traditional data management research, these issues become essential topics in modern machine learning applications. We thus study fairness measures and unfairness mitigation techniques that can be applied before, during, or after model training. We believe that the data management community is well poised to solve these problems.

Motivation & Objective

• Motivate the shift to data-centric AI where data quality critically influences model performance.
• Survey data collection techniques, including acquisition, labeling, and improving existing data for deep learning.
• Survey data validation, cleaning, and integration methods and their impact on robustness and accuracy.
• Discuss robust training techniques to cope with noisy, poisoned, or biased data.
• Highlight fairness measures and mitigation techniques applicable before, during, or after model training.

Proposed method

• Classify data-centric AI topics and map techniques to data types and workflow (data collection, validation/cleaning/integration, robust training, fairness).
• Summarize representative and influential techniques across data acquisition, labeling, validation, cleaning, sanitization, and mitigation methods.
• Present a taxonomy (Table 1) and a workflow decision tree linking techniques across the data-centric AI lifecycle.
• Discuss practical systems and frameworks (e.g., TF Data Validation, SeeDB, ActiveClean) and notable methods (GANs, Mixup, data programming).
• Draw connections between data management and ML communities to address bias, robustness, and ethics in AI.

Experimental results

Research questions

• RQ1 What data collection strategies best support deep learning when feature engineering is less central?
• RQ2 How can data validation, cleaning, and integration improve downstream model accuracy and robustness in deep learning?
• RQ3 What robust and fair training techniques are effective for handling noisy, poisoned, or biased data in supervised learning?
• RQ4 How should data-centric AI teams orchestrate data quality practices across acquisition, labeling, cleaning, sanitization, and mitigation?
• RQ5 What are the key open challenges and future directions at the intersection of data management and AI ethics in deep learning?

Key findings

• Data collection is foundational for deep learning performance, with data discovery, augmentation, and generation as core approaches.
• Data labeling can leverage existing labels, manual labeling, semi-supervised learning, crowdsourcing, and weak supervision, including data programming.
• Data validation and visualization aid human-in-the-loop checks, with schema-based validation and automated anomaly detection gaining traction.
• Data cleaning and data sanitization can improve or sometimes hurt model accuracy; selecting appropriate cleaning strategies and considering robustness is crucial.
• Fairness and bias measures are essential and can be integrated before, during, or after training; mitigation techniques span pre-processing, in-processing, and post-processing.
• There is a need for holistic orchestration of data-centric AI techniques across the entire ML lifecycle to address robustness, fairness, and data quality.