[Paper Review] BoostClean: Automated Error Detection and Repair for Machine Learning

Sanjay Krishnan, Michael J. Franklin+2 more|arXiv (Cornell University)|Nov 3, 2017

Machine Learning and Data Classification41 references58 citations

TL;DR

BoostClean automatically detects domain value violations in training and test data and uses boosting to select an ensemble of detectors and repairs that improve downstream model accuracy, achieving up to 9% absolute improvement and substantial speedups.

ABSTRACT

Predictive models based on machine learning can be highly sensitive to data error. Training data are often combined with a variety of different sources, each susceptible to different types of inconsistencies, and new data streams during prediction time, the model may encounter previously unseen inconsistencies. An important class of such inconsistencies is domain value violations that occur when an attribute value is outside of an allowed domain. We explore automatically detecting and repairing such violations by leveraging the often available clean test labels to determine whether a given detection and repair combination will improve model accuracy. We present BoostClean which automatically selects an ensemble of error detection and repair combinations using statistical boosting. BoostClean selects this ensemble from an extensible library that is pre-populated general detection functions, including a novel detector based on the Word2Vec deep learning model, which detects errors across a diverse set of domains. Our evaluation on a collection of 12 datasets from Kaggle, the UCI repository, real-world data analyses, and production datasets that show that Boost- Clean can increase absolute prediction accuracy by up to 9% over the best non-ensembled alternatives. Our optimizations including parallelism, materialization, and indexing techniques show a 22.2x end-to-end speedup on a 16-core machine.

Motivation & Objective

• Address pervasive dirty data in ML pipelines by focusing on domain value violations that occur when values lie outside their allowed domains.
• Automatic detection of dirty records using a library of detectors and featurizers including a Word2Vec-based detector.
• Automatically selecting an ensemble of repairs via boosting to maximize predictive accuracy on unseen test data.
• Evaluate BoostClean on diverse datasets to quantify accuracy gains and runtime improvements.
• Provide an extensible framework with pre-populated detectors and repairs while allowing domain-specific customization.

Proposed method

• Represent detectors as predicates that mark candidate dirty records and map them to repairs via conditional repair rules.
• Use a boosting-based procedure to select a sequence of conditional repairs that maximize test accuracy on weighted data.
• Treat cleaning as feature generation by applying repairs before training and use boosted ensemble of classifiers trained with cleaned data.
• Incorporate IsoDetect, which converts featurizations into detector generators using learned thresholds via Isolation Forests.
• Include a novel Word2Vec-based detector to capture multi-attribute error patterns across domains.
• Provide an end-to-end workflow with detection, repair, training, and deployment components.

Experimental results

Research questions

• RQ1 Can BoostClean reliably identify and repair domain value violations that harm classifier accuracy on unseen test data?
• RQ2 How much predictive performance gain can be achieved by boosting an ensemble of detectors and repairs compared to non-ensembled baselines?
• RQ3 What is the runtime performance of BoostClean and its scalability on multi-core hardware?
• RQ4 How effective is the Word2Vec-based detector and the IsoDetect library in broad settings?
• RQ5 To what extent can pre-populated detector/repair libraries generalize across datasets?

Key findings

• BoostClean can increase absolute prediction accuracy by up to 9% over the best non-ensembled alternatives on unseen test data.
• Optimizations such as parallelism, materialization, and indexing yield a 22.2x end-to-end speedup on a 16-core machine.
• An extensible detector/repair library and boosting-based selection effectively identify and apply conditional repairs.
• The Word2Vec-based detector detects errors across diverse domains and achieves high detection coverage compared to hand-written rules on multiple datasets.
• IsoDetect efficiently converts featurizations into detector generators using Isolation Forests, enabling scalable outlier-based detection.