[论文解读] BoostClean: Automated Error Detection and Repair for Machine Learning

Sanjay Krishnan, Michael J. Franklin+2 more|arXiv (Cornell University)|Nov 3, 2017

Machine Learning and Data Classification参考文献 41被引用 58

一句话总结

BoostClean 自动检测训练数据和测试数据中的领域值违规，并使用提升来选择检测器和修复的集合，从而提升下游模型的准确性，达到高达 9% 的绝对提升并实现显著的加速。

ABSTRACT

Predictive models based on machine learning can be highly sensitive to data error. Training data are often combined with a variety of different sources, each susceptible to different types of inconsistencies, and new data streams during prediction time, the model may encounter previously unseen inconsistencies. An important class of such inconsistencies is domain value violations that occur when an attribute value is outside of an allowed domain. We explore automatically detecting and repairing such violations by leveraging the often available clean test labels to determine whether a given detection and repair combination will improve model accuracy. We present BoostClean which automatically selects an ensemble of error detection and repair combinations using statistical boosting. BoostClean selects this ensemble from an extensible library that is pre-populated general detection functions, including a novel detector based on the Word2Vec deep learning model, which detects errors across a diverse set of domains. Our evaluation on a collection of 12 datasets from Kaggle, the UCI repository, real-world data analyses, and production datasets that show that Boost- Clean can increase absolute prediction accuracy by up to 9% over the best non-ensembled alternatives. Our optimizations including parallelism, materialization, and indexing techniques show a 22.2x end-to-end speedup on a 16-core machine.

研究动机与目标

• 通过聚焦当数值超出允许取值域时发生的领域值违规来解决机器学习流水线中普遍存在的脏数据问题。
• 使用包含 Word2Vec 基于检测器的库对脏记录进行自动检测。
• 通过提升自动选择修复集合，以在未见测试数据上最大化预测准确性。
• 在多样化数据集上评估 BoostClean，以量化准确性提升和运行时改进。
• 提供一个可扩展的框架，包含预填充的检测器和修复，并允许领域特定的定制。

提出的方法

• 将检测器表示为标记候选脏记录的谓词，并通过有条件的修复规则将它们映射到修复。
• 使用基于提升的过程来选择一系列有条件的修复，以在带权数据上最大化测试准确性。
• 通过在训练前应用修复将清理视为特征生成，并使用用清理后数据训练的提升集成分类器。
• 整合 IsoDetect，它通过 Isolation Forests 学到的阈值将特征化转化为检测器生成器。
• 包含一种新颖的基于 Word2Vec 的检测器，以捕捉跨域的多属性错误模式。
• 提供一个端到端工作流，包含检测、修复、训练和部署组件。

实验结果

研究问题

• RQ1 Can BoostClean reliably identify and repair domain value violations that harm classifier accuracy on unseen test data?
• RQ2 How much predictive performance gain can be achieved by boosting an ensemble of detectors and repairs compared to non-ensembled baselines?
• RQ3 What is the runtime performance of BoostClean and its scalability on multi-core hardware?
• RQ4 How effective is the Word2Vec-based detector and the IsoDetect library in broad settings?
• RQ5 To what extent can pre-populated detector/repair libraries generalize across datasets?

主要发现

• BoostClean can increase absolute prediction accuracy by up to 9% over the best non-ensembled alternatives on unseen test data.
• Optimizations such as parallelism, materialization, and indexing yield a 22.2x end-to-end speedup on a 16-core machine.
• An extensible detector/repair library and boosting-based selection effectively identify and apply conditional repairs.
• The Word2Vec-based detector detects errors across diverse domains and achieves high detection coverage compared to hand-written rules on multiple datasets.
• IsoDetect efficiently converts featurizations into detector generators using Isolation Forests, enabling scalable outlier-based detection.