[Paper Review] Auditing: Active Learning with Outcome-Dependent Query Costs

Sivan Sabato, Anand D. Sarwate+1 more|arXiv (Cornell University)|Jun 10, 2013

Machine Learning and Algorithms24 references3 citations

TL;DR

This paper introduces 'auditing'—a novel active learning framework where labeling costs depend on the outcome, specifically incurring cost only for negative labels. The authors design algorithms for thresholds and rectangles that achieve significantly lower auditing complexity than standard active learning, demonstrating that outcome-dependent costs enable more efficient learning in scenarios like fraud detection.

ABSTRACT

We propose a learning setting in which unlabeled data is free, and the cost of a label depends on its value, which is not known in advance. We study binary classification in an extreme case, where the algorithm only pays for negative labels. Our motivation are applications such as fraud detection, in which investigating an honest transaction should be avoided if possible. We term the setting auditing, and consider the auditing complexity of an algorithm: the number of negative labels the algorithm requires in order to learn a hypothesis with low relative error. We design auditing algorithms for simple hypothesis classes (thresholds and rectangles), and show that with these algorithms, the auditing complexity can be significantly lower than the active label complexity. We also discuss a general competitive approach for auditing and possible modifications to the framework.

Motivation & Objective

• Address the challenge of labeling cost in machine learning where the cost depends on the label outcome, particularly when negative labels are expensive.
• Motivate the framework using real-world applications such as fraud detection, where investigating honest (negative) transactions incurs avoidable costs.
• Propose a new learning setting—'auditing'—where only negative labels incur cost, and define auditing complexity as the number of such labels required.
• Analyze and compare auditing complexity to standard active learning label complexity, showing potential for significant reductions in cost.
• Design and evaluate auditing algorithms for simple hypothesis classes (thresholds and rectangles), and propose a general competitive analysis framework.

Proposed method

• Define the auditing setting where unlabeled data is free, and only negative labels incur cost, modeling scenarios like fraud detection.
• Introduce auditing complexity as the number of negative labels an algorithm must query to achieve low error relative to the best hypothesis.
• Design specific auditing algorithms for threshold and rectangle hypothesis classes using active learning principles adapted to outcome-dependent costs.
• Use competitive analysis to bound the auditing complexity of a greedy algorithm relative to the optimal algorithm, proving a logarithmic approximation factor.
• Leverage version space reduction and cost-aware query selection to minimize negative label queries while maintaining hypothesis accuracy.
• Prove that the greedy auditing algorithm achieves an auditing cost within a factor of $ (\ln(|\mathcal{H}|-1) + 1) \cdot \mathrm{OPT} $ of the optimal cost, where $ \mathrm{OPT} $ is the minimal possible auditing cost.

Experimental results

Research questions

• RQ1 Can auditing complexity be significantly lower than standard active learning label complexity in settings where only negative labels incur cost?
• RQ2 Do optimal active learning algorithms inherently have low auditing complexity, or are specialized auditing algorithms required?
• RQ3 How can a general competitive analysis framework be constructed for auditing in arbitrary hypothesis classes?
• RQ4 What is the theoretical worst-case performance guarantee for a greedy auditing algorithm compared to the optimal algorithm?
• RQ5 To what extent can hypothesis class structure (e.g., thresholds, rectangles) be exploited to minimize negative label queries?

Key findings

• For threshold and rectangle hypothesis classes, the proposed auditing algorithms achieve auditing complexity that is significantly lower than standard active learning label complexity.
• The greedy auditing algorithm achieves an auditing cost within a factor of $ \ln(|\mathcal{H}|-1) + 1 $ of the optimal cost, proving a logarithmic competitive ratio.
• The auditing complexity can be substantially reduced in practice, especially when the fraction of positive labels is small, due to the avoidance of negative label queries.
• The framework demonstrates that outcome-dependent costs necessitate a rethinking of active learning strategies, as standard algorithms may not minimize auditing cost.
• Theoretical bounds show that even in the worst case, the greedy auditing algorithm performs within a logarithmic factor of the optimal strategy.
• The results suggest that auditing is not just a variant of active learning but a distinct setting requiring specialized algorithms to minimize cost under outcome-dependent labeling.