[Paper Review] Active Learning for Cost-Sensitive Classification

Akshay Krishnamurthy, Alekh Agarwal+3 more|arXiv (Cornell University)|Mar 3, 2017

Machine Learning and Algorithms18 citations

TL;DR

This paper proposes COAL, an active learning algorithm for cost-sensitive multiclass classification that reduces labeling effort by querying only labels with potentially low costs. By regressing to label-specific costs and focusing only on promising candidates, COAL achieves significant improvements in test cost and labeling efficiency over passive and active baselines, with strong theoretical guarantees for any regression family using squared loss optimization.

ABSTRACT

We design an active learning algorithm for cost-sensitive multiclass classification: problems where different errors have different costs. Our algorithm, COAL, makes predictions by regressing to each label’s cost and predicting the smallest. On a new example, it uses a set of regressors that perform well on past data to estimate possible costs for each label. It queries only the labels that could be the best, ignoring the sure losers. We prove COAL can be efficiently implemented for any regression family that admits squared loss optimization; it also enjoys strong guarantees with respect to predictive performance and labeling effort. Our experiment with COAL show significant improvements in labeling effort and test cost over passive and active baselines.

Motivation & Objective

• To address the challenge of reducing labeling effort in cost-sensitive multiclass classification, where misclassification costs vary across classes.
• To design an active learning framework that intelligently selects which labels to query, avoiding redundant or low-potential labels.
• To ensure strong theoretical guarantees on predictive performance and labeling efficiency under any regression family with squared loss optimization.
• To empirically demonstrate superior performance in test cost and labeling effort compared to passive and active learning baselines.

Proposed method

• COAL predicts the best label by regressing to the cost of each class and selecting the minimum predicted cost.
• It employs a set of regressors trained on past data to estimate the cost of each label for a new example.
• The algorithm limits querying to only those labels that could potentially be optimal, ignoring clearly suboptimal (sure loser) labels.
• COAL is efficiently implementable using any regression family that supports squared loss optimization.
• It leverages theoretical guarantees on predictive performance and labeling effort reduction through cost-aware active sampling.

Experimental results

Research questions

• RQ1 Can active learning be effectively adapted to cost-sensitive multiclass classification to reduce labeling effort?
• RQ2 How can we efficiently identify and query only the most promising labels, avoiding costly queries on clearly suboptimal options?
• RQ3 What theoretical guarantees can be provided for predictive performance and labeling efficiency in this setting?
• RQ4 How does COAL compare in practice to passive and active learning baselines in terms of test cost and labeling effort?

Key findings

• COAL significantly reduces labeling effort compared to passive learning and existing active learning baselines.
• The algorithm achieves lower test cost than both passive and active learning baselines, demonstrating improved predictive performance under cost-sensitive evaluation.
• COAL's efficiency is maintained across different regression families due to its compatibility with squared loss optimization.
• By focusing only on potentially optimal labels, COAL avoids querying 'sure losers,' leading to a more efficient labeling process.