[Paper Review] Active Learning for Binary Classification with Abstention

Shubhanshu Shekhar, Mohammad Ghavamzadeh+1 more|arXiv (Cornell University)|Jun 1, 2019

Machine Learning and Algorithms32 references6 citations

TL;DR

This paper proposes active learning algorithms for binary classification with abstention across three settings—fixed-cost and two bounded-rate variants—using a non-parametric framework. It establishes minimax near-optimal excess risk bounds and introduces an adaptive strategy for unknown smoothness parameters, with a computationally efficient variant for high-dimensional problems.

ABSTRACT

We construct and analyze active learning algorithms for the problem of binary classification with abstention. We consider three abstention settings: \emph{fixed-cost} and two variants of \emph{bounded-rate} abstention, and for each of them propose an active learning algorithm. All the proposed algorithms can work in the most commonly used active learning models, i.e., \emph{membership-query}, \emph{pool-based}, and \emph{stream-based} sampling. We obtain upper-bounds on the excess risk of our algorithms in a general non-parametric framework and establish their minimax near-optimality by deriving matching lower-bounds. Since our algorithms rely on the knowledge of some smoothness parameters of the regression function, we then describe a new strategy to adapt to these unknown parameters in a data-driven manner. Since the worst case computational complexity of our proposed algorithms increases exponentially with the dimension of the input space, we conclude the paper with a computationally efficient variant of our algorithm whose computational complexity has a polynomial dependence over a smaller but rich class of learning problems.

Motivation & Objective

• Address the lack of active learning methods for binary classification with abstention, a setting where models can defer decisions to reduce error costs.
• Design algorithms that work across three abstention settings: fixed-cost, bounded-rate with known marginal, and bounded-rate with unknown marginal.
• Establish theoretical guarantees by deriving upper and matching lower bounds on excess risk, proving minimax near-optimality.
• Develop a data-driven adaptive strategy to estimate unknown smoothness parameters of the regression function without prior knowledge.
• Propose a computationally efficient variant with polynomial complexity for high-dimensional problems, maintaining strong performance on a rich class of learning tasks.

Proposed method

• Formulate the problem using a regression function $\eta(x) = P(Y=1|X=x)$ and define abstaining classifiers that map inputs to label distributions including a $\Delta$ (abstain) option.
• Design active learning algorithms for three settings: fixed-cost (known $\lambda$), bounded-rate with known $P_X$, and bounded-rate with unknown $P_X$, using pool-, stream-, and membership-query models.
• Use a non-parametric framework with smoothness assumptions on $\eta(x)$, including Hölder-type regularity and bounded density for $P_X$, to derive excess risk bounds.
• Introduce a novel adaptive scheme that estimates unknown smoothness parameters via data-driven selection of bandwidths and thresholds, improving robustness.
• Apply a noisy binary search strategy to refine uncertainty regions around decision thresholds, reducing the number of required labels.
• Derive a computationally efficient variant by restricting the search space to a low-dimensional manifold or structured input space, reducing complexity from exponential to polynomial in dimension $D$.

Experimental results

Research questions

• RQ1 Can active learning strategies for binary classification with abstention achieve minimax optimal excess risk in non-parametric settings?
• RQ2 How does the performance of active learning with abstention compare to passive learning in terms of excess risk under fixed-cost and bounded-rate settings?
• RQ3 What is the impact of unknown smoothness parameters on algorithm performance, and can they be adaptively estimated from data?
• RQ4 How does the ambient dimension $D$ affect the computational complexity of active learning with abstention, and can this be mitigated?
• RQ5 Can a computationally efficient variant be designed that maintains strong theoretical guarantees while reducing exponential dependence on $D$?

Key findings

• The proposed active learning algorithms achieve upper bounds on excess risk that are minimax near-optimal, matching the lower bounds derived for both fixed-cost and bounded-rate settings.
• In the fixed-cost and bounded-rate settings, the excess risk scales as $O(n^{-\frac{\alpha_0 \beta}{2\alpha_0 \beta + D}})$ under smoothness and density assumptions, where $\alpha_0$ and $\beta$ are smoothness parameters.
• The adaptive strategy successfully estimates unknown smoothness parameters without prior knowledge, maintaining the same excess risk rate as the non-adaptive version.
• The worst-case computational complexity of the main algorithms grows exponentially with input dimension $D$, but this is mitigated in a restricted class of problems via a polynomial-time variant.
• The Detectability (DE) assumption is necessary to avoid large unclassified regions; without it, the algorithm’s convergence rate degrades due to poor marginal density near decision thresholds.
• Under the additional assumption that $P_X$ has a bounded density below, the effective dimension in the risk bound reduces to $\max\{0, D - \beta \alpha_0\}$, improving sample efficiency in low-dimensional manifolds.