[Paper Review] A Compression Technique for Analyzing Disagreement-Based Active Learning

Yair Wiener, Steve Hanneke+1 more|arXiv (Cornell University)|Apr 5, 2014

Machine Learning and Algorithms39 references3 citations

TL;DR

This paper introduces a new characterization of label complexity in disagreement-based active learning using the version space compression set size—the smallest subset of training data that induces the same version space as the full dataset. The approach provides tighter, logarithmically optimal bounds than prior methods and simplifies analysis by eliminating the need for characterizing set complexity, with applications to linear separators under mixtures of Gaussians and axis-aligned rectangles under product densities.

ABSTRACT

We introduce a new and improved characterization of the label complexity of disagreement-based active learning, in which the leading quantity is the version space compression set size. This quantity is defined as the size of the smallest subset of the training data that induces the same version space. We show various applications of the new characterization, including a tight analysis of CAL and refined label complexity bounds for linear separators under mixtures of Gaussians and axis-aligned rectangles under product densities. The version space compression set size, as well as the new characterization of the label complexity, can be naturally extended to agnostic learning problems, for which we show new speedup results for two well known active learning algorithms.

Motivation & Objective

• To develop a new, simplified characterization of label complexity in disagreement-based active learning.
• To replace the reliance on characterizing set complexity with a more natural and tighter measure: the version space compression set size.
• To derive improved, nearly optimal label complexity bounds for specific learning problems such as linear separators and axis-aligned rectangles.
• To extend the analysis to agnostic and noisy learning settings, demonstrating broader applicability.
• To show that the new characterization improves upon existing bounds by reducing the dependence on VC dimension.

Proposed method

• Define the version space compression set size as the minimal subset of labeled data that preserves the same version space as the full dataset.
• Use this measure as the leading complexity term in a new characterization of label complexity, replacing the disagreement coefficient in some analyses.
• Prove that the version space compression set size and the disagreement coefficient are within a factor of the VC dimension of each other, up to logarithmic factors.
• Apply the new characterization to derive tighter label complexity bounds for linear separators under mixtures of Gaussians and axis-aligned rectangles under product densities.
• Extend the framework to agnostic learning by relating the disagreement coefficient to the version space compression set size, enabling analysis in noisy settings.
• Demonstrate that the new method simplifies prior analyses by eliminating the need for characterizing set complexity, a key technical innovation.

Experimental results

Research questions

• RQ1 Can the label complexity of disagreement-based active learning be characterized more tightly using a single, interpretable complexity measure?
• RQ2 How does the version space compression set size compare to the disagreement coefficient and other existing complexity measures?
• RQ3 Can the new characterization yield improved, nearly optimal label complexity bounds for specific learning problems?
• RQ4 Is the version space compression set size applicable and useful in agnostic, noisy learning settings?
• RQ5 Can the new framework simplify prior theoretical analyses that relied on multiple complexity measures?

Key findings

• The new characterization achieves label complexity bounds that are tight up to logarithmic factors in the realizable case, unlike prior methods that could be off by a factor of the VC dimension.
• The version space compression set size is shown to be within a factor of the VC dimension (up to logarithmic factors) of the disagreement coefficient, establishing a strong theoretical link between the two measures.
• For linear separators under mixtures of Gaussians, the paper derives new, refined label complexity bounds using the version space compression set size, improving upon prior results.
• For axis-aligned rectangles under product densities, the paper provides tighter label complexity bounds than previously known, again through the new characterization.
• The framework is successfully extended to agnostic learning, enabling new speedup results for well-known active learning algorithms in noisy settings.
• The method eliminates the need for characterizing set complexity, significantly simplifying the theoretical analysis of disagreement-based active learning.