[Paper Review] Active classification with comparison queries

Daniel M. Kane, Shachar Lovett+2 more|arXiv (Cornell University)|Apr 11, 2017

Machine Learning and Algorithms6 references3 citations

TL;DR

This paper introduces active learning with comparison queries—where the learner can ask which of two examples is closer to the decision boundary—demonstrating that for half-spaces with large margin or bounded bit-description, only O(log n) queries are needed to label n examples, achieving exponential improvement over standard active learning. The key contribution is a new combinatorial measure, inference dimension, which characterizes query complexity under such queries.

ABSTRACT

We study an extension of active learning in which the learning algorithm may ask the annotator to compare the distances of two examples from the boundary of their label-class. For example, in a recommendation system application (say for restaurants), the annotator may be asked whether she liked or disliked a specific restaurant (a label query); or which one of two restaurants did she like more (a comparison query). We focus on the class of half spaces, and show that under natural assumptions, such as large margin or bounded bit-description of the input examples, it is possible to reveal all the labels of a sample of size $n$ using approximately $O(\log n)$ queries. This implies an exponential improvement over classical active learning, where only label queries are allowed. We complement these results by showing that if any of these assumptions is removed then, in the worst case, $Ω(n)$ queries are required. Our results follow from a new general framework of active learning with additional queries. We identify a combinatorial dimension, called the \emph{inference dimension}, that captures the query complexity when each additional query is determined by $O(1)$ examples (such as comparison queries, each of which is determined by the two compared examples). Our results for half spaces follow by bounding the inference dimension in the cases discussed above.

Motivation & Objective

• To investigate whether comparison queries—asking which of two examples is closer to the decision boundary—can significantly reduce query complexity in active learning.
• To identify conditions under which comparison queries yield exponential improvements over standard active learning with only label queries.
• To develop a general theoretical framework for active learning with additional query types, particularly those determined by a constant number of examples.
• To characterize the query complexity of learning half-spaces using comparison queries under natural assumptions such as large margin or bounded input description length.
• To establish tight bounds by showing that without such assumptions, Ω(n) queries are necessary in the worst case.

Proposed method

• Introduce a new combinatorial measure called inference dimension to characterize the query complexity of active learning when additional queries (like comparisons) are allowed.
• Define inference dimension as the maximum size of a set S such that no subset of S can infer the label of a point in S using only label or comparison queries.
• Prove that for concept classes with inference dimension d, O(d) queries suffice to label any realizable sample of size n.
• Establish a connection between inference dimension and geometric properties such as margin and bit-description complexity of inputs.
• Use explicit constructions of functions and vectors to prove lower bounds on inference dimension under various assumptions.
• Leverage linear algebra and functional analysis to construct families of functions that are indistinguishable under comparison and label queries on all but one point.

Experimental results

Research questions

• RQ1 Can comparison queries—asking which of two examples is closer to the decision boundary—reduce the number of queries needed to label a dataset in active learning?
• RQ2 Under what conditions does the use of comparison queries lead to an exponential reduction in query complexity compared to standard active learning?
• RQ3 Is there a general theoretical framework that captures the query complexity of active learning when additional query types are available?
• RQ4 What is the worst-case query complexity of learning half-spaces when only comparison and label queries are allowed, without any structural assumptions?
• RQ5 Can the inference dimension framework be used to derive tight upper and lower bounds on query complexity for specific concept classes?

Key findings

• For half-spaces with a large margin (at least 1/8), the query complexity is still Ω(n) in the worst case, showing that margin alone does not enable exponential savings with comparison queries.
• When inputs have bounded bit-description length, the query complexity is O(log n), achieving an exponential improvement over standard active learning.
• The inference dimension of a concept class is an upper bound on the number of queries needed to label any realizable sample of size n.
• There exist concept classes with large margin and inference dimension Ω(n), proving that margin alone does not reduce query complexity in the worst case.
• Without any structural assumptions, such as margin or bounded description length, Ω(n) queries are required in the worst case, even with comparison queries.
• The inference dimension framework provides a tight characterization of query complexity for active learning with comparison queries, especially when queries are determined by a constant number of examples.