[Paper Review] Robust Interactive Learning

Maria-Florina Balcan, Steve Hanneke|arXiv (Cornell University)|Nov 6, 2011

Machine Learning and Algorithms2 references3 citations

TL;DR

This paper introduces and analyzes class-conditional queries—where an algorithm requests an example of a specific label from a given set—as a generalization of standard active learning. It provides nearly tight bounds on query complexity in both agnostic and bounded noise settings, showing reductions by a factor related to the noise rate, and demonstrates adaptivity to unknown noise rates with minimal cost.

ABSTRACT

In this paper we propose and study a generalization of the standard active-learning model where a more general type of query, class conditional query, is allowed. Such queries have been quite useful in applications, but have been lacking theoretical understanding. In this work, we characterize the power of such queries under two well-known noise models. We give nearly tight upper and lower bounds on the number of queries needed to learn both for the general agnostic setting and for the bounded noise model. We further show that our methods can be made adaptive to the (unknown) noise rate, with only negligible loss in query complexity.

Motivation & Objective

• To formalize and analyze the power of class-conditional queries in active learning under noisy data.
• To characterize the query complexity of learning in the agnostic and bounded noise models using this query type.
• To show that the method can adapt to unknown noise rates with negligible loss in efficiency.
• To establish theoretical foundations for interactive learning models that go beyond standard label queries.

Proposed method

• Proposes a query model allowing requests for any example with a specified label within a subset of unlabeled data.
• Analyzes query complexity under two noise models: general agnostic and bounded noise.
• Uses the Natarajan dimension to characterize concept class complexity.
• Applies the splitting index framework (Dasgupta, 2005) to relate query complexity to the informativeness of examples.
• Develops algorithms that use Find-Mistake oracles to identify misclassified examples efficiently.
• Establishes upper and lower bounds via reductions to known results in active learning.

Experimental results

Research questions

• RQ1 How does the query complexity of class-conditional queries compare to standard active learning under noise?
• RQ2 What is the theoretical power of class-conditional queries in the agnostic and bounded noise settings?
• RQ3 Can algorithms using class-conditional queries adapt to unknown noise rates without significant loss in efficiency?
• RQ4 What is the minimal number of queries required to achieve a given error rate in the presence of noise?

Key findings

• In the agnostic setting with noise rate η, the query complexity is Ω(dη²/ǫ²) and ˜O(dη²/ǫ²), representing a reduction by a factor of η compared to passive learning.
• In the bounded noise model, the query complexity is reduced by a factor proportional to the noise bound α compared to standard active learning.
• The proposed methods are adaptive to unknown noise rates with only negligible loss in query complexity.
• For interval classifiers on [0,1], the query complexity of class-conditional queries is Ω(α/ǫ), matching the lower bound up to constants.
• Theoretical bounds show that class-conditional queries offer consistent improvements over active learning, particularly in noisy settings.
• The results extend to related query types, such as sample-based equivalence queries, with only a factor of k loss.