[论文解读] The Relationship Between Agnostic Selective Classification Active Learning and the Disagreement Coefficient

Roei Gelbhart, Ran El‐Yaniv|arXiv (Cornell University)|Mar 19, 2017

Machine Learning and Algorithms被引用 6

一句话总结

本文提出 ILESS，一种新颖的逐点竞争性选择（PCS）学习算法，在不依赖伯恩斯坦性质等限制性假设的前提下，实现了在全知设置下的快速拒绝率。它建立了快速拒绝率、多对数分歧系数与新型基于分歧的主动学习算法 Active-ILESS 中指数级加速之间的等价关系，统一了选择分类与主动学习在分歧系数框架下的理论洞见。

ABSTRACT

A selective classifier (f,g) comprises a classification function f and a binary selection function g, which determines if the classifier abstains from prediction, or uses f to predict. The classifier is called pointwise-competitive if it classifies each point identically to the best classifier in hindsight (from the same class), whenever it does not abstain. The quality of such a classifier is quantified by its rejection mass, defined to be the probability mass of the points it rejects. A "fast" rejection rate is achieved if the rejection mass is bounded from above by O(1/m) where m is the number of labeled examples used to train the classifier (and O hides logarithmic factors). Pointwise-competitive selective (PCS) classifiers are intimately related to disagreement-based active learning and it is known that in the realizable case, a fast rejection rate of a known PCS algorithm (called Consistent Selective Strategy) is equivalent to an exponential speedup of the well-known CAL active algorithm. We focus on the agnostic setting, for which there is a known algorithm called LESS that learns a PCS classifier and achieves a fast rejection rate (depending on Hanneke's disagreement coefficient) under strong assumptions. We present an improved PCS learning algorithm called ILESS for which we show a fast rate (depending on Hanneke's disagreement coefficient) without any assumptions. Our rejection bound smoothly interpolates the realizable and agnostic settings. The main result of this paper is an equivalence between the following three entities: (i) the existence of a fast rejection rate for any PCS learning algorithm (such as ILESS); (ii) a poly-logarithmic bound for Hanneke's disagreement coefficient; and (iii) an exponential speedup for a new disagreement-based active learner called ActiveiLESS.

研究动机与目标

• 消除全知选择分类中对强假设（如伯恩斯坦性质）的依赖。
• 在无先验分布约束的全知设置下，为逐点竞争性选择学习建立快速拒绝率。
• 通过分歧系数框架统一选择分类与主动学习。
• 证明快速拒绝、有界分歧系数与主动学习中标注复杂度指数级降低之间的等价性。

提出的方法

• 提出 ILESS，一种新型逐点竞争性选择分类算法，通过基于分歧的采样自适应选择信息丰富的样本。
• 采用基于批次的主动学习策略（Batch-ILESS），模拟 Active-ILESS 以控制误差与拒绝率。
• 利用基于 VC 维与对数因子的置信半径界，控制选择分类器的泛化误差。
• 应用集中不等式（如切尔诺夫不等式）以在分布变化下界 empirical risk 与真实风险之间的偏差。
• 通过 $\mathcal{P_{X,Y}}(G_{t-1})$ 引入分布漂移机制，以在迭代过程中保持真实风险最小化器 $f^*$ 的一致性。
• 建立三个核心要素之间的等价关系：快速拒绝率、多对数分歧系数与主动学习中的指数级加速。

实验结果

研究问题

• RQ1 在不假设伯恩斯坦性质的前提下，逐点竞争性选择分类器能否在全知设置下实现快速拒绝率？
• RQ2 选择分类中的快速拒绝与主动学习中标注复杂度的指数级降低之间是否存在直接等价关系？
• RQ3 分歧系数如何影响全知情况下选择学习与主动学习的样本复杂度？
• RQ4 能否仅通过分歧系数来界定 PCS 算法的拒绝率，而无需额外的分布假设？
• RQ5 分歧系数与基于分歧的主动学习算法性能之间存在何种关系？

主要发现

• ILESS 在全知设置下实现了由 $ \tilde{O}(1/m) $ 有界的快速拒绝率，且无需依赖伯恩斯坦性质。
• ILESS 的拒绝率可根据分歧系数在可实现情况与全知情况之间平滑插值。
• 任何 PCS 算法的快速拒绝率等价于 Hanneke 分歧系数的多对数有界性。
• 当且仅当分歧系数为多对数有界时，Active-ILESS 才能实现标注复杂度的指数级加速。
• Batch-ILESS 的拒绝率由 $ \theta_{f^*}(R_0) \cdot R_0 $ 有界，其中 $ R_0 = 2R(f^*) + 44 \cdot \frac{B}{m} + 12 \cdot \sqrt{\frac{B}{m} \cdot R(f^*)} $，且 $ B = 4d\ln(\frac{8m^2e}{d\delta}) $。
• 即使在全知设置下，分歧系数也能完全刻画选择分类与主动学习之间样本复杂度的权衡。