[论文解读] Distribution-Independent Reliable Learning

Varun Kanade, Justin Thaler|arXiv (Cornell University)|Feb 20, 2014

Machine Learning and Algorithms参考文献 14被引用 12

一句话总结

本文引入了一元多项式逼近作为分布无关的可靠盲学习的新工具，使在强误差保证下高效学习多数函数及多数函数的析取式成为可能。通过利用阈值度与近似度之间的中间复杂度，首次提供了完全可靠学习可能严格优于标准盲学习的证据。

ABSTRACT

We study several questions in the reliable agnostic learning framework of Kalai et al. (2009), which captures learning tasks in which one type of error is costlier than others. A positive reliable classifier is one that makes no false positive errors. The goal in the positive reliable agnostic framework is to output a hypothesis with the following properties: (i) its false positive error rate is at most $ε$, (ii) its false negative error rate is at most $ε$ more than that of the best positive reliable classifier from the class. A closely related notion is fully reliable agnostic learning, which considers partial classifiers that are allowed to predict "unknown" on some inputs. The best fully reliable partial classifier is one that makes no errors and minimizes the probability of predicting "unknown", and the goal in fully reliable learning is to output a hypothesis that is almost as good as the best fully reliable partial classifier from a class. For distribution-independent learning, the best known algorithms for PAC learning typically utilize polynomial threshold representations, while the state of the art agnostic learning algorithms use point-wise polynomial approximations. We show that one-sided polynomial approximations, an intermediate notion between polynomial threshold representations and point-wise polynomial approximations, suffice for learning in the reliable agnostic settings. We then show that majorities can be fully reliably learned and disjunctions of majorities can be positive reliably learned, through constructions of appropriate one-sided polynomial approximations. Our fully reliable algorithm for majorities provides the first evidence that fully reliable learning may be strictly easier than agnostic learning. Our algorithms also satisfy strong attribute-efficiency properties, and provide smooth tradeoffs between sample complexity and running time.

研究动机与目标

• 开发在分布无关假设下高效可靠的盲学习算法。
• 解决其中一种错误类型（如假阳性）成本高于其他类型的机器学习任务。
• 通过一种新的复杂度度量——一元近似度，识别可高效实现可靠学习的概念类。
• 证明完全可靠学习在某些概念类上可能严格优于标准盲学习。

提出的方法

• 引入一元近似度作为介于阈值度与近似度之间的中间复杂度度量。
• 设计基于 $ L_1 $-风格回归与合页损失的学习算法，以处理可靠学习中不对称的错误成本。
• 为多数函数及其交集构造显式的一元多项式逼近。
• 利用对偶性与多项式逼近理论，证明关键概念类的一元近似度的界。
• 将一元逼近的概念应用于推导样本复杂度、运行时间与误差保证之间的权衡。
• 利用已知的阈值度与近似度的下界，确立该方法在 DNF 和半空间等某些类上的局限性。

实验结果

研究问题

• RQ1 一元多项式逼近是否能比现有盲学习方法实现更高效的可靠学习？
• RQ2 对于某些概念类，完全可靠学习是否严格优于标准盲学习？
• RQ3 多数函数及其交集的一元近似度是多少？与它们的近似度和阈值度相比如何？
• RQ4 能否通过一元逼近在可靠学习中实现样本复杂度与运行时间之间的平滑权衡？
• RQ5 是否存在一元近似度显著小于近似度的概念类，从而提升学习效率？

主要发现

• 多数函数可在时间 $ 2^{o(n)} $ 内被完全可靠学习，首次提供了完全可靠学习可能严格优于盲学习的证据。
• 析取式多数函数可通过一元多项式逼近实现正向可靠学习，获得强误差保证。
• 多数函数的一元近似度为 $ ilde{O}(n^{1/2}) $，远低于其近似度，从而实现高效学习。
• 对于半空间，一元近似度可能高达 $ ilde{ heta}(n) $，表明该方法在高权重重的半空间上存在局限性。
• 多项式大小的 DNF 可在时间 $ 2^{ ilde{O}( heta(n^{1/2}))} $ 内实现正向可靠学习，且样本复杂度与运行时间之间存在平滑权衡。
• 由于负向一元近似度存在 $ ilde{ heta}((n/ ext{log} n)^{2/3}) $ 的下界，该方法无法高效实现 DNF 的负向可靠学习。