[论文解读] Learning without Interaction Requires Separation.

Amit Daniely, Vitaly Feldman|arXiv (Cornell University)|Sep 24, 2018

Machine Learning and Algorithms被引用 9

一句话总结

本文证明，在局部差分隐私或有限通信约束下的非交互式学习要求低边际复杂度，从而在边际复杂度较大的类别中建立了交互式与非交互式学习之间的分离。本文提出了一种关于边际复杂度的多项式时间的标签非自适应SQ算法，通过边际复杂度刻画了非交互式可学习性。

ABSTRACT

One of the key resources in large-scale learning systems is the number of rounds of communication between the server and the clients holding the data points. We study this resource for systems with two types of constraints on the communication from each of the clients: local differential privacy and limited number of bits communicated. For both models the number of rounds of communications is captured by the number of rounds of interaction when solving the learning problem in the statistical query (SQ) model. For many learning problems known efficient algorithms require many rounds of interaction. Yet little is known on whether this is actually necessary. In the context of classification in the PAC learning model, Kasiviswanathan et al. (2008) constructed an artificial class of functions that is PAC learnable with respect to a fixed distribution but cannot be learned by an efficient non-interactive (or one-round) SQ algorithm. Here we show that a similar separation holds for any class with large margin complexity that is closed under negation, without assumptions on the distribution. That is, classes of functions that cannot be represented as large-margin linear separators. In particular this is true for linear separators and decision lists. To prove this separation we show that non-interactive SQ algorithms can only learn function classes of low margin complexity. Our lower bound also holds against a stronger class of algorithms that for which only queries that depend on labels are non-interactive (we refer to them as label-non-adaptive). We complement this lower bound with a new efficient and label-non-adaptive SQ learning algorithm whose complexity is polynomial in the margin complexity. We thus obtain a new characterization of margin complexity that might be of independent interest.

研究动机与目标

• 确定在通信或隐私约束下，非交互式学习是否在本质上受限于交互式学习。
• 研究在统计查询（SQ）模型中，具有大边际复杂度的类别是否无法在无交互的情况下被高效学习。
• 利用边际复杂度表征可通过非交互式或标签非自适应SQ算法学习的函数类别。
• 开发一种高效且标签非自适应的SQ算法，其复杂度关于边际复杂度呈多项式依赖。

提出的方法

• 作者通过证明非交互式SQ算法只能学习边际复杂度较低的函数类，分析了其局限性。
• 他们引入了一种更强的标签非自适应算法模型，其中仅依赖标签的查询是非交互式的。
• 构建了一种新的高效SQ学习算法，该算法为标签非自适应，且其复杂度关于边际复杂度呈多项式关系。
• 证明依赖于使用边际复杂度作为关键参数，对非交互式查询在学习过程中的影响进行有界控制。
• 分析将分布式学习中的通信约束与SQ模型联系起来，表明轮次复杂度对应于SQ查询中的交互性。

实验结果

研究问题

• RQ1 在局部差分隐私或比特受限通信下，非交互式学习算法能否高效学习具有大边际复杂度的函数类？
• RQ2 在闭合于否定运算且具有大边际复杂度的类别中，SQ模型下交互式与非交互式学习之间是否存在可证明的分离？
• RQ3 非交互式可学习性的精确表征在边际复杂度方面是什么？
• RQ4 能否设计一种标签非自适应SQ算法，使其复杂度关于边际复杂度呈多项式依赖，同时保持高效性？

主要发现

• 即使在类别对否定运算封闭且可PAC学习的情况下，非交互式SQ算法也无法学习具有大边际复杂度的函数类。
• 交互式与非交互式学习之间存在分离：具有大边际复杂度的类别需要交互才能被高效学习。
• 可通过非交互式或标签非自适应SQ算法学习的函数类，恰好由低边际复杂度所表征。
• 构建了一种高效标签非自适应SQ算法，其复杂度关于目标类的边际复杂度呈多项式关系。
• 该下界在更强的模型下依然成立，即仅依赖标签的查询是非交互式的，从而证实了该分离的稳健性。