[论文解读] Adaptive Learning with Robust Generalization Guarantees

Rachel Cummings, Katrina Ligett+3 more|arXiv (Cornell University)|Feb 24, 2016

Privacy-Preserving Technologies in Data参考文献 5被引用 4

一句话总结

本文引入了三种逐步增强的泛化概念——鲁棒泛化、差分隐私和完美泛化——这些概念对后处理和自适应组合均具有鲁棒性。论文证明了所有PAC可学习的概念类均可在近似最优样本复杂度下通过鲁棒泛化实现学习，并表明鲁棒泛化严格弱于差分隐私但严格强于完美泛化，其在自适应学习和隐私保护机器学习中具有应用价值。

ABSTRACT

The traditional notion of generalization---i.e., learning a hypothesis whose empirical error is close to its true error---is surprisingly brittle. As has recently been noted in [DFH+15b], even if several algorithms have this guarantee in isolation, the guarantee need not hold if the algorithms are composed adaptively. In this paper, we study three notions of generalization---increasing in strength---that are robust to postprocessing and amenable to adaptive composition, and examine the relationships between them. We call the weakest such notion Robust Generalization. A second, intermediate, notion is the stability guarantee known as differential privacy. The strongest guarantee we consider we call Perfect Generalization. We prove that every hypothesis class that is PAC learnable is also PAC learnable in a robustly generalizing fashion, with almost the same sample complexity. It was previously known that differentially private algorithms satisfy robust generalization. In this paper, we show that robust generalization is a strictly weaker concept, and that there is a learning task that can be carried out subject to robust generalization guarantees, yet cannot be carried out subject to differential privacy. We also show that perfect generalization is a strictly stronger guarantee than differential privacy, but that, nevertheless, many learning tasks can be carried out subject to the guarantees of perfect generalization.

研究动机与目标

• 为解决传统泛化在自适应和后处理学习设置下的脆弱性问题。
• 形式化并比较三种逐步增强的泛化保证：鲁棒泛化、差分隐私和完美泛化。
• 证明鲁棒泛化可在无限域上实现近似最优样本复杂度的PAC学习。
• 表明鲁棒泛化严格弱于差分隐私，但严格强于完美泛化。
• 建立完美泛化支持自适应组合，并可通过完美泛化机制的组合实现。

提出的方法

• 引入'鲁棒泛化'作为保证：任何后处理器都无法生成经验误差与真实误差显著不同的假设。
• 将'完美泛化'定义为强于差分隐私的保证，确保任何后处理后的假设均保持泛化能力。
• 使用拉普拉斯机制作为满足鲁棒泛化的差分私密算法的典型示例。
• 利用群体隐私将差分隐私扩展至完美泛化，尽管存在参数膨胀。
• 通过二项分布的尾部界推导强泛化下隐私参数的下界。
• 采用基于模拟器的论证和集中不等式，将差分隐私参数与强泛化参数关联。

实验结果

研究问题

• RQ1 是否所有PAC可学习的概念类均可实现接近最优样本复杂度的鲁棒泛化？
• RQ2 鲁棒泛化是否严格弱于差分隐私？若是，能否用于解决差分隐私无法解决的学习任务？
• RQ3 完美泛化是否严格强于差分隐私？能否通过自适应组合实现？
• RQ4 差分隐私与强泛化在参数膨胀方面有何关系？
• RQ5 能否在不导致过度参数膨胀的情况下实现完美泛化，特别是通过机制组合？

主要发现

• 所有PAC可学习的概念类均可通过样本复杂度渐近接近最优的鲁棒泛化学习算法实现。
• 存在一个学习任务可在鲁棒泛化下解决，但无法在差分隐私下解决，证明鲁棒泛化严格弱于差分隐私。
• 完美泛化严格强于差分隐私，其存在性通过存在满足完美泛化但不满足差分隐私的机制得到证明。
• 拉普拉斯机制满足鲁棒泛化，且为$(\beta, \tilde{\theta}(\beta, \frac{1}{\beta}))$-强泛化，$\beta$-强泛化参数的下界为$\tilde{\theta}(\beta \times \frac{1}{\beta})$。
• 群体隐私表明，任何$(\beta, \tilde{\theta}(\beta, \frac{1}{\beta}))$-差分私密机制均为$(0, n\beta, ne^{(n-1)\beta}\tilde{\theta}(\beta, \frac{1}{\beta}))$-完美泛化，尽管存在显著的参数膨胀。
• 对$(\beta, \tilde{\theta}(\beta, \frac{1}{\beta}))$-完美泛化机制进行组合，其泛化参数优于通过群体隐私直接约减的结果，表明这是一条更具可扩展性的完美泛化实现路径。