[论文解读] High probability generalization bounds for uniformly stable algorithms with nearly optimal rate

Vitaly Feldman, J. Vondrák|arXiv (Cornell University)|Feb 27, 2019

Stochastic Gradient Optimization Techniques参考文献 44被引用 40

一句话总结

该论文证明了 gamma-均匀稳定算法在对数开销下几乎紧致的高概率泛化界，优于先前界限，并为 SGD 与正则化 ERM 提供强有力的结果。

ABSTRACT

Algorithmic stability is a classical approach to understanding and analysis of the generalization error of learning algorithms. A notable weakness of most stability-based generalization bounds is that they hold only in expectation. Generalization with high probability has been established in a landmark paper of Bousquet and Elisseeff (2002) albeit at the expense of an additional $\sqrt{n}$ factor in the bound. Specifically, their bound on the estimation error of any $γ$-uniformly stable learning algorithm on $n$ samples and range in $[0,1]$ is $O(γ\sqrt{n \log(1/δ)} + \sqrt{\log(1/δ)/n})$ with probability $\geq 1-δ$. The $\sqrt{n}$ overhead makes the bound vacuous in the common settings where $γ\geq 1/\sqrt{n}$. A stronger bound was recently proved by the authors (Feldman and Vondrak, 2018) that reduces the overhead to at most $O(n^{1/4})$. Still, both of these results give optimal generalization bounds only when $γ= O(1/n)$. We prove a nearly tight bound of $O(γ\log(n)\log(n/δ) + \sqrt{\log(1/δ)/n})$ on the estimation error of any $γ$-uniformly stable algorithm. It implies that for algorithms that are uniformly stable with $γ= O(1/\sqrt{n})$, estimation error is essentially the same as the sampling error. Our result leads to the first high-probability generalization bounds for multi-pass stochastic gradient descent and regularized ERM for stochastic convex problems with nearly optimal rate --- resolving open problems in prior work. Our proof technique is new and we introduce several analysis tools that might find additional applications.

研究动机与目标

• 动机与分析具有均匀稳定性的学习算法的高概率泛化界限。
• 通过将开销从 sqrt(n) 降至 delta 和 n 的多对数因子来改进先前的界限。
• 在随机凸设定下展示对多轮 SGD 与正则化 ERM 的适用性。
• 提供通过范围缩减和数据集划分分解估计误差的新证明技巧。

提出的方法

• 为数据相关函数定义 gamma-均匀稳定性，并将其与泛化误差 Delta_s(M) 联系起来。
• 引入一个 leave-one-out 无偏变换 L(s,z)，并具有受控稳定性。
• 通过钳位和均值减法实现范围缩减，同时保持零均值和稳定性。
• 使用递归的两步方案（范围缩减和数据集规模缩减）来界定估计误差。
• 证明主聚合界限具有指数尾：Delta_s(M) <= c( gamma log(n) log(n/delta) + sqrt(log(1/delta)/n) ).
• 将该界限应用于随机凸优化设置，并推导强凸 ERM 与 SGD 的推论。

实验结果

研究问题

• RQ1 是否可以仅通过对数开销实现 gamma-均匀稳定算法的高概率泛化界？
• RQ2 在实际中，稳定性参数 gamma 如何与对数因子相互作用以影响估计误差？
• RQ3 新的界限是否在随机凸问题中的多轮 SGD 与正则化 ERM 上提供有意义的保证？
• RQ4 新的范围缩减（钳位）技术是否能够同时保持无偏与稳定性？
• RQ5 结果如何为预测私有学习及相关的隐私保护设定提供指导？

主要发现

• 建立了一个高概率界限：Pr[Delta_s(M) >= c( gamma log(n) log(n/delta) + sqrt(log(1/delta)/n) )] <= delta.
• 证明当 gamma = O(1/√n) 时，估计误差在对数因子上的程度与取样误差相匹配。
• 在 SGD 的全梯度下降和随机凸问题中的正则化 ERM 上推导出近乎最优的高概率泛化界。
• 给出伪结论，在正则化参数 lambda = log(n)/√n 时提供高概率保证。
• 通过预测私有学习及相关界限，展示对差分隐私情境的适用性。