[论文解读] Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints

Wenlong Mou, Liwei Wang+2 more|arXiv (Cornell University)|Jul 19, 2017

Stochastic Gradient Optimization Techniques参考文献 23被引用 55

一句话总结

论文为非凸学习中的 SGLD 推导了两种算法相关的泛化界限，使用稳定性和 PAC-Bayesian 方法，界限不直接依赖模型维度，且依赖聚合步长。

ABSTRACT

Algorithm-dependent generalization error bounds are central to statistical learning theory. A learning algorithm may use a large hypothesis space, but the limited number of iterations controls its model capacity and generalization error. The impacts of stochastic gradient methods on generalization error for non-convex learning problems not only have important theoretical consequences, but are also critical to generalization errors of deep learning. In this paper, we study the generalization errors of Stochastic Gradient Langevin Dynamics (SGLD) with non-convex objectives. Two theories are proposed with non-asymptotic discrete-time analysis, using Stability and PAC-Bayesian results respectively. The stability-based theory obtains a bound of $O\left(\frac{1}{n}L\sqrt{βT_k} ight)$, where $L$ is uniform Lipschitz parameter, $β$ is inverse temperature, and $T_k$ is aggregated step sizes. For PAC-Bayesian theory, though the bound has a slower $O(1/\sqrt{n})$ rate, the contribution of each step is shown with an exponentially decaying factor by imposing $\ell^2$ regularization, and the uniform Lipschitz constant is also replaced by actual norms of gradients along trajectory. Our bounds have no implicit dependence on dimensions, norms or other capacity measures of parameter, which elegantly characterizes the phenomenon of "Fast Training Guarantees Generalization" in non-convex settings. This is the first algorithm-dependent result with reasonable dependence on aggregated step sizes for non-convex learning, and has important implications to statistical learning aspects of stochastic gradient methods in complicated models such as deep learning.

研究动机与目标

• 了解随机梯度 Langevin 动力学（SGLD）在非凸学习中的泛化影响。
• 使用两种理论框架：稳定性和 PAC-Bayes，提供非渐近、与算法相关的界限。
• 证明界限可以与维度无关，并依赖聚合步长而非参数范数。
• 将理论与深度学习训练中非凸性和随机性突出的实际含义联系起来。

提出的方法

• 将学习目标建模为正则化经验风险 F_n(w) = (1/n) sum_i f_i(w) + R(w)。
• 分析 SGLD 更新式 w_{k+1} = w_k - eta_k g_hat_k(w) + sqrt(2 eta_k / beta) N(0, I_d)。
• 使用两种分析框架：统一稳定性（导致快速的 O(1/n) 速率）和 PAC-Bayesian 理论（在轨迹自适应项下得到 O(1/√n) 速率）。
• 将离散时间的 SGLD 与连续时间的 Langevin 方程及其 Fokker-Planck 描述联系起来，通过海灵格距离和 KL 散度来界定分布变化。
• 强调所得界限与参数维度无关，并依赖于聚合步长与沿轨迹的梯度范数。

实验结果

研究问题

• RQ1 SGLD 如何影响非凸学习情境中的泛化误差？
• RQ2 我们能否利用稳定性和 PAC-Bayesian 技术为 SGLD 获得非渐近、与算法相关的泛化界限？
• RQ3 界限是否依赖于聚合步长而非模型维度或参数范数，以及沿轨迹的梯度范数如何影响它们？
• RQ4 稳定性基与基于 PAC-Bayes 的界限在非凸随机优化中的权衡是什么？

主要发现

• 基于稳定性的界限给出 O(1/n) 速率，随 L、beta 以及累积步长的平方根而变化。
• PAC-Bayesian 界限给出 O(1/√n) 速率，在迭代过程中有指数衰减因子，并依赖于轨迹上的梯度范数。
• 连续时间 Langevin 分析为理想化情形提供 O(L C sqrt(beta T)/(sqrt{2} n)) 的界限，凸显聚合时间 T 的作用。
• 离散时间 SGLD 的稳健性分析表明，在随机数据抽样下，邻近数据集之间的平方海灵格距离可以被控制，从而获得有利的泛化界限。
• 界限不明确依赖参数维度或参数范数，支持在非凸情境中“快速训练保证泛化”的直觉。