[论文解读] Why gradient clipping accelerates training: A theoretical justification for adaptivity

Jingzhao Zhang, Tianxing He+2 more|arXiv (Cornell University)|May 28, 2019

Stochastic Gradient Optimization Techniques参考文献 57被引用 140

一句话总结

本文提出了一种放宽的平滑性条件，在该条件下局部梯度的Lipschitz常数可以随梯度范数增长，并证明在该条件下梯度裁剪和归一化梯度方法的收敛速度可以比固定步长梯度下降更快，并在NLP和视觉任务中给出经验验证。

ABSTRACT

We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that gradient smoothness, a concept central to the analysis of first-order optimization algorithms that is often assumed to be a constant, demonstrates significant variability along the training trajectory of deep neural networks. Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient. These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a fixed bound on smoothness. These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used Lipschitz smoothness assumption. Under the new condition, we prove that two popular methods, namely, \\emph{gradient clipping} and \\emph{normalized gradient}, converge arbitrarily faster than gradient descent with fixed stepsize. We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.

研究动机与目标

• Motivate why adaptive gradient methods perform well in deep neural network training.
• Introduce a new relaxed smoothness condition that allows the Hessian norm to grow with gradient norm.
• Prove convergence and rate results for clipped gradient descent and normalized gradient descent under the new condition.
• Provide stochastic and deterministic convergence analyses comparing clipped GD to standard GD.
• Empirically validate the theory in NLP language modeling and image classification tasks.

提出的方法

• Define a relaxed (L0,L1)-smoothness condition: ||∇2f(x)|| ≤ L0 + L1||∇f(x)||.
• Analyze gradient descent with fixed stepsize, clipped gradient descent, and normalized gradient descent under the new condition.
• Prove upper and lower bounds on convergence rates for deterministic GD and clipped GD (Theorems 3, 4, 6).
• Extend the analysis to stochastic settings, deriving convergence guarantees for stochastic clipped GD and SGD (Theorems 7, 8).
• Relate clipped GD to normalized GD and discuss practical parameter settings (γ, ηc, ηn) for equivalence up to constants.]
• research_questions(可翻译成中文)：如下为中文翻译：

实验结果

研究问题

• RQ1 在局部平滑性随梯度范数增长的放宽条件下，是否能为自适应梯度方法提供更快的收敛保证？
• RQ2 在放宽的平滑性条件下，梯度裁剪和归一化梯度方法是否比固定步长梯度下降收敛更快？
• RQ3 这些理论结果在神经网络训练中常见的随机设置下如何扩展？
• RQ4 有哪些实证证据支持所提出的放宽平滑性条件及其与梯度裁剪在NLP和CV任务中的有效性之间的联系？
• RQ5 这些发现如何解释为什么自适应方法在实际中优于SGD？

主要发现

• 在新的 (L0,L1)-平滑性条件下，裁剪GD的收敛速度可以任意快于固定步长GD（定理3）。
• 在放宽平滑性框架下，固定步长GD的收敛速度可以比裁剪GD慢得多（定理4）。
• 确定性GD在固定步长下的上界与L0、L1相关，而裁剪GD显示出改进的收敛速度（定理6）。
• 随机裁剪GD和SGD表明裁剪可以比固定步长的SGD更快（定理7和定理8）。
• 经验性的NLP实验（AWD-LSTM语言建模）表明梯度平滑性与梯度范数相关，符合理论预测；裁剪在语言建模中加速收敛，在CV任务中也可提升结果。