[Paper Review] Why gradient clipping accelerates training: A theoretical justification for adaptivity

Jingzhao Zhang, Tianxing He+2 more|arXiv (Cornell University)|May 28, 2019

Stochastic Gradient Optimization Techniques57 references140 citations

TL;DR

The paper introduces a relaxed smoothness condition where local gradient Lipschitz constants can grow with the gradient norm and proves that gradient clipping and normalized gradient methods can converge faster than fixed-step gradient descent under this condition, with empirical validation in NLP and vision tasks.

ABSTRACT

We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that gradient smoothness, a concept central to the analysis of first-order optimization algorithms that is often assumed to be a constant, demonstrates significant variability along the training trajectory of deep neural networks. Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient. These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a fixed bound on smoothness. These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used Lipschitz smoothness assumption. Under the new condition, we prove that two popular methods, namely, \\emph{gradient clipping} and \\emph{normalized gradient}, converge arbitrarily faster than gradient descent with fixed stepsize. We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.

Motivation & Objective

• Motivate why adaptive gradient methods perform well in deep neural network training.
• Introduce a new relaxed smoothness condition that allows the Hessian norm to grow with gradient norm.
• Prove convergence and rate results for clipped gradient descent and normalized gradient descent under the new condition.
• Provide stochastic and deterministic convergence analyses comparing clipped GD to standard GD.
• Empirically validate the theory in NLP language modeling and image classification tasks.

Proposed method

• Define a relaxed (L0,L1)-smoothness condition: ||∇2f(x)|| ≤ L0 + L1||∇f(x)||.
• Analyze gradient descent with fixed stepsize, clipped gradient descent, and normalized gradient descent under the new condition.
• Prove upper and lower bounds on convergence rates for deterministic GD and clipped GD (Theorems 3, 4, 6).
• Extend the analysis to stochastic settings, deriving convergence guarantees for stochastic clipped GD and SGD (Theorems 7, 8).
• Relate clipped GD to normalized GD and discuss practical parameter settings (γ, ηc, ηn) for equivalence up to constants.

Experimental results

Research questions

• RQ1 Can a relaxed smoothness condition, where local smoothness grows with gradient norm, yield faster convergence guarantees for adaptive gradient methods?
• RQ2 Do gradient clipping and normalized gradient methods converge faster than fixed-step gradient descent under the relaxed smoothness condition?
• RQ3 How do these theoretical results extend to stochastic settings typical in neural network training?
• RQ4 What empirical evidence supports the proposed relaxed smoothness condition and its link to gradient clipping effectiveness in NLP and CV tasks?
• RQ5 How do these findings explain why adaptive methods outperform SGD in practice?

Key findings

• Clipped GD converges arbitrarily faster than fixed-step GD under the new (L0,L1)-smoothness condition (Theorem 3).
• Fixed-step GD can be arbitrarily slower than clipped GD under the relaxed smoothness framework (Theorem 4).
• Deterministic GD with fixed stepsize has upper bounds tied to L0 and L1, whereas clipped GD exhibits improved rates (Theorem 6).
• Stochastic clipped GD and SGD show that clipping can be faster than SGD with fixed stepsize (Theorems 7 and 8).
• Empirical NLP experiments (AWD-LSTM language modeling) show gradient smoothness correlates with gradient norm, matching the theory; clipping accelerates convergence in LM and can improve CV results as well.