[Paper Review] Robustness to Unbounded Smoothness of Generalized SignSGD

Michael Crawshaw, Mingrui Liu+3 more|arXiv (Cornell University)|Aug 23, 2022

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes a generalized SignSGD algorithm that achieves convergence rates comparable to gradient-clipped SGD under relaxed $(L_0, L_1)$ smoothness conditions—where gradients and Hessian norms can grow unboundedly—without requiring explicit gradient clipping. The method leverages momentum to implicitly control unbounded smoothness and gradient norms, enabling robust performance on deep learning models like Transformers, matching Adam’s accuracy while outperforming other optimizers.

ABSTRACT

Traditional analyses in non-convex optimization typically rely on the smoothness assumption, namely requiring the gradients to be Lipschitz. However, recent evidence shows that this smoothness condition does not capture the properties of some deep learning objective functions, including the ones involving Recurrent Neural Networks and LSTMs. Instead, they satisfy a much more relaxed condition, with potentially unbounded smoothness. Under this relaxed assumption, it has been theoretically and empirically shown that the gradient-clipped SGD has an advantage over the vanilla one. In this paper, we show that clipping is not indispensable for Adam-type algorithms in tackling such scenarios: we theoretically prove that a generalized SignSGD algorithm can obtain similar convergence rates as SGD with clipping but does not need explicit clipping at all. This family of algorithms on one end recovers SignSGD and on the other end closely resembles the popular Adam algorithm. Our analysis underlines the critical role that momentum plays in analyzing SignSGD-type and Adam-type algorithms: it not only reduces the effects of noise, thus removing the need for large mini-batch in previous analyses of SignSGD-type algorithms, but it also substantially reduces the effects of unbounded smoothness and gradient norms. We also compare these algorithms with popular optimizers on a set of deep learning tasks, observing that we can match the performance of Adam while beating the others.

Motivation & Objective

• To address the limitations of traditional smoothness assumptions in non-convex optimization, particularly for deep learning models like Transformers and LSTMs that exhibit unbounded smoothness.
• To demonstrate that gradient clipping is not necessary for convergence under relaxed $(L_0, L_1)$ smoothness conditions, especially for Adam-type algorithms.
• To propose and theoretically analyze a generalized SignSGD algorithm that interpolates between SignSGD and Adam, eliminating the need for explicit clipping.
• To empirically validate that momentum alone can implicitly stabilize training under unbounded smoothness, explaining why clipping is ineffective in Adam.

Proposed method

• Introduces a generalized SignSGD algorithm parameterized to recover SignSGD at one extreme and closely resemble Adam at the other, using adaptive momentum and normalized gradient signs.
• Employs a coordinate-wise $(L_0, L_1)$ smoothness assumption to better capture the heterogeneous gradient behavior across layers and parameters in deep networks.
• Theoretically proves convergence under unbounded smoothness by showing that momentum induces an exponential decay on gradient norms and smoothness terms.
• Uses momentum to reduce noise effects and suppress the impact of unbounded gradient norms, effectively replacing explicit clipping.
• Employs grid search and repeated training runs to tune hyperparameters (learning rate, weight decay, $eta_2$) and compare performance across optimizers on Transformer models.
• Validates the method on WMT’16 German-English translation and Wikitext-2 tasks, measuring training loss, perplexity, and accuracy across five random seeds.

Experimental results

Research questions

• RQ1 Can a SignSGD-type algorithm achieve convergence rates comparable to clipped SGD under the relaxed $(L_0, L_1)$ smoothness condition without explicit gradient clipping?
• RQ2 What is the role of momentum in mitigating the effects of unbounded smoothness and gradient norms in non-convex optimization?
• RQ3 Does Adam’s robustness to unbounded smoothness stem from implicit clipping, and can this be replicated in a SignSGD framework?
• RQ4 Can a unified algorithm be designed that bridges SignSGD and Adam, offering both theoretical robustness and empirical performance?
• RQ5 Does the generalized SignSGD algorithm match or exceed the performance of Adam on large-scale Transformer models under unbounded smoothness?

Key findings

• The proposed generalized SignSGD algorithm achieves a test perplexity of 7.2731 ± 0.0870 on the WMT’16 German-English translation task, outperforming all baselines including SGD with momentum and clipping.
• The algorithm matches Adam’s test accuracy of 68.9828 ± 0.2786 while achieving a significantly lower training loss of 1.6263 ± 0.0024 compared to Adam’s 1.4303 ± 0.0009.
• Empirical results show that gradient clipping has no measurable effect on Adam’s performance when training large Transformer models, suggesting implicit clipping via second-moment normalization.
• The coordinate-wise $(L_0, L_1)$ smoothness assumption is empirically validated on Transformers, with distinct $L_0$ and $L_1$ values observed across layers and parameters.
• Momentum is shown to play a critical role in suppressing unbounded smoothness and gradient norms, enabling stable convergence without explicit clipping.
• SGD with momentum and clipping performs worse than the proposed method, indicating that clipping is not beneficial when momentum is already present and effectively controls gradient growth.