[Paper Review] Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Jinghui Chen, Dongruo Zhou+4 more|arXiv (Cornell University)|Jun 18, 2018

Stochastic Gradient Optimization Techniques43 references90 citations

TL;DR

Padam introduces a partially adaptive momentum method that unifies Adam/Amsgrad with SGD-Momentum, achieving fast convergence while matching SGD in generalization across benchmarks.

ABSTRACT

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter $p$, to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to a stationary point in the stochastic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain a fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.

Motivation & Objective

• Motivate the generalization gap observed with adaptive gradient methods like Adam and Amsgrad compared to SGD with momentum.
• Propose Padam, a partially adaptive momentum method, to balance adaptiveness and generalization.
• Provide a convergence analysis of Padam in stochastic nonconvex optimization.
• Empirically evaluate Padam on standard benchmarks (CIFAR-10, ImageNet, Penn Treebank) to compare generalization and convergence.
• Discuss practical implications for choosing optimizer in deep learning training.

Proposed method

• Introduce Padam by adding a partial adaptive exponent p in the second-moment normalization, interpolating between SGD-Momentum (p=0) and Amsgrad (p=1/2).
• Update rules: m_t = β1 m_{t-1} + (1−β1) g_t; v_t = β2 v_{t-1} + (1−β2) g_t^2; v̂_t = max(v̂_{t-1}, v_t); x_{t+1} = x_t − α_t m_t / v̂_t^p.
• Provide convergence guarantees for Padam in stochastic nonconvex optimization with rate characterized by E[||∇f(x_out)||^2] ≤ ... (expression in theorem).
• Explain the small learning rate dilemma in adaptive methods and how partial adaptivity mitigates it, enabling larger effective learning rates.
• Present empirical protocol including hyperparameter grids (p in {2/5, 1/4, 1/5, 1/8, 1/16}, β1=0.9, β2=0.999/0.9999, etc.).
• Compare Padam to SGD-Momentum, Adam, Amsgrad, AdamW, Yogi, AdaBound across image classification and language modeling tasks.

Experimental results

Research questions

• RQ1 Can a partially adaptive learning-rate scheme reduce the generalization gap observed for Adam/Amsgrad while preserving fast convergence?
• RQ2 Does Padam provide convergence guarantees in stochastic nonconvex optimization and what is the rate dependence on dimension and iteration?
• RQ3 How does Padam perform empirically on modern architectures (CIFAR-10/ImageNet, LSTMs) compared to SGD-Momentum and other Adam variants?
• RQ4 What is the impact of the partial adaptive parameter p on early convergence and final generalization?

Key findings

• Padam with a properly chosen p achieves fast convergence like Adam/Amsgrad while generalizing as well as SGD-Momentum.
• Convergence rate of Padam in stochastic nonconvex optimization is established, with rate depending on dimension d and iteration T.
• Empirical results show Padam often attaining the best test accuracy on CIFAR-10 (VGGNet/WideResNet) and competitive Top-1 accuracy on ImageNet (VGGNet); Padam achieves the best or near-best results across tested models.
• On Penn Treebank, Padam attains the lowest test perplexity among compared optimizers for both 2-layer and 3-layer LSTM models.
• Padam tends to avoid the generalization gap seen in Adam/Amsgrad and outperforms AdaBound/Yogi in several benchmarks.
• Overall, Padam provides a practical optimizer choice offering both fast training and strong generalization.