[Paper Review] On the Variance of the Adaptive Learning Rate and Beyond

Liyuan Liu, Haoming Jiang+5 more|arXiv (Cornell University)|Aug 8, 2019

Advanced Neural Network Applications28 references607 citations

TL;DR

The paper analyzes why learning rate warmup helps adaptive optimizers like Adam by reducing the variance of the adaptive learning rate in early training, and introduces Rectified Adam (RAdam) to explicitly rectify this variance with solid theoretical grounding and strong empirical results.

ABSTRACT

The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: https://github.com/LiyuanLucasLiu/RAdam.

Motivation & Objective

• Identify the root cause of training instability with adaptive optimizers in the early training stage.
• Provide theoretical justification for warmup as variance reduction.
• Propose a variance-rectified variant of Adam (RAdam) and analyze its properties.
• Empirically validate RAdam across language modeling, image classification, and machine translation tasks.

Proposed method

• Present a generic adaptive optimization framework parameterized by momentum and adaptive step size.
• Analyze the variance of the adaptive learning rate and show it is large when sample size is small.
• Introduce two variance-reducing variants (Adam-2k and Adam-eps) to support the variance issue empirically.
• Derive a rectification factor r_t to normalize the variance of the adaptive learning rate based on a measured rho_t (effective SMA length).
• Propose Rectified Adam (RAdam) which applies the variance rectification term when rho_t > 4 and otherwise uses unadapted momentum update.
• Provide algorithms (Algorithm 2) for RAdam with practical steps and bias correction.

Experimental results

Research questions

• RQ1 Does the early-stage high variance of the adaptive learning rate cause instability or convergence to bad local optima in Adam?”,“Can warmup be theoretically interpreted as variance reduction for adaptive optimizers?”,“Can we design a principled rectification to stabilize the adaptive learning rate without tuning extra hyperparameters?”,“How does the proposed RAdam perform across language modeling, image classification, and neural machine translation compared to vanilla Adam and warmup baselines?”],
• RQ2 key_findings':['Adaptive learning rate variance is large in early training due to limited samples, leading to unstable updates.','Warmup can be interpreted as a variance reduction technique for adaptive optimizers.','Rectified Adam (RAdam) reduces early-stage variance and matches or exceeds Adam performance across tasks, with robustness to learning-rate changes.','RAdam achieves consistent improvements over vanilla Adam on language modeling (One Billion Word) and image classification (CIFAR10, ImageNet).','RAdam enables comparable performance to Adam with warmup on neural machine translation datasets (IWSLT’14 De-En/En-De, WMT’16 En-De) while requiring less hyperparameter tuning.','Simulated and theoretical analyses support the variance-rectification mechanism and its practical efficacy.

Key findings

• Image classification: CIFAR10 and ImageNet show RAdam outperforms vanilla Adam in accuracy and training dynamics.
• Language modeling: One Billion Word shows RAdam improves convergence speed and final performance versus Adam.
• NMT: RAdam achieves BLEU scores comparable to Adam with warmup across multiple datasets.
• RAdam demonstrates robust performance when initial learning rates vary, more so than Adam or SGD in several settings.