[Paper Review] On the adequacy of untuned warmup for adaptive optimization

Jerry Ma, Denis Yarats|arXiv (Cornell University)|Oct 9, 2019

Advanced Bandit Algorithms Research31 references22 citations

TL;DR

This paper demonstrates that simple linear warmup for Adam—specifically over $2/(1-eta_2)$ training iterations—performs as well as the more complex RAdam algorithm, which claims to automatically rectify adaptive learning rate variance. The authors show that RAdam is equivalent to four initial momentum steps followed by Adam with a fixed warmup, and that update magnitude instability in early training, not gradient variance, explains the need for warmup.

ABSTRACT

Adaptive optimization algorithms such as Adam are widely used in deep learning. The stability of such algorithms is often improved with a warmup schedule for the learning rate. Motivated by the difficulty of choosing and tuning warmup schedules, recent work proposes automatic variance rectification of Adam's adaptive learning rate, claiming that this rectified approach ("RAdam") surpasses the vanilla Adam algorithm and reduces the need for expensive tuning of Adam with warmup. In this work, we refute this analysis and provide an alternative explanation for the necessity of warmup based on the magnitude of the update term, which is of greater relevance to training stability. We then provide some "rule-of-thumb" warmup schedules, and we demonstrate that simple untuned warmup of Adam performs more-or-less identically to RAdam in typical practical settings. We conclude by suggesting that practitioners stick to linear warmup with Adam, with a sensible default being linear warmup over $2 / (1 - β_2)$ training iterations.

Motivation & Objective

• To challenge the claim that RAdam eliminates the need for learning rate warmup by re-evaluating the underlying rationale for warmup in adaptive optimization.
• To investigate whether the instability in Adam’s early training is better explained by update magnitude dynamics than by gradient variance.
• To propose simple, untuned warmup schedules for Adam that match RAdam’s performance across diverse models and hyperparameters.
• To provide a practical, default warmup schedule that avoids costly hyperparameter tuning while maintaining training stability and convergence.

Proposed method

• Re-express RAdam as Adam with four initial heavy-ball momentum steps and a fixed warmup schedule, showing it is not a fundamentally new algorithm.
• Analyze the magnitude of parameter updates in Adam during early training using simulations, even at a simulated local minimum.
• Propose rule-of-thumb warmup schedules based on the hyperparameter $\beta_2$, specifically linear warmup over $2/(1 - \beta_2)$ iterations.
• Compare training dynamics and performance of untuned linear warmup with RAdam across multiple models and hyperparameter settings.
• Use simulation-based analysis to show that update magnitudes in Adam converge to a stationary distribution quickly, suggesting early instability is not due to gradient variance.
• Evaluate the effectiveness of the proposed warmup schedule in both standard and extreme hyperparameter regimes, including different $\beta_2$ values and learning rates.

Experimental results

Research questions

• RQ1 Is the need for learning rate warmup in Adam truly driven by gradient variance, as claimed by RAdam’s authors?
• RQ2 Can a simple, untuned warmup schedule for Adam achieve performance comparable to RAdam across diverse deep learning settings?
• RQ3 What is the actual cause of instability in Adam’s early training—gradient variance or update magnitude dynamics?
• RQ4 How does the magnitude of parameter updates in Adam evolve during early training, and does it stabilize quickly?
• RQ5 Can a default warmup schedule be derived that eliminates the need for hyperparameter tuning while maintaining training stability?

Key findings

• RAdam is mathematically equivalent to four initial momentum steps followed by Adam with a fixed warmup schedule, not a fundamentally new optimization method.
• The variance-based explanation for warmup is flawed because it ignores the correlation between first and second moment estimators in Adam.
• Even at a simulated local minimum, Adam exhibits significant non-regularity in early-stage update magnitudes, indicating a root cause of instability beyond gradient variance.
• Linear warmup over $2/(1 - \beta_2)$ iterations achieves performance and training dynamics indistinguishable from RAdam across a wide range of models and hyperparameters.
• The effective warmup period in RAdam and the proposed rule-of-thumb schedules is approximately 1000 iterations for $\beta_2 = 0.999$, despite theoretical convergence of update magnitudes in ~40 iterations.
• Late-stage update magnitudes in Adam are largely invariant to gradient variance or stationarity, suggesting that learning rate decay is the primary lever for improving late-stage convergence.