[Paper Review] Ranger21: a synergistic deep learning optimizer

Less Wright, Nestor Demeure|arXiv (Cornell University)|Jun 25, 2021

Advanced Neural Network Applications22 references4 citations

TL;DR

Ranger21 is a synergistic deep learning optimizer that combines AdamW with eight complementary optimization techniques—such as adaptive gradient clipping, weight decay reweighting, and Lookahead—resulting in faster convergence, smoother training curves, and the ability to train a ResNet50 on ImageNet2012 without batch normalization, where AdamW fails entirely.

ABSTRACT

As optimizers are critical to the performances of neural networks, every year a large number of papers innovating on the subject are published. However, while most of these publications provide incremental improvements to existing algorithms, they tend to be presented as new optimizers rather than composable algorithms. Thus, many worthwhile improvements are rarely seen out of their initial publication. Taking advantage of this untapped potential, we introduce Ranger21, a new optimizer which combines AdamW with eight components, carefully selected after reviewing and testing ideas from the literature. We found that the resulting optimizer provides significantly improved validation accuracy and training speed, smoother training curves, and is even able to train a ResNet50 on ImageNet2012 without Batch Normalization layers. A problem on which AdamW stays systematically stuck in a bad initial state.

Motivation & Objective

• To address the fragmentation in deep learning optimizer research, where incremental improvements are often presented as standalone optimizers rather than composable components.
• To investigate whether combining multiple, independently validated optimization techniques could yield synergistic performance gains beyond individual components.
• To demonstrate that a single, unified optimizer can outperform established baselines like AdamW on challenging benchmarks, including architectures previously considered difficult to train.
• To provide a publicly available, modular optimizer that enables practitioners to benefit from state-of-the-art optimization techniques without re-implementing them.
• To explore the potential of meta-optimization and modular design for future adaptive optimizer construction.

Proposed method

• Ranger21 is built on AdamW as its core optimization engine, maintaining adaptive learning rates via exponential moving averages of gradients and squared gradients.
• It integrates adaptive gradient clipping to stabilize training by dynamically capping large gradients, preventing divergence in high-variance mini-batches.
• It applies weight decay reweighting to decouple weight decay from the learning rate, improving generalization and training stability.
• It incorporates the Lookahead optimizer to improve convergence by updating a 'fast' weight vector via a slow, moving average update rule.
• It uses the Rectified Adam (RAdam) technique to correct the variance of the adaptive learning rate in the early training phase.
• It applies gradient normalization and a modified learning rate schedule to further stabilize optimization dynamics and improve generalization.

Experimental results

Research questions

• RQ1 Can combining multiple composable optimization improvements into a single optimizer yield synergistic performance gains beyond individual components?
• RQ2 Does a unified optimizer built from proven techniques outperform AdamW on standard benchmarks like ImageNet2012?
• RQ3 Can Ranger21 train a ResNet50 without batch normalization layers, where AdamW fails to converge?
• RQ4 How does Ranger21 compare to AdamW in terms of training speed, validation accuracy, and training curve smoothness?
• RQ5 To what extent can modular, composable optimization components be generalized across different architectures and datasets?

Key findings

• Ranger21 achieved significantly higher final validation accuracy on ImageNet2012 compared to AdamW, demonstrating superior generalization.
• Training with Ranger21 reached key validation accuracy levels up to 3x faster than with AdamW, indicating substantial speed improvements.
• Ranger21 produced smoother training curves with reduced loss oscillations, indicating more stable optimization dynamics.
• Unlike AdamW, which failed to converge on a Normalizer-Free ResNet50 (loss remained constant at ~0.1), Ranger21 successfully trained the model with steady loss reduction over 60 epochs.
• The optimizer's performance was consistent across multiple random seeds, confirming robustness and not reliance on chance initialization.
• Ranger21 is publicly available with ready-to-use PyTorch and Flax implementations, enabling immediate adoption and community feedback.