[Paper Review] Step-size Adaptation Using Exponentiated Gradient Updates

Ehsan Amid, Rohan Anil+2 more|arXiv (Cornell University)|Jan 31, 2022

Machine Learning and ELM4 citations

TL;DR

This paper proposes a meta-optimization framework that adaptively tunes global step-size and per-coordinate gains using unnormalized exponentiated gradient updates (EGU), enabling automatic learning rate schedule discovery in deep learning. The method achieves state-of-the-art performance without manual learning rate scheduling, matching or exceeding baseline results on CIFAR-10 and ImageNet with ResNet50 and MobileNetV1 models.

ABSTRACT

Optimizers like Adam and AdaGrad have been very successful in training large-scale neural networks. Yet, the performance of these methods is heavily dependent on a carefully tuned learning rate schedule. We show that in many large-scale applications, augmenting a given optimizer with an adaptive tuning method of the step-size greatly improves the performance. More precisely, we maintain a global step-size scale for the update as well as a gain factor for each coordinate. We adjust the global scale based on the alignment of the average gradient and the current gradient vectors. A similar approach is used for updating the local gain factors. This type of step-size scale tuning has been done before with gradient descent updates. In this paper, we update the step-size scale and the gain variables with exponentiated gradient updates instead. Experimentally, we show that our approach can achieve compelling accuracy on standard models without using any specially tuned learning rate schedule. We also show the effectiveness of our approach for quickly adapting to distribution shifts in the data during training.

Motivation & Objective

• To address the critical dependency of deep learning optimizers on manually tuned learning rate schedules.
• To unify existing step-size adaptation methods under a rigorous, generalizable framework.
• To enable automatic discovery of effective learning rate schedules through adaptive gain and scale parameters.
• To improve robustness to data distribution shifts during training.
• To demonstrate compatibility with a wide range of base optimizers, including Adam, AdaGrad, and SGD Momentum.

Proposed method

• Introduces a meta-optimization framework that maintains a global step-size scale and per-coordinate gain factors as learnable hyperparameters.
• Updates the global scale and local gains using unnormalized exponentiated gradient updates (EGU), which enforce non-negativity and enable multiplicative, sparse updates.
• The EGU update rule is: θ^{t+1} = θ^t ⊙ exp(−η ∇_θ f(θ^t)), ensuring stable, non-negative parameter updates.
• The framework accepts any pre-conditioned gradient from a base optimizer (e.g., Adam, AdaGrad, SGD Momentum), making it modular and widely applicable.
• Uses an exponential moving average (EMA) of gradients to compute alignment signals for scale and gain updates.
• The method is agnostic to the internal optimizer and can be applied as a plug-in meta-algorithm.

Experimental results

Research questions

• RQ1 Can a unified, principled method for step-size adaptation outperform manual learning rate scheduling in large-scale deep learning?
• RQ2 How effective is EGU-based tuning of global scale and per-coordinate gains in discovering effective learning rate schedules automatically?
• RQ3 Can the proposed method improve generalization and adapt to data distribution shifts without retraining?
• RQ4 How does the method perform across diverse architectures (e.g., MobileNetV1, ResNet50) and datasets (CIFAR-10, ImageNet)?
• RQ5 Does the EGU-based update mechanism provide advantages over heuristic or gradient-based tuning methods in terms of convergence and stability?

Key findings

• On CIFAR-10 with MobileNetV1, Funneled Momentum achieved 89.61% top-1 accuracy without a learning rate schedule, matching the 89.51% accuracy of the baseline with manual scheduling.
• On ImageNet with ResNet50, Funneled Momentum achieved 72.39% top-1 accuracy without scheduling, outperforming AdaGrad-EMA (70.70%) and significantly surpassing standard Momentum without scheduling (53.80%).
• The method automatically recovers a decay-like schedule for the global step-size across all layers, with varying decay rates per layer, as shown in Figure 4.
• Per-coordinate gain values dynamically ramp up initially for ~5k steps before decaying, indicating adaptive exploration of relevant features.
• The method is robust to distribution shifts, as demonstrated by rapid adaptation during training.
• The code is publicly released at https://users.soe.ucsc.edu/~eamid/funnel.html for reproducibility and further research.