[Paper Review] Accumulated Gradient Normalization

Joeri Hermans, Gerasimos Spanakis+1 more|arXiv (Cornell University)|Oct 6, 2017

Stochastic Gradient Optimization Techniques2 references6 citations

TL;DR

This paper proposes Accumulated Gradient Normalization (AGN), a distributed optimizer that improves convergence and stability in asynchronous data parallel training by normalizing accumulated first-order gradients before sending them to a parameter server. By reducing gradient magnitude and aligning update directions, AGN mitigates the negative effects of parameter staleness and implicit momentum, outperforming baselines like EASGD and DYN-SGD in both accuracy and temporal efficiency under high concurrency and communication constraints.

ABSTRACT

This work addresses the instability in asynchronous data parallel optimization. It does so by introducing a novel distributed optimizer which is able to efficiently optimize a centralized model under communication constraints. The optimizer achieves this by pushing a normalized sequence of first-order gradients to a parameter server. This implies that the magnitude of a worker delta is smaller compared to an accumulated gradient, and provides a better direction towards a minimum compared to first-order gradients, which in turn also forces possible implicit momentum fluctuations to be more aligned since we make the assumption that all workers contribute towards a single minima. As a result, our approach mitigates the parameter staleness problem more effectively since staleness in asynchrony induces (implicit) momentum, and achieves a better convergence rate compared to other optimizers such as asynchronous EASGD and DynSGD, which we show empirically.

Motivation & Objective

• To address instability in asynchronous data parallel optimization due to parameter staleness and implicit momentum.
• To reduce communication frequency between workers and parameter servers without sacrificing convergence or stability.
• To improve convergence speed and final model accuracy in highly concurrent distributed training setups.
• To develop a metric—temporal efficiency—that evaluates optimization performance beyond final accuracy, incorporating stability and time-to-accuracy.
• To demonstrate that normalized accumulated gradients provide better update directions than individual first-order gradients in asynchronous settings.

Proposed method

• AGN computes a sequence of first-order gradients locally on each worker without communication during the accumulation phase.
• These accumulated gradients are normalized by their L2 norm to reduce magnitude and improve directionality.
• The normalized accumulated gradient is then sent to the parameter server for model update, reducing the impact of stale or divergent updates.
• The method assumes all workers contribute toward a common minimum, aligning implicit momentum effects across workers.
• The parameter server aggregates updates using a simple FIFO queue, maintaining consistency with standard asynchronous data parallelism.
• Temporal efficiency is introduced as a metric to compare optimizers by integrating training metric areas over time.

Experimental results

Research questions

• RQ1 How does normalizing accumulated gradients affect convergence and stability in asynchronous distributed training?
• RQ2 Can accumulated gradient normalization reduce the negative impact of parameter staleness and implicit momentum in high-concurrency settings?
• RQ3 How does AGN compare to EASGD and DYN-SGD in terms of final accuracy, convergence speed, and temporal efficiency?
• RQ4 Does the use of normalized accumulated gradients lead to better alignment of update directions across workers?
• RQ5 Can AGN maintain performance under varying communication frequencies and numbers of workers?

Key findings

• AGN outperforms EASGD in 69.73% of experimental cases across different hyperparameter settings (number of workers and communication frequency).
• AGN achieves higher training and validation accuracy than EASGD, especially under high concurrency and reduced communication frequency.
• In the MNIST experiment with 40 workers and λ=40, AGN reached 97.88% test accuracy in 257.62 seconds, outperforming EASGD in both speed and accuracy.
• The temporal efficiency metric shows AGN maintains better performance over time, indicating improved stability and convergence dynamics.
• AGN reduces the negative effects of implicit momentum by aligning gradient directions across workers, even under high staleness.
• The method remains robust across diverse distributed hyperparameter configurations, suggesting practical advantages in real-world deployment.