[Paper Review] Demon: Improved Neural Network Training with Momentum Decay

John Chen, C. Wolfe+2 more|arXiv (Cornell University)|Oct 11, 2019

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Demon, a novel momentum decay schedule for stochastic gradient descent with momentum (SGDM) and Adam optimizers, which dynamically reduces momentum over training time using a non-linear decay rule. Demon achieves state-of-the-art performance across 28 diverse deep learning tasks, outperforming all standard learning rate and momentum schedules, including cosine decay and OneCycle, with 39% Top-1 and 85% Top-3 finishes, while requiring no additional hyperparameter tuning and incurring negligible computational cost.

ABSTRACT

Momentum is a widely used technique for gradient-based optimizers in deep learning. In this paper, we propose a decaying momentum ( extsc{Demon}) rule. We conduct the first large-scale empirical analysis of momentum decay methods for modern neural network optimization, in addition to the most popular learning rate decay schedules. Across 28 relevant combinations of models, epochs, datasets, and optimizers, extsc{Demon} achieves the highest number of Top-1 and Top-3 finishes at 39\% and 85\% respectively, almost doubling the second-placed learning rate cosine schedule at 17\% and 60\%, respectively. extsc{Demon} also outperforms other widely used schedulers including, but not limited to, the learning rate step schedule, linear schedule, OneCycle schedule, and exponential schedule. Compared with the widely used learning rate step schedule, extsc{Demon} is observed to be less sensitive to parameter tuning, which is critical to training neural networks in practice. Results are demonstrated across a variety of settings and architectures, including image classification, generative models, and language models. extsc{Demon} is easy to implement, requires no additional tuning, and incurs almost no extra computational overhead compared to the vanilla counterparts. Code is readily available.

Motivation & Objective

• To address the challenge of hyperparameter sensitivity in deep learning optimization, particularly the reliance on extensive tuning of learning rate and momentum schedules.
• To improve model generalization and training stability across diverse architectures and datasets by introducing a principled momentum decay strategy.
• To reduce the computational and human cost of hyperparameter search by designing a momentum schedule that performs consistently well without additional tuning.
• To evaluate the effectiveness of momentum decay against standard learning rate schedules in real-world deep learning benchmarks, including image classification, language modeling, and generative modeling.

Proposed method

• Demon implements a non-linear momentum decay schedule where the momentum coefficient β_t decreases from an initial value β_init to 0 over training, following the formula: β_t = β_init * (1 - t/T) / [(1 - β_init) + β_init * (1 - t/T)].
• The decay rule is designed to maintain high momentum early in training for faster convergence and reduce it later to improve generalization and stability.
• The method is compatible with both SGDM and Adam optimizers, with separate algorithmic formulations for each, enabling broad applicability.
• The approach is computationally efficient, requiring only a single additional computation per iteration to compute β_t, with no extra memory or significant overhead.
• The method is implemented in PyTorch and released publicly, enabling easy integration into existing training pipelines.
• The authors conduct a large-scale empirical evaluation across 28 model-dataset-optimizer-epoch combinations, comparing Demon to 10 established learning rate and momentum schedules.

Experimental results

Research questions

• RQ1 Does a non-linear momentum decay schedule outperform standard learning rate decay schedules in terms of final model accuracy and robustness across diverse deep learning tasks?
• RQ2 How does Demon compare to widely used learning rate schedules such as cosine, step, linear, and OneCycle in terms of Top-1 and Top-3 performance across multiple benchmarks?
• RQ3 To what extent does Demon reduce the sensitivity to hyperparameter tuning compared to standard schedules like learning rate step decay?
• RQ4 Can Demon be effectively applied to both SGDM and Adam optimizers, and does it maintain consistent performance improvements across different architectures and datasets?
• RQ5 Is the performance gain of Demon attributable to a simple effective learning rate adjustment, or does it stem from a more complex dynamic behavior in the optimization trajectory?

Key findings

• Demon achieved the highest number of Top-1 finishes (39.29%) across 28 experiments, significantly outperforming the next-best method, the learning rate cosine schedule (17.86%).
• Demon achieved 85.71% Top-3 finishes, nearly doubling the performance of the second-best method (60.71% for learning rate cosine schedule).
• In image classification (ResNet-20 on CIFAR-10), Demon SGDM reduced test error from 8.71% (SGD ELR) to 8.71% with a 0.5% improvement over baseline SGD ELR.
• On the BERT base model fine-tuned on GLUE, Demon Adam achieved a 0.6-point improvement on the CoLA task (79.7 vs. 79.1) and 0.5-point gain on QNLI (90.0 vs. 89.0), with no additional tuning.
• On VAE training for MNIST, Demon SGDM reduced generalization loss from 138.29 to 136.55 at 100 epochs, while SGD ELR diverged, indicating Demon’s stability.
• Demon outperformed all other momentum and learning rate schedules, including OneCycle, step, linear, and exponential decay, across vision, NLP, and generative modeling tasks.