[Paper Review] kDecay: Just adding k-decay items on Learning-Rate Schedule to improve Neural Networks

Tao Zhang, Wei Li|arXiv (Cornell University)|Apr 13, 2020

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes k-decay, a simple yet effective method to improve deep neural network training by modifying the learning rate (LR) schedule using the k-th order derivative of the LR function. By adding a k-decay term that increases the rate of change of the LR—especially in later training stages—it boosts model accuracy without additional computational cost, achieving up to 2.07% accuracy gain on CIFAR-100 and 1.25% on ImageNet.

ABSTRACT

Recent work has shown that optimizing the Learning Rate (LR) schedule can be a very accurate and efficient way to train deep neural networks. We observe that the rate of change (ROC) of LR has correlation with the training process, but how to use this relationship to control the training to achieve the purpose of improving accuracy? We propose a new method, k-decay, just add an extra item to the commonly used and easy LR schedule(exp, cosine and polynomial), is effectively improves the performance of these schedule, also better than the state-of-the-art algorithms of LR shcedule such as SGDR, CLR and AutoLRS. In the k-decay, by adjusting the hyper-parameter \(k\), to generate different LR schedule, when k increases, the performance is improved. We evaluate the k-decay method on CIFAR And ImageNet datasets with different neural networks (ResNet, Wide ResNet). Our experiments show that this method can improve on most of them. The accuracy has been improved by 1.08\% on the CIFAR-10 dataset and by 2.07 \% on the CIFAR-100 dataset. On the ImageNet, accuracy is improved by 1.25\%. Our method is not only a general method to be applied other LR Shcedule, but also has no additional computational cost.

Motivation & Objective

• To improve deep neural network performance by rethinking the learning rate (LR) schedule beyond standard decay methods.
• To investigate how the rate of change (ROC) of the LR correlates with training dynamics and model accuracy.
• To develop a general, low-cost method that enhances existing LR schedules without increasing hyperparameter complexity.
• To demonstrate that controlling the ROC of the LR through higher-order derivatives leads to consistent performance gains across datasets and architectures.

Proposed method

• The method introduces a k-decay term by modifying the k-th order derivative of the base LR schedule function η(t), adding an increment Δf^(k)(t) to η^(k)(t).
• The new LR schedule is derived by solving the modified k-th order derivative equation, resulting in a new function η'(t) = η_o(k,t) that depends on the hyperparameter k.
• The approach is applied as an additional term to standard LR schedules such as polynomial, cosine, and exponential decay, enhancing their performance without altering their core structure.
• The hyperparameter k controls the ROC of the LR: increasing k increases the rate of change of the LR in later training stages, accelerating convergence near the end.
• The method is mathematically grounded in calculus, with the k-decay term derived via analytical integration of the modified k-th derivative.
• The approach is computationally efficient, requiring no additional forward or backward passes, making it applicable to any existing LR schedule.

Experimental results

Research questions

• RQ1 How does the rate of change (ROC) of the learning rate correlate with model training performance, especially in later training stages?
• RQ2 Can improving the ROC of the learning rate through higher-order derivatives lead to consistent accuracy gains across different datasets and architectures?
• RQ3 How does the hyperparameter k influence the performance of the k-decay method, and what is its optimal range for different models?
• RQ4 Can k-decay be generalized to various existing LR schedules (e.g., polynomial, cosine, exponential) without increasing computational cost?
• RQ5 Does k-decay outperform state-of-the-art adaptive LR scheduling methods like SGDR, CLR, and AutoLRS in terms of accuracy and training stability?

Key findings

• k-decay improves top-1 accuracy by 1.25% on ImageNet using ResNet-50, outperforming SGDR, AutoLRS, and CLR by 1.03%, 0.96%, and 0.95% respectively.
• On CIFAR-100, k-decay achieves a 2.07% accuracy improvement over the base polynomial schedule, with a top-1 error rate of 23.11% when k=2.0.
• On CIFAR-10, the method improves accuracy by 1.08% compared to the base polynomial schedule, demonstrating consistent gains across datasets.
• The test error rate decreases with increasing k up to a threshold k_v, after which performance degrades, indicating model-specific sensitivity to k.
• Deeper models (e.g., ResNet-101) have a lower k_v threshold (3) than shallower ones (e.g., ResNet-47, k_v≈7), showing increased sensitivity to high k values.
• Loss curves show that higher k values lead to slower convergence in early stages but faster final convergence due to steeper LR decay, resulting in lower final loss and better generalization.