[Paper Review] AdaS: Adaptive Scheduling of Stochastic Gradients

Mahdi S. Hosseini, Konstantinos N. Plataniotis|arXiv (Cornell University)|Jun 11, 2020

Stochastic Gradient Optimization Techniques29 references10 citations

TL;DR

AdaS proposes an adaptive learning rate scheduling method for stochastic gradient descent (SGD) that dynamically adjusts the learning rate based on the rate of change in 'knowledge gain'—a metric derived from the low-rank factorization of convolutional layer weights. By monitoring knowledge gain and mapping condition across layers, AdaS achieves faster convergence and superior generalization compared to adaptive optimizers like Adam, without requiring a validation set for early stopping.

ABSTRACT

The choice of step-size used in Stochastic Gradient Descent (SGD) optimization is empirically selected in most training procedures. Moreover, the use of scheduled learning techniques such as Step-Decaying, Cyclical-Learning, and Warmup to tune the step-size requires extensive practical experience--offering limited insight into how the parameters update--and is not consistent across applications. This work attempts to answer a question of interest to both researchers and practitioners, namely extit{"how much knowledge is gained in iterative training of deep neural networks?"} Answering this question introduces two useful metrics derived from the singular values of the low-rank factorization of convolution layers in deep neural networks. We introduce the notions of extit{"knowledge gain"} and extit{"mapping condition"} and propose a new algorithm called Adaptive Scheduling (AdaS) that utilizes these derived metrics to adapt the SGD learning rate proportionally to the rate of change in knowledge gain over successive iterations. Experimentation reveals that, using the derived metrics, AdaS exhibits: (a) faster convergence and superior generalization over existing adaptive learning methods; and (b) lack of dependence on a validation set to determine when to stop training. Code is available at \url{https://github.com/mahdihosseini/AdaS}.

Motivation & Objective

• To address the limitations of fixed and scheduled learning rates in SGD, which often lead to poor convergence and generalization.
• To introduce new metrics—knowledge gain and mapping condition—that quantify the quality of learning within individual convolutional layers.
• To develop an adaptive learning rate scheduling framework that improves convergence speed and generalization without relying on validation data.
• To reduce dependency on manual hyperparameter tuning by enabling automatic, layer-specific learning rate adaptation.

Proposed method

• The method computes low-rank factorizations of convolutional layer weight matrices to extract singular values and derive the 'knowledge gain' metric, representing the rate of information accumulation in the network.
• The 'mapping condition' is defined as the ratio of the largest to smallest singular values in the factorization, indicating the conditioning of the layer's input-output transformation.
• The learning rate is scheduled adaptively based on the rate of change in knowledge gain across iterations, ensuring larger updates when knowledge gain increases rapidly.
• The algorithm applies independent learning rate scheduling per convolutional block, enabling fine-grained adaptation to layer-specific learning dynamics.
• Knowledge gain and mapping condition are monitored throughout training to guide learning rate adjustments and determine training termination.
• The approach is compatible with SGD with momentum and can be integrated with existing adaptive optimizers like Adam for enhanced robustness.

Experimental results

Research questions

• RQ1 How can we quantify the amount of knowledge gained during iterative training of deep neural networks?
• RQ2 What role does the conditioning of the weight transformation (mapping condition) play in generalization and convergence?
• RQ3 Can learning rate scheduling based on knowledge gain improve convergence speed and test performance compared to fixed or heuristic schedules?
• RQ4 Is it possible to eliminate the need for a validation set in early stopping by using intrinsic training metrics?
• RQ5 How does adaptive learning rate scheduling based on layer-specific knowledge gain affect generalization across different optimizers?

Key findings

• AdaS achieved a test accuracy of 95.16% on CIFAR-10 using ResNet-18, outperforming Adam (93.22%) and SGD with cyclical learning rate (94.13%).
• AdaS reduced training loss to 8.69e-4, significantly lower than Adam (5.66e-3) and RMSProp (6.35e-3), indicating faster convergence.
• Knowledge gain in AdaS reached 0.2938 with β=0.95, showing higher information accumulation compared to Adam (0.2973) and RMSProp (0.3011), indicating more effective learning.
• The mapping condition κ in AdaS remained below 9 for all settings, suggesting better-conditioned layer transformations compared to Adam (18.484), which correlates with improved generalization.
• AdaS achieved superior generalization without requiring a validation set, as training termination was determined directly from training loss using the derived metrics.
• The method demonstrated robustness to initial learning rate choice and can be combined with adaptive optimizers like Adam for further performance gains.