[Paper Review] Reinforced stochastic gradient descent for deep neural network learning

Haiping Huang, Taro Toyoizumi|arXiv (Cornell University)|Jan 27, 2017

Stochastic Gradient Optimization Techniques26 references3 citations

TL;DR

This paper proposes Reinforced Stochastic Gradient Descent (R-SGD), a novel SGD variant that stochastically reinforces current gradients with accumulated past gradients using a time-increasing probability. R-SGD significantly accelerates learning, reduces overfitting, and achieves generalization performance comparable to Adam while using half the memory, outperforming standard SGD and adaptive methods on both synthetic and MNIST benchmarks.

ABSTRACT

Stochastic gradient descent (SGD) is a standard optimization method to minimize a training error with respect to network parameters in modern neural network learning. However, it typically suffers from proliferation of saddle points in the high-dimensional parameter space. Therefore, it is highly desirable to design an efficient algorithm to escape from these saddle points and reach a parameter region of better generalization capabilities. Here, we propose a simple extension of SGD, namely reinforced SGD, which simply adds previous first-order gradients in a stochastic manner with a probability that increases with learning time. As verified in a simple synthetic dataset, this method significantly accelerates learning compared with the original SGD. Surprisingly, it dramatically reduces over-fitting effects, even compared with state-of-the-art adaptive learning algorithm---Adam. For a benchmark handwritten digits dataset, the learning performance is comparable to Adam, yet with an extra advantage of requiring one-fold less computer memory. The reinforced SGD is also compared with SGD with fixed or adaptive momentum parameter and Nesterov's momentum, which shows that the proposed framework is able to reach a similar generalization accuracy with less computational costs. Overall, our method introduces stochastic memory into gradients, which plays an important role in understanding how gradient-based training algorithms can work and its relationship with generalization abilities of deep networks.

Motivation & Objective

• To address the challenge of SGD getting trapped in saddle points and plateaus in high-dimensional loss landscapes during deep neural network training.
• To improve generalization performance and convergence speed without increasing computational cost or memory usage.
• To explore whether stochastic incorporation of historical gradient information can enhance exploration and escape from poor local minima.
• To compare R-SGD's performance against state-of-the-art adaptive optimizers like Adam and momentum-based methods in terms of accuracy and efficiency.
• To investigate the relationship between gradient memory and generalization in deep learning through empirical evaluation.

Proposed method

• R-SGD introduces a time-dependent reinforcement probability that increases with training steps, stochastically combining the current gradient with a weighted sum of past gradients.
• The method uses a probabilistic mechanism to decide whether to apply the reinforced gradient (current + past) or only the current gradient at each update step.
• The reinforcement is implemented by adding a scaled version of accumulated past gradients to the current gradient with a probability that grows over time.
• The algorithm maintains a running average of past gradients and applies them stochastically, introducing controlled memory into the optimization process.
• The framework is evaluated on a fully-connected deep network with sigmoid activations and tested on both synthetic non-linear mapping and MNIST digit classification tasks.
• The method is compared with vanilla SGD, Adam, and momentum-based SGD variants using training/test loss and generalization accuracy as metrics.

Experimental results

Research questions

• RQ1 Can stochastic reinforcement of past gradients improve convergence speed and generalization in deep neural network training?
• RQ2 Does R-SGD outperform Adam in terms of generalization while using less memory?
• RQ3 How does the inclusion of historical gradient information affect the optimization trajectory in high-dimensional, non-convex loss landscapes?
• RQ4 Can R-SGD effectively escape saddle points and plateaus that hinder standard SGD?
• RQ5 How does the performance of R-SGD compare to momentum-based and adaptive methods across different initialization conditions?

Key findings

• On a synthetic non-linear mapping task, R-SGD significantly accelerates learning and reduces overfitting compared to standard SGD and Adam.
• On the MNIST benchmark, R-SGD achieves generalization performance comparable to Adam, but with one-fold less memory usage.
• R-SGD reaches similar generalization accuracy as Adam and SGD with adaptive momentum, but with lower computational cost.
• Bilinear interpolation analysis shows that R-SGD explores a sharper subspace in weight space than Adam, suggesting different generalization properties.
• The method outperforms SGD with fixed or adaptive momentum and Nesterov’s momentum in terms of convergence speed and generalization.
• R-SGD's performance remains robust across different random initializations, indicating consistent optimization behavior.