[Paper Review] Adaptive Stochastic Optimization

Frank E. Curtis, Katya Scheinberg|arXiv (Cornell University)|Jan 18, 2020

Stochastic Gradient Optimization Techniques27 references4 citations

TL;DR

This paper proposes an analytical framework for adaptive stochastic optimization methods that aim to reduce the need for manual hyperparameter tuning in large-scale machine learning. By leveraging adaptive stepsize strategies and variance reduction, the framework enables convergence with improved computational efficiency, achieving optimal complexity guarantees under stochastic conditions.

ABSTRACT

Optimization lies at the heart of machine learning and signal processing. Contemporary approaches based on the stochastic gradient method are non-adaptive in the sense that their implementation employs prescribed parameter values that need to be tuned for each application. This article summarizes recent research and motivates future work on adaptive stochastic optimization methods, which have the potential to offer significant computational savings when training large-scale systems.

Motivation & Objective

• Address the high computational cost and manual tuning burden associated with non-adaptive stochastic optimization in large-scale machine learning and signal processing.
• Develop a theoretical foundation for adaptive stochastic optimization that extends deterministic adaptive methods to the stochastic regime.
• Enable convergence rate guarantees for adaptive stochastic algorithms under realistic, non-iid data sampling conditions.
• Overcome the challenge of defining valid stopping times in the stochastic setting, which is critical for complexity analysis.
• Pave the way for practical, efficient optimization algorithms that dynamically adjust stepsizes and batch sizes without prior tuning.

Proposed method

• Introduce a novel analytical framework for adaptive stochastic optimization, inspired by deterministic adaptive methods but adapted to handle stochastic gradients.
• Use a stochastic trust-region-like approach with adaptive stepsize updates based on gradient estimates and error bounds.
• Define stopping criteria based on $ε$-stationarity, requiring $\|\nabla f(x_{k+1})\| \leq \varepsilon$, while accounting for random sampling in the gradient estimates.
• Employ a recursive stepsize update rule that adjusts based on the observed progress and error tolerance, using a single constant $\gamma$ for both increases and decreases.
• Integrate variance reduction techniques implicitly through the adaptive framework to control gradient noise and improve convergence stability.
• Analyze the algorithm under the assumption that gradient estimates are unbiased with bounded variance, ensuring convergence in expectation.

Experimental results

Research questions

• RQ1 Can an adaptive stochastic optimization framework be designed to achieve optimal convergence complexity comparable to deterministic methods?
• RQ2 How can stopping times be redefined in the stochastic regime to maintain theoretical guarantees when gradient estimates are random?
• RQ3 What are the conditions under which adaptive stochastic algorithms can achieve $\mathcal{O}(\varepsilon^{-3/2})$ complexity for $\varepsilon$-stationarity?
• RQ4 How can the framework be extended to handle complex subproblem solutions, such as those in trust-region or cubic regularization methods, under stochastic conditions?
• RQ5 What role do error bounds and adaptive parameter updates (e.g., different $\gamma$ for increase vs. decrease) play in improving practical performance and reducing tuning overhead?

Key findings

• The proposed analytical framework enables convergence rate guarantees for adaptive stochastic optimization, extending deterministic adaptive methods to the stochastic regime.
• The framework addresses the challenge of defining valid stopping times in the stochastic setting, which is critical for complexity analysis and practical implementation.
• Adaptive stochastic algorithms can achieve optimal complexity bounds—such as $\mathcal{O}(\varepsilon^{-3/2})$ for $\varepsilon$-stationarity—when properly designed and analyzed.
• Variance reduction techniques are implicitly embedded in the adaptive framework, improving convergence stability without requiring explicit batch-size tuning.
• The framework is extensible to advanced methods like TRACE and stochastic trust-region algorithms, though challenges remain in maintaining complexity guarantees under stochastic subproblem solutions.
• The use of adaptive stepsizes and dynamic batch sizes significantly reduces the need for manual hyperparameter tuning, offering substantial computational savings in large-scale ML applications.