[Paper Review] Nonlinear Two-Time-Scale Stochastic Approximation: Convergence and Finite-Time Performance

Thinh T. Doan|arXiv (Cornell University)|Nov 3, 2020

Advanced Thermodynamics and Statistical Mechanics4 citations

TL;DR

This paper establishes the finite-time convergence rate of nonlinear two-time-scale stochastic approximation under standard assumptions, showing that the mean square error decays at a rate of $\mathcal{O}(1/k^{2/3})$ by leveraging a Lyapunov function inspired by singularly perturbed systems theory. The analysis carefully balances fast and slow time-scale step sizes to manage coupling between iterates, providing a rigorous performance guarantee for a widely used class of stochastic optimization algorithms.

ABSTRACT

Two-time-scale stochastic approximation, a generalized version of the popular stochastic approximation, has found broad applications in many areas including stochastic control, optimization, and machine learning. Despite its popularity, theoretical guarantees of this method, especially its finite-time performance, are mostly achieved for the linear case while the results for the nonlinear counterpart are very sparse. Motivated by the classic control theory for singularly perturbed systems, we study in this paper the asymptotic convergence and finite-time analysis of the nonlinear two-time-scale stochastic approximation. Under some fairly standard assumptions, we provide a formula that characterizes the rate of convergence of the main iterates to the desired solutions. In particular, we show that the method achieves a convergence in expectation at a rate $\\mathcal{O}(1/k^{2/3})$, where $k$ is the number of iterations. The key idea in our analysis is to properly choose the two step sizes to characterize the coupling between the fast and slow-time-scale iterates.

Motivation & Objective

• To close the theoretical gap in finite-time performance guarantees for nonlinear two-time-scale stochastic approximation, which previously lacked results beyond the linear case.
• To derive an explicit convergence rate formula for the mean square error of the iterates in solving coupled nonlinear stochastic equations.
• To understand how the choice of two distinct step sizes—fast and slow—impacts the convergence speed and stability of the algorithm.
• To extend classical singular perturbation theory from control to stochastic approximation, enabling analysis of coupled fast-slow dynamics under noise.
• To provide a theoretical foundation for applications in reinforcement learning and distributed optimization where two-time-scale methods are prevalent.

Proposed method

• A Lyapunov function is constructed to analyze the coupled dynamics of fast and slow iterates, capturing the interaction between the two time scales.
• The analysis assumes standard regularity conditions on the operators F and G, including Lipschitz continuity and strong monotonicity.
• Step sizes $\alpha_k$ and $\beta_k$ are chosen such that $\beta_k \ll \alpha_k$, with $\alpha_k = \Theta(k^{-1/3})$ and $\beta_k = \Theta(k^{-2/3})$ to balance convergence speed and noise averaging.
• A recursive inequality is derived for the expected squared error of the estimation error vector, which is then bounded using product terms and exponential inequalities.
• The method uses the inequality $1 + x \leq \exp(x)$ and its dual $1 + x \geq \exp(-x)$ to control product terms arising from the recursive error bound.
• The final convergence rate is derived by summing the error recursion and applying integral test bounds on the step size sequences.

Experimental results

Research questions

• RQ1 What is the finite-time convergence rate of nonlinear two-time-scale stochastic approximation under standard assumptions?
• RQ2 How do the relative magnitudes of the fast and slow time-scale step sizes affect the convergence speed and stability of the algorithm?
• RQ3 Can the convergence rate of the nonlinear two-time-scale SA be explicitly characterized, and if so, what is the functional form of this rate?
• RQ4 To what extent can techniques from singularly perturbed systems theory be adapted to analyze nonlinear stochastic approximation with coupled dynamics?
• RQ5 Is it possible to achieve a faster convergence rate than $\mathcal{O}(1/k^{2/3})$ in the nonlinear case, as seen in the linear case?

Key findings

• The mean square error of the two-time-scale stochastic approximation algorithm converges to zero at a rate of $\mathcal{O}(1/k^{2/3})$, where $k$ is the number of iterations.
• This convergence rate is achieved when the fast-time-scale step size $\alpha_k$ scales as $\Theta(k^{-1/3})$ and the slow-time-scale step size $\beta_k$ scales as $\Theta(k^{-2/3})$.
• The analysis establishes that the coupling between fast and slow iterates is effectively managed through careful step size selection, ensuring stability and convergence.
• The Lyapunov-based approach successfully bounds the error propagation across iterations, even in the presence of noisy oracle feedback.
• The derived convergence rate is tight under the given assumptions and matches the best-known rate in the linear two-time-scale case, though the nonlinear case requires fundamentally different analysis.
• The results provide a theoretical justification for the use of two-time-scale methods in reinforcement learning and distributed optimization, where such algorithms are commonly deployed.