[Paper Review] Finite-Time Convergence Rates of Nonlinear Two-Time-Scale Stochastic Approximation under Markovian Noise

Thinh T. Doan|arXiv (Cornell University)|Apr 4, 2021

Markov Chains and Monte Carlo Methods42 references4 citations

TL;DR

This paper establishes the finite-time convergence rate of nonlinear two-time-scale stochastic approximation under Markovian noise, proving an ${\cal O}(1/k^{2/3})$ convergence rate in expectation for mean square error. The analysis leverages Lyapunov functions and geometric mixing time to handle dependent data and bias, extending singular perturbation theory to non-asymptotic settings with Markov-driven sampling.

ABSTRACT

We study the so-called two-time-scale stochastic approximation, a simulation-based approach for finding the roots of two coupled nonlinear operators. Our focus is to characterize its finite-time performance in a Markov setting, which often arises in stochastic control and reinforcement learning problems. In particular, we consider the scenario where the data in the method are generated by Markov processes, therefore, they are dependent. Such dependent data result to biased observations of the underlying operators. Under some fairly standard assumptions on the operators and the Markov processes, we provide a formula that characterizes the convergence rate of the mean square errors generated by the method to zero. Our result shows that the method achieves a convergence in expectation at a rate $\mathcal{O}(1/k^{2/3})$, where $k$ is the number of iterations. Our analysis is mainly motivated by the classic singular perturbation theory for studying the asymptotic convergence of two-time-scale systems, that is, we consider a Lyapunov function that carefully characterizes the coupling between the two iterates. In addition, we utilize the geometric mixing time of the underlying Markov process to handle the bias and dependence in the data. Our theoretical result complements for the existing literature, where the rate of nonlinear two-time-scale stochastic approximation under Markovian noise is unknown.

Motivation & Objective

• To characterize the finite-time convergence performance of nonlinear two-time-scale stochastic approximation when data are generated by a Markov process.
• To address the challenge of dependent, biased observations arising from Markovian sampling in stochastic approximation algorithms.
• To derive a non-asymptotic convergence rate for the mean square error of the iterates under standard assumptions on the operators and Markov process.
• To extend singular perturbation theory to finite-time analysis in the presence of Markovian noise.
• To quantify the impact of step size selection and mixing time on convergence speed in two-time-scale SA.

Proposed method

• Uses a Lyapunov function to analyze the coupling between fast and slow iterates in the two-time-scale system.
• Incorporates the geometric mixing time of the underlying Markov process to quantify and control the bias from dependent samples.
• Employs a time-scale separation with $\beta_k \ll \alpha_k$, where $\alpha_k$ and $\beta_k$ are step sizes for the fast and slow iterates, respectively.
• Introduces a modified error process $\hat{z}_k = \hat{x}_k + \hat{y}_k$ to track the combined deviation of both iterates from their fixed points.
• Applies recursive inequalities and bounds on error terms using constants $B$, $\mu_F$, $\mu_G$, and mixing time $\tau(\alpha_k)$ to derive convergence bounds.
• Uses exponential weighting via $w_k$ to control the growth of error terms and derive uniform bounds on the expected squared error.

Experimental results

Research questions

• RQ1 What is the finite-time convergence rate of nonlinear two-time-scale stochastic approximation when the data are generated by a Markov process?
• RQ2 How does the geometric mixing time of the Markov process affect the bias and convergence of the algorithm?
• RQ3 Can the singular perturbation framework be extended to provide non-asymptotic convergence rates under Markovian sampling?
• RQ4 What step size choices ensure optimal convergence speed while handling dependent observations?
• RQ5 How do the coupling between fast and slow iterates and the bias from non-i.i.d. data jointly influence the mean square error?

Key findings

• The method achieves a finite-time convergence rate of ${\cal O}(1/k^{2/3})$ in expectation for the mean square error of the iterates.
• The convergence rate is derived under standard assumptions on the operators and the Markov process, including geometric ergodicity.
• The geometric mixing time of the Markov chain is explicitly used to control the bias introduced by dependent samples.
• The analysis shows that the error bound depends on the product of step sizes $\alpha_k\beta_k$, $\alpha_k\alpha_{k;\tau(\alpha_k)}$, and $\beta_k^2$, with coefficients depending on operator Lipschitz constants and fixed point norms.
• The Lyapunov function approach successfully captures the interplay between the two time scales and ensures stability and convergence.
• The result fills a critical gap in the literature by providing the first finite-time convergence rate for nonlinear two-time-scale SA under Markovian noise.