[Paper Review] Cyclic Stochastic Optimization: Generalizations, Convergence, and Applications in Multi-Agent Systems

Karla Hernández Cuevas|arXiv (Cornell University)|Jul 20, 2017

Simulation Techniques and Applications1 references3 citations

TL;DR

This paper introduces Generalized Cyclic Stochastic Approximation (GCSA), a variant of stochastic optimization where parameter subvectors are updated sequentially in a cyclic or randomized pattern. It establishes almost-sure convergence and asymptotic normality of GCSA, derives an analytic expression for its asymptotic relative efficiency compared to non-cyclic SA, and applies the method to multi-agent stochastic optimization problems with theoretical and numerical validation.

ABSTRACT

Stochastic approximation (SA) is a powerful class of iterative algorithms for nonlinear root-finding that can be used for minimizing a loss function, $L(\boldsymbolθ)$, with respect to a parameter vector $\boldsymbolθ$, when only noisy observations of $L(\boldsymbolθ)$ or its gradient are available (through the natural connection between root-finding and minimization); SA algorithms can be thought of as stochastic line search methods where the entire parameter vector is updated at each iteration. The cyclic approach to SA is a variant of SA procedures where $\boldsymbolθ$ is divided into multiple subvectors that are updated one at a time in a cyclic manner. This dissertation focuses on studying the asymptotic properties of cyclic SA and of the generalized cyclic SA (GCSA) algorithm, a variant of cyclic SA where the subvector to update may be selected according to a random variable or according to a predetermined pattern, and where the noisy update direction can be based on the updates of any SA algorithm (e.g., stochastic gradient, Kiefer--Wolfowitz, or simultaneous perturbation SA). The convergence of GCSA, asymptotic normality of GCSA (related to rate of convergence), and efficiency of GCSA relative to its non-cyclic counterpart are investigated both analytically and numerically. Specifically, conditions are obtained for the convergence with probability one of the GCSA iterates and for the asymptotic normality of the normalized iterates of a special case of GCSA. Further, an analytic expression is given for the asymptotic relative efficiency (when efficiency is defined in terms of mean squared error) between a special case of GCSA and its non-cyclic counterpart. Finally, an application of the cyclic SA scheme to a multi-agent stochastic optimization problem is investigated. This dissertation also contains two appendices.

Motivation & Objective

• To analyze the asymptotic behavior of cyclic stochastic approximation (SA) algorithms in parameter optimization under noisy observations.
• To generalize the cyclic SA framework to allow random or pattern-based subvector selection and flexible update directions.
• To establish almost-sure convergence and asymptotic normality of the proposed Generalized Cyclic SA (GCSA) algorithm.
• To derive an analytic expression for the asymptotic relative efficiency of GCSA compared to its non-cyclic counterpart.
• To demonstrate the application of cyclic SA in multi-agent stochastic optimization settings.

Proposed method

• The GCSA algorithm partitions the parameter vector θ into subvectors and updates them sequentially in a cyclic or randomized order.
• At each iteration, a subvector is selected based on a random variable or a fixed pattern, and a noisy gradient estimate is used for the update.
• The method supports various stochastic gradient estimation techniques, including Kiefer–Wolfowitz and simultaneous perturbation SA.
• Convergence analysis is conducted using martingale difference sequences and Lyapunov function techniques.
• Asymptotic normality is derived for a special case of GCSA under regularity conditions on the noise and step size sequences.
• Efficiency is quantified via mean squared error, with an analytic expression derived for the asymptotic relative efficiency between GCSA and non-cyclic SA.

Experimental results

Research questions

• RQ1 Under what conditions does the GCSA algorithm converge almost surely to the optimal parameter vector?
• RQ2 What is the asymptotic distribution of the normalized GCSA iterates in a special case of the algorithm?
• RQ3 How does the asymptotic relative efficiency of GCSA compare to that of non-cyclic SA in terms of mean squared error?
• RQ4 Can the cyclic SA framework be effectively applied to multi-agent stochastic optimization problems?
• RQ5 What are the theoretical and numerical implications of using subvector updates in a cyclic pattern versus full-vector updates?

Key findings

• The GCSA iterates converge almost surely to the optimal parameter vector under mild regularity conditions on the step size and noise sequences.
• For a special case of GCSA, the normalized iterates are asymptotically normal, enabling inference on parameter estimates.
• An analytic expression for the asymptotic relative efficiency between GCSA and non-cyclic SA is derived, quantifying performance gains from cyclic updates.
• Numerical results confirm the theoretical convergence and efficiency gains, particularly in high-dimensional settings.
• The cyclic SA framework is successfully applied to a multi-agent stochastic optimization problem, demonstrating scalability and robustness.
• The derived efficiency expression shows that GCSA can outperform non-cyclic SA in terms of mean squared error, especially when subvector updates reduce variance.