[Paper Review] Fast Stochastic Variance Reduced ADMM for Stochastic Composition Optimization

Yue Yu, Longbo Huang|arXiv (Cornell University)|May 11, 2017

Stochastic Gradient Optimization Techniques12 references3 citations

TL;DR

This paper proposes com-SVR-ADMM, the first ADMM-based algorithm for stochastic composition optimization with linear constraints, leveraging stochastic variance reduction to achieve linear convergence for strongly convex, Lipschitz smooth problems and improved rates of $O(\log S/S)$ for convex cases, outperforming prior methods in convergence speed and robustness.

ABSTRACT

We consider the stochastic composition optimization problem proposed in \cite{wang2017stochastic}, which has applications ranging from estimation to statistical and machine learning. We propose the first ADMM-based algorithm named com-SVR-ADMM, and show that com-SVR-ADMM converges linearly for strongly convex and Lipschitz smooth objectives, and has a convergence rate of $O( \log S/S)$, which improves upon the $O(S^{-4/9})$ rate in \cite{wang2016accelerating} when the objective is convex and Lipschitz smooth. Moreover, com-SVR-ADMM possesses a rate of $O(1/\sqrt{S})$ when the objective is convex but without Lipschitz smoothness. We also conduct experiments and show that it outperforms existing algorithms.

Motivation & Objective

• To address the lack of efficient, scalable algorithms for stochastic composition optimization with linear constraints in machine learning and estimation.
• To extend variance reduction techniques—previously used in standard stochastic optimization—to the ADMM framework for composition objectives.
• To develop a method that achieves faster convergence than existing sublinear-rate algorithms, particularly in convex and strongly convex settings.
• To incorporate regularization and constraints naturally through ADMM, enabling broader applicability to real-world problems like risk-averse learning and dynamic programming.

Proposed method

• Proposes com-SVR-ADMM, an ADMM-based algorithm that integrates stochastic variance reduction (SVR) into the augmented Lagrangian framework for stochastic composition problems.
• Uses a two-loop structure: an outer loop for updating dual variables and an inner loop for variance-reduced gradient estimation of the composite objective.
• Applies SVR to reduce the variance of stochastic gradients in both the inner and outer functions, improving convergence stability and speed.
• Introduces a novel Lyapunov function analysis to bound the expected suboptimality and constraint violation, leveraging the structure of the augmented Lagrangian.
• Employs adaptive step-sizes and a recursive averaging scheme to maintain convergence under non-i.i.d. sampling and finite-sample settings.
• Derives convergence guarantees by analyzing the decrease in expected primal-dual gap and constraint violation over iterations.

Experimental results

Research questions

• RQ1 Can variance reduction techniques be effectively adapted to the ADMM framework for stochastic composition problems?
• RQ2 What convergence rate can be achieved by combining SVR with ADMM in the context of stochastic composition optimization?
• RQ3 How does com-SVR-ADMM compare to existing sublinear-rate algorithms in terms of convergence speed and robustness?
• RQ4 Can the algorithm handle convex objectives without Lipschitz smoothness, and what rate is achievable in that case?
• RQ5 Does the inclusion of linear constraints and regularization improve the practical performance on real-world problems like risk-averse learning or dynamic programming?

Key findings

• com-SVR-ADMM achieves linear convergence for strongly convex and Lipschitz smooth objectives, a significant improvement over prior sublinear rates.
• For convex and Lipschitz smooth objectives, the convergence rate is $O(\log S / S)$, which improves upon the $O(S^{-4/9})$ rate in Wang et al. (2016b).
• When the objective is convex but not Lipschitz smooth, the algorithm achieves a convergence rate of $O(1/\sqrt{S})$.
• Empirical results show com-SVR-ADMM outperforms existing algorithms in both convergence speed and solution quality across multiple benchmark problems.
• The algorithm maintains stability and fast convergence even under non-uniform sampling and finite-sample settings, demonstrating robustness.
• Theoretical analysis confirms that constraint violation and suboptimality decay at the stated rates, validated by the Lyapunov function and recursive bounding techniques.