[Paper Review] Accelerated Stochastic ADMM with Variance Reduction

Chao Zhang, Zebang Shen+4 more|arXiv (Cornell University)|Nov 13, 2016

Sparse and Compressive Sensing Techniques21 references3 citations

TL;DR

This paper proposes Accelerated SVRG-ADMM (ASVRG-ADMM), a novel stochastic ADMM algorithm that combines variance reduction with Nesterov-style acceleration to achieve an optimal O(1/N²) convergence rate with respect to the smoothness constant L_f. The method uses auxiliary variables and adaptive momentum to accelerate convergence in large-scale empirical risk minimization problems.

ABSTRACT

Alternating Direction Method of Multipliers (ADMM) is a popular method for solving large-scale Machine Learning problems. Stochastic ADMM was proposed to reduce the per iteration computational complexity, which is more suitable for big data problems. Recently, variance reduction techniques have been integrated with stochastic ADMM in order to get a faster convergence rate, such as SAG-ADMM and SVRG-ADMM. However, their convergence rate is still suboptimal w.r.t the smoothness constant. In this paper, we propose an accelerated stochastic ADMM algorithm with variance reduction, which enjoys a faster convergence than all the existing stochastic ADMM algorithms. We theoretically analyse its convergence rate and show its dependence on the smoothness constant is optimal. We also empirically validate its effectiveness and show its priority over other stochastic ADMM algorithms.

Motivation & Objective

• Address the suboptimal convergence rate of existing stochastic ADMM methods, particularly their O(1/√N) rate, which limits performance on large datasets.
• Improve convergence by integrating variance reduction techniques (like SVRG) with Nesterov-style acceleration to achieve faster convergence with respect to the smoothness constant L_f.
• Develop a method that maintains low per-iteration cost while achieving optimal convergence rates, especially critical when L_f is large due to big data.
• Theoretical and empirical validation is conducted to demonstrate superiority over SAG-ADMM, SVRG-ADMM, and deterministic accelerated ADMM in big data settings.

Proposed method

• Proposes ASVRG-ADMM, a stochastic ADMM variant that combines variance reduction (SVRG) with Nesterov-type acceleration via auxiliary variables to improve convergence.
• Uses a momentum-based update rule with adaptive weights α₂,s and α₃,s that evolve over iterations to accelerate convergence while maintaining stability.
• Introduces a linearized augmented Lagrangian approach to simplify x-update, enabling efficient computation by precomputing (L̄I + β₁AᵀA)⁻¹.
• Employs a hybrid update strategy where the x-update uses a combination of the current iterate, a previous averaged iterate, and a momentum term to reduce variance and accelerate convergence.
• The algorithm maintains a set of auxiliary points ỹ_s and ỹ_s^ag to track the momentum and variance-reduced gradient estimates, improving convergence speed.
• The step size parameters η_s and ρ_s are dynamically adjusted to increase the effective step size over time, enhancing convergence rate.

Experimental results

Research questions

• RQ1 Can variance reduction and Nesterov acceleration be effectively combined in a stochastic ADMM framework to achieve faster convergence than existing methods?
• RQ2 Does the proposed ASVRG-ADMM achieve an optimal convergence rate of O(1/N²) with respect to the smoothness constant L_f, as in deterministic accelerated methods?
• RQ3 How does the algorithm perform in practice on large-scale empirical risk minimization problems with high L_f values, such as those arising from big datasets?
• RQ4 Can the algorithm maintain low per-iteration computational cost while achieving faster convergence compared to SAG-ADMM and SVRG-ADMM?

Key findings

• ASVRG-ADMM achieves an O(1/N²) convergence rate with respect to the smoothness constant L_f, which is optimal and matches the best deterministic accelerated methods.
• When L_f is large (e.g., on the covtype dataset with L_f ≈ 2.02×10⁷), ASVRG-ADMM converges significantly faster than SAG-ADMM and SVRG-ADMM, even after 50 effective passes of data.
• On smaller datasets (a9a, w8a, ijcnn1) with lower L_f, ASVRG-ADMM converges faster than both baselines within 20 effective passes, demonstrating consistent superiority.
• The algorithm's performance is robust across different data distributions and scales, with objective values decreasing more rapidly than competing methods across all four datasets tested.
• The method maintains low computational overhead by precomputing matrix inverses and using adaptive step sizes, ensuring scalability to large-scale problems.
• When m=1, ASVRG-ADMM reduces to the deterministic accelerated ADMM of Ouyang et al. (2015), confirming consistency with prior work.