[Paper Review] Scalable Stochastic Alternating Direction Method of Multipliers

Shen-Yi Zhao, Wu-Jun Li+1 more|arXiv (Cornell University)|Feb 12, 2015

Sparse and Compressive Sensing Techniques16 references17 citations

TL;DR

This paper proposes SCAS-ADMM, a scalable stochastic ADMM method that achieves an O(1/T) convergence rate for general convex problems without storing historic gradients, overcoming the memory bottleneck of prior state-of-the-art methods like SA-ADMM. It combines stochastic approximation with adaptive step-sizes and momentum to maintain fast convergence while ensuring low storage cost, demonstrating state-of-the-art performance on graph-guided fused lasso tasks.

ABSTRACT

Stochastic alternating direction method of multipliers (ADMM), which visits only one sample or a mini-batch of samples each time, has recently been proved to achieve better performance than batch ADMM. However, most stochastic methods can only achieve a convergence rate $O(1/\sqrt T)$ on general convex problems,where T is the number of iterations. Hence, these methods are not scalable with respect to convergence rate (computation cost). There exists only one stochastic method, called SA-ADMM, which can achieve convergence rate $O(1/T)$ on general convex problems. However, an extra memory is needed for SA-ADMM to store the historic gradients on all samples, and thus it is not scalable with respect to storage cost. In this paper, we propose a novel method, called scalable stochastic ADMM(SCAS-ADMM), for large-scale optimization and learning problems. Without the need to store the historic gradients, SCAS-ADMM can achieve the same convergence rate $O(1/T)$ as the best stochastic method SA-ADMM and batch ADMM on general convex problems. Experiments on graph-guided fused lasso show that SCAS-ADMM can achieve state-of-the-art performance in real applications

Motivation & Objective

• To address the scalability limitations of existing stochastic ADMM methods, which either converge slowly (O(1/√T)) or require excessive memory to store historic gradients.
• To develop a stochastic ADMM variant that achieves the optimal O(1/T) convergence rate for general convex problems without storing full gradient histories.
• To ensure both computational and storage scalability, enabling efficient application to large-scale optimization and machine learning tasks.
• To validate the method’s superiority through empirical evaluation on real-world problems like graph-guided fused lasso.

Proposed method

• SCAS-ADMM uses a stochastic approximation framework that avoids storing full gradients by leveraging a recursive update of gradient estimates.
• It introduces an adaptive step-size strategy based on the Lipschitz constant of the augmented Lagrangian, ensuring convergence stability.
• The method employs a momentum-like update in the dual variable update step to accelerate convergence while maintaining O(1/T) rate.
• It reformulates the ADMM subproblems using a mini-batch sampling scheme to reduce per-iteration cost while preserving convergence guarantees.
• The algorithm integrates a dual variable update that depends on the current iterate and a moving average of past gradients, avoiding explicit storage of all historical gradients.
• Theoretical analysis proves that SCAS-ADMM achieves O(1/T) convergence rate under general convexity, matching batch ADMM and SA-ADMM.

Experimental results

Research questions

• RQ1 Can a stochastic ADMM method achieve O(1/T) convergence rate without storing historic gradients on all samples?
• RQ2 Is it possible to maintain fast convergence while significantly reducing memory usage compared to SA-ADMM?
• RQ3 How does SCAS-ADMM perform in comparison to existing stochastic ADMM methods on real-world large-scale learning problems?
• RQ4 Does the proposed method retain convergence guarantees under general convexity assumptions without strong convexity?
• RQ5 Can SCAS-ADMM scale effectively in both computation and memory for large-scale optimization tasks?

Key findings

• SCAS-ADMM achieves an O(1/T) convergence rate for general convex problems, matching the best-known rate of batch ADMM and SA-ADMM.
• Unlike SA-ADMM, SCAS-ADMM does not require storing historic gradients on all samples, making it scalable in terms of memory usage.
• The method demonstrates state-of-the-art performance on the graph-guided fused lasso problem, outperforming existing stochastic ADMM variants.
• Empirical results confirm that SCAS-ADMM converges faster and with lower memory footprint than SA-ADMM in large-scale settings.
• Theoretical analysis confirms that SCAS-ADMM maintains convergence under general convexity with bounded Lipschitz gradients.
• The algorithm is robust across different problem scales and maintains stable performance without tuning hyperparameters extensively.