[Paper Review] Accelerated Proximal Stochastic Dual Coordinate Ascent for Regularized Loss Minimization

Shai Shalev‐Shwartz, Tong Zhang|arXiv (Cornell University)|Sep 10, 2013

Stochastic Gradient Optimization Techniques18 citations

TL;DR

This paper proposes an accelerated proximal stochastic dual coordinate ascent (AP-SDCA) algorithm for regularized loss minimization in machine learning. By combining proximal updates with an inner-outer iteration scheme, it achieves improved convergence rates—nearly linear in data size for well-conditioned problems and significantly faster than prior methods for ill-conditioned cases—across SVM, logistic regression, ridge regression, Lasso, and multiclass SVM.

ABSTRACT

We introduce a proximal version of the stochastic dual coordinate ascent method and show how to accelerate the method using an inner-outer iteration procedure. We analyze the runtime of the framework and obtain rates that improve state-of-the-art results for various key machine learning optimization problems including SVM, logistic regression, ridge regression, Lasso, and multiclass SVM. Experiments validate our theoretical findings.

Motivation & Objective

• Address the need for faster optimization algorithms in large-scale machine learning with regularized loss minimization.
• Extend the stochastic dual coordinate ascent (SDCA) framework to handle general strongly convex regularizers and smooth vector-valued loss functions.
• Develop an accelerated variant that achieves faster convergence than state-of-the-art methods like SGD and AGD.
• Provide theoretical runtime guarantees with explicit dependence on problem condition number, data size, and accuracy.
• Enable application to non-smooth regularizers (e.g., L1) via small L2 regularization and smoothing techniques.

Proposed method

• Introduce a proximal version of stochastic dual coordinate ascent (SDCA) to handle general strongly convex regularizers beyond the squared norm.
• Use an inner-outer iteration procedure to accelerate convergence, where the outer loop maintains dual variables and the inner loop performs proximal updates.
• Establish a duality gap analysis using the expected sub-optimality of the dual objective, leveraging strong convexity and smoothness assumptions.
• Apply Markov's inequality and union bounds to derive high-probability convergence guarantees with confidence parameter δ.
• Use smoothing techniques for non-smooth loss functions (e.g., hinge loss in SVM) to achieve improved runtime in terms of ε and λ.
• Derive runtime bounds by analyzing the decay rate of dual sub-optimality, showing geometric convergence under appropriate conditions.

Experimental results

Research questions

• RQ1 Can the SDCA framework be extended to handle general strongly convex regularizers, including non-smooth ones like L1?
• RQ2 What is the optimal convergence rate for regularized loss minimization when the problem condition number is large?
• RQ3 How can acceleration be achieved in the dual coordinate ascent framework for both smooth and non-smooth loss functions?
• RQ4 Can the algorithm achieve nearly linear time complexity in data size for well-conditioned problems?
• RQ5 What is the trade-off between convergence speed and accuracy when using randomized sampling and averaging in the dual space?

Key findings

• The proposed AP-SDCA algorithm achieves a runtime of Õ(d(n + min{1/(λγ), √(n/(λγ))})) for smooth, strongly convex problems, improving over SGD and AGD.
• For SVM with hinge loss, the algorithm achieves Õ(d(n + min{1/(λε), √(n/(λε))})) runtime, significantly improving over SGD’s O(d/(λε)) when 1/(λε) ≫ n.
• For Lasso with L1 regularization, the runtime is Õ(d(n + min{1/(εγ), √(n/(εγ))})), improving over SGD and FISTA in high-dimensional settings.
• The algorithm achieves nearly linear time complexity Õ(dn) when the condition number is O(n), matching state-of-the-art results.
• For ill-conditioned problems (condition number ≫ n), the runtime reduces to Õ(d√(n/(λγ))), significantly faster than AGD’s Õ(dn√(1/(λγ))).
• High-probability convergence guarantees are established with confidence 1−δ, using Markov’s inequality and iterative refinement over multiple rounds.