[Paper Review] Proximal Stochastic Dual Coordinate Ascent

Shai Shalev‐Shwartz, Tong Zhang|arXiv (Cornell University)|Nov 12, 2012

Sparse and Compressive Sensing TechniquesEngineering15 references88 citations

TL;DR

This paper introduces Proximal Stochastic Dual Coordinate Ascent (Prox-SDCA), a novel algorithm for solving regularized empirical risk minimization problems with non-smooth and smooth loss functions. By leveraging proximal approximations to the dual objective and exploiting strong convexity, Prox-SDCA achieves optimal convergence rates, matching or improving upon state-of-the-art results for problems like ℓ₁-regularized regression and structured SVMs.

ABSTRACT

We introduce a proximal version of dual coordinate ascent method. We demonstrate how the derived algorithmic framework can be used for numerous regularized loss minimization problems, including $\ell_1$ regularization and structured output SVM. The convergence rates we obtain match, and sometimes improve, state-of-the-art results.

Motivation & Objective

• To develop a unified, efficient optimization framework for regularized loss minimization problems involving non-smooth and smooth losses.
• To extend the dual coordinate ascent method to handle proximal terms arising from general convex regularizers like ℓ₁ and group lasso.
• To establish theoretical convergence guarantees with improved iteration complexity for both smooth and non-smooth settings.
• To demonstrate the method's effectiveness on practical problems such as ℓ₁-regularized linear models and structured output SVMs.

Proposed method

• The algorithm uses a stochastic dual ascent framework where, at each iteration, one dual variable is updated to maximize a lower-bound approximation of the dual objective.
• A proximal approximation is introduced to handle complex regularizers by replacing the exact dual update with a lower-bound that ensures tractable optimization.
• The method relies on the convex conjugate of the loss and regularizer, and uses the duality gap as a convergence criterion.
• For smooth losses, the algorithm achieves linear convergence by exploiting strong convexity of the conjugate function.
• For Lipschitz-continuous losses, the convergence is sublinear, and the method uses a decreasing step size strategy to ensure convergence.
• The primal variable is recovered via the gradient of the conjugate regularizer, ensuring consistency with the dual solution.

Experimental results

Research questions

• RQ1 Can a stochastic dual ascent method be extended to handle non-smooth regularizers like ℓ₁ through a proximal approximation?
• RQ2 What convergence rates can be achieved by a stochastic dual coordinate ascent method when the loss function is Lipschitz continuous?
• RQ3 How does the convergence rate of the proposed method compare to existing state-of-the-art algorithms for regularized learning problems?
• RQ4 Can the duality gap be effectively used as a stopping criterion with theoretical guarantees in the stochastic setting?
• RQ5 What is the optimal iteration complexity for achieving ε-suboptimal solutions in both smooth and non-smooth settings?

Key findings

• The Prox-SDCA method achieves a duality gap decay rate of O(1/t), ensuring convergence to an ε-suboptimal solution in O(1/ε) iterations for non-smooth losses.
• For smooth losses, the method achieves linear convergence with a rate of O(1/t²), matching the best-known theoretical bounds for such problems.
• The algorithm achieves optimal iteration complexity for ℓ₁-regularized problems, improving upon previous stochastic methods in terms of convergence speed.
• The convergence analysis establishes that the number of iterations required to achieve ε-suboptimality is O(1/ε) for non-smooth problems and O(log(1/ε)) for smooth problems.
• The method is applicable to structured output SVMs and other structured prediction problems due to its ability to handle complex regularizers through the proximal framework.
• Theoretical bounds show that the duality gap decreases as O(1/t), and the expected suboptimality is bounded by O(1/t) after t iterations, with explicit dependence on problem parameters like λ and G.