[Paper Review] Random extrapolation for primal-dual coordinate descent

Ahmet Alacaoglu, Olivier Fercoq+1 more|arXiv (Cornell University)|Jul 13, 2020

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper proposes PURE-CD, a primal-dual coordinate descent method that uses random extrapolation to adaptively handle both sparse and dense data matrices. It achieves linear convergence under metric subregularity and optimal O(1/k) sublinear convergence in the general convex-concave case, outperforming existing methods in both settings without parameter tuning.

ABSTRACT

We introduce a randomly extrapolated primal-dual coordinate descent method that adapts to sparsity of the data matrix and the favorable structures of the objective function. Our method updates only a subset of primal and dual variables with sparse data, and it uses large step sizes with dense data, retaining the benefits of the specific methods designed for each case. In addition to adapting to sparsity, our method attains fast convergence guarantees in favorable cases extit{without any modifications}. In particular, we prove linear convergence under metric subregularity, which applies to strongly convex-strongly concave problems and piecewise linear quadratic functions. We show almost sure convergence of the sequence and optimal sublinear convergence rates for the primal-dual gap and objective values, in the general convex-concave case. Numerical evidence demonstrates the state-of-the-art empirical performance of our method in sparse and dense settings, matching and improving the existing methods.

Motivation & Objective

• To address the limitation of existing primal-dual coordinate descent methods that either suffer from high per-iteration cost on sparse data or restrict step sizes on dense data.
• To unify the advantages of sparse-efficient and dense-efficient methods into a single framework that adapts to data sparsity without manual tuning.
• To prove linear convergence under metric subregularity, which covers strongly convex-strongly concave problems and piecewise linear-quadratic objectives.
• To establish almost sure convergence and optimal O(1/k) sublinear convergence rates for the primal-dual gap and objective values in the general convex-concave case.

Proposed method

• Introduces random extrapolation as a key mechanism to dynamically adjust step sizes based on data sparsity and problem structure.
• Uses coordinate descent with selective updates on a random subset of primal and dual variables, guided by the sparsity pattern of the data matrix A.
• Employs proximal operators for g and h*, and gradient updates for f, with step sizes adapted via the extrapolation mechanism.
• Leverages a randomized selection strategy with probability laws dependent on block sparsity, ensuring efficiency in both sparse and dense regimes.
• Derives convergence guarantees using a Lyapunov function and probabilistic analysis, including bounds on expected iterates and duality gap.
• Incorporates a dual variable update with extrapolation that improves convergence speed while maintaining stability.

Experimental results

Research questions

• RQ1 Can a primal-dual coordinate descent method simultaneously achieve low per-iteration cost on sparse data and large step sizes on dense data?
• RQ2 Does random extrapolation enable adaptation to unknown problem structures without modifying step sizes?
• RQ3 What convergence rates can be guaranteed under metric subregularity, and how do they compare to existing methods?
• RQ4 Can almost sure convergence and optimal O(1/k) sublinear rates be established in the general convex-concave case?

Key findings

• PURE-CD achieves linear convergence under metric subregularity, which includes strongly convex-strongly concave problems and piecewise linear-quadratic objectives such as Lasso and SVMs.
• The method attains optimal O(1/k) sublinear convergence rate for the primal-dual gap and objective values in the general convex-concave case.
• Almost sure convergence of the iterates to a solution is proven, even without strong convexity.
• Numerical experiments show state-of-the-art empirical performance across sparse, dense, and moderately sparse data regimes.
• The method maintains low per-iteration cost on sparse data and large step sizes on dense data, outperforming prior methods that sacrifice one for the other.