[Paper Review] Parallel Coordinate Descent for L1-Regularized Loss Minimization

Joseph K. Bradley, Aapo Kyrola+2 more|arXiv (Cornell University)|May 26, 2011

Sparse and Compressive Sensing TechniquesEngineering3 references216 citations

TL;DR

This paper proposes Shotgun, a parallel coordinate descent algorithm for L1-regularized loss minimization that achieves linear speedups by updating multiple coordinates simultaneously. It proves convergence bounds predicting linear scalability up to a problem-dependent limit, validated empirically on Lasso and sparse logistic regression, where Shotgun outperforms state-of-the-art solvers on large-scale datasets.

ABSTRACT

We propose Shotgun, a parallel coordinate descent algorithm for minimizing L1-regularized losses. Though coordinate descent seems inherently sequential, we prove convergence bounds for Shotgun which predict linear speedups, up to a problem-dependent limit. We present a comprehensive empirical study of Shotgun for Lasso and sparse logistic regression. Our theoretical predictions on the potential for parallelism closely match behavior on real data. Shotgun outperforms other published solvers on a range of large problems, proving to be one of the most scalable algorithms for L1.

Motivation & Objective

• To address the growing need for scalable optimization in high-dimensional machine learning problems with L1 regularization.
• To enable efficient parallelization of coordinate descent, an algorithm traditionally considered inherently sequential.
• To develop a theoretically grounded, highly scalable solver for L1-regularized problems such as Lasso and sparse logistic regression.
• To empirically validate theoretical speedup predictions and compare performance against state-of-the-art solvers.

Proposed method

• Proposes Shotgun, a parallel coordinate descent algorithm that performs P simultaneous coordinate updates on features in a multicore setting.
• Uses a transformation to convert the L1-regularized problem into one with non-negative variables and a twice-differentiable regularizer for theoretical analysis.
• Applies a convergence analysis that proves linear speedups up to a problem-dependent maximum number of parallel updates P*.
• Employs atomic updates to a shared residual vector Ax to maintain consistency during parallel updates.
• Derives a theoretical estimate for the optimal number of parallel updates P* based on data properties such as feature norms and sparsity.
• Implements and evaluates Shotgun on real and synthetic datasets using both Lasso and sparse logistic regression.

Experimental results

Research questions

• RQ1 Can coordinate descent, traditionally sequential, be effectively parallelized for L1-regularized problems?
• RQ2 What theoretical convergence guarantees and speedup predictions can be derived for parallel coordinate descent in this setting?
• RQ3 How does the performance of Shotgun compare to state-of-the-art solvers like SGD, SMIDAS, and Shooting on large-scale datasets?
• RQ4 What are the hardware-level bottlenecks limiting runtime speedup despite strong iteration-speedup?
• RQ5 Can a hybrid algorithm combining Shotgun and SGD achieve scalability in both number of samples and features?

Key findings

• Shotgun achieves nearly linear speedup in the number of iterations with respect to the number of parallel updates P, closely matching theoretical predictions.
• On the rcv1 dataset (d > n), Shotgun with P=8 converged over 4 times faster in runtime than SGD, despite SGD's initial advantage on the zeta dataset (n ≫ d).
• Theoretical analysis predicts a maximum number of effective parallel updates P*, which can be computed directly from data and matches empirical behavior.
• Runtime speedup is limited by memory bandwidth and latency, not algorithmic convergence, due to high memory-to-computation ratio and lack of data locality.
• Shotgun outperforms other published solvers on 35 real and synthetic datasets, establishing it as one of the most scalable L1-regularized optimization algorithms.
• The self-speedup in iterations is nearly linear (e.g., ~8× for P=8), but actual runtime speedup is limited to 2×–4× due to memory wall effects.