[Paper Review] L0Learn: A Scalable Package for Sparse Learning using L0 Regularization

Hussein Hazimeh, Rahul Mazumder+1 more|arXiv (Cornell University)|Feb 10, 2022

Machine Learning and Data Classification4 citations

TL;DR

L0Learn is a scalable C++-based package with R and Python interfaces that implements fast, approximate algorithms for sparse linear regression and classification using ℓ₀ regularization. It combines coordinate descent with local combinatorial optimization to achieve state-of-the-art runtimes and statistical performance—matching or outperforming glmnet, picasso, abess, and ncvreg on synthetic high-dimensional datasets with up to 10⁵ features.

ABSTRACT

We present L0Learn: an open-source package for sparse linear regression and classification using $\ell_0$ regularization. L0Learn implements scalable, approximate algorithms, based on coordinate descent and local combinatorial optimization. The package is built using C++ and has user-friendly R and Python interfaces. L0Learn can address problems with millions of features, achieving competitive run times and statistical performance with state-of-the-art sparse learning packages. L0Learn is available on both CRAN and GitHub (https://cran.r-project.org/package=L0Learn and https://github.com/hazimehh/L0Learn).

Motivation & Objective

• Address the computational challenge of ℓ₀-regularized learning, which is NP-hard but offers superior statistical properties for sparse model selection.
• Develop a practical, scalable solution for high-dimensional problems with millions of features where global optimality is computationally prohibitive.
• Improve upon existing approximate methods like ℓ₁-regularization (e.g., glmnet) and nonconvex penalties (e.g., MCP, SCAD) by combining coordinate descent with local combinatorial optimization for better solution quality.
• Enable efficient, user-friendly sparse learning in real-world applications such as genomics and healthcare through production-ready R and Python APIs.
• Support both ℓ₀ℓ₁ and ℓ₀ℓ₂ regularization to balance sparsity and shrinkage, enhancing model robustness in low-signal regimes.

Proposed method

• Leverages cyclic coordinate descent (CD) as the core optimization engine, adapted for discontinuous ℓ₀ objectives with convergence guarantees to a local minimizer.
• Introduces a local combinatorial optimization phase that explores neighborhood solutions around a CD solution to improve objective value and reduce false positives.
• Supports multiple loss functions: squared error (regression), logistic loss (classification), and squared-hinge loss (classification), enabling broad applicability.
• Implements a two-parameter regularization path: ℓ₀ penalty for sparsity and ℓ₁ or ℓ₂ penalty for shrinkage, with automatic grid tuning over λ and γ.
• Uses a C++ backend for high-performance computation and exposes the functionality via efficient, user-friendly R and Python interfaces on CRAN and GitHub.
• Employs warm-starting and efficient updates in CD to accelerate convergence, especially in high-dimensional settings with sparse solutions.

Experimental results

Research questions

• RQ1 Can a fast, approximate ℓ₀-regularized solver achieve competitive statistical performance and runtime efficiency compared to state-of-the-art ℓ₁ and nonconvex penalized methods?
• RQ2 How does the integration of local combinatorial optimization with coordinate descent improve solution quality in ℓ₀-regularized problems compared to CD alone?
• RQ3 To what extent can ℓ₀Learn scale to problems with up to 10⁵ features while maintaining low runtime and high sparsity accuracy?
• RQ4 Does the ℓ₀ℓ₂ regularization formulation outperform ℓ₀ℓ₁ or standard ℓ₁ in terms of prediction error, false positive control, and support recovery across diverse signal-to-noise ratios?
• RQ5 Can L0Learn serve as an effective warm-starting heuristic for exact ℓ₀ solvers, improving their convergence in mixed-integer programming frameworks?

Key findings

• L0Learn achieved the fastest runtime across all tested feature dimensions (p = 10³, 10⁴, 10⁵), with median times of 0.09s, 0.49s, and 4.4s respectively, outperforming glmnet (0.55s, 0.94s, 8.0s) and picasso (1.46s, 2.92s, 15.5s).
• L0Learn achieved the lowest prediction error (PE × 10² = 9.4) across all p, significantly outperforming glmnet (19.8), picasso (19.8), abess (11.0), and ncvreg (9.4–9.6), with minimal variance.
• L0Learn reported zero false positives (FP = 0) in all settings, while glmnet (154–485), picasso (157–483), and abess (2–3) had substantial false detection rates.
• Support size (SS) was perfectly recovered at 50 in all cases, matching the true sparsity level, while abess and ncvreg showed overfitting with SS = 52–54.
• The package successfully handled p = 10⁵ features without memory failure, while abess failed to converge on the largest problem due to memory constraints.
• L0Learn’s ℓ₀ℓ₂ formulation with default parameters matched or exceeded the performance of tuned ℓ₁ (glmnet) and MCP (ncvreg) methods, demonstrating the value of ℓ₀ regularization with shrinkage.