[Paper Review] GAP Safe Screening Rules for Sparse-Group-Lasso

Eugène Ndiaye, Olivier Fercoq+2 more|arXiv (Cornell University)|Feb 19, 2016

Statistical Methods and Inference13 citations

TL;DR

This paper introduces the first provably safe screening rules for Sparse-Group Lasso, leveraging geometric properties of the ε-norm to identify and discard inactive features and groups early in coordinate descent optimization. The method significantly reduces computation time by efficiently computing dual gaps without compromising solution accuracy.

ABSTRACT

For statistical learning in high dimension, sparse regularizations have proven useful to boost both computational and statistical efficiency. In some contexts, it is natural to handle more refined structures than pure sparsity, such as for instance group sparsity. Sparse-Group Lasso has recently been introduced in the context of linear regression to enforce sparsity both at the feature and at the group level. We propose the first (provably) safe screening rules for Sparse-Group Lasso, i.e., rules that allow to discard early in the solver features/groups that are inactive at optimal solution. Thanks to efficient dual gap computations relying on the geometric properties of $\epsilon$-norm, safe screening rules for Sparse-Group Lasso lead to significant gains in term of computing time for our coordinate descent implementation.

Motivation & Objective

• To address the computational inefficiency of Sparse-Group Lasso in high-dimensional settings by enabling early elimination of irrelevant features and groups.
• To develop safe screening rules that guarantee no optimal solution components are discarded, ensuring solution correctness.
• To exploit the geometric structure of the ε-norm to compute dual gaps efficiently for screening decisions.
• To integrate screening rules into coordinate descent solvers to accelerate convergence without sacrificing accuracy.

Proposed method

• The method derives safe screening rules based on the dual gap, computed using the geometric properties of the ε-norm in sparse-group regularization.
• It formulates conditions under which features or groups can be safely discarded based on bounds derived from the dual problem.
• The screening rules are integrated into a coordinate descent algorithm, enabling iterative elimination of inactive components.
• Dual gap computations are optimized using the structure of the ε-norm, enabling fast and reliable screening decisions.
• The approach ensures provable safety by relying on duality gap estimates that are tight and computable during optimization.

Experimental results

Research questions

• RQ1 Can safe screening rules be developed for Sparse-Group Lasso that guarantee no optimal solution components are removed?
• RQ2 How can the geometric structure of the ε-norm be leveraged to compute dual gaps efficiently for screening?
• RQ3 What is the impact of early elimination of inactive features and groups on the convergence speed of coordinate descent in Sparse-Group Lasso?
• RQ4 Can the screening rules be integrated into existing solvers without compromising solution quality?

Key findings

• The proposed screening rules are provably safe, ensuring that no features or groups active in the optimal solution are incorrectly discarded.
• The method achieves significant reductions in computing time by eliminating inactive components early in the optimization process.
• Dual gap computations based on ε-norm geometry enable efficient and accurate screening decisions during coordinate descent.
• The integration of screening rules into the coordinate descent framework leads to faster convergence without loss of solution accuracy.