[Paper Review] Hybrid safe-strong rules for efficient optimization in lasso-type problems

Yaohui Zeng, Tianbao Yang+1 more|arXiv (Cornell University)|Apr 27, 2017

Machine Learning and Algorithms27 references4 citations

TL;DR

This paper proposes hybrid safe-strong rules (HSSR) that combine the reliability of safe screening rules with the efficiency of sequential strong rules (SSR) to accelerate lasso-type optimization. By integrating safe rules into SSR, the method drastically reduces post-convergence KKT checks, achieving up to 70% faster computation than state-of-the-art methods on correlated and high-dimensional data.

ABSTRACT

The lasso model has been widely used for model selection in data mining, machine learning, and high-dimensional statistical analysis. However, with the ultrahigh-dimensional, large-scale data sets now collected in many real-world applications, it is important to develop algorithms to solve the lasso that efficiently scale up to problems of this size. Discarding features from certain steps of the algorithm is a powerful technique for increasing efficiency and addressing the Big Data challenge. In this paper, we propose a family of hybrid safe-strong rules (HSSR) which incorporate safe screening rules into the sequential strong rule (SSR) to remove unnecessary computational burden. In particular, we present two instances of HSSR, namely SSR-Dome and SSR-BEDPP, for the standard lasso problem. We further extend SSR-BEDPP to the elastic net and group lasso problems to demonstrate the generalizability of the hybrid screening idea. Extensive numerical experiments with synthetic and real data sets are conducted for both the standard lasso and the group lasso problems. Results show that our proposed hybrid rules can substantially outperform existing state-of-the-art rules.

Motivation & Objective

• Address the computational bottleneck in solving ultrahigh-dimensional lasso problems common in Big Data applications.
• Overcome limitations of existing screening rules—safe rules are conservative, while strong rules risk discarding active features.
• Develop a hybrid framework that leverages the safety of geometric bounds with the aggressiveness of strong rules to maximize efficiency without sacrificing correctness.
• Demonstrate generalizability of the hybrid approach to elastic net and group lasso problems.
• Minimize unnecessary KKT checks post-optimization by pre-eliminating features via safe rules within the SSR framework.

Proposed method

• Propose a hybrid screening framework, HSSR, that applies safe rules to filter features before applying sequential strong rules (SSR), reducing the number of features requiring KKT validation.
• Introduce two specific HSSR instances: SSR-Dome and SSR-BEDPP, which integrate the Dome test and EDPP safe rules into the SSR framework.
• Leverage the dual formulation of the lasso to define compact regions for the dual solution, enabling safe feature elimination via the KKT condition: $|\mathbf{x}_j^T\boldsymbol{\theta}(\lambda)| < \lambda \Rightarrow \widehat{\beta}_j = 0$.
• Use the unit-slope assumption in SSR to derive strong rules that skip updates for features unlikely to be active, with a post-check for correctness.
• Extend SSR-BEDPP to elastic net and group lasso by adapting the safe region and dual bounds to handle grouped and ridge-regularized structures.
• Formulate the hybrid rule as a two-stage process: (1) safe screening removes features guaranteed to be zero, (2) SSR further prunes features with high probability of being zero, minimizing KKT checks.

Experimental results

Research questions

• RQ1 Can combining safe and strong rules significantly reduce computational time in solving lasso problems without compromising solution correctness?
• RQ2 How does feature correlation affect the performance of hybrid screening rules compared to standalone safe or strong rules?
• RQ3 To what extent can the hybrid framework be generalized to other lasso-type problems such as elastic net and group lasso?
• RQ4 What is the trade-off between screening aggressiveness and reliability in hybrid rules, especially under high correlation?
• RQ5 Can the hybrid approach substantially reduce the number of KKT checks required after optimization, thereby accelerating convergence?

Key findings

• The proposed SSR-BEDPP rule reduced average computing time by 53% compared to standard SSR and 44% compared to SEDPP on synthetic data with low to moderate correlation.
• On data with pairwise correlation up to 0.5, hybrid rules (SSR-Dome and SSR-BEDPP) were 30–70% faster than either safe or strong rules alone.
• In high-correlation settings (ρ = 0.8), strong rules became ineffective, and so did the hybrid rules, indicating a performance ceiling under extreme dependence.
• For real-world data, the hybrid rules demonstrated strong practical performance, confirming effectiveness in realistic correlation structures.
• The hybrid approach reduced the number of KKT checks by over 90% in low-correlation regimes, significantly accelerating the final convergence step.
• SSR-BEDPP achieved the fastest runtime across all methods, with a mean time of 3.22 seconds (SE 0.05) on 100 λ values under uncorrelated design, outperforming basic PCD (10.92s) and AC (6.15s).