[Paper Review] Structured Sparse Regression via Greedy Hard-Thresholding

Prateek Jain, Nikhil Rao+1 more|arXiv (Cornell University)|Feb 19, 2016

Sparse and Compressive Sensing Techniques24 references13 citations

TL;DR

This paper proposes a greedy iterative hard thresholding (IHT) method for structured sparse regression with overlapping groups, using submodular optimization to replace NP-hard projections with efficient greedy approximations. The method achieves strong theoretical guarantees even under high condition numbers (e.g., $κ \geq 100$), enabling fast, accurate recovery of group-sparse signals with orders-of-magnitude speedup over convex and greedy baselines.

ABSTRACT

Several learning applications require solving high-dimensional regression problems where the relevant features belong to a small number of (overlapping) groups. For very large datasets and under standard sparsity constraints, hard thresholding methods have proven to be extremely efficient, but such methods require NP hard projections when dealing with overlapping groups. In this paper, we show that such NP-hard projections can not only be avoided by appealing to submodular optimization, but such methods come with strong theoretical guarantees even in the presence of poorly conditioned data (i.e. say when two features have correlation $\geq 0.99$), which existing analyses cannot handle. These methods exhibit an interesting computation-accuracy trade-off and can be extended to significantly harder problems such as sparse overlapping groups. Experiments on both real and synthetic data validate our claims and demonstrate that the proposed methods are orders of magnitude faster than other greedy and convex relaxation techniques for learning with group-structured sparsity.

Motivation & Objective

• To address the inefficiency and theoretical limitations of existing iterative hard thresholding (IHT) methods for group-sparse regression with overlapping groups.
• To eliminate the need for NP-hard projections in IHT by leveraging submodular optimization for efficient, greedy approximations.
• To provide theoretical convergence guarantees under general restricted strong convexity (RSC)/restricted strong smoothness (RSS) conditions, even when the data covariance has high condition number ($\kappa$).
• To extend the method to more complex sparsity structures, including sparse overlapping groups and hierarchical group structures.
• To empirically validate that approximate projections in IHT yield both speed and accuracy advantages over convex relaxation and other greedy methods.

Proposed method

• Uses a greedy projection step based on submodular function maximization to approximate the NP-hard projection onto the set of $k^*$-group-sparse vectors with overlapping groups.
• Applies iterative hard thresholding with a fully corrective (FC) step to refine the solution at each iteration, improving convergence and accuracy.
• Employs a novel analysis framework that extends sparse regression guarantees from exact projections to approximate greedy projections, ensuring convergence under general RSC/RSS conditions.
• Generalizes the method to the sparse overlapping group (SoG) setting by using a modified projection rule that accounts for both group sparsity and individual feature sparsity.
• Introduces a hierarchical extension for group structures with nested or overlapping hierarchies, maintaining theoretical guarantees and computational efficiency.
• Uses a data-dependent thresholding rule in the hard thresholding step, selecting features based on their contribution to residual reduction.

Experimental results

Research questions

• RQ1 Can greedy projections based on submodular optimization replace NP-hard projections in IHT for overlapping group-sparse regression while preserving theoretical convergence guarantees?
• RQ2 Does the proposed IHT method achieve consistent recovery of the true sparse vector $\bm{w}^*$ even when the data covariance matrix has a high condition number ($\kappa \gg 1$)?
• RQ3 How does the performance of IHT with greedy projections compare to convex relaxation (e.g., group lasso) and other greedy methods (e.g., GOMP, CoGEnT) in terms of speed and accuracy?
• RQ4 Can the IHT framework be extended to more complex sparsity structures such as sparse overlapping groups and hierarchical group structures?
• RQ5 What is the trade-off between computational efficiency and estimation accuracy in IHT when using approximate projections?

Key findings

• The proposed IHT method with greedy projections recovers an $\epsilon$-approximation to $\bm{w}^*$ using $n = \Omega\left((s\log\frac{1}{\epsilon} + \kappa^{2}k^{*}\log M)\log\frac{1}{\epsilon}\right)$ samples, even for arbitrarily large $\kappa$, unlike prior methods that require $\kappa \leq 3$.
• For noisy settings, the method guarantees $\|\hat{\bm{w}} - \bm{w}^*\| \leq 2\epsilon + \lambda \cdot \kappa \sqrt{\frac{s + \kappa^{2}k^{*}\log M}{n}}$, which is a consistent estimator even for large $\kappa$, unlike prior IHT methods that fail to be consistent under the same conditions.
• On synthetic data with $\kappa = 10$, IHT-FC achieved a reconstruction error of 21.65% in 0.1601 seconds, outperforming CoGEnT (23.53% error, 0.1414 sec) and GOMP (25.01% error, 0.2891 sec), while being 10× faster than CoGEnT.
• In poorly conditioned settings ($\kappa = 200$), IHT achieved exact recovery of $\bm{w}^*$ with $n \sim 1500$ measurements, while GOMP failed to recover $\bm{w}^*$ even once in 10 runs.
• On the breast cancer tumor classification dataset, the IHT-FC method achieved the lowest misclassification rate among all compared methods, demonstrating competitive performance despite approximate projections.
• The method achieves orders of magnitude speedup over convex relaxation and other greedy methods, with IHT running in 0.0400 seconds on synthetic data versus 6.4538 seconds for FW.