[Paper Review] A Constructive Approach to High-dimensional Regression

Jian Huang, Yuling Jiao+2 more|arXiv (Cornell University)|Jan 18, 2017

Stochastic Gradient Optimization Techniques3 citations

TL;DR

This paper proposes SDAR, a constructive algorithm for high-dimensional sparse linear regression that iteratively detects supports and solves root-finding problems based on KKT conditions of ℓ₀-penalized least squares. It achieves exponential decay in ℓ₂ and ℓ∞ estimation errors under sparsity and coherence conditions, with convergence in O(√J log R) and O(log R) steps respectively, and recovers the oracle estimator exactly under known sparsity levels at O(np) cost per iteration.

ABSTRACT

We develop a constructive approach to estimating sparse, high-dimensional linear regression models. The approach is a computational algorithm motivated from the KKT conditions for the $\ell_0$-penalized least squares solutions. It generates a sequence of solutions iteratively, based on support detection using primal and dual information and root finding. We refer to the algorithm as SDAR for brevity. Under a sparse Rieze condition on the design matrix and certain other conditions, we show that with high probability, the $\ell_2$ estimation error of the solution sequence decays exponentially to the minimax error bound in $O(\sqrt{J}\log(R))$ steps; and under a mutual coherence condition and certain other conditions, the $\ell_{\infty}$ estimation error decays to the optimal error bound in $O(\log(R))$ steps, where $J$ is the number of important predictors, $R$ is the relative magnitude of the nonzero target coefficients. Computational complexity analysis shows that the cost of SDAR is $O(np)$ per iteration. Moreover the oracle least squares estimator can be exactly recovered with high probability at the same cost if we know the sparsity level. We also consider an adaptive version of SDAR to make it more practical in applications. Numerical comparisons with Lasso, MCP and greedy methods demonstrate that SDAR is competitive with or outperforms them in accuracy and efficiency.

Motivation & Objective

• Address the computational challenge of solving ℓ₀-penalized least squares in high-dimensional regression, which is NP-hard.
• Develop a stable and efficient algorithm that approximates the sparse solution without relying on convex relaxations like Lasso.
• Achieve fast convergence with theoretical guarantees on estimation error decay under sparsity and coherence conditions.
• Enable exact recovery of the oracle least squares estimator when the sparsity level is known.
• Improve upon existing methods by combining support detection with primal-dual root finding for better accuracy and efficiency.

Proposed method

• Propose SDAR (Support Detection and Root finding), a computational algorithm motivated by the KKT conditions of ℓ₀-penalized least squares.
• Iteratively detect the support of the true coefficient vector using primal and dual information from the current iterate.
• Solve a root-finding problem at each step to refine the solution, ensuring convergence to the sparse solution.
• Use a sparse Rieze condition and mutual coherence condition as theoretical assumptions to guarantee convergence rates.
• Introduce an adaptive version of SDAR to enhance practical performance without prior knowledge of sparsity.
• Analyze computational complexity, showing O(np) cost per iteration, making it scalable for high-dimensional settings.

Experimental results

Research questions

• RQ1 Can a nonconvex, ℓ₀-based approach for high-dimensional regression achieve fast and stable convergence with theoretical error bounds?
• RQ2 Does the proposed algorithm SDAR converge exponentially in ℓ₂ and ℓ∞ estimation error under sparsity and coherence assumptions?
• RQ3 Can SDAR exactly recover the oracle least squares estimator when the true sparsity level is known?
• RQ4 How does SDAR compare in accuracy and efficiency to Lasso, MCP, and greedy methods in numerical experiments?
• RQ5 What is the computational cost of SDAR, and can it scale efficiently in high-dimensional settings?

Key findings

• Under a sparse Rieze condition, the ℓ₂ estimation error of SDAR decays exponentially to the minimax error bound in O(√J log R) steps, where J is the number of important predictors.
• Under a mutual coherence condition, the ℓ∞ estimation error decays to the optimal error bound in O(log R) steps, with R being the relative magnitude of nonzero coefficients.
• SDAR recovers the oracle least squares estimator exactly with high probability when the sparsity level s is known, at a cost of O(np) per iteration.
• The computational complexity of SDAR is O(np) per iteration, making it scalable for large p and n.
• Numerical comparisons show SDAR is competitive or superior to Lasso, MCP, and greedy methods in both accuracy and efficiency.
• The adaptive version of SDAR improves practical performance by dynamically adjusting thresholds without requiring prior knowledge of sparsity.