[Paper Review] GEP-MSCRA for computing the group zero-norm regularized least squares estimator

Shujun Bi, Shaohua Pan|arXiv (Cornell University)|Apr 26, 2018

Sparse and Compressive Sensing Techniques7 references3 citations

TL;DR

This paper proposes GEP-MSCRA, a multi-stage convex relaxation algorithm for computing the group zero-norm regularized least squares estimator via global exact penalty reformulation of an MPEC. It establishes theoretical guarantees under restricted strong convexity and demonstrates superior performance in reducing prediction error and achieving group sparsity compared to SLEP and MALSAR on synthetic and real multi-task learning data.

ABSTRACT

This paper concerns with the group zero-norm regularized least squares estimator which, in terms of the variational characterization of the zero-norm, can be obtained from a mathematical program with equilibrium constraints (MPEC). By developing the global exact penalty for the MPEC, this estimator is shown to arise from an exact penalization problem that not only has a favorable bilinear structure but also implies a recipe to deliver equivalent DC estimators such as the SCAD and MCP estimators. We propose a multi-stage convex relaxation approach (GEP-MSCRA) for computing this estimator, and under a restricted strong convexity assumption on the design matrix, establish its theoretical guarantees which include the decreasing of the error bounds for the iterates to the true coefficient vector and the coincidence of the iterates after finite steps with the oracle estimator. Finally, we implement the GEP-MSCRA with the subproblems solved by a semismooth Newton augmented Lagrangian method (ALM) and compare its performance with that of SLEP and MALSAR, the solvers for the weighted $\ell_{2,1}$-norm regularized estimator, on synthetic group sparse regression problems and real multi-task learning problems. Numerical comparison indicates that the GEP-MSCRA has significant advantage in reducing error and achieving better sparsity than the SLEP and the MALSAR do.

Motivation & Objective

• To develop a computationally tractable method for the group zero-norm regularized least squares estimator, which is combinatorially challenging due to the nonconvex zero-norm.
• To establish a global exact penalty reformulation of the MPEC arising from the variational characterization of the zero-norm, enabling exact penalization with favorable bilinear structure.
• To propose a multi-stage convex relaxation approach (GEP-MSCRA) that delivers equivalent DC estimators such as SCAD and MCP, with theoretical convergence guarantees.
• To empirically validate GEP-MSCRA’s superiority in prediction accuracy and group sparsity over SLEP and MALSAR on synthetic and real-world multi-task learning problems.

Proposed method

• Reformulate the group zero-norm regularized least squares estimator as a mathematical program with equilibrium constraints (MPEC) using the variational characterization of the zero-norm.
• Develop a global exact penalty for the MPEC, transforming the nonconvex problem into a single nonconvex program with a favorable bilinear structure.
• Propose GEP-MSCRA, a multi-stage convex relaxation algorithm that iteratively solves convex subproblems derived from the exact penalized formulation.
• Solve each subproblem using a semismooth Newton augmented Lagrangian method (ALM) for high-accuracy and fast convergence.
• Leverage the connection between the exact penalized problem and DC estimators like SCAD and MCP, enabling equivalent solutions through relaxation.
• Use warm-starting strategies in numerical experiments, initializing each $λ$-problem with the solution from the previous $λ$-value to enhance efficiency.

Experimental results

Research questions

• RQ1 Can the group zero-norm regularized least squares estimator be reformulated as a global exact penalty problem to enable efficient computation?
• RQ2 Does the proposed GEP-MSCRA algorithm achieve theoretical convergence to the oracle estimator under restricted strong convexity?
• RQ3 How does GEP-MSCRA compare to SLEP and MALSAR in terms of prediction error and group sparsity across synthetic and real multi-task learning problems?
• RQ4 To what extent does the performance of GEP-MSCRA depend on initialization, and how does it compare to MALSAR’s warm-start dependency?
• RQ5 Can the exact penalized formulation unify the computation of SCAD and MCP estimators within a single framework?

Key findings

• GEP-MSCRA reduces prediction error by at least 20% compared to MALSAR when MALSAR does not use warm-starting, and matches or exceeds MALSAR’s performance even with warm-starting.
• The prediction error of GEP-MSCRA decreases with increasing training sample size, while MALSAR’s error increases or fails to improve, especially when only 35% of data is used for training.
• GEP-MSCRA achieves group sparsity with fewer than 5 active groups across all tested training sample sizes, whereas MALSAR fails to produce group-sparse solutions under the same conditions.
• The GEP-MSCRA solution is robust to initialization, as its performance does not degrade when starting from $x^0 = 0$, unlike MALSAR which heavily relies on warm-starting.
• Under restricted strong convexity, the iterates of GEP-MSCRA exhibit decreasing error bounds toward the true coefficient vector and eventually coincide with the oracle estimator after finite steps.
• The exact penalized formulation reveals that SCAD and MCP estimators also arise from the same global exact penalty framework, establishing a unifying computational and theoretical foundation.