[Paper Review] A note on the group lasso and a sparse group lasso

Jerome H. Friedman, Trevor Hastie+1 more|arXiv (Cornell University)|Jan 5, 2010

Statistical Methods and Inference5 references656 citations

TL;DR

This paper introduces the sparse group lasso, a convex optimization method that combines L1 and group lasso penalties to achieve sparsity at both group and individual feature levels. It proposes an efficient coordinate descent algorithm that works for non-orthonormal design matrices, enabling effective variable selection in high-dimensional linear models with grouped predictors.

ABSTRACT

We consider the group lasso penalty for the linear model. We note that the standard algorithm for solving the problem assumes that the model matrices in each group are orthonormal. Here we consider a more general penalty that blends the lasso (L1) with the group lasso ("two-norm"). This penalty yields solutions that are sparse at both the group and individual feature levels. We derive an efficient algorithm for the resulting convex problem based on coordinate descent. This algorithm can also be used to solve the general form of the group lasso, with non-orthonormal model matrices.

Motivation & Objective

• Address the limitation of the standard group lasso, which does not induce sparsity within groups, by proposing a penalty that selects both entire groups and individual predictors.
• Overcome the restrictive assumption in existing group lasso algorithms that predictors within each group must be orthonormal.
• Develop a computationally efficient algorithm for solving the sparse group lasso problem that also generalizes to the standard group lasso with non-orthonormal design matrices.
• Provide a unified computational framework that enables sparse selection at both group and individual levels in high-dimensional linear models.

Proposed method

• Propose the sparse group lasso criterion: minimize the sum of squared residuals plus a combination of group lasso (L2 norm over groups) and lasso (L1 norm over individual coefficients) penalties.
• Use block coordinate descent to optimize the objective, updating one group at a time while holding others fixed.
• For each group, first check if the group should be entirely zero by evaluating a condition based on the norm of the group-wise subgradient; if not, proceed to individual coefficient updates.
• For individual coefficients within a group, apply soft-thresholding via one-dimensional optimization when the gradient exceeds the lasso penalty threshold.
• Derive a condition based on minimizing a quadratic function over the subgradient variables to determine whether the entire group should be set to zero.
• The algorithm is applicable to both the sparse group lasso and the standard group lasso, even when the design matrices in each group are not orthonormal.

Experimental results

Research questions

• RQ1 Can a penalty be constructed that induces sparsity at both the group and individual feature levels in high-dimensional linear models?
• RQ2 How can the group lasso be generalized to handle non-orthonormal predictor matrices in each group without compromising solution accuracy?
• RQ3 What efficient optimization algorithm can be used to solve the resulting convex problem with mixed L1 and group L2 penalties?
• RQ4 Does the proposed method outperform the standard lasso and group lasso in terms of variable selection accuracy when both group-level and individual-level sparsity are desired?
• RQ5 Can the algorithm be adapted to work with arbitrary design matrices, including those with correlated predictors within groups?

Key findings

• The sparse group lasso criterion successfully induces sparsity at both the group and individual feature levels, enabling selection of entire groups and individual predictors within groups.
• The proposed coordinate descent algorithm efficiently solves the sparse group lasso problem and generalizes to the standard group lasso even with non-orthonormal design matrices.
• In a simulation with 100 predictors in 10 groups of 10, the sparse group lasso achieved lower misclassification rates for both groups and individual coefficients compared to lasso and group lasso.
• The method correctly identified the true underlying group structure and individual non-zero coefficients, with the number of misclassified groups and individual features being significantly lower than with lasso or group lasso alone.
• The algorithm maintains computational efficiency and convergence even when predictors within groups are correlated, unlike orthonormalization-based approaches which can distort the solution.
• Theoretical analysis confirms that the algorithm converges to the global minimum due to the convexity of the objective and the use of block coordinate descent with exact line search for individual coefficients.