[Paper Review] Structured Variable Selection with Sparsity-Inducing Norms

Rodolphe Jenatton, Jean-Yves Audibert+1 more|arXiv (Cornell University)|Apr 22, 2009

Sparse and Compressive Sensing Techniques71 references458 citations

TL;DR

This paper introduces structured sparsity-inducing norms that extend the $ε_1$-norm and group $ε_1$-norm by allowing overlapping groups, enabling the modeling of complex, prior-specific nonzero coefficient patterns in linear models. The key contribution is a framework linking group structures to desired sparsity patterns, along with an active set algorithm and theoretical consistency results for variable selection in high and low-dimensional settings.

ABSTRACT

We consider the empirical risk minimization problem for linear supervised learning, with regularization by structured sparsity-inducing norms. These are defined as sums of Euclidean norms on certain subsets of variables, extending the usual $\ell_1$-norm and the group $\ell_1$-norm by allowing the subsets to overlap. This leads to a specific set of allowed nonzero patterns for the solutions of such problems. We first explore the relationship between the groups defining the norm and the resulting nonzero patterns, providing both forward and backward algorithms to go back and forth from groups to patterns. This allows the design of norms adapted to specific prior knowledge expressed in terms of nonzero patterns. We also present an efficient active set algorithm, and analyze the consistency of variable selection for least-squares linear regression in low and high-dimensional settings.

Motivation & Objective

• To address the limitation of standard $ε_1$-norm regularization in ignoring structural relationships among variables.
• To develop sparsity-inducing norms that encode complex, prior-specific nonzero patterns (e.g., spatial, hierarchical, or contiguous structures) via overlapping group structures.
• To establish a formal link between the group structure defining the norm and the resulting set of allowed nonzero patterns in solutions.
• To design an efficient active set algorithm for solving the resulting optimization problems.
• To analyze the consistency of variable selection in both low- and high-dimensional settings for least-squares regression.

Proposed method

• Proposes structured norms as sums of Euclidean norms over overlapping subsets (groups) of variables, generalizing the $ε_1$-norm and group $ε_1$-norm.
• Introduces forward and backward algorithms to map from group structures to allowed nonzero patterns and vice versa, enabling norm design based on prior knowledge.
• Develops an active set algorithm that efficiently solves the optimization problem by iteratively updating the set of active variables.
• Derives optimality conditions using subdifferential calculus, showing that a solution satisfies a dual condition involving the subgradient of the norm.
• Uses the dual norm structure to characterize the optimality condition, leveraging the fact that the dual of a sum of disjoint norms is the maximum of the individual dual norms.
• Applies the framework to real-world problems such as neuroimaging, face recognition, and genomics, where structural priors (spatial, temporal, hierarchical) are critical for performance and interpretability.

Experimental results

Research questions

• RQ1 How can sparsity-inducing norms be designed to encode complex, prior-specific nonzero patterns in linear models, such as spatial locality or hierarchical relationships?
• RQ2 What is the precise relationship between the group structure defining the norm and the set of allowed nonzero coefficient patterns in the solution?
• RQ3 Can an efficient active set algorithm be developed to solve the resulting structured sparsity optimization problem?
• RQ4 Under what conditions is variable selection consistent in both low- and high-dimensional settings when using structured norms?
• RQ5 How do overlapping groups in the norm affect the sparsity pattern and the theoretical properties of the estimator?

Key findings

• The proposed structured norms, defined as sums of Euclidean norms over overlapping groups, allow for the explicit encoding of complex prior knowledge about nonzero patterns in regression solutions.
• Forward and backward algorithms are established to map between group structures and allowed nonzero patterns, enabling the design of norms tailored to specific structural priors.
• An active set algorithm is proposed that efficiently solves the optimization problem by iteratively updating the active set of variables, improving computational performance.
• Theoretical consistency of variable selection is established for least-squares regression in both low- and high-dimensional regimes, under appropriate conditions on the design matrix and sparsity.
• The dual norm characterization shows that the optimality condition involves the maximum of dual norms over disjoint supports, enabling efficient subgradient computation.
• Empirical validation across neuroimaging, face recognition, and genomics demonstrates improved performance and interpretability when using structured norms over unstructured $ε_1$-norms.