[Paper Review] Structured sparsity-inducing norms through submodular functions

Francis Bach|arXiv (Cornell University)|Aug 25, 2010

Stochastic Gradient Optimization Techniques48 references59 citations

TL;DR

This paper introduces a general framework for structured sparsity-inducing norms using nondecreasing submodular set-functions, showing that their convex envelope is derived from the Lovász extension. The key contribution is a unified theoretical and algorithmic approach—featuring subgradients, proximal operators, and support recovery conditions—for a broad class of structured norms, including overlapping group norms and new non-factorial priors.

ABSTRACT

Sparse methods for supervised learning aim at finding good linear predictors from as few variables as possible, i.e., with small cardinality of their supports. This combinatorial selection problem is often turned into a convex optimization problem by replacing the cardinality function by its convex envelope (tightest convex lower bound), in this case the L1-norm. In this paper, we investigate more general set-functions than the cardinality, that may incorporate prior knowledge or structural constraints which are common in many applications: namely, we show that for nondecreasing submodular set-functions, the corresponding convex envelope can be obtained from its \lova extension, a common tool in submodular analysis. This defines a family of polyhedral norms, for which we provide generic algorithmic tools (subgradients and proximal operators) and theoretical results (conditions for support recovery or high-dimensional inference). By selecting specific submodular functions, we can give a new interpretation to known norms, such as those based on rank-statistics or grouped norms with potentially overlapping groups; we also define new norms, in particular ones that can be used as non-factorial priors for supervised learning.

Motivation & Objective

• To develop a general framework for structured sparsity-inducing norms that go beyond cardinality-based sparsity.
• To establish a theoretical link between submodular set-functions and convex optimization for sparse learning.
• To provide generic algorithmic tools (subgradients and proximal operators) applicable to a wide class of structured norms.
• To derive theoretical guarantees for support recovery and high-dimensional inference under the proposed framework.
• To unify and reinterpret existing norms (e.g., overlapping group norms, rank-based norms) and introduce new non-factorial priors for supervised learning.

Proposed method

• Leverages the Lovász extension of nondecreasing submodular functions to construct the convex envelope of the set-function penalty $ F({\rm Supp}(w)) $, enabling convex relaxation.
• Derives the proximal operator and subgradient of the resulting norm $ \Omega(w) $, enabling efficient optimization via proximal algorithms.
• Uses the decomposition property $ \Omega(w) = \Omega_J(w_J) + \Omega^J(w_{J^c}) $ to analyze support recovery and high-dimensional consistency.
• Applies restricted eigenvalue and compatibility conditions to derive theoretical bounds on estimation error and support recovery.
• Demonstrates that the dual norm $ \Omega^*(z) $ can be computed via extreme points of the unit ball, enabling efficient computation of subgradients.
• Reinterprets known norms (e.g., grouped norms, rank-statistics) as special cases of submodular penalties, and constructs new norms for non-factorial priors.

Experimental results

Research questions

• RQ1 How can structured sparsity be formalized beyond cardinality using set-functions?
• RQ2 What is the convex envelope of a nondecreasing submodular set-function $ F({\rm Supp}(w)) $, and how can it be computed efficiently?
• RQ3 Can generic algorithmic tools (proximal operators, subgradients) be derived for this class of norms?
• RQ4 Under what conditions does the optimization procedure recover the true support in high-dimensional settings?
• RQ5 How do the proposed norms compare to greedy approaches and existing structured norms in terms of performance and interpretability?

Key findings

• The convex envelope of $ w \mapsto F({\rm Supp}(w)) $ on the $ \ell_\infty $-ball is given by the Lovász extension of $ F $, enabling convex relaxation of structured sparsity.
• The resulting norm $ \Omega(w) $ is a polyhedral norm with computable subgradients and proximal operators, enabling efficient optimization via standard proximal solvers.
• Support recovery is guaranteed when the dual norm of the noise satisfies $ \Omega^*(q) \leq \lambda \rho(J)/2 $, with $ \rho(J) $ controlling the compatibility of the design matrix.
• The estimation error is bounded by $ \Omega_J(\Delta_J) \leq \frac{6c(J)^2\lambda}{\kappa\rho(J)} $, where $ \kappa $ is the restricted eigenvalue and $ c(J) $ the norm compatibility constant.
• The framework recovers and reinterprets known norms such as overlapping group lasso and rank-based penalties as special cases of submodular penalties.
• Empirical results show the proposed method outperforms greedy approaches in simulation studies, particularly in support recovery and estimation accuracy.