[Paper Review] Taking Advantage of Sparsity in Multi-Task Learning

Karim Lounici, Massimiliano Pontil+2 more|arXiv (Cornell University)|Mar 9, 2009

Sparse and Compressive Sensing Techniques19 references125 citations

TL;DR

This paper proposes a group Lasso-based multi-task learning method that exploits shared sparsity patterns across multiple regression tasks. Under restricted eigenvalue and coherence conditions, it establishes sharp prediction and estimation error bounds that eliminate dependence on the number of predictors M when the number of tasks T grows, enabling consistent variable selection and optimal rates of convergence even in high-dimensional settings with M ≫ n.

ABSTRACT

We study the problem of estimating multiple linear regression equations for the purpose of both prediction and variable selection. Following recent work on multi-task learning Argyriou et al. [2008], we assume that the regression vectors share the same sparsity pattern. This means that the set of relevant predictor variables is the same across the different equations. This assumption leads us to consider the Group Lasso as a candidate estimation method. We show that this estimator enjoys nice sparsity oracle inequalities and variable selection properties. The results hold under a certain restricted eigenvalue condition and a coherence condition on the design matrix, which naturally extend recent work in Bickel et al. [2007], Lounici [2008]. In particular, in the multi-task learning scenario, in which the number of tasks can grow, we are able to remove completely the effect of the number of predictor variables in the bounds. Finally, we show how our results can be extended to more general noise distributions, of which we only require the variance to be finite.

Motivation & Objective

• Address the challenge of high-dimensional regression in multi-task learning where the number of predictors M greatly exceeds the number of observations n.
• Exploit the assumption that all tasks share the same set of relevant predictor variables (structured sparsity) to improve estimation and variable selection.
• Establish theoretical guarantees for prediction accuracy and correct sparsity pattern recovery under minimal assumptions on the design matrix.
• Demonstrate that the number of tasks T can be leveraged to eliminate the dependence of error bounds on M, even when M ≫ n.
• Extend results to general noise distributions with finite variance, not requiring Gaussianity.

Proposed method

• Formulate a multi-task regression model where each task t has its own design matrix $X_t$, response vector $y_t$, and regression coefficient vector $eta^*_t$.
• Introduce a group Lasso estimator that minimizes the average squared residual error across all tasks, penalized by a mixed $(2,1)$-norm on the coefficient vectors to enforce shared sparsity.
• Use a regularization parameter $\lambda = \sigma \sqrt{\frac{\log^{1+\delta} M}{nT}}$ with $\delta > 0$ to control the trade-off between fit and sparsity.
• Apply restricted eigenvalue (RE) and coherence conditions on the design matrices to ensure stable recovery of the true coefficient vectors.
• Leverage concentration inequalities and moment bounds (via Lemma A.2) to derive high-probability bounds on estimation error and sparsity pattern recovery.
• Modify the estimator and thresholding rule to achieve exact support recovery with high probability under stronger conditions on the minimum signal strength of active predictors.

Experimental results

Research questions

• RQ1 Can shared sparsity across multiple regression tasks be exploited to improve prediction accuracy and variable selection in high-dimensional settings?
• RQ2 What are the theoretical error bounds for group Lasso in multi-task learning when $M \gg n$ and $T$ is large?
• RQ3 Does the number of tasks $T$ help eliminate the dependence of error bounds on the number of predictors $M$?
• RQ4 Under what conditions can the group Lasso estimator consistently recover the true sparsity pattern $J(\beta^*)$?
• RQ5 How do the theoretical guarantees extend to non-Gaussian noise with only finite variance?

Key findings

• The prediction error is bounded by $\frac{16}{\kappa^2} \sigma^2 s \frac{\log^{1+\delta} M}{n}$ with high probability, and this bound is independent of $M$ when $T$ grows.
• The estimation error in the $(2,1)$-norm satisfies $\frac{1}{\sqrt{T}} \|\hat{\beta} - \beta^*\|_{2,1} \leq \frac{16}{\kappa^2} \sigma s \sqrt{\frac{\log^{1+\delta} M}{n}}$ with high probability.
• The group Lasso estimator achieves exact support recovery (i.e., $\hat{J} = J(\beta^*)$) with high probability when the minimum signal strength exceeds $2c \sqrt{\frac{\log^{1+\delta} M}{n}}$, where $c = \left(\frac{3}{2} + \frac{1}{7(\alpha-1)}\right)\sigma$.
• Under the restricted eigenvalue condition RE(2s), the prediction error bound becomes $\frac{160}{\kappa^4(2s)} \sigma^2 s \frac{\log^{1+\delta} M}{n}$, showing optimal rate of convergence.
• The results extend to sub-Gaussian noise with finite variance, and the dependence on $M$ cannot be fully removed in this case unless $T$ grows.
• The probability of the event $\mathcal{A}^c$ (failure of concentration) is bounded by $\frac{(2e\log M - e)c'}{\log^{1+\delta} M}$, which decays with $M$ and $T$.