[Paper Review] Multi-Task Feature Learning Via Efficient l2,1-Norm Minimization

Jun Liu, Shuiwang Ji+1 more|arXiv (Cornell University)|May 9, 2012

Gaussian Processes and Bayesian Inference27 references553 citations

TL;DR

This paper proposes an efficient optimization method for multi-task feature learning using l2,1-norm regularization, which promotes shared sparsity across tasks. By reformulating the non-smooth problem into two smooth convex forms and solving them via Nesterov's optimal first-order method, the approach achieves faster convergence with analytically computable or linear-time Euclidean projections, significantly improving computational efficiency on benchmark datasets.

ABSTRACT

The problem of joint feature selection across a group of related tasks has applications in many areas including biomedical informatics and computer vision. We consider the l2,1-norm regularized regression model for joint feature selection from multiple tasks, which can be derived in the probabilistic framework by assuming a suitable prior from the exponential family. One appealing feature of the l2,1-norm regularization is that it encourages multiple predictors to share similar sparsity patterns. However, the resulting optimization problem is challenging to solve due to the non-smoothness of the l2,1-norm regularization. In this paper, we propose to accelerate the computation by reformulating it as two equivalent smooth convex optimization problems which are then solved via the Nesterov's method-an optimal first-order black-box method for smooth convex optimization. A key building block in solving the reformulations is the Euclidean projection. We show that the Euclidean projection for the first reformulation can be analytically computed, while the Euclidean projection for the second one can be computed in linear time. Empirical evaluations on several data sets verify the efficiency of the proposed algorithms.

Motivation & Objective

• To address the challenge of joint feature selection across multiple related tasks in machine learning.
• To overcome the computational difficulty posed by the non-smooth l2,1-norm regularization in multi-task regression.
• To develop a scalable and efficient optimization framework that accelerates convergence for large-scale multi-task learning problems.
• To enable practical deployment of l2,1-norm regularized models by reducing computational complexity through smooth reformulations.

Proposed method

• Reformulate the non-smooth l2,1-norm regularized multi-task learning problem into two equivalent smooth convex optimization problems.
• Apply Nesterov's optimal first-order method to solve the smooth reformulations, ensuring fast convergence rates.
• Compute the Euclidean projection for the first reformulation analytically, enabling closed-form updates.
• Implement a linear-time algorithm for computing the Euclidean projection in the second reformulation, ensuring scalability.
• Leverage the structure of the l2,1-norm to encourage shared sparsity patterns across multiple predictors.
• Use the probabilistic interpretation of the l2,1-norm as a prior from the exponential family to justify the model's statistical foundation.

Experimental results

Research questions

• RQ1 Can the non-smooth l2,1-norm regularization problem in multi-task learning be efficiently solved using smooth optimization techniques?
• RQ2 What is the computational complexity of the Euclidean projection in the reformulated optimization problems, and can it be computed efficiently?
• RQ3 How does the proposed method compare in convergence speed and scalability to existing approaches for multi-task feature learning?
• RQ4 Can the smooth reformulation preserve the shared sparsity property of the original l2,1-norm model?

Key findings

• The proposed method achieves faster convergence than traditional first-order methods due to the use of Nesterov's optimal method on smooth reformulations.
• The Euclidean projection in the first reformulation can be computed in closed form, enabling efficient updates.
• The Euclidean projection in the second reformulation is computable in linear time, ensuring scalability to large-scale problems.
• Empirical evaluations on multiple datasets demonstrate significant speedups in computation time while maintaining or improving feature selection performance.