[Paper Review] Multi-Stage Multi-Task Feature Learning

Pinghua Gong, Jieping Ye+1 more|arXiv (Cornell University)|Oct 22, 2012

Sparse and Compressive Sensing Techniques41 references126 citations

TL;DR

This paper proposes a non-convex multi-stage multi-task feature learning (MSMTFL) algorithm using a capped-$\ell_1$ regularizer to improve sparse feature selection over convex alternatives. By iteratively refining feature estimates through stage-wise optimization, MSMTFL achieves a tighter parameter estimation error bound under weaker conditions than prior work, with theoretical guarantees on convergence and reproducibility.

ABSTRACT

Multi-task sparse feature learning aims to improve the generalization performance by exploiting the shared features among tasks. It has been successfully applied to many applications including computer vision and biomedical informatics. Most of the existing multi-task sparse feature learning algorithms are formulated as a convex sparse regularization problem, which is usually suboptimal, due to its looseness for approximating an $\ell_0$-type regularizer. In this paper, we propose a non-convex formulation for multi-task sparse feature learning based on a novel non-convex regularizer. To solve the non-convex optimization problem, we propose a Multi-Stage Multi-Task Feature Learning (MSMTFL) algorithm; we also provide intuitive interpretations, detailed convergence and reproducibility analysis for the proposed algorithm. Moreover, we present a detailed theoretical analysis showing that MSMTFL achieves a better parameter estimation error bound than the convex formulation. Empirical studies on both synthetic and real-world data sets demonstrate the effectiveness of MSMTFL in comparison with the state of the art multi-task sparse feature learning algorithms.

Motivation & Objective

• Address the suboptimal performance of convex multi-task sparse learning formulations that loosely approximate the $\ell_0$-type regularizer.
• Overcome the limitations of existing convex methods, such as restrictive incoherence conditions and loose error bounds.
• Develop a non-convex formulation using a capped-$\ell_1$ regularizer to better approximate the true sparsity-inducing $\ell_0$ norm.
• Design a multi-stage optimization algorithm (MSMTFL) that improves solution quality iteratively while ensuring reproducibility.
• Provide theoretical guarantees on convergence, solution uniqueness, and improved parameter estimation error bounds compared to convex counterparts.

Proposed method

• Propose a non-convex regularizer, the capped-$\ell_1$ norm, as a tighter approximation to the $\ell_0$-type regularizer for multi-task feature learning.
• Design the MSMTFL algorithm as a multi-stage optimization process that alternates between updating feature weights and refining active sets.
• Use a thresholding mechanism at each stage to identify relevant features, based on the magnitude of weight vectors across tasks.
• Apply a sequence of decreasing regularization parameters $\lambda$ across stages to progressively refine feature selection.
• Introduce a convergence analysis showing that the error bound improves with each iteration under the sparse eigenvalue condition.
• Establish solution uniqueness (reproducibility) under a mild condition, enabling reliable theoretical analysis of the algorithm.

Experimental results

Research questions

• RQ1 Can a non-convex formulation based on a capped-$\ell_1$ regularizer outperform convex multi-task sparse learning in terms of parameter estimation error?
• RQ2 Does the multi-stage optimization strategy in MSMTFL lead to progressively better solutions and tighter error bounds than single-step convex methods?
• RQ3 Under what conditions is the solution of the non-convex MSMTFL algorithm reproducible across different initializations?
• RQ4 How does the error bound of MSMTFL compare to that of convex formulations like $\ell_1 + \ell_{1,\infty}$ under weaker assumptions?
• RQ5 Can the proposed algorithm achieve better feature selection performance on both synthetic and real-world datasets compared to state-of-the-art methods?

Key findings

• The MSMTFL algorithm achieves a progressively improving parameter estimation error bound across stages, with the bound at stage $\ell$ being tighter than at stage $\ell-1$.
• Theoretical analysis shows that MSMTFL achieves a better error bound than convex formulations under the sparse eigenvalue condition, which is weaker than the incoherence condition used in prior work.
• The solution of MSMTFL is unique (i.e., reproducible) under a mild condition, resolving a key challenge in non-convex optimization for multi-task learning.
• Empirical results on synthetic and real-world datasets demonstrate that MSMTFL outperforms state-of-the-art multi-task sparse learning algorithms in feature selection accuracy and generalization.
• The algorithm shows robust performance across various data regimes, including high-dimensional settings with sparse true features.
• The multi-stage design enables better convergence behavior and improved identification of shared and task-specific features compared to single-step convex solvers.