[Paper Review] $l_{2,p}$ Matrix Norm and Its Application in Feature Selection

Liping Wang, Songcan Chen|arXiv (Cornell University)|Mar 16, 2013

Sparse and Compressive Sensing TechniquesEngineering18 references22 citations

TL;DR

This paper introduces a generalized non-convex $l_{2,p}$ matrix pseudo-norm for feature selection, extending $l_{2,1}$-norm regularization to $p \in (0,1)$, and proposes a unified, convergent algorithm that efficiently solves both convex ($p=1$) and non-convex ($0<p<1$) optimization problems. Experiments on gene expression data show that $p=0.5$ yields superior sparsity and classification accuracy compared to $p=1$, validating the method’s effectiveness in enhancing feature selection performance.

ABSTRACT

Recently, $l_{2,1}$ matrix norm has been widely applied to many areas such as computer vision, pattern recognition, biological study and etc. As an extension of $l_1$ vector norm, the mixed $l_{2,1}$ matrix norm is often used to find jointly sparse solutions. Moreover, an efficient iterative algorithm has been designed to solve $l_{2,1}$-norm involved minimizations. Actually, computational studies have showed that $l_p$-regularization ($0

Motivation & Objective

• To generalize $l_{2,1}$-norm regularization to non-convex $l_{2,p}$-norms for improved sparsity in feature selection.
• To develop a unified algorithm that efficiently solves $l_{2,p}$-norm optimization problems across all $p \in (0,1]$, including both convex and non-convex cases.
• To prove uniform convergence of the proposed algorithm for all $p \in (0,1]$, overcoming limitations of prior methods requiring separate solvers for convex and non-convex regimes.
• To empirically validate that $p \in (0,1)$, particularly $p=0.5$, enhances sparsity and classification accuracy over $p=1$ in real-world biological data.

Proposed method

• Proposes a mixed $l_{2,p}$ matrix pseudo-norm as a generalization of $l_{2,1}$-norm to non-convex $p \in (0,1)$, defined as $\|Y\|_{2,p}^p = \sum_{i=1}^m \|y_i\|_2^p$ for matrix $Y$ with rows $y_i$.
• Designs a unified iterative algorithm that handles both convex ($p=1$) and non-convex ($0<p<1$) cases without requiring separate optimization strategies.
• Employs a proximal-like update scheme with closed-form solutions at each step, ensuring computational efficiency and convergence.
• Proves uniform convergence of the algorithm for all $p \in (0,1]$, including the non-Lipschitz and non-convex case when $0<p<1$, using theoretical analysis.
• Applies the algorithm to $l_{2,p}$-regularized least squares problems in feature selection, with objective function $\|X - YB\|_F^2 + \alpha \|Y\|_{2,p}^p$.
• Uses $p=0.25, 0.5, 0.75, 1$ in experiments and evaluates performance via 5-fold cross-validation with SVM classifiers on four gene expression datasets.

Experimental results

Research questions

• RQ1 Can the $l_{2,p}$-norm with $p \in (0,1)$ provide better sparsity than the standard $l_{2,1}$-norm in feature selection tasks?
• RQ2 Is it possible to design a single, unified algorithm that efficiently solves $l_{2,p}$-norm optimization problems across both convex ($p=1$) and non-convex ($0<p<1$) regimes?
• RQ3 Does the proposed algorithm maintain convergence guarantees for all $p \in (0,1]$, including the non-convex and non-Lipschitz case when $0<p<1$?
• RQ4 How does the choice of $p$ affect the classification accuracy and sparsity of selected features in real biological datasets?

Key findings

• The $l_{2,p}$-norm with $p=0.5$ achieves the lowest classification error on average across all datasets, outperforming $p=1$ in feature selection performance.
• For the ALLAML dataset, $p=0.5$ achieves a classification error of 4.0% with 20 features, compared to 5.43% for $p=1$, indicating improved accuracy with sparser features.
• On the LUNG dataset, $p=0.5$ reduces error from 2.95% ($p=1$) to 1.98% with 20 features, demonstrating consistent improvement.
• The unified algorithm converges within approximately 20 iterations across all $p$ values, with consistent convergence speed and objective reduction across datasets.
• The average classification error for $p=0.5$ is 3.0675% with 40 features and 3.1625% with 80 features, outperforming $p=1$ in all cases.
• Empirical results show that $p=0.5$ yields a better sparse pattern than $p=1$, confirming that non-convex $l_{2,p}$-norms can enhance feature selection quality.