[Paper Review] Regularization and feature selection for large dimensional data

Nand Kishore Sharma, Prathamesh Verlekar+2 more|arXiv (Cornell University)|Dec 6, 2017

Gene expression and cancer classification14 references3 citations

TL;DR

This paper evaluates five embedded feature selection methods using L1, L2, or combined regularization in SVMs for high-dimensional data, comparing performance across sparsity, correlation, and execution time. Elastic Net (EN) consistently outperforms others in accuracy and speed, demonstrating robustness across diverse data types and correlation levels.

ABSTRACT

Feature selection has evolved to be an important step in several machine learning paradigms. In domains like bio-informatics and text classification which involve data of high dimensions, feature selection can help in drastically reducing the feature space. In cases where it is difficult or infeasible to obtain sufficient number of training examples, feature selection helps overcome the curse of dimensionality which in turn helps improve performance of the classification algorithm. The focus of our research here are five embedded feature selection methods which use either the ridge regression, or Lasso regression, or a combination of the two in the regularization part of the optimization function. We evaluate five chosen methods on five large dimensional datasets and compare them on the parameters of sparsity and correlation in the datasets and their execution times.

Motivation & Objective

• To evaluate the impact of different regularization techniques—L1, L2, and combined L1/L2—on feature selection performance in high-dimensional data.
• To compare embedded methods (using SVM with regularization) against filter methods (SC and Golub) on real-world datasets with varying sparsity and correlation.
• To assess algorithmic performance in terms of classification accuracy (BSR), sparsity, correlation sensitivity, and execution time.
• To identify which regularization strategy offers the best balance of accuracy, efficiency, and robustness across diverse data characteristics.

Proposed method

• Employs five embedded feature selection methods: L1-SVM, L2-SVM, L21, Local Learning (LL), and Elastic Net (EN), all based on SVM with different regularization norms.
• Uses a unified optimization framework: min_w,b L(w,b) + λR(w,b), where L is the loss function (e.g., hinge loss) and R is the regularizer (L1, L2, or mixed).
• Applies the methods to five large-dimensional datasets: Arabidopsis, Arcene, Dexter, Dorothea (100k features), and Gisette, representing sparse and dense data.
• Measures performance using Balanced Sensitivity Rate (BSR), sparsity of selected features, correlation sensitivity, and execution time.
• Compares results across data characteristics: sparsity (sparse vs. dense), correlation (low vs. high), and dimensionality (up to 100k features).
• Uses SC and Golub as filter method baselines for comparison, evaluating their performance relative to embedded methods.

Experimental results

Research questions

• RQ1 How do L1, L2, and combined L1/L2 regularization perform in terms of classification accuracy (BSR) across datasets with varying correlation levels?
• RQ2 Which embedded method achieves the best trade-off between feature selection sparsity and classification performance on both sparse and dense datasets?
• RQ3 How does execution time vary across the five embedded methods, particularly in ultra-high-dimensional settings (e.g., Dorothea with 100k features)?
• RQ4 To what extent does the performance of L1 regularization degrade in high-correlation datasets, and how do EN and L21 compare?
• RQ5 Does combining L1 and L2 regularization in the Elastic Net framework yield superior performance compared to using it only in the loss or regularization term?

Key findings

• Elastic Net (EN) achieved the highest Balanced Sensitivity Rate (BSR) in three out of five datasets, demonstrating superior overall performance across diverse data types.
• EN significantly outperformed other methods in execution time, completing feature selection in an average of 5.3 seconds on the smallest dataset (Arcene), compared to 45 seconds for the next fastest method.
• On high-correlation datasets like Dorothea, L1-SVM performed worst, while SC (a correlation-based filter) performed best, but EN still ranked second, indicating robustness to correlation.
• For low-correlation datasets (e.g., Dexter, Gisette), Local Learning (LL) and L1-SVM were top performers, with EN ranking second in two of the three, highlighting its versatility.
• On sparse datasets, EN and LL achieved the best BSR, while on dense datasets, LL, L1-SVM, and EN were top performers, showing strong adaptability.
• The L21 method, which applies both norms to both loss and regularization, underperformed compared to EN, which applies both norms only to the regularization term, suggesting this configuration is more effective.