[Paper Review] Faster feature selection with a Dropping Forward-Backward algorithm

Thu L. N. Nguyen|arXiv (Cornell University)|Oct 17, 2019

Sparse and Compressive Sensing Techniques10 references4 citations

TL;DR

This paper proposes a novel Dropping Forward-Backward (DFB) algorithm for faster feature selection in high-dimensional data, which accelerates training by dynamically dropping irrelevant features during forward selection. The method reduces computation time by up to 65.77% compared to stepwise selection while maintaining comparable error rates and feature set quality, particularly effective in $p \gg n$ scenarios.

ABSTRACT

In this era of big data, feature selection techniques, which have long been proven to simplify the model, makes the model more comprehensible, speed up the process of learning, have become more and more important. Among many developed methods, forward and stepwise feature selection regression remained widely used due to their simplicity and efficiency. However, they all involving rescanning all the un-selected features again and again. Moreover, many times, the backward steps in stepwise deem unnecessary, as we will illustrate in our example. These remarks motivate us to introduce a novel algorithm that may boost the speed up to 65.77% compared to the stepwise procedure while maintaining good performance in terms of the number of selected features and error rates. Also, our experiments illustrate that feature selection procedures may be a better choice for high-dimensional problems where the number of features highly exceeds the number of samples.

Motivation & Objective

• To address inefficiencies in traditional forward and stepwise feature selection, which repeatedly rescan unselected features and may include redundant features.
• To develop a faster, more efficient feature selection method that maintains low error rates and minimal feature count.
• To demonstrate that feature selection can outperform feature extraction (e.g., PCA) in $p \gg n$ problems due to better covariance estimation.
• To show that runtime depends not only on data dimensionality but also on model sparsity, influencing algorithm design.
• To provide a practical, configurable algorithm with tunable thresholds ($\alpha$, $\beta$) and support for early stopping or feature ranking.

Proposed method

• The DFB algorithm combines forward selection with dynamic backward elimination: features are added sequentially based on improvement in a criterion, then immediately tested for removal if they no longer contribute significantly.
• After each forward addition, the algorithm checks and removes features that no longer meet the $\beta$-to-remove threshold, preventing accumulation of redundant features.
• The method uses a two-phase loop: forward phase adds features, followed by a 'dropping' phase that removes non-contributing features before the next forward step.
• The algorithm terminates when no further improvement is possible beyond the $\alpha$ threshold, or when a maximum number of features is reached.
• It supports configurable thresholds ($\alpha$ for entry, $\beta$ for removal), and can output feature rankings based on forward-phase inclusion order.
• The approach is evaluated using LDA and other classifiers on both simulated and real-world datasets, including Biodegradation, Ionosphere, Optic, Satellite, and Parkinson.

Experimental results

Research questions

• RQ1 Can a hybrid forward-backward feature selection method significantly reduce training time without degrading model performance?
• RQ2 How does the runtime of feature selection depend on data dimensionality and model sparsity?
• RQ3 In $p \gg n$ problems, is feature selection a better alternative to feature extraction methods like PCA?
• RQ4 Does dynamic removal of redundant features during forward selection improve efficiency compared to standard stepwise or forward-backward methods?
• RQ5 How do the $\alpha$ and $\beta$ thresholds affect the speed and stability of the feature selection process?

Key findings

• On simulated data with $p=70$, the DFB algorithm was 21.89% faster than the forward-backward method and 23.11% faster than stepwise selection, with identical feature counts (4 features).
• On the Optic dataset, the DFB method ran in 24.045 seconds, achieving 34.23% of the time taken by stepwise selection and 66.29% of the forward-backward method.
• For the Parkinson dataset, DFB reduced runtime from 25.486 seconds (stepwise) to 2.691 seconds, a 89.3% speedup, while selecting only 10 features compared to 24 in stepwise.
• In all real-world datasets, DFB achieved error rates equal to or better than stepwise and forward-backward methods, particularly outperforming PCA in LDA classification.
• The algorithm's runtime was not strictly proportional to feature count; for example, Parkinson (753 features) ran faster than Optic (64 features), indicating strong dependence on model sparsity.
• The number of features selected remained stable across methods, with DFB never exceeding the stepwise count by more than twice in 1000 simulations.