[Paper Review] Efficient Wrapper Feature Selection using Autoencoder and Model Based Elimination

Sharan Ramjee, Aly El Gamal|arXiv (Cornell University)|May 28, 2019

Machine Learning in Bioinformatics26 references4 citations

TL;DR

This paper proposes AMBER, a computationally efficient wrapper feature selection method that combines autoencoders for redundancy detection and a single-train ranker model for relevance scoring, enabling greedy backward elimination without retraining. AMBER achieves higher classification accuracy than baseline methods on multiple datasets, including a 99% accuracy on RadioML2016.10b after feature reduction, demonstrating superior performance with reduced computational cost compared to traditional wrappers.

ABSTRACT

We propose a computationally efficient wrapper feature selection method - called Autoencoder and Model Based Elimination of features using Relevance and Redundancy scores (AMBER) - that uses a single ranker model along with autoencoders to perform greedy backward elimination of features. The ranker model is used to prioritize the removal of features that are not critical to the classification task, while the autoencoders are used to prioritize the elimination of correlated features. We demonstrate the superior feature selection ability of AMBER on 4 well known datasets corresponding to different domain applications via comparing the classification accuracies with other computationally efficient state-of-the-art feature selection techniques. Interestingly, we find that the ranker model that is used for feature selection does not necessarily have to be the same as the final classifier that is trained on the selected features. Finally, we note how a smaller number of features can lead to higher accuracies on some datasets, and hypothesize that overfitting the ranker model on the training set facilitates the selection of more salient features.

Motivation & Objective

• To address the high computational cost of traditional wrapper feature selection methods that retrain classifiers at each iteration.
• To improve feature selection accuracy by integrating redundancy detection (via autoencoders) into wrapper methods, which typically ignore feature correlation.
• To maintain high classification performance while reducing training time by training the ranker model only once.
• To explore the impact of overfitting the ranker model on feature relevance score differentiation.
• To balance computational efficiency (like filter methods) with high performance (like wrappers), especially in high-dimensional settings.

Proposed method

• AMBER uses a single, pre-trained ranker model (e.g., a neural network) to assign relevance scores to features based on input layer weight magnitudes after training.
• Autoencoders are trained on the dataset to identify and group correlated features, enabling the detection of redundant features.
• Features are ranked for removal using a composite score based on relevance (from the ranker model) and redundancy (from autoencoder reconstruction error).
• Greedy backward elimination is performed by iteratively removing the least relevant and most redundant features in each step.
• The final feature subset is selected based on the ranker model's performance on the reduced feature set, simulating model accuracy without full retraining.
• Overfitting the ranker model via extended training epochs enhances the distinction between weight magnitudes of salient and non-salient features, improving selection accuracy.

Experimental results

Research questions

• RQ1 Can a single-train ranker model combined with autoencoders achieve competitive feature selection accuracy while significantly reducing computational cost?
• RQ2 Does integrating redundancy detection via autoencoders improve feature selection performance in wrapper methods compared to standard wrappers?
• RQ3 Can overfitting the ranker model enhance the differentiation of relevance scores between salient and non-salient features?
• RQ4 How does AMBER compare in accuracy and efficiency to state-of-the-art filter and wrapper methods on diverse real-world datasets?
• RQ5 To what extent does feature reduction via AMBER mitigate overfitting and improve generalization in high-dimensional datasets?

Key findings

• AMBER achieved higher classification accuracy than all compared state-of-the-art methods on four benchmark datasets, including a 99% accuracy on the RadioML2016.10b dataset after reducing features from 256 to 64.
• On the Wisconsin Breast Cancer dataset, AMBER improved accuracy beyond the full-feature model, suggesting it mitigated overfitting by removing noisy or redundant features.
• In the RadioML2016.10b dataset, accuracy increased from 93% with all 256 features to 99% with 64 selected features, indicating effective noise and redundancy reduction.
• PCA visualizations confirmed that AMBER improved class separability in the reduced feature space, particularly for the AM-DSM and WBFM classes that were previously misclassified.
• Overfitting the ranker model for more training epochs increased the difference in input layer weight magnitudes between salient and non-salient features, enhancing relevance score discrimination.
• AMBER required significantly less time than traditional wrapper methods (which retrain at each iteration) but slightly more than the fastest filter methods, achieving a strong balance between speed and accuracy.