[Paper Review] Nonparametric Feature Selection by Random Forests and Deep Neural Networks

Xiaojun Mao, Liuhua Peng+1 more|arXiv (Cornell University)|Jan 18, 2022

Neural Networks and Applications4 citations

TL;DR

This paper proposes NFSRD, a nonparametric feature selection method that combines random forests and deep neural networks using kernel-based two-sample tests to identify relevant features. It achieves superior performance in feature detection, false positive control, and computational efficiency compared to existing methods, with theoretical guarantees under regularity conditions.

ABSTRACT

Random forests are a widely used machine learning algorithm, but their computational efficiency is undermined when applied to large-scale datasets with numerous instances and useless features. Herein, we propose a nonparametric feature selection algorithm that incorporates random forests and deep neural networks, and its theoretical properties are also investigated under regularity conditions. Using different synthetic models and a real-world example, we demonstrate the advantage of the proposed algorithm over other alternatives in terms of identifying useful features, avoiding useless ones, and the computation efficiency. Although the algorithm is proposed using standard random forests, it can be widely adapted to other machine learning algorithms, as long as features can be sorted accordingly.

Motivation & Objective

• To address the computational inefficiency of random forests on high-dimensional datasets with many irrelevant features.
• To develop a theoretically grounded, nonparametric feature selection method that avoids strong assumptions on tree splits.
• To improve feature selection accuracy by integrating deep neural networks with random forests using nonparametric two-sample testing.
• To enhance computational efficiency through subsampling while maintaining selection accuracy.
• To provide a generalizable framework applicable to other machine learning algorithms beyond standard random forests.

Proposed method

• The method uses deep neural networks to perform nonparametric two-sample tests to assess feature relevance by comparing feature distributions between response classes.
• It applies maximum mean discrepancy (MMD) in a reproducing kernel Hilbert space (RKHS) to measure distribution differences without parametric assumptions.
• Feature selection is performed via a significance test using the MMD statistic, with p-values computed under the null hypothesis of no difference.
• Subsampling is employed to reduce computational cost, improving scalability to large datasets while preserving statistical power.
• The approach does not require median-based splits or strong assumptions on tree structure, unlike prior methods such as minimum depth selection.
• Theoretical properties are established under regularity conditions, including consistency of feature selection and asymptotic normality of the test statistic.

Figure 1: Boxplots for the computation time of different methods based on 200 Monte Carlo simulations, where the features are independently generated. The horizontal segments within the boxplots represent the median computation time. “BRT” represent Boruta, “MVS” stands for the minimum-depth variabl

Experimental results

Research questions

• RQ1 Can a nonparametric feature selection method based on deep neural networks and random forests outperform existing permutation-based and depth-based methods in identifying relevant features?
• RQ2 How does the proposed NFSRD method perform in terms of false discovery rate and true positive detection across synthetic and real-world datasets?
• RQ3 To what extent does subsampling improve computational efficiency without compromising feature selection accuracy?
• RQ4 What are the theoretical properties of the proposed method under regularity conditions, particularly regarding consistency and asymptotic distribution?
• RQ5 Can the framework be generalized to other machine learning algorithms beyond standard random forests?

Key findings

• The NFSRD method significantly outperforms alternative methods in detecting useful features while avoiding irrelevant ones, especially in high-dimensional settings with many useless features.
• On the superconductivity dataset, NFSRD selected 13 features at α=0.01 and 10 at α=0.05, all among the top 20 most important features identified by XGBoost, whereas BRT selected all features regardless of significance.
• Subsampling reduced computational cost and memory usage while maintaining high accuracy, making the method scalable to large datasets.
• Theoretical analysis confirms the consistency of the feature selection procedure and the asymptotic normality of the MMD test under regularity conditions.
• The method is robust to feature correlation and does not require strong assumptions on tree splits, unlike minimum depth-based approaches.
• The NFSRD framework is generalizable and can be adapted to other machine learning algorithms as long as features can be ranked or sorted.