[Paper Review] DeepPINK: reproducible feature selection in deep neural networks

Yang Lu, Yingying Fan+2 more|arXiv (Cornell University)|Sep 4, 2018

Explainable Artificial Intelligence (XAI)69 citations

TL;DR

DeepPINK integrates model-X knockoffs with a pairwise-connected neural network to achieve false discovery rate control in feature selection for DNNs and demonstrates higher power than naive knockoff-augmented models on simulations and real data.

ABSTRACT

Deep learning has become increasingly popular in both supervised and unsupervised machine learning thanks to its outstanding empirical performance. However, because of their intrinsic complexity, most deep learning methods are largely treated as black box tools with little interpretability. Even though recent attempts have been made to facilitate the interpretability of deep neural networks (DNNs), existing methods are susceptible to noise and lack of robustness. Therefore, scientists are justifiably cautious about the reproducibility of the discoveries, which is often related to the interpretability of the underlying statistical models. In this paper, we describe a method to increase the interpretability and reproducibility of DNNs by incorporating the idea of feature selection with controlled error rate. By designing a new DNN architecture and integrating it with the recently proposed knockoffs framework, we perform feature selection with a controlled error rate, while maintaining high power. This new method, DeepPINK (Deep feature selection using Paired-Input Nonlinear Knockoffs), is applied to both simulated and real data sets to demonstrate its empirical utility.

Motivation & Objective

• Motivate interpretability and reproducibility in deep neural nets through controlled-feature selection.
• Leverage the model-X knockoffs framework to control the FDR in high-dimensional settings.
• Develop a DNN architecture that allows competition between original features and their knockoffs to improve power.
• Demonstrate empirical utility on simulated data and real datasets (HIV-1 mutations, gut microbiome).

Proposed method

• Introduce a pairwise-connected layer in a DNN where each input feature is paired with its knockoff counterpart.
• Define feature importance measures Z_j and  Z_j using filter weights and subsequent MLP layer weights.
• Compute knockoff statistics W_j = Z_j^2 -  Z_j^2 and apply the knockoff thresholding rules T or T+ for FDR control.
• Train the network with L1 regularization and Adam optimization, using an MLP with two hidden layers.
• Construct knockoffs under a Gaussian design and estimate the precision matrix when needed (ISEE for scalability).
• Compare against baselines (MLP, DeepLIFT, RF, SVR) in simulations, evaluating empirical FDR and power.

Experimental results

Research questions

• RQ1 Can DeepPINK achieve finite-sample FDR control while maintaining high power in feature selection for DNNs?
• RQ2 Does integrating a pairwise-knockoff layer improve feature selection performance over naïve knockoff feeding in DNNs?
• RQ3 How does DeepPINK perform under linear and nonlinear data-generating processes (linear model and Single-Index model) across varying dimensionality?
• RQ4 Do the methods identify known relevant features in real biological datasets (HIV-1 mutations, gut microbiome) at a controlled FDR?

Key findings

• DeepPINK consistently controls FDR below the target level q=0.2 in simulations.
• DeepPINK achieves higher power than competing methods (MLP, DeepLIFT, RF, SVR) across most settings in the synthetic experiments.
• The pairwise-connected layer and the competition between X_j and  X_j significantly boost power relative to naive knockoff-augmented networks.
• In real-data applications, DeepPINK identifies mutations and microbiome features with support from existing literature, often outperforming or matching Knockoff and BHq in FDR control and discovery.
• The approach avoids strong model assumptions about the underlying data-generating process and is applicable to other network architectures beyond MLP.