[Paper Review] PowerShap: A Power-Full Shapley Feature Selection Method

Jarne Verhaeghe, Jeroen Van Der Donckt+2 more|Ghent University Academic Bibliography (Ghent University)|Jun 16, 2022

Machine Learning and Data Classification4 citations

TL;DR

PowerShap is a novel, fast wrapper feature selection method that combines Shapley values with statistical power calculations and hypothesis testing to identify informative features efficiently. It achieves predictive performance comparable to state-of-the-art wrapper methods while being significantly faster—often reducing execution time by half or more—through intelligent iteration control and an automatic hyperparameter tuning mode.

ABSTRACT

Feature selection is a crucial step in developing robust and powerful machine learning models. Feature selection techniques can be divided into two categories: filter and wrapper methods. While wrapper methods commonly result in strong predictive performances, they suffer from a large computational complexity and therefore take a significant amount of time to complete, especially when dealing with high-dimensional feature sets. Alternatively, filter methods are considerably faster, but suffer from several other disadvantages, such as (i) requiring a threshold value, (ii) not taking into account intercorrelation between features, and (iii) ignoring feature interactions with the model. To this end, we present powershap, a novel wrapper feature selection method, which leverages statistical hypothesis testing and power calculations in combination with Shapley values for quick and intuitive feature selection. Powershap is built on the core assumption that an informative feature will have a larger impact on the prediction compared to a known random feature. Benchmarks and simulations show that powershap outperforms other filter methods with predictive performances on par with wrapper methods while being significantly faster, often even reaching half or a third of the execution time. As such, powershap provides a competitive and quick algorithm that can be used by various models in different domains. Furthermore, powershap is implemented as a plug-and-play and open-source sklearn component, enabling easy integration in conventional data science pipelines. User experience is even further enhanced by also providing an automatic mode that automatically tunes the hyper-parameters of the powershap algorithm, allowing to use the algorithm without any configuration needed.

Motivation & Objective

• To address the high computational cost of traditional wrapper feature selection methods in high-dimensional datasets.
• To overcome filter method limitations such as reliance on arbitrary thresholds, lack of inter-feature interaction modeling, and ignoring model-specific feature interactions.
• To develop a method that combines the predictive strength of wrappers with the speed of filters using statistical power and Shapley values.
• To enable plug-and-play integration into scikit-learn workflows with minimal configuration through an automatic tuning mode.
• To provide a scalable, open-source solution for efficient feature selection across diverse machine learning domains.

Proposed method

• PowerShap uses Shapley values to quantify each feature’s contribution to model predictions, leveraging model-agnostic interpretability.
• It applies statistical hypothesis testing to determine whether a feature’s Shapley value significantly differs from that of a random baseline feature.
• The method employs power calculations to dynamically determine the minimum number of iterations required to achieve sufficient statistical power, reducing unnecessary computation.
• An automatic mode tunes hyperparameters (e.g., number of iterations) based on power requirements, enabling configuration-free usage.
• A recursive convergence mode excludes previously selected features and re-runs the process to detect additional informative features, improving detection in high-dimensional settings.
• The algorithm is implemented as a scikit-learn-compatible estimator, enabling seamless integration into standard ML pipelines.

Experimental results

Research questions

• RQ1 Can a wrapper feature selection method achieve performance on par with state-of-the-art methods while significantly reducing computational time?
• RQ2 Can statistical power calculations be effectively used to optimize the number of iterations in Shapley-based feature selection, improving efficiency?
• RQ3 Does the automatic hyperparameter tuning mode in PowerShap maintain high performance without requiring user input?
• RQ4 How does PowerShap compare to filter and wrapper baselines in terms of feature selection accuracy and execution speed across diverse datasets?
• RQ5 To what extent does the recursive convergence mode improve detection of informative features in high-dimensional, complex datasets?

Key findings

• PowerShap achieves predictive performance comparable to state-of-the-art wrapper methods like ShapICant, while reducing execution time by up to half or more in benchmark experiments.
• In a simulation with 500 features and 450 informative features, PowerShap’s convergence mode increased the percentage of detected informative features from 38% to 73% without introducing noise features.
• The automatic mode of PowerShap enables effective feature selection without any user configuration, maintaining strong performance across diverse datasets.
• PowerShap outperforms traditional filter methods in feature selection quality, especially in scenarios where feature intercorrelations and model interactions matter.
• The method demonstrates robustness across classification and regression benchmarks, with consistent performance gains in both speed and accuracy.
• The open-source, scikit-learn-compatible implementation enables easy integration into existing data science workflows, enhancing accessibility and adoption.