[Paper Review] Entropic Variable Boosting for Explainability and Interpretability in Machine Learning.

François Bachoc, Fabrice Gamboa+2 more|arXiv (Cornell University)|Oct 18, 2018

Explainable Artificial Intelligence (XAI)Computer Science27 references3 citations

TL;DR

This paper introduces Entropic Variable Boosting, a scalable method for explaining black-box machine learning models by systematically stressing each input variable while preserving data distribution. By reweighting reference samples rather than generating new ones, the approach efficiently computes variable importance, demonstrating improved interpretability on standard datasets with minimal computational overhead.

ABSTRACT

In this paper, we present a new explainability formalism to make clear the impact of each variable on the predictions given by black-box decision rules. Our method consists in evaluating the decision rules on test samples generated in such a way that each variable is stressed incrementally while preserving the original distribution of the machine learning problem. We then propose a new computation-ally efficient algorithm to stress the variables, which only reweights the reference observations and predictions. This makes our methodology scalable to large datasets. Results obtained on standard machine learning datasets are presented and discussed.

Motivation & Objective

• To address the lack of clear, distribution-preserving methods for assessing individual variable impacts in black-box machine learning models.
• To develop a scalable approach that avoids expensive data generation while maintaining fidelity to the original data distribution.
• To provide a computationally efficient alternative to existing feature attribution methods for model explainability.
• To enable precise, incremental stress-testing of each variable to quantify its contribution to model predictions.

Proposed method

• The method generates test samples by incrementally stressing each input variable while preserving the original joint distribution of the dataset.
• It introduces a reweighting strategy that modifies reference observations and predictions instead of generating new synthetic data.
• The approach leverages entropy-based measures to quantify the impact of variable stress on model outputs.
• Variable importance is derived from changes in model predictions under controlled stress conditions across the input space.
• The algorithm is designed to be computationally efficient, enabling scalability to large datasets by avoiding full data re-sampling.
• The method maintains distributional consistency by ensuring that stressed variables remain within plausible statistical bounds.

Experimental results

Research questions

• RQ1 How can variable importance be measured in a way that preserves the original data distribution during stress-testing?
• RQ2 Can a reweighting-based approach achieve comparable interpretability to data generation methods with significantly reduced computational cost?
• RQ3 To what extent does the method reveal meaningful, incremental impacts of individual variables on model predictions?
• RQ4 How does the method scale across large datasets compared to traditional feature attribution techniques?

Key findings

• The proposed method achieves scalable variable importance estimation by reweighting existing data points instead of generating new ones.
• Results on standard machine learning datasets demonstrate that the method effectively captures variable impacts while preserving data distribution.
• The approach shows computational efficiency, making it suitable for large-scale datasets where data generation would be prohibitive.
• The method provides a clear, incremental assessment of variable influence, enhancing model interpretability without sacrificing fidelity.