[Paper Review] Learning to Explain: An Information-Theoretic Perspective on Model Interpretation

Jianbo Chen, Le Song+2 more|arXiv (Cornell University)|Feb 21, 2018

Machine Learning and Data Classification274 citations

TL;DR

The paper presents an instancewise feature selector that maximizes mutual information between a learned feature subset and the model output, using a variational lower bound and a continuous relaxation for efficient training and explanation.

ABSTRACT

We introduce instancewise feature selection as a methodology for model interpretation. Our method is based on learning a function to extract a subset of features that are most informative for each given example. This feature selector is trained to maximize the mutual information between selected features and the response variable, where the conditional distribution of the response variable given the input is the model to be explained. We develop an efficient variational approximation to the mutual information, and show the effectiveness of our method on a variety of synthetic and real data sets using both quantitative metrics and human evaluation.

Motivation & Objective

• Define instancewise feature selection as selecting a subset of features per instance that informs the model output.
• Propose an information-theoretic objective to maximize mutual information between selected features and the model response.
• Develop a tractable variational lower bound and a neural network-based explainer for efficient training.
• Demonstrate effectiveness and efficiency on synthetic and real datasets with quantitative metrics and human evaluation.

Proposed method

• Formulate explanations as selecting a k-feature subset S per input X to maximize I(X_S; Y).
• Introduce a variational lower bound for mutual information using a family Q of distributions for Y|X_S.
• Parameterize Q with a single neural network g_alpha to model P_m(Y|X_S).
• Use a continuous relaxation (Gumbel-softmax / Concrete) to sample subsets S and enable differentiable optimization.
• Jointly optimize explainer parameters theta and model-approximation parameters alpha via stochastic gradient methods.
• Explain later by ranking features according to the learned weights w_theta(X) and selecting the top-k features.

Experimental results

Research questions

• RQ1 Can instancewise feature subsets be learned that maximize mutual information with the model output for each instance?
• RQ2 Is a tractable variational bound sufficient to train an explainer that is model-agnostic and efficient at inference?
• RQ3 Does the proposed L2X approach outperform existing instancewise explanation methods in accuracy and speed?
• RQ4 How does the method perform on synthetic nonlinear data and real-world datasets (text and images) for explanation quality and human alignment?

Key findings

• L2X outperforms baselines (Saliency, DeepLIFT, SHAP, LIME) on nonlinear and switching feature datasets in terms of feature ranking quality.
• The approach yields competitive or superior post-hoc accuracy and aligns well with human judgments on sentiment and MNIST experiments.
• L2X is more efficient at explanation time, requiring only a single forward pass to produce explanations per instance.
• Training time becomes a smaller portion of total time as data scale increases, improving relative efficiency.
• The method enables model-agnostic explanations while maintaining high fidelity to the original model predictions.