[Paper Review] Learning Global Additive Explanations for Neural Nets Using Model Distillation

Sarah Tan, Rich Caruana+3 more|arXiv (Cornell University)|Sep 27, 2018

Explainable Artificial Intelligence (XAI)22 references83 citations

TL;DR

This paper proposes a method to learn global additive explanations for neural networks using model distillation, enabling feature shapes that capture general model behavior beyond local predictions. The approach yields interpretable, quantitative insights into how input features influence model outputs across the entire dataset.

ABSTRACT

Interpretability has largely focused on local explanations, i.e. explaining why a model made a particular prediction for a sample. These explanations are appealing due to their simplicity and local fidelity. However, they do not provide information about the general behavior of the model. We propose to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions. These global explanations take the form of feature shapes, which are more expressive than feature attributions. Through careful experimentation, we show qualitatively and quantitatively that global additive explanations are able to describe model behavior and yield insights about models such as neural nets. A visualization of our approach applied to a neural net as it is trained is available at this https URL.

Motivation & Objective

• To address the limitation of local explanations in capturing the general behavior of neural networks.
• To develop a method that learns global, additive explanations describing feature contributions across the entire input space.
• To use model distillation to transfer knowledge from a complex model to a simpler, interpretable surrogate model.
• To produce feature shapes—more expressive than attributions—that reveal consistent relationships between inputs and predictions.
• To provide qualitative and quantitative insights into neural network behavior through global interpretability.

Proposed method

• Leverage model distillation to train a surrogate model that mimics the predictions of a trained neural network.
• Train the surrogate model to output additive explanations by optimizing for feature-wise contributions.
• Use the distilled surrogate model to learn feature shapes that represent the global influence of each input feature.
• Ensure fidelity by aligning the surrogate's predictions with the original model's outputs across diverse inputs.
• Visualize the resulting feature shapes to interpret how features collectively affect predictions.
• Apply the method during training to observe the evolution of feature importance over time.

Experimental results

Research questions

• RQ1 Can model distillation effectively produce global additive explanations that reflect the true behavior of a neural network?
• RQ2 How do the learned feature shapes compare to local attributions in capturing model behavior?
• RQ3 To what extent can global explanations reveal consistent, interpretable patterns in model decision-making?
• RQ4 How does the method perform across different datasets and model architectures?
• RQ5 Can the evolution of feature shapes during training provide insights into learning dynamics?

Key findings

• Global additive explanations derived via model distillation successfully capture the general behavior of neural networks across diverse inputs.
• Feature shapes provide more expressive and consistent interpretations than local attributions, revealing stable relationships between features and predictions.
• The method enables visualization of how feature importance evolves during training, offering insights into learning dynamics.
• Quantitative evaluation shows strong alignment between the surrogate model's explanations and the original model's behavior.
• The approach yields interpretable, global insights that are both qualitatively meaningful and quantitatively reliable.
• The method generalizes across datasets and architectures, demonstrating robustness and scalability.