[Paper Review] Towards Robust Interpretability with Self-Explaining Neural Networks

David Alvarez-Melis, Tommi Jaakkola|arXiv (Cornell University)|Jun 20, 2018

Explainable Artificial Intelligence (XAI)Computer Science22 references419 citations

TL;DR

The paper proposes self-explaining neural networks (Senn) that embed interpretability into the model via interpretable basis concepts and a locality-driven regularization, enforcing explicit, faithful, and stable explanations during training.

ABSTRACT

Most recent work on interpretability of complex machine learning models has focused on estimating $ extit{a posteriori}$ explanations for previously trained models around specific predictions. $ extit{Self-explaining}$ models where interpretability plays a key role already during learning have received much less attention. We propose three desiderata for explanations in general -- explicitness, faithfulness, and stability -- and show that existing methods do not satisfy them. In response, we design self-explaining models in stages, progressively generalizing linear classifiers to complex yet architecturally explicit models. Faithfulness and stability are enforced via regularization specifically tailored to such models. Experimental results across various benchmark datasets show that our framework offers a promising direction for reconciling model complexity and interpretability.

Motivation & Objective

• Define three core desiderata for explanations: explicitness, faithfulness, and stability.
• Develop self-explaining models that are architecturally interpretable and locally linear in a learned basis.
• Introduce regularization to enforce faithful, stable explanations while preserving predictive performance.
• Learn interpretable basis concepts via an autoencoder and grounding prototypes to provide human-understandable explanations.

Proposed method

• Generalize linear models to f(x)=theta(x)^T h(x) where theta depends on input x and h(x) are interpretable basis concepts.
• Introduce aggregation function g over the terms theta_i(x) h_i(x) with properties: monotone, additive, and non-mading interactions (P1-P5).
• Impose local stability by enforcing a local relation: gradient of f approximates theta(x0) in a neighborhood (L_theta regularization).
• Learn h(x) as either raw inputs or higher-level concepts via an autoencoder, with grounding and diversity constraints (L_h) and prototype-based grounding.
• Train end-to-end with a combined loss: L_y + lambda L_theta + xi L_h (and potentially other regularizers).
• Optionally realize theta with neural networks to retain high modeling capacity while preserving interpretability at the concept level.

Experimental results

Research questions

• RQ1 How can we design models whose explanations are explicit, faithful, and stable by construction?
• RQ2 Can we extend linear interpretability to complex models through input-dependent coefficients and interpretable basis concepts?
• RQ3 Do gradient-based regularizations that align model sensitivity with concept-based explanations improve robustness without sacrificing accuracy?
• RQ4 How useful are autoencoder-based, groundable, and prototype-grounded concepts as interpretable units for explaining predictions?

Key findings

• Self-explaining models yield explanations that are immediate and interpretable via the learned concepts and their relevance scores.
• The proposed gradient regularization L_theta improves stability and faithfulness of explanations, with a tradeoff controlled by lambda.
• Grounding concepts through prototypes and autoencoder-based learning provides meaningful, human-understandable explanations.
• Across MNIST, UCI, and Compas datasets, Senn matches non-interpretable baselines in accuracy while delivering robust, concept-based explanations.
• Senn outperforms post-hoc interpretability methods (LIME, SHAP, occlusion, etc.) in robustness of explanations to input perturbations.
• The framework supports end-to-end training and flexible incorporation of higher-level concepts beyond raw features.