[Paper Review] NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks

Isaac Ahern, Adam Noack+4 more|arXiv (Cornell University)|Sep 10, 2019

Explainable Artificial Intelligence (XAI)26 references37 citations

TL;DR

NormLIME introduces a normalized aggregation of local LIME explanations to produce global and class-specific feature importances for deep networks, showing strong human preference and faithful feature ranking.

ABSTRACT

The problem of explaining deep learning models, and model predictions generally, has attracted intensive interest recently. Many successful approaches forgo global approximations in order to provide more faithful local interpretations of the model's behavior. LIME develops multiple interpretable models, each approximating a large neural network on a small region of the data manifold and SP-LIME aggregates the local models to form a global interpretation. Extending this line of research, we propose a simple yet effective method, NormLIME for aggregating local models into global and class-specific interpretations. A human user study strongly favored class-specific interpretations created by NormLIME to other feature importance metrics. Numerical experiments confirm that NormLIME is effective at recognizing important features.

Motivation & Objective

• Provide a metric to aggregate local explanations into global and class-specific interpretations for deep neural networks.
• Extend LIME/SP-LIME with a normalization scheme to improve interpretability at multiple granularity levels.
• Demonstrate through human studies and KAR-style evaluation that NormLIME yields accurate and understandable explanations.

Proposed method

• Build local explanations around data points using a sparse linear model as in LIME.
• Aggregate local explanations into global feature importances via a normalization-based weighting scheme (Equations 3–5).
• Partition local explanations by class labels to obtain class-specific salience (Equations 7–9).
• Compare NormLIME against baselines (LIME, SmoothGrad, VarGrad) in human studies on MNIST.
• Evaluate explanations with Keep-and-Retrain (KAR) style experiments on raw pixels and convolutional features.
• Discuss theoretical relationship to existing global explanation methods (SP-LIME, L2 vs L1 normalization).

Experimental results

Research questions

• RQ1 Can NormLIME produce global and class-specific explanations that align with human intuition?
• RQ2 Does NormLIME outperform baseline feature-importance methods in human evaluations?
• RQ3 Are NormLIME explanations faithful indicators of features critical to model performance as shown by KAR-style evaluation?
• RQ4 How does normalization impact the quality of class-specific salience in multi-class settings?

Key findings

• In a human study on MNIST, NormLIME received 939 votes versus 438 for LIME, 151 for VarGrad, and 132 for SmoothGrad.
• Participants preferred class-specific explanations from NormLIME over baselines with statistical significance (p<0.001 in Tukey HSD).
• NormLIME outperformed other methods in KAR-style evaluation, especially on convolutional features, with best performance when removing 70% of features achieving 99.31% accuracy.
• NormLIME explanations showed that normalization aids in revealing differences among important pixels, improving readability for class-level interpretations.
• Across evaluations, gradient-based methods lagged behind additive local-model approaches like SHAP, LIME, and NormLIME, with NormLIME excelling in class-specific interpretability.