[Paper Review] How Much Can I Trust You? -- Quantifying Uncertainties in Explaining Neural Networks

Kirill Bykov, Marina Höhne+3 more|arXiv (Cornell University)|Jun 16, 2020

Explainable Artificial Intelligence (XAI)Computer Science37 references18 citations

TL;DR

This paper proposes B-LRP, a novel framework that quantifies uncertainty in neural network explanations by adapting any existing explanation method—like LRP—to Bayesian neural networks (BNNs). By sampling from the BNN's posterior, B-LRP generates explanation distributions, enabling percentile-based confidence levels (e.g., 5th percentile for conservative explanations), which significantly reduce spurious relevance in saliency maps and improve reliability in safety-critical applications.

ABSTRACT

Explainable AI (XAI) aims to provide interpretations for predictions made by learning machines, such as deep neural networks, in order to make the machines more transparent for the user and furthermore trustworthy also for applications in e.g. safety-critical areas. So far, however, no methods for quantifying uncertainties of explanations have been conceived, which is problematic in domains where a high confidence in explanations is a prerequisite. We therefore contribute by proposing a new framework that allows to convert any arbitrary explanation method for neural networks into an explanation method for Bayesian neural networks, with an in-built modeling of uncertainties. Within the Bayesian framework a network's weights follow a distribution that extends standard single explanation scores and heatmaps to distributions thereof, in this manner translating the intrinsic network model uncertainties into a quantification of explanation uncertainties. This allows us for the first time to carve out uncertainties associated with a model explanation and subsequently gauge the appropriate level of explanation confidence for a user (using percentiles). We demonstrate the effectiveness and usefulness of our approach extensively in various experiments, both qualitatively and quantitatively.

Motivation & Objective

• To address the lack of uncertainty quantification in existing explanation methods for deep neural networks.
• To enable trust-aware interpretation by modeling uncertainty in explanations, particularly in safety-critical domains.
• To extend any existing explanation method (e.g., LRP) to work with Bayesian neural networks, producing probabilistic explanation outputs.
• To provide users with adjustable confidence levels (via percentiles) for explanation reliability, such as 5th percentile for high-caution decisions.
• To empirically validate that uncertainty-aware explanations reduce false positives in saliency maps, improving fidelity and trust.

Proposed method

• The method transfers uncertainty from Bayesian neural networks (BNNs) to any existing explanation method via knowledge distillation from the BNN’s posterior distribution.
• For each input, multiple forward passes are performed using Monte Carlo dropout or posterior sampling to generate a distribution of explanation scores (e.g., relevance maps).
• The resulting explanation distribution is used to compute percentiles (e.g., 5th, 50th, 95th), enabling risk-adjusted explanations—lower percentiles yield more conservative, reliable regions.
• The framework is applied to LRP, resulting in B-LRP, which outputs a distribution of relevance maps instead of a single heatmap.
• Uncertainty is visualized spatially, highlighting regions of high confidence (red) and low confidence (blue), such as eliminating false relevance to irrelevant objects like gas lamps.
• The method is general and compatible with any model-agnostic or model-aware explanation method and any approximate inference in BNNs.

Experimental results

Research questions

• RQ1 Can we quantify uncertainty in neural network explanations, which are currently treated as deterministic and potentially misleading?
• RQ2 How can we adapt existing explanation methods like LRP to produce uncertainty-aware outputs using Bayesian neural networks?
• RQ3 To what extent can percentile-based explanation thresholds (e.g., 5th percentile) improve reliability by filtering out spurious relevance in saliency maps?
• RQ4 Does uncertainty-aware explanation improve performance in standard evaluation benchmarks such as pixel-flipping or feature importance ranking?
• RQ5 Can the proposed method reduce the 'clever Hans' effect, where models rely on spurious artifacts in data, by highlighting only the most certain features?

Key findings

• The 5th percentile B-LRP explanation outperformed standard LRP in the pixel-flipping evaluation on MNIST, correctly identifying the x% most important pixels for x ≤ 12.
• On ImageNet, the 5th percentile B-LRP explanation assigned relevance only to the castle object, eliminating false relevance to the gas lamppost, unlike standard LRP.
• The high-certainty (5th percentile) explanation was visually and quantitatively more accurate, confirming reduced spurious relevance and improved fidelity.
• B-LRP successfully revealed the 'clever Hans' effect by showing that standard LRP incorrectly attributed relevance to artifacts like gas lamps present in training data.
• Using 100 posterior samples provided stable uncertainty estimates for coarse-grained percentile analysis, though finer percentiles (e.g., 1st) require more samples.
• The method is general and can be applied to any explanation method and BNN inference technique, enabling risk-adjusted interpretation in diverse applications.