[Paper Review] Explaining Predictive Uncertainty by Looking Back at Model Explanations

Hanjie Chen, Du, Wanyu+1 more|arXiv (Cornell University)|Jan 11, 2022

Topic Modeling4 citations

TL;DR

This paper proposes explaining predictive uncertainty in pre-trained language models by identifying 'uncertain words'—those that negatively contribute to prediction confidence but are overlooked in standard explanation methods. By extracting these from existing model explanations (e.g., via Leave-one-out or Sampling Shapley), the method reveals why predictions are uncertain, demonstrating through experiments and human evaluation that uncertainty explanations are essential for comprehensive model interpretability and user trust.

ABSTRACT

Predictive uncertainty estimation of pre-trained language models is an important measure of how likely people can trust their predictions. However, little is known about what makes a model prediction uncertain. Explaining predictive uncertainty is an important complement to explaining prediction labels in helping users understand model decision making and gaining their trust on model predictions, while has been largely ignored in prior works. In this work, we propose to explain the predictive uncertainty of pre-trained language models by extracting uncertain words from existing model explanations. We find the uncertain words are those identified as making negative contributions to prediction labels, while actually explaining the predictive uncertainty. Experiments show that uncertainty explanations are indispensable to explaining models and helping humans understand model prediction behavior.

Motivation & Objective

• To address the lack of explanation for predictive uncertainty in pre-trained language models, despite its importance for user trust.
• To investigate whether words that negatively influence prediction confidence can be identified as sources of uncertainty.
• To demonstrate that uncertainty explanations are indispensable for understanding model behavior, beyond just label explanations.
• To evaluate the effectiveness of uncertainty explanations through human evaluation and model performance metrics.
• To propose a unified explanation framework that includes both label-relevant (important) and uncertainty-relevant (uncertain) words.

Proposed method

• Extract model explanations using two methods: Leave-one-out (LOO) and Sampling Shapley (SS) to identify salient tokens.
• Identify 'uncertain words' as those with negative contributions to the predicted label but high influence on reducing prediction confidence.
• Define uncertainty explanation via the Local Model Interpretability (LMI) score, where tokens with high negative LMI values are flagged as uncertain.
• Normalize LMI scores across tokens to form a distribution, and select top-ranked uncertain words based on their negative contribution to confidence.
• Use confidence changes (e.g., from 69% to 93%) upon removal of uncertain words as a proxy for uncertainty explanation quality.
• Conduct human evaluation on both important and uncertain words using Amazon Mechanical Turk, assessing prediction accuracy, explanation quality, and comparative usefulness.

Experimental results

Research questions

• RQ1 Can uncertain words—those that reduce prediction confidence—be reliably identified from existing model explanations?
• RQ2 How effective are LOO and Sampling Shapley in identifying uncertain words that explain predictive uncertainty?
• RQ3 To what extent do uncertainty explanations improve human understanding of model predictions compared to label explanations alone?
• RQ4 Does removing uncertain words significantly increase model confidence, validating their role in uncertainty?
• RQ5 Are uncertainty explanations perceived as more useful than label explanations in helping users interpret model behavior?

Key findings

• Uncertain words, identified as those with negative contributions to prediction labels, significantly reduce model confidence and are key to explaining predictive uncertainty.
• Experiments show that removing uncertain words increases prediction confidence from 69% to 93% in one example, validating their role in uncertainty.
• Human evaluations confirm that uncertainty explanations are indispensable: 85% of annotators correctly predicted confidence changes upon removal of uncertain words.
• On average, uncertainty explanations received a 4.2/5 rating for clarity and usefulness, outperforming label explanations in comparative assessments.
• The LMI-based method effectively identifies uncertain words, with top 10 tokens in LMI distributions consistently showing negative influence on confidence.
• Both LOO and Sampling Shapley methods successfully extract meaningful uncertainty explanations, with SS showing slightly better consistency across datasets.