[Paper Review] Rethinking Attention-Model Explainability through Faithfulness Violation Test

Yibing Liu, Haoliang Li+4 more|arXiv (Cornell University)|Jan 28, 2022

Explainable Artificial Intelligence (XAI)12 citations

TL;DR

This paper introduces a faithfulness violation test to evaluate attention-based explanation methods by assessing polarity consistency—whether explanation weights correctly reflect whether input features contribute to or suppress model predictions. The method reveals that most attention explanations, especially raw attention, suffer from high faithfulness violations, with gradient-based methods like Attention ⊙ Gradient showing significantly lower violation rates, highlighting the need to prioritize polarity consistency in explainability evaluation.

ABSTRACT

Attention mechanisms are dominating the explainability of deep models. They produce probability distributions over the input, which are widely deemed as feature-importance indicators. However, in this paper, we find one critical limitation in attention explanations: weakness in identifying the polarity of feature impact. This would be somehow misleading -- features with higher attention weights may not faithfully contribute to model predictions; instead, they can impose suppression effects. With this finding, we reflect on the explainability of current attention-based techniques, such as Attentio$\odot$Gradient and LRP-based attention explanations. We first propose an actionable diagnostic methodology (henceforth faithfulness violation test) to measure the consistency between explanation weights and the impact polarity. Through the extensive experiments, we then show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue, especially the raw attention. Empirical analyses on the factors affecting violation issues further provide useful observations for adopting explanation methods in attention models.

Motivation & Objective

• To address the critical gap in existing faithfulness evaluations that overlook polarity consistency in attention-based explanations.
• To diagnose whether explanation weights accurately reflect whether input features contribute to or suppress model predictions.
• To evaluate the performance of widely used explanation methods (e.g., Attention Rollout, LRP-based methods) under this new criterion.
• To identify architectural and methodological factors contributing to faithfulness violations in attention explanations.
• To advocate for polarity consistency as a necessary criterion in the design and evaluation of faithful explanation methods.

Proposed method

• Proposes a diagnostic faithfulness violation test that evaluates whether the sign of the top attention weight correctly indicates the actual impact polarity (contribution or suppression) of the corresponding input feature.
• For each sample, the method identifies the highest-magnitude attention weight and checks if its sign matches the true impact direction, determined via confidence perturbation after removing the top-10% weighted input components.
• Uses confidence perturbation (ΔC) to measure the effect of removing high-attention regions: ΔC < 0 indicates suppression, ΔC > 0 indicates contribution.
• Applies the test across nine explanation methods, including raw attention, Attention ⊙ Gradient, and LRP-based variants, on six diverse tasks and nine datasets.
• Employs replacement functions to mitigate out-of-distribution (OOD) issues in perturbation, ensuring robustness in faithfulness evaluation.
• Compares the violation test with existing faithfulness metrics, demonstrating that prior metrics fail to detect polarity inconsistencies.

Experimental results

Research questions

• RQ1 To what extent do existing attention-based explanation methods fail to preserve the polarity of feature impact, i.e., whether high attention weights correctly indicate whether a feature contributes to or suppresses the model prediction?
• RQ2 How does the proposed faithfulness violation test compare to existing faithfulness evaluation metrics in detecting polarity inconsistencies?
• RQ3 Which explanation methods exhibit the lowest faithfulness violation rates, and what architectural or methodological factors contribute to their improved performance?
• RQ4 How do model complexity and the ability to identify impact polarity influence the likelihood of faithfulness violations?
• RQ5 Can gradient-based attention explanations like Attention ⊙ Gradient reduce faithfulness violations compared to raw attention or LRP-based methods?

Key findings

• Raw attention exhibits a high faithfulness violation ratio of 40% on the VQA v2.0 dataset, indicating that in 40% of cases, high-attention regions suppress model confidence rather than support it.
• The Attention ⊙ Gradient method reduces violation rates to 2% on Yelp, 3% on AgNews, and 6% on VQA v2.0, demonstrating significantly improved polarity consistency.
• Methods based on absolute gradient values (e.g., Attention ⊙ |∇α|) and sign-based gradient (e.g., Attention ⊙ sign(∇α)) show moderate improvements, with violation ratios of 15%, 7%, and 25% on Yelp, AgNews, and VQA, respectively.
• The study finds that model architecture complexity and the inherent capability to identify impact polarity are key factors influencing the frequency of faithfulness violations.
• Existing faithfulness evaluation metrics fail to detect polarity inconsistencies, underscoring the necessity of the proposed violation test for comprehensive evaluation.
• Empirical results show that even with high importance correlation, explanation methods can be misleading if they do not preserve polarity consistency, as demonstrated in the VQA dataset's red-point violation cases.