[Paper Review] Why do explanations fail? A typology and discussion on failures in XAI

Clara Bove, Thibault Laugel+3 more|arXiv (Cornell University)|May 22, 2024

Scientific Computing and Data Management4 citations

TL;DR

This paper proposes a holistic typology of failures in eXplainable AI (XAI), distinguishing between system-specific and user-specific failures to address the complex, overlapping causes of explanation breakdowns. By analyzing technical limitations and human interpretation issues, it identifies key failure categories and advocates for improved transparency, user-centered interfaces, and interdisciplinary design to enhance explanation quality and reliability in ML systems.

ABSTRACT

As Machine Learning models achieve unprecedented levels of performance, the XAI domain aims at making these models understandable by presenting end-users with intelligible explanations. Yet, some existing XAI approaches fail to meet expectations: several issues have been reported in the literature, generally pointing out either technical limitations or misinterpretations by users. In this paper, we argue that the resulting harms arise from a complex overlap of multiple failures in XAI, which existing ad-hoc studies fail to capture. This work therefore advocates for a holistic perspective, presenting a systematic investigation of limitations of current XAI methods and their impact on the interpretation of explanations. % By distinguishing between system-specific and user-specific failures, we propose a typological framework that helps revealing the nuanced complexities of explanation failures. Leveraging this typology, we discuss some research directions to help practitioners better understand the limitations of XAI systems and enhance the quality of ML explanations.

Motivation & Objective

• To address the fragmented understanding of XAI failures by identifying overlapping technical and human-centered limitations.
• To develop a systematic, hierarchical typology of XAI failures that distinguishes between system-specific and user-specific failure types.
• To guide AI practitioners in diagnosing the root causes of explanation failures through a structured framework.
• To promote more transparent, user-centered XAI design by highlighting communication gaps and interaction limitations.
• To identify research directions that enhance explanation quality through improved transparency and interactive user interfaces.

Proposed method

• Conducting a comprehensive literature review of recent XAI studies focusing on limitations in explanations, interfaces, and evaluations.
• Proposing a two-tiered typology of XAI failures: system-specific (e.g., robustness, faithfulness, stability) and user-specific (e.g., misinterpretation, overtrust, cognitive biases).
• Analyzing how failures emerge from interactions between system design and user cognition, especially in the absence of clear communication about method assumptions.
• Emphasizing the need for improved transparency in XAI methods through design principles such as ML transparency and factual documentation, akin to Model Cards.
• Advocating for the development of advanced XAI user interfaces (XUIs) that support interactive, conversational, and narrative-based explanation delivery.
• Leveraging the typology to identify research avenues focused on diagnosing failure origins and mitigating combined failure effects.

Experimental results

Research questions

• RQ1 What are the primary categories of failure in XAI, and how do they overlap or interact?
• RQ2 How do system-level limitations such as instability or lack of faithfulness contribute to explanation failures?
• RQ3 In what ways do user-level factors—such as prior beliefs, cognitive biases, or lack of understanding—lead to misinterpretation of explanations?
• RQ4 How do communication gaps between XAI systems and users exacerbate explanation failures?
• RQ5 What design principles and interface modalities (e.g., conversational, visual) can reduce user-specific failures and improve explanation comprehension?

Key findings

• The paper identifies a critical gap in XAI research: failures are rarely studied holistically, leading to misdiagnosis and ineffective mitigation strategies.
• System-specific failures include issues like lack of faithfulness, robustness, and stability in explanation methods, which can mislead users even when explanations are technically accurate.
• User-specific failures arise from mismatches between explanation content and user needs, such as overtrust in explanations or incorrect inferences due to cognitive biases.
• Many failures stem not from flawed explanations per se, but from users’ lack of understanding of how explanation methods work, especially regarding feature interactions and trade-offs like locality vs. stability.
• The absence of interactive, conversational, or narrative-based interfaces limits users’ ability to explore and verify explanations, increasing reliance on external reasoning and risk of misinterpretation.
• The authors advocate for standardized documentation of XAI methods—similar to Model Cards—to improve transparency and help users understand methodological assumptions and limitations.