[Paper Review] Explaining Any ML Model? -- On Goals and Capabilities of XAI

Moritz Renftle, Holger Trittenbach+2 more|arXiv (Cornell University)|Jun 28, 2022

Explainable Artificial Intelligence (XAI)28 references4 citations

TL;DR

This paper reorients XAI research by framing it through users' actual questions about ML models, identifying that only one core question—how to represent a complex model as a simple function of interpretable attributes—is currently addressed by XAI algorithms. It reveals two key challenges: model approximation and translation of technical features into human-understandable terms, arguing that future XAI must holistically address both to meet real-world needs.

ABSTRACT

An increasing ubiquity of machine learning (ML) motivates research on algorithms to explain ML models and their predictions -- so-called eXplainable Artificial Intelligence (XAI). Despite many survey papers and discussions, the goals and capabilities of XAI algorithms are far from being well understood. We argue that this is because of a problematic reasoning scheme in XAI literature: XAI algorithms are said to complement ML models with desired properties, such as "interpretability", or "explainability". These properties are in turn assumed to contribute to a goal, like "trust" in an ML system. But most properties lack precise definitions and their relationship to such goals is far from obvious. The result is a reasoning scheme that obfuscates research results and leaves an important question unanswered: What can one expect from XAI algorithms? In this article, we clarify the goals and capabilities of XAI algorithms from a concrete perspective: that of their users. Explaining ML models is only necessary if users have questions about them. We show that users can ask diverse questions, but that only one of them can be answered by current XAI algorithms. Answering this core question can be trivial, difficult or even impossible, depending on the ML application. Based on these insights, we outline which capabilities policymakers, researchers and society can reasonably expect from XAI algorithms.

Motivation & Objective

• To clarify the true goals and capabilities of XAI algorithms by shifting focus from abstract properties like 'interpretability' to the actual questions users ask about ML models.
• To identify which of these user questions are currently answerable by existing XAI methods, countering the vague and normative reasoning prevalent in XAI literature.
• To expose the limitations of current XAI approaches, which primarily address only one specific question: representing complex models via interpretable attributes.
• To highlight two fundamental challenges in XAI—approximation and translation—that must be addressed jointly for meaningful progress.
• To guide policymakers, researchers, and society toward realistic expectations of XAI by grounding its potential in concrete, user-centered goals.

Proposed method

• Analyzes seven distinct user questions about ML models using a spam detection use case, categorizing them by intent and complexity.
• Identifies that only one question—'How can one represent a complex ML model as a simple function of interpreted attributes?'—is currently addressed by XAI algorithms.
• Introduces two core challenges: (1) approximation, where surrogate models must balance fidelity and simplicity; and (2) translation, where technical model features (e.g., neuron activations) must be linked to human-interpretable concepts (e.g., objects, shapes).
• Reviews existing XAI methods such as LIME, SHAP, and integrated gradients as approximation techniques, and methods in computer vision and NLP that enable feature translation.
• Proposes that future XAI should integrate both approximation and translation, exemplified by neurosymbolic approaches like Neural Prototype Trees.
• Uses a user-centric lens to reframe XAI not as a pursuit of abstract properties, but as a technical problem of answering specific, actionable questions.

Experimental results

Research questions

• RQ1 Which questions do users actually ask about ML models, and how do these questions relate to the goals of XAI?
• RQ2 Which of these user questions can current XAI algorithms realistically answer?
• RQ3 What are the fundamental technical challenges that limit the effectiveness of XAI algorithms in answering user questions?
• RQ4 To what extent do existing XAI methods address both the approximation and translation challenges simultaneously?
• RQ5 How can XAI research be reoriented to align with realistic user needs rather than normative or ill-defined properties like 'interpretability'?

Key findings

• Only one user question—'How can one represent a complex ML model as a simple function of interpreted attributes?'—is currently addressed by XAI algorithms.
• The two main challenges in XAI are approximation (balancing fidelity and simplicity of surrogate models) and translation (mapping technical features to human-understandable concepts).
• Existing XAI methods such as LIME, SHAP, and integrated gradients primarily address the approximation challenge, but often fail to solve the translation challenge.
• Translation is underexplored; methods that link neuron activations in CNNs to visual concepts (e.g., shapes, objects) exist but require large annotated datasets.
• Jointly addressing approximation and translation remains an open challenge, especially in multimodal domains like vision, NLP, and physical process modeling.
• Neurosymbolic AI approaches, such as Neural Prototype Trees, offer a promising path toward integrated solutions by combining neural learning with symbolic interpretation.