[Paper Review] The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?

Jasmijn Bastings, Katja Filippova|arXiv (Cornell University)|Oct 12, 2020

Explainable Artificial Intelligence (XAI)24 references4 citations

TL;DR

This position paper argues that input saliency methods are superior to attention mechanisms for explaining model predictions, especially for model developers seeking faithful, token-level relevance scores. Despite attention's ease of extraction, saliency methods better fulfill the goal of identifying the most influential input tokens, and the authors advocate shifting focus from attention to saliency for interpretability in NLP.

ABSTRACT

There is a recent surge of interest in using attention as explanation of model predictions, with mixed evidence on whether attention can be used as such. While attention conveniently gives us one weight per input token and is easily extracted, it is often unclear toward what goal it is used as explanation. We find that often that goal, whether explicitly stated or not, is to find out what input tokens are the most relevant to a prediction, and that the implied user for the explanation is a model developer. For this goal and user, we argue that input saliency methods are better suited, and that there are no compelling reasons to use attention, despite the coincidence that it provides a weight for each input. With this position paper, we hope to shift some of the recent focus on attention to saliency methods, and for authors to clearly state the goal and user for their explanations.

Motivation & Objective

• To challenge the widespread use of attention as explanation in NLP, especially when saliency methods already serve the same purpose.
• To identify the implicit goal of attention-as-explanation: identifying the most relevant input tokens for a prediction.
• To clarify that the intended user of such explanations is typically a model developer, for whom faithfulness is critical.
• To argue that saliency methods are better suited than attention for achieving faithful, token-level relevance attribution.
• To encourage researchers to explicitly define the goal and user of explanations to improve interpretability research.

Proposed method

• Analyzes the implicit goal of attention-as-explanation: identifying the most important input tokens for a prediction, particularly for model developers.
• Compares attention weights to gradient-based saliency methods (e.g., Integrated Gradients, Grad-CAM) in terms of faithfulness to model behavior.
• Evaluates attention against established saliency benchmarks, noting that saliency methods are designed specifically for input relevance attribution.
• Reviews empirical evidence showing attention often correlates poorly with gradient-based importance measures and fails under adversarial perturbations.
• Highlights that saliency methods require only a single forward and backward pass—making them computationally efficient, similar to attention.
• Proposes that attention should not be used as a proxy for saliency, despite its convenience in providing per-token weights.

Experimental results

Research questions

• RQ1 Why is attention commonly used as an explanation when saliency methods already exist for the same purpose?
• RQ2 What is the true goal of attention-as-explanation in practice, and who is the intended user?
• RQ3 How does the faithfulness of attention weights compare to that of established saliency methods in identifying relevant input tokens?
• RQ4 Are there compelling reasons to prefer attention over saliency methods for input relevance explanation?
• RQ5 What alternative explanation methods might better serve different interpretability goals beyond per-token saliency?

Key findings

• The primary goal of attention-as-explanation is to identify the most relevant input tokens for a prediction, a task that saliency methods are explicitly designed to address.
• Empirical studies show that attention weights often correlate poorly with gradient-based importance measures, indicating low faithfulness.
• Attention mechanisms can be manipulated to produce deceptive weights that misrepresent model reliance on input tokens, even when the model still makes correct predictions.
• Saliency methods such as Integrated Gradients and Grad-CAM are more reliable and faithful in identifying input features critical to predictions.
• Despite attention's computational convenience, saliency methods are equally efficient—requiring only a forward and backward pass—and are not more complex to implement.
• The use of attention as explanation often assumes a causal definition of explanation, which it fails to satisfy, as attention reflects attention at a single point in the computation, not causal influence.