[Paper Review] Attention Interpretability Across NLP Tasks

Shikhar Vashishth, Shyam Upadhyay+2 more|arXiv (Cornell University)|Sep 24, 2019

Explainable Artificial Intelligence (XAI)31 references69 citations

TL;DR

The paper provides a systematic analysis showing that attention weights can be interpretable and correlate with feature importance across diverse NLP tasks, but only when attention genuinely influences the model's prediction, with single-sequence tasks behaving like gating rather than explanatory weights.

ABSTRACT

The attention layer in a neural network model provides insights into the model's reasoning behind its prediction, which are usually criticized for being opaque. Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019; Vig & Belinkov, 2019). Amid such confusion arises the need to understand attention mechanism more systematically. In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not). Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.

Motivation & Objective

• Extend attention interpretability analysis to diverse NLP tasks beyond text classification.
• Identify conditions under which attention weights correlate with feature importance.
• Explain why attention can be a gate in single-sequence tasks but not in more complex tasks.
• Validate interpretability through manual evaluation across multiple datasets and architectures.

Proposed method

• Evaluate attention in single-sequence, pair-sequence, and generation tasks using various attention mechanisms (Bahdanau-style, hierarchical, self-attention, and Transformer).
• Compare models with uniform, random, and permuted attention weights to assess impact on predictions.
• Provide theoretical interpretation of attention as a gating unit for single-sequence tasks.
• Use manual annotation to assess human interpretability of attention weights.
• Analyze correlations between attention weights and feature importance via JS divergence when inputs are ablated.
• Examine layer-wise effects in self-attention models on performance.

Experimental results

Research questions

• RQ1 When are attention weights interpretable across NLP tasks?
• RQ2 Do attention weights correlate with feature importance measures across tasks?
• RQ3 How does altering attention weights affect model predictions in single-sequence vs. pair-sequence and generation tasks?
• RQ4 Are attention weights human-interpretable to end users across different NLP tasks?
• RQ5 How do self-attention layers (e.g., Transformer) respond to permuted attention weights across layers?

Key findings

• Attention is not always explanatory in single-sequence tasks, where weights can act as gating rather than reasoning signals.
• Altering attention weights degrades performance much more in pair-sequence and generation tasks than in single-sequence tasks.
• Attention weights correlate with feature importance for tasks where attention is essential to prediction, especially in non-gating scenarios.
• Self-attention models (Transformers) show substantial performance changes when attention weights are permuted across layers, indicating genuine reliance on attention in these architectures.
• Manual human evaluation shows attention weights are generally meaningful in single-sequence tasks but less so in certain pair-sequence settings.