[Paper Review] On the Interpretability of Attention Networks

Lakshmi Narayan Pandey, Rahul Vashisht+1 more|arXiv (Cornell University)|Dec 30, 2022

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper introduces the Selective Dependence Classification (SDC) task and a Focus-Classify Attention Model (FCAM) to enable objective, quantitative evaluation of attention model interpretability. It demonstrates that standard attention models can be highly accurate yet poorly interpretable, and shows that sparsity-inducing methods like sparsemax significantly improve interpretability without sacrificing accuracy.

ABSTRACT

Attention mechanisms form a core component of several successful deep learning architectures, and are based on one key idea: ''The output depends only on a small (but unknown) segment of the input.'' In several practical applications like image captioning and language translation, this is mostly true. In trained models with an attention mechanism, the outputs of an intermediate module that encodes the segment of input responsible for the output is often used as a way to peek into the `reasoning` of the network. We make such a notion more precise for a variant of the classification problem that we term selective dependence classification (SDC) when used with attention model architectures. Under such a setting, we demonstrate various error modes where an attention model can be accurate but fail to be interpretable, and show that such models do occur as a result of training. We illustrate various situations that can accentuate and mitigate this behaviour. Finally, we use our objective definition of interpretability for SDC tasks to evaluate a few attention model learning algorithms designed to encourage sparsity and demonstrate that these algorithms help improve interpretability.

Motivation & Objective

• To define a formal, quantitative framework for measuring interpretability in attention models, moving beyond subjective assessments.
• To identify and analyze error modes where attention models achieve high accuracy but fail to be interpretable due to misaligned attention distributions.
• To evaluate the effectiveness of various attention learning algorithms—especially those promoting sparsity—in improving interpretability on SDC tasks.
• To demonstrate that interpretability and accuracy are distinct objectives, and that high accuracy does not imply high interpretability.
• To provide empirical benchmarks for attention model interpretability using controlled datasets (CIFAR-SDC and synthetic SDC).

Proposed method

• Introduces the Selective Dependence Classification (SDC) problem, a simplified classification task where only one input segment (the 'foreground') is relevant to the output.
• Proposes the Focus-Classify Attention Model (FCAM), a two-stage architecture: a focus network computes attention weights α, and a classifier uses the attended input segment.
• Defines interpretability via the focus score: the attention weight α_j* for the true foreground segment j* must be high and dominant.
• Uses three metrics to quantify interpretability: Fraction of True (FT), NNZ (non-zero elements), and entropy of attention vector α.
• Employs loss functions with sparsity regularization (e.g., entropy regularization) and alternative activation functions (sparsemax, spherical softmax) to encourage attention concentration on the true segment.
• Evaluates models on two datasets: a synthetic SDC dataset and CIFAR-SDC, with repeated runs to ensure statistical reliability.

Experimental results

Research questions

• RQ1 Can we define a quantitative, objective measure of interpretability for attention models in classification tasks?
• RQ2 Under what conditions can an attention model be accurate yet non-interpretable, and what are the underlying failure modes?
• RQ3 Do attention learning algorithms designed to promote sparsity (e.g., sparsemax, entropy regularization) improve interpretability without degrading accuracy?
• RQ4 How does the choice of input to the focus network (raw input vs. final hidden layer) affect interpretability?
• RQ5 Is there a trade-off between accuracy and interpretability in hard attention mechanisms?

Key findings

• Standard attention models (e.g., vanilla softmax) achieve high test accuracy (~95%) on CIFAR-SDC but exhibit poor interpretability, with only ~80% of test instances correctly identifying the true foreground via attention.
• Sparsemax and spherical softmax activation functions improve the Fraction of True (FT) metric by over 10 percentage points (e.g., from 81.11% to 91.43%), indicating significantly better attention localization.
• Entropy regularization and other sparsity-inducing methods show no measurable improvement in FT, despite reducing attention vector entropy and NNZ, suggesting they do not enhance interpretability.
• Using the final hidden layer of the focus network instead of raw input improves FT by up to 10% in low-dimensional synthetic data, indicating feature-level aggregation enhances interpretability.
• Hard attention models achieve high interpretability (FT ~87.63%) but suffer from significantly lower accuracy (74.45%), highlighting a strong trade-off between interpretability and performance.
• The results confirm that interpretability and accuracy are decoupled: models can be highly accurate yet poorly interpretable, and vice versa, necessitating explicit interpretability objectives.