[Paper Review] Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability

Ninghao Liu, Yunsong Meng+3 more|arXiv (Cornell University)|Sep 16, 2020

Explainable Artificial Intelligence (XAI)38 references4 citations

TL;DR

This paper proposes a Definition Driven Pipeline (DDP) for post-hoc interpretability in NLP, defining interpretation via adversarial perturbations to ensure alignment between interpretation algorithms and evaluation metrics. It demonstrates that metric bias can distort faithfulness assessments, and shows that DDP unifies existing methods while enabling better alignment with human rationales through adversarial training.

ABSTRACT

Recent years have witnessed an increasing number of interpretation methods being developed for improving transparency of NLP models. Meanwhile, researchers also try to answer the question that whether the obtained interpretation is faithful in explaining mechanisms behind model prediction? Specifically, (Jain and Wallace, 2019) proposes that "attention is not explanation" by comparing attention interpretation with gradient alternatives. However, it raises a new question that can we safely pick one interpretation method as the ground-truth? If not, on what basis can we compare different interpretation methods? In this work, we propose that it is crucial to have a concrete definition of interpretation before we could evaluate faithfulness of an interpretation. The definition will affect both the algorithm to obtain interpretation and, more importantly, the metric used in evaluation. Through both theoretical and experimental analysis, we find that although interpretation methods perform differently under a certain evaluation metric, such a difference may not result from interpretation quality or faithfulness, but rather the inherent bias of the evaluation metric.

Motivation & Objective

• To address the lack of consistent evaluation standards for post-hoc interpretation methods in NLP.
• To identify and mitigate inherent biases in existing evaluation metrics that distort faithfulness assessments.
• To unify diverse interpretation methods under a single, principled definition based on adversarial attacks.
• To improve alignment between model interpretations and human-specified rationales through adversarial training.
• To provide a systematic framework for developing and evaluating faithful, definition-driven interpretation methods.

Proposed method

• Defining interpretation as the minimal perturbation required to change model predictions, using adversarial attacks as the core definition.
• Deriving existing interpretation methods (e.g., gradient-based, attention-based) as special cases of the DDP framework under specific constraints.
• Formulating a metric for faithfulness that is intrinsically aligned with the definition, ensuring fair evaluation.
• Introducing a retraining strategy that optimizes interpretation to align with human-annotated rationales, using the CSA metric as a supervision signal.
• Applying the DDP framework to multiple models (LSTM, BERT, LSTM_att) and datasets (SST-2, Yelp, AGNews) to validate consistency and fairness.
• Using adversarial training with a fixed epsilon to generate perturbations and compute importance scores as ||x^n - x^{*n}||_2.

Experimental results

Research questions

• RQ1 Can a unified definition of interpretation resolve inconsistencies in evaluating faithfulness across different methods?
• RQ2 To what extent do evaluation metrics introduce bias that distorts the perceived performance of interpretation methods?
• RQ3 How can interpretation be systematically aligned with human-specified rationales in model explanations?
• RQ4 Are commonly used interpretation methods like attention and gradients truly faithful, or is their evaluation skewed by metric design?
• RQ5 Can adversarial perturbations serve as a principled, general definition for post-hoc interpretation in NLP?

Key findings

• The faithfulness of interpretation methods is highly sensitive to the choice of evaluation metric, with significant performance differences arising not from method quality but from metric bias.
• Many existing interpretation methods, including Integrated Gradient and SmoothGrad, are mathematically equivalent to the DDP framework under specific constraints, demonstrating their unification under a single definition.
• Adversarial training with human-annotated rationales improves alignment between model interpretations and human cognition, increasing similarity from 0.48 to 0.72 on SST-2 after retraining.
• On the Yelp dataset, the DDP-based evaluation reveals that VaGrad and SmoothGrad outperform other methods under the CSA metric, while performance rankings shift under ERA and MMA metrics, highlighting metric dependency.
• The two properties proposed by Jain and Wallace (2019) for faithfulness are not universally valid—they depend on the underlying definition of interpretation, which DDP formalizes.
• Model accuracy slightly decreases after adversarial retraining (e.g., from 0.81 to 0.791 on SST-2), confirming a known trade-off between robustness and accuracy in adversarial training.