[Paper Review] Local Interpretations for Explainable Natural Language Processing: A Survey

Siwen Luo, Hamish Ivison+2 more|arXiv (Cornell University)|Mar 20, 2021

Topic ModelingComputer Science215 references17 citations

TL;DR

This survey provides a comprehensive analysis of local interpretability methods in natural language processing, categorizing them into feature importance, natural language explanations, and hidden state probing. It identifies key challenges in evaluation, including metric inconsistency and reliance on subjective human assessment, and calls for unified, user-centric evaluation frameworks to improve faithfulness and trustworthiness in model explanations for both correct and incorrect predictions.

ABSTRACT

As the use of deep learning techniques has grown across various fields over the past decade, complaints about the opaqueness of the black-box models have increased, resulting in an increased focus on transparency in deep learning models. This work investigates various methods to improve the interpretability of deep neural networks for Natural Language Processing (NLP) tasks, including machine translation and sentiment analysis. We provide a comprehensive discussion on the definition of the term interpretability and its various aspects at the beginning of this work. The methods collected and summarised in this survey are only associated with local interpretation and are specifically divided into three categories: 1) interpreting the model's predictions through related input features; 2) interpreting through natural language explanation; 3) probing the hidden states of models and word representations.

Motivation & Objective

• To provide a systematic review of local interpretability methods in NLP, focusing on explainability for individual predictions rather than global model behavior.
• To clarify the distinctions between interpretability, explainability, and faithfulness in the context of deep neural networks.
• To identify critical limitations in current evaluation practices, particularly the overreliance on human evaluation and inconsistent automatic metrics.
• To advocate for the development of comprehensive evaluation frameworks that assess faithfulness, stability, and comprehensibility across both correct and incorrect predictions.
• To emphasize the need for interpretable models that can explain not only correct decisions but also errors, to build trust and enable self-correcting AI systems.

Proposed method

• Categorizes local interpretability techniques into three main types: feature importance, natural language explanations (NLE), and probing of hidden states and word representations.
• Reviews feature importance methods such as LIME and Grad-CAM, which highlight input tokens most influential to a prediction.
• Examines natural language explanation generation, where models produce textual rationales for predictions using sequence-to-sequence or prompt-based approaches.
• Analyzes probing techniques that analyze internal representations (e.g., attention heads, hidden states) to interpret model reasoning.
• Evaluates existing automatic and human-based evaluation methods, highlighting inconsistencies and limitations in metrics like BLEU for NLE and DFFOT/SUFF for faithfulness.
• Proposes a framework for future evaluation that is reproducible, user-centric, and capable of assessing multiple interpretability dimensions across prediction types.

Experimental results

Research questions

• RQ1 How can local interpretability methods in NLP be systematically categorized and compared?
• RQ2 What are the key differences between interpretability, explainability, and faithfulness in the context of deep learning models?
• RQ3 Why is the evaluation of interpretability methods inconsistent, and what are the main limitations of current automatic and human evaluation approaches?
• RQ4 How can interpretability methods be extended to explain not only correct predictions but also model errors?
• RQ5 What are the essential components of a robust, reproducible, and user-centric evaluation framework for model interpretability?

Key findings

• Current automatic evaluation metrics for interpretability, such as BLEU for natural language explanations, often fail to capture semantic faithfulness, leading to misleading conclusions about explanation quality.
• Evaluation metrics like DFFOT and SUFF produce contradictory results on the same model and dataset, indicating a lack of consistency and reliability in automatic evaluation.
• Human evaluation remains the gold standard for assessing interpretability but is subjective, time-consuming, and not easily reproducible across studies.
• There is a significant research gap in evaluating interpretability dimensions such as stability and trustworthiness, which are currently under-explored in automated frameworks.
• The majority of interpretability research focuses only on explaining correct predictions, neglecting the critical need to explain model errors, which limits model trust and usability.
• The future of explainable AI lies in developing models that can generate faithful, comprehensible, and stable explanations for both correct and incorrect predictions to enable self-correcting, trustworthy AI systems.