[Paper Review] Interpreting Deep Learning Models in Natural Language Processing: A Review

Xiaofei Sun, Diyi Yang+7 more|arXiv (Cornell University)|Oct 20, 2021

Topic Modeling216 references25 citations

TL;DR

This survey provides a comprehensive taxonomy and empirical review of interpretation methods for neural NLP models, covering training-based, test-based, and hybrid approaches, and discusses current deficiencies and future directions.

ABSTRACT

Neural network models have achieved state-of-the-art performances in a wide range of natural language processing (NLP) tasks. However, a long-standing criticism against neural network models is the lack of interpretability, which not only reduces the reliability of neural NLP systems but also limits the scope of their applications in areas where interpretability is essential (e.g., health care applications). In response, the increasing interest in interpreting neural NLP models has spurred a diverse array of interpretation methods over recent years. In this survey, we provide a comprehensive review of various interpretation methods for neural models in NLP. We first stretch out a high-level taxonomy for interpretation methods in NLP, i.e., training-based approaches, test-based approaches, and hybrid approaches. Next, we describe sub-categories in each category in detail, e.g., influence-function based methods, KNN-based methods, attention-based models, saliency-based methods, perturbation-based methods, etc. We point out deficiencies of current methods and suggest some avenues for future research.

Motivation & Objective

• Provide a high-level taxonomy of interpretation methods for neural NLP models.
• Survey training-based, test-based, and hybrid interpretation techniques with representative methods.
• Discuss limitations, challenges, and open problems to guide future research.
• Highlight practical developments such as efficient influence-function implementations and landmark-based methods.

Proposed method

• Define a two-dimensional taxonomy: training-focused vs. test-focused, and joint vs. post-hoc interpretation.
• Detail training-based subcategories: influence functions, KNN-based interpretation, and kernel-based explanations.
• Detail test-based subcategories: saliency maps, attention-based explanations, and explanation generation.
• Describe hybrid approaches that combine training and testing perspectives.
• Present representative works and how they fit into the taxonomy.
• Compare joint versus post-hoc implementations and provide illustrative frameworks (e.g., attention as joint, post-hoc probes).

Experimental results

Research questions

• RQ1 What are the major categories and subcategories of interpretation methods for neural NLP models?
• RQ2 How do training-focused, test-focused, and hybrid methods differ in their objectives and mechanisms?
• RQ3 What are the strengths, limitations, and representative techniques within each category?
• RQ4 What are the open problems and directions for future research in neural NLP interpretability?

Key findings

• The paper provides a comprehensive taxonomy of interpretability methods for neural NLP, distinguishing training-based, test-based, and hybrid approaches.
• Influence functions, KNN-based methods, and kernel-based explanations are key training-based techniques discussed.
• Saliency maps, attention-based explanations, and explanation generation are core test-based methods reviewed.
• Efficient implementations (e.g., turn-over dropout and algorithmic optimizations) significantly accelerate influence-function based interpretation, achieving up to 80x speedups.
• Landmark-based interpretations combine Layer-wise Relevance Propagation with kernel architectures to produce explanations and templates, enabling joint and post-hoc interpretations.
• The survey highlights deficiencies of current methods and outlines open problems and future research directions to improve reliability and applicability of neural NLP interpretability.