[Paper Review] Understanding Neural Networks through Representation Erasure

Jiwei Li, Will Monroe+1 more|arXiv (Cornell University)|Dec 24, 2016

Topic Modeling23 references462 citations

TL;DR

Proposes a framework to interpret neural NLP models by erasing parts of representations (input dimensions, words, or hidden units) and analyzing the impact on decisions, including a reinforcement learning method to erase minimal words to flip predictions.

ABSTRACT

While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.

Motivation & Objective

• Motivate interpretability for neural NLP models and identify which representations contribute to decisions.
• Develop a general erasure-based analysis framework applicable to inputs and hidden representations.
• Show how erasure reveals linguistic feature usage, word importance, and model architecture behavior.
• Provide error-analysis tools by identifying representations whose removal improves decisions.

Proposed method

• Define an importance score I(d) measuring the impact of erasing a word-vector dimension on log-likelihood of the correct label (Eq. 1).
• Apply erasure to input word dimensions, words, or hidden units across NLP tasks from lexical to document level.
• Train four-layer networks on word embeddings (Word2Vec, GloVe) for various tagging tasks and compute dimension-wise importance.
• Extend to word-level analysis in sentiment tasks using erasure to identify influential sentiment indicators.
• Introduce a reinforcement learning approach to find a minimal subset of words whose removal changes the model’s prediction (Eq. 2–7).
• Evaluate on POS, NER, chunking, prefixes/suffixes, word-shape, sentiment, and hotel-Review aspect data to illustrate interpretability and error analysis.

Experimental results

Research questions

• RQ1 Which input dimensions, words, and hidden units contribute most to neural NLP decisions across tasks?
• RQ2 How does erasing parts of the representation affect model performance and error analysis?
• RQ3 Can we identify minimal word subsets whose removal flips model decisions using reinforcement learning?
• RQ4 How do different architectures (RNN, LSTM, Bi-LSTM) differ in susceptibility to erasures?
• RQ5 What linguistic insights about feature representations can erasure reveal (POS, NER, sentiment indicators)?

Key findings

• Erasing input vector dimensions reveals task-dependent importance patterns and shared dimensions across tasks.
• GloVe-based models rely on a small set of frequency-related dimensions, while Word2Vec shows no single dominant frequency dimension.
• Higher network layers distribute importance more broadly, while input layers concentrate information in a few dimensions.
• Erasing sentiment-indicative words often has stronger effects for Bi-LSTM and outperforms RNNs, indicating better focus on sentiment cues.
• Removing words can yield negative importance, meaning some words hinder correct decisions and can be used for error analysis.
• Reinforcement learning can identify minimal word subsets that flip predictions, providing interpretable rationales for decisions.