[Paper Review] InterpNET: Neural Introspection for Interpretable Deep Learning

Shane Barratt|arXiv (Cornell University)|Oct 26, 2017

Topic Modeling14 references17 citations

TL;DR

This paper introduces InterpNET, a novel neural network module that generates natural language explanations for deep learning classifications by leveraging internal layer activations of a pre-trained classifier. By training a language-generating RNN on these activations, InterpNET achieves a state-of-the-art METEOR score of 37.9 on the CUB bird classification and explanation dataset, significantly outperforming prior methods.

ABSTRACT

Humans are able to explain their reasoning. On the contrary, deep neural networks are not. This paper attempts to bridge this gap by introducing a new way to design interpretable neural networks for classification, inspired by physiological evidence of the human visual system's inner-workings. This paper proposes a neural network design paradigm, termed InterpNET, which can be combined with any existing classification architecture to generate natural language explanations of the classifications. The success of the module relies on the assumption that the network's computation and reasoning is represented in its internal layer activations. While in principle InterpNET could be applied to any existing classification architecture, it is evaluated via an image classification and explanation task. Experiments on a CUB bird classification and explanation dataset show qualitatively and quantitatively that the model is able to generate high-quality explanations. While the current state-of-the-art METEOR score on this dataset is 29.2, InterpNET achieves a much higher METEOR score of 37.9.

Motivation & Objective

• To bridge the interpretability gap between human reasoning and black-box deep neural networks.
• To develop a general-purpose module that generates natural language explanations for any pre-trained classification architecture.
• To investigate whether internal layer activations contain sufficient representational information for generating meaningful, human-interpretable explanations.
• To improve upon existing captioning and explanation models by grounding explanations in the network's reasoning process, not just input observations.
• To establish a framework for interpretable AI that supports both accurate classification and faithful explanation generation.

Proposed method

• InterpNET uses a pre-trained classification network to extract hierarchical internal activations from intermediate layers, which are concatenated into a reasoning representation r(x).
• The representation r(x) is fed into a language-generating RNN (specifically an LRCN2f architecture) trained in a supervised manner to produce natural language explanations E(x,y).
• The method is designed to mirror the feed-forward processing in the human visual cortex, where internal representations guide reasoning and explanation.
• The model is trained end-to-end on supervised trios (x, y, E(x,y)), where x is an image, y is the true class, and E(x,y) is a human-annotated explanation.
• Multiple variants of InterpNET are evaluated, differing in the number of hidden layers used in the classification network (0 to 3), to study the impact of representational depth on explanation quality.
• The language generator is trained using cross-entropy loss on word-level predictions, with a dictionary mapping word indices to English words.

Experimental results

Research questions

• RQ1 Can internal activations of a deep neural network be used to generate high-quality, natural language explanations for its classifications?
• RQ2 Does incorporating multi-level internal representations (beyond just final class probabilities) improve explanation quality compared to input-only or output-only baselines?
• RQ3 How does the depth of the classification network’s hidden layers affect the quality and coherence of generated explanations?
• RQ4 Can a modular, plug-in approach like InterpNET be applied to any existing classification architecture to enable interpretable reasoning?
• RQ5 To what extent do the generated explanations correlate with human judgments of semantic similarity and relevance?

Key findings

• InterpNET achieves a METEOR score of 37.9 on the CUB dataset, significantly outperforming the previous state-of-the-art model, which scored 29.2.
• InterpNET 2, which uses two hidden layers in the classification network, achieves the highest overall performance across all metrics, including a CIDEr score of 82.1.
• The model with only class probabilities (InterpNET 0) still outperforms the captioning baseline, achieving a METEOR score of 35.0, indicating that class statistics alone carry strong explanatory signal.
• Increasing the number of hidden layers beyond two (e.g., InterpNET 3) leads to a drop in both explanation quality and classification accuracy, suggesting a trade-off between model expressivity and generalization.
• InterpNET variants consistently outperform the captioning baseline and the baseline model for generating visual explanations, demonstrating the value of reasoning representations over raw input features.
• The results provide empirical support for the hypothesis that the reasoning process of a neural network is encoded in its internal activations, enabling interpretable explanation generation.