[Paper Review] Hierarchical interpretations for neural network predictions

Chandan Singh, William J. Murdoch+1 more|arXiv (Cornell University)|Jun 14, 2018

Explainable Artificial Intelligence (XAI)44 references69 citations

TL;DR

ACD introduces agglomerative contextual decomposition to produce a hierarchical, group-based interpretation of DNN predictions, enabling phrase/patch level insights and robust visualizations across NLP and vision tasks.

ABSTRACT

Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables. However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications. To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method, agglomerative contextual decomposition (ACD). Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction. This hierarchy is optimized to identify clusters of features that the DNN learned are predictive. Using examples from Stanford Sentiment Treebank and ImageNet, we show that ACD is effective at diagnosing incorrect predictions and identifying dataset bias. Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.

Motivation & Objective

• Motivate the need for interpretable explanations of DNN predictions beyond single-feature importance.
• Develop a general method to extract interactions among groups of features for arbitrary DNN architectures.
• Create a hierarchical visualization framework to display predictive interactions at multiple granularities.
• Demonstrate utility of ACD in diagnosing incorrect predictions, detecting dataset bias, and evaluating trust/adversarial robustness.

Proposed method

• Generalize Contextual Decomposition (CD) to arbitrary DNNs by decomposing logits g(x) into beta(x) and gamma(x) across layers (Equations 1–6).
• Partition biases in convolutional layers and adapt Decomposition rules for ReLU and max-pooling to produce layer-wise CD components (Equations 5–11).
• Define agglomerative clustering using CD scores as the joining metric to build a hierarchical interpretation (Algorithm 1).
• Iteratively add top-scoring groups (within k% of the top) and generate candidate groups by extending current groups with adjacent features (text) or patches (images).
• Terminate the hierarchy according to application-specific criteria (e.g., stop when all words selected for sentiment; stop after predefined iterations for images).
• The method remains model-agnostic aside from requiring a group-level importance score function (CD) to drive clustering.

Experimental results

Research questions

• RQ1 Can hierarchical group-based explanations reveal non-linear feature interactions learned by DNNs?
• RQ2 Does agglomerative contextual decomposition (ACD) produce intuitive, trustworthy interpretations across NLP and vision models?
• RQ3 Are ACD hierarchies robust to adversarial perturbations compared to non-hierarchical interpretations?
• RQ4 Can ACD help diagnose incorrect predictions and dataset biases in real datasets like SST, MNIST, and ImageNet?

Key findings

• ACD yields intuitive visualizations that reveal meaningful phrases and image patches contributing to predictions.
• Human studies show ACD helps users identify more accurate models and rank ACD as more trustworthy than prior methods.
• ACD hierarchies demonstrate robustness to adversarial perturbations, indicating they capture fundamental input features rather than noise.
• Qualitative examples diagnose incorrect SST predictions and identify dataset biases in ImageNet (e.g., skate features aiding puck classification).
• Table 1 illustrates top-scoring phrases of varying lengths discovered by ACD on SST (examples of positive and negative phrases).