[Paper Review] YASENN: Explaining Neural Networks via Partitioning Activation Sequences

Yaroslav Zharov, Denis Korzhenkov+2 more|arXiv (Cornell University)|Nov 7, 2018

Explainable Artificial Intelligence (XAI)33 references3 citations

TL;DR

YASENN is a model-specific method for interpreting feed-forward neural networks by partitioning activation sequences into discrete streams using layer-wise gradient boosting decision trees. It encodes inputs as sequences of tree leaf indices, enabling interpretable analysis via downstream models like logistic regression, and reveals nuanced decision patterns—such as reliance on absence of negative terms—while maintaining low computational cost and deterministic training.

ABSTRACT

We introduce a novel approach to feed-forward neural network interpretation based on partitioning the space of sequences of neuron activations. In line with this approach, we propose a model-specific interpretation method, called YASENN. Our method inherits many advantages of model-agnostic distillation, such as an ability to focus on the particular input region and to express an explanation in terms of features different from those observed by a neural network. Moreover, examination of distillation error makes the method applicable to the problems with low tolerance to interpretation mistakes. Technically, YASENN distills the network with an ensemble of layer-wise gradient boosting decision trees and encodes the sequences of neuron activations with leaf indices. The finite number of unique codes induces a partitioning of the input space. Each partition may be described in a variety of ways, including examination of an interpretable model (e.g. a logistic regression or a decision tree) trained to discriminate between objects of those partitions. Our experiments provide an intuition behind the method and demonstrate revealed artifacts in neural network decision making.

Motivation & Objective

• To address the critical need for interpretable deep learning in high-stakes domains like medicine, finance, and autonomous systems where model transparency is legally and ethically required.
• To overcome limitations of existing interpretation methods—such as instability in gradient-based approaches, lack of model-specific insight in model-agnostic methods, and restricted architecture in inherently interpretable networks.
• To develop a method that leverages internal neural network mechanics for more accurate and trustworthy interpretation, while remaining agnostic to network architecture.
• To enable deliberate, human-interpretable explanations even in low-tolerance settings for interpretation errors, such as regulatory or safety-critical applications.
• To provide a systematic way to analyze and describe decision-making patterns in neural networks through structured partitioning of activation sequences.

Proposed method

• YASENN distills a trained neural network using an ensemble of layer-wise gradient boosting decision trees, where each tree corresponds to a hidden layer and uses its neuron activations as input.
• For each input sample, the method collects the sequence of leaf indices from all trees, forming a unique code that partitions the input space into discrete streams or clusters.
• The finite number of unique codes induces a discrete partitioning of the input space, where each partition corresponds to a distinct activation pattern across layers.
• Each partition (code) is then analyzed using a human-interpretable model—such as logistic regression or a decision tree—trained to discriminate between samples in different partitions.
• The method supports extensions for modified feature spaces and low-tolerance interpretation tasks by analyzing distillation error and refining code selection.
• The approach is deterministic and efficient due to the use of gradient-boosted trees, avoiding the stochasticity common in other distillation or sampling-based methods.

Experimental results

Research questions

• RQ1 Can partitioning the space of activation sequences into discrete, interpretable streams reveal meaningful and human-understandable patterns in neural network decisions?
• RQ2 How does YASENN’s model-specific distillation approach compare to model-agnostic methods in terms of fidelity, stability, and interpretability?
• RQ3 To what extent can YASENN detect and explain subtle decision-making artifacts, such as reliance on the absence of negative terms rather than presence of positive ones?
• RQ4 Can YASENN be effectively applied to diverse data types and tasks, including tabular and sequential data, without architectural constraints?
• RQ5 How does the method perform in low-tolerance settings for interpretation errors, and can it be enhanced with error-aware refinement mechanisms?

Key findings

• YASENN successfully identified two distinct streams in a text classification task where the model predicted high probability for the positive class, yet the underlying reasoning differed: one relied on presence of positive words, the other on absence of negative words.
• The method revealed that the neural network’s decision in one stream was not based on explicit positive sentiment but on the lack of negative terms, suggesting potential fragility in decision logic.
• One-vs-all logistic regression models trained on individual streams achieved AUC scores of 0.95 and 0.89, confirming that the partitioned streams were discriminative and meaningful.
• The method demonstrated robustness to noise and adaptability to modified input manifolds due to the distillation process, which inherently regularizes the model.
• YASENN’s deterministic training and use of decision trees enabled fast, reproducible, and low-complexity interpretation without sacrificing model-specific insight.
• The approach revealed that different activation patterns across layers can lead to similar predictions, highlighting the importance of analyzing the full sequence of activations rather than final outputs alone.