[Paper Review] Interpreting Deep Classifier by Visual Distillation of Dark Knowledge

Kai Xu, Dae Hoon Park+2 more|arXiv (Cornell University)|Mar 11, 2018

Explainable Artificial Intelligence (XAI)Computer Science28 references17 citations

TL;DR

This paper proposes DarkSight, a novel visualization method that interprets deep classifiers by distilling their predictive probability distributions into low-dimensional embeddings. By jointly performing model compression and dimensionality reduction, DarkSight preserves cluster structure, global relationships, outliers, and local fidelity—outperforming t-SNE in revealing classifier confidence, confusion patterns, and atypical predictions across MNIST, CIFAR-10, and CIFAR-100 datasets.

ABSTRACT

Interpreting black box classifiers, such as deep networks, allows an analyst to validate a classifier before it is deployed in a high-stakes setting. A natural idea is to visualize the deep network's representations, so as to "see what the network sees". In this paper, we demonstrate that standard dimension reduction methods in this setting can yield uninformative or even misleading visualizations. Instead, we present DarkSight, which visually summarizes the predictions of a classifier in a way inspired by notion of dark knowledge. DarkSight embeds the data points into a low-dimensional space such that it is easy to compress the deep classifier into a simpler one, essentially combining model compression and dimension reduction. We compare DarkSight against t-SNE both qualitatively and quantitatively, demonstrating that DarkSight visualizations are more informative. Our method additionally yields a new confidence measure based on dark knowledge by quantifying how unusual a given vector of predictions is.

Motivation & Objective

• To address the limitations of standard dimension reduction techniques like t-SNE in visualizing deep classifier behavior, which often mislead due to poor cluster preservation and fidelity.
• To develop a method that visualizes the full vector of predicted class probabilities (dark knowledge) to improve interpretability of black-box classifiers.
• To enable analysts to diagnose model reliability, identify misclassified or unusual inputs, and understand class similarities and decision boundaries.
• To jointly optimize model compression and dimensionality reduction so that a simple classifier can mimic the black-box model’s predictions in the low-dimensional space.

Proposed method

• DarkSight jointly trains a low-dimensional embedding and a lightweight classifier to match the full predictive probability vectors of the original deep classifier.
• The method uses a model compression objective: the lightweight classifier’s predictions on embedded inputs must match the original classifier’s outputs on raw inputs.
• Embeddings are optimized end-to-end using backpropagation, ensuring that the low-dimensional space preserves key interpretability properties like cluster preservation and local fidelity.
• The framework enables visualization of predictive confidence, class confusion, and outliers by analyzing the spatial distribution of probability vectors in the embedding space.
• It leverages dark knowledge—the full probability distribution—not just the top prediction, to inform the structure of the visualization.
• The method supports interactive exploration of high-dimensional datasets (e.g., CIFAR-100 with 100 classes) through panning and zooming in visualizations.

Experimental results

Research questions

• RQ1 Can visualizations based on dark knowledge reveal more reliable insights into deep classifier behavior than standard dimension reduction like t-SNE?
• RQ2 How well do visualizations preserve cluster structure, global relationships between classes, and outlier detection capabilities?
• RQ3 Can the method identify and highlight data points with unusual prediction vectors that may indicate model uncertainty or misclassification?
• RQ4 To what extent does the low-dimensional embedding maintain local fidelity, ensuring nearby points have similar predictive distributions?
• RQ5 Can the method serve as a diagnostic tool for identifying model weaknesses such as confusion between similar classes or atypical inputs?

Key findings

• DarkSight visualizations preserve cluster structure and monotonically decrease confidence from cluster centers, enabling reliable assessment of classifier confidence.
• Points near cluster centers (e.g., typical digits) have higher confidence, while those at cluster edges (e.g., ambiguous or atypical digits) show lower confidence, as confirmed by prediction vectors with high second-best probabilities.
• The method successfully identifies predictive outliers—data points with unusual probability distributions—such as digits with multiple high-probability predictions, which are visualized at cluster edges or in isolated regions.
• In CIFAR-100, DarkSight visualizations clearly show a one-dimensional manifold connecting semantically similar classes (e.g., truck → car → bird → dog), with smooth transitions in prediction vectors along the path.
• Misclassified points tend to cluster together in the visualization, revealing shared features (e.g., long top strokes in digits misclassified as '7'), suggesting diagnostic insights for model improvement.
• Compared to t-SNE, DarkSight avoids misleading separations and better reflects actual model behavior, particularly in preserving global structure and identifying ambiguous or rare cases.