[Paper Review] Bayes-TrEx: Model Transparency by Example

Serena Booth, Yilun Zhou+2 more|arXiv (Cornell University)|Feb 19, 2020

Adversarial Robustness in Machine Learning63 references4 citations

TL;DR

Bayes-TrEx is a model-agnostic method that generates distribution-conforming examples to improve neural network interpretability by visualizing class boundaries, identifying in-distribution adversarial examples, and exposing model overconfidence. It combines a classifier with a data generator to produce representative inputs at known confidence levels, demonstrating effectiveness on CLEVR, MNIST, and Fashion-MNIST.

ABSTRACT

Post-hoc explanation methods are gaining popularity as tools for interpreting, understanding, and debugging neural networks. Most post-hoc methods explain decisions in response to individual inputs. These individual inputs are typically drawn from the test set; however, the test set may be biased or may only sparsely invoke some model behaviours. To address these challenges, we introduce Bayes-TrEx, a model-agnostic method for generating distribution-conforming examples of known prediction confidence. Using a classifier prediction and a data generator, Bayes-TrEx can be used to visualize class boundaries; to find in-distribution adversarial examples; to understand novel-class extrapolation; and to expose neural network overconfidence. We demonstrate Bayes-TrEx with rendered data (CLEVR) and organic data (MNIST, Fashion-MNIST). Code: this http URL.

Motivation & Objective

• Address limitations of post-hoc explanation methods that rely on sparse or biased test set inputs.
• Enable visualization of model behavior across the full distribution of inputs, not just individual test samples.
• Expose neural network overconfidence by generating examples at known prediction confidence levels.
• Support understanding of in-distribution adversarial examples and novel-class extrapolation.
• Provide a model-agnostic framework applicable to diverse data modalities, including synthetic and real-world data.

Proposed method

• Leverages a pre-trained classifier and a differentiable data generator to synthesize inputs that conform to the data distribution.
• Uses Bayesian inference to condition the data generator on a target prediction confidence level, ensuring generated examples are representative.
• Employs a variational inference framework to optimize the generator such that the generated data achieves desired confidence scores.
• Applies the method to both synthetic (CLEVR) and real-world (MNIST, Fashion-MNIST) datasets to validate generalization.
• Visualizes decision boundaries by generating examples across confidence levels, revealing model uncertainty and overconfidence.
• Enables discovery of in-distribution adversarial examples by searching for inputs near decision boundaries with high confidence predictions.

Experimental results

Research questions

• RQ1 Can we generate representative, distribution-conforming examples that reflect known model confidence levels?
• RQ2 How can we visualize class boundaries and model behavior across the full input distribution using synthetic examples?
• RQ3 To what extent can Bayes-TrEx detect in-distribution adversarial examples that are misclassified with high confidence?
• RQ4 How does the method expose neural network overconfidence in a controlled and interpretable way?
• RQ5 Can the approach generalize across different data modalities, including synthetic and real-world image datasets?

Key findings

• Bayes-TrEx successfully generates data examples that conform to the underlying data distribution while achieving specified prediction confidence levels.
• The method enables clear visualization of decision boundaries by generating inputs across a range of confidence scores.
• It identifies in-distribution adversarial examples that are misclassified with high confidence, revealing model brittleness.
• The approach effectively exposes model overconfidence by generating examples with high predicted probabilities that are actually incorrect.
• The method generalizes across data types, demonstrating utility on both synthetic (CLEVR) and real-world (MNIST, Fashion-MNIST) datasets.
• Generated examples provide interpretable insights into model behavior, including extrapolation to novel classes and failure modes.