[Paper Review] Bayes-TrEx: a Bayesian Sampling Approach to Model Transparency by Example

Serena Booth, Yilun Zhou+2 more|arXiv (Cornell University)|Feb 19, 2020

Adversarial Robustness in Machine Learning4 citations

TL;DR

Bayes-TrEx proposes a Bayesian sampling framework that generates in-distribution examples with specific prediction confidences to enhance model transparency beyond test-set analysis. By leveraging MCMC on a hierarchical Bayesian model, it uncovers high-confidence misclassifications, ambiguous cases, and extrapolation behaviors—demonstrating superior discovery of model failure modes compared to test-set inspection.

ABSTRACT

Post-hoc explanation methods are gaining popularity for interpreting, understanding, and debugging neural networks. Most analyses using such methods explain decisions in response to inputs drawn from the test set. However, the test set may have few examples that trigger some model behaviors, such as high-confidence failures or ambiguous classifications. To address these challenges, we introduce a flexible model inspection framework: Bayes-TrEx. Given a data distribution, Bayes-TrEx finds in-distribution examples with a specified prediction confidence. We demonstrate several use cases of Bayes-TrEx, including revealing highly confident (mis)classifications, visualizing class boundaries via ambiguous examples, understanding novel-class extrapolation behavior, and exposing neural network overconfidence. We use Bayes-TrEx to study classifiers trained on CLEVR, MNIST, and Fashion-MNIST, and we show that this framework enables more flexible holistic model analysis than just inspecting the test set. Code is available at https://github.com/serenabooth/Bayes-TrEx.

Motivation & Objective

• To address the limitation of post-hoc explanation methods that rely solely on test-set inputs, which may lack examples of rare or high-confidence failure modes.
• To enable holistic model analysis by generating diverse, in-distribution examples that trigger specific prediction confidences.
• To improve model transparency by example, particularly for behaviors like overconfidence, ambiguous classifications, and novel-class extrapolation.
• To provide a flexible framework that works with learned data distributions (e.g., VAEs, GANs) or manually defined ones, applicable across vision benchmarks.
• To support debugging and interpretation by generating examples that can be further analyzed with saliency maps and other local explanation techniques.

Proposed method

• Bayes-TrEx formulates the problem of finding in-distribution examples with target prediction confidences as posterior inference in a hierarchical Bayesian model.
• It uses Markov Chain Monte Carlo (MCMC) sampling to explore the posterior distribution over latent representations, ensuring generated examples are both plausible and match the desired confidence level.
• The method relaxes the exact level set constraint by widening the confidence interval, making posterior sampling computationally feasible.
• It integrates with data distributions learned via VAEs or GANs for image data, or uses manually defined distributions for structured data like CLEVR.
• The framework supports multiple use cases by specifying different confidence targets (e.g., balanced, high-confidence, or ambiguous predictions).
• Generated examples are further analyzed using existing local explanation methods such as SmoothGrad to interpret model attention and decision rationale.

Experimental results

Research questions

• RQ1 Can a Bayesian sampling approach generate in-distribution examples that reveal high-confidence misclassifications missed in standard test sets?
• RQ2 How effective is Bayes-TrEx in uncovering ambiguous examples that visualize class boundaries compared to test-set inspection?
• RQ3 To what extent can Bayes-TrEx expose model overconfidence and novel-class extrapolation behaviors not evident from confusion matrices?
• RQ4 How do the generated examples compare to test-set examples in terms of authenticity and representativeness for model debugging?
• RQ5 Can the framework be generalized across different data distributions and model architectures, including those trained on CLEVR, MNIST, and Fashion-MNIST?

Key findings

• Bayes-TrEx discovered 51 high-confidence misclassifications in Fashion-MNIST, far exceeding the 93 total test-set examples, with most test-set failures due to data mislabeling rather than model errors.
• In CLEVR, Bayes-TrEx identified a scene with only cylinders and cubes that the model classified as containing a sphere with 97.1% confidence, revealing a critical failure mode.
• The framework consistently found more high-confidence misclassifications and ambiguous examples than test-set analysis, which often lacked such cases due to data sparsity.
• For novel-class extrapolation, Bayes-TrEx revealed that models mistakenly classified bags as ankle boots—a behavior absent in the test set confusion matrix, indicating hidden failure modes.
• Saliency maps applied to generated examples confirmed that model attention was misdirected to small, visually similar objects (e.g., a red cylinder mistaken for a sphere), validating the interpretability of the findings.
• The method demonstrated that test-set-based evaluations can miss critical failure patterns, while Bayes-TrEx enables more comprehensive and trustworthy model transparency through systematic, distribution-aware example generation.