[Paper Review] Understanding Measures of Uncertainty for Adversarial Example Detection

Lewis Smith, Yarin Gal|arXiv (Cornell University)|Mar 22, 2018

Adversarial Robustness in Machine Learning16 references183 citations

TL;DR

The paper analyzes how different uncertainty measures (predictive entropy, mutual information, softmax variance) behave for adversarial example detection, explains why MI is effective, identifies dropout limitations, and proposes probabilistic ensembles to improve uncertainty estimates, with MNIST and Cats vs Dogs experiments.

ABSTRACT

Measuring uncertainty is a promising technique for detecting adversarial examples, crafted inputs on which the model predicts an incorrect class with high confidence. But many measures of uncertainty exist, including predictive en- tropy and mutual information, each capturing different types of uncertainty. We study these measures, and shed light on why mutual information seems to be effective at the task of adversarial example detection. We highlight failure modes for MC dropout, a widely used approach for estimating uncertainty in deep models. This leads to an improved understanding of the drawbacks of current methods, and a proposal to improve the quality of uncertainty estimates using probabilistic model ensembles. We give illustrative experiments using MNIST to demonstrate the intuition underlying the different measures of uncertainty, as well as experiments on a real world Kaggle dogs vs cats classification dataset.

Motivation & Objective

• Investigate how different uncertainty measures respond to adversarial inputs and on-manifold vs off-manifold data.
• Explain why mutual information may outperform other uncertainty metrics for detecting adversarial examples.
• Identify limitations of MC dropout in capturing full Bayesian uncertainty.
• Propose ensemble-based extensions to improve uncertainty estimation.
• Demonstrate findings on MNIST and a real-world Cats vs Dogs dataset.

Proposed method

• Review Bayesian neural networks and variational inference with dropout as an approximation.
• Define and compare uncertainty measures: predictive entropy, mutual information (MI), and MC dropout-based estimates.
• Relate softmax variance to MI through theoretical expansion to explain empirical effectiveness.
• Visualize uncertainty in latent and image spaces to identify failure modes of dropout.
• Evaluate uncertainty-based detection on MNIST interpolations and the ASIRRA Cats vs Dogs dataset using MC dropout and ensembles.

Experimental results

Research questions

• RQ1 How do predictive entropy, MI, and softmax variance behave for adversarial vs. non-adversarial inputs?
• RQ2 Can MI distinguish inputs far from the data manifold from on-manifold ambiguous inputs, unlike predictive entropy?
• RQ3 What are the failure modes of MC dropout in representing model uncertainty, and can ensembles mitigate them?
• RQ4 Do uncertainty-based defenses generalize from toy datasets to realistic datasets like Cats vs Dogs?

Key findings

• Mutual information and predictive entropy both rise for inputs far from the training data manifold, but MI more reliably distinguishes off-manifold adversarial inputs from on-manifold ambiguous inputs.
• Softmax variance is related to MI and can explain why simple variance-based uncertainty sometimes works as a proxy for MI, though it may miss nuances MI captures.
• MC dropout underestimates posterior uncertainty, revealing gaps in latent space where the model is overconfident on nonsensical samples; ensembles of dropout models alleviate some of these gaps.
• Ensembles of dropout models (multiple initializations) better approximate the posterior and reduce spuriously confident predictions, improving robustness to adversarial inputs.
• On a real-world Cats vs Dogs dataset, MI-based uncertainty achieves meaningful discrimination between adversarial and benign inputs, outperforming entropy in some settings; entropy can yield high false positives due to inherent aleatoric uncertainty.
• The work underscores that improving uncertainty estimates via principled Bayesian approaches can enhance robustness to adversarial examples even without adversarial-specific training.