[Paper Review] On Attacking Out-Domain Uncertainty Estimation in Deep Neural Networks

Huimin Zeng, Zhenrui Yue+4 more|arXiv (Cornell University)|Oct 3, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes a white-box adversarial attack that undermines out-of-domain uncertainty estimation in deep neural networks by generating imperceptible perturbations that fool state-of-the-art uncertainty estimation models into assigning high confidence to out-of-distribution (OOD) inputs. Despite strong OOD detection performance, methods like Deep Ensemble, DUQ, DUE, and SNGP are shown to be highly vulnerable, with confidence scores rising above 90% on OOD data under attack.

ABSTRACT

In many applications with real-world consequences, it is crucial to develop reliable uncertainty estimation for the predictions made by the AI decision systems. Targeting at the goal of estimating uncertainty, various deep neural network (DNN) based uncertainty estimation algorithms have been proposed. However, the robustness of the uncertainty returned by these algorithms has not been systematically explored. In this work, to raise the awareness of the research community on robust uncertainty estimation, we show that state-of-the-art uncertainty estimation algorithms could fail catastrophically under our proposed adversarial attack despite their impressive performance on uncertainty estimation. In particular, we aim at attacking the out-domain uncertainty estimation: under our attack, the uncertainty model would be fooled to make high-confident predictions for the out-domain data, which they originally would have rejected. Extensive experimental results on various benchmark image datasets show that the uncertainty estimated by state-of-the-art methods could be easily corrupted by our attack.

Motivation & Objective

• To expose the critical vulnerability of state-of-the-art uncertainty estimation methods in deep neural networks to adversarial attacks targeting out-of-distribution (OOD) inputs.
• To demonstrate that even highly accurate uncertainty estimation models can be manipulated into making overconfident predictions on OOD data through small, imperceptible perturbations.
• To raise awareness in the research community about the urgent need to improve the robustness of uncertainty estimation in safety-critical AI applications such as autonomous driving and medical diagnosis.
• To provide a prototype white-box attack framework that systematically evaluates the robustness of uncertainty estimation models under adversarial perturbations.
• To inspire future research into robust uncertainty estimation algorithms and defense mechanisms against such attacks.

Proposed method

• Proposes a white-box adversarial attack targeting the uncertainty estimation head of DNNs, specifically designed to mislead models into assigning high confidence to out-of-distribution (OOD) inputs.
• Uses a gradient-based optimization method to generate adversarial perturbations that shift OOD inputs into the high-confidence region of the model’s prediction space.
• Employs a loss function that maximizes the predicted confidence for OOD samples while minimizing the confidence for in-distribution (ID) samples, ensuring the attack remains effective and targeted.
• Applies the attack to multiple state-of-the-art uncertainty estimation methods: Deep Ensemble, DUQ (RBF-based), DUE (Gaussian Process-based), and SNGP (Spectral-Normalized GP).
• Evaluates the attack across multiple benchmark datasets (e.g., CIFAR-10, SVHN, NotMNIST) with varying perturbation magnitudes (ε) to assess scalability and robustness.
• Visualizes the shift in confidence distributions before and after attack, demonstrating a significant increase in model confidence for OOD samples.

Experimental results

Research questions

• RQ1 To what extent can state-of-the-art uncertainty estimation methods in deep neural networks be fooled by adversarial perturbations on out-of-distribution inputs?
• RQ2 How does the magnitude of perturbation (ε) affect the success rate and confidence increase of the attack across different uncertainty estimation models?
• RQ3 Can the proposed attack consistently degrade the reliability of uncertainty estimation across diverse architectures and uncertainty estimation techniques?
• RQ4 What is the impact of adversarial perturbations on key uncertainty metrics such as entropy and rejection rate in OOD detection?
• RQ5 How do adversarially trained models (e.g., adversarially trained Deep Ensemble) compare in robustness to the proposed attack compared to standard models?

Key findings

• The proposed adversarial attack successfully increases the predictive confidence of state-of-the-art uncertainty estimation models on out-of-distribution (OOD) inputs, with confidence scores rising above 90% in multiple cases.
• For the DUQ model on NotMNIST, entropy dropped from 1.02 (clean) to 0.80 and 0.72 under increasing ε (0.016 to 0.063), indicating a significant reduction in uncertainty.
• On SVHN, the adversarially trained Deep Ensemble showed increased entropy and rejection rate with larger ε, indicating inconsistent behavior, but still produced high-confidence wrong predictions.
• Visualization of confidence distributions showed that models like DUE, which were originally only ~50% confident on OOD images, became over 90% confident after the attack.
• The attack was effective across diverse uncertainty estimation methods, including Deep Ensemble, DUQ, DUE, and SNGP, demonstrating broad vulnerability.
• The results reveal a critical security gap: even models with strong OOD detection performance can be manipulated into making overconfident, potentially catastrophic predictions on OOD inputs.