[Paper Review] Disrupting Deep Uncertainty Estimation Without Harming Accuracy

Ido Galil, Ran El‐Yaniv|arXiv (Cornell University)|Oct 26, 2021

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces ACE (Attack on Confidence Estimation), a novel, low-magnitude perturbation attack that undermines deep neural network uncertainty estimation without degrading classification accuracy. By subtly shifting correct predictions closer to the decision boundary and incorrect ones further away, ACE causes models to overconfidently misclassify while preserving original labels, severely degrading reliability of softmax, MC-Dropout, Deep Ensembles, and SelectiveNet.

ABSTRACT

Deep neural networks (DNNs) have proven to be powerful predictors and are widely used for various tasks. Credible uncertainty estimation of their predictions, however, is crucial for their deployment in many risk-sensitive applications. In this paper we present a novel and simple attack, which unlike adversarial attacks, does not cause incorrect predictions but instead cripples the network's capacity for uncertainty estimation. The result is that after the attack, the DNN is more confident of its incorrect predictions than about its correct ones without having its accuracy reduced. We present two versions of the attack. The first scenario focuses on a black-box regime (where the attacker has no knowledge of the target network) and the second scenario attacks a white-box setting. The proposed attack is only required to be of minuscule magnitude for its perturbations to cause severe uncertainty estimation damage, with larger magnitudes resulting in completely unusable uncertainty estimations. We demonstrate successful attacks on three of the most popular uncertainty estimation methods: the vanilla softmax score, Deep Ensembles and MC-Dropout. Additionally, we show an attack on SelectiveNet, the selective classification architecture. We test the proposed attack on several contemporary architectures such as MobileNetV2 and EfficientNetB0, all trained to classify ImageNet.

Motivation & Objective

• To expose a new vulnerability in deep uncertainty estimation methods that does not affect model accuracy but severely compromises confidence calibration.
• To develop a black-box and white-box attack that manipulates uncertainty scores using imperceptible perturbations.
• To demonstrate that widely used uncertainty estimation techniques—softmax, MC-Dropout, Deep Ensembles, and SelectiveNet—are highly susceptible to such attacks.
• To show that even tiny perturbations (ε = 0.005) can drastically increase selective risk, rendering uncertainty estimation useless in practice.
• To motivate future research into robust uncertainty estimation and detection mechanisms for such attacks.

Proposed method

• ACE constructs adversarial perturbations that shift correctly predicted samples closer to the decision boundary and misclassified samples further away, without flipping predictions.
• The attack operates in both black-box (query-based) and white-box (model-architecture aware) settings, using only the model’s confidence scores as feedback.
• For softmax-based uncertainty, the attack directly manipulates the confidence score via gradient-based optimization on the model’s output logits.
• For models using MC-Dropout or Deep Ensembles, the attack targets the ensemble or dropout-based confidence estimation by perturbing input features.
• For SelectiveNet, the attack targets the selector head or the softmax score used as input to the selector, depending on the attack variant.
• Perturbations are optimized using a loss function that maximizes selective risk while preserving original predictions and minimizing perturbation magnitude.

Experimental results

Research questions

• RQ1 Can a perturbation attack be designed to degrade uncertainty estimation without altering model accuracy or prediction labels?
• RQ2 How effective is the attack across different uncertainty estimation methods, including softmax, MC-Dropout, Deep Ensembles, and SelectiveNet?
• RQ3 Does the attack remain effective in both black-box and white-box settings with minimal perturbation magnitude?
• RQ4 What is the impact of the attack on selective classification performance, particularly in terms of selective risk and coverage?
• RQ5 Are certain uncertainty estimation methods more resilient to such attacks than others?

Key findings

• ACE successfully undermines uncertainty estimation in all tested methods—softmax, MC-Dropout, Deep Ensembles, and SelectiveNet—without changing original predictions.
• With ε = 0.005, the selective risk for SelectiveNet with a VGG16 backbone increased from 0.0067 (clean) to 0.0829, indicating a 12x increase in risk for non-rejected predictions.
• For ResNet18-based SelectiveNet, the selective risk rose from 0.0012 to 0.0747 under a direct attack, demonstrating a 62x increase in risk with minimal perturbation.
• The indirect attack via the softmax score was more harmful than the direct attack on the selector head, with selective risk increasing to 0.0829 vs. 0.0772 for VGG16.
• Even at ε = 0.0005, selective risk increased significantly (e.g., from 0.0012 to 0.0059 for ResNet18), showing that damage occurs at sub-pixel perturbation levels.
• The attack causes a dramatic drop in reliability: models become more confident in incorrect predictions than in correct ones, undermining trust in uncertainty-aware systems.