[Paper Review] Generating and Detecting True Ambiguity: A Forgotten Danger in DNN Supervision Testing

Michael Weiß, André García Gómez+1 more|arXiv (Cornell University)|Jul 21, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces AmbiGuess, a novel method to generate model-agnostic, human-annotated ambiguous images for testing DNN supervisors, using gradient-guided sampling in a regularized adversarial autoencoder's latent space. The key contribution is the first comprehensive empirical study showing that supervisors excel at detecting specific uncertainty types—softmax-based methods are strong on ambiguity but weak on adversarial/corrupted inputs, while OOD detectors perform poorly on true ambiguity.

ABSTRACT

Deep Neural Networks (DNNs) are becoming a crucial component of modern software systems, but they are prone to fail under conditions that are different from the ones observed during training (out-of-distribution inputs) or on inputs that are truly ambiguous, i.e., inputs that admit multiple classes with nonzero probability in their labels. Recent work proposed DNN supervisors to detect high-uncertainty inputs before their possible misclassification leads to any harm. To test and compare the capabilities of DNN supervisors, researchers proposed test generation techniques, to focus the testing effort on high-uncertainty inputs that should be recognized as anomalous by supervisors. However, existing test generators aim to produce out-of-distribution inputs. No existing model- and supervisor independent technique targets the generation of truly ambiguous test inputs, i.e., inputs that admit multiple classes according to expert human judgment. In this paper, we propose a novel way to generate ambiguous inputs to test DNN supervisors and used it to empirically compare several existing supervisor techniques. In particular, we propose AmbiGuess to generate ambiguous samples for image classification problems. AmbiGuess is based on gradient-guided sampling in the latent space of a regularized adversarial autoencoder. Moreover, we conducted what is -- to the best of our knowledge -- the most extensive comparative study of DNN supervisors, considering their capabilities to detect 4 distinct types of high-uncertainty inputs, including truly ambiguous ones. We find that the tested supervisors' capabilities are complementary: Those best suited to detect true ambiguity perform worse on invalid, out-of-distribution and adversarial inputs and vice-versa.

Motivation & Objective

• Address the lack of standardized, human-verified ambiguous test data for evaluating DNN supervisors, especially for aleatoric uncertainty.
• Identify that existing test generation techniques focus on out-of-distribution (OOD) inputs but ignore truly ambiguous inputs where multiple labels are valid.
• Provide a model- and supervisor-independent approach to generate diverse, labeled ambiguous images for image classification tasks.
• Conduct the most extensive comparative evaluation of DNN supervisors to date, assessing their ability to detect four distinct types of high-uncertainty inputs: nominal, ambiguous, adversarial, and corrupted.
• Enable fair and reliable assessment of new DNN supervisors by releasing two pre-compiled ambiguous datasets for MNIST and Fashion-MNIST.

Proposed method

• Propose AmbiGuess, a gradient-guided sampling technique in the latent space of a regularized adversarial autoencoder to generate ambiguous images.
• Use a latent space that preserves semantic structure while enabling controlled perturbations toward ambiguous regions via class-conditional gradients.
• Ensure generated samples are ambiguous to a hypothetical oracle (e.g., human expert) by validating label diversity through human judgment and statistical consistency.
• Generate two datasets—AmbiMNIST and AmbiFashionMNIST—based on MNIST and Fashion-MNIST, respectively, with human-verified ambiguous labels.
• Apply four metrics (e.g., label entropy, inter-annotator agreement, model confidence, and diversity) to validate ambiguity and data quality.
• Evaluate 16 DNN supervisors across four input types: nominal, ambiguous, adversarial, and corrupted, using standard metrics like AUC-ROC and F1-score.

Experimental results

Research questions

• RQ1 Can we generate diverse, labeled, and human-verified ambiguous images for image classification tasks in a way that is independent of the DNN and supervisor?
• RQ2 How do existing DNN supervisors perform in detecting truly ambiguous inputs compared to other uncertainty types such as adversarial, corrupted, or invalid inputs?
• RQ3 Are there complementary detection capabilities among different supervisor techniques, with some excelling at ambiguity detection but underperforming on other uncertainty types?
• RQ4 To what extent do softmax-based supervisors (e.g., MC-Dropout, Ensembles) outperform OOD detection-based supervisors in detecting true ambiguity?
• RQ5 Can the proposed AmbiGuess-generated datasets be used to fairly and reliably assess new DNN supervisors for aleatoric uncertainty detection?

Key findings

• AmbiGuess significantly outperformed the only existing ambiguous dataset (AmbiguousMNIST) on four quantitative ambiguity indicators, including label entropy and inter-annotator agreement.
• Softmax-based supervisors (e.g., MC-Dropout, Ensembles) achieved high AUC-ROC scores (e.g., >0.95) on ambiguous inputs but performed poorly on adversarial and corrupted inputs, with AUC-ROC below 0.60 in some cases.
• OOD detection-based supervisors (e.g., Surprise Adequacy, Autoencoder-based) showed strong performance on adversarial and corrupted inputs (AUC-ROC >0.90) but failed to detect in-distribution ambiguous inputs, with AUC-ROC <0.55.
• The study revealed a clear trade-off: supervisors optimized for one type of uncertainty (e.g., ambiguity) are less effective for others (e.g., OOD), indicating complementary detection capabilities.
• The results were consistent across four different DNN architectures (e.g., LeNet, VGG-like, ResNet), confirming the robustness of the findings despite architectural variation.
• The authors released two open-source ambiguous datasets (AmbiMNIST and AmbiFashionMNIST) with permissive licenses to support future research and tool evaluation.