[Paper Review] A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification

Paul F. Jaeger, Carsten T. Lüth+2 more|arXiv (Cornell University)|Nov 28, 2022

Adversarial Robustness in Machine Learning18 citations

TL;DR

This paper calls for a unified evaluation framework for confidence scoring functions (CSFs) in image classification failure detection, revealing that current isolated evaluation protocols across Misclassification Detection, OoD Detection, and other subfields lead to inconsistent, incomplete benchmarking. The authors conduct a large-scale empirical study showing that simple softmax response outperforms complex methods, exposing flaws in existing evaluation practices and advocating for a holistic, standardized benchmark to enable safer deployment of ML systems.

ABSTRACT

Reliable application of machine learning-based decision systems in the wild is one of the major challenges currently investigated by the field. A large portion of established approaches aims to detect erroneous predictions by means of assigning confidence scores. This confidence may be obtained by either quantifying the model's predictive uncertainty, learning explicit scoring functions, or assessing whether the input is in line with the training distribution. Curiously, while these approaches all state to address the same eventual goal of detecting failures of a classifier upon real-life application, they currently constitute largely separated research fields with individual evaluation protocols, which either exclude a substantial part of relevant methods or ignore large parts of relevant failure sources. In this work, we systematically reveal current pitfalls caused by these inconsistencies and derive requirements for a holistic and realistic evaluation of failure detection. To demonstrate the relevance of this unified perspective, we present a large-scale empirical study for the first time enabling benchmarking confidence scoring functions w.r.t all relevant methods and failure sources. The revelation of a simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation in the abundance of publicized research on confidence scoring. Code and trained models are at https://github.com/IML-DKFZ/fd-shifts.

Motivation & Objective

• To identify and address inconsistencies in evaluation protocols across failure detection subfields (e.g., MisD, OoD-D, SC, PUQ), which hinder fair comparison and progress.
• To demonstrate that current evaluation practices exclude relevant methods and failure sources, reducing practical relevance and reliability of reported results.
• To advocate for a unified, comprehensive evaluation protocol analogous to robustness benchmarks, enabling holistic assessment of confidence scoring functions (CSFs) across all realistic failure modes.
• To provide empirical evidence through a large-scale study showing that simple baselines like softmax response perform best, challenging the validity of current research trends.

Proposed method

• Systematically analyze and compare existing evaluation protocols across four failure detection subfields: Misclassification Detection (MisD), Out-of-Distribution Detection (OoD-D), Selective Classification (SC), and Predictive Uncertainty Quantification (PUQ).
• Propose a holistic evaluation framework that unifies these disparate protocols into a single, consistent benchmark covering all relevant failure sources and CSF types.
• Conduct a large-scale empirical study comparing 15 CSFs across 12 distribution shifts on CIFAR-10, CIFAR-100, and SVHN datasets, using standardized training and evaluation protocols.
• Implement and evaluate CSFs including softmax response, Mahalanobis score, Maximum Logit Score (MLS), and ConfidNet, using consistent data augmentation and model selection strategies.
• Use AUROC_f as the primary metric to assess failure detection performance, with additional analysis on fail-average-precision and AURC for robustness.
• Perform ablation studies on model selection strategies (e.g., last epoch vs. best epoch) and hyperparameter choices to ensure fair and reproducible comparisons.

Experimental results

Research questions

• RQ1 Why do current evaluation protocols for failure detection in image classification lead to inconsistent and incomplete benchmarking across different research subfields?
• RQ2 To what extent do existing evaluation protocols exclude relevant confidence scoring functions (CSFs) and failure sources, undermining practical relevance?
• RQ3 How does the performance of simple baselines like softmax response compare to complex, state-of-the-art CSFs across diverse distribution shifts and failure modes?
• RQ4 What are the key shortcomings in the OoD-D task formulation that deviate from the goal of detecting classification failures in real-world deployment?
• RQ5 Can a unified, holistic evaluation protocol be designed and validated to enable fair, comprehensive, and realistic benchmarking of CSFs?

Key findings

• The softmax response baseline outperforms all other confidence scoring functions across all tested datasets and distribution shifts, achieving the highest AUROC_f scores.
• The Mahalanobis score and Maximum Logit Score (MLS) show strong performance, but remain outperformed by the simple softmax baseline.
• Model selection via epoch selection (e.g., using the last epoch) consistently outperforms more complex strategies like selecting the best epoch based on validation metrics.
• The original ConfidNet configuration yields highly volatile results due to small validation sets, and simpler model selection strategies improve stability and performance.
• The evaluation of Mahalanobis scores shows high variance across runs, with standard deviations reaching up to 7.86, indicating instability in reported results.
• The OoD-D task formulation fundamentally deviates from the goal of failure detection, as it focuses on distribution shift detection rather than misclassification detection, leading to misleading performance assessments.