[Paper Review] Improving Confidence Estimates for Unfamiliar Examples

Zhizhong Li, Derek Hoiem|arXiv (Cornell University)|Apr 9, 2018

Adversarial Robustness in Machine Learning50 references4 citations

TL;DR

This paper addresses overconfident predictions in deep learning models when classifying unfamiliar but valid examples by evaluating calibration techniques, ensembles, distillation, and Bayesian methods. The ensemble of temperature-scaled models achieves the best performance, significantly reducing label, calibration, and likelihood errors across both familiar and unfamiliar data distributions.

ABSTRACT

Intuitively, unfamiliarity should lead to lack of confidence. In reality, current algorithms often make highly confident yet wrong predictions when faced with relevant but unfamiliar examples. A classifier we trained to recognize gender is 12 times more likely to be wrong with a 99% confident prediction if presented with a subject from a different age group than those seen during training. In this paper, we compare and evaluate several methods to improve confidence estimates for unfamiliar and familiar samples. We propose a testing methodology of splitting unfamiliar and familiar samples by attribute (age, breed, subcategory) or sampling (similar datasets collected by different people at different times). We evaluate methods including confidence calibration, ensembles, distillation, and a Bayesian model and use several metrics to analyze label, likelihood, and calibration error. While all methods reduce over-confident errors, the ensemble of calibrated models performs best overall, and T-scaling performs best among the approaches with fastest inference. Our code is available at https://github.com/lizhitwo/ConfidenceEstimates . $\color{red}{ ext{Please see UPDATED ERRATA.}}$

Motivation & Objective

• To investigate and address the problem of deep neural networks making highly confident yet incorrect predictions on unfamiliar but semantically valid test samples.
• To develop and evaluate a systematic methodology for splitting test data into familiar and unfamiliar subsets based on attributes (e.g., age, breed) or sampling variation (e.g., different data collection times).
• To compare the effectiveness of confidence calibration, ensembles, distillation, and Bayesian uncertainty estimation in improving reliability of prediction confidence for out-of-distribution but valid inputs.
• To demonstrate that standard calibration on i.i.d. validation sets leads to overconfidence on unfamiliar data, necessitating specialized evaluation and tuning strategies.

Proposed method

• Proposes a novel evaluation framework that splits test data into familiar and unfamiliar subsets using attribute-based (e.g., age, breed) or sampling-based (e.g., different data collection periods) splits.
• Applies temperature scaling (T-scaling) for confidence calibration, optimizing the temperature parameter on a validation set to improve calibration and likelihood scores.
• Employs model ensembles of T-scaled classifiers to improve uncertainty estimation and reduce overconfidence, leveraging the variance in predictions across models.
• Introduces knowledge distillation and generative distillation (G-distillation) to transfer the performance of ensembles into a single model, aiming to preserve accuracy and calibration with lower inference cost.
• Uses Bayesian deep learning via Monte Carlo dropout (MC-Dropout) to estimate predictive uncertainty and compare it with other methods.
• Employs standard metrics including label error, expected calibration error (ECE), negative log-likelihood (NLL), and Brier score to evaluate model performance on both familiar and unfamiliar data.

Experimental results

Research questions

• RQ1 How do deep neural networks perform in terms of confidence calibration and prediction accuracy when tested on unfamiliar but valid examples not seen during training?
• RQ2 To what extent can confidence calibration (e.g., T-scaling) reduce overconfident errors on unfamiliar data compared to baseline models?
• RQ3 How do ensemble methods, distillation, and Bayesian models compare in improving confidence estimation for unfamiliar samples?
• RQ4 Does calibration tuned on i.i.d. validation data lead to overconfident predictions on unfamiliar data, and can this be mitigated by using a differently sampled validation set?
• RQ5 Can knowledge distillation effectively transfer the robustness of ensemble models to a single model while preserving low calibration and label error on unfamiliar data?

Key findings

• Unfamiliar samples exhibit significantly higher calibration error and label error than familiar samples, with models being 12 times more likely to make a wrong 99% confident prediction on unseen age groups.
• The ensemble of temperature-scaled models achieves the lowest label error, calibration error, NLL, and Brier score across all datasets and splits, outperforming all other methods.
• T-scaling alone reduces likelihood and calibration error on both familiar and unfamiliar data, and performs best among single-model approaches with minimal inference cost.
• Distillation and G-distillation do not consistently outperform T-scaling, offering no significant gain in performance despite increased training complexity.
• The Bayesian method (Kendall et al.) performs second best overall, with only slightly higher label error than the ensemble and comparable calibration improvements.
• Calibrating on an i.i.d. validation set leads to overestimation of confidence for unfamiliar samples, suggesting that a validation set with distributional shift is necessary for optimal calibration.