[Paper Review] Calibration of Neural Networks

Ruslan Vasilev, A. G. D’yakonov|arXiv (Cornell University)|Mar 19, 2023

Neural Networks and Applications4 citations

TL;DR

This paper presents a comprehensive survey and empirical evaluation of confidence calibration methods for neural networks, comparing post-hoc techniques like histogram binning, isotonic regression, and temperature scaling with training-time approaches using modified loss functions. The key contribution is a systematic benchmark across diverse datasets and architectures, showing that temperature scaling and improved cross-entropy losses significantly enhance calibration, with V-scaling and M-scaling-b methods achieving state-of-the-art Brier scores on CIFAR-100 and TinyImageNet.

ABSTRACT

Neural networks solving real-world problems are often required not only to make accurate predictions but also to provide a confidence level in the forecast. The calibration of a model indicates how close the estimated confidence is to the true probability. This paper presents a survey of confidence calibration problems in the context of neural networks and provides an empirical comparison of calibration methods. We analyze problem statement, calibration definitions, and different approaches to evaluation: visualizations and scalar measures that estimate whether the model is well-calibrated. We review modern calibration techniques: based on post-processing or requiring changes in training. Empirical experiments cover various datasets and models, comparing calibration methods according to different criteria.

Motivation & Objective

• Address the critical need for reliable confidence estimation in neural network predictions for real-world applications such as healthcare and autonomous systems.
• Systematically evaluate and compare existing calibration methods—both post-hoc and training-based—on modern deep learning architectures.
• Provide a unified empirical benchmark across multiple datasets (CIFAR-10, CIFAR-100, TinyImageNet, ImageNet) and models to assess calibration performance under varying conditions.
• Analyze how different loss functions (cross-entropy, focal loss, label smoothing) impact model calibration and generalization.
• Establish a clear methodology for evaluating calibration using visualizations (reliability diagrams) and scalar metrics (Brier score, ECE).

Proposed method

• Define model calibration via the formal criterion: P(y = ŷ | ŷ confidence = p) = p for all p ∈ [0,1], ensuring predicted confidence matches true accuracy.
• Use reliability diagrams (calibration curves) to visually assess calibration by binning confidence scores and plotting average confidence vs. actual accuracy across bins.
• Apply scalar metrics such as Expected Calibration Error (ECE) and Brier Score to quantitatively evaluate calibration quality, with lower values indicating better calibration.
• Implement post-hoc calibration techniques including histogram binning, isotonic regression, and temperature scaling (T-scaling), which learn a monotonic transformation of logits.
• Introduce and evaluate novel variants like V-scaling and M-scaling-b, which apply scaling to logits with additional constraints to improve calibration on multi-class problems.
• Train models with modified loss functions—cross-entropy, focal loss (FL), and label smoothing (LS)—to assess their intrinsic calibration properties without post-processing.

Experimental results

Research questions

• RQ1 How do different post-hoc calibration methods (e.g., histogram binning, isotonic regression, temperature scaling) compare in terms of calibration performance across diverse neural network architectures?
• RQ2 What is the impact of training-time loss modifications—such as focal loss and label smoothing—on the intrinsic calibration of deep neural networks?
• RQ3 How do calibration metrics like Brier Score and Expected Calibration Error (ECE) correlate with visual reliability diagrams across different datasets and model types?
• RQ4 To what extent do model architecture and dataset complexity (e.g., CIFAR-10 vs. ImageNet) influence the effectiveness of calibration techniques?
• RQ5 Which calibration method achieves the best trade-off between calibration improvement and generalization performance across multiple benchmarks?

Key findings

• Temperature scaling (T-scaling) consistently outperforms histogram binning and isotonic regression in reducing Expected Calibration Error (ECE) across all datasets and models.
• Models trained with focal loss (FL) and label smoothing (LS) exhibit significantly better intrinsic calibration than standard cross-entropy (CE), with Brier scores reduced by up to 20% on CIFAR-100.
• On CIFAR-100, the V-scaling-b method achieved the lowest Brier score of 0.596, outperforming all other post-hoc methods and even some training-time baselines.
• ResNet50 with label smoothing (LS 0.05) achieved a Brier score of 0.621 on TinyImageNet, significantly improving over the standard CE baseline (0.645), demonstrating the benefit of regularization.
• V-scaling and M-scaling-b methods showed robust performance across all datasets, with Brier scores consistently below 0.7 on TinyImageNet and below 0.32 on ImageNet for MobileNetV2.
• Reliability diagrams confirmed that models trained with focal loss and label smoothing exhibit flatter, more diagonal calibration curves, indicating better alignment between predicted confidence and actual accuracy.