[Paper Review] Revisiting the Calibration of Modern Neural Networks

Matthias Minderer, Josip Djolonga+6 more|arXiv (Cornell University)|Jun 15, 2021

Adversarial Robustness in Machine Learning51 references69 citations

TL;DR

The paper re-evaluates predictive uncertainty calibration of recent image classifiers, finding that modern non-convolutional architectures (e.g., ViT, MLP-Mixer) are well-calibrated and that calibration trends depend more on architecture than model size or pretraining alone, especially under distribution shift.

ABSTRACT

Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.

Motivation & Objective

• Motivate and reassess whether state-of-the-art image classifiers remain well-calibrated given rapid architectural advances.
• Systematically relate calibration to accuracy across diverse model families and distribution conditions.
• Identify architectural factors, beyond size and pretraining data, that influence calibration properties.
• Provide a large dataset and code to enable broad calibration assessment across models and datasets.

Proposed method

• Compare a wide range of modern image-classification model families (including convolutional and non-convolutional architectures) on ImageNet-scale tasks.
• Evaluate calibration using Expected Calibration Error (ECE) with 100 equal-mass bins, including reliability diagrams and alternative metrics (NLL, Brier score).
• Apply post-hoc temperature scaling to separate intrinsic calibration from confidence bias and assess its effects across model families.
• Analyze the influence of model size and pretraining amount/dataset on calibration, controlling for accuracy.
• Assess calibration under distribution shift using ImageNet-C and other out-of-distribution benchmarks, and examine cross-dataset consistency.

Experimental results

Research questions

• RQ1 Do modern state-of-the-art image classifiers remain well-calibrated, or is calibration degraded with increasing accuracy as previously suggested?
• RQ2 How does calibration vary across model families (convolutional vs. non-convolutional) after temperature scaling?
• RQ3 To what extent do model size and pretraining data explain calibration differences between architectures, especially under distribution shift?

Key findings

• Best current models, including non-convolutional MLP-Mixer and Vision Transformers, are well calibrated and robust to distribution shift compared to past models.
• In-distribution calibration degrades slightly with increasing model size, but this is outweighed by accuracy gains.
• Under distribution shift, calibration improves with model size, reversing the in-distribution trend.
• Accuracy and calibration correlate under distribution shift, suggesting accuracy optimization can benefit calibration.
• Model size and pretraining amount alone do not fully explain calibration differences between model families; architecture is a major determinant.