[Paper Review] Unsupervised Temperature Scaling: An Unsupervised Post-Processing Calibration Method of Deep Networks

Azadeh Sadat Mozafari, Hugo Siqueira Gomes+2 more|arXiv (Cornell University)|May 1, 2019

Cutaneous Melanoma Detection and Management25 references3 citations

TL;DR

This paper proposes Unsupervised Temperature Scaling (UTS), a post-processing method that calibrates deep neural networks using only unlabeled test samples, eliminating the need for labeled validation data. UTS achieves state-of-the-art calibration performance comparable to supervised Temperature Scaling while preserving model accuracy and computational efficiency, with significant improvements in confidence reliability on skin lesion detection.

ABSTRACT

The great performances of deep learning are undeniable, with impressive results over a wide range of tasks. However, the output confidence of these models is usually not well-calibrated, which can be an issue for applications where confidence on the decisions is central to providing trust and reliability (e.g., autonomous driving or medical diagnosis). For models using softmax at the last layer, Temperature Scaling (TS) is a state-of-the-art calibration method, with low time and memory complexity as well as demonstrated effectiveness. TS relies on a T parameter to rescale and calibrate values of the softmax layer, whose parameter value is computed from a labelled dataset. We are proposing an Unsupervised Temperature Scaling (UTS) approach, which does not depend on labelled samples to calibrate the model, which allows, for example, the use of a part of a test samples to calibrate the pre-trained model before going into inference mode. We provide theoretical justifications for UTS and assess its effectiveness on a wide range of deep models and datasets. We also demonstrate calibration results of UTS on skin lesion detection, a problem where a well-calibrated output can play an important role for accurate decision-making.

Motivation & Objective

• To address the challenge of model miscalibration in deep learning, where softmax outputs are overconfident and poorly reflect true class probabilities.
• To eliminate the dependency on labeled data for post-training calibration, which is costly and impractical in real-world applications like medical diagnosis.
• To develop a method that maintains the low complexity and accuracy preservation of Temperature Scaling while enabling calibration using only unlabeled test samples.
• To validate the effectiveness of UTS across diverse architectures and datasets, especially in high-stakes applications such as skin lesion detection.

Proposed method

• UTS estimates the optimal temperature parameter T by minimizing Negative Log Likelihood (NLL) on a small, unlabeled subset of test data, bypassing the need for labels.
• It identifies samples near the decision boundary—where class probabilities are balanced—as reliable candidates for calibration, leveraging their symmetric posterior distribution.
• A threshold θₖ is computed as the sum of the mean and standard deviation of softmax scores for class k across misclassified samples, selecting high-confidence, ambiguous predictions.
• The method uses the empirical distribution of confidence scores from unlabeled samples to approximate the true calibration distribution, enabling T estimation without labels.
• The temperature scaling transformation S_y(x) = exp(h_y / T) / Σ_j exp(h_j / T) is applied post-inference to recalibrate confidence outputs.
• The approach is theoretically justified by showing that samples near decision boundaries have symmetric posterior likelihoods, making them suitable for unsupervised calibration.

Experimental results

Research questions

• RQ1 Can deep neural network calibration be effectively performed without labeled data, using only unlabeled test samples?
• RQ2 Does UTS achieve calibration performance comparable to supervised Temperature Scaling while maintaining low computational cost?
• RQ3 How does UTS perform on real-world, high-stakes applications such as skin lesion classification where labeling is expensive?
• RQ4 Can samples near the decision boundary be reliably used to estimate the optimal temperature parameter T in an unsupervised manner?
• RQ5 What is the impact of UTS on confidence calibration for misclassified versus correctly classified samples?

Key findings

• UTS consistently improves calibration over uncalibrated models and matches or exceeds the performance of supervised Temperature Scaling on multiple datasets and architectures.
• On CIFAR-10 with ResNet18, UTS reduces ECE from 0.043 to 0.031 and NLL from 0.427 to 0.301, outperforming TS in some cases.
• For skin lesion detection on ISIC 2018, UTS significantly reduces confidence for misclassified samples while preserving high confidence for correctly classified ones.
• The method maintains model accuracy after calibration, confirming that temperature scaling does not alter the model’s predictive ranking.
• UTS achieves better calibration than TS on 7 out of 12 model-dataset combinations in the experiments, demonstrating robustness across diverse settings.
• The use of unlabeled data for calibration is validated through theoretical and empirical analysis, showing that boundary samples provide reliable estimates of the true calibration distribution.