[Paper Review] Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

Meelis Kull, Miquel Perelló-Nieto+4 more|arXiv (Cornell University)|Oct 28, 2019

Machine Learning and Data ClassificationComputer Science162 citations

TL;DR

This paper introduces Dirichlet calibration, a native multiclass post-hoc calibration method that learns a Dirichlet-based calibration map to produce well-calibrated multiclass probabilities for any classifier, outperforming temperature scaling in many settings.

ABSTRACT

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.

Motivation & Objective

• Clarify fundamental calibration concepts for multiclass settings and highlight limitations of existing methods.
• Introduce a natively multiclass calibration approach grounded in Dirichlet distributions.
• Provide practical, implementable parametrisations with interpretability for the calibration maps.
• Evaluate Dirichlet calibration against a range of non-neural and neural models across multiple datasets and metrics.

Proposed method

• Propose Dirichlet calibration by modeling P(p̂(X)|Y=j) as Dirichlet(α^(j)) for each class j.
• Derive a canonical calibration function μ̂_Dir(q) using Bayes’ rule, with a generative form and two equivalent parametrisations: a linear form μ̂_DirLin(q;W,b) = σ(W ln q + b) and a canonical form μ̂_Dir(q;A,c) = σ(A ln(q/1/k) + ln c).
• Prove equivalence of the three parametrisations (DirGen, DirLin, Dir).
• Offer interpretability of the canonical form via interpretation points on the probability simplex and analysis of how matrix A and vector c affect calibration and class-confusion boundaries.
• Introduce an Off-Diagonal and Intercept Regularisation (ODIR) to combat overfitting when calibrating deep nets, added to the log-loss objective as L = (1/n)∑ logloss(μ̂_DirLin(p̂(xi)); yi) + λ(1/(k(k−1))∑i≠j wij^2) + μ(1/k)∑j bj^2.
• Provide straightforward implementation in standard frameworks (e.g., a log-transform layer followed by a dense layer and softmax) and note clipping for zero probabilities.

Experimental results

Research questions

• RQ1 Can a native multiclass calibration map outperform binary/unary calibrated approaches in multiclass settings?
• RQ2 Do Dirichlet-calibration-based maps offer advantages over temperature scaling across diverse datasets and model families?
• RQ3 Is the proposed OD-regularisation essential to prevent overfitting in high-parameter calibration on deep networks?
• RQ4 How do Dirichlet calibration maps compare with matrix scaling and vector scaling when calibrated on logits vs probabilities?
• RQ5 What interpretability insights can be gained from the canonical parametrisation about class-wise biases and confusion patterns?

Key findings

• Dirichlet_L2 generally performs well across non-neural models and datasets, often ranking among the best calibrators.
• Dirichlet_ODIR (with OD-regularisation) frequently beats temperature scaling on deep nets, especially for classwise calibration (cw-ECE).
• On CIFAR-10, Dirichlet variants outperform temperature scaling in cw-ECE and log-loss in several settings; on CIFAR-100, Dir-ODIR improves cw-ECE over TempS though log-loss may rise modestly.
• Matrix scaling with ODIR can outperform Dir-ODIR on some metrics, but benefits of capturing off-diagonal class dependencies are evident in several cases (e.g., improved log-loss when modeling pairwise effects).
• ODIR regularisation mitigates overfitting and helps maintain calibration quality on high-dimensional calibration maps (e.g., with 100 classes).
• Calibration maps provide interpretability by showing how calibration parameters shift probabilities near canonical points on the simplex; interpretation points include the simplex center and facet centers.