[Paper Review] Distribution-free calibration guarantees for histogram binning without sample splitting

Chirag Gupta, Aaditya Ramdas|arXiv (Cornell University)|May 10, 2021

Neural Networks and Applications26 references4 citations

TL;DR

This paper establishes the first distribution-free calibration guarantees for histogram binning (uniform-mass binning) without sample splitting, using a Markov property of order statistics. It proves that double-dipping the data—using the same data for binning and probability estimation—yields strong theoretical calibration bounds, and introduces validity plots as a new tool for assessing calibration performance, outperforming standard ECE metrics.

ABSTRACT

We prove calibration guarantees for the popular histogram binning (also called uniform-mass binning) method of Zadrozny and Elkan [2001]. Histogram binning has displayed strong practical performance, but theoretical guarantees have only been shown for sample split versions that avoid 'double dipping' the data. We demonstrate that the statistical cost of sample splitting is practically significant on a credit default dataset. We then prove calibration guarantees for the original method that double dips the data, using a certain Markov property of order statistics. Based on our results, we make practical recommendations for choosing the number of bins in histogram binning. In our illustrative simulations, we propose a new tool for assessing calibration -- validity plots -- which provide more information than an ECE estimate. Code for this work will be made publicly available at https://github.com/aigen/df-posthoc-calibration.

Motivation & Objective

• To close the theoretical gap in calibration guarantees for histogram binning by proving distribution-free bounds without sample splitting.
• To demonstrate that the statistical cost of sample splitting is practically significant, especially in low-data regimes.
• To introduce validity plots as a more informative alternative to ECE for evaluating calibration performance.
• To provide practical recommendations for choosing the number of bins in histogram binning based on theoretical and empirical results.

Proposed method

• Leverages the Markov property of order statistics to derive distribution-free calibration guarantees for uniform-mass binning (UMD) without sample splitting.
• Introduces validity plots, which estimate the probability that the calibration error is bounded by ε across all prediction bins, providing a conditional calibration assessment.
• Uses a randomized version of UMD to ensure absolute continuity of the predicted scores, improving theoretical marginal calibration guarantees.
• Employs a novel theoretical framework to bound the expected calibration error without requiring data splitting, relying on order statistics and concentration inequalities.
• Compares UMD with UMS (sample-split version), fixed-width binning, isotonic regression, and scaling-binning using validity plots on the UCI credit default dataset.
• Validates theoretical bounds via simulations and shows that the theoretical curve closely matches empirical performance, even when the bound is slightly underestimated due to estimation error.

Experimental results

Research questions

• RQ1 Can distribution-free calibration guarantees be established for histogram binning when the same data is used for both binning and probability estimation (i.e., without sample splitting)?
• RQ2 How does the performance of double-dipping histogram binning (UMD) compare to sample-split versions (UMS) in terms of calibration and statistical efficiency?
• RQ3 Can validity plots provide more informative calibration assessment than standard metrics like ECE?
• RQ4 What is the theoretical relationship between the number of bins and the calibration error in UMD, and how can it guide practical implementation?
• RQ5 Do the theoretical calibration bounds for UMD closely match empirical performance in real-world datasets?

Key findings

• The paper proves the first distribution-free calibration guarantees for uniform-mass binning without sample splitting, using the Markov property of order statistics.
• Validity plots reveal that UMD consistently outperforms UMS and other baselines like isotonic regression and fixed-width binning in conditional calibration.
• For n=7K, UMD outperforms scaling-binning when ε ≤ 0.05, and the theoretical bound closely matches empirical performance, with only minor underestimation due to estimation error.
• The theoretical curve for UMD crosses the empirical validity plot at n=7K, indicating that the bound is so tight that 5K test points are insufficient to fully verify it.
• The results show that the statistical cost of sample splitting is non-negligible, especially in low-data regimes, and that double-dipping can be theoretically justified.
• The proposed validity plots provide more informative and reliable calibration assessment than ECE, as they capture the full distribution of calibration errors across prediction bins.