[Paper Review] Non-Parametric Calibration of Probabilistic Regression

Hao Song, Meelis Kull+1 more|arXiv (Cornell University)|Jun 20, 2018

Neural Networks and Applications2 references3 citations

TL;DR

This paper proposes non-parametric calibration methods for probabilistic regression to improve predictive density estimates by calibrating cumulative distribution functions (CDFs) derived from regression models. It introduces two empirical approaches (e-logistic and e-beta) and a Gaussian process classifier-based method (GPC) that learn smooth calibration functions, significantly enhancing model performance on predictive likelihood without assuming parametric forms for the target distribution.

ABSTRACT

The task of calibration is to retrospectively adjust the outputs from a machine learning model to provide better probability estimates on the target variable. While calibration has been investigated thoroughly in classification, it has not yet been well-established for regression tasks. This paper considers the problem of calibrating a probabilistic regression model to improve the estimated probability densities over the real-valued targets. We propose to calibrate a regression model through the cumulative probability density, which can be derived from calibrating a multi-class classifier. We provide three non-parametric approaches to solve the problem, two of which provide empirical estimates and the third providing smooth density estimates. The proposed approaches are experimentally evaluated to show their ability to improve the performance of regression models on the predictive likelihood.

Motivation & Objective

• To define and formalize calibration for probabilistic regression, extending the concept from classification to continuous target variables.
• To address the challenge of improving probability density estimates when regression models make incorrect distributional assumptions (e.g., Gaussian residuals).
• To develop non-parametric calibration techniques that do not require prior knowledge of the true distribution family of the target variable.
• To evaluate whether post-hoc calibration of predicted CDFs leads to better-calibrated and more accurate density estimates.
• To compare the performance of empirical calibration (e-logistic, e-beta) and smooth calibration (GPC) across diverse regression datasets and settings.

Proposed method

• Calibration is performed on the cumulative distribution function (CDF) of predicted densities, leveraging the link between calibrated CDFs and calibrated density functions.
• Two empirical calibration methods—e-logistic and e-beta—are proposed, adapting logistic and beta calibration from classification to regression via one-vs-rest CDF prediction.
• The GPC-based method uses a Gaussian process classifier to learn a smooth, non-parametric calibration function over predicted CDF values, enabling continuous and flexible correction.
• The calibration framework uses a set of discrete target values (8 to 64) to discretize the continuous CDF, with predictions mapped to these values for training.
• For GPC, up to 5,000 CDF values from the base model are used during training to maintain computational efficiency, though this may reduce performance on large target sets.
• The method avoids parametric assumptions on the density function, making it suitable for complex, unknown, or non-Gaussian target distributions.

Experimental results

Research questions

• RQ1 Can the concept of calibration from classification be meaningfully extended to probabilistic regression with continuous targets?
• RQ2 Does calibrating the cumulative distribution function (CDF) lead to better-calibrated and more accurate conditional density estimates?
• RQ3 How do empirical calibration methods (e-logistic, e-beta) compare to a smooth, non-parametric GPC-based approach in improving predictive likelihood?
• RQ4 What is the impact of the number of discretized target values on calibration performance?
• RQ5 Can calibration correct for mismatched distributional assumptions (e.g., Gaussian residuals) in standard regression models like OLS?

Key findings

• The proposed calibration methods consistently improve predictive likelihood compared to uncalibrated models, with the GPC-based method achieving the best performance across most datasets and settings.
• The e-beta method outperforms e-logistic in most cases, particularly due to its ability to model non-sigmoidal calibration functions, which better capture complex density shapes.
• Performance of empirical methods (e-logistic, e-beta) degrades with a high number of target values (e.g., 64), likely due to increased variance and overfitting from empirical estimation.
• The GPC method shows superior performance with larger numbers of target values, though performance drops at 64 values due to the use of only 5,000 CDF samples, indicating a trade-off between speed and accuracy.
• In settings with very few target values (e.g., 8), calibration methods underperform the uncalibrated model, as insufficient CDF information limits effective learning.
• The results confirm that non-parametric calibration of CDFs is a viable and effective alternative to retraining or re-modeling, especially when the true distribution is unknown or misspecified.