[Paper Review] Estimating Uncertainty Intervals from Collaborating Networks

Tianhui Zhou, Yitong Li+2 more|arXiv (Cornell University)|Feb 12, 2020

Statistical Methods and Inference4 citations

TL;DR

This paper proposes Collaborating Networks (CN), a novel deep learning framework that estimates uncertainty intervals in regression by training two neural networks: one to approximate the conditional cumulative distribution function (CDF) and another to approximate its inverse. The method achieves both well-calibrated and sharp predictive intervals, outperforming existing approaches on synthetic and real-world datasets, including A1c forecasting in diabetic patients.

ABSTRACT

Effective decision making requires understanding the uncertainty inherent in a prediction. In regression, this uncertainty can be estimated by a variety of methods; however, many of these methods are laborious to tune, generate overconfident uncertainty intervals, or lack sharpness (give imprecise intervals). We address these challenges by proposing a novel method to capture predictive distributions in regression by defining two neural networks with two distinct loss functions. Specifically, one network approximates the cumulative distribution function, and the second network approximates its inverse. We refer to this method as Collaborating Networks (CN). Theoretical analysis demonstrates that a fixed point of the optimization is at the idealized solution, and that the method is asymptotically consistent to the ground truth distribution. Empirically, learning is straightforward and robust. We benchmark CN against several common approaches on two synthetic and six real-world datasets, including forecasting A1c values in diabetic patients from electronic health records, where uncertainty is critical. In the synthetic data, the proposed approach essentially matches ground truth. In the real-world datasets, CN improves results on many performance metrics, including log-likelihood estimates, mean absolute errors, coverage estimates, and prediction interval widths.

Motivation & Objective

• To address the challenge of obtaining well-calibrated and sharp uncertainty intervals in regression tasks using deep learning.
• To develop a method that avoids overconfidence and maintains consistency with the true underlying conditional distribution.
• To create a robust, trainable framework that simultaneously optimizes for both calibration and precision in predictive intervals.
• To demonstrate the method's effectiveness across diverse datasets, including complex real-world applications like A1c forecasting from electronic health records.

Proposed method

• The framework uses two neural networks: one estimates the conditional CDF, and the other estimates the inverse CDF, forming a collaborative pair.
• The CDF network is trained using a loss based on the empirical distribution function, while the inverse CDF network is trained using a separate, complementary loss function.
• The networks are optimized jointly with distinct objectives, similar to GANs, ensuring mutual consistency and convergence to the true distribution.
• Theoretical analysis shows that the fixed point of optimization corresponds to the true conditional CDF and inverse CDF, with asymptotic consistency under mild regularity conditions.
• The method supports both one-dimensional and multivariate output distributions, including complex transformations like polar-to-Cartesian coordinate systems.
• The approach enables sampling from the estimated distribution via inverse CDF inversion and supports flexible interval types (e.g., one-sided, two-sided) and coverage levels.

Experimental results

Research questions

• RQ1 Can a deep learning framework jointly learn the conditional CDF and its inverse to produce well-calibrated and sharp uncertainty intervals?
• RQ2 Does the Collaborating Networks method achieve asymptotic consistency with the true underlying distribution?
• RQ3 How does the method compare to existing uncertainty estimation techniques in terms of calibration, sharpness, and log-likelihood on real-world and synthetic data?
• RQ4 Can the framework handle complex, nonlinear transformations of output distributions, such as those arising from circular or multivariate distributions?

Key findings

• On synthetic data, the CN method closely matches the ground truth CDF, demonstrating accurate recovery of the true conditional distribution.
• In real-world datasets, including A1c forecasting from electronic health records, CN achieved superior performance in log-likelihood, mean absolute error, and prediction interval width.
• The method showed improved coverage estimates, indicating better calibration compared to baseline models, especially in high-stakes medical prediction tasks.
• The framework maintained robust and stable training across diverse data distributions, including bivariate Gaussians and von Mises-based circular distributions.
• Empirical results confirmed that the method produces sharper intervals without sacrificing calibration, resolving a common trade-off in uncertainty estimation.
• Theoretical analysis confirmed that the true CDF and inverse CDF form a fixed point of the optimization, supporting the method's consistency and convergence.