[Paper Review] Accurate Uncertainty Estimation and Decomposition in Ensemble Learning

Jeremiah Zhe Liu, John Paisley+2 more|arXiv (Cornell University)|Nov 11, 2019

Air Quality Monitoring and Forecasting34 citations

TL;DR

Proposes Bayesian Nonparametric Ensemble (BNE) to augment ensembles for flexible calibration of aleatoric, parametric, and structural uncertainties, with theoretical guarantees and applications to air pollution prediction.

ABSTRACT

Ensemble learning is a standard approach to building machine learning systems that capture complex phenomena in real-world data. An important aspect of these systems is the complete and valid quantification of model uncertainty. We introduce a Bayesian nonparametric ensemble (BNE) approach that augments an existing ensemble model to account for different sources of model uncertainty. BNE augments a model's prediction and distribution functions using Bayesian nonparametric machinery. It has a theoretical guarantee in that it robustly estimates the uncertainty patterns in the data distribution, and can decompose its overall predictive uncertainty into distinct components that are due to different sources of noise and error. We show that our method achieves accurate uncertainty estimates under complex observational noise, and illustrate its real-world utility in terms of uncertainty decomposition and model bias detection for an ensemble in predict air pollution exposures in Eastern Massachusetts, USA.

Motivation & Objective

• Motivate the need for complete uncertainty quantification in ensemble learning.
• Develop a Bayesian nonparametric augmentation to mitigate prediction and distributional misspecification in ensembles.
• Provide theoretical guarantees for consistent aleatoric uncertainty estimation and uncertainty decomposition.
• Demonstrate the method on synthetic nonlinear tasks and a real-world air-pollution prediction ensemble.
• Show how BNE enables bias detection and uncertainty interpretation across regions.

Proposed method

• Augment a base ensemble Y = sum_k f_k(x) w_k + epsilon with a flexible residual delta(x) modeled as a Gaussian Process to capture structural biases.
• Calibrate the distributional assumption by introducing a nonparametric calibration function G, modeled as a constrained Gaussian Process to map Phi_epsilon(y|x, mu) to F(y|x, mu).
• Form the full BNE as F*(y|x) = G[Phi_epsilon(y|x, mu)], with mu = sum_k f_k(x) w_k + delta(x).
• Use shift-invariant kernels (e.g., Matérn 3/2) for delta and G, enabling uncertainty to reflect distance from training data and distributional flexibility.
• Perform posterior inference via Hamiltonian Monte Carlo with kernel matrix pre-orthogonalization; discuss computational complexity and scalable schemes.
• Provide posterior-based bias-correction terms D_delta and D_G to quantify and decompose prediction biases.

Experimental results

Research questions

• RQ1 How can an ensemble be augmented to accurately quantify aleatoric, parametric, and structural uncertainties?
• RQ2 Can a nonparametric calibration function improve the ensemble's modeling of complex distributions beyond Gaussian assumptions?
• RQ3 How does the residual correction and calibration function contribute to predictive bias detection and uncertainty decomposition?
• RQ4 Do the proposed Bayesian nonparametric components ensure calibration of predictive intervals and consistency of aleatoric uncertainty estimation?
• RQ5 How does BNE perform in synthetic nonlinear settings and in real-world air pollution prediction tasks?

Key findings

• BNE provides flexible modeling of the data distribution via a calibration function G and a residual process delta, improving both prediction and uncertainty characterization.
• The framework yields a consistent estimator for the aleatoric uncertainty under broad conditions, via the CGP (constrained Gaussian process) formulation.
• BNE enables decomposition of epistemic uncertainty into parametric and structural components (delta and G), and further into structural components delta vs. G, for bias diagnosis.
• Experiments show improved calibration and predictive accuracy under complex observational noise, compared to baselines like stacking, BAEs, and nonparametric KDE/mixtures.
• An application to a three-model PM2.5 exposure ensemble in Eastern Massachusetts demonstrates practical bias detection and spatial uncertainty decomposition.