[Paper Review] Uncertainty in Neural Networks: Bayesian Ensembling.

Tim Pearce, Mohamed Zaki+2 more|arXiv (Cornell University)|Oct 12, 2018

Gaussian Processes and Bayesian InferenceComputer Science28 references64 citations

TL;DR

This paper proposes a modified neural network ensembling method that achieves Bayesian uncertainty quantification by regularizing model parameters toward values drawn from a prior distribution. The approach provides a scalable, practical alternative to traditional Bayesian neural networks, offering well-calibrated uncertainty estimates without sacrificing performance.

ABSTRACT

Understanding the uncertainty of a neural network's (NN) predictions is essential for many applications. The Bayesian framework provides a principled approach to this, however applying it to NNs is challenging due to the large number of parameters and data. Ensembling NNs provides a practical and scalable method for uncertainty quantification. Its drawback is that its justification is heuristic rather than Bayesian. In this work we propose one modification to the usual ensembling process, that does result in Bayesian behaviour: regularising parameters about values drawn from a prior distribution. Hence, we present an easily implementable, scalable technique for performing approximate Bayesian inference in NNs.

Motivation & Objective

• To address the challenge of uncertainty quantification in deep neural networks, which is critical for reliable decision-making in real-world applications.
• To bridge the gap between heuristic ensembling and principled Bayesian inference in neural networks.
• To develop a scalable and easily implementable method that approximates Bayesian inference in large neural networks.
• To provide a regularization-based ensembling technique that justifies ensembling within a Bayesian framework.

Proposed method

• The method modifies standard ensembling by introducing a regularization term that pulls each model's parameters toward values sampled from a prior distribution.
• Each model in the ensemble is trained with a weight decay loss that encourages parameters to stay close to independently drawn prior samples.
• The regularization is applied independently per model, ensuring that each network learns a distinct but prior-informed representation.
• The final prediction is obtained by averaging predictions across the ensemble, with uncertainty estimated from the variance of predictions.
• The approach is scalable and compatible with standard deep learning frameworks, requiring minimal architectural changes.
• The method effectively transforms heuristic ensembling into a Bayesian approximation by embedding prior knowledge into the optimization process.

Experimental results

Research questions

• RQ1 Can ensembling be formally justified within a Bayesian framework through a simple modification to the training process?
• RQ2 Does regularizing neural network parameters toward prior samples improve uncertainty calibration without degrading predictive accuracy?
• RQ3 Can this method achieve well-calibrated uncertainty estimates in large neural networks with minimal computational overhead?
• RQ4 How does the proposed method compare to standard ensembling and full Bayesian inference in terms of uncertainty quality and scalability?

Key findings

• The proposed method achieves Bayesian behavior in ensembling by explicitly regularizing parameters toward prior samples, providing a principled justification for ensemble predictions.
• The approach yields well-calibrated uncertainty estimates, with predictive variance that reliably reflects true prediction error.
• The method is scalable and practical, requiring only minor modifications to standard training pipelines.
• Empirical results show that the uncertainty estimates are more reliable than those from standard ensembling, particularly in out-of-distribution settings.