[Paper Review] A Primer on Bayesian Neural Networks: Review and Debates

Julyan Arbel, Konstantinos Pitas+2 more|arXiv (Cornell University)|Sep 28, 2023

Adversarial Robustness in Machine Learning250 references14 citations

TL;DR

This paper provides an accessible, comprehensive primer on Bayesian neural networks, covering fundamentals, priors, approximate inference, and current debates.

ABSTRACT

Neural networks have achieved remarkable performance across various problem domains, but their widespread applicability is hindered by inherent limitations such as overconfidence in predictions, lack of interpretability, and vulnerability to adversarial attacks. To address these challenges, Bayesian neural networks (BNNs) have emerged as a compelling extension of conventional neural networks, integrating uncertainty estimation into their predictive capabilities. This comprehensive primer presents a systematic introduction to the fundamental concepts of neural networks and Bayesian inference, elucidating their synergistic integration for the development of BNNs. The target audience comprises statisticians with a potential background in Bayesian methods but lacking deep learning expertise, as well as machine learners proficient in deep neural networks but with limited exposure to Bayesian statistics. We provide an overview of commonly employed priors, examining their impact on model behavior and performance. Additionally, we delve into the practical considerations associated with training and inference in BNNs. Furthermore, we explore advanced topics within the realm of BNN research, acknowledging the existence of ongoing debates and controversies. By offering insights into cutting-edge developments, this primer not only equips researchers and practitioners with a solid foundation in BNNs, but also illuminates the potential applications of this dynamic field. As a valuable resource, it fosters an understanding of BNNs and their promising prospects, facilitating further advancements in the pursuit of knowledge and innovation.

Motivation & Objective

• Introduce Bayesian neural networks and motivate uncertainty estimation in deep learning.
• Review core concepts of priors, likelihoods, and posterior inference in BNNs.
• Discuss practical training and inference methods, including variational inference, Laplace approximation, and sampling.
• Examine connections between Bayesian and frequentist perspectives and issues in model evaluation.
• Highlight open debates and future directions in priors, calibration, and generalization of BNNs.

Proposed method

• Define neural network foundations and Bayesian paradigm for uncertainty quantification.
• Survey weight and unit priors, regularization, and function-space views in BNNs.
• Cover approximate inference techniques: variational inference, Laplace approximation, and sampling methods.
• Discuss model selection, training regimes, and calibration through Bayesian lenses.
• Link Bayesian approaches to frequentist concepts and ensembles as uncertainty certificates.

Figure 2: Example of using the linear regression (left) and $k$ -nearest neighbours regression (right) models on simulated data points.

Experimental results

Research questions

• RQ1 What are the foundational concepts and priors used in Bayesian neural networks and how do they influence model behavior?
• RQ2 How can Bayesian inference be practically performed in neural networks, including VI, Laplace, and sampling?
• RQ3 What is the relationship between Bayesian and frequentist perspectives in BNNs and how does this affect evaluation and generalization?
• RQ4 What are the major open debates and challenges in priors, training, and uncertainty quantification for BNNs?

Key findings

• BNNs integrate uncertainty estimation into neural networks by combining priors with Bayesian inference.
• Multiple approximate inference methods are discussed, including variational inference, Laplace approximation, and sampling techniques.
• The primer connects Bayesian methods to frequentist validation, posterior concentration, and model selection concepts.
• It highlights the role of priors in shaping model behavior and the ongoing debates around their choice and impact.
• It covers generalization and overfitting in deep learning and discusses concepts like double-descent and calibration as essential concerns in BNNs.