[Paper Review] Bayesian Deep Learning and a Probabilistic Perspective of Generalization

Andrew Gordon Wilson, Pavel Izmailov|arXiv (Cornell University)|Feb 20, 2020

Gaussian Processes and Bayesian Inference81 references182 citations

TL;DR

The paper reframes generalization in deep learning through Bayesian marginalization, showing deep ensembles and multimodal marginalization (MultiSWAG) improve predictive accuracy and calibration by approximating the Bayesian predictive distribution across multiple basins of attraction.

ABSTRACT

The key distinguishing property of a Bayesian approach is marginalization, rather than using a single setting of weights. Bayesian marginalization can particularly improve the accuracy and calibration of modern deep neural networks, which are typically underspecified by the data, and can represent many compelling but different solutions. We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead. We also investigate the prior over functions implied by a vague distribution over neural network weights, explaining the generalization properties of such models from a probabilistic perspective. From this perspective, we explain results that have been presented as mysterious and distinct to neural network generalization, such as the ability to fit images with random labels, and show that these results can be reproduced with Gaussian processes. We also show that Bayesian model averaging alleviates double descent, resulting in monotonic performance improvements with increased flexibility. Finally, we provide a Bayesian perspective on tempering for calibrating predictive distributions.

Motivation & Objective

• Motivate a probabilistic view of generalization based on model support and inductive biases.
• Argue that marginalization over models provides better predictive distributions than single-weight optimization.
• Demonstrate that deep ensembles approximate Bayesian marginalization and propose methods to marginalize within basins of attraction.
• Show how priors over functions induced by weight priors explain generalization phenomena and mysterious results for random labels.

Proposed method

• Frame generalization as a two-dimensional concept: model support and inductive biases.
• Define Bayesian model averaging (BMA) and relate it to marginalization over weights.
• Interpret deep ensembles as an approximate BMA that captures diversity across basins of attraction.
• Introduce MultiSWAG, a multimodal posterior approximation that aggregates multiple SWAG basins into a mixture of Gaussians.
• Compare marginalization approaches to conventional Monte Carlo and variational methods, focusing on function-space diversity and predictive calibration.

Experimental results

Research questions

• RQ1 How does marginalization over neural network parameters affect predictive accuracy and calibration compared to single-point optimization?
• RQ2 Can deep ensembles be reconciled with Bayesian inference as a practical approximation to Bayesian model averaging?
• RQ3 Does multimodal marginalization (MultiSWAG) improve performance over single-basin methods and standard ensembles, especially under distribution shifts?
• RQ4 What priors over functions are induced by common weight priors, and how do they influence generalization and the ability to fit random labels?
• RQ5 How does tempering influence calibration and predictive uncertainty in Bayesian deep learning?

Key findings

• Deep ensembles approximate Bayesian marginalization and provide improved calibration and accuracy by representing diverse basins of attraction.
• Multimodal marginalization (MultiSWAG) yields significant gains over single-basin approaches and standard ensembles, especially under corruptions and distribution shifts.
• Bayesian model averaging with multimodal posteriors alleviates double descent and leads to monotonic improvements with model flexibility.
• Priors over functions induced by weight priors can have reasonable inductive biases, explaining phenomena like fitting random labels without losing generalization on clean data.
• Gaussian processes can reproduce some neural-network-like generalization phenomena, suggesting results are not unique to networks but arise from distribution over functions with large support and appropriate inductive biases.