[Paper Review] Multiplicative Normalizing Flows for Variational Bayesian Neural Networks

Christos Louizos, Max Welling|arXiv (Cornell University)|Mar 6, 2017

Gaussian Processes and Bayesian Inference40 references234 citations

TL;DR

Introduces multiplicative normalizing flows (MNFs) to build flexible variational posteriors for Bayesian neural networks, improving predictive accuracy and uncertainty over mean-field approaches while preserving local reparameterization tricks.

ABSTRACT

We reinterpret multiplicative noise in neural networks as auxiliary random variables that augment the approximate posterior in a variational setting for Bayesian neural networks. We show that through this interpretation it is both efficient and straightforward to improve the approximation by employing normalizing flows while still allowing for local reparametrizations and a tractable lower bound. In experiments we show that with this new approximation we can significantly improve upon classical mean field for Bayesian neural networks on both predictive accuracy as well as predictive uncertainty.

Motivation & Objective

• Motivate robust uncertainty estimation in neural networks when labeled data are scarce.
• Develop a scalable variational inference framework for Bayesian NNs with flexible posterior posteriors.
• Incorporate multiplicative noise and normalizing flows to augment the approximate posterior without losing local reparameterization advantages.
• Evaluate MNFs against dropout, deep ensembles, and prior/posterior choices on MNIST, CIFAR-10, and toy regression tasks.

Proposed method

• Model weights with a conditional Gaussian q(W|z) where W is modulated by a latent z drawn from q(z).
• Apply normalizing flows to q(z) (using masked RealNVP with IAF-style updates) to enrich the mixing distribution.
• Use an auxiliary distribution r(z|W) to bound the entropy term and obtain a tractable variational bound.
• Maintain local reparameterizations for efficient gradient-based optimization of the variational objective.
• Derive KL-divergence bounds and practical forward passes for fully connected and convolutional layers under MNFs.
• Provide forward-propagation algorithms that incorporate z-flow into layer computations (Equations 3–6).

Experimental results

Research questions

• RQ1 Does the MNF posterior better approximate the true posterior over weights than mean-field Gaussian posteriors?
• RQ2 Do MNFs improve predictive accuracy and calibrated uncertainty compared to standard Bayesian NN approaches (e.g., dropout, matrix Gaussians) and non-Bayesian baselines?
• RQ3 How do MNFs perform under adversarial attacks and out-of-distribution scenarios compared to competing uncertainty methods?
• RQ4 What is the impact of different priors (Gaussian vs. log-uniform) on MNF-based uncertainty and sparsity?
• RQ5 Can MNFs achieve comparable performance to deep ensembles while offering superior uncertainty estimates and efficiency?

Key findings

• MNFs significantly improve predictive accuracy and uncertainty over mean-field posteriors.
• MNF-based models provide more realistic predictive uncertainties on notMNIST and CIFAR-10 compared with dropout and deep ensembles in the reported setups.
• Under adversarial perturbations, MNFs show increased predictive uncertainty rather than high confidence, unlike some baseline methods.
• Using MNFs with appropriate priors (e.g., Gaussian vs. log-uniform) yields different sparsity and uncertainty profiles, highlighting prior choice importance.
• MNFs achieve competitive accuracy with Dropout while offering improved uncertainty behavior, and can approach the performance of Deep Ensembles in some settings.