[Paper Review] Feed-forward Uncertainty Propagation in Belief and Neural Networks

Alexander Shekhovtsov, Boris Flach+1 more|arXiv (Cornell University)|Mar 28, 2018

Adversarial Robustness in Machine Learning2 references3 citations

TL;DR

This paper introduces a feed-forward uncertainty propagation method that extends neural networks to track both mean and variance through layers, enabling approximate Bayesian inference without iterative computation. The approach supports end-to-end training, improves robustness under input noise and dropout, and enhances stability and calibration in deep networks while maintaining computational efficiency.

ABSTRACT

We propose a feed-forward inference method applicable to belief and neural networks. In a belief network, the method estimates an approximate factorized posterior of all hidden units given the input. In neural networks the method propagates uncertainty of the input through all the layers. In neural networks with injected noise, the method analytically takes into account uncertainties resulting from this noise. Such feed-forward analytic propagation is differentiable in parameters and can be trained end-to-end. Compared to standard NN, which can be viewed as propagating only the means, we propagate the mean and variance. The method can be useful in all scenarios that require knowledge of the neuron statistics, e.g. when dealing with uncertain inputs, considering sigmoid activations as probabilities of Bernoulli units, training the models regularized by injected noise (dropout) or estimating activation statistics over the dataset (as needed for normalization methods). In the experiments we show the possible utility of the method in all these tasks as well as its current limitations.

Motivation & Objective

• To enable uncertainty-aware inference in deep neural networks by propagating both mean and variance through layers.
• To provide a differentiable, end-to-end trainable method that approximates Bayesian posterior distributions in belief networks and neural networks.
• To improve model robustness under input uncertainty, adversarial noise, and dropout by analytically modeling variance propagation.
• To support practical applications such as dropout regularization, normalization techniques, and probabilistic interpretation of sigmoid activations.
• To bridge the gap between probabilistic graphical models and deep learning by enabling uncertainty-aware training and inference in standard neural network architectures.

Proposed method

• Introduces variance-propagating layers that extend standard neural network layers to track both mean and variance of activations.
• Uses a first-order approximation to propagate uncertainty through non-linearities, assuming independence between inputs and applying the delta method for variance estimation.
• Applies the method to sigmoid belief networks by treating hidden units as Bernoulli variables and approximating posterior expectations via mean and variance propagation.
• Enables analytic dropout by modeling the expected output under multiplicative Bernoulli noise, avoiding Monte Carlo sampling during inference.
• Supports end-to-end training by ensuring the uncertainty propagation is differentiable with respect to network weights.
• Employs a factorized posterior approximation to estimate uncertainty in belief networks, avoiding costly iterative inference.

Experimental results

Research questions

• RQ1 Can feed-forward uncertainty propagation provide a scalable alternative to iterative Bayesian inference in deep networks?
• RQ2 How does propagating mean and variance improve model robustness under input noise or adversarial perturbations?
• RQ3 Can analytic uncertainty propagation improve generalization and performance in dropout training compared to standard dropout?
• RQ4 To what extent does uncertainty propagation enhance the reliability of sigmoid activations as probabilistic outputs?
• RQ5 How does uncertainty propagation affect normalization and initialization in deep networks?

Key findings

• The proposed method, called AP2, achieves better approximation of Monte Carlo output distributions than standard mean-only propagation, especially under input noise.
• In CIFAR-10 experiments, analytic dropout with AP2 propagation improved validation accuracy over standard dropout and baseline training, particularly at higher drop rates (0.3).
• The method significantly improves stability under input Gaussian noise and adversarial attacks, reducing overconfidence in incorrect predictions.
• Models trained with AP2 propagation show lower confidence odds (ratio of predicted to true class probability) under noise, indicating better calibration.
• Batch normalization was found to reduce stability under noise, despite lowering training loss, highlighting the trade-off between optimization and robustness.
• The method’s approximation quality degrades with depth, suggesting limitations in deep architectures unless combined with calibration techniques.