[Paper Review] Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning

Tim Pearce, Nicolas Anastassacos+2 more|arXiv (Cornell University)|May 29, 2018

Adversarial Robustness in Machine Learning13 references20 citations

TL;DR

This paper proposes anchored ensembles—neural network ensembles where individual networks are regularized toward their initial weights rather than zero—enabling analytical Bayesian inference. The method converges to a Gaussian process in the infinite-width, infinite-ensemble limit and enables stable, uncertainty-driven exploration in reinforcement learning via Thompson sampling, outperforming ε-greedy in learning stability and policy quality.

ABSTRACT

The use of ensembles of neural networks (NNs) for the quantification of predictive uncertainty is widespread. However, the current justification is intuitive rather than analytical. This work proposes one minor modification to the normal ensembling methodology, which we prove allows the ensemble to perform Bayesian inference, hence converging to the corresponding Gaussian Process as both the total number of NNs, and the size of each, tend to infinity. This working paper provides early-stage results in a reinforcement learning setting, analysing the practicality of the technique for an ensemble of small, finite number. Using the uncertainty estimates produced by anchored ensembles to govern the exploration-exploitation process results in steadier, more stable learning.

Motivation & Objective

• To address the lack of analytical justification for neural network ensembling in uncertainty quantification.
• To resolve the issue of ensemble member collapse under training, where all models converge to identical predictions.
• To enable practical Bayesian inference in neural networks without full posterior computation.
• To improve exploration in reinforcement learning using uncertainty estimates from anchored ensembles.
• To provide a scalable, low-cost alternative to Bayesian neural networks and Monte Carlo methods.

Proposed method

• Regularize each neural network's parameters toward their initial values instead of zero, preserving prior information.
• Maintain ensemble diversity by anchoring each model to its initialization, preventing convergence to identical solutions.
• Use maximum a posteriori (MAP) estimation under a multivariate Gaussian prior on parameters to enable analytical Bayesian inference.
• Prove that anchored ensembles converge to the corresponding Gaussian Process as both network width and ensemble size tend to infinity.
• Apply uncertainty estimates from the anchored ensemble to guide Thompson sampling in reinforcement learning.
• Use the variance of ensemble predictions as a measure of predictive uncertainty for exploration.

Experimental results

Research questions

• RQ1 Can anchoring neural network parameters to their initial values enable analytical Bayesian inference in ensembles?
• RQ2 Does anchoring preserve model diversity and prevent ensemble collapse under training?
• RQ3 How does the performance of anchored ensembles compare to standard ensembles and other Bayesian approximation methods in reinforcement learning?
• RQ4 Can uncertainty estimates from anchored ensembles lead to more stable and effective exploration in RL?
• RQ5 How quickly do anchored ensembles converge to their corresponding Gaussian process limit?

Key findings

• Anchored ensembles converge to the corresponding Gaussian Process in the limit of infinite width and infinite ensemble size, providing a theoretical foundation for Bayesian inference.
• The method enables analytical Bayesian inference under a multivariate Gaussian prior assumption, with a formal proof provided in the supplementary material.
• In reinforcement learning, Thompson sampling using anchored ensemble uncertainty leads to a more stable, exponential decay learning curve, unlike the sigmoidal shape of ε-greedy.
• The anchored ensemble approach promotes safer exploration by maintaining high uncertainty (and thus exploration) in rare or non-critical states, while exploiting in critical states.
• The method outperforms ε-greedy in learning stability, with no sudden performance drops observed around episode 800.
• Anchored ensembles did not significantly improve performance over standard ensembles in the noiseless CartPole environment, suggesting benefits may emerge in more complex or noisy environments.