[Paper Review] Decomposition of Uncertainty for Active Learning and Reliable Reinforcement Learning in Stochastic Systems.

Stefan Depeweg, José Miguel Hernández-Lobato+2 more|arXiv (Cornell University)|Oct 19, 2017

Machine Learning and Algorithms16 references22 citations

TL;DR

This paper proposes a decomposition of predictive uncertainty in Bayesian neural networks with latent variables into epistemic (model) and aleatoric (data) components. By leveraging this decomposition in active learning and reinforcement learning, the method improves uncertainty calibration and decision reliability, demonstrating enhanced performance in uncertainty-aware active learning and robust RL policies.

ABSTRACT

Bayesian neural networks (BNNs) with latent variables are probabilistic models which can automatically identify complex stochastic patterns in the data. We study in these models a decomposition of predictive uncertainty into its epistemic and aleatoric components. We show how such a decomposition arises naturally in a Bayesian active learning scenario and develop a new objective for reliable reinforcement learning (RL) with an epistemic and aleatoric risk element. Our experiments illustrate the usefulness of the resulting decomposition in active learning and reliable RL.

Motivation & Objective

• To decompose predictive uncertainty in Bayesian neural networks with latent variables into epistemic and aleatoric components.
• To enable more reliable active learning by using uncertainty decomposition to prioritize informative data samples.
• To develop a risk-aware reinforcement learning objective that explicitly accounts for both epistemic and aleatoric uncertainty.
• To improve policy reliability in stochastic environments by incorporating uncertainty decomposition into RL training.
• To empirically validate the utility of the uncertainty decomposition in active learning and reliable RL settings.

Proposed method

• Uses Bayesian neural networks with latent variables to model complex stochastic patterns in data.
• Applies variational inference to approximate posterior distributions over network weights and latent variables.
• Derives a decomposition of predictive variance into epistemic and aleatoric components using the law of total variance.
• Introduces a novel objective function for reinforcement learning that combines expected return with epistemic and aleatoric risk terms.
• Employs uncertainty estimates to guide active learning by selecting samples with high epistemic uncertainty for labeling.
• Trains and evaluates models on stochastic environments to assess uncertainty calibration and policy reliability.

Experimental results

Research questions

• RQ1 How can predictive uncertainty in Bayesian neural networks be systematically decomposed into epistemic and aleatoric components in the presence of latent variables?
• RQ2 To what extent does uncertainty decomposition improve sample efficiency in active learning scenarios?
• RQ3 Can explicit modeling of both epistemic and aleatoric uncertainty enhance policy reliability in reinforcement learning?
• RQ4 How does the proposed risk-aware RL objective compare to standard RL baselines in terms of robustness to distributional shift?
• RQ5 What is the impact of uncertainty decomposition on uncertainty calibration and decision-making in stochastic environments?

Key findings

• The decomposition of predictive uncertainty into epistemic and aleatoric components arises naturally in Bayesian active learning, enabling more informative data selection.
• The proposed uncertainty decomposition improves sample efficiency in active learning by prioritizing instances with high epistemic uncertainty.
• The risk-aware RL objective incorporating both uncertainty types leads to more robust policies in stochastic environments.
• The method demonstrates improved uncertainty calibration, reducing overconfidence in predictions under distributional shift.
• Empirical results show that models using the decomposition achieve better generalization and reliability in both active learning and RL benchmarks.
• The approach enables reliable exploration in reinforcement learning by distinguishing between uncertainty due to lack of knowledge (epistemic) and inherent data noise (aleatoric).