[Paper Review] Estimating Risk and Uncertainty in Deep Reinforcement Learning

William R. Clements, Bastien Van Delft+3 more|arXiv (Cornell University)|May 23, 2019

Reinforcement Learning in Robotics59 references55 citations

TL;DR

The paper presents a theoretical framework to disentangle and unbiasedly estimate epistemic and aleatoric uncertainty in return distributions for deep RL, and introduces an uncertainty-aware DQN (UA-DQN) that improves safe learning and performance.

ABSTRACT

Reinforcement learning agents are faced with two types of uncertainty. Epistemic uncertainty stems from limited data and is useful for exploration, whereas aleatoric uncertainty arises from stochastic environments and must be accounted for in risk-sensitive applications. We highlight the challenges involved in simultaneously estimating both of them, and propose a framework for disentangling and estimating these uncertainties on learned Q-values. We derive unbiased estimators of these uncertainties and introduce an uncertainty-aware DQN algorithm, which we show exhibits safe learning behavior and outperforms other DQN variants on the MinAtar testbed.

Motivation & Objective

• Motivate the need to distinguish epistemic uncertainty (data-limited) from aleatoric uncertainty (environmental randomness) in RL.
• Develop a theoretical framework to separately estimate these uncertainties on learned Q-values within distributional RL.
• Provide unbiased estimators for both uncertainties based on a Bayesian interpretation of quantile networks.
• Demonstrate practical use of uncertainty estimates in an uncertainty-aware DQN that improves safety and performance.

Proposed method

• Model the quantile forecasts of the return distribution with a neural network producing N quantiles and trained via quantile regression loss.
• Frame quantile learning as Bayesian inference with a likelihood based on an asymmetric Laplace distribution.
• Define epistemic uncertainty as the variance of quantile outputs over posterior samples and aleatoric uncertainty as the variance of the posterior-mean quantile outputs.
• Propose unbiased estimators of both uncertainties using only two posterior samples of network weights (two auxiliary networks).
• Decompose total uncertainty into the sum of epistemic and aleatoric components and validate unbiasedness (Propositions 2.1–2.3).
• Introduce UA-DQN, which uses auxiliary networks to estimate uncertainties and adjusts action selection for risk (aleatoric) and exploration (epistemic).

Experimental results

Research questions

• RQ1 Can epistemic and aleatoric uncertainties be disentangled and estimated simultaneously in distributional RL?
• RQ2 Can unbiased estimators for both types of uncertainty be derived within a practical framework?
• RQ3 Do uncertainty estimates improve risk-sensitive decision making and exploration in deep Q-learning?
• RQ4 Does an uncertainty-aware DQN (UA-DQN) outperform standard QR-DQN and other DQN variants on benchmark tasks?

Key findings

• A theoretical framework is developed to disentangle epistemic and aleatoric uncertainties for return distributions in RL.
• Unbiased estimators for both uncertainties are derived, relying on two posterior weight samples and distributional quantile outputs.
• The total uncertainty on the return distribution equals the sum of epistemic and aleatoric uncertainties in the proposed framework.
• Two auxiliary networks enable practical estimation of uncertainties with unbiased properties.
• UA-DQN uses aleatoric uncertainty to penalize risky actions and epistemic uncertainty to drive exploration, improving safety and exploration.
• Empirical results show UA-DQN outperforms QR-DQN and other DQN variants on MinAtar, with safer learning behavior in a windy gridworld risk scenario.