[Paper Review] Pessimism About Unknown Unknowns Inspires Conservatism

Michael K. Cohen, Marcus Hütter|arXiv (Cornell University)|Jun 15, 2020

Advanced Bandit Algorithms Research14 references4 citations

TL;DR

This paper proposes a Bayesian reinforcement learning agent that uses pessimism over unknown world-models to avoid novel failure modes. By deferring to a mentor when uncertain and optimizing for worst-case expected reward across a set of models, the agent ensures it rarely causes unprecedented events—achieving asymptotic performance at least as good as the mentor while deferring less over time.

ABSTRACT

If we could define the set of all bad outcomes, we could hard-code an agent which avoids them; however, in sufficiently complex environments, this is infeasible. We do not know of any general-purpose approaches in the literature to avoiding novel failure modes. Motivated by this, we define an idealized Bayesian reinforcement learner which follows a policy that maximizes the worst-case expected reward over a set of world-models. We call this agent pessimistic, since it optimizes assuming the worst case. A scalar parameter tunes the agent's pessimism by changing the size of the set of world-models taken into account. Our first main contribution is: given an assumption about the agent's model class, a sufficiently pessimistic agent does not cause "unprecedented events" with probability $1-δ$, whether or not designers know how to precisely specify those precedents they are concerned with. Since pessimism discourages exploration, at each timestep, the agent may defer to a mentor, who may be a human or some known-safe policy we would like to improve. Our other main contribution is that the agent's policy's value approaches at least that of the mentor, while the probability of deferring to the mentor goes to 0. In high-stakes environments, we might like advanced artificial agents to pursue goals cautiously, which is a non-trivial problem even if the agent were allowed arbitrary computing power; we present a formal solution.

Motivation & Objective

• To address the challenge of avoiding unknown unknown failure modes in complex, non-Markovian environments where formal specification of all risks is infeasible.
• To design a reinforcement learning agent that remains conservative in the face of unprecedented events, even when designers cannot formally specify all potential failure modes.
• To ensure the agent's long-term performance approaches or exceeds that of a mentor while minimizing reliance on mentorship over time.
• To formalize a safety guarantee that the agent will not cause any event never before observed with high probability, unless the mentor initiates it.

Proposed method

• The agent maintains a Bayesian posterior over a countable set of world-models and mentor-models, updating recursively based on interaction history.
• At each timestep, the agent computes a pessimistic value as the minimum expected return across the top β fraction of world-models in its posterior.
• The agent defers to a mentor if the mentor’s expected value in a sampled world-model exceeds the pessimistic value plus noise, or if the pessimistic value is zero.
• The agent uses Thompson-sampling-inspired query probability to balance exploration and mentorship, favoring deference when uncertainty is high.
• The agent’s policy is defined as the one maximizing the pessimistic value among the top world-models, ensuring worst-case robustness.
• The model class is assumed to be rich enough (e.g., semicomputable stochastic models) to include plausible world- and mentor-models, enabling broad applicability.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent be designed to avoid unprecedented failure modes even when those failures are not formally specifiable?
• RQ2 Can such an agent achieve performance at least as good as a mentor while deferring to the mentor less over time?
• RQ3 Is it possible to formally guarantee that the agent will not cause a novel event that has never occurred before, with high probability?
• RQ4 How can pessimism be formalized in a Bayesian RL framework to ensure safety without sacrificing long-term performance?

Key findings

• The pessimistic agent’s policy value asymptotically approaches at least that of the mentor, ensuring long-term performance is not worse than the mentor’s.
• The probability of deferring to the mentor approaches zero as time increases, indicating the agent becomes increasingly autonomous.
• For any event E in a given complexity class C, the agent can be made to avoid causing E for the first time with arbitrarily high probability, provided the event has not occurred before.
• The agent’s safety guarantee holds even when designers cannot formally specify the failure modes they wish to avoid, due to the pessimism-based worst-case analysis.
• The agent’s behavior is robust to wireheading-type world-models: pessimism suppresses the incentive to hijack reward signals, as the worst-case value of such actions remains low.
• The theoretical framework is formal and general, relying only on countable model classes and Bayesian updating, making it applicable to complex, non-Markovian environments.