[Paper Review] Parenting: Safe Reinforcement Learning from Human Input

Christopher Frye, Ilya Feige|arXiv (Cornell University)|Feb 18, 2019

Reinforcement Learning in Robotics19 references4 citations

TL;DR

This paper introduces the Parenting algorithm, a safe reinforcement learning framework inspired by human parenting that combines human intervention, preference feedback, and direct policy learning to prevent reward hacking, unsafe exploration, and negative side effects. By introducing a novel maturation process, the agent learns to outperform its parent over time while maintaining safety and generalizing to new environments.

ABSTRACT

Autonomous agents trained via reinforcement learning present numerous safety concerns: reward hacking, negative side effects, and unsafe exploration, among others. In the context of near-future autonomous agents, operating in environments where humans understand the existing dangers, human involvement in the learning process has proved a promising approach to AI Safety. Here we demonstrate that a precise framework for learning from human input, loosely inspired by the way humans parent children, solves a broad class of safety problems in this context. We show that our Parenting algorithm solves these problems in the relevant AI Safety gridworlds of Leike et al. (2017), that an agent can learn to outperform its parent as it "matures", and that policies learnt through Parenting are generalisable to new environments.

Motivation & Objective

• Address key AI safety concerns in reinforcement learning, including unsafe exploration, reward hacking, negative side effects, and unsafe interruptibility.
• Develop a framework that enables human involvement in training autonomous agents without relying on a precisely specified reward function.
• Overcome the limitations of preference-based reward modeling, which can suffer from shift ambiguity and require task-specific hyperparameter tuning.
• Enable agents to surpass their parents' performance through a maturation process that gradually optimizes policies using feedback on longer behavioral sequences.
• Ensure generalization of safe policies to new environments, reducing the need for repeated human intervention across tasks.

Proposed method

• Use human guidance to intervene in real time and prevent dangerous actions during exploration, emulating parental redirection.
• Collect human preferences on clips of agent behavior to provide feedback on sequences of actions, analogous to post-hoc parental feedback.
• Apply supervised learning to directly train the agent’s policy using labeled data from human interventions and preferences, avoiding reward function specification.
• Implement maturation: a progressive training scheme where feedback is given on increasingly longer clips of behavior, enabling long-term policy optimization.
• Use value iteration-like dynamics to ensure that the agent’s policy improves over time, with the value function $ V_T $ converging to optimality as sequence length increases.
• Maintain symmetry with human preferences by avoiding reward model shifts, thus eliminating the need for hyperparameter tuning of reward function means.

Experimental results

Research questions

• RQ1 Can a combination of human intervention and preference feedback safely train an RL agent without relying on a pre-specified reward function?
• RQ2 How can unsafe exploration and reward hacking be mitigated in reinforcement learning when the reward function is imperfect or ambiguous?
• RQ3 Can a learning agent progressively outperform its parent policy through a structured, human-guided maturation process?
• RQ4 Does the use of direct policy learning from human feedback avoid the shift ambiguity problem inherent in preference-based reward modeling?
• RQ5 To what extent can policies learned via Parenting generalize to new environments without retraining or re-intervention?

Key findings

• The Parenting algorithm successfully mitigates all five core AI safety problems—unsafe exploration, reward hacking, negative side effects, unsafe interruptibility, and absent supervisor—within the AI Safety gridworlds of Leike et al. (2017).
• The agent’s policy, trained via direct supervised learning from human feedback, is robust to reward function shifts and avoids the need for task-specific hyperparameter tuning.
• Through the maturation process, the agent’s performance improves over time and eventually surpasses that of its parent policy, even though initial training is myopic.
• The value function $ V_T $ used in maturation converges to the optimal value function in a manner analogous to value iteration, ensuring long-term optimality.
• Policies trained via Parenting generalize to new environments, reducing the need for repeated human input across different tasks.
• The framework avoids the ambiguity in preference-based reward modeling by directly learning policies from human feedback, preserving the symmetry of human preferences.