[Paper Review] Scalable agent alignment via reward modeling: a research direction

Jan Leike, David Krueger+4 more|arXiv (Cornell University)|Nov 19, 2018

Reinforcement Learning in Robotics120 references124 citations

TL;DR

A proposal for scalable agent alignment through reward modeling, separating learning user intentions (reward model) from policy optimization, with recursive reward modeling to handle complex domains and trust-building avenues.

ABSTRACT

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents.

Motivation & Objective

• Frame the agent alignment problem in terms of learning a reward function from user feedback and optimizing it with reinforcement learning.
• Propose reward modeling as a separation of what to achieve (the objective) from how to achieve it (the policy).
• Identify challenges in scaling reward modeling to complex domains and outline potential mitigation strategies.
• Introduce recursive reward modeling to enable evaluation assistance for increasingly capable agents.
• Discuss desiderata (scalable, economical, pragmatic) and assumptions guiding the research direction.

Proposed method

• Define reward modeling as training a reward model from user feedback that provides rewards to an RL agent.
• Use RL to optimize the learned reward function, thereby separating the objective from the policy.
• Examine recursive reward modeling where higher-level agents assist in evaluating outcomes for training subsequent agents.
• Survey design specification problems (off-switches, side-effects, absent supervisor, containment breach, etc.) and argue reward modeling can address them.
• Propose a framework for trust-building via design choices, testing, interpretability, verification, and guarantees.

Experimental results

Research questions

• RQ1 Can learning a reward function from user feedback scale to complex, real-world domains while preserving alignment?
• RQ2 What challenges (feedback amount, distribution shift, reward hacking, etc.) emerge as reward modeling scales, and how can they be mitigated?
• RQ3 Does recursive reward modeling enable evaluation-assisted training for progressively capable agents without accumulating error?
• RQ4 What mechanisms (design choices, testing, interpretability, formal verification) can credibly increase trust in reward-modeled agents?
• RQ5 Under what assumptions does reward modeling provide a practical, scalable, and economical path to agent alignment?

Key findings

• Reward modeling can separate the user objective from agent behavior, potentially easing credit assignment and enabling alignment with less feedback per interaction.
• Recursive reward modeling offers a pathway to scale evaluation to complex domains by using downstream agents to assist in evaluation.
• The approach aims to meet the desiderata of being scalable, economical, and pragmatic, offering a minimal viable path toward practical alignment.
• Acknowledges that success is not guaranteed and clarifies scope, excluding questions about preference aggregation or disobedience when multiple users are involved.
• Identifies core challenges (amount of feedback, distribution shift, reward hacking, unacceptable outcomes, reward–result gap) and outlines potential mitigating approaches.