[Paper Review] Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification

Benjamin Eysenbach, Sergey Levine+1 more|arXiv (Cornell University)|Mar 23, 2021

Reinforcement Learning in Robotics41 references4 citations

TL;DR

This paper proposes Recursive Classification for Example-based control (RCE), a reinforcement learning method that replaces manual reward function design with direct learning from success examples. Instead of learning a reward function first, RCE uses recursive classification to predict future success probability, satisfying a novel data-driven Bellman equation; it outperforms prior imitation and reward-learning methods on complex manipulation tasks, including image-based environments.

ABSTRACT

Reinforcement learning (RL) algorithms assume that users specify tasks by manually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, we derive a control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning a reward function and then optimizing this reward function using another RL algorithm. In contrast, our method directly learns a value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer hyperparameters to tune and lines of code to debug. We show that our method satisfies a new data-driven Bellman equation, where examples take the place of the typical reward function term. Experiments show that our approach outperforms prior methods that learn explicit reward functions.

Motivation & Objective

• To address the challenge of manually designing reward functions in reinforcement learning, which is labor-intensive and requires technical expertise.
• To enable users to specify tasks by providing only examples of successful outcomes, rather than defining mathematical reward functions.
• To develop a direct, end-to-end method for policy learning that avoids the bias and complexity of intermediate reward function learning.
• To establish theoretical foundations for a new Bellman equation where success examples replace reward terms.
• To empirically demonstrate superior performance on complex manipulation tasks, including those with image observations.

Proposed method

• RCE directly learns a value function that predicts the probability of future success using transitions and success examples, without learning an intermediate reward function.
• It employs a recursive classification framework that satisfies a new data-driven Bellman equation, replacing the standard reward term with success example signals.
• The method uses a classifier $ C_{\theta} $ to estimate the ratio of future success probabilities, with predictions constrained to [0, 0.5] to reflect the probability ratio form.
• The algorithm uses temporal difference learning with a target network and a loss function that combines immediate prediction and bootstrapped future estimates.
• It assumes a generative process for success examples and introduces a robust variant that minimizes squared Hellinger distance to handle ambiguous example collection processes.
• The method is end-to-end trainable and does not require fine-tuning of image encoders, using random weights for better performance in image-based tasks.

Experimental results

Research questions

• RQ1 Can reinforcement learning be made more accessible by replacing reward functions with user-provided success examples?
• RQ2 Can a direct policy learning method that bypasses reward function learning achieve better performance than two-stage approaches?
• RQ3 What theoretical foundation supports a Bellman equation where success examples replace the reward term?
• RQ4 How does the method generalize to new environments with varying goal locations and object shapes?
• RQ5 What assumptions about example collection are necessary for stable learning, and how can they be made robust?

Key findings

• RCE outperforms state-of-the-art imitation learning methods (AIRL, DAC, SQIL) and reward-learning baselines (ORIL, PURL, VICE) on complex manipulation tasks.
• On the Sawyer push environment, RCE achieved an average return of 0.2, significantly outperforming a modified C-learning variant that achieved only 0.07.
• RCE successfully learned to solve tasks such as picking up a hammer and knocking in a nail—tasks that none of the baselines could solve.
• The method generalizes to new environments with varying shapes and goal locations when using image observations, demonstrating robust success generalization.
• RCE is robust to hyperparameter choices such as learning rate, discount factor, batch size, and weight initialization, with minimal need for tuning.
• Using random image encoders outperformed fine-tuned encoders, suggesting that representation learning may be a separate challenge for future work.