[Paper Review] Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations

Daniel S. Brown, Wonjoon Goo+1 more|arXiv (Cornell University)|Jul 9, 2019

Reinforcement Learning in Robotics63 references32 citations

TL;DR

The paper introduces D-REX, a method that automatically generates ranked demonstrations via noise-injected behavioral cloning to learn reward functions, enabling policies that surpass the demonstrator without extra supervision. It provides theory on when better-than-demonstrator imitation is possible and demonstrates strong empirical gains on MuJoCo and Atari benchmarks.

ABSTRACT

The performance of imitation learning is typically upper-bounded by the performance of the demonstrator. While recent empirical results demonstrate that ranked demonstrations allow for better-than-demonstrator performance, preferences over demonstrations may be difficult to obtain, and little is known theoretically about when such methods can be expected to successfully extrapolate beyond the performance of the demonstrator. To address these issues, we first contribute a sufficient condition for better-than-demonstrator imitation learning and provide theoretical results showing why preferences over demonstrations can better reduce reward function ambiguity when performing inverse reinforcement learning. Building on this theory, we introduce Disturbance-based Reward Extrapolation (D-REX), a ranking-based imitation learning method that injects noise into a policy learned through behavioral cloning to automatically generate ranked demonstrations. These ranked demonstrations are used to efficiently learn a reward function that can then be optimized using reinforcement learning. We empirically validate our approach on simulated robot and Atari imitation learning benchmarks and show that D-REX outperforms standard imitation learning approaches and can significantly surpass the performance of the demonstrator. D-REX is the first imitation learning approach to achieve significant extrapolation beyond the demonstrator's performance without additional side-information or supervision, such as rewards or human preferences. By generating rankings automatically, we show that preference-based inverse reinforcement learning can be applied in traditional imitation learning settings where only unlabeled demonstrations are available.

Motivation & Objective

• Provide a theoretical condition under which imitation learning can exceed demonstrator performance.
• Show that demonstration rankings reduce reward-function ambiguity in IRL.
• Develop a practical method (D-REX) that generates rankings automatically without human labels.
• Empirically validate D-REX on simulated robotics and Atari benchmarks.
• Compare D-REX against standard imitation learning and demonstrator performance.

Proposed method

• Use behavioral cloning to learn a policy from unlabeled demonstrations.
• Inject noise into the cloned policy to create trajectories with varying performance levels.
• Automatically derive trajectory rankings from noise levels (more noise → worse performance).
• Apply Trajectory-ranked Reward Extrapolation (T-REX) to learn a reward function from the automatic rankings.
• Optimize a policy with reinforcement learning using the learned reward function.

Experimental results

Research questions

• RQ1 Under what conditions can imitation learning exceed the demonstrator’s performance?
• RQ2 Can automatically generated, unlabeled rankings via noise injection provide enough signal to recover a reward function that enables extrapolation beyond the demonstrator?
• RQ3 Does D-REX outperform standard imitation learning and the demonstrator on diverse benchmarks without extra supervision?

Key findings

• D-REX often achieves better-than-demonstrator performance across MuJoCo and Atari tasks, outperforming BC and GAIL in most cases.
• Automatic noisy perturbations produce monotonic degradation in policy performance, enabling reliable ranking (epsilon-greedy noise).
• Rewards learned from automatic rankings correlate with ground-truth returns and reveal semantically meaningful features.
• D-REX yields significant extrapolation: average improvement around 39% across Atari tasks when excluding a known loophole game, and substantial gains on MuJoCo benchmarks.
• Worst-case performance of D-REX is better than the demonstrator or standard IRL-based imitation methods in their experiments.
• D-REX is the first approach to achieve extrapolation beyond the demonstrator without rewards or human preferences.