[Paper Review] Asymmetric self-play for automatic goal discovery in robotic manipulation

OpenAI OpenAI, Matthias Plappert+14 more|arXiv (Cornell University)|Jan 13, 2021

Reinforcement Learning in Robotics57 references21 citations

TL;DR

This paper proposes asymmetric self-play to enable a single goal-conditioned policy to discover and solve diverse robotic manipulation tasks without human-provided goals or curricula. By training two agents—Alice to propose challenging goals and Bob to solve them—the method generates a curriculum implicitly, enabling zero-shot generalization to unseen tasks like stacking blocks and setting a table using only sparse rewards and behavioral cloning from Alice’s trajectories.

ABSTRACT

We train a single, goal-conditioned policy that can solve many robotic manipulation tasks, including tasks with previously unseen goals and objects. We rely on asymmetric self-play for goal discovery, where two agents, Alice and Bob, play a game. Alice is asked to propose challenging goals and Bob aims to solve them. We show that this method can discover highly diverse and complex goals without any human priors. Bob can be trained with only sparse rewards, because the interaction between Alice and Bob results in a natural curriculum and Bob can learn from Alice's trajectory when relabeled as a goal-conditioned demonstration. Finally, our method scales, resulting in a single policy that can generalize to many unseen tasks such as setting a table, stacking blocks, and solving simple puzzles. Videos of a learned policy is available at https://robotics-self-play.github.io.

Motivation & Objective

• To train a single goal-conditioned policy capable of solving a wide range of previously unseen robotic manipulation tasks.
• To eliminate reliance on human-designed goals or reward shaping by automatically generating a diverse and challenging goal distribution.
• To enable efficient learning with sparse rewards by leveraging trajectories from a goal-setting agent as demonstrations.
• To scale the approach to complex, real-world-like manipulation tasks involving diverse objects and configurations.
• To achieve zero-shot generalization to holdout tasks involving unseen goals and previously unseen objects.

Proposed method

• Train two reinforcement learning agents—Alice (goal setter) and Bob (goal solver)—in a shared robotic environment with shared initial states.
• Alice proposes goals by manipulating objects in the environment, while Bob attempts to solve the same goal from the same initial state.
• Bob is trained using a combination of sparse dense rewards and behavioral cloning (BC) on Alice’s trajectories, which are relabeled as goal-conditioned demonstrations.
• The training uses a multi-goal setting per episode, where Alice proposes 5 goals and Bob solves them sequentially until failure, promoting policy generalization.
• A curriculum emerges naturally from the interaction: goals become progressively more complex as Bob improves, and suboptimal demonstrations are filtered out.
• The BC loss is clipped to stabilize training and prevent large policy updates per optimizer step.

Experimental results

Research questions

• RQ1 Can asymmetric self-play generate a diverse and challenging distribution of goals without human priors in robotic manipulation?
• RQ2 Can a single goal-conditioned policy trained via asymmetric self-play generalize zero-shot to unseen tasks involving novel goals and objects?
• RQ3 Does using Alice’s trajectories as demonstrations via behavioral cloning significantly improve sample efficiency and learning stability with sparse rewards?
• RQ4 How does multi-goal training per episode compare to single-goal training in terms of policy generalization and training stability?
• RQ5 What components of the asymmetric self-play framework (e.g., BC, demonstration filtering, curriculum) are essential for successful zero-shot generalization?

Key findings

• The asymmetric self-play framework enables a single policy to achieve zero-shot generalization to 10+ unseen manipulation tasks, including stacking blocks, setting a table, and solving simple puzzles.
• Without asymmetric self-play (no ABC), Bob fails completely on all holdout tasks, demonstrating that ABC is essential for learning.
• Clipping the behavioral cloning loss improves training stability and accelerates learning, particularly on pick-and-place and stack tasks.
• Filtering suboptimal demonstrations during behavioral cloning reduces instability and improves performance, especially on complex tasks like flip.
• Using multiple goals per episode leads to faster and more stable learning than a single-goal baseline, suggesting internal memory benefits from repeated exposure.
• The method generalizes to complex, diverse objects and configurations when scaled with increased compute, demonstrating robustness and scalability.