[Paper Review] It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generation

Yuqing Du, Pieter Abbeel+1 more|arXiv (Cornell University)|Feb 22, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Curriculum Self Play (CuSP), a symmetric multi-agent self-play framework that uses two off-policy students and two regret-maximizing goal generators to automatically generate diverse, progressively challenging, and continually revisited goals. CuSP outperforms prior methods in zero-shot generalization to out-of-distribution goals by balancing exploration and anti-catastrophic forgetting through minimax regret optimization.

ABSTRACT

We are interested in training general-purpose reinforcement learning agents that can solve a wide variety of goals. Training such agents efficiently requires automatic generation of a goal curriculum. This is challenging as it requires (a) exploring goals of increasing difficulty, while ensuring that the agent (b) is exposed to a diverse set of goals in a sample efficient manner and (c) does not catastrophically forget previously solved goals. We propose Curriculum Self Play (CuSP), an automated goal generation framework that seeks to satisfy these desiderata by virtue of a multi-player game with four agents. We extend the asymmetric curricula learning in PAIRED (Dennis et al., 2020) to a symmetrized game that carefully balances cooperation and competition between two off-policy student learners and two regret-maximizing teachers. CuSP additionally introduces entropic goal coverage and accounts for the non-stationary nature of the students, allowing us to automatically induce a curriculum that balances progressive exploration with anti-catastrophic exploitation. We demonstrate that our method succeeds at generating an effective curricula of goals for a range of control tasks, outperforming other methods at zero-shot test-time generalization to novel out-of-distribution goals.

Motivation & Objective

• To address the challenge of efficiently training general-purpose reinforcement learning agents that can solve a wide variety of tasks.
• To automatically generate a curriculum of goals that balances progressive difficulty and diversity while preventing catastrophic forgetting.
• To overcome limitations of prior methods—such as reliance on discriminators or sample inefficiency in asymmetric self-play—by introducing a symmetrized, multi-agent framework.
• To enable sample-efficient, scalable curriculum generation that adapts dynamically to the agent’s learning progress.
• To incorporate both cooperation (scaffolding) and competition (challenge) in goal generation through dual, asymmetric goal generators.

Proposed method

• CuSP employs a symmetric four-agent setup: two off-policy goal-conditioned students (Alice and Bob) and two regret-maximizing goal generators (G_A and G_B), each aligned with one student.
• Each goal generator is optimized using a minimax regret objective: G_A is rewarded by the difference in returns between Alice and Bob (R^A - R^B), encouraging it to propose goals that are challenging for Bob but feasible for Alice.
• This design creates a dynamic balance: G_A promotes progressive difficulty for Bob while supporting Alice’s learning, and vice versa, ensuring both diversity and feasibility.
• The framework incorporates entropic regularization to promote goal space coverage and explicitly accounts for non-stationarity in student policies to prevent forgetting.
• The method operates in a self-play loop where students learn from goals generated by teachers, and teachers adapt based on relative performance, enabling autonomous curriculum progression.
• Goal generation is constrained within a pre-specified goal space, allowing integration of human priors to ensure safety and validity.

Experimental results

Research questions

• RQ1 Can a symmetric multi-agent self-play framework effectively generate a curriculum that balances progressive difficulty and goal diversity in goal-conditioned reinforcement learning?
• RQ2 How can self-play be extended to non-zero-sum settings to simultaneously support cooperation and competition in curriculum generation?
• RQ3 To what extent can minimax regret-based goal generation mitigate catastrophic forgetting while maintaining sample efficiency?
• RQ4 How does CuSP compare to prior methods like Asymmetric Self-Play (ASP) and GAN-based goal generation in zero-shot generalization to out-of-distribution goals?
• RQ5 What emergent behaviors or skills arise from the interaction of multiple agents in the CuSP framework?

Key findings

• CuSP outperforms baseline methods in zero-shot generalization to out-of-distribution goals, demonstrating superior generalization capability across multiple control environments.
• The method successfully maintains performance on previously solved goals, indicating effective mitigation of catastrophic forgetting through continual re-exposure and dynamic curriculum adjustment.
• CuSP generates a diverse and progressively challenging curriculum, with goal space coverage enhanced by entropic regularization and symmetric teacher dynamics.
• The framework enables the emergence of task-specific skills without explicit shaping, suggesting that the self-play dynamics support complex behavior discovery.
• Empirical results show that CuSP achieves higher sample efficiency and better performance on unseen goals compared to ASP and other curriculum learning baselines.
• Ablation studies confirm that both the symmetric design and the regret-based objective are critical for performance, with ablation of either significantly degrading results.