[Paper Review] TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL

Clément Romac, Rémy Portelas+2 more|arXiv (Cornell University)|Mar 17, 2021

Reinforcement Learning in Robotics60 references4 citations

TL;DR

This paper introduces TeachMyAgent, a benchmark for Automatic Curriculum Learning (ACL) in Deep Reinforcement Learning, featuring procedural environments like a Box2D bipedal walker and a complex Parkour environment. It evaluates multiple ACL algorithms, showing ALP-GMM and Covar-GMM significantly outperform random task selection, while the Parkour environment remains a challenging open problem for current methods.

ABSTRACT

Training autonomous agents able to generalize to multiple tasks is a key target of Deep Reinforcement Learning (DRL) research. In parallel to improving DRL algorithms themselves, Automatic Curriculum Learning (ACL) study how teacher algorithms can train DRL agents more efficiently by adapting task selection to their evolving abilities. While multiple standard benchmarks exist to compare DRL agents, there is currently no such thing for ACL algorithms. Thus, comparing existing approaches is difficult, as too many experimental parameters differ from paper to paper. In this work, we identify several key challenges faced by ACL algorithms. Based on these, we present TeachMyAgent (TA), a benchmark of current ACL algorithms leveraging procedural task generation. It includes 1) challenge-specific unit-tests using variants of a procedural Box2D bipedal walker environment, and 2) a new procedural Parkour environment combining most ACL challenges, making it ideal for global performance assessment. We then use TeachMyAgent to conduct a comparative study of representative existing approaches, showcasing the competitiveness of some ACL algorithms that do not use expert knowledge. We also show that the Parkour environment remains an open problem. We open-source our environments, all studied ACL algorithms (collected from open-source code or re-implemented), and DRL students in a Python package available at https://github.com/flowersteam/TeachMyAgent.

Motivation & Objective

• To address the lack of standardized benchmarks for evaluating Automatic Curriculum Learning (ACL) algorithms in Deep Reinforcement Learning (DRL).
• To identify key challenges in ACL, such as task space complexity, feasibility, and generalization, that hinder algorithm comparison and progress.
• To develop a unified benchmark with procedural task generation to enable fair, reproducible, and scalable evaluation of ACL methods.
• To evaluate existing ACL algorithms across diverse DRL student models and task morphologies, revealing performance gaps and limitations.
• To open-source environments, algorithms, and DRL students to accelerate research and foster community-wide comparison.

Proposed method

• Design a benchmark with two procedural environments: a Box2D-based bipedal walker with unit-test variants and a new Parkour environment combining multiple ACL challenges.
• Implement 8 representative ACL algorithms—ALP-GMM, Covar-GMM, GoalGAN, RIAC, SPDL, Setter-Solver, ADR, and Random—for direct comparison.
• Use procedural content generation (PCG) via CPPNs to create diverse, scalable, and challenging task spaces for curriculum learning.
• Evaluate all ACL methods across 3 DRL student morphologies (bipedal walker, fish, climber) and 16 random seeds per setup to ensure statistical robustness.
• Apply statistical testing (Welch’s t-test) at regular intervals to determine significant performance differences between ACL methods and random baseline.
• Open-source all environments, algorithms, and DRL models via a Python package at https://github.com/flowersteam/TeachMyAgent.

Experimental results

Research questions

• RQ1 Can a standardized benchmark improve the comparability and reproducibility of ACL algorithm evaluations in Deep RL?
• RQ2 Which ACL algorithms perform best across diverse DRL student architectures and task morphologies?
• RQ3 To what extent do ACL methods without expert knowledge outperform random task selection in complex, procedural environments?
• RQ4 Why do current ACL methods fail to master complex locomotion tasks like climbing in the Parkour environment?
• RQ5 Can simplifying the task space reveal the true potential of existing ACL algorithms?

Key findings

• ALP-GMM significantly outperforms random task selection across all morphologies, mastering 42.7% of tasks on the bipedal walker and 26.4% overall in the Parkour environment.
• Covar-GMM and RIAC show performance close to ALP-GMM, with 35.7% and 31.2% mastery on the bipedal walker, respectively, indicating strong generalization.
• SPDL and Setter-Solver perform similarly to random, with 30.6% and 28.75% mastery on the bipedal walker, respectively, and fail to significantly outperform random.
• ADR reaches a performance plateau early and achieves only 14.7% mastery on the bipedal walker, significantly worse than random in some cases.
• In a simplified task space without water, the random teacher masters 6% of tasks (up to 30% in best seed), suggesting that current ACL methods fail to discover feasible subspaces in complex environments.
• The Parkour environment remains an open challenge, as no ACL method achieves more than 1% mastery on the climbing morphology, indicating limitations in current curriculum discovery.