[Paper Review] Automated curricula through setter-solver interactions

Sébastien Racanière, Andrew K. Lampinen+4 more|arXiv (Cornell University)|Sep 27, 2019

Reinforcement Learning in Robotics28 references34 citations

TL;DR

The paper presents a setter-solver framework for automatic curriculum generation in goal-conditioned RL with dynamic, sparsely rewarded environments, highlighting goal validity, feasibility, and coverage as key objectives, and demonstrates conditioning on environment observations and targeting desired goal distributions.

ABSTRACT

Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance. But in dynamic or sparsely rewarding environments these correlations are often too small, or rewarding events are too infrequent to make learning feasible. Human education instead relies on curricula--the breakdown of tasks into simpler, static challenges with dense rewards--to build up to complex behaviors. While curricula are also useful for artificial agents, hand-crafting them is time consuming. This has lead researchers to explore automatic curriculum generation. Here we explore automatic curriculum generation in rich, dynamic environments. Using a setter-solver paradigm we show the importance of considering goal validity, goal feasibility, and goal coverage to construct useful curricula. We demonstrate the success of our approach in rich but sparsely rewarding 2D and 3D environments, where an agent is tasked to achieve a single goal selected from a set of possible goals that varies between episodes, and identify challenges for future work. Finally, we demonstrate the value of a novel technique that guides agents towards a desired goal distribution. Altogether, these results represent a substantial step towards applying automatic task curricula to learn complex, otherwise unlearnable goals, and to our knowledge are the first to demonstrate automated curriculum generation for goal-conditioned agents in environments where the possible goals vary between episodes.

Motivation & Objective

• Motivate the need for automated curricula in complex, dynamic, and sparsely rewarded tasks.
• Propose a setter-solver framework that generates goal curricula for a solver agent.
• Define and optimize three setter objectives: goal validity, goal feasibility, and goal coverage.
• Explore conditioning on environment observations to handle varying environments.
• Investigate targeting a known desired distribution of tasks when available.

Proposed method

• Introduce a three-component model: a solver (goal-conditioned RL agent), a setter (generative model of goals), and a judge (feasibility predictor).
• Train the solver with goals generated by the setter using distributed policy-gradient learning (V-trace with entropy regularization).
• Define goal validity as the existence of a solver policy achieving the goal; feasibility as the current probability the solver will achieve it; and coverage as the variability of generated goals.
• Setter losses: (i) validity loss to bias toward achievable goals, (ii) feasibility loss to align goals with judge predictions, (iii) coverage loss to maximize diversity in goals.
• Optional extensions include targeting a known desired goal distribution via a Wasserstein discriminator and conditioning setter/judge on environment observations to handle varying environments.

Experimental results

Research questions

• RQ1 Can automatic curricula improve learning in goal-conditioned RL in dynamic, sparse-reward environments?
• RQ2 Do goal validity, feasibility, and coverage jointly enable effective curricula in complex tasks?
• RQ3 Does conditioning the setter/judge on environment observations help in environments where goals vary by episode?
• RQ4 Can targeting a known desired distribution of goals accelerate mastery of specific tasks?
• RQ5 How does the proposed approach compare to existing methods like Goal GAN in complex, variable environments?

Key findings

• All three setter losses (validity, feasibility, coverage) are necessary for good performance in complex environments.
• Conditioning the setter and judge on environment observations improves performance in varying environments.
• Targeting a known desired distribution can accelerate learning for certain tasks, though benefits depend on task structure.
• The method outperforms baselines and prior approaches on 3D and grid-world experiments with sparse rewards.
• Compared to Goal GAN, the setter-solver approach is more stable and effective on the tasks tested.
• The work demonstrates automated curriculum generation for goal-conditioned agents in environments where goals vary between episodes.