[Paper Review] Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design

Matthew Thomas Jackson, Minqi Jiang+6 more|arXiv (Cornell University)|Oct 4, 2023

Advanced Bandit Algorithms ResearchDecision Sciences3 citations

TL;DR

This paper proposes GROOVE, a method for discovering general reinforcement learning algorithms by using adversarial environment design to meta-train a policy optimizer. By introducing algorithmic regret (AR) as a novel metric to guide curriculum learning, GROOVE significantly improves zero-shot generalization on unseen environments like Atari, outperforming prior methods such as LPG in both robustness and transfer performance.

ABSTRACT

The past decade has seen vast progress in deep reinforcement learning (RL) on the back of algorithms manually designed by human researchers. Recently, it has been shown that it is possible to meta-learn update rules, with the hope of discovering algorithms that can perform well on a wide range of RL tasks. Despite impressive initial results from algorithms such as Learned Policy Gradient (LPG), there remains a generalization gap when these algorithms are applied to unseen environments. In this work, we examine how characteristics of the meta-training distribution impact the generalization performance of these algorithms. Motivated by this analysis and building on ideas from Unsupervised Environment Design (UED), we propose a novel approach for automatically generating curricula to maximize the regret of a meta-learned optimizer, in addition to a novel approximation of regret, which we name algorithmic regret (AR). The result is our method, General RL Optimizers Obtained Via Environment Design (GROOVE). In a series of experiments, we show that GROOVE achieves superior generalization to LPG, and evaluate AR against baseline metrics from UED, identifying it as a critical component of environment design in this setting. We believe this approach is a step towards the discovery of truly general RL algorithms, capable of solving a wide range of real-world environments.

Motivation & Objective

• To address the generalization gap in meta-learned reinforcement learning algorithms when applied to unseen environments.
• To investigate how characteristics of the meta-training distribution impact generalization performance in policy meta-optimization (PMO).
• To develop a method that automatically generates informative, challenging curricula for meta-training optimizers.
• To introduce algorithmic regret (AR) as a proxy metric for task informativeness in environment design for PMO.
• To demonstrate that GROOVE enables superior zero-shot transfer to out-of-distribution environments like Atari, compared to existing methods such as LPG.

Proposed method

• Proposes a novel formulation of policy meta-optimization (PMO) using the Meta-UPOMDP framework to model the interaction between meta-optimizer, environment curator, and agent.
• Introduces algorithmic regret (AR), a novel regret approximation tailored for PMO, which measures the performance gap between a meta-learned optimizer and an optimal baseline on a given task.
• Employs an adversarial curriculum generation mechanism where a level curator is trained to maximize the regret of the meta-learned optimizer, using a gradient-based update rule.
• Uses a two-stage training loop: during meta-training, agents are trained on dynamically curated environments, and the meta-optimizer is updated based on agent returns and AR.
• Leverages a fast JAX-based implementation of GROOVE, reducing meta-training time to 3 hours on a single V100 GPU, enabling broader academic access.
• Adapts ideas from Unsupervised Environment Design (UED) to the meta-RL setting, extending PLR to support meta-optimization and environment curation.

Figure 1: Out-of-distribution performance on Atari—after meta-training exclusively on Grid-World levels, our method (GROOVE) significantly outperforms LPG on Atari. Improvement is measured as a percentage of mean human-normalized return over 5 seeds.

Experimental results

Research questions

• RQ1 How do characteristics of the meta-training distribution affect the generalization performance of meta-learned RL optimizers?
• RQ2 Can algorithmic regret (AR) serve as an effective proxy metric for identifying informative and challenging environments in policy meta-optimization?
• RQ3 Does adversarial environment curation via AR lead to more robust and generalizable RL optimizers compared to standard or random curriculum design?
• RQ4 How does GROOVE compare to LPG in terms of zero-shot generalization to out-of-distribution environments such as Atari?
• RQ5 What is the impact of the antagonist agent in AR on the quality of the meta-training curriculum?

Key findings

• GROOVE achieves significantly improved out-of-distribution generalization on Atari games after meta-training exclusively on Grid-World environments, outperforming LPG by a substantial margin in human-normalized return.
• Algorithmic regret (AR) is identified as a critical component in environment design for PMO, outperforming baseline UED metrics on Min-Atar, where prior metrics fail to surpass random sampling.
• Ablation studies show that both PLR and LPG are insufficient without AR, and that the antagonist agent in AR is essential for effective curriculum generation.
• GROOVE demonstrates increased robustness to in-distribution challenges, including difficult Grid-World levels, indicating stronger generalization capacity.
• The method reduces meta-training time to 3 hours on a single V100 GPU, a 10x speedup over the original LPG implementation, enabling wider academic adoption.
• The open-source release of GROOVE and LPG in JAX lowers computational barriers, facilitating future research in general RL algorithm discovery.

Figure 2: GROOVE meta-training (left) and meta-testing (right). During meta-training, levels are sampled from both the level curator and sampler. Agents are trained by an optimizer for multiple updates, before a meta-optimizer updates the optimizer based on agent return. At the end of an agent’s lif