[Paper Review] Leveraging Procedural Generation to Benchmark Reinforcement Learning

Karl Cobbe, Christopher Hesse+2 more|arXiv (Cornell University)|Dec 3, 2019

Reinforcement Learning in Robotics25 references171 citations

TL;DR

Procgen Benchmark provides 16 procedurally generated, diverse game-like environments to evaluate RL agents on generalization and sample efficiency, showing larger models improve both metrics and that diverse distributions are essential.

ABSTRACT

We introduce Procgen Benchmark, a suite of 16 procedurally generated game-like environments designed to benchmark both sample efficiency and generalization in reinforcement learning. We believe that the community will benefit from increased access to high quality training environments, and we provide detailed experimental protocols for using this benchmark. We empirically demonstrate that diverse environment distributions are essential to adequately train and evaluate RL agents, thereby motivating the extensive use of procedural content generation. We then use this benchmark to investigate the effects of scaling model size, finding that larger models significantly improve both sample efficiency and generalization.

Motivation & Objective

• Motivate the need for diverse, high-quality RL training environments to measure generalization.
• Introduce Procgen Benchmark with 16 procedurally generated environments.
• Provide experimental protocols for evaluating sample efficiency and generalization with Procgen.
• Demonstrate how environment diversity and model scaling affect RL performance.

Proposed method

• Define 16 procedurally generated environments with shared 15-action space and 64x64x3 observations.
• Use deterministic state transition functions with procedural level variability to require robust policies.
• Provide experimental protocol: PPO baseline trained for 200M timesteps on full distributions; easy vs hard difficulty.
• Measure generalization via training on finite level sets (e.g., 500 levels) and testing on full distribution.
• Assess sample efficiency and generalization with varying training set sizes and level distributions.
• Scale model size using IMPALA-style CNN architectures (1x, 2x, 4x channel counts) and compare to Nature-CNN; adjust learning rates accordingly.

Experimental results

Research questions

• RQ1 How does training set size affect generalization across procedurally generated levels?
• RQ2 Does procedural generation improve the ability to generalize beyond training distributions compared to fixed-level benchmarks?
• RQ3 What is the impact of model size on sample efficiency and generalization in diverse RL environments?
• RQ4 How does PPO compare to Rainbow on Procgen benchmarks in terms of stability and performance?

Key findings

• Agents overfit strongly to small training sets; generalization improves with access to about 10,000 levels in most environments.
• Training on a fixed deterministic sequence of levels yields a large train/test gap, highlighting the need for diverse distributions.
• Larger models significantly improve both sample efficiency and generalization across environments.
• Scaling IMPALA-style channels by factors increases performance; the smaller Nature-CNN often fails to train on Procgen.
• PPO provides more consistent performance across Procgen than Rainbow, though Rainbow outperforms in several environments under tuned settings.