[Paper Review] Prioritized Level Replay

Minqi Jiang, Edward Grefenstette+1 more|arXiv (Cornell University)|Oct 8, 2020

Reinforcement Learning in Robotics33 references4 citations

TL;DR

Prioritized Level Replay (PLR) improves sample efficiency and generalization in procedural content generation (PCG) environments by dynamically selecting training levels based on estimated future learning potential, using TD-errors as a proxy. It achieves state-of-the-art performance on the ProcGen benchmark, matching prior SOTA in test return and improving generalization by over 76% when combined with prior methods.

ABSTRACT

Environments with procedurally generated content serve as important benchmarks for testing systematic generalization in deep reinforcement learning. In this setting, each level is an algorithmically created environment instance with a unique configuration of its factors of variation. Training on a prespecified subset of levels allows for testing generalization to unseen levels. What can be learned from a level depends on the current policy, yet prior work defaults to uniform sampling of training levels independently of the policy. We introduce Prioritized Level Replay (PLR), a general framework for selectively sampling the next training level by prioritizing those with higher estimated learning potential when revisited in the future. We show TD-errors effectively estimate a level's future learning potential and, when used to guide the sampling procedure, induce an emergent curriculum of increasingly difficult levels. By adapting the sampling of training levels, PLR significantly improves sample efficiency and generalization on Procgen Benchmark--matching the previous state-of-the-art in test return--and readily combines with other methods. Combined with the previous leading method, PLR raises the state-of-the-art to over 76% improvement in test return relative to standard RL baselines.

Motivation & Objective

• To address the challenge of poor generalization in deep reinforcement learning on procedurally generated environments.
• To improve sample efficiency by selectively sampling training levels based on their future learning potential.
• To develop a method that dynamically adapts level sampling during training without relying on predefined difficulty rankings.
• To enable better generalization to unseen levels by focusing on those that offer the most informative learning signals.

Proposed method

• PLR uses temporal-difference (TD) errors from past trajectories to estimate each level’s future learning potential.
• It maintains level scores S and staleness timestamps C to balance learning potential and recency of experience.
• The next training level is sampled from a distribution that combines both the estimated learning potential (S) and staleness (C), with a trade-off parameter ρ.
• It supports both unseen level sampling (from the full training set) and replay sampling (from previously seen levels) via a unified sampling procedure.
• The method is compatible with any off-policy RL algorithm and can be combined with data augmentation or other generalization techniques.
• Level scores are updated after each episode using a scoring function that incorporates TD-error and trajectory length.

Experimental results

Research questions

• RQ1 Can dynamically prioritizing training levels based on estimated learning potential improve generalization in procedural environments?
• RQ2 How does using TD-error as a proxy for future learning potential affect sample efficiency and test performance?
• RQ3 Can a curriculum emerge organically from level sampling guided by learning potential without prior difficulty annotations?
• RQ4 To what extent does PLR improve generalization when combined with existing state-of-the-art methods?
• RQ5 Does the balance between learning potential and staleness (via ρ) significantly affect performance across diverse PCG environments?

Key findings

• PLR matches the previous state-of-the-art in test return on the ProcGen benchmark, achieving performance comparable to the best existing methods.
• When combined with the previous leading method, PLR improves test return by over 76% relative to standard RL baselines.
• PLR significantly improves sample efficiency and generalization in 10 to 11 out of 16 games, depending on the prioritization strategy (rank or proportional).
• The method induces an emergent curriculum where increasingly difficult levels are prioritized as training progresses, based on learning potential estimates.
• Performance gains are statistically significant (p < 0.05) in 10–11 of 16 games, demonstrating consistent improvement across diverse environments.
• The combination of level score and staleness in the replay distribution is essential for performance gains, as shown by ablation studies.