[Paper Review] Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation

Niels Justesen, Rubén Rodríguez Torrado+4 more|arXiv (Cornell University)|Jun 28, 2018

Reinforcement Learning in Robotics33 references119 citations

TL;DR

The paper investigates how procedural content generation (PCG) and Progressive PCG (PPCG) during training affect generalization of deep reinforcement learning agents across new, unseen levels, including human-designed ones, in multiple 2D games. PPCG can improve generalization and performance on some games, while distribution mismatch between training and test levels remains a key challenge.

ABSTRACT

Deep reinforcement learning (RL) has shown impressive results in a variety of domains, learning directly from high-dimensional sensory streams. However, when neural networks are trained in a fixed environment, such as a single level in a video game, they will usually overfit and fail to generalize to new levels. When RL models overfit, even slight modifications to the environment can result in poor agent performance. This paper explores how procedurally generated levels during training can increase generality. We show that for some games procedural level generation enables generalization to new levels within the same distribution. Additionally, it is possible to achieve better performance with less data by manipulating the difficulty of the levels in response to the performance of the agent. The generality of the learned behaviors is also evaluated on a set of human-designed levels. The results suggest that the ability to generalize to human-designed levels highly depends on the design of the level generators. We apply dimensionality reduction and clustering techniques to visualize the generators' distributions of levels and analyze to what degree they can produce levels similar to those designed by a human.

Motivation & Objective

• Demonstrate that deep RL agents overfit when trained on fixed, limited levels in 2D arcade games.
• Assess whether procedurally generated levels improve generalization within the training distribution and beyond.
• Introduce Progressive PCG to adapt level difficulty to the agent’s performance during training.
• Analyze how distributions of generated levels compare to human-designed levels using dimensionality reduction and clustering.

Proposed method

• Utilize A2C (Advantage Actor-Critic) with a CNN-based network on the General Video Game AI (GVG-AI) framework linked to OpenAI Gym.
• Implement four training regimens: single human-designed level; multiple human-designed levels; PCG with fixed difficulty; Progressive PCG that adjusts difficulty based on win/loss signals.
• Develop constructive PCG generators for Boulderdash, Frogs, Solarfox, and Zelda with a controllable difficulty parameter that affects active level size, object counts, and layout complexity.
• Introduce PPCG where level difficulty starts at 0 and increases after a win (decreases after a loss) with a shared difficulty across parallel workers.
• Evaluate generalization by testing trained policies on 30 pre-generated levels at difficulties 0.5 and 1, plus five human-designed levels; compare to random and max-score baselines.
• Apply PCA and DBSCAN to visualize and analyze the distribution of generated levels relative to human-designed levels.

Experimental results

Research questions

• RQ1 Does training on a fixed set of levels cause overfitting and poor generalization to unseen levels?
• RQ2 Can procedurally generated levels during training improve generalization within the generated level distribution, and can PPCG further enhance this generalization?
• RQ3 How does the generator design and the level difficulty progression impact transfer to human-designed levels?
• RQ4 Do generated levels approximate the distribution of human-designed levels, as assessed by dimensionality reduction and clustering?

Key findings

• Agents trained on a single level exhibit strong training-level scores but poor generalization to test levels, indicating overfitting.
• Procedurally generated levels enable generalization to unseen levels within the training distribution in some games, with PPCG further aiding learning by matching difficulty to agent performance.
• PPCG achieves notable improvements in Frogs (57% win rate on hard levels) and Zelda across procedurally generated and human-designed levels, while in Solarfox and Boulderdash fixed-difficulty PCG can outperform PPCG in certain settings.
• Generalization to human-designed levels is uneven and highly dependent on level-generator design and the alignment between generator-distributions and human-designed layouts.
• Dimensionality reduction and clustering reveal distinct clusters in generated levels, with some human-designed levels appearing as outliers, indicating distribution gaps between generated and human-designed content.