[Paper Review] Procedural Level Generation Improves Generality of Deep Reinforcement Learning

Niels Justesen, Rubén Rodríguez Torrado+4 more|arXiv (Cornell University)|Jun 28, 2018

Reinforcement Learning in Robotics17 citations

TL;DR

This paper proposes using procedural level generation (PLG) during training to improve the generalization of deep reinforcement learning agents in video games. By training agents on diverse, dynamically generated levels, the approach enhances performance on unseen levels—especially when combined with adaptive difficulty scaling—while analysis of generator distributions reveals that human-level similarity depends heavily on generator design.

ABSTRACT

Deep reinforcement learning (RL) has shown impressive results in a variety of domains, learning directly from high-dimensional sensory streams. However, when neural networks are trained in a fixed environment, such as a single level in a video game, they will usually overfit and fail to generalize to new levels. When RL models overfit, even slight modifications to the environment can result in poor agent performance. This paper explores how procedurally generated levels during training can increase generality. We show that for some games procedural level generation enables generalization to new levels within the same distribution. Additionally, it is possible to achieve better performance with less data by manipulating the difficulty of the levels in response to the performance of the agent. The generality of the learned behaviors is also evaluated on a set of human-designed levels. The results suggest that the ability to generalize to human-designed levels highly depends on the design of the level generators. We apply dimensionality reduction and clustering techniques to visualize the generators' distributions of levels and analyze to what degree they can produce levels similar to those designed by a human.

Motivation & Objective

• To address the overfitting of deep RL agents trained on fixed game levels, which limits their ability to generalize to new environments.
• To investigate whether procedural level generation during training can enhance generalization to unseen levels within the same distribution.
• To evaluate the transfer performance of trained agents on human-designed levels and understand the role of generator design in this generalization.
• To analyze the distribution of generated levels using dimensionality reduction and clustering to assess similarity to human-designed levels.

Proposed method

• Training deep reinforcement learning agents on procedurally generated levels instead of fixed, static levels to improve robustness and generalization.
• Implementing adaptive difficulty scheduling that modifies level generation based on agent performance to optimize data efficiency and learning speed.
• Applying dimensionality reduction (e.g., autoencoders or t-SNE) to visualize the latent space of generated levels and assess diversity and coverage.
• Using clustering techniques to analyze the distribution of generated levels and compare them to human-designed level clusters.
• Evaluating trained agents on both procedurally generated and human-designed test levels to measure zero-shot generalization performance.
• Designing level generators with varying degrees of structural and mechanical complexity to assess their impact on generalization.

Experimental results

Research questions

• RQ1 Can procedural level generation during training improve generalization of deep RL agents to new, unseen levels within the same game distribution?
• RQ2 How does adaptive difficulty scheduling during PLG affect data efficiency and final performance of RL agents?
• RQ3 To what extent can agents trained on procedural levels generalize to human-designed levels, and how does this depend on the design of the level generator?
• RQ4 How similar are the distributions of procedurally generated levels to those of human-designed levels, as measured by clustering and dimensionality reduction?

Key findings

• Agents trained with procedural level generation generalize significantly better to new levels within the same distribution compared to those trained on fixed levels.
• Adaptive difficulty scheduling during training reduces data requirements and improves final performance by maintaining optimal challenge levels.
• Generalization to human-designed levels is highly dependent on the design of the level generator, with some generators producing levels closer to human-made content than others.
• Dimensionality reduction and clustering reveal that only certain generators produce level distributions that are structurally and mechanically similar to human-designed levels.
• The study identifies that generator design directly influences the agent's ability to transfer knowledge to human-made content, highlighting a key bottleneck in generalization.