[Paper Review] Quantifying Generalization in Reinforcement Learning

Karl Cobbe, Oleg Klimov+3 more|arXiv (Cornell University)|Dec 6, 2018

Reinforcement Learning in RoboticsComputer Science20 references195 citations

TL;DR

This paper introduces CoinRun, a procedurally generated RL benchmark to quantify generalization, reveals substantial overfitting with standard training setups, and shows that architectural choices and regularization techniques can improve generalization in RL.

ABSTRACT

In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice offers relatively little insight into an agent's ability to generalize. We address this issue by using procedurally generated environments to construct distinct training and test sets. Most notably, we introduce a new environment called CoinRun, designed as a benchmark for generalization in RL. Using CoinRun, we find that agents overfit to surprisingly large training sets. We then show that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.

Motivation & Objective

• Motivate the need for explicit generalization metrics in deep RL by separating training and test environments.
• Show that agents overfit to large training sets and quantify generalization gaps on CoinRun.
• Evaluate how architectural choices and regularization techniques affect RL generalization on procedurally generated tasks.

Proposed method

• Introduce CoinRun, a procedurally generated environment with a fixed reward structure to assess zero-shot generalization.
• Train multiple agents on varying numbers of training levels and measure zero-shot test performance without fine-tuning.
• Compare CNN architectures (Nature-CNN vs. IMPALA-CNN) and scale to IMPALA-Large to assess generalization differences.
• Apply regularization techniques from supervised learning (L2, dropout, data augmentation, batch normalization) and assess their impact on generalization.
• Inject stochasticity via environment or policy (epsilon-greedy, entropy bonuses) to study effects on generalization.
• Explore additional environments (CoinRun-Platforms, RandomMazes) to corroborate overfitting findings.

Experimental results

Research questions

• RQ1 To what extent do RL agents overfit when training and testing are performed on distinct sets of procedurally generated levels?
• RQ2 How do architectural choices (CNN variants) influence zero-shot generalization in CoinRun?
• RQ3 What is the impact of regularization methods (L2, dropout, data augmentation, batch normalization) on generalization in RL?
• RQ4 Does increasing stochasticity in the environment or policy improve generalization, and by how much?
• RQ5 Do results generalize to other procedurally generated tasks beyond CoinRun?

Key findings

• Agents exhibit substantial overfitting when trained on fewer than 4,000 training levels, with gaps persisting even at 16,000 levels.
• The IMPALA-CNN architecture generalizes better than Nature-CNN across training sets and test time.
• Deeper variants (IMPALA-Large) can further improve generalization.
• L2 regularization and dropout reduce the generalization gap, with dropout at p=0.1 and L2 weight w=1e-4 being notably effective.
• Data augmentation (Cutout-like masking) and batch normalization significantly boost generalization.
• Introducing stochasticity via epsilon-greedy actions or entropy bonuses improves generalization, often more than regularization alone.
• Combining data augmentation, batch normalization, and L2 regularization yields modest additional gains; combining stochasticity with regularization is less consistently beneficial.
• Regularization and architectural improvements on CoinRun-Platforms and RandomMazes corroborate substantial overfitting and generalization challenges in RL.