[Paper Review] A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning

Amy Zhang, Nicolas Ballas+1 more|arXiv (Cornell University)|Jun 20, 2018

Reinforcement Learning in Robotics11 references93 citations

TL;DR

The paper studies generalization vs memorization in deep reinforcement learning for continuous domains, introduces seed-based and randomized-reward tests, and shows that sufficient training diversity enables generalization in simulated environments while natural data pose greater generalization challenges.

ABSTRACT

The risks and perils of overfitting in machine learning are well known. However most of the treatment of this, including diagnostic tools and remedies, was developed for the supervised learning case. In this work, we aim to offer new perspectives on the characterization and prevention of overfitting in deep Reinforcement Learning (RL) methods, with a particular focus on continuous domains. We examine several aspects, such as how to define and diagnose overfitting in MDPs, and how to reduce risks by injecting sufficient training diversity. This work complements recent findings on the brittleness of deep RL methods and offers practical observations for RL researchers and practitioners.

Motivation & Objective

• Motivate and characterize overfitting in deep reinforcement learning (RL) within continuous domains.
• Define practical notions of generalization in RL, including within-task and out-of-task scenarios.
• Develop diagnostic tests (seed randomization, reward randomization, distribution shifts) to detect memorization.
• Investigate how training diversity affects generalization across model-free and model-based RL methods.
• Provide actionable guidelines for evaluating and improving robustness of RL policies.

Proposed method

• Formalize RL generalization in terms of training vs. test seeds to separate stochasticity sources.
• Use randomized reward experiments to assess memorization tendencies under limited seed variation.
• Evaluate generalization by expanding initial state distributions and injecting observation noise.
• Compare model-free (DQN/PPO) and model-based RL approaches across discrete and continuous action spaces.
• Apply a range of environments (Cartpole, Acrobot, Reacher, Thrower) and natural image-based tasks to study generalization.
• Provide implementation and hyperparameter details to enable replication.

Experimental results

Research questions

• RQ1 How does seed diversity affect generalization versus memorization in continuous RL tasks?
• RQ2 Can randomized rewards reveal memorization tendencies in deep RL models, and how does seed count influence this?
• RQ3 How do distributional shifts in initial states and observation noise impact transfer/generalization performance?
• RQ4 Do model-based RL approaches improve or worsen generalization under limited training seeds?
• RQ5 What are the differences in generalization behavior between simulated-domain tasks and natural-image grounded tasks?

Key findings

• Deep RL can overfit in both simple simulated tasks and natural-image based tasks when training diversity is limited.
• Increasing the number of training seeds generally improves generalization in simulated domains, even with continuous actions.
• Randomized reward experiments reveal memorization especially under few training seeds, but larger seed sets reduce memorization effects.
• Out-of-task generalization improves with more training environments but degrades as domain shift increases (initial state and observation noise).
• Model-based RL can propagate or amplify biases in limited-seed regimes, showing worse generalization in some continuous control tasks.
• Natural image tasks exhibit a larger generalization gap than synthetic tasks, highlighting the need for new benchmarks.