[Paper Review] Assessing Generalization in Deep Reinforcement Learning

Charles Packer, Katelyn Gao+4 more|arXiv (Cornell University)|Oct 29, 2018

Reinforcement Learning in Robotics42 references114 citations

TL;DR

This paper provides a reproducible benchmark and protocol to study generalization in deep RL, comparing vanilla A2C/PPO to generalization-focused methods (EPOpt, RL2) across interpolation and extrapolation in varied control tasks.

ABSTRACT

Deep reinforcement learning (RL) has achieved breakthrough results on many tasks, but agents often fail to generalize beyond the environment they were trained in. As a result, deep RL algorithms that promote generalization are receiving increasing attention. However, works in this area use a wide variety of tasks and experimental setups for evaluation. The literature lacks a controlled assessment of the merits of different generalization schemes. Our aim is to catalyze community-wide progress on generalization in deep RL. To this end, we present a benchmark and experimental protocol, and conduct a systematic empirical study. Our framework contains a diverse set of environments, our methodology covers both in-distribution and out-of-distribution generalization, and our evaluation includes deep RL algorithms that specifically tackle generalization. Our key finding is that `vanilla' deep RL algorithms generalize better than specialized schemes that were proposed specifically to tackle generalization.

Motivation & Objective

• Motivate the need for a controlled, reproducible generalization benchmark in deep RL.
• Introduce a diverse set of control-task environments with parameterized dynamics changes.
• Evaluate vanilla and generalized RL algorithms under in-distribution and out-of-distribution settings.
• Provide interpretable generalization metrics (default, interpolation, extrapolation) and baselines for fair comparison.

Proposed method

• Define a fixed MDP formalism and distributions over environments to study interpolation and extrapolation.
• Benchmark six algorithms (A2C, PPO, EPOpt-A2C, EPOpt-PPO, RL2-A2C, RL2-PPO) on six environments with controlled parameter variations.
• Use two network architectures (FF and RC) to study effect of representation on generalization.
• Train/test under nine training-testing regime pairs (D/R/E training; D/R/E testing) with standardized episode counts.
• Compute three generalization metrics: Default (DD), Interpolation (RR), and Extrapolation (GM of DR/DE/RE).
• Present a reproducible experimental protocol including hyperparameter sweeps and multiple seeds.

Experimental results

Research questions

• RQ1 How well do vanilla deep RL agents generalize to unseen environment variations (interpolation) and to more extreme variations (extrapolation)?
• RQ2 Do specialized generalization schemes (EPOpt, RL2) outperform vanilla algorithms on these benchmarks?
• RQ3 How do architecture choices (FF vs RC) influence generalization performance across tasks?
• RQ4 Under what conditions does robustness/adaptation-based generalization provide advantages or fail to train?

Key findings

• Vanilla A2C and PPO often generalize better than their EPOpt and RL2 counterparts under the proposed protocols.
• Extrapolation is consistently harder than interpolation across tasks and algorithms.
• Training on a distribution of environments improves interpolation performance, but extrapolation remains challenging.
• EPOpt improves generalization over PPO on some continuous-action tasks (e.g., Hopper, Pendulum, HalfCheetah) but not universally across all environments or with A2C.
• RL2 variants were difficult to train and generally underperformed compared to vanilla baselines under the same resources.
• RC (recurrent) architectures affect PPO performance and can hinder training in fixed-environment settings, highlighting interaction between architecture and algorithm.