[Paper Review] Gotta Learn Fast: A New Benchmark for Generalization in RL

Alex Nichol, Vicki Pfau+3 more|arXiv (Cornell University)|Apr 10, 2018

Reinforcement Learning in RoboticsComputer Science21 references85 citations

TL;DR

Introduces a Sonic the Hedgehog–based RL benchmark (Gym Retro Sonic) with train/test splits to evaluate transfer and few-shot generalization, plus baseline results.

ABSTRACT

In this report, we present a new reinforcement learning (RL) benchmark based on the Sonic the Hedgehog (TM) video game franchise. This benchmark is intended to measure the performance of transfer learning and few-shot learning algorithms in the RL domain. We also present and evaluate some baseline algorithms on the new benchmark.

Motivation & Objective

• Motivate the need for RL benchmarks with explicit train/test splits to measure cross-task generalization.
• Propose a meta-learning friendly RL dataset built from multiple similar tasks (Sonic levels).
• Provide baseline algorithms to gauge transfer and few-shot learning capabilities in RL.

Proposed method

• Construct a Sonic-based RL benchmark using Gym Retro with a train/test split across 58 saved level states.
• Use frame skipping and sticky frame skip to simulate realistic timing and stochasticity.
• Define rewards based on horizontal progress with a completion bonus to encourage fast completion.
• Evaluate multiple baselines including Rainbow (DQN variant), PPO, and a non-visual JERK baseline as a transfer/few-shot benchmark.
• Implement joint training (transfer) setups where training on multiple levels initializes test-level learning.
• Provide detailed level-wise and aggregate performance metrics for comparison.

Experimental results

Research questions

• RQ1 How well do RL agents generalize across similar tasks (levels) in a fixed distribution of Sonic games?
• RQ2 Can transfer learning and few-shot RL approaches reduce sample complexity on unseen levels compared to training from scratch?
• RQ3 What baseline transfer strategies (joint training, feature reuse) most effectively improve test performance?

Key findings

• Human players achieve the highest aggregate test score among baselines (7438.2 ± 624.2).
• Joint training transfer (Joint PPO/Rainbow) generally improves test performance over non-joint training baselines, with Joint PPO achieving 3926.3 ± 78.1 aggregate (test) and Joint Rainbow 3704.2 ± 151.1.
• Rainbow with reward preprocessing outperforms standard PPO on the test set (Rainbow: 2748.6 ± 102.2 vs PPO: 1488.8 ± 42.8).
• JERK, a non-deep-learning scripted approach, performs competitively, sometimes outperforming vanilla PPO due to structured exploration (1904.0 ± 21.9 aggregate).
• Joint Rainbow and Joint PPO demonstrate transfer from training to test levels, indicating some generalization, though the best transfer still trails human performance.