[Paper Review] Generalization and Regularization in DQN

Jesse Farebrother, Marlos C. Machado+1 more|arXiv (Cornell University)|Sep 29, 2018

Reinforcement Learning in Robotics25 references100 citations

TL;DR

The paper evaluates generalization in DQN using Atari 2600 game flavours, showing DQN overfits to training flavours and that regularization plus fine-tuning can yield more general, reusable representations improving sample efficiency.

ABSTRACT

Deep reinforcement learning algorithms have shown an impressive ability to learn complex control policies in high-dimensional tasks. However, despite the ever-increasing performance on popular benchmarks, policies learned by deep reinforcement learning algorithms can struggle to generalize when evaluated in remarkably similar environments. In this paper we propose a protocol to evaluate generalization in reinforcement learning through different modes of Atari 2600 games. With that protocol we assess the generalization capabilities of DQN, one of the most traditional deep reinforcement learning algorithms, and we provide evidence suggesting that DQN overspecializes to the training environment. We then comprehensively evaluate the impact of dropout and $\ell_2$ regularization, as well as the impact of reusing learned representations to improve the generalization capabilities of DQN. Despite regularization being largely underutilized in deep reinforcement learning, we show that it can, in fact, help DQN learn more general features. These features can be reused and fine-tuned on similar tasks, considerably improving DQN's sample efficiency.

Motivation & Objective

• Assess generalization of DQN across subtly different Atari 2600 game flavours (modes/difficulties).
• Quantify overfitting tendencies of DQN to a training flavour.
• Evaluate regularization techniques (dropout and L2) on DQN performance across flavours.
• Investigate whether regularized representations can be reused and fine-tuned on related tasks to improve sample efficiency.

Proposed method

• Introduce a protocol using Atari 2600 flavours (modes and difficulties) within the ALE to test generalization.
• Train DQN in a default flavour (m0d0) for 50M frames and evaluate on other flavours.
• Apply dropout to first four layers and L2 weight regularization during training; perform grid search to select hyperparameters.
• Compare performance of regularized policies to non-regularized baselines across flavours.
• Explore two transfer learning strategies: (i) fine-tuning entire network, (ii) fine-tuning early layers, after pretraining with regularization.

Experimental results

Research questions

• RQ1 Can DQN policies trained on a single Atari flavour generalize to visually/dynamically similar flavours?
• RQ2 Do traditional regularization techniques improve cross-flavour generalization and/or enable more reusable representations in DQN?
• RQ3 Does pretraining with regularization yield better fine-tuning performance on new flavours than training from scratch?
• RQ4 To what extent do regularized representations reduce sample complexity when transferring to related tasks?

Key findings

• DQN policies show poor generalization across flavours and exhibit overfitting to the training flavour in several games (e.g., Freeway).
• Regularization (dropout + L2) during training improves cross-flavour evaluation in several cases and can enhance sample efficiency, though it does not by itself guarantee cross-flavour generalization.
• Regularized representations can serve as better initializations for fine-tuning on new flavours, often outperforming scratch initialization under equivalent or lower total training frames.
• Fine-tuning the entire network after regularized pretraining yields strong gains in several games (notably HERO and Space Invaders), suggesting learned general features.
• Fine-tuning only early layers after regularized pretraining also helps, indicating some layer-wise transferability of features.