[Paper Review] Model-Free Episodic Control

Charles Blundell, Benigno Uría+7 more|arXiv (Cornell University)|Jun 14, 2016

Memory and Neural Mechanisms35 references172 citations

TL;DR

The paper introduces a non-parametric episodic control mechanism that rapidly stores and replays high-return experiences to learn policies much faster than deep RL methods, demonstrated on Atari and Labyrinth environments.

ABSTRACT

State of the art deep reinforcement learning algorithms take many millions of interactions to attain human-level performance. Humans, on the other hand, can very quickly exploit highly rewarding nuances of an environment upon first discovery. In the brain, such rapid learning is thought to depend on the hippocampus and its capacity for episodic memory. Here we investigate whether a simple model of hippocampal episodic control can learn to solve difficult sequential decision-making tasks. We demonstrate that it not only attains a highly rewarding strategy significantly faster than state-of-the-art deep reinforcement learning algorithms, but also achieves a higher overall reward on some of the more challenging domains.

Motivation & Objective

• Motivate fast, one-shot-like learning in reinforcement learning by leveraging hippocampal-inspired episodic control.
• Develop a simple, non-parametric episodic memory (Q_EC) that stores and replays high-return state-action traces.
• Address memory and generalization challenges of tabular RL with a nearest-neighbours scheme and representations.
• Demonstrate data-efficient learning and competitive or superior performance to baseline deep RL methods in fixed environments.

Proposed method

• Maintain a growing non-parametric table Q_EC(s,a) storing the highest return achieved for each state-action pair.
• Use k-nearest-neighbours in a per-action buffer to generalize Q_EC for novel states via平均ing the k nearest stored states’ values (Eq. 2).
• Embed observations with a feature mapping phi (random projection or VAE-based latent features) to determine state space distances.
• Execute episodes using the current Q_EC-implied policy, then perform backward replay over the episode to update Q_EC entries (Eq. 1).
• Limit memory with least-recently-used forgetting and cap buffers per action to manage growth.
• Discuss representation learning with VAEs as a means to capture salient factors of variation and improve distance metrics (Appendix on VAEs).

Experimental results

Research questions

• RQ1 Can model-free episodic control achieve rapid learning by reusing high-return experiences without gradient-based updates?
• RQ2 How do memory size, nearest-neighbour parameters (k), and representation choices affect performance and generalization in deterministic and stochastic-like environments?
• RQ3 Does episodic control remain effective in more realistic 3D environments where states are rarely revisited?
• RQ4 What is the impact of simple exploration (epsilon-greedy) when paired with episodic memory for data efficiency?

Key findings

• Episodic control learns significantly faster than state-of-the-art deep RL algorithms during initial learning in Atari games and Labyrinth levels.
• With k-NN buffers and simple representations, the method outperforms DQN, DQN with prioritized replay, and A3C in early learning phases across several games.
• In deterministic Atari environments, many states, actions are revisited, enabling effective use of episodic memory (e.g., 10-60% of states re-encountered).
• In Labyrinth, episodic control achieves one-shot-like learning and can surpass or rapidly approach A3C performance, especially in sparse reward settings (Double-T-Maze).
• Using VAEs for representations can provide generalization benefits in Labyrinth, with larger k improving final performance; Atari shows mixed results depending on task structure.
• A simple epsilon-greedy exploration strategy suffices for fast learning when paired with episodic control.