[论文解读] Neural Episodic Control

Alexander Pritzel, Benigno Uría+6 more|arXiv (Cornell University)|Mar 6, 2017

Reinforcement Learning in Robotics参考文献 41被引用 92

一句话总结

NEC 使用 per-action differentiable neural dictionaries to store and rapidly back up Q-values from recent experiences, achieving substantially faster data-efficient learning on Atari games than several deep RL baselines.

ABSTRACT

Deep reinforcement learning methods attain super-human performance in a wide range of environments. Such methods are grossly inefficient, often taking orders of magnitudes more data than humans to achieve reasonable performance. We propose Neural Episodic Control: a deep reinforcement learning agent that is able to rapidly assimilate new experiences and act upon them. Our agent uses a semi-tabular representation of the value function: a buffer of past experience containing slowly changing state representations and rapidly updated estimates of the value function. We show across a wide range of environments that our agent learns significantly faster than other state-of-the-art, general purpose deep reinforcement learning agents.

研究动机与目标

• Address data inefficiency in deep reinforcement learning by accelerating reward propagation and value estimation.
• Leverage a semi-tabular memory that combines slow-changing state representations with fast-updating value estimates.
• Enable rapid assimilation of new experiences via an append-only, memory-based Q-function akin to episodic memory.
• Investigate how fast memory updates interact with N-step returns and a shared CNN embedding to improve learning speed.

提出的方法

• Introduce a Differentiable Neural Dictionary (DND) per action that stores (key, value) pairs.
• Process states with a shared convolutional neural network to produce a key h for lookup in each action’s DND.
• Retrieve Q(s,a) as a weighted sum of values in the DND using a nearest-neighbor kernel over keys.
• Write new (h, Q^(N)(s,a)) pairs to the corresponding action’s DND; update existing keys via Q-learning like a tabular updater.
• Use N-step Q-learning targets Q^(N)(s,a)= sum_{j=0}^{N-1} gamma^j r_{t+j} + gamma^N max_a' Q(s_{t+N}, a'), with the max taken by querying all memories.
• Train end-to-end with a differentiable network by minimizing the L2 loss between predicted Q(s,a) and Q^(N)(s,a) over mini-batches from a replay buffer.

实验结果

研究问题

• RQ1 Can a memory-augmented, semi-tabular value function accelerate data-efficient learning in deep RL environments like Atari?
• RQ2 How does adding a fast-updating memory (DND) per action influence reward propagation and learning speed compared to standard DQN/A3C baselines?
• RQ3 What is the impact of N-step Q-learning and differentiable memory on final performance and data efficiency across diverse Atari games?
• RQ4 Does an append-only, large-scale memory with approximate nearest-neighbor access provide practical benefits over episodic resets in memory?

主要发现

• NEC learns significantly faster in the small data regime across Atari games than DQN, A3C, and several λ-return baselines.
• Across early learning, NEC outperforms all baselines; at around 40 million frames, DQN with Prioritised Replay can surpass NEC on average.
• NEC achieves human-level performance in about 25% of the tested games within 10 million frames, indicating strong data efficiency.
• NEC and MFEC both explore episodic-like value estimation; NEC, however, uses a reward-guided embedding to improve interpolation of values.
• NEC generally outperforms MFEC and Prioritised Replay in learning speed and data efficiency, especially before around 5-10 million frames.