[Paper Review] Episodic Memory Deep Q-Networks

Zichuan Lin, Tianqi Zhao+2 more|arXiv (Cornell University)|May 19, 2018

Reinforcement Learning in Robotics16 references4 citations

TL;DR

This paper proposes Episodic Memory Deep Q-Networks (EMDQN), a biologically inspired reinforcement learning algorithm that enhances deep Q-learning by using episodic memory to supervise training. By distilling high-reward trajectories into the Q-network via a learnable weighting factor λ, EMDQN achieves state-of-the-art performance on Atari games using 1/5 the data required by standard DQN, significantly improving sample efficiency and reducing Q-value overestimation.

ABSTRACT

Reinforcement learning (RL) algorithms have made huge progress in recent years by leveraging the power of deep neural networks (DNN). Despite the success, deep RL algorithms are known to be sample inefficient, often requiring many rounds of interaction with the environments to obtain satisfactory performance. Recently, episodic memory based RL has attracted attention due to its ability to latch on good actions quickly. In this paper, we present a simple yet effective biologically inspired RL algorithm called Episodic Memory Deep Q-Networks (EMDQN), which leverages episodic memory to supervise an agent during training. Experiments show that our proposed method can lead to better sample efficiency and is more likely to find good policies. It only requires 1/5 of the interactions of DQN to achieve many state-of-the-art performances on Atari games, significantly outperforming regular DQN and other episodic memory based RL algorithms.

Motivation & Objective

• To improve sample efficiency in deep reinforcement learning by integrating episodic memory into deep Q-networks.
• To address the slow convergence and high data requirements of standard DQN in near-deterministic environments.
• To reduce Q-value overestimation in function-approximated Q-learning without relying on double Q-learning.
• To explore a biologically inspired fusion of parametric (DQN) and non-parametric (episodic control) learning systems for better policy learning.

Proposed method

• EMDQN introduces a learnable weighting factor λ to combine the standard DQN loss with a supervision signal derived from episodic memory.
• Episodic memory stores high-reward state-action trajectories from past episodes and provides a target value for Q-network updates.
• The Q-network is trained using a hybrid loss: a standard DQN loss and a memory-supervised loss that pulls Q-values toward the return of the best historical trajectory.
• The episodic memory is updated during training by storing the most rewarding sequences observed in each episode.
• The method uses a target network and experience replay, similar to DQN, but injects episodic memory signals to accelerate learning and stabilize training.
• The value of λ is fixed during training, but the paper suggests dynamic adjustment could further improve performance.

Experimental results

Research questions

• RQ1 Can episodic memory improve the sample efficiency of deep Q-learning in Atari environments?
• RQ2 How does episodic memory influence the stability and convergence of Q-value estimation in deep RL?
• RQ3 Can episodic memory naturally alleviate Q-value overestimation without using double Q-learning?
• RQ4 What is the impact of combining parametric and non-parametric learning systems in deep reinforcement learning?

Key findings

• EMDQN achieves state-of-the-art performance on Atari games using only 1/5 of the data required by standard DQN.
• The method significantly reduces training time and sample complexity while maintaining or improving final performance.
• EMDQN exhibits lower variance in Q-value learning curves compared to DQN, indicating more stable learning signals.
• The algorithm naturally mitigates Q-value overestimation, avoiding the catastrophic reward drops seen in DQN during training.
• In games like Battlezone, performance slightly declines in later training, suggesting potential for dynamic λ adjustment to balance exploration and exploitation.
• The episodic memory signal provides full reward propagation, unlike local bounds used in prior optimality tightening methods.