[Paper Review] Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation

Qingfeng Lan, Yangchen Pan+2 more|arXiv (Cornell University)|May 22, 2022

Machine Learning and ELM4 citations

TL;DR

This paper proposes MeDQN, a memory-efficient reinforcement learning algorithm that reduces catastrophic forgetting in deep Q-networks by consolidating knowledge from the target Q-network to the current Q-network via a consolidation loss. With a replay buffer at least 10× smaller than standard DQN, MeDQN achieves comparable or better performance on both tabular and image-based environments, significantly reducing memory usage while maintaining high sample efficiency.

ABSTRACT

Artificial neural networks are promising for general function approximation but challenging to train on non-independent or non-identically distributed data due to catastrophic forgetting. The experience replay buffer, a standard component in deep reinforcement learning, is often used to reduce forgetting and improve sample efficiency by storing experiences in a large buffer and using them for training later. However, a large replay buffer results in a heavy memory burden, especially for onboard and edge devices with limited memory capacities. We propose memory-efficient reinforcement learning algorithms based on the deep Q-network algorithm to alleviate this problem. Our algorithms reduce forgetting and maintain high sample efficiency by consolidating knowledge from the target Q-network to the current Q-network. Compared to baseline methods, our algorithms achieve comparable or better performance in both feature-based and image-based tasks while easing the burden of large experience replay buffers.

Motivation & Objective

• To address catastrophic forgetting in deep reinforcement learning when training on non-i.i.d. data due to limited memory capacity.
• To reduce the memory burden of large experience replay buffers in on-device and edge RL applications.
• To maintain high sample efficiency and learning stability without relying on large-scale data storage.
• To enable effective knowledge retention in continual RL settings with minimal memory overhead.

Proposed method

• Introduce a knowledge consolidation loss that transfers value function knowledge from the target Q-network to the current Q-network.
• Use a tunable hyperparameter to balance learning new experiences and retaining old knowledge.
• Replace large experience replay buffers with uniform or real-state sampling, reducing memory usage by up to 10×.
• Apply the consolidation loss during training updates to stabilize learning and reduce forgetting.
• Design two variants: MeDQN(U) for uniform state sampling and MeDQN(R) for real-state sampling, tailored to different task types.
• Maintain standard DQN architecture but modify the learning objective to include knowledge consolidation, preserving compatibility with existing frameworks.

Experimental results

Research questions

• RQ1 Can knowledge consolidation from the target network reduce forgetting in DQN without relying on large replay buffers?
• RQ2 How does the performance of MeDQN compare to standard DQN when using a replay buffer 10× smaller?
• RQ3 Does knowledge consolidation improve sample efficiency and learning stability in both low- and high-dimensional control tasks?
• RQ4 Can MeDQN maintain performance with minimal memory usage in image-based environments like Atari and MinAtar?
• RQ5 What is the optimal balance between learning new knowledge and remembering past knowledge in continual RL?

Key findings

• MeDQN(R) with a buffer size of 100k outperformed standard DQN with a 1M buffer in three Atari games: Name This Game, Phoenix, and Qbert.
• In Seaquest (MinAtar), MeDQN(R) achieved a return of 25.82 ± 1.69 with a 100k buffer, outperforming DQN’s 20.48 ± 0.75 with a 1M buffer.
• MeDQN(R) matched DQN’s performance on two other Atari games despite using only 10% of the replay buffer memory.
• Memory usage was reduced from 7GB to 0.7GB in Atari games, confirming a 10× reduction in memory footprint without performance degradation.
• MeDQN(U) and MeDQN(R) showed improved computation efficiency due to reduced forgetting and fewer required updates.
• The method achieved state-of-the-art performance with minimal memory overhead, demonstrating strong robustness across diverse environments.