[Paper Review] Continual Reinforcement Learning with Multi-Timescale Replay

Christos Kaplanis, Claudia Clopath+1 more|arXiv (Cornell University)|Apr 16, 2020

Domain Adaptation and Few-Shot Learning28 references9 citations

TL;DR

This paper proposes a multi-timescale replay (MTR) buffer that improves continual reinforcement learning by maintaining experiences across multiple timescales, enabling better balance between retaining old knowledge and adapting to new data. The MTR framework, especially when combined with invariant risk minimization (IRM), outperforms baselines in nonstationary environments, particularly in HalfCheetah with fluctuating gravity.

ABSTRACT

In this paper, we propose a multi-timescale replay (MTR) buffer for improving continual learning in RL agents faced with environments that are changing continuously over time at timescales that are unknown to the agent. The basic MTR buffer comprises a cascade of sub-buffers that accumulate experiences at different timescales, enabling the agent to improve the trade-off between adaptation to new data and retention of old knowledge. We also combine the MTR framework with invariant risk minimization, with the idea of encouraging the agent to learn a policy that is robust across the various environments it encounters over time. The MTR methods are evaluated in three different continual learning settings on two continuous control tasks and, in many cases, show improvement over the baselines.

Motivation & Objective

• Address catastrophic forgetting in deep reinforcement learning agents operating in nonstationary environments with unknown timescales of distribution shift.
• Develop a replay buffer that maintains experiences across multiple timescales to balance adaptation to new data and retention of old knowledge.
• Explore the integration of invariant risk minimization (IRM) with MTR to learn policies robust across time-varying environments.
• Evaluate the MTR framework in continual learning settings where task boundaries are absent or undetectable, aligning with real-world deployment constraints.

Proposed method

• Design a cascade of sub-buffers, each accumulating experiences at distinct timescales, to store experiences over varying temporal ranges.
• Implement a probabilistic memory transfer mechanism where experiences are moved from one sub-buffer to the next based on a coin flip, simulating gradual forgetting.
• Use a power-law decay in memory retention probability, approximating a 1/t decay, to model psychological evidence of memory strength over time.
• Combine the MTR buffer with invariant risk minimization (IRM) to encourage policies that generalize across different environmental distributions encountered over time.
• Train agents using experience replay from the MTR buffer, sampling experiences from all sub-buffers to maintain a balanced distribution of old and new data.
• Apply the MTR framework to continuous control tasks (HalfCheetah and Ant) under three continual learning settings, including environments with time-varying dynamics like gravity.

Experimental results

Research questions

• RQ1 Can a multi-timescale replay buffer improve continual reinforcement learning performance in environments with unknown and varying distribution shifts?
• RQ2 How does the MTR buffer compare to single-timescale or dual-timescale replay methods in balancing retention and adaptation?
• RQ3 Does combining MTR with invariant risk minimization (IRM) lead to more robust policies in nonstationary environments?
• RQ4 How sensitive is the MTR method to hyperparameters such as the number of sub-buffers and memory transfer probability?
• RQ5 In what types of environments does the MTR-IRM combination provide the greatest performance gains?

Key findings

• The basic MTR agent demonstrated the most consistent performance across all continual learning settings tested on both HalfCheetah and Ant tasks.
• The MTR-IRM agent achieved the best performance in two of the more nonstationary settings on the HalfCheetah task, particularly under fluctuating gravity conditions.
• On the Ant task, the MTR-IRM agent underperformed compared to other methods, suggesting that the utility of IRM may depend on task-specific dynamics and environment shift characteristics.
• The MTR buffer’s memory retention probability approximates a 1/t decay, aligning with psychological models of memory forgetting.
• The method effectively mitigates catastrophic forgetting in environments where data distribution changes occur unpredictably and at varying timescales.
• The performance of MTR is sensitive to hyperparameters β_mtr (memory transfer probability) and n_b (number of sub-buffers), indicating a need for careful tuning.