[Paper Review] Policy Consolidation for Continual Reinforcement Learning

Christos Kaplanis, Murray Shanahan+1 more|arXiv (Cornell University)|Feb 1, 2019

Domain Adaptation and Few-Shot Learning10 citations

TL;DR

This paper proposes Policy Consolidation (PC), a continual reinforcement learning method that mitigates catastrophic forgetting by distilling an agent’s policy across multiple timescales using a cascade of hidden networks. The approach improves continual learning performance across single-task, alternating-task, and multi-agent self-play environments without requiring task boundaries or prior knowledge of distribution shifts.

ABSTRACT

We propose a method for tackling catastrophic forgetting in deep reinforcement learning that is extit{agnostic} to the timescale of changes in the distribution of experiences, does not require knowledge of task boundaries, and can adapt in extit{continuously} changing environments. In our extit{policy consolidation} model, the policy network interacts with a cascade of hidden networks that simultaneously remember the agent's policy at a range of timescales and regularise the current policy by its own history, thereby improving its ability to learn without forgetting. We find that the model improves continual learning relative to baselines on a number of continuous control tasks in single-task, alternating two-task, and multi-agent competitive self-play settings.

Motivation & Objective

• Address catastrophic forgetting in deep reinforcement learning when the data distribution changes continuously over time.
• Develop a method that does not rely on task boundaries or predefined task segmentation.
• Enable continual learning in non-stationary environments such as single-task training with evolving dynamics and multi-agent self-play.
• Improve policy retention and performance by leveraging multi-timescale knowledge distillation.
• Design a scalable and agnostic approach to continual learning that works across diverse RL settings.

Proposed method

• Implement a cascade of hidden networks that maintain policy representations at different timescales, from recent to long-term.
• Use bidirectional knowledge distillation between the policy network and each hidden network to regularize the current policy using historical knowledge.
• Integrate proximal policy optimization (PPO) to stabilize training and control policy updates.
• Apply knowledge distillation using KL divergence to align action distributions between the policy and each hidden network.
• Train the policy network with a surrogate objective that penalizes large policy changes, ensuring stability.
• Dynamically update the hidden networks at varying timescales to capture both short-term and long-term policy behavior.

Experimental results

Research questions

• RQ1 Can a continual RL agent maintain performance on previously learned tasks when the data distribution evolves continuously without task boundaries?
• RQ2 How effective is multi-timescale policy distillation in reducing catastrophic forgetting across diverse RL environments?
• RQ3 Does the absence of task boundary detection affect the robustness and generalization of continual learning?
• RQ4 How does Policy Consolidation compare to existing continual learning baselines in non-stationary settings like self-play and alternating tasks?
• RQ5 Can the method be applied effectively in environments with gradual, non-discrete distribution shifts?

Key findings

• The PC model significantly improves continual learning performance across all evaluated settings: single-task, alternating two-task, and multi-agent self-play.
• In alternating two-task environments, PC outperforms baselines in maintaining performance on previously learned tasks without task boundary detection.
• The method reduces catastrophic forgetting by stabilizing policy updates through multi-timescale distillation, even when the environment dynamics shift gradually.
• The model achieves better average returns compared to standard PPO and other continual learning baselines in non-stationary training regimes.
• Empirical results show that the cascade of hidden networks effectively preserves policy knowledge across timescales, enhancing long-term retention.
• The approach is robust to unpredictable changes in data distribution and does not require prior knowledge of when or how the environment changes.