[Paper Review] Continual Learning Using World Models for Pseudo-Rehearsal

Nicholas Ketz, Soheil Kolouri+1 more|arXiv (Cornell University)|Mar 6, 2019

Model Reduction and Neural Networks31 references4 citations

TL;DR

This paper proposes a label-free continual learning method for world models using pseudo-rehearsal, where internally generated rollouts of past experiences are interleaved with real environment interactions to prevent catastrophic forgetting. The approach enables stable unsupervised temporal prediction and partial continual learning in policy gradient agents across sequential Atari games without task segmentation.

ABSTRACT

The utility of learning a dynamics/world model of the environment in reinforcement learning has been shown in a many ways. When using neural networks, however, these models suffer catastrophic forgetting when learned in a lifelong or continual fashion. Current solutions to the continual learning problem require experience to be segmented and labeled as discrete tasks, however, in continuous experience it is generally unclear what a sufficient segmentation of tasks would be. Here we propose a method to continually learn these internal world models through the interleaving of internally generated episodes of past experiences (i.e., pseudo-rehearsal). We show this method can sequentially learn unsupervised temporal prediction, without task labels, in a disparate set of Atari games. Empirically, this interleaving of the internally generated rollouts with the external environment's observations leads to a consistent reduction in temporal prediction loss compared to non-interleaved learning and is preserved over repeated random exposures to various tasks. Similarly, using a network distillation approach, we show that modern policy gradient based reinforcement learning algorithms can use this internal model to continually learn to optimize reward based on the world model's representation of the environment.

Motivation & Objective

• To address catastrophic forgetting in lifelong reinforcement learning without requiring explicit task segmentation.
• To explore whether pseudo-rehearsal can preserve world model performance across sequential, unlabeled environments.
• To evaluate the utility of internal world models in supporting continual policy learning for deep reinforcement learning agents.
• To investigate the role of simulated rollouts in maintaining representation stability and enabling transfer.

Proposed method

• The method uses a world model architecture (M) trained on raw pixel transitions to predict future states, actions, and rewards.
• Pseudo-rehearsal is implemented by generating synthetic rollouts from a preserved copy of the world model (M*) and interleaving them with real experience during training.
• The internal model is trained via unsupervised temporal prediction loss on both real and simulated transitions.
• A separate agent (C) is trained using policy gradients, with the world model's latent representation as input for decision-making.
• The training process alternates between updating the world model and the agent, using both real and simulated data.
• The approach avoids task labeling by treating the environment stream as a continuous, unlabeled sequence of experiences.

Experimental results

Research questions

• RQ1 Can pseudo-rehearsal stabilize world model learning in a continual, unlabeled environment without task segmentation?
• RQ2 How does interleaving simulated rollouts with real experience affect temporal prediction loss over time?
• RQ3 To what extent can the internal world model support continual policy learning in a reinforcement learning agent?
• RQ4 What is the relationship between the quality of pseudo-rehearsal rollouts and performance retention in the world model?
• RQ5 Does the method enable transfer of knowledge across different Atari games without explicit task boundaries?

Key findings

• The interleaved training with pseudo-rehearsal led to a consistent reduction in temporal prediction loss compared to non-interleaved training across repeated exposures.
• The world model maintained stable performance on previously seen Atari games even after multiple random exposures to new tasks, indicating reduced catastrophic forgetting.
• For the controlling agent, pseudo-rehearsal improved cumulative reward in Task 1 (River Raid), with a measurable performance drop in non-interleaved training during later epochs.
• No significant performance difference was observed in Task 2 (Tutankham) due to lack of reward, and Task 3 (Crazy Climber) showed no discernible benefit, possibly due to low complexity or high variance.
• The results suggest that while the world model benefits significantly from pseudo-rehearsal, the agent's continual learning is more sensitive to architectural choices such as action space representation and training dynamics.
• The study demonstrates that pseudo-rehearsal can be effective in a label-free continual learning setting, particularly for unsupervised world model training.