[Paper Review] Continual State Representation Learning for Reinforcement Learning using Generative Replay

Hugo Caselles-Dupré, M. Ortíz+1 more|arXiv (Cornell University)|Oct 9, 2018

Reinforcement Learning in Robotics17 references19 citations

TL;DR

This paper proposes a continual state representation learning method for reinforcement learning using variational autoencoders (VAEs) and generative replay to prevent catastrophic forgetting. By automatically detecting environment changes via Welch’s t-test on VAE reconstruction errors and generating past experience samples, the model maintains bounded system size and achieves forward transfer, outperforming fine-tuning and raw input baselines in RL performance.

ABSTRACT

We consider the problem of building a state representation model in a continual fashion. As the environment changes, the aim is to efficiently compress the sensory state's information without losing past knowledge. The learned features are then fed to a Reinforcement Learning algorithm to learn a policy. We propose to use Variational Auto-Encoders for state representation, and Generative Replay, i.e. the use of generated samples, to maintain past knowledge. We also provide a general and statistically sound method for automatic environment change detection. Our method provides efficient state representation as well as forward transfer, and avoids catastrophic forgetting. The resulting model is capable of incrementally learning information without using past data and with a bounded system size.

Motivation & Objective

• To address catastrophic forgetting in continual reinforcement learning by maintaining stable state representations across evolving environments.
• To enable autonomous, data-efficient learning without access to past experience data.
• To develop a statistically sound, automatic method for detecting environment changes during continual learning.
• To evaluate whether learned state representations support efficient and transferable policy learning in RL.
• To demonstrate forward transfer and robustness to distribution shifts using VAEs with generative replay.

Proposed method

• Uses variational autoencoders (VAEs) to learn compressed, disentangled state representations from sensory inputs.
• Applies generative replay by sampling from the VAE’s latent space of previous environments to augment new experience, preventing forgetting.
• Employs Welch’s t-test on VAE reconstruction error distributions to automatically detect environment changes, avoiding arbitrary thresholds.
• Trains the VAE on a joint distribution of real new data and generated past data, maintaining a bounded system size.
• Uses a statistical hypothesis test (null hypothesis: equal mean reconstruction error) with p-value threshold of 0.01 to detect distribution shifts.
• Evaluates performance via RL training using PPO with representations from VAEs trained under different regimes.

Experimental results

Research questions

• RQ1 Can generative replay with VAEs effectively prevent catastrophic forgetting in continual state representation learning?
• RQ2 Does the proposed automatic environment change detection method reliably identify distribution shifts without prior knowledge?
• RQ3 Do the learned state representations enable efficient and high-performing policy learning in RL?
• RQ4 Is there forward transfer, where knowledge from earlier environments improves performance on later tasks?
• RQ5 How does the method compare to fine-tuning and raw input baselines in terms of reconstruction fidelity and RL sample efficiency?

Key findings

• The proposed method achieved 100% success rate in detecting environment changes when change was present and 99.5% when absent, using Welch’s t-test on reconstruction errors.
• Generative Replay reduced mean squared error (MSE) for environment 1 reconstruction to 3.3×10⁻⁴, compared to 1.3×10⁻³ for fine-tuning, indicating effective forgetting mitigation.
• On environment 2, both methods achieved similar MSE (6.4×10⁻⁴ vs. 9.3×10⁻⁴), confirming no degradation in current task performance.
• In PPO training, the Generative Replay VAE representation led to significantly higher final mean rewards than fine-tuning and raw inputs, especially on the second environment.
• The method demonstrated forward transfer: policies trained on features from the first environment performed well on the second, indicating useful generalization.
• The VAE with generative replay outperformed all baselines in sample efficiency and final performance, confirming its effectiveness for continual RL.