[Paper Review] The Impact of Non-stationarity on Generalisation in Deep Reinforcement Learning

Maximilian Igl, Gregory Farquhar+3 more|arXiv (Cornell University)|Jun 10, 2020

Reinforcement Learning in Robotics28 references15 citations

TL;DR

This paper identifies non-stationarity in deep reinforcement learning—caused by evolving policies and changing training targets—as a key factor undermining generalization. It proposes Iterated Relearning (ITER), which repeatedly transfers knowledge from a trained policy to a newly initialized network, reducing non-stationarity and improving performance on ProcGen and Multiroom benchmarks.

ABSTRACT

Non-stationarity arises in Reinforcement Learning (RL) even in stationary environments. Most RL algorithms collect new data throughout training, using a non-stationary behaviour policy. Furthermore, training targets in RL can change even with a fixed state distribution when the policy, critic, or bootstrap values are updated. We study these types of non-stationarity in supervised learning settings as well as in RL, finding that they can lead to worse generalisation performance when using deep neural network function approximators. Consequently, to improve generalisation of deep RL agents, we propose Iterated Relearning (ITER). ITER augments standard RL training by repeated knowledge transfer of the current policy into a freshly initialised network, which thereby experiences less non-stationarity during training. Experimentally, we show that ITER improves performance on the challenging generalisation benchmarks ProcGen and Multiroom.

Motivation & Objective

• To investigate how non-stationarity in policy and training targets negatively affects generalization in deep RL with function approximators.
• To analyze the impact of non-stationarity in both supervised learning and RL settings using deep networks.
• To develop a training method that reduces non-stationarity during policy learning to enhance generalization performance.
• To empirically validate improvements in generalization on challenging benchmarks such as ProcGen and Multiroom.

Proposed method

• ITER repeatedly transfers the current policy's knowledge into a newly initialized neural network to reduce non-stationarity during training.
• The method uses knowledge distillation to transfer policy behavior from a trained network to a fresh, randomly initialized network at regular intervals.
• Training proceeds in cycles: a policy is trained, then its knowledge is transferred to a new network, which is then fine-tuned from scratch.
• The process iteratively reduces the impact of non-stationary targets and behavior policies on the learning process.
• The approach maintains standard RL training objectives but introduces periodic reinitialization and knowledge transfer to stabilize learning dynamics.
• The method is applied to both value-based and policy-based RL algorithms in the experiments.

Experimental results

Research questions

• RQ1 How does non-stationarity in policy and training targets affect generalization in deep RL with function approximators?
• RQ2 To what extent does knowledge distillation into a newly initialized network reduce non-stationarity during training?
• RQ3 Can iterative retraining improve generalization on challenging, diverse RL benchmarks like ProcGen and Multiroom?
• RQ4 How does ITER compare to standard training in terms of sample efficiency and robustness to distributional shift?
• RQ5 Does the reduction in non-stationarity lead to more stable and generalizable policies across diverse environments?

Key findings

• Non-stationarity in policy and training targets significantly degrades generalization performance in deep RL, even in stationary environments.
• ITER improves performance on the ProcGen benchmark, achieving higher mean and median scores compared to standard training.
• On the Multiroom environment, ITER demonstrates superior generalization across unseen test levels, indicating improved robustness.
• The method reduces the negative impact of non-stationary targets and behavior policies, leading to more stable learning dynamics.
• Knowledge transfer into newly initialized networks effectively mitigates the effects of non-stationarity during training.
• ITER achieves consistent improvements across multiple environments without modifying the underlying RL algorithm or hyperparameters.