[Paper Review] Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detection

Lucas N. Alegre, Ana L. C. Bazzan+1 more|arXiv (Cornell University)|May 19, 2021

Data Stream Mining Techniques5 citations

TL;DR

This paper proposes MBCD, a model-based reinforcement learning algorithm that uses online high-confidence change-point detection to enable rapid adaptation in non-stationary environments with continuous states and actions. By detecting context changes with minimal delay while bounding false alarms, MBCD outperforms state-of-the-art model-free, model-based, and meta-learning methods—especially in fully online settings without pre-training or when facing out-of-distribution contexts.

ABSTRACT

Non-stationary environments are challenging for reinforcement learning algorithms. If the state transition and/or reward functions change based on latent factors, the agent is effectively tasked with optimizing a behavior that maximizes performance over a possibly infinite random sequence of Markov Decision Processes (MDPs), each of which drawn from some unknown distribution. We call each such MDP a context. Most related works make strong assumptions such as knowledge about the distribution over contexts, the existence of pre-training phases, or a priori knowledge about the number, sequence, or boundaries between contexts. We introduce an algorithm that efficiently learns policies in non-stationary environments. It analyzes a possibly infinite stream of data and computes, in real-time, high-confidence change-point detection statistics that reflect whether novel, specialized policies need to be created and deployed to tackle novel contexts, or whether previously-optimized ones might be reused. We show that (i) this algorithm minimizes the delay until unforeseen changes to a context are detected, thereby allowing for rapid responses; and (ii) it bounds the rate of false alarm, which is important in order to minimize regret. Our method constructs a mixture model composed of a (possibly infinite) ensemble of probabilistic dynamics predictors that model the different modes of the distribution over underlying latent MDPs. We evaluate our algorithm on high-dimensional continuous reinforcement learning problems and show that it outperforms state-of-the-art (model-free and model-based) RL algorithms, as well as state-of-the-art meta-learning methods specially designed to deal with non-stationarity.

Motivation & Objective

• To address the challenge of efficient policy learning in non-stationary environments where MDP dynamics and rewards change unpredictably due to unknown latent factors.
• To detect context changes with minimal delay while maintaining a low false alarm rate, enabling rapid deployment of specialized policies.
• To operate effectively in fully online settings without requiring a pre-training phase or prior knowledge of context distribution or boundaries.
• To outperform existing model-free, model-based, and meta-learning RL algorithms in high-dimensional continuous control tasks under non-stationarity.
• To enable sample-efficient adaptation when encountering novel, out-of-distribution contexts not seen during training.

Proposed method

• The algorithm constructs a mixture model of probabilistic dynamics predictors, each specialized to a distinct context mode in the unknown distribution of underlying MDPs.
• It employs online high-confidence change-point detection statistics to monitor prediction errors and detect shifts in system dynamics in real time.
• Change-point detection is based on sequential hypothesis testing with a confidence threshold, ensuring low false alarm rates while minimizing detection delay.
• When a change is detected, the system triggers the creation and deployment of a new, context-specific policy, reusing previously optimized models when appropriate.
• The method operates without requiring a pre-training phase and dynamically adapts to arbitrarily different contexts drawn from unknown distributions.
• It combines model-based planning with context-aware policy adaptation, enabling rapid response to environmental shifts.

Experimental results

Research questions

• RQ1 Can online change-point detection enable faster and more reliable detection of context changes in non-stationary MDPs compared to existing methods?
• RQ2 How does the proposed method perform in fully online settings without a pre-training phase?
• RQ3 Can the algorithm maintain high performance when faced with out-of-distribution contexts not seen during training?
• RQ4 How does the method compare to state-of-the-art model-free and model-based RL algorithms in non-stationary continuous control tasks?
• RQ5 To what extent does the method reduce regret by minimizing both detection delay and false alarms?

Key findings

• MBCD achieves approximately constant and high performance across changing contexts in the non-stationary maze domain, even without a pre-training phase, while state-of-the-art RL algorithms like MBPO and SAC degrade significantly.
• In the fully online setting (no pre-training), MBCD outperforms MBPO and SAC, demonstrating robustness and adaptability in real-time adaptation.
• When tested on out-of-distribution contexts not seen during training, MBCD degrades gracefully with a short adaptation period, whereas meta-learning methods like ReBAL and GrBAL fail catastrophically.
• The algorithm maintains a low false alarm rate while achieving minimal detection delay, enabling rapid response to environmental shifts.
• MBCD significantly outperforms meta-learning methods designed for non-stationarity, particularly when the test distribution differs from the training distribution.
• The method enables efficient, continual online learning in non-stationary environments without requiring prior knowledge of context boundaries or distributions.