[Paper Review] Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended version

Erwan Lecarpentier, Emmanuel Rachelson|arXiv (Cornell University)|Apr 22, 2019

Reinforcement Learning in Robotics31 references4 citations

TL;DR

This paper proposes Risk-Averse Tree-Search (RATS), a model-based reinforcement learning algorithm for non-stationary Markov Decision Processes (NSMDPs) with Lipschitz-continuous time-varying dynamics. RATS plans under worst-case model evolution by treating environment changes as adversarial, ensuring robust performance across all possible evolutions and maximizing the worst-case return, as empirically validated in a grid-world benchmark with varying environmental drifts.

ABSTRACT

This work tackles the problem of robust zero-shot planning in non-stationary stochastic environments. We study Markov Decision Processes (MDPs) evolving over time and consider Model-Based Reinforcement Learning algorithms in this setting. We make two hypotheses: 1) the environment evolves continuously with a bounded evolution rate; 2) a current model is known at each decision epoch but not its evolution. Our contribution can be presented in four points. 1) we define a specific class of MDPs that we call Non-Stationary MDPs (NSMDPs). We introduce the notion of regular evolution by making an hypothesis of Lipschitz-Continuity on the transition and reward functions w.r.t. time; 2) we consider a planning agent using the current model of the environment but unaware of its future evolution. This leads us to consider a worst-case method where the environment is seen as an adversarial agent; 3) following this approach, we propose the Risk-Averse Tree-Search (RATS) algorithm, a zero-shot Model-Based method similar to Minimax search; 4) we illustrate the benefits brought by RATS empirically and compare its performance with reference Model-Based algorithms.

Motivation & Objective

• To address robust planning in non-stationary stochastic environments where the MDP evolves over time with bounded rate of change.
• To formalize a class of Non-Stationary MDPs (NSMDPs) with Lipschitz-continuous transition and reward functions over time.
• To develop a planning algorithm that operates under model uncertainty—knowing only the current model but not its future evolution—by assuming worst-case environmental evolution.
• To design a method that maximizes the minimal achievable return across all possible future evolutions of the environment, ensuring robustness.
• To empirically validate that the proposed approach outperforms standard model-based methods in worst-case performance guarantees.

Proposed method

• Define Non-Stationary MDPs (NSMDPs) as time-dependent MDPs with time-varying transition and reward functions, assuming Lipschitz continuity in time for both.
• Formulate a worst-case planning approach where the environment is treated as an adversarial agent that evolves to minimize the agent’s return.
• Propose the Risk-Averse Tree-Search (RATS) algorithm, a minimax-style search that plans over a tree of possible future model evolutions under the worst-case assumption.
• Use a search depth parameter $d_{ ext{max}}$ to control the approximation of the worst-case future model, with leaf nodes representing terminal states.
• Apply RATS to compute policies that maximize the lower bound on return across all possible evolutions, ensuring robustness.
• Compare RATS against baseline model-based algorithms (e.g., DP-snapshot, DP-NSMDP) using a grid-world environment with controlled environmental drift via parameter $\epsilon$.

Experimental results

Research questions

• RQ1 Can a model-based reinforcement learning agent achieve robust performance in non-stationary environments when it only knows the current model and not its future evolution?
• RQ2 How can worst-case planning be formalized in NSMDPs with time-varying dynamics under Lipschitz continuity constraints?
• RQ3 Does a minimax-style planning approach that assumes adversarial environmental evolution lead to better worst-case performance guarantees than standard expected-return maximization?
• RQ4 To what extent does the proposed method reduce sensitivity to environmental drift compared to conventional model-based planners?
• RQ5 Can the worst-case return be effectively approximated using a finite-depth tree search under bounded model evolution?

Key findings

• RATS achieves consistently high performance across all values of $\epsilon$, which controls the direction of environmental drift, demonstrating robustness to model evolution.
• DP-snapshot performs well only when $\epsilon=0$ (favorable drift) but fails catastrophically when $\epsilon=1$, highlighting its lack of robustness.
• The return distribution of RATS shows a significantly reduced left tail compared to baseline methods, indicating lower risk of low returns.
• RATS achieves the highest Conditional Value at Risk (CVaR) at 5%, outperforming both DP-snapshot and DP-NSMDP, which optimize for expected return instead.
• The worst-case return of RATS is maximized across all possible evolutions, confirming that it successfully optimizes for the minimal achievable outcome.
• Empirical results show that RATS maintains strong performance even when the environment evolves adversarially, validating its worst-case planning approach.