[Paper Review] Non-Stationary Reinforcement Learning: The Blessing of (More) Optimism

Wang Chi Cheung, David Simchi‐Levi+1 more|arXiv (Cornell University)|Jun 7, 2019

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper proposes SWUCRL2-CW and BORL algorithms for non-stationary reinforcement learning in Markov decision processes with time-varying rewards and transitions bounded by variation budgets. By introducing a novel confidence widening technique that increases optimism, the algorithms achieve optimal dynamic regret bounds without requiring prior knowledge of variation budgets, outperforming existing methods in both theory and practice.

ABSTRACT

We consider un-discounted reinforcement learning (RL) in Markov decision processes (MDPs) under temporal drifts, ie, both the reward and state transition distributions are allowed to evolve over time, as long as their respective total variations, quantified by suitable metrics, do not exceed certain variation budgets. This setting captures the endogeneity, exogeneity, uncertainty, and partial feedback in sequential decision-making scenarios, and finds applications in vehicle remarketing and real-time bidding. We first develop the Sliding Window Upper-Confidence bound for Reinforcement Learning with Confidence Widening (SWUCRL2-CW) algorithm, and establish its dynamic regret bound when the variation budgets are known. In addition, we propose the Bandit-over-Reinforcement Learning (BORL) algorithm to adaptively tune the SWUCRL2-CW algorithm to achieve the same dynamic regret bound, but in a parameter-free manner, ie, without knowing the variation budgets. Finally, we conduct numerical experiments to show that our proposed algorithms achieve superior empirical performance compared to existing algorithms. Notably, the interplay between endogeneity and exogeneity presents a unique challenge, absent in existing (stationary and non-stationary) stochastic online learning settings, when we apply the conventional Optimism in Face of Uncertainty principle to design algorithms with provably low dynamic regret for RL in drifting MDPs. We overcome the challenge by a novel confidence widening technique that incorporates additional optimism into our learning algorithms to ensure low dynamic regret bounds. To extend our theoretical findings, we apply our framework to inventory control problems, and demonstrate how one can alternatively leverage special structures on the state transition distributions to bypass the difficulty in exploring time-varying environments.

Motivation & Objective

• To address un-discounted reinforcement learning in non-stationary MDPs where rewards and transition distributions evolve over time within bounded variation budgets.
• To overcome the challenge of balancing endogeneity (policy-dependent dynamics) and exogeneity (time-varying external factors) in applying optimism-based learning.
• To design a parameter-free algorithm that adapts to unknown variation budgets while maintaining optimal dynamic regret performance.
• To demonstrate the effectiveness of confidence widening in improving exploration and regret bounds in drifting environments.
• To show empirical superiority of the proposed algorithms over baseline methods like UCRL2 and UCRL2.S in dynamic environments.

Proposed method

• Introduces the SWUCRL2-CW algorithm, which combines sliding window estimation with confidence widening to handle time-varying MDPs when variation budgets are known.
• Employs a novel confidence widening technique that injects additional optimism into value function estimates to ensure low dynamic regret.
• Designs the BORL algorithm as a master algorithm that adaptively tunes the SWUCRL2-CW algorithm using EXP3.P, enabling parameter-free operation without prior knowledge of variation budgets.
• Uses sliding window estimators to track recent reward and transition statistics, reducing sensitivity to long-term drifts.
• Applies the Optimism in Face of Uncertainty (OFU) principle in a modified form, enhanced by confidence widening to counteract the interplay between endogeneity and exogeneity.
• Leverages structural properties in single-item inventory control with fixed costs to bypass the need for confidence widening in specific cases.

Experimental results

Research questions

• RQ1 Can optimism-based reinforcement learning achieve low dynamic regret in non-stationary MDPs with both time-varying rewards and transitions?
• RQ2 How can the interplay between endogenous policy effects and exogenous environmental drifts be managed to maintain low regret?
• RQ3 Can confidence widening be used to enhance optimism in a way that improves regret bounds beyond standard OFU principles?
• RQ4 Is it possible to design a parameter-free algorithm that matches the regret performance of a tuned algorithm in non-stationary RL?
• RQ5 To what extent can structural properties of state transitions (e.g., in inventory control) reduce the need for confidence widening?

Key findings

• The SWUCRL2-CW algorithm achieves a dynamic regret bound of $ \tilde{O}(\sqrt{B_r T} + \sqrt{B_p T}) $ when variation budgets $ B_r $ and $ B_p $ are known, matching the theoretical lower bound up to logarithmic factors.
• The BORL algorithm achieves the same $ \tilde{O}(\sqrt{B_r T} + \sqrt{B_p T}) $ dynamic regret bound without requiring knowledge of $ B_r $ and $ B_p $, making it parameter-free.
• In numerical experiments, both SWUCRL2-CW and BORL outperform UCRL2 and UCRL2.S by at least 20% in cumulative rewards under low and moderate variations.
• In scenarios with high variation in transition distributions ($ B_p = \Theta(T^{0.5}) $), the improvement was 12%, showing robustness to transition drifts.
• Surprisingly, BORL outperformed SWUCRL2-CW in some settings despite not knowing variation budgets, indicating the adaptive EXP3.P master algorithm effectively leveraged benign environments.
• The confidence widening technique was shown to be critical in maintaining low regret, especially when exogenous drifts in transitions were more severe than in rewards.