[Paper Review] Acting in Delayed Environments with Non-Stationary Markov Policies

Esther Derman, Gal Dalal+1 more|arXiv (Cornell University)|Jan 28, 2021

Reinforcement Learning in Robotics31 references4 citations

TL;DR

This paper introduces a non-stationary Markov policy framework for reinforcement learning in environments with delayed action execution, proving that deterministic non-stationary policies in the original state space are optimal and sufficient. It proposes Delayed-Q, a model-based Q-learning algorithm that avoids state augmentation, achieving fast convergence and superior performance over state-augmented and oblivious methods even under high delays (m=25) in tabular, physical, and Atari environments.

ABSTRACT

The standard Markov Decision Process (MDP) formulation hinges on the assumption that an action is executed immediately after it was chosen. However, assuming it is often unrealistic and can lead to catastrophic failures in applications such as robotic manipulation, cloud computing, and finance. We introduce a framework for learning and planning in MDPs where the decision-maker commits actions that are executed with a delay of $m$ steps. The brute-force state augmentation baseline where the state is concatenated to the last $m$ committed actions suffers from an exponential complexity in $m$, as we show for policy iteration. We then prove that with execution delay, deterministic Markov policies in the original state-space are sufficient for attaining maximal reward, but need to be non-stationary. As for stationary Markov policies, we show they are sub-optimal in general. Consequently, we devise a non-stationary Q-learning style model-based algorithm that solves delayed execution tasks without resorting to state-augmentation. Experiments on tabular, physical, and Atari domains reveal that it converges quickly to high performance even for substantial delays, while standard approaches that either ignore the delay or rely on state-augmentation struggle or fail due to divergence. The code is available at github.com/galdl/rl_delay_basic and github.com/galdl/rl_delay_atari.

Motivation & Objective

• To address the lack of theoretical and practical frameworks for reinforcement learning with delayed action execution, a common issue in robotics, finance, and cloud computing.
• To challenge the assumption that state augmentation is necessary for handling execution delay, which suffers from exponential complexity in delay length m.
• To prove that deterministic non-stationary Markov policies in the original state space are sufficient and optimal under delayed execution.
• To develop a practical, model-based algorithm that avoids state augmentation and scales to continuous and high-dimensional environments like Atari.

Proposed method

• Introduces a delayed MDP formulation where actions are executed m steps after selection, modeling the resulting non-Markovian process.
• Proves that deterministic non-stationary policies in the original state space are sufficient for optimal performance, while stationary policies are suboptimal.
• Proposes Delayed-Q, a model-based Q-learning algorithm that learns non-stationary Q-values directly in the original state space without state augmentation.
• Uses a planning-based approach to estimate the most likely present state from delayed observations, enabling efficient value updates.
• Employs a value iteration-style update that accounts for the delayed execution by propagating rewards and transitions over m-step horizons.
• Applies the algorithm to tabular MDPs, continuous control environments (e.g., CartPole), and the Atari suite, demonstrating scalability and robustness.

Experimental results

Research questions

• RQ1 Can deterministic non-stationary Markov policies in the original state space achieve optimal performance in MDPs with delayed action execution?
• RQ2 Is state augmentation necessary for handling execution delay, or can optimal policies be learned directly in the original state space?
• RQ3 Can a model-based Q-learning algorithm be designed to avoid the exponential complexity of state augmentation while maintaining performance under high delays?
• RQ4 How does the proposed method compare to state-augmented, oblivious, and RNN-based policies in terms of convergence and final performance across diverse environments?

Key findings

• Delayed-Q outperforms both state-augmented and oblivious Q-learning methods in 39 out of 42 experiments across tabular, physical, and Atari environments.
• In the Atari environment 'Enduro', Delayed-Q achieves 33±2 points at m=25, while Augmented-Q and Oblivious-Q achieve only 29±4 and 0.5±0.2 respectively.
• On 'MsPacman', Delayed-Q attains 1354±86 at m=5, significantly outperforming Augmented-Q (1083±60) and Oblivious-Q (1319±35).
• In 'Zaxxon' at m=25, Delayed-Q scores 1418±148, far exceeding Oblivious-Q (605±66) and Augmented-Q (431±77).
• The RNN-based policy, which is unaware of the delay, fails to improve performance over oblivious baselines, confirming that delay-awareness is essential.
• Delayed-Q converges quickly and stably even at m=25, while state-augmented methods diverge or fail due to exponential state-space growth.