[Paper Review] Delays in Reinforcement Learning

Pierre Liotet|arXiv (Cornell University)|Sep 20, 2023

Complex Systems and Decision MakingDecision Sciences3 citations

TL;DR

This paper proposes a formal framework for modeling and analyzing delays in reinforcement learning using delayed Markov decision processes (MDPs). It introduces a continuous-time abstraction where actions are applied after a delay Δ, and establishes equivalence between delayed and standard MDPs via a transformed MDP with augmented state-action histories. The key contribution is proving that optimal policies in the delayed MDP correspond to optimal policies in the original MDP, enabling standard RL algorithms to be applied to delayed environments with theoretical guarantees.

ABSTRACT

Delays are inherent to most dynamical systems. Besides shifting the process in time, they can significantly affect their performance. For this reason, it is usually valuable to study the delay and account for it. Because they are dynamical systems, it is of no surprise that sequential decision-making problems such as Markov decision processes (MDP) can also be affected by delays. These processes are the foundational framework of reinforcement learning (RL), a paradigm whose goal is to create artificial agents capable of learning to maximise their utility by interacting with their environment. RL has achieved strong, sometimes astonishing, empirical results, but delays are seldom explicitly accounted for. The understanding of the impact of delay on the MDP is limited. In this dissertation, we propose to study the delay in the agent's observation of the state of the environment or in the execution of the agent's actions. We will repeatedly change our point of view on the problem to reveal some of its structure and peculiarities. A wide spectrum of delays will be considered, and potential solutions will be presented. This dissertation also aims to draw links between celebrated frameworks of the RL literature and the one of delays.

Motivation & Objective

• To formalize the impact of delays in reinforcement learning, particularly action delays, in a mathematically rigorous way.
• To address the challenge that standard RL algorithms fail in environments with delayed actions or observations.
• To establish a theoretical equivalence between a delayed MDP and a standard MDP through state augmentation.
• To enable the application of standard RL algorithms to delayed environments by transforming the problem into an equivalent standard MDP.
• To provide theoretical guarantees on performance equivalence between policies in the original and transformed MDPs.

Proposed method

• Introduces a continuous-time delayed MDP model where actions are applied after a delay Δ, using a state transition kernel that accounts for the delay.
• Defines a transformed MDP, denoted as $\widetilde{\mathcal{M}}$, with an augmented state space that includes past actions to encode the delay.
• Uses a state-action occupancy measure $\mu^\pi_\gamma$ to represent the expected discounted time spent in each state-action pair, enabling performance analysis.
• Constructs a transition kernel $\widetilde{p}(x'|x,a)$ that maps the current augmented state $x = (s, a_1, \dots, a_{\lceil\Delta\rceil})$ to the next state, incorporating the delay through a sequence of deterministic transitions.
• Defines a delayed reward function $\widetilde{r}(x,a)$ as the expected reward from the original MDP, conditioned on the delayed state and action.
• Proves that the optimal policy in the transformed MDP $\widetilde{\mathcal{M}}$ corresponds to the optimal policy in the original MDP $\mathcal{M}$, ensuring theoretical equivalence.

Experimental results

Research questions

• RQ1 How can delays in action execution be formally modeled within the framework of Markov decision processes?
• RQ2 What is the relationship between a delayed MDP and a standard MDP in terms of optimal policies and value functions?
• RQ3 Can a delayed MDP be transformed into an equivalent standard MDP such that standard RL algorithms can be applied without loss of optimality?
• RQ4 What conditions ensure that the performance of a policy in the delayed MDP matches its performance in the original MDP?
• RQ5 How does the delay affect the state-action occupancy measure and the expected return in the MDP?

Key findings

• The optimal policy in the transformed MDP $\widetilde{\mathcal{M}}$ is equivalent to the optimal policy in the original MDP $\mathcal{M}$, ensuring that standard RL algorithms can be applied to delayed environments with no loss of optimality.
• The state-action occupancy measure $\mu^\pi_\gamma$ in the original MDP can be recovered from the transformed MDP, preserving the expected discounted return.
• The transformed MDP $\widetilde{\mathcal{M}}$ maintains the same expected discounted return $J^\pi_\gamma$ as the original MDP, as shown by the identity $J^\pi_\gamma = \int_{\mathcal{S},\mathcal{A}} r(s,a) \, d\mu^\pi_\gamma(ds,da)$.
• For any delay $\Delta \in \mathbb{R}_{\geq 0}$, the Wasserstein distance between the delayed transition distribution $b_\Delta(\cdot|s,a)$ and the Dirac delta at $s$ is bounded by $\Delta L_T$, ensuring stability under small delays.
• The framework allows for the use of standard RL algorithms in environments with action delays by transforming the problem into an equivalent standard MDP with augmented states.
• The theoretical equivalence between the original and transformed MDPs holds regardless of whether $\Delta$ is an integer or not, enabling a unified treatment of delays.