[Paper Review] Navigating to the Best Policy in Markov Decision Processes

Aymen Al Marjani, Aurélien Garivier+1 more|arXiv (Cornell University)|Jun 5, 2021

Reinforcement Learning in Robotics40 references4 citations

TL;DR

This paper proposes MDP-NaS, the first algorithm for best policy identification in Markov Decision Processes under online navigation constraints, where transitions are observed sequentially rather than queried. It establishes a problem-dependent lower bound and proves instance-specific sample complexity with convergence rates dependent on MDP ergodicity, using a novel sampling rule mixing uniform and plug-in policies to track optimal state-action frequencies via an ergodic theorem for non-homogeneous Markov chains.

ABSTRACT

We investigate the classical active pure exploration problem in Markov Decision Processes, where the agent sequentially selects actions and, from the resulting system trajectory, aims at identifying the best policy as fast as possible. We propose a problem-dependent lower bound on the average number of steps required before a correct answer can be given with probability at least $1-δ$. We further provide the first algorithm with an instance-specific sample complexity in this setting. This algorithm addresses the general case of communicating MDPs; we also propose a variant with a reduced exploration rate (and hence faster convergence) under an additional ergodicity assumption. This work extends previous results relative to the \emph{generative setting}~\cite{pmlr-v139-marjani21a}, where the agent could at each step query the random outcome of any (state, action) pair. In contrast, we show here how to deal with the \emph{navigation constraints}, induced by the \emph{online setting}. Our analysis relies on an ergodic theorem for non-homogeneous Markov chains which we consider of wide interest in the analysis of Markov Decision Processes.

Motivation & Objective

• To address the best policy identification (BPI) problem in infinite-horizon discounted MDPs under online, sequential observation constraints, where actions determine the next state via natural transitions.
• To derive a problem-dependent lower bound on the expected stopping time for correct policy identification with probability at least $1 - \delta$.
• To design an algorithm with instance-specific sample complexity in the online setting, overcoming limitations of prior generative model assumptions.
• To analyze the minimal forced-exploration rate required for consistent estimation of the oracle policy under different MDP communication structures (e.g., ergodic vs. merely communicating).
• To establish theoretical guarantees using a novel ergodic theorem for non-homogeneous Markov chains to ensure convergence of sampling frequencies to the target oracle allocation.

Proposed method

• Adapt the information-theoretic lower bound from [MP21] to the online setting by restricting the algorithm's strategy set to $\Omega(\mathcal{M})$, reflecting navigation constraints.
• Propose MDP-NaS, a sampling rule that mixes the uniform policy with a plug-in estimate of the oracle policy to ensure convergence of state-action frequencies to the optimal allocation $\omega^\star$.
• Use a non-homogeneous Markov chain framework to prove that the sampling frequency $N_{sa}(t)/t$ converges to $\omega^\star$ under the proposed rule, relying on a new ergodic theorem for such chains.
• Establish that under ergodic MDPs, a slow exploration rate of $1/\sqrt{t}$ suffices for consistency, while for merely communicating MDPs, a slower rate of $t^{-1/(m+1)}$ is required, with $m$ related to the MDP's structure.
• Leverage perturbation bounds for Markov chains to relate deviations in transition kernels to deviations in stationary distributions, ensuring robustness of the plug-in policy estimator.
• Apply Poisson equation techniques to bound the sensitivity of stationary distributions to model perturbations, enabling stability analysis of the sampling rule.

Experimental results

Research questions

• RQ1 How does the sample complexity of best policy identification change when moving from a generative model to an online navigation setting?
• RQ2 What is the problem-dependent lower bound on the expected stopping time for identifying an $\varepsilon$-optimal policy with probability at least $1 - \delta$ in the online MDP setting?
• RQ3 Can a sampling rule be designed that ensures convergence of state-action sampling frequencies to the oracle allocation $\omega^\star$ under sequential, non-queriable transitions?
• RQ4 What is the minimal forced-exploration rate required to guarantee consistency of the plug-in policy estimator in communicating MDPs?
• RQ5 How does the ergodicity of the MDP influence the required exploration rate and convergence speed of the algorithm?

Key findings

• The paper establishes a problem-dependent lower bound on the expected stopping time for BPI in online MDPs, extending prior results from the generative model setting.
• MDP-NaS achieves instance-specific sample complexity in the asymptotic regime $\delta \to 0$, with the first such guarantee in the online navigation setting.
• Under ergodic MDPs, a forced-exploration rate as low as $1/\sqrt{t}$ is sufficient for consistency of the plug-in estimator, significantly faster than in the general communicating case.
• For general communicating MDPs, a slower exploration rate of $t^{-1/(m+1)}$ is required, where $m$ is a structural parameter of the MDP that can scale as $S-1$ in the worst case.
• The convergence of sampling frequencies to $\omega^\star$ is proven via a new ergodic theorem for non-homogeneous Markov chains, which is of independent interest in MDP analysis.
• The analysis shows that the sensitivity of stationary distributions to model perturbations is bounded by a constant $\kappa_{\mathcal{M}}$ depending only on the MDP, enabling robust policy estimation.