[Paper Review] Active Exploration in Markov Decision Processes

Jean Tarbouriech, Alessandro Lazaric|arXiv (Cornell University)|Feb 28, 2019

Advanced Bandit Algorithms Research22 citations

TL;DR

This paper introduces active exploration in Markov Decision Processes (MDPs), where an agent must strategically navigate to estimate state means accurately under unknown noise levels. By combining Frank-Wolfe UCB with a novel policy improvement heuristic, the proposed FW-AME algorithm achieves a regret rate of $\widetilde{O}(t^{-1/3})$, demonstrating that active exploration in MDPs is significantly more complex than in multi-armed bandits due to the need for controlled exploration and mixing policies.

ABSTRACT

We introduce the active exploration problem in Markov decision processes (MDPs). Each state of the MDP is characterized by a random value and the learner should gather samples to estimate the mean value of each state as accurately as possible. Similarly to active exploration in multi-armed bandit (MAB), states may have different levels of noise, so that the higher the noise, the more samples are needed. As the noise level is initially unknown, we need to trade off the exploration of the environment to estimate the noise and the exploitation of these estimates to compute a policy maximizing the accuracy of the mean predictions. We introduce a novel learning algorithm to solve this problem showing that active exploration in MDPs may be significantly more difficult than in MAB. We also derive a heuristic procedure to mitigate the negative effect of slowly mixing policies. Finally, we validate our findings on simple numerical simulations.

Motivation & Objective

• To formalize the active exploration problem in MDPs, where the goal is to estimate state means accurately under unknown and heterogeneous noise levels.
• To address the challenge of balancing exploration for noise estimation and exploitation for accurate mean prediction in a constrained MDP setting.
• To develop a learning algorithm that achieves vanishing regret under known MDP dynamics and ergodicity.
• To mitigate the negative impact of slowly mixing policies on estimation accuracy through a heuristic convex optimization procedure.
• To validate theoretical findings via numerical simulations on synthetic Garnet MDPs with controlled mixing properties.

Proposed method

• The FW-AME algorithm extends Frank-Wolfe UCB to MDPs by integrating policy optimization with active exploration, using a confidence-bound-based selection of actions.
• The algorithm maintains empirical estimates of state variances and uses them to guide exploration toward high-noise states requiring more samples.
• A novel heuristic based on FMH-SDP (Fast Mixing Heuristic - Semidefinite Programming) is introduced to compute faster-mixing, reversible policies that improve estimation stability.
• The regret analysis incorporates error decomposition into two components: estimation error from empirical policy distribution and approximation error from the FMH-SDP relaxation.
• The algorithm operates in episodes, with policy updates based on accumulated observations, and uses a confidence interval that shrinks over time to ensure convergence.
• Theoretical analysis shows that the regret scales as $\widetilde{O}(t^{-1/3})$, which is the first such rate for active exploration in MDPs under known dynamics.

Experimental results

Research questions

• RQ1 How does active exploration in MDPs differ fundamentally from active exploration in multi-armed bandits due to the need for state transitions?
• RQ2 Can a regret-minimizing algorithm be designed for active exploration in MDPs when dynamics are known and the MDP is ergodic?
• RQ3 What is the impact of slowly mixing policies on the accuracy of mean estimation, and how can this be mitigated?
• RQ4 Can a convex optimization framework be used to compute faster-mixing policies that improve estimation performance?
• RQ5 What is the optimal regret rate achievable in active exploration for MDPs with unknown state variances?

Key findings

• The FW-AME algorithm achieves a regret rate of $\widetilde{O}(t^{-1/3})$, which is the first such rate established for active exploration in MDPs with known dynamics.
• The regret analysis shows that the error in policy estimation is bounded by $O(1/\sqrt{\gamma(\widehat{\psi})\tau_k})$, where $\gamma$ measures mixing speed and $\tau_k$ is the episode length.
• The FMH-SDP heuristic reduces the approximation error $\delta_{\tau_k}$ by $O(1/\sqrt{\tau_k})$, maintaining the same regret order while improving mixing properties.
• Numerical simulations on Garnet MDPs confirm that the algorithm adapts to state-specific noise levels and achieves better estimation accuracy than baseline methods.
• The exponent $\theta = 1/3$ in the regret bound $\widetilde{O}(1/t^{1/3})$ is maximized when the episode length $m=3$, indicating optimal trade-off between exploration and exploitation.
• The method is robust to unknown state variances, as the algorithm dynamically adjusts sampling based on estimated noise, improving accuracy in high-variance states.