[Paper Review] Learning Adversarial MDPs with Bandit Feedback and Unknown Transition

Chi Jin, Tiancheng Jin+3 more|arXiv (Cornell University)|Dec 3, 2019

Advanced Bandit Algorithms ResearchDecision Sciences34 references34 citations

TL;DR

The paper introduces UOB-REPS, an efficient algorithm for episodic adversarial MDPs with unknown transitions and bandit feedback, achieving near-√T regret and matching full-information bounds up to polylog factors.

ABSTRACT

We consider the problem of learning in episodic finite-horizon Markov decision processes with an unknown transition function, bandit feedback, and adversarial losses. We propose an efficient algorithm that achieves $\mathcal{ ilde{O}}(L|X|\sqrt{|A|T})$ regret with high probability, where $L$ is the horizon, $|X|$ is the number of states, $|A|$ is the number of actions, and $T$ is the number of episodes. To the best of our knowledge, our algorithm is the first to ensure $\mathcal{ ilde{O}}(\sqrt{T})$ regret in this challenging setting; in fact it achieves the same regret bound as (Rosenberg & Mansour, 2019a) that considers an easier setting with full-information feedback. Our key technical contributions are two-fold: a tighter confidence set for the transition function, and an optimistic loss estimator that is inversely weighted by an $ extit{upper occupancy bound}$.

Motivation & Objective

• Motivate learning in episodic finite-horizon MDPs with unknown transitions and adversarial losses.
• Develop an algorithm with tight confidence sets for transitions and optimistic loss estimation under bandit feedback.
• Achieve sublinear regret bounds comparable to full-information settings while handling bandit feedback.

Proposed method

• Model the problem via occupancy measures to convert policy learning into online linear optimization over the occupancy space.
• Construct a tighter confidence set for the transition function that bounds per-next-state probabilities.
• Introduce an optimistic, biased loss estimator using an upper occupancy bound computed from the confidence set and implicit exploration.
• Apply Online Mirror Descent over occupancy measures with a KL-divergence-based Bregman divergence to update policies.
• Compute the upper occupancy bounds efficiently via backward dynamic programming and a greedy optimization routine.

Experimental results

Research questions

• RQ1 Can we achieve sublinear regret in episodic adversarial MDPs with unknown transitions under bandit feedback?
• RQ2 How can confidence sets for transitions and optimistic loss estimators be combined to enable Online Mirror Descent in this setting?
• RQ3 What is the resulting regret bound and under what conditions does it hold with high probability?

Key findings

• Propose UOB-REPS, achieving regret R_T = O~(L|X|√(|A|T)) with high probability.
• Show the new per-(x,a,x′) confidence bounds tighten previous results and enable tighter analysis.
• Introduce upper occupancy bound-based loss estimators that remain tractable under unknown transitions.
• Demonstrate that the algorithm attains O~(√T) regret in this challenging bandit setting, matching the full-information bound up to polylog factors.
• Provide an efficient implementation pathway via backward DP for occupancy-bound computations and LP-style optimizations.