[Paper Review] Exploration Bonus for Regret Minimization in Undiscounted Discrete and Continuous Markov Decision Processes

Jian Qian, Ronan Fruit+2 more|arXiv (Cornell University)|Dec 11, 2018

Advanced Bandit Algorithms Research13 references4 citations

TL;DR

This paper introduces SCAL+ and C-SCAL+, efficient exploration algorithms for undiscounted Markov Decision Processes (MDPs) that use exploration bonuses to achieve sublinear regret. By directly planning on the empirically estimated MDP with a bonus term, they match the regret bounds of prior optimistic algorithms (e.g., SCAL, UCCRL) while significantly reducing computational complexity, making them the first implementable algorithms with regret guarantees in continuous-state undiscounted MDPs.

ABSTRACT

We introduce and analyse two algorithms for exploration-exploitation in discrete and continuous Markov Decision Processes (MDPs) based on exploration bonuses. SCAL$^+$ is a variant of SCAL (Fruit et al., 2018) that performs efficient exploration-exploitation in any unknown weakly-communicating MDP for which an upper bound C on the span of the optimal bias function is known. For an MDP with $S$ states, $A$ actions and $Γ\leq S$ possible next states, we prove that SCAL$^+$ achieves the same theoretical guarantees as SCAL (i.e., a high probability regret bound of $\widetilde{O}(C\sqrt{ΓSAT})$), with a much smaller computational complexity. Similarly, C-SCAL$^+$ exploits an exploration bonus to achieve sublinear regret in any undiscounted MDP with continuous state space. We show that C-SCAL$^+$ achieves the same regret bound as UCCRL (Ortner and Ryabko, 2012) while being the first implementable algorithm with regret guarantees in this setting. While optimistic algorithms such as UCRL, SCAL or UCCRL maintain a high-confidence set of plausible MDPs around the true unknown MDP, SCAL$^+$ and C-SCAL$^+$ leverage on an exploration bonus to directly plan on the empirically estimated MDP, thus being more computationally efficient.

Motivation & Objective

• To develop efficient exploration-exploitation algorithms for undiscounted MDPs with provable regret guarantees.
• To address the computational inefficiency of existing optimistic algorithms that maintain high-confidence sets of plausible MDPs.
• To extend the use of exploration bonuses—common in bandits and discounted MDPs—to undiscounted, general infinite-horizon MDPs.
• To provide the first implementable algorithm with regret guarantees in continuous-state undiscounted MDPs.
• To match the theoretical regret bounds of SCAL and UCCRL while drastically reducing computational complexity.

Proposed method

• SCAL+ uses an exploration bonus added to the empirical reward in a discrete MDP, enabling direct planning on the estimated MDP instead of maintaining a high-confidence set of plausible MDPs.
• The exploration bonus in SCAL+ is derived from the span of the optimal bias function and scales as $\widetilde{O}(c / \sqrt{N(s,a)})$, where $c$ is an upper bound on the bias span.
• C-SCAL+ extends the exploration bonus approach to continuous-state MDPs, using function approximation and a bonus term to ensure sublinear regret.
• Both algorithms rely on the optimism in the face of uncertainty (OFU) principle but avoid the computational overhead of maintaining a set of plausible MDPs.
• The algorithms use episode-based learning with stopping times to ensure sufficient exploration and maintain high-probability confidence in the estimated MDP.
• Theoretical analysis proves that SCAL+ achieves $\widetilde{O}(c\sqrt{\Gamma SAT})$ regret with much lower computational cost than SCAL, and C-SCAL+ matches the regret bound of UCCRL in continuous MDPs.

Experimental results

Research questions

• RQ1 Can exploration bonuses be effectively used in undiscounted MDPs to achieve sublinear regret without maintaining a high-confidence set of MDPs?
• RQ2 Can the computational complexity of optimistic algorithms like SCAL and UCCRL be reduced while preserving their regret guarantees?
• RQ3 Is it possible to design an implementable algorithm with regret guarantees in continuous-state undiscounted MDPs using exploration bonuses?
• RQ4 How does the use of an exploration bonus based on the bias span compare to confidence-set-based methods in terms of regret and efficiency?
• RQ5 Can the OFU principle be applied efficiently in undiscounted MDPs using direct planning on the estimated MDP with a bonus?

Key findings

• SCAL+ achieves the same regret bound as SCAL—$\widetilde{O}(c\sqrt{\Gamma SAT})$—but with significantly reduced computational complexity by avoiding high-confidence set maintenance.
• C-SCAL+ achieves the same regret bound as UCCRL in continuous-state undiscounted MDPs, making it the first implementable algorithm with such guarantees.
• The exploration bonus in SCAL+ scales as $\widetilde{O}(c / \sqrt{N(s,a)})$, where $c$ is an upper bound on the span of the optimal bias function.
• The use of direct planning on the estimated MDP with a bonus term enables efficient computation while maintaining theoretical regret guarantees.
• Theoretical analysis confirms that both algorithms maintain high-probability confidence in the estimated MDP through episode-based learning and stopping times.
• The results demonstrate that exploration bonuses can be effectively extended to undiscounted MDPs, bridging a gap between bandit-style methods and full RL algorithms.