[Paper Review] Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizon

Zihan Zhang, Xiangyang Ji+1 more|arXiv (Cornell University)|Sep 28, 2020

Advanced Bandit Algorithms Research46 references30 citations

TL;DR

The paper introduces MVP, a monotonic value propagation algorithm with a Bernstein-type bonus that achieves near-bandit sample complexity for episodic RL, with regret near the contextual bandit lower bound and logarithmic dependence on the horizon H.

ABSTRACT

Episodic reinforcement learning and contextual bandits are two widely studied sequential decision-making problems. Episodic reinforcement learning generalizes contextual bandits and is often perceived to be more difficult due to long planning horizon and unknown state-dependent transitions. The current paper shows that the long planning horizon and the unknown state-dependent transitions (at most) pose little additional difficulty on sample complexity. We consider the episodic reinforcement learning with $S$ states, $A$ actions, planning horizon $H$, total reward bounded by $1$, and the agent plays for $K$ episodes. We propose a new algorithm, extbf{M}onotonic extbf{V}alue extbf{P}ropagation (MVP), which relies on a new Bernstein-type bonus. Compared to existing bonus constructions, the new bonus is tighter since it is based on a well-designed monotonic value function. In particular, the \emph{constants} in the bonus should be subtly setting to ensure optimism and monotonicity. We show MVP enjoys an $O\left(\left(\sqrt{SAK} + S^2A ight) \poly\log \left(SAHK ight) ight)$ regret, approaching the $Ω\left(\sqrt{SAK} ight)$ lower bound of \emph{contextual bandits} up to logarithmic terms. Notably, this result 1) \emph{exponentially} improves the state-of-the-art polynomial-time algorithms by Dann et al. [2019] and Zanette et al. [2019] in terms of the dependency on $H$, and 2) \emph{exponentially} improves the running time in [Wang et al. 2020] and significantly improves the dependency on $S$, $A$ and $K$ in sample complexity.

Motivation & Objective

• Assess whether episodic RL can match contextual bandits in sample efficiency under bounded total reward.
• Develop a computationally efficient algorithm with logarithmic dependence on the horizon H.
• Introduce a Bernstein-type exploration bonus ensuring optimism and monotonic value propagation.
• Provide theoretical guarantees: regret and PAC bounds close to CB lower bounds up to logarithmic factors.

Proposed method

• Propose Monotonic Value Propagation (MVP), a UCB-based model-based algorithm with a new Bernstein-type bonus.
• Define Q_h(s,a) = hat{r}(s,a) + hat{P}_{s,a} V_{h+1} + b_h(s,a) and ensure optimism.
• Introduce a monotonicity property: Q_h(V_{h+1}) is increasing in V_{h+1}, enabling horizon-independent propagation of optimism.
• Use a trigger-based doubling update framework to update rewards and transitions and propagate estimates across episodes.
• Derive a recursive variance-bound technique to control total variance across the horizon via higher-order moment expansion.
• Establish regret and PAC bounds: Regret(K) = O((sqrt(SAK) + S^2A) polylog(SAHK/δ)) and PAC-RL bound O((SA/ε^2) + (S^2A/ε)) polylog factors.

Experimental results

Research questions

• RQ1 Does episodic RL incur additional sample complexity over contextual bandits under bounded total reward?
• RQ2 Can we design a computationally efficient algorithm with regret and PAC guarantees that match CB lower bounds up to logarithmic factors?
• RQ3 Can horizon dependence be made logarithmic rather than polynomial while maintaining near-optimal sample complexity?
• RQ4 What exploration bonus structure ensures optimism and monotonic value propagation across the horizon?

Key findings

• MVP achieves regret O((sqrt(SAK) + S^2A) polylog(SAHK)) with high probability.
• A standard reduction shows an ε-suboptimal policy can be found in O((SA/ε^2) + (S^2A/ε)) polylog(SAH/εδ) episodes.
• The algorithm is computationally efficient (polynomial time) and achieves logarithmic dependence on H in its bounds.
• A new Bernstein-type bonus, combined with a monotonicity property, tightens the optimism required for near-bandit performance.
• The results substantially close the gap between RL and CB, and improve upon previous polynomial-time algorithms in H-dependency and S,A,K scaling.