[Paper Review] Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mapping

Dongruo Zhou, Jiafan He+1 more|arXiv (Cornell University)|Jun 23, 2020

Reinforcement Learning in Robotics37 references30 citations

TL;DR

The paper introduces the Upper-Confidence Linear Kernel Reinforcement Learning (UCLK) algorithm for discounted MDPs with linear kernel transitions, achieving a regret of ern~$ ilde{O}(d\sqrt{T}/(1-\gamma)^2)$, and proves a matching lower bound showing near-optimality.

ABSTRACT

Modern tasks in reinforcement learning have large state and action spaces. To deal with them efficiently, one often uses predefined feature mapping to represent states and actions in a low-dimensional space. In this paper, we study reinforcement learning for discounted Markov Decision Processes (MDPs), where the transition kernel can be parameterized as a linear function of certain feature mapping. We propose a novel algorithm that makes use of the feature mapping and obtains a $ ilde O(d\sqrt{T}/(1-γ)^2)$ regret, where $d$ is the dimension of the feature space, $T$ is the time horizon and $γ$ is the discount factor of the MDP. To the best of our knowledge, this is the first polynomial regret bound without accessing the generative model or making strong assumptions such as ergodicity of the MDP. By constructing a special class of MDPs, we also show that for any algorithms, the regret is lower bounded by $Ω(d\sqrt{T}/(1-γ)^{1.5})$. Our upper and lower bound results together suggest that the proposed reinforcement learning algorithm is near-optimal up to a $(1-γ)^{-0.5}$ factor.

Motivation & Objective

• Motivate design of RL algorithms that scale to large state/action spaces via feature mappings in discounted MDPs.
• Study a linear kernel MDP where transition probabilities are linear in a known feature map and an unknown parameter.
• Develop an online algorithm that learns a near-optimal value function without a generative model or ergodicity assumptions.
• Provide upper and lower regret bounds to establish optimality of the proposed approach within the linear kernel framework.

Proposed method

• Introduce the Upper-Confidence Linear Kernel Reinforcement Learning (UCLK) algorithm that operates in epochs with adaptive length.
• Formulate a regularized least-squares estimator for the unknown parameter theta* in each epoch.
• Construct a confidence set \mathcal{C}_k around the estimator and perform Extended Value Iteration (EVI) over \mathcal{C}_k \cap \mathcal{B} to obtain Q_k.
• Use Extended Value Iteration to compute Q_k via maximization over feasible parameters within the confidence set, yielding a near-optimal policy.
• Update the statistical matrices \Sigma_t and b_t online and switch epochs when the determinant of \Sigma_t doubles, controlling switching error.
• Prove that the regret is \tilde{O}(d sqrt(T)/(1-\gamma)^2) with high probability under appropriate choices of \beta and U.

Experimental results

Research questions

• RQ1 Can we design provably efficient RL algorithms with feature mappings for discounted MDPs without relying on a generative model or strong ergodicity assumptions?
• RQ2 What are the regret guarantees for a linear kernel MDP when using a confidence-augmented planning method like UCLK?
• RQ3 How close can upper and lower bounds get to characterize the optimal dependence on feature dimension d, horizon T, and discount factor \gamma?

Key findings

• UCLK attains a regret bound of \tilde{O}(d sqrt(T)/(1-\gamma)^2) for learning the optimal value function in linear kernel MDPs.
• The regret bound is independent of the state and action space cardinalities, highlighting scalability to large problems.
• A matching (problem-dependent) lower bound shows regret at least \Omega(d sqrt(T)/(1-\gamma)^{1.5}) for any algorithm, up to a (1-\gamma)^{-0.5} factor.
• The upper and lower bounds together imply near-optimality of UCLK with respect to d and T, and near-optimality in \gamma up to a (1-\gamma)^{-0.5} factor.
• The results apply to a broad class of MDPs that can be represented as linear combinations of base models, unifying several prior models under the linear kernel MDP framework.