[Paper Review] On Online Learning in Kernelized Markov Decision Processes

Sayak Ray Chowdhury, Aditya Gopalan|arXiv (Cornell University)|May 21, 2018

Advanced Bandit Algorithms Research41 references4 citations

TL;DR

This paper proposes GP-UCRL and posterior sampling algorithms for online reinforcement learning in continuous-state and continuous-action Markov decision processes (MDPs) using Gaussian process priors to model smooth transition and reward functions. It establishes sublinear regret bounds that depend on novel kernel-dependent information gain metrics, generalizing kernel bandit results to MDPs and providing the first frequentist regret guarantees for nonparametric kernelized MDPs.

ABSTRACT

We consider online learning for minimizing regret in unknown, episodic Markov decision processes (MDPs) with continuous states and actions. We develop variants of the UCRL and posterior sampling algorithms that employ nonparametric Gaussian process priors to generalize across the state and action spaces. When the transition and reward functions of the true MDP are members of the associated Reproducing Kernel Hilbert Spaces of functions induced by symmetric psd kernels (frequentist setting), we show that the algorithms enjoy sublinear regret bounds. The bounds are in terms of explicit structural parameters of the kernels, namely a novel generalization of the information gain metric from kernelized bandit, and highlight the influence of transition and reward function structure on the learning performance. Our results are applicable to multidimensional state and action spaces with composite kernel structures, and generalize results from the literature on kernelized bandits, and the adaptive control of parametric linear dynamical systems with quadratic costs.

Motivation & Objective

• To develop online learning algorithms for episodic MDPs with continuous states and actions where dynamics and rewards are smooth and belong to Reproducing Kernel Hilbert Spaces (RKHS).
• To extend kernelized bandit results—specifically information gain metrics—to the temporal and state-dependent setting of MDPs.
• To provide finite-time, high-probability regret bounds for online RL in kernelized MDPs under a frequentist setting.
• To generalize existing results on parametric LQR and UCRL/PSRL to nonparametric, infinite-dimensional, kernel-structured MDPs.
• To leverage concentration inequalities for RKHS-valued martingales to derive theoretical guarantees.

Proposed method

• Propose GP-UCRL and posterior sampling (PSRL) variants that use Gaussian process priors to model unknown transition and reward functions in continuous MDPs.
• Model the true transition and reward functions as elements of RKHSs induced by symmetric positive definite kernels, enabling nonparametric generalization across states and actions.
• Define a novel generalization of information gain—γT(R) and γmT(P)—for kernelized MDPs, capturing the maximum information gained about the true dynamics and rewards after T interactions.
• Use concentration bounds for RKHS-valued martingales to control estimation error in both value function and policy regret, relying on posterior variance and kernel structure.
• Construct high-probability confidence sets for the true reward and transition functions using GP posterior variance and confidence width parameters βR,l and βP,l.
• Derive regret bounds by decomposing the difference between optimal and learned policy values into estimation errors in reward and transition functions, then bounding each using kernel-specific information gain terms.

Experimental results

Research questions

• RQ1 Can online reinforcement learning in continuous-state and continuous-action MDPs achieve sublinear regret when the true transition and reward functions are smooth and belong to an RKHS?
• RQ2 How does the choice of kernel influence the regret performance in kernelized MDPs, and can this be quantified via a generalized information gain metric?
• RQ3 Can the theoretical framework of kernelized bandits be extended to the sequential decision-making setting of MDPs with state evolution and finite-horizon planning?
• RQ4 What is the finite-time regret bound for GP-UCRL and posterior sampling in kernelized MDPs, and how does it depend on structural properties of the kernel?
• RQ5 Can concentration inequalities for RKHS-valued processes be leveraged to derive high-probability regret bounds in nonparametric MDPs?

Key findings

• The regret of GP-UCRL is bounded with high probability by Õ(γT(R) + γmT(P))√T, where γT(R) and γmT(P) are novel information gain metrics for reward and transition functions.
• For the squared exponential kernel, the information gain terms γT(R) and γmT(P) scale as polylog(T), leading to regret bounds of Õ(√T log^d T) for d-dimensional state and action spaces.
• The posterior sampling algorithm (PSRL) achieves a Bayes regret bound of O(αR,τ exp(γH−1(kR,Z))√(γT(kR,Z)T)) + O(αP,τ exp(γmH−1(kP,Z̃))√(γmT(kP,Z̃)T)) with high probability.
• The regret bounds explicitly depend on the kernel structure through the information gain, showing that smoother kernels (e.g., squared exponential) lead to better regret performance.
• The analysis generalizes kernel bandit results to MDPs and extends parametric LQR regret bounds to nonparametric, kernel-based MDPs with infinite-dimensional function spaces.
• The results are the first frequentist regret guarantees for online learning in kernelized MDPs, establishing a theoretical foundation for nonparametric RL with smoothness assumptions.