[Paper Review] Optimal policy evaluation using kernel-based temporal difference methods

Yaqi Duan, Mengdi Wang+1 more|arXiv (Cornell University)|Sep 24, 2021

Statistical Methods and Inference49 references4 citations

TL;DR

This paper proposes a kernel-based temporal difference method for optimal policy evaluation in infinite-horizon discounted Markov reward processes, using a regularized kernel least-squares temporal difference estimator that reduces to solving a linear system with kernel matrices. The key contribution is a non-asymptotic $L^2(\mu)$ error bound with explicit dependence on kernel eigenvalues and Bellman residual variance, showing the rate is minimax optimal and revealing that scaling with the effective horizon $H = (1 - \gamma)^{-1}$ can be sub-cubic depending on the kernel and problem instance.

ABSTRACT

We study methods based on reproducing kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process (MRP). We study a regularized form of the kernel least-squares temporal difference (LSTD) estimate; in the population limit of infinite data, it corresponds to the fixed point of a projected Bellman operator defined by the associated reproducing kernel Hilbert space. The estimator itself is obtained by computing the projected fixed point induced by a regularized version of the empirical operator; due to the underlying kernel structure, this reduces to solving a linear system involving kernel matrices. We analyze the error of this estimate in the $L^2(μ)$-norm, where $μ$ denotes the stationary distribution of the underlying Markov chain. Our analysis imposes no assumptions on the transition operator of the Markov chain, but rather only conditions on the reward function and population-level kernel LSTD solutions. We use empirical process theory techniques to derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator, as well as the instance-dependent variance of the Bellman residual error. In addition, we prove minimax lower bounds over sub-classes of MRPs, which shows that our rate is optimal in terms of the sample size $n$ and the effective horizon $H = (1 - γ)^{-1}$. Whereas existing worst-case theory predicts cubic scaling ($H^3$) in the effective horizon, our theory reveals that there is in fact a much wider range of scalings, depending on the kernel, the stationary distribution, and the variance of the Bellman residual error. Notably, it is only parametric and near-parametric problems that can ever achieve the worst-case cubic scaling.

Motivation & Objective

• To provide a sharp statistical characterization of kernel-based policy evaluation methods in infinite-horizon Markov reward processes.
• To analyze the estimation error of a regularized kernel least-squares temporal difference estimator in the $L^2(\mu)$-norm, where $\mu$ is the stationary distribution.
• To derive non-asymptotic error bounds that explicitly depend on kernel eigenvalues and the variance of the Bellman residual.
• To establish minimax lower bounds to show the optimality of the derived error rate in terms of sample size $n$ and effective horizon $H = (1 - \gamma)^{-1}$.
• To reveal that worst-case cubic scaling in $H$ ($H^3$) only occurs in parametric or near-parametric settings, while richer kernels allow for significantly better scaling.

Proposed method

• The method employs a regularized form of the kernel least-squares temporal difference (LSTD) estimate, which corresponds to the fixed point of a projected Bellman operator in a reproducing kernel Hilbert space (RKHS).
• The empirical estimator is computed by solving a linear system involving kernel matrices, leveraging the representer theorem to ensure computational tractability.
• The analysis uses empirical process theory to bound the $L^2(\mu)$ estimation error between the empirical and population-level kernel LSTD solutions.
• The error bound is derived under minimal assumptions—only on the reward function and the population solution—without requiring structural assumptions on the Markov chain's transition operator.
• The method incorporates the eigenstructure of the kernel integral operator and the variance of the Bellman residual to refine the error dependence on problem-specific features.
• Minimax lower bounds are derived over sub-classes of MRPs to establish the optimality of the proposed upper bound.

Experimental results

Research questions

• RQ1 What is the non-asymptotic $L^2(\mu)$ estimation error of the regularized kernel LSTD estimator in infinite-horizon MRP settings?
• RQ2 How does the error depend on the eigenvalues of the kernel operator and the variance of the Bellman residual?
• RQ3 Can the worst-case cubic scaling in the effective horizon $H = (1 - \gamma)^{-1}$ be avoided, and if so, under what conditions?
• RQ4 Is the proposed error bound minimax optimal over relevant classes of MRP problems?
• RQ5 Under what conditions does the error scaling transition from sub-cubic to parametric or near-parametric rates?

Key findings

• The paper establishes a non-asymptotic upper bound on the $L^2(\mu)$ estimation error that explicitly depends on the eigenvalues of the kernel operator and the variance of the Bellman residual.
• The upper bound reveals that the scaling with the effective horizon $H = (1 - \gamma)^{-1}$ is not universally cubic; instead, it can be significantly better depending on the kernel and problem instance.
• The proposed error bound is minimax optimal, as confirmed by matching lower bounds derived over sub-classes of MRPs.
• Cubic scaling in $H$ ($H^3$) occurs only in parametric or near-parametric problems, indicating that richer kernel classes can avoid this worst-case behavior.
• The analysis shows that the estimation error is controlled by the interplay between the eigengap of the kernel operator and the concentration of the stationary distribution, with tighter bounds achievable when eigenvalues decay slowly.
• The method achieves a constant-order $\ell_1$-norm of the perturbed eigenvector, ensuring stability in the kernel-based representation under perturbations.