[Paper Review] Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound

Lin F. Yang, Mengdi Wang|arXiv (Cornell University)|May 24, 2019

Reinforcement Learning in RoboticsComputer Science95 citations

TL;DR

Introduces MatrixRL, an online RL algorithm that learns a low-dimensional transition core in feature space (and its kernelized version) to achieve near-optimal regret bounds that depend on the feature dimension rather than the state-action space size.

ABSTRACT

Exploration in reinforcement learning (RL) suffers from the curse of dimensionality when the state-action space is large. A common practice is to parameterize the high-dimensional value and policy functions using given features. However existing methods either have no theoretical guarantee or suffer a regret that is exponential in the planning horizon $H$. In this paper, we propose an online RL algorithm, namely the MatrixRL, that leverages ideas from linear bandit to learn a low-dimensional representation of the probability transition model while carefully balancing the exploitation-exploration tradeoff. We show that MatrixRL achieves a regret bound ${O}\\big(H^2d\\log T\\sqrt{T}\\big)$ where $d$ is the number of features. MatrixRL has an equivalent kernelized version, which is able to work with an arbitrary kernel Hilbert space without using explicit features. In this case, the kernelized MatrixRL satisfies a regret bound ${O}\\big(H^2\\widetilde{d}\\log T\\sqrt{T}\\big)$, where $\\widetilde{d}$ is the effective dimension of the kernel space. To our best knowledge, for RL using features or kernels, our results are the first regret bounds that are near-optimal in time $T$ and dimension $d$ (or $\\widetilde{d}$) and polynomial in the planning horizon $H$.

Motivation & Objective

• Address the curse of dimensionality in RL by leveraging feature representations for provable exploration efficiency.
• Develop an online RL algorithm that learns a low-dimensional transition core via regression and optimistic planning.
• Extend the approach to kernel spaces to handle infinite-dimensional feature representations.
• Provide regret bounds that scale with the number of features (or effective kernel dimension) rather than the size of the state-action space.

Proposed method

• Model the transition dynamics with a core matrix M* so that P(·|s,a)=φ(s,a)ᵀ M* ψ(·).
• Estimate M* using ridge regression to form M_n as in (2).
• Construct an optimistic Q-function via a matrix confidence ball B_n to balance exploration and exploitation (Eq. 4).
• Provide two regret bounds: Theorem 1 with a 2→1 matrix norm and Theorem 2 with a Frobenius-norm ball, yielding O(H² d^{3/2} √T) or O(H² d √T) regret respectively.
• Kernelize MatrixRL by replacing explicit features with kernels k_φ and k_ψ, yielding KernelMatrixRL with regret O(H² ẑ√T) where ẑ is the effective kernel dimension (Theorem 3).
• Offer practical implementation notes including closed-form confidence bounds and space complexity O(d²).

Experimental results

Research questions

• RQ1 Can feature-based representations enable efficient exploration in RL with regret bounds independent of the full state-action space?
• RQ2 How can one estimate a transition core using ridge regression and use a confidence ball to achieve optimistic RL in feature space?
• RQ3 Does a kernelized version extend these guarantees to infinite-dimensional feature spaces and what is the resulting regret?
• RQ4 What are the concrete regret bounds in terms of the feature dimension d (or effective kernel dimension ẑ) and horizon H?
• RQ5 Can the outlined approach be implemented efficiently in practice for high-dimensional problems?

Key findings

• MatrixRL achieves regret bounds of O(H² d log T √T) (Theorem 1) or O(H² d √T) under stronger regularity (Theorem 2).
• KernelMatrixRL achieves regret O(H² ẑ log T √T) with ẑ the effective kernel dimension (Theorem 3).
• Regret bounds are near-optimal in time T and dimension d (or ẑ) and polynomial in the planning horizon H, without dependence on the size of the state-action space.
• Algorithm operates in space O(d²) and updates in closed form per step.
• The kernelized version can handle infinite-dimensional spaces via kernels, maintaining sublinear regret in T with respect to the effective dimension.