[Paper Review] Optimism in Reinforcement Learning with Generalized Linear Function Approximation

Yining Wang, Ruosong Wang+2 more|arXiv (Cornell University)|Dec 9, 2019

Advanced Bandit Algorithms Research29 references54 citations

TL;DR

The paper introduces a provably efficient RL algorithm that uses generalized linear function approximation with an optimistic Bellman backup, achieving a sublinear regret bound of tilde O(H sqrt(d^3 T)).

ABSTRACT

We design a new provably efficient algorithm for episodic reinforcement learning with generalized linear function approximation. We analyze the algorithm under a new expressivity assumption that we call "optimistic closure," which is strictly weaker than assumptions from prior analyses for the linear setting. With optimistic closure, we prove that our algorithm enjoys a regret bound of $ ilde{O}(\sqrt{d^3 T})$ where $d$ is the dimensionality of the state-action features and $T$ is the number of episodes. This is the first statistically and computationally efficient algorithm for reinforcement learning with generalized linear functions.

Motivation & Objective

• Motivate learning in episodic MDPs with infinite state spaces where function approximation is needed.
• Propose a simple GLM-based Q-learning variant that remains sample-efficient via optimism.
• Introduce an expressivity assumption called optimistic closure that weakens prior dynamics-based requirements.
• Establish regret guarantees for the proposed algorithm under the new assumption.

Proposed method

• Use a GLM class to approximate the optimal Q-function with a known feature map and link function.
• Maintain optimistic Q-value estimates via a backward dynamic programming step and an optimism bonus.
• Update Q-function parameters by solving constrained least-squares problems at each time step.
• Define an optimistic Q-function via a min{1, f(phi(s,a)^T hat{theta}) + gamma ||phi(s,a)||_{Lambda^{-1}}} bound.
• Deploy a greedy policy with respect to the current optimistic Q-function and iterate over episodes.

Experimental results

Research questions

• RQ1 Can optimistic closure provide sufficient expressivity to enable provably efficient RL with generalized linear function approximation?
• RQ2 How does GLM-based optimism compare to linear MDP assumptions in terms of guarantees and practicality?
• RQ3 What regret bounds are achievable for episodic RL when using GLMs under optimistic closure?
• RQ4 Does the proposed approach extend beyond linear models to more general function classes while remaining computationally feasible?

Key findings

• Proposes LSVI-UCB, a GLM-based dynamic programming algorithm for episodic RL.
• Proves a regret bound of tilde O(H sqrt(d^3 T)) under optimistic closure with mild regularity conditions.
• Shows that optimistic closure implies realizability and that it is strictly weaker than the linear MDP assumption.
• Demonstrates that the method generalizes prior tabular and linear-function results, enabling GLM-based RL.
• Corollary: in the tabular setting with standard features, the regret bound becomes tilde O(H |S|^3 |A|^3 T) (grounded in the paper's framing).