[Paper Review] Bilinear Classes: A Structural Framework for Provable Generalization in RL

Simon S. Du, Sham M. Kakade+5 more|arXiv (Cornell University)|Mar 19, 2021

Reinforcement Learning in Robotics41 references21 citations

TL;DR

This paper introduces Bilinear Classes, a novel structural framework that enables provable sample-efficient reinforcement learning through function approximation. By modeling the Bellman error as a bilinear form, the framework unifies existing efficient RL models and introduces new ones—like Linear $Q^*/V^*$ and RKHS-based models—while achieving polynomial sample complexity that depends on a supervised learning generalization error, extending to infinite-dimensional settings via information-theoretic quantities.

ABSTRACT

This work introduces Bilinear Classes, a new structural framework, which permit generalization in reinforcement learning in a wide variety of settings through the use of function approximation. The framework incorporates nearly all existing models in which a polynomial sample complexity is achievable, and, notably, also includes new models, such as the Linear $Q^*/V^*$ model in which both the optimal $Q$-function and the optimal $V$-function are linear in some known feature space. Our main result provides an RL algorithm which has polynomial sample complexity for Bilinear Classes; notably, this sample complexity is stated in terms of a reduction to the generalization error of an underlying supervised learning sub-problem. These bounds nearly match the best known sample complexity bounds for existing models. Furthermore, this framework also extends to the infinite dimensional (RKHS) setting: for the the Linear $Q^*/V^*$ model, linear MDPs, and linear mixture MDPs, we provide sample complexities that have no explicit dependence on the explicit feature dimension (which could be infinite), but instead depends only on information theoretic quantities.

Motivation & Objective

• To develop a general structural framework that captures a wide range of reinforcement learning models with provable sample efficiency.
• To unify existing efficient RL frameworks—such as Bellman rank, witness rank, and Bellman complete—under a single, more general condition.
• To extend sample efficiency to new model classes, including Linear $Q^*/V^*$ and RKHS-based MDPs, where prior frameworks fail.
• To provide sample complexity bounds that depend on a supervised learning generalization error, enabling transferable analysis across models.
• To extend finite-dimensional results to infinite-dimensional function spaces using information-theoretic measures like critical information gain.

Proposed method

• Propose Bilinear Classes as a structural condition where the Bellman error admits a bilinear form in two functions: one from the hypothesis class and one from a discriminant class.
• Design the BiLin-UCB algorithm, which uses a bilinear optimization framework to minimize Bellman error via confidence intervals derived from the bilinear structure.
• Reduce the sample complexity of RL to the generalization error of a supervised learning subproblem, enabling transfer of known generalization bounds.
• Introduce the critical information gain as a non-parametric complexity measure that replaces explicit feature dimension dependence in infinite-dimensional settings.
• Apply the framework to new models: Linear $Q^*/V^*$, RKHS Linear MDPs, and Linear Mixture MDPs, showing sample complexity independent of explicit feature dimension.
• Demonstrate that the framework subsumes prior frameworks (e.g., Bellman rank, witness rank) and extends to models not captured by them, such as factored MDPs and low occupancy complexity MDPs.

Experimental results

Research questions

• RQ1 Can a single structural framework unify existing provably sample-efficient RL models and extend to new ones?
• RQ2 Can the sample complexity of RL be reduced to a supervised learning generalization error in a way that applies across diverse function approximation settings?
• RQ3 Can the framework achieve sample efficiency in infinite-dimensional function spaces (e.g., RKHS) without explicit dependence on feature dimension?
• RQ4 Are there new model classes—such as Linear $Q^*/V^*$—that are provably learnable under this framework but not under prior ones?
• RQ5 Can the framework capture models like factored MDPs and low occupancy complexity MDPs, which are not covered by Bellman rank or witness rank?

Key findings

• The Bilinear Classes framework generalizes Bellman rank and witness rank, and subsumes all known model classes with polynomial sample complexity.
• The BiLin-UCB algorithm achieves polynomial sample complexity that depends only on the generalization error of a supervised learning subproblem.
• For Linear $Q^*/V^*$, Linear MDPs, and Linear Mixture MDPs, the framework provides sample complexity bounds that depend only on information-theoretic quantities like critical information gain, not on explicit feature dimension.
• The framework extends to infinite-dimensional RKHS settings, enabling non-parametric generalization in function approximation without explicit dimension dependence.
• The framework captures new models such as the Linear $Q^*/V^*$ model, which was not covered by prior frameworks and is provably learnable under this structure.
• Theoretical analysis shows that the framework includes models like factored MDPs and low occupancy complexity MDPs, which are not captured by Bellman rank or witness rank, and provides tighter sample complexity bounds than prior methods in these cases.