[Paper Review] What are the Statistical Limits of Offline RL with Linear Function Approximation?

Ruosong Wang, Dean P. Foster+1 more|arXiv (Cornell University)|Oct 22, 2020

Reinforcement Learning in Robotics63 references37 citations

TL;DR

The paper proves that under realizability with linear Q-functions and bounded feature coverage, offline RL requires exponential samples in the horizon to evaluate any policy; it analyzes why sample-efficient offline policy evaluation is impossible without stronger conditions.

ABSTRACT

Offline reinforcement learning seeks to utilize offline (observational) data to guide the learning of (causal) sequential decision making strategies. The hope is that offline reinforcement learning coupled with function approximation methods (to deal with the curse of dimensionality) can provide a means to help alleviate the excessive sample complexity burden in modern sequential decision making problems. However, the extent to which this broader approach can be effective is not well understood, where the literature largely consists of sufficient conditions. This work focuses on the basic question of what are necessary representational and distributional conditions that permit provable sample-efficient offline reinforcement learning. Perhaps surprisingly, our main result shows that even if: i) we have realizability in that the true value function of \emph{every} policy is linear in a given set of features and 2) our off-policy data has good coverage over all features (under a strong spectral condition), then any algorithm still (information-theoretically) requires a number of offline samples that is exponential in the problem horizon in order to non-trivially estimate the value of \emph{any} given policy. Our results highlight that sample-efficient offline policy evaluation is simply not possible unless significantly stronger conditions hold; such conditions include either having low distribution shift (where the offline data distribution is close to the distribution of the policy to be evaluated) or significantly stronger representational conditions (beyond realizability).

Motivation & Objective

• Assess whether realizability and good feature coverage suffice for sample-efficient offline RL with linear function approximation.
• Establish fundamental limits on offline policy evaluation in the linear setting.
• Illustrate error amplification and identify conditions under which efficiency is possible.
• Provide insights into how LSPE behaves under offline data and realizability.

Proposed method

• State and formalize a hardness theorem showing exponential sample complexity in horizon H under Assumptions 1 and 2.
• Construct a hard MDP instance with linear Q-functions and bounded feature norms to prove information-theoretic limits.
• Analyze Least-Squares Policy Evaluation (LSPE) to illustrate geometric error amplification under offline data.
• Introduce Assumption 3 (low distribution shift) and discuss its sufficiency for achieving sample efficiency with LSPE.
• Discuss the relationship between offline LSVE/LSVI and their online/offline counterparts (LSPI) in terms of sample complexity.

Experimental results

Research questions

• RQ1 Can realizability plus good feature coverage guarantee polynomial sample complexity for offline policy evaluation with linear function approximation?
• RQ2 Under what distributional or representational strengthenings can offline RL become sample-efficient?
• RQ3 How does error propagate (amplify) in offline policy evaluation with LSPE under the realizability assumption?
• RQ4 What concrete hard instances demonstrate the limits of offline RL with linear function approximation?

Key findings

• Any algorithm requires exponential samples in the horizon to non-trivially estimate the value of any given policy under the stated assumptions.
• LSPE has exponential variance in the horizon H despite being an unbiased estimator under the construction.
• There is an exponential separation between offline LSVI and LSPI in sample complexity under the offline, linear, realizability setting.
• Two hard instances (sparse reward and deterministic dynamics) show simultaneous potential amplification in reward and transition estimation errors.
• Sample-efficient offline policy evaluation is only possible under stronger conditions such as low distribution shift or stronger representation assumptions beyond realizability.