[Paper Review] Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RL

Andrea Zanette|arXiv (Cornell University)|Dec 14, 2020

Reinforcement Learning in Robotics55 references18 citations

TL;DR

This paper establishes exponential information-theoretic lower bounds for batch reinforcement learning in linear function approximation settings, showing that even with exact rewards, transitions, and the optimal data distribution, batch RL remains exponentially harder than online RL. The key result is an exponential separation between batch and online learning, demonstrating fundamental limitations in learning near-optimal policies or evaluating target policies using only offline data, despite strong assumptions like realizability and exact feedback.

ABSTRACT

Several practical applications of reinforcement learning involve an agent learning from past data without the possibility of further exploration. Often these applications require us to 1) identify a near optimal policy or to 2) estimate the value of a target policy. For both tasks we derive \emph{exponential} information-theoretic lower bounds in discounted infinite horizon MDPs with a linear function representation for the action value function even if 1) \emph{realizability} holds, 2) the batch algorithm observes the exact reward and transition \emph{functions}, and 3) the batch algorithm is given the \emph{best} a priori data distribution for the problem class. Our work introduces a new `oracle + batch algorithm' framework to prove lower bounds that hold for every distribution. The work shows an exponential separation between batch and online reinforcement learning.

Motivation & Objective

• To investigate whether batch reinforcement learning (RL) is fundamentally limited compared to online RL, even under strong assumptions.
• To determine if a batch algorithm can efficiently learn near-optimal policies or evaluate target policies when given the best possible data distribution.
• To establish information-theoretic lower bounds that hold for every batch data distribution, under realizability and exact feedback.
• To demonstrate an exponential separation between batch and online RL in infinite-horizon MDPs with linear function approximation.
• To formalize a novel 'oracle + batch algorithm' framework to derive strong lower bounds that are distribution-agnostic.

Proposed method

• Introduce an 'oracle + batch algorithm' framework where an oracle selects the best data distribution in advance, and the batch algorithm learns from it.
• Construct a family of MDPs with a single critical state and linearly parameterized value functions to create hard instances.
• Use hyperspherical caps in the action space to encode the unknown direction of the optimal policy, making it difficult to identify without sufficient samples.
• Leverage a helper lemma showing that for any unit vector, at least one canonical basis vector has an inner product of at least $1/\sqrt{d}$, enabling detection of the optimal action.
• Prove lower bounds by showing that any batch algorithm must make exponentially many queries to achieve sub-optimality bounds of $1/\sqrt{d}$.
• Demonstrate that even with exact feedback and optimal data, the batch algorithm cannot reliably identify the correct policy or value function without exponential sample complexity.

Experimental results

Research questions

• RQ1 Can batch RL achieve efficient learning of near-optimal policies when given the best possible data distribution, even with exact feedback and realizability?
• RQ2 Is there a fundamental exponential gap between batch and online RL in linear function approximation settings?
• RQ3 Can information-theoretic lower bounds be established that hold for all data distributions, not just worst-case ones?
• RQ4 Does the structure of the action space (e.g., hyperspherical caps) lead to inherent hardness in policy identification and off-policy evaluation?
• RQ5 Can the oracle framework be used to prove strong lower bounds that are independent of the data distribution?

Key findings

• The paper establishes exponential lower bounds for both off-policy evaluation and best policy identification in batch RL, even when the algorithm has access to the exact reward and transition functions.
• Even with the best a priori data distribution, the batch algorithm requires exponentially many samples in the feature dimension $d$ to achieve a sub-optimality gap of $1/\sqrt{d}$.
• The lower bounds hold under realizability and exact feedback, showing that the difficulty is not due to model misspecification or data noise.
• The exponential separation between batch and online RL is proven by showing that online algorithms can adaptively query to identify the optimal policy in polynomial time.
• The results imply that batch RL cannot be made efficient in general, even with strong assumptions, due to the inherent information-theoretic limitations of offline data.