[Paper Review] Towards Instance-Optimal Offline Reinforcement Learning with Pessimism

Minghao Yin, Yu-Xiang Wang|arXiv (Cornell University)|Oct 17, 2021

Reinforcement Learning in Robotics50 references4 citations

TL;DR

This paper introduces an instance-optimal offline reinforcement learning framework using pessimism, deriving a tight suboptimality bound that adapts to individual MDP instances. The proposed Adaptive Pessimistic Value Iteration (APVI) algorithm achieves a bound nearly matching $ O\left(\sum_{h=1}^{H}\sum_{s_h,a_h} d^{\pi^\star}_h(s_h,a_h) \sqrt{\frac{\mathrm{Var}_{P_{s_h,a_h}}(V^{\star}_{h+1}+r_h)}{d^{\mu}_h(s_h,a_h)}} \sqrt{\frac{1}{n}} \right) $, which recovers known minimax and problem-dependent results under various assumptions and establishes a per-instance information-theoretic lower bound.

ABSTRACT

We study the offline reinforcement learning (offline RL) problem, where the goal is to learn a reward-maximizing policy in an unknown Markov Decision Process (MDP) using the data coming from a policy $μ$. In particular, we consider the sample complexity problems of offline RL for finite-horizon MDPs. Prior works study this problem based on different data-coverage assumptions, and their learning guarantees are expressed by the covering coefficients which lack the explicit characterization of system quantities. In this work, we analyze the Adaptive Pessimistic Value Iteration (APVI) algorithm and derive the suboptimality upper bound that nearly matches \[ O\left(\sum_{h=1}^H\sum_{s_h,a_h}d^{π^\star}_h(s_h,a_h)\sqrt{\frac{\mathrm{Var}_{P_{s_h,a_h}}{(V^\star_{h+1}+r_h)}}{d^μ_h(s_h,a_h)}}\sqrt{\frac{1}{n}} ight). \] In complementary, we also prove a per-instance information-theoretical lower bound under the weak assumption that $d^μ_h(s_h,a_h)>0$ if $d^{π^\star}_h(s_h,a_h)>0$. Different from the previous minimax lower bounds, the per-instance lower bound (via local minimaxity) is a much stronger criterion as it applies to individual instances separately. Here $π^\star$ is a optimal policy, $μ$ is the behavior policy and $d_h^μ$ is the marginal state-action probability. We call the above equation the intrinsic offline reinforcement learning bound since it directly implies all the existing optimal results: minimax rate under uniform data-coverage assumption, horizon-free setting, single policy concentrability, and the tight problem-dependent results. Later, we extend the result to the assumption-free regime (where we make no assumption on $ μ$) and obtain the assumption-free intrinsic bound. Due to its generic form, we believe the intrinsic bound could help illuminate what makes a specific problem hard and reveal the fundamental challenges in offline RL.

Motivation & Objective

• To develop a provably efficient offline RL method that adapts to individual problem instances rather than relying on worst-case assumptions.
• To derive a suboptimality bound that captures the intrinsic difficulty of each MDP instance, based on behavior policy coverage and value variance.
• To establish a per-instance information-theoretic lower bound via local minimaxity, which is stronger than traditional minimax bounds.
• To extend the analysis to an assumption-free regime, where no data-coverage assumptions are made on the behavior policy.
• To unify and recover existing results—minimax, horizon-free, single policy concentrability, and problem-dependent bounds—under a single, intrinsic bound.

Proposed method

• Proposes the Adaptive Pessimistic Value Iteration (APVI) algorithm, which incorporates pessimism in value iteration to handle distributional shift in offline data.
• Derives an upper bound on suboptimality that depends on the optimal policy’s state-action visitation $ d^{\pi^\star}_h $, behavior policy coverage $ d^{\mu}_h $, and the variance of the next-value function $ \mathrm{Var}_{P_{s_h,a_h}}(V^{\star}_{h+1}+r_h) $.
• Uses a novel reduction to two-point optimal-value estimation to prove a per-instance lower bound, establishing local minimax optimality.
• Introduces a pessimistic augmented MDP to analyze the assumption-free regime, enabling bounds without any data-coverage assumptions.
• Employs Hellinger distance and second-order Taylor expansion to control distributional shifts and ensure statistical consistency of the estimated transition models.
• Applies a local instance-dependent analysis to derive a lower bound that holds for each individual MDP instance, not just in expectation over worst-case instances.

Experimental results

Research questions

• RQ1 Can we derive a suboptimality bound in offline RL that adapts to the intrinsic difficulty of individual MDP instances rather than relying on worst-case assumptions?
• RQ2 How does the proposed bound relate to and recover existing results such as minimax rates, horizon-free bounds, and single-policy concentrability?
• RQ3 What is the fundamental statistical limit of offline RL for a given instance, and can it be characterized via a per-instance lower bound?
• RQ4 Can we establish a provably efficient algorithm in the assumption-free regime, where no data-coverage assumptions are made on the behavior policy?
• RQ5 Does the derived bound reflect the true system-level complexity of offline RL, including the roles of value variance and coverage?

Key findings

• The APVI algorithm achieves a suboptimality upper bound that nearly matches $ O\left(\sum_{h=1}^{H}\sum_{s_h,a_h} d^{\pi^\star}_h(s_h,a_h) \sqrt{\frac{\mathrm{Var}_{P_{s_h,a_h}}(V^{\star}_{h+1}+r_h)}{d^{\mu}_h(s_h,a_h)}} \sqrt{\frac{1}{n}} \right) $, which is shown to be instance-optimal.
• The paper proves a per-instance information-theoretic lower bound under the weak assumption $ d^\mu_h(s_h,a_h) > 0 $ whenever $ d^{\pi^\star}_h(s_h,a_h) > 0 $, which is stronger than traditional minimax bounds.
• The intrinsic bound recovers known results: minimax rate under uniform coverage, horizon-free bounds, single policy concentrability, and problem-dependent bounds.
• In the assumption-free regime, the paper derives a bound that holds without any data-coverage assumptions, relying only on the behavior policy's support and the structure of the MDP.
• The analysis shows that the fundamental challenge in offline RL lies in the interplay between the optimal policy’s visitation, the behavior policy’s coverage, and the variance of the value function, which the intrinsic bound captures precisely.
• The derived bound is shown to be tight in multiple regimes, demonstrating that the proposed method achieves instance-optimal sample complexity.