[Paper Review] Near-Optimal Offline Reinforcement Learning via Double Variance Reduction

Minghao Yin, Yu Bai+1 more|arXiv (Cornell University)|Feb 2, 2021

Reinforcement Learning in RoboticsComputer Science60 references18 citations

TL;DR

This paper proposes OPDVR, a novel offline reinforcement learning algorithm using double variance reduction to achieve near-optimal sample complexity. It proves that OPDVR finds an $\epsilon$-optimal policy with $\widetilde{O}(H^{2}/d_{m}\epsilon^{2})$ offline episodes in finite-horizon stationary MDPs, improving over prior work by an $H$-factor and matching a new information-theoretic lower bound up to logarithmic factors.

ABSTRACT

We consider the problem of offline reinforcement learning (RL) -- a well-motivated setting of RL that aims at policy optimization using only historical data. Despite its wide applicability, theoretical understandings of offline RL, such as its optimal sample complexity, remain largely open even in basic settings such as \emph{tabular} Markov Decision Processes (MDPs). In this paper, we propose Off-Policy Double Variance Reduction (OPDVR), a new variance reduction based algorithm for offline RL. Our main result shows that OPDVR provably identifies an $ε$-optimal policy with $\widetilde{O}(H^2/d_mε^2)$ episodes of offline data in the finite-horizon stationary transition setting, where $H$ is the horizon length and $d_m$ is the minimal marginal state-action distribution induced by the behavior policy. This improves over the best known upper bound by a factor of $H$. Moreover, we establish an information-theoretic lower bound of $Ω(H^2/d_mε^2)$ which certifies that OPDVR is optimal up to logarithmic factors. Lastly, we show that OPDVR also achieves rate-optimal sample complexity under alternative settings such as the finite-horizon MDPs with non-stationary transitions and the infinite horizon MDPs with discounted rewards.

Motivation & Objective

• To close the gap in theoretical understanding of offline reinforcement learning sample complexity in tabular MDPs.
• To develop an algorithm that achieves optimal sample complexity for offline policy optimization in finite-horizon stationary MDPs.
• To establish tight information-theoretic lower bounds to certify optimality of the proposed algorithm.
• To extend the algorithm’s optimality to non-stationary and infinite-horizon discounted MDPs.
• To resolve initialization dependence issues in prior variance reduction methods for offline RL.

Proposed method

• OPDVR uses a stochastic value iteration framework with double variance reduction to stabilize learning from offline data.
• It applies a doubling technique to overcome initialization-dependent performance issues in standard variance reduction algorithms.
• The algorithm performs minibatch stochastic updates on value functions using historical trajectories, reducing estimation variance.
• It leverages a modified form of variance reduction inspired by Sidford et al. (2018a), adapted for off-policy data and stationary transitions.
• The method ensures minimax optimality by preserving concentration bounds under arbitrary initialization.
• It incorporates conditional variance decomposition to analyze error propagation across time steps in the MDP.

Experimental results

Research questions

• RQ1 What is the optimal sample complexity for offline reinforcement learning in finite-horizon stationary MDPs?
• RQ2 Can a variance reduction-based algorithm achieve this optimal complexity in the offline setting?
• RQ3 How does the sample complexity of offline RL differ between stationary and non-stationary transition models?
• RQ4 Can the proposed algorithm maintain optimality in infinite-horizon discounted MDPs?
• RQ5 What are the theoretical limitations of prior variance reduction methods in offline RL, and how can they be overcome?

Key findings

• OPDVR achieves an $\widetilde{O}(H^{2}/d_{m}\epsilon^{2})$ sample complexity for finding an $\epsilon$-optimal policy in finite-horizon stationary MDPs, improving over the prior best bound by an $H$ factor.
• The paper establishes a matching information-theoretic lower bound of $\Omega(H^{2}/d_{m}\epsilon^{2})$, proving OPDVR is optimal up to logarithmic factors.
• In the finite-horizon non-stationary setting, OPDVR achieves $\widetilde{O}(H^{3}/d_{m}\epsilon^{2})$ sample complexity, which is optimal up to logarithmic factors.
• For infinite-horizon discounted MDPs, OPDVR achieves $\widetilde{O}((1-\gamma)^{-3}/d_{m}\epsilon^{2})$ sample complexity in terms of steps, which is optimal up to logarithmic factors.
• The algorithm resolves a critical flaw in prior variance reduction methods by using a doubling procedure that removes dependency on initialization.
• The analysis confirms that OPDVR is the first offline algorithm to achieve $O(H^2)$ horizon dependence in the stationary setting, formally separating it from non-stationary settings.