[Paper Review] Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization

Haoran Xu, Lì Jiāng+5 more|arXiv (Cornell University)|Mar 28, 2023

Data Stream Mining Techniques4 citations

TL;DR

This paper proposes an Implicit Value Regularization (IVR) framework that enables in-sample offline reinforcement learning by implicitly regularizing the value function using only dataset actions, avoiding out-of-distribution (OOD) action extrapolation. The framework leads to two new algorithms, Sparse Q-learning (SQL) and Exponential Q-learning (EQL), which achieve state-of-the-art performance, especially in noisy and small-data regimes, by introducing sparsity that improves robustness to poor-quality data.

ABSTRACT

Most offline reinforcement learning (RL) methods suffer from the trade-off between improving the policy to surpass the behavior policy and constraining the policy to limit the deviation from the behavior policy as computing $Q$-values using out-of-distribution (OOD) actions will suffer from errors due to distributional shift. The recently proposed extit{In-sample Learning} paradigm (i.e., IQL), which improves the policy by quantile regression using only data samples, shows great promise because it learns an optimal policy without querying the value function of any unseen actions. However, it remains unclear how this type of method handles the distributional shift in learning the value function. In this work, we make a key finding that the in-sample learning paradigm arises under the extit{Implicit Value Regularization} (IVR) framework. This gives a deeper understanding of why the in-sample learning paradigm works, i.e., it applies implicit value regularization to the policy. Based on the IVR framework, we further propose two practical algorithms, Sparse $Q$-learning (SQL) and Exponential $Q$-learning (EQL), which adopt the same value regularization used in existing works, but in a complete in-sample manner. Compared with IQL, we find that our algorithms introduce sparsity in learning the value function, making them more robust in noisy data regimes. We also verify the effectiveness of SQL and EQL on D4RL benchmark datasets and show the benefits of in-sample learning by comparing them with CQL in small data regimes.

Motivation & Objective

• Address the fundamental trade-off in offline RL between policy performance and distributional shift due to out-of-distribution (OOD) actions.
• Understand why in-sample learning methods like IQL work by framing them within a unified implicit value regularization (IVR) framework.
• Develop practical in-sample algorithms that maintain value regularization but avoid OOD action queries, improving robustness in low-data and noisy settings.
• Demonstrate that sparsity in value function learning enhances performance when datasets contain noisy or suboptimal transitions.
• Validate the proposed methods on standard D4RL benchmarks and custom small/noisy data regimes, showing superiority over CQL and IQL.

Proposed method

• Propose the Implicit Value Regularization (IVR) framework, which formulates policy learning as a constrained optimization problem with behavior regularizers.
• Show that the optimal policy under IVR has a closed-form solution expressed as a weighted mixture of the behavior policy, where weights are derived from state- and action-value functions.
• Transform the intractable state-value function computation into a convex optimization problem solvable via dataset samples, enabling in-sample learning.
• Adapt existing value regularization techniques from CQL and AWR into a complete in-sample learning paradigm, resulting in SQL and EQL.
• Introduce sparsity in the state-value function by filtering out actions with low Q-values below a threshold, enhancing robustness to noisy data.
• Train the value functions exclusively on dataset actions, eliminating reliance on OOD action value estimation and reducing extrapolation errors.

Experimental results

Research questions

• RQ1 Why does in-sample learning, as in IQL, achieve strong performance despite avoiding OOD action value queries?
• RQ2 Can a unified theoretical framework explain the success of in-sample learning methods in offline RL?
• RQ3 How does introducing sparsity in value function learning improve robustness in noisy or low-quality datasets?
• RQ4 Does in-sample learning provide a performance advantage over out-of-sample learning in small-data regimes with limited state coverage?
• RQ5 Can existing value regularization techniques be adapted into a complete in-sample learning framework without sacrificing performance?

Key findings

• The in-sample learning paradigm used in IQL is formally explained by the IVR framework, which reveals that IQL implicitly applies value regularization with a hyperparameter τ controlling the strength of regularization.
• SQL and EQL, derived from the IVR framework, outperform IQL in noisy data regimes: on Walker2d with only 5% expert ratio, SQL and EQL achieve near-expert performance (≈100 return), while IQL drops sharply.
• In small-data regimes (e.g., AntMaze with hard discarding), CQL suffers from exponential growth in Bellman error (up to 300.5) and performance drop (0% return), while SQL maintains low error (1.9) and high return (24.2).
• SQL and EQL achieve lower Bellman error than CQL across all difficulty levels in the AntMaze benchmark, indicating reduced value extrapolation error due to in-sample learning.
• The sparsity introduced in SQL’s value function learning filters out low-quality actions, making the method more robust to noisy transitions, especially when expert data is scarce.
• On D4RL and Atari benchmarks, SQL and EQL achieve state-of-the-art performance, confirming the benefits of in-sample learning with implicit regularization in diverse offline RL settings.