[Paper Review] When does return-conditioned supervised learning work for offline reinforcement learning?

David Brandfonbrener, Alberto Bietti+3 more|arXiv (Cornell University)|Jun 2, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper investigates when return-conditioned supervised learning (RCSL) succeeds in offline reinforcement learning, showing it achieves optimal policies only under strong assumptions—nearly deterministic dynamics, known return conditioning values, and sufficient return coverage in the dataset. Empirical results confirm these theoretical limits, revealing RCSL fails in stochastic or poorly covered environments, despite its simplicity and scalability compared to dynamic programming methods.

ABSTRACT

Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL). RCSL algorithms learn the distribution of actions conditioned on both the state and the return of the trajectory. Then they define a policy by conditioning on achieving high return. In this paper, we provide a rigorous study of the capabilities and limitations of RCSL, something which is crucially missing in previous work. We find that RCSL returns the optimal policy under a set of assumptions that are stronger than those needed for the more traditional dynamic programming-based algorithms. We provide specific examples of MDPs and datasets that illustrate the necessity of these assumptions and the limits of RCSL. Finally, we present empirical evidence that these limitations will also cause issues in practice by providing illustrative experiments in simple point-mass environments and on datasets from the D4RL benchmark.

Motivation & Objective

• To rigorously analyze the theoretical guarantees and failure modes of return-conditioned supervised learning (RCSL) in offline reinforcement learning.
• To identify the necessary and sufficient conditions under which RCSL recovers the optimal policy, comparing them to traditional dynamic programming (DP) methods.
• To empirically validate theoretical limitations using synthetic environments and D4RL benchmark datasets.
• To clarify the relationship between RCSL, behavior cloning (BC), and DP-based algorithms in terms of performance and robustness.
• To assess whether RCSL can serve as a general-purpose solution for offline RL or is limited to specific, well-structured settings.

Proposed method

• RCSL trains a policy to predict actions conditioned on both state and trajectory return using maximum likelihood estimation on a dataset of expert trajectories.
• The method minimizes the empirical negative log-likelihood loss: $\hat{L}(\pi) = -\sum_{\tau \in \mathcal{D}} \sum_{t=1}^{H} \log \pi(a_t | s_t, g(\tau))$.
• At inference, the policy is conditioned on a target return via a function $f(s)$, defining $\pi_f(a|s) = \pi(a|s, f(s))$.
• Theoretical analysis derives conditions under which RCSL recovers the optimal policy, including nearly deterministic dynamics and full return support in the dataset.
• Empirical evaluation uses point-mass environments and D4RL benchmark datasets to test performance under varying return coverage, stochasticity, and behavior policy quality.
• Comparative experiments include DP-based methods (TD3+BC, IQL), BC-based baselines, and RCSL variants (e.g., Decision Transformer), with hyperparameter tuning and multiple random seeds.

Experimental results

Research questions

• RQ1 Under what conditions does RCSL recover the optimal policy, and how do these compare to the assumptions required by dynamic programming (DP) methods?
• RQ2 What are the theoretical failure modes of RCSL, and how do they manifest in practice?
• RQ3 How does RCSL perform relative to BC and DP-based algorithms in environments with sparse rewards, stochastic dynamics, or poor dataset coverage?
• RQ4 Can RCSL generalize to real-world offline RL benchmarks, or is its success limited to specific, high-quality data regimes?
• RQ5 To what extent does RCSL inherit the simplicity of supervised learning while avoiding the pitfalls of behavior cloning?

Key findings

• RCSL recovers the optimal policy only under stronger assumptions than DP, including nearly deterministic dynamics and full support of the target return in the dataset.
• In environments with stochastic dynamics or sparse returns, RCSL fails to learn even positive returns, as seen in the antmaze sparse-reward setting where no high-return trajectories exist.
• On the D4RL benchmark, RCSL underperforms DP-based methods like TD3+BC and IQL on most tasks, especially those with high stochasticity (e.g., antmaze-medium-play).
• RCSL performs well only when the behavior policy is high-quality but suboptimal, such as in the pen-human dataset, where dense rewards and limited coverage allow return-conditioning to improve over BC.
• Theoretical analysis shows that RCSL’s reliance on trajectory-level return information makes it fundamentally limited by the best trajectories in the dataset, requiring exponential sample complexity in horizon to match optimal performance.
• Despite its simplicity and scalability, RCSL is not a general solution for offline RL, as it cannot recover from poor dataset coverage or stochastic dynamics, unlike DP methods.