[Paper Review] Offline RL Policies Should be Trained to be Adaptive

Dibya Ghosh, Anurag Ajay+2 more|arXiv (Cornell University)|Jul 5, 2022

Machine Learning and Algorithms4 citations

TL;DR

This paper argues that offline reinforcement learning (RL) policies should be adaptive rather than purely conservative, as epistemic uncertainty from limited datasets induces implicit partial observability, necessitating history-dependent adaptation. The authors propose APE-V, an ensemble-based method that adaptively selects among value functions during inference, achieving superior performance on offline RL benchmarks like Procgen Mazes and D4RL by outperforming conservative baselines through dynamic strategy switching.

ABSTRACT

Offline RL algorithms must account for the fact that the dataset they are provided may leave many facets of the environment unknown. The most common way to approach this challenge is to employ pessimistic or conservative methods, which avoid behaviors that are too dissimilar from those in the training dataset. However, relying exclusively on conservatism has drawbacks: performance is sensitive to the exact degree of conservatism, and conservative objectives can recover highly suboptimal policies. In this work, we propose that offline RL methods should instead be adaptive in the presence of uncertainty. We show that acting optimally in offline RL in a Bayesian sense involves solving an implicit POMDP. As a result, optimal policies for offline RL must be adaptive, depending not just on the current state but rather all the transitions seen so far during evaluation.We present a model-free algorithm for approximating this optimal adaptive policy, and demonstrate the efficacy of learning such adaptive policies in offline RL benchmarks.

Motivation & Objective

• To challenge the prevailing reliance on conservative, static policies in offline RL, which can be suboptimal due to sensitivity to conservatism hyperparameters.
• To formalize offline RL as a Bayesian problem where epistemic uncertainty leads to implicit partial observability, necessitating adaptive policies.
• To develop a practical algorithm that enables within-episode adaptation in offline RL, improving test-time performance over static policies.
• To demonstrate that adaptive policies can outperform conservative policies even in data-scarce settings by dynamically selecting among value function hypotheses.

Proposed method

• Formalize offline RL as a POMDP induced by epistemic uncertainty, where the agent's belief over MDPs evolves based on observed transitions.
• Derive a policy gradient objective for adaptive history-based policies, using a posterior over MDPs and a distribution over value functions to model uncertainty.
• Propose APE-V, a model-free algorithm that maintains an ensemble of Q-functions and uses a belief distribution over them to guide action selection during evaluation.
• Implement within-episode adaptation by dynamically updating the belief state based on observed transitions, allowing the policy to switch strategies when initial predictions are incorrect.
• Use a policy gradient update that optimizes the adaptive policy using the implicit POMDP objective, enabling it to learn from both the dataset and real-time feedback during evaluation.
• Apply the method to offline RL benchmarks, including Procgen Mazes and D4RL, using SAC-n as the base value function learner and comparing to conservative baselines like CQL and SAC-n.

Experimental results

Research questions

• RQ1 Can static, conservative policies in offline RL be arbitrarily suboptimal due to their inability to adapt to new information during evaluation?
• RQ2 Does implicit partial observability arising from dataset uncertainty necessitate adaptive behavior in offline RL?
• RQ3 Can a model-free, ensemble-based method effectively approximate the Bayes-optimal adaptive policy in offline RL?
• RQ4 Does within-episode adaptation using belief updates over value functions lead to improved test-time performance compared to static conservative policies?
• RQ5 In what settings does adaptation provide a significant advantage over conservatism in offline RL?

Key findings

• APE-V achieves 79% success rate in solving held-out mazes in the Procgen environment, outperforming both conservative and non-adaptive methods.
• In a low-data regime with only 200 training mazes, APE-V achieves 31% success rate on held-out mazes, compared to 23% for conservative methods.
• On D4RL, APE-V outperforms SAC-n and CQL on medium-performance and medium-replay datasets, achieving higher average return than any single ensemble member.
• The adaptive policy in D4RL consistently outperforms any fixed choice of value function, demonstrating the benefit of dynamic belief updating during inference.
• Adaptation provides marginal gains on tasks with low ambiguity in data distributions, such as expert- or random-policy collected data, where multiple hypotheses are less salient.
• The results confirm that adaptation is necessary for optimal offline RL performance, as static policies can be arbitrarily suboptimal under uncertainty.