[Paper Review] NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning

Rongjun Qin, Songyi Gao+6 more|arXiv (Cornell University)|Feb 1, 2021

Reinforcement Learning in RoboticsComputer Science31 references25 citations

TL;DR

NeoRL presents near real-world offline RL benchmarks with multi-domain, finite-data datasets and extra test data, showing that deterministic behavior policies often rival or outperform many offline RL methods on these tasks.

ABSTRACT

Offline reinforcement learning (RL) aims at learning a good policy from a batch of collected data, without extra interactions with the environment during training. However, current offline RL benchmarks commonly have a large reality gap, because they involve large datasets collected by highly exploratory policies, and the trained policy is directly evaluated in the environment. In real-world situations, running a highly exploratory policy is prohibited to ensure system safety, the data is commonly very limited, and a trained policy should be well validated before deployment. In this paper, we present a near real-world offline RL benchmark, named NeoRL, which contains datasets from various domains with controlled sizes, and extra test datasets for policy validation. We evaluate existing offline RL algorithms on NeoRL and argue that the performance of a policy should also be compared with the deterministic version of the behavior policy, instead of the dataset reward. The empirical results demonstrate that the tested offline RL algorithms become less competitive to the deterministic policy on many datasets, and the offline policy evaluation hardly helps. The NeoRL suit can be found at http://polixir.ai/research/neorl. We hope this work will shed some light on future research and draw more attention when deploying RL in real-world systems.

Motivation & Objective

• Motivate offline RL for real-world, safety-conscious settings with limited data.
• Provide a unified, near real-world benchmark suite spanning multiple domains.
• Evaluate existing offline RL methods and compare them against the deterministic behavior policy.
• Highlight the limited utility of offline policy evaluation (OPE) in these realistic contexts.
• Offer datasets and evaluation protocols to guide deployment-ready RL research.

Proposed method

• Construct near real-world offline RL datasets from diverse domains with controllable data sizes and add an extra test dataset for policy validation.
• Generate multi-level policies (expert, low/medium/high returns) and collect training data by suboptimal policies with injected noise to simulate real-world data distributions.
• Provide a unified API for accessing state, action, reward, next state, and a reward function interface per task.
• Benchmark model-free and model-based offline RL methods against the deterministic behavior policy and the expert.
• Use online evaluation for final policy ranking and offline policy evaluation (FQE) for offline model selection, including KL constraints and perturbation mechanisms to stay near data support.

Experimental results

Research questions

• RQ1 Do current offline RL methods outperform the deterministic version of the behavior policy on near real-world offline benchmarks?
• RQ2 How does data quality and quantity affect the relative performance of model-free versus model-based offline RL methods in realistic tasks?
• RQ3 Can offline policy evaluation methods like FQE reliably rank policies and guide model selection in near real-world datasets?
• RQ4 What is the impact of enforcing proximity to the behavior policy (via KL penalties or perturbation constraints) on offline RL performance?
• RQ5 How do different domains (robotics, industrial, finance, city management) influence the effectiveness of offline RL algorithms?

Key findings

• BC often matches or outperforms many offline RL methods on the NeoRL tasks, indicating strong baselines from the deterministic behavior policy.
• CQL emerges as the best among several offline methods on about one third of tasks, but gains over the deterministic policy are limited on MuJoCo and IB environments.
• Model-based methods can underperform without proper constraints (e.g., KL) and may suffer from model exploitation, though MOPO with ensembles can mitigate this in some tasks.
• MOPO shows significant improvements in some tasks (e.g., HalfCheetah variants) but is sensitive to rollout length and penalty hyperparameters.
• Offline model selection with FQE can rank policies but often fails to outperform online evaluation, highlighting challenges in offline evaluation reliability.