[Paper Review] Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization

Tatsuya Matsushima, Hiroki Furuta+3 more|arXiv (Cornell University)|Jun 5, 2020

Reinforcement Learning in RoboticsComputer Science62 references50 citations

TL;DR

Introduces BREMEN, a model-based offline RL method with implicit KL regularization and an ensemble of dynamics models to achieve high deployment efficiency (5–10 deployments) while maintaining competitive sample efficiency.

ABSTRACT

Most reinforcement learning (RL) algorithms assume online access to the environment, in which one may readily interleave updates to the policy with experience collection using that policy. However, in many real-world applications such as health, education, dialogue agents, and robotics, the cost or potential risk of deploying a new data-collection policy is high, to the point that it can become prohibitive to update the data-collection policy more than a few times during learning. With this view, we propose a novel concept of deployment efficiency, measuring the number of distinct data-collection policies that are used during policy learning. We observe that naïvely applying existing model-free offline RL algorithms recursively does not lead to a practical deployment-efficient and sample-efficient algorithm. We propose a novel model-based algorithm, Behavior-Regularized Model-ENsemble (BREMEN) that can effectively optimize a policy offline using 10-20 times fewer data than prior works. Furthermore, the recursive application of BREMEN is able to achieve impressive deployment efficiency while maintaining the same or better sample efficiency, learning successful policies from scratch on simulated robotic environments with only 5-10 deployments, compared to typical values of hundreds to millions in standard RL baselines. Codes and pre-trained models are available at https://github.com/matsuolab/BREMEN .

Motivation & Objective

• Motivate deployment efficiency as a practical metric for RL in high-cost deployment settings (health, robotics, dialogue, education).
• Develop an algorithm that learns successful policies with very few data-collection policy changes.
• Achieve strong performance with small offline datasets by leveraging model ensembles and conservative updates.

Proposed method

• Propose Behavior-Regularized Model-ENsemble (BREMEN) combining an ensemble of deterministic dynamics models with a policy updated via trust-region optimization.
• Use imaginary rollouts from the model ensemble to train a policy, reducing dependence on real environment interactions.
• Initialize the policy with behavior cloning of the latest data to implicitly regularize against distribution shift.
• Apply KL-based trust-region updates to constrain policy improvement and regularize learning (without explicit KL penalties on the objective).
• Train dynamics models on collected data; at deployment, collect a batch, update the model ensemble, estimate a behavior policy from the data, re-initialize the policy, and perform T offline KL-constrained updates using imaginary rollouts.

Experimental results

Research questions

• RQ1 Can deployment efficiency be used as a practical metric to reduce data collection costs and risks in RL?
• RQ2 Does a model-based offline approach with ensemble dynamics and implicit KL regularization outperform traditional online/offline RL methods under deployment constraints?
• RQ3 How does BREMEN perform in standard offline RL benchmarks with varying dataset sizes (1M, 100K, 50K) and in deployment-constrained scenarios?
• RQ4 What is the impact of behavior-cloning initialization and implicit KL regularization on mitigating model-bias and distribution shift?

Key findings

• BREMEN achieves high deployment efficiency, learning successful policies with only 5–10 deployments on MuJoCo continuous-control tasks.
• In offline-batch settings, BREMEN attains competitive performance with 1M transition datasets and outperforms baselines when trained on smaller datasets (10–20x smaller).
• Compared to SAC, ME-TRPO, BCQ, and BRAC in deployment-constrained setups, BREMEN shows markedly better progress under limited deployments.
• Behavior cloning initialization combined with conservative trust-region updates provides implicit KL regularization that outperforms explicit KL penalties in this setting.
• BREMEN’s offline performance on standard benchmarks approaches state-of-the-art model-free/offline methods, while requiring far fewer deployments.