[Paper Review] MOPO: Model-based Offline Policy Optimization

Tianhe Yu, Garrett Thomas+6 more|arXiv (Cornell University)|May 27, 2020

Reinforcement Learning in Robotics71 references217 citations

TL;DR

MOPO introduces an offline model-based RL method that penalizes rewards by estimated model uncertainty to safely generalize beyond the offline data distribution, outperforming prior model-free and model-based approaches on D4RL and out-of-distribution tasks.

ABSTRACT

Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. This problem setting offers the promise of utilizing such datasets to acquire policies without any costly or dangerous active exploration. However, it is also challenging, due to the distributional shift between the offline training data and those states visited by the learned policy. Despite significant recent progress, the most successful prior methods are model-free and constrain the policy to the support of data, precluding generalization to unseen states. In this paper, we first observe that an existing model-based RL algorithm already produces significant gains in the offline setting compared to model-free approaches. However, standard model-based RL methods, designed for the online setting, do not provide an explicit mechanism to avoid the offline setting's distributional shift issue. Instead, we propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics. We theoretically show that the algorithm maximizes a lower bound of the policy's return under the true MDP. We also characterize the trade-off between the gain and risk of leaving the support of the batch data. Our algorithm, Model-based Offline Policy Optimization (MOPO), outperforms standard model-based RL algorithms and prior state-of-the-art model-free offline RL algorithms on existing offline RL benchmarks and two challenging continuous control tasks that require generalizing from data collected for a different task. The code is available at https://github.com/tianheyu927/mopo.

Motivation & Objective

• Motivate offline RL that can generalize beyond the data support and beyond the target task.
• Develop a model-based offline RL algorithm that manages distributional shift via uncertainty penalties.
• Provide theoretical guarantees that MOPO maximizes a lower bound on the true return.
• Propose a practical MOPO implementation with ensemble-based uncertainty to penalize rewards.
• Evaluate MOPO on standard offline RL benchmarks and tasks requiring out-of-distribution generalization.

Proposed method

• Build on MBPO by incorporating an uncertainty-based reward penalty derived from model error estimates.
• Define an uncertainty-penalized reward: tilde{r}(s,a) = r(s,a) - lambda * u(s,a).
• Estimate dynamics with an ensemble of probabilistic models and use the maximum ensemble variance as u(s,a).
• Train a policy on the uncertainty-penalized MDP to maximize the conservative return.
• Provide a theoretical bound: eta_M(hat{pi}) >= max_pi { eta_M(pi) - 2 lambda epsilon_u(pi) }.
• Offer practical guidelines for implementing MOPO, including how lambda relates to the error estimator and how it is computed.

Experimental results

Research questions

• RQ1 Can offline model-based RL generalize beyond the data support better than model-free offline methods?
• RQ2 How should uncertainty about dynamics be quantified and incorporated into the reward to balance risk and return?
• RQ3 Does MOPO outperform existing model-free offline methods on standard benchmarks and in out-of-distribution tasks?
• RQ4 What theoretical guarantees can be provided for MOPO’s performance relative to the true MDP?

Key findings

• MOPO substantially outperforms model-free offline RL algorithms on the D4RL benchmark across several datasets.
• MOPO also excels on tasks requiring generalization to out-of-distribution states, outperforming baselines and showing the ability to reach unseen states.
• Two main results: (i) MOPO’s uncertainty-penalized framework yields conservative yet effective policy optimization; (ii) vanilla MBPO can outperform SAC in offline settings, supporting model-based approaches for batch RL.
• An explicit trade-off between potential gain and risk is characterized, with a bound relating learned policy performance to model error along trajectories.