[Paper Review] Model-Based Offline Meta-Reinforcement Learning with Regularization

Sen Lin, Jialin Wan+3 more|arXiv (Cornell University)|Feb 7, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes MerPO, a model-based offline meta-reinforcement learning framework with regularized policy optimization to balance exploration via a meta-policy and exploitation of offline datasets. By introducing a meta-regularized actor-critic method that interpolates between behavior policy and meta-policy regularizers, MerPO ensures guaranteed performance improvement over both policies, achieving safer and more generalizable policy learning in offline meta-RL settings.

ABSTRACT

Existing offline reinforcement learning (RL) methods face a few major challenges, particularly the distributional shift between the learned policy and the behavior policy. Offline Meta-RL is emerging as a promising approach to address these challenges, aiming to learn an informative meta-policy from a collection of tasks. Nevertheless, as shown in our empirical studies, offline Meta-RL could be outperformed by offline single-task RL methods on tasks with good quality of datasets, indicating that a right balance has to be delicately calibrated between "exploring" the out-of-distribution state-actions by following the meta-policy and "exploiting" the offline dataset by staying close to the behavior policy. Motivated by such empirical analysis, we explore model-based offline Meta-RL with regularized Policy Optimization (MerPO), which learns a meta-model for efficient task structure inference and an informative meta-policy for safe exploration of out-of-distribution state-actions. In particular, we devise a new meta-Regularized model-based Actor-Critic (RAC) method for within-task policy optimization, as a key building block of MerPO, using conservative policy evaluation and regularized policy improvement; and the intrinsic tradeoff therein is achieved via striking the right balance between two regularizers, one based on the behavior policy and the other on the meta-policy. We theoretically show that the learnt policy offers guaranteed improvement over both the behavior policy and the meta-policy, thus ensuring the performance improvement on new tasks via offline Meta-RL. Experiments corroborate the superior performance of MerPO over existing offline Meta-RL methods.

Motivation & Objective

• To address the critical challenge in offline meta-RL where existing methods fail to generalize well across datasets of varying quality, especially when behavior policies are high-quality.
• To identify and resolve the unbalanced exploration-exploitation tradeoff in offline Meta-RL, where over-reliance on the meta-policy leads to distributional shift, while over-fidelity to the behavior policy limits generalization.
• To develop a method that guarantees performance improvement over both the behavior policy and the meta-policy, ensuring safe and reliable policy updates in offline settings.
• To enable efficient task structure inference and safe out-of-distribution state-action exploration through a learned meta-model and synthetic rollouts.

Proposed method

• Proposes a meta-regularized model-based actor-critic (RAC) method for within-task policy optimization, using conservative policy evaluation and regularized policy improvement.
• Introduces a novel interpolation between two regularizers: one based on the behavior policy and one based on the meta-policy, to balance exploration and exploitation.
• Learns a meta-model from offline datasets to enable fast and accurate task-specific model learning and efficient structure inference across tasks.
• Uses the learned dynamics model to generate synthetic rollouts for safe exploration of out-of-distribution state-actions.
• Employs a regularized policy improvement module that combines entropy regularization and distributional constraints to prevent over-optimism and distributional shift.
• Theoretical analysis shows that the resulting policy guarantees performance improvement over both the behavior policy and the meta-policy with high probability under mild conditions.

Experimental results

Research questions

• RQ1 How can we design an offline meta-RL algorithm that safely balances exploration using a meta-policy and exploitation of a given offline dataset?
• RQ2 Why do existing offline Meta-RL methods underperform compared to single-task RL on high-quality datasets, and how can this be mitigated?
• RQ3 Can a model-based approach with conservative dynamics modeling and regularized policy optimization ensure guaranteed performance improvement over both the behavior policy and the meta-policy?
• RQ4 How can synthetic data generated from a meta-model improve generalization in offline meta-RL?
• RQ5 What is the optimal tradeoff between behavior policy fidelity and meta-policy guidance in terms of regularizer weights for performance guarantees?

Key findings

• MerPO achieves guaranteed performance improvement over both the behavior policy and the meta-policy with high probability, as theoretically proven under mild conditions.
• The method outperforms state-of-the-art offline Meta-RL baselines, including FOCAL and COMBO, in generalization across diverse offline datasets.
• On tasks with high-quality datasets, MerPO surpasses COMBO, a strong single-task offline RL method, demonstrating superior generalization capability.
• The performance gain is attributed to the effective balance between meta-policy exploration and behavior policy exploitation via the dual regularizer interpolation.
• Empirical results confirm that MerPO maintains safe policy improvement even when the behavior policy is strong, avoiding performance degradation due to distributional shift.
• The theoretical analysis provides a design guideline for selecting regularizer weights to ensure performance improvement, with positive improvement guaranteed when the meta-policy regularizer weight is appropriately tuned.