[Paper Review] Meta-Model-Based Meta-Policy Optimization

T. HIRAOKA, Takahisa Imagawa+4 more|arXiv (Cornell University)|Jun 4, 2020

Reinforcement Learning in Robotics41 references4 citations

TL;DR

This paper proposes Meta-Model-Based Meta-Policy Optimization (M3PO), a model-based meta-RL method that provides theoretical performance guarantees by extending Janner et al. (2019)’s theorems to meta-RL settings. M3PO uses branched rollouts in a meta-model to improve sample efficiency and generalization, outperforming prior methods like PEARL and M2PO in continuous-control benchmarks, especially in long-term adaptation and directional control tasks.

ABSTRACT

Model-based meta-reinforcement learning (RL) methods have recently been shown to be a promising approach to improving the sample efficiency of RL in multi-task settings. However, the theoretical understanding of those methods is yet to be established, and there is currently no theoretical guarantee of their performance in a real-world environment. In this paper, we analyze the performance guarantee of model-based meta-RL methods by extending the theorems proposed by Janner et al. (2019). On the basis of our theoretical results, we propose Meta-Model-Based Meta-Policy Optimization (M3PO), a model-based meta-RL method with a performance guarantee. We demonstrate that M3PO outperforms existing meta-RL methods in continuous-control benchmarks.

Motivation & Objective

• To address the lack of theoretical performance guarantees in model-based meta-RL methods, which often suffer from model bias despite strong empirical results.
• To extend the theoretical framework of Janner et al. (2019) on model-based rollouts to the meta-RL setting, particularly for partially observable Markov decision processes (POMDPs).
• To design a practical meta-RL algorithm that leverages theoretical guarantees to improve sample efficiency and robustness in multi-task reinforcement learning.
• To empirically validate that model adaptation via a meta-model enhances meta-policy performance, especially in long-horizon and directional control tasks.

Proposed method

• Formalize the meta-RL problem as a special case of POMDPs to enable theoretical analysis of model-based rollouts.
• Extend Janner et al. (2019)’s performance guarantee theorems to meta-RL, comparing branched rollouts against full-model rollouts.
• Propose M3PO, a practical algorithm that uses a context-aware meta-model and branched rollouts for action planning with theoretical performance bounds.
• Implement a gradual adaptation schedule for the meta-model’s influence (via hyperparameter β) to stabilize training and improve long-term performance.
• Use ensemble Gaussian models for transition dynamics and apply model predictive control (MPC) with meta-learned context for fast adaptation.
• Compare M3PO with M2PO (non-adaptive model baseline) and PEARL to isolate the impact of meta-model adaptation.

Experimental results

Research questions

• RQ1 Can theoretical performance guarantees from model-based rollouts in standard RL be extended to meta-RL settings?
• RQ2 Is branched rollout more tightly guaranteed than full-model rollout in meta-RL, and does this justify its use?
• RQ3 Does model adaptation via a meta-model significantly improve meta-policy performance in long-horizon and multi-directional control tasks?
• RQ4 How does the gradual adaptation of the meta-model influence training stability and final performance?
• RQ5 What is the relative contribution of the meta-model versus the base model in achieving sample efficiency and robustness?

Key findings

• M3PO outperforms PEARL and M2PO in long-term training on Halfcheetah-pier, Walker2D-randomparams, and Humanoid-direc, achieving comparable or better final returns.
• The gradual adaptation schedule (M3PO-h) improves long-term performance, matching or exceeding PEARL’s performance in all three environments.
• M3PO achieves directional control in Humanoid-direc, while M2PO fails and learns a 'standing' policy due to lack of adaptive dynamics.
• M2PO shows lower TD-errors during early training platetus, suggesting its non-adaptive model provides more stable optimization in some cases.
• The use of a meta-model significantly improves performance in Halfcheetah-fwd-bwd and directional control, demonstrating the value of dynamic adaptation.
• The theoretical analysis confirms that branched rollouts offer tighter performance guarantees than full-model rollouts in meta-RL, justifying their use in M3PO.