[Paper Review] Model-Based Reinforcement Learning via Meta-Policy Optimization

Ignasi Clavera, Jonas Rothfuss+4 more|arXiv (Cornell University)|Sep 14, 2018

Machine Learning and Data ClassificationComputer Science117 citations

TL;DR

MB-MPO learns an ensemble of dynamics models and meta-learns a policy that can quickly adapt to any model with one gradient step, achieving model-free-level asymptotic performance with far less data.

ABSTRACT

Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization (MB-MPO), an approach that foregoes the strong reliance on accurate learned dynamics models. Using an ensemble of learned dynamic models, MB-MPO meta-learns a policy that can quickly adapt to any model in the ensemble with one policy gradient step. This steers the meta-policy towards internalizing consistent dynamics predictions among the ensemble while shifting the burden of behaving optimally w.r.t. the model discrepancies towards the adaptation step. Our experiments show that MB-MPO is more robust to model imperfections than previous model-based approaches. Finally, we demonstrate that our approach is able to match the asymptotic performance of model-free methods while requiring significantly less experience.

Motivation & Objective

• Motivate data-efficient reinforcement learning for complex control tasks.
• Address model-bias by meta-learning a policy that adapts to a distribution of dynamics models.
• Enable robust performance and fast adaptation using an ensemble of learned dynamics.
• Demonstrate that meta-optimization can match model-free asymptotic performance with far less experience.

Proposed method

• Learn an ensemble of neural network dynamics models predicting state changes Δs.
• Formulate a distribution of MDPs using the ensemble and apply gradient-based meta-learning to optimize a policy across these models.
• Use the MAML objective to train a pre-update policy θ that can be quickly adapted to each model k via one gradient step: θ′k = θ + α ∇θ Jk(θ).
• Estimate Jk(θ) by simulating imagined trajectories in the k-th model with the pre-update policy.
• Perform outer policy optimization with a policy-gradient method (TRPO) to improve θ toward the meta-objective across all models.
• Collect real-environment data to train the ensemble and periodically update models with aggregated data.

Experimental results

Research questions

• RQ1 Can MB-MPO achieve asymptotic performance comparable to model-free methods while requiring significantly less real-world data?
• RQ2 Does meta-learning over an ensemble of learned dynamics models improve robustness to model bias compared to traditional model-based approaches?
• RQ3 How does policy adaptation within each model affect data collection efficiency and learning speed?
• RQ4 Is MB-MPO robust to biased or imperfect dynamic models and long-horizon predictions?

Key findings

• MB-MPO matches the asymptotic performance of state-of-the-art model-free methods using 10–100× less data across six Mujoco tasks.
• MB-MPO outperforms recent model-based baselines (ME-TRPO, MB-MPC) in convergence speed and final performance, especially on tasks requiring long-horizon planning.
• There is a strong positive correlation between model ensemble uncertainty and policy plasticity (KL divergence between pre- and post-update policies).
• MB-MPO remains robust under biased/noisy dynamics models and can learn with strong model imperfections where ME-TRPO fails.
• Adapting policies to each ensemble model and then updating the meta-policy yields faster convergence and better sample efficiency than learning from scratch or without adaptation.
• Real-world data needs to reach high performance in easier domains within ~30 minutes and up to ~90 minutes in harder domains.