[Paper Review] Model-Ensemble Trust-Region Policy Optimization

Thanard Kurutach, Ignasi Clavera+3 more|arXiv (Cornell University)|Feb 28, 2018

Reinforcement Learning in Robotics31 references218 citations

TL;DR

ME-TRPO uses a model ensemble and trust-region policy optimization to achieve state-of-the-art sample efficiency in model-based deep RL, matching model-free performance with ~100x less data.

ABSTRACT

Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity, which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and to date have succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared to model-free deep RL methods on challenging continuous control benchmark tasks.

Motivation & Objective

• Motivate reducing sample complexity in reinforcement learning by leveraging learned dynamics models.
• Investigate instability in vanilla model-based deep RL when using neural networks for both model and policy.
• Develop a robust training framework that maintains model uncertainty and stabilizes policy updates.
• Demonstrate that ensemble models and TRPO improve stability and performance across challenging tasks.

Proposed method

• Introduce a model ensemble of neural dynamics predictors to capture uncertainty.
• Train all models on collected real-data and sample fictitious roll-outs from the ensemble.
• Replace backpropagation through time with likelihood-ratio gradient estimators for policy optimization.
• Use Trust Region Policy Optimization (TRPO) to constrain policy updates over imagined trajectories.
• Validate policy updates by monitoring performance across all ensemble models and stop when improvement falls below a threshold.
• Collect real-environment data iteratively to refine the ensemble and re-train the policy.

Experimental results

Research questions

• RQ1 How does model-based RL with neural dynamics compare to state-of-the-art model-free methods in sample efficiency and final performance?
• RQ2 Can an ensemble of dynamics models regularize policy learning and mitigate model bias?
• RQ3 Does replacing BPTT with likelihood-ratio gradient estimation stabilize training for long-horizon tasks?
• RQ4 How does TRPO perform within a model-based, ensemble-regularized framework compared to other policy gradient methods?

Key findings

• The method matches model-free performance with approximately 100x less real-world data.
• Vanilla model-based deep RL suffers from instability and model bias, particularly over long horizons.
• Using an ensemble of dynamics models provides regularization and reduces overfitting to any single model.
• Replacing BPTT with TRPO yields more stable and effective policy learning.
• Increasing the number of models in the ensemble improves performance, especially on complex tasks like Half-Cheetah and Ant.