[Paper Review] Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Kurtland Chua, Roberto Calandra+2 more|arXiv (Cornell University)|May 30, 2018

Reinforcement Learning in RoboticsComputer Science49 references185 citations

TL;DR

PESTS uses uncertainty-aware probabilistic neural network dynamics with trajectory sampling to achieve model-based RL that matches model-free asymptotic performance with far fewer samples on standard benchmarks.

ABSTRACT

Model-based reinforcement learning (RL) algorithms can attain excellent sample efficiency, but often lag behind the best model-free algorithms in terms of asymptotic performance. This is especially true with high-capacity parametric function approximators, such as deep networks. In this paper, we study how to bridge this gap, by employing uncertainty-aware dynamics models. We propose a new algorithm called probabilistic ensembles with trajectory sampling (PETS) that combines uncertainty-aware deep network dynamics models with sampling-based uncertainty propagation. Our comparison to state-of-the-art model-based and model-free deep RL algorithms shows that our approach matches the asymptotic performance of model-free algorithms on several challenging benchmark tasks, while requiring significantly fewer samples (e.g., 8 and 125 times fewer samples than Soft Actor Critic and Proximal Policy Optimization respectively on the half-cheetah task).

Motivation & Objective

• Motivate sample-efficient reinforcement learning by integrating uncertainty into deep model-based dynamics.
• Develop an uncertainty-aware dynamics model that combines neural networks with ensembles to capture aleatoric and epistemic uncertainty.
• Propose trajectory sampling within a model-predictive control framework to propagate uncertainty during planning.
• Demonstrate that the method achieves asymptotic performance comparable to model-free methods with substantially fewer samples.

Proposed method

• Propose probabilistic ensembles with trajectory sampling (PETS) that uses an ensemble of bootstrapped probabilistic neural networks to model dynamics.
• Separate and capture aleatoric uncertainty (via probabilistic nets) and epistemic uncertainty (via ensembles).
• Use trajectory sampling with multiple particles and bootstrap indices to propagate uncertainty through time.
• Apply model predictive control with cross-entropy method (CEM) to optimize action sequences at each step based on expected rewards over predicted trajectories.
• Train the model on data and update via trial-based data collection, evaluating against model-free and GP-based baselines.

Experimental results

Research questions

• RQ1 Can uncertainty-aware deep neural network dynamics models close the performance gap between model-based and model-free RL while reducing sample complexity?
• RQ2 What is the impact of modeling aleatoric vs epistemic uncertainty on planning and learning efficiency?
• RQ3 How do different uncertainty propagation methods affect planning quality and data efficiency?
• RQ4 Are ensembles of probabilistic networks practical and scalable to high-dimensional control tasks?
• RQ5 Do PETS-style methods achieve asymptotic performance comparable to state-of-the-art model-free methods on standard benchmarks with significantly fewer samples?

Key findings

• PETS matches the asymptotic performance of state-of-the-art model-free methods on benchmark tasks while using far fewer samples.
• Eight times fewer samples are needed on half-cheetah compared to Soft Actor Critic in their experiments.
• PETS outperforms prior model-based methods and some model-free baselines in several tasks, demonstrating the value of incorporating uncertainty into modeling and planning.
• The combination of probabilistic ensembles and trajectory sampling provides robust handling of both aleatoric and epistemic uncertainty, improving data efficiency.
• Moment matching can be competitive in low-dimensional tasks but is less reliable in higher-dimensional tasks like half-cheetah.
• Model choice (PE ensembles) has a larger impact on performance than the specific uncertainty propagation technique used.