[Paper Review] Sample-efficient reinforcement learning using deep Gaussian processes

Charles W. L. Gadd, Markus Heinonen+2 more|arXiv (Cornell University)|Nov 2, 2020

Gaussian Processes and Bayesian Inference25 references4 citations

TL;DR

This paper proposes a deep Gaussian process (DGP)-based model for sample-efficient reinforcement learning by combining the uncertainty quantification of Gaussian processes with the representational capacity of deep learning. The method enables fast, stable learning in complex control tasks—demonstrating superior early sample efficiency, including solving the half-cheetah task with only one episode of interaction, outperforming prior GP-based and neural network-based baselines.

ABSTRACT

Reinforcement learning provides a framework for learning to control which actions to take towards completing a task through trial-and-error. In many applications observing interactions is costly, necessitating sample-efficient learning. In model-based reinforcement learning efficiency is improved by learning to simulate the world dynamics. The challenge is that model inaccuracies rapidly accumulate over planned trajectories. We introduce deep Gaussian processes where the depth of the compositions introduces model complexity while incorporating prior knowledge on the dynamics brings smoothness and structure. Our approach is able to sample a Bayesian posterior over trajectories. We demonstrate highly improved early sample-efficiency over competing methods. This is shown across a number of continuous control tasks, including the half-cheetah whose contact dynamics have previously posed an insurmountable problem for earlier sample-efficient Gaussian process based models.

Motivation & Objective

• To address the challenge of sample inefficiency in reinforcement learning, especially in costly real-world environments.
• To combine the uncertainty-aware modeling of Gaussian processes with the representational power of deep neural networks.
• To enable fast, reliable learning in non-stationary and complex dynamical systems using minimal interaction data.
• To overcome the limitations of previous GP-based models that fail on complex tasks like half-cheetah due to insufficient model capacity.
• To demonstrate improved sample efficiency and robustness in benchmark continuous control environments.

Proposed method

• The method employs deep Gaussian processes (DGPs) to model environment dynamics, with multiple layers of GP regressors forming a hierarchical composition.
• It uses Bayesian inference to compute a posterior over trajectories, enabling uncertainty propagation over time.
• The model incorporates prior knowledge about system smoothness and dynamics through structured kernel design.
• A novel inference scheme enables efficient approximate posterior sampling over trajectories, supporting model-based planning.
• The approach uses deep composition to increase model capacity while retaining principled uncertainty estimates.
• It integrates with model predictive control (MPC) for action selection, using sampled trajectories to optimize reward.

Experimental results

Research questions

• RQ1 Can deep Gaussian processes improve sample efficiency in model-based reinforcement learning compared to standard GP and neural network baselines?
• RQ2 Can a deep GP model effectively capture complex, non-smooth dynamics such as those in contact-rich environments like half-cheetah?
• RQ3 Does incorporating prior knowledge about system smoothness and structure enhance learning stability and performance in low-data regimes?
• RQ4 How does the DGP-based method compare in early learning speed and final performance across multiple continuous control benchmarks?
• RQ5 Can the method achieve strong performance with only a single episode of interaction, unlike prior GP-based approaches?

Key findings

• The DGP-MPC method achieved significantly faster initial learning rates than competing methods, accumulating positive reward from the very first episode in the half-cheetah environment.
• In the half-cheetah task, GP-MM and GP-DS failed to accumulate any reward, while PETS required substantially more interactions to reach comparable performance.
• The method demonstrated consistent and high mean reward across the first 10 episodes, outperforming all baselines in learning speed and stability.
• The DGP-based approach achieved strong performance with only a few episodes, showing superior early sample efficiency compared to PETS and other GP-based models.
• The model maintained robust uncertainty estimates even in complex, non-stationary dynamics, enabling reliable planning from minimal data.
• The method's performance was stable across 10 independent random seeds, with low variability, indicating consistent learning behavior.