[Paper Review] Meta Reinforcement Learning with Latent Variable Gaussian Processes

Steindór Sæmundsson, Katja Hofmann+1 more|arXiv (Cornell University)|Mar 20, 2018

Gaussian Processes and Bayesian Inference34 references105 citations

TL;DR

The paper introduces a model-based meta-RL approach using a latent-variable Gaussian Process to capture task variation, enabling online latent inference and MPC planning, achieving substantial data efficiency and transfer to new tasks.

ABSTRACT

Learning from small data sets is critical in many practical applications where data collection is time consuming or expensive, e.g., robotics, animal experiments or drug design. Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Often, this relationship between tasks is hard coded or relies in some other way on human expertise. In this paper, we frame meta learning as a hierarchical latent variable model and infer the relationship between tasks automatically from data. We apply our framework in a model-based reinforcement learning setting and show that our meta-learning model effectively generalizes to novel tasks by identifying how new tasks relate to prior ones from minimal data. This results in up to a 60% reduction in the average interaction time needed to solve tasks compared to strong baselines.

Motivation & Objective

• Frame meta-learning within a probabilistic, model-based RL setting to transfer knowledge across related dynamical systems.
• Learn a dynamics model conditioned on a latent representation of task specifications.
• Use variational inference and sparse GP methods to enable online latent-variable adaptation and data-efficient planning.
• Demonstrate transfer and few-shot learning effectiveness on cart-pole and double-pendulum tasks.

Proposed method

• Model the dynamics with Gaussian Processes conditioned on a latent task variable h_p, forming x_{t+1} = f(x_t, c_t, h_p) + ε where ε ~ N(0,E).
• Adopt MPC to plan control sequences by minimizing an expected finite-horizon cost with Gaussian approximations of state distributions.
• Represent the global dynamics via a latent embedding h_p drawn from p(h_p) and infer h_p online from data collected during trajectories.
• Use variational inference with a sparse GP (inducing points) to approximate the posterior over the GP f and the latent variables H, enabling scalable learning across many tasks.
• Train with a hierarchical Bayesian model, optimizing an evidence lower bound (ELBO) that includes KL terms for q(H) and q(U).
• Leverage mini-batch stochastic optimization (Adam) and online latent-variable updates for real-time adaptation.

Experimental results

Research questions

• RQ1 Does conditioning the GP on a latent variable disentangle global versus task-specific dynamics?
• RQ2 Can the latent embeddings generalize to unseen task configurations to improve predictive performance?
• RQ3 Does the ML-GP approach improve data efficiency in model-based RL across related tasks?
• RQ4 How well do the inferred latent embeddings reflect meaningful task structure (e.g., length and mass in pendulum systems)?

Key findings

• ML-GP outperforms sparse GP baselines and standard GPs in predictive RMSE and NLL on unseen tasks.
• The learned latent embeddings disentangle task-specific variation from global dynamics, with interpretable structure in latent space.
• In data-efficient RL, ML-GP achieves faster learning and lower interaction time than independently trained models (SGP-I) and outperforms SGP-ML in adapting to new dynamics.
• Across cart-pole and double-pendulum tasks, ML-GP reduces total training and testing time, and shows superior transfer/few-shot performance.
• Latent embeddings inferred from limited observations are informative for predicting future states and guiding MPC.