[Paper Review] Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables

Kate Rakelly, Aurick Zhou+3 more|arXiv (Cornell University)|Mar 19, 2019

Reinforcement Learning in Robotics227 citations

TL;DR

PEARL introduces an off-policy meta-reinforcement learning algorithm that uses probabilistic latent context to rapidly adapt to new tasks, achieving 20-100X meta-training sample efficiency and improved asymptotic performance across six continuous-control benchmarks.

ABSTRACT

Deep reinforcement learning algorithms require large amounts of experience to learn an individual task. While in principle meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of experience, several major challenges preclude their practicality. Current methods rely heavily on on-policy experience, limiting their sample efficiency. The also lack mechanisms to reason about task uncertainty when adapting to new tasks, limiting their effectiveness in sparse reward problems. In this paper, we address these challenges by developing an off-policy meta-RL algorithm that disentangles task inference and control. In our approach, we perform online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience. This probabilistic interpretation enables posterior sampling for structured and efficient exploration. We demonstrate how to integrate these task variables with off-policy RL algorithms to achieve both meta-training and adaptation efficiency. Our method outperforms prior algorithms in sample efficiency by 20-100X as well as in asymptotic performance on several meta-RL benchmarks.

Motivation & Objective

• Reduce sample inefficiency in meta-RL by shifting to off-policy training.
• Infer task uncertainty online via probabilistic latent context to enable structured exploration.
• Decouple task inference from control to leverage off-policy RL with efficient meta-training.
• Enable rapid trajectory-level adaptation at test time through posterior sampling over task context.

Proposed method

• Introduce a probabilistic latent context Z that conditions the policy: π(a|s,z).
• Use an amortized variational encoder qφ(z|c) to infer the posterior p(z|c) from recent experience c.
• Model the context with a permutation-invariant encoder as a product of Gaussian factors over individual transitions.
• Employ posterior sampling by drawing z from qφ(z|c) at test time and holding it fixed for an episode to enable structured exploration.
• Train the encoder separately from the actor/critic using off-policy data, with the actor/critic updated via SAC-like objectives.
• Ground the approach in an off-policy meta-RL framework by decoupling context sampling from RL data collection (Algorithm 1).

Experimental results

Research questions

• RQ1 How can we perform efficient off-policy meta-training for RL while maintaining rapid adaptation to new tasks?
• RQ2 Does probabilistic latent context enable effective, temporally extended exploration in sparse-reward, unseen-tasks settings?
• RQ3 To what extent does decoupling task inference from control improve sample efficiency and final performance in meta-RL?
• RQ4 How does posterior sampling over task contexts compare to prior methods for exploration in meta-RL?
• RQ5 What are the critical data-sampling strategies for training the encoder and the policy in off-policy meta-RL?

Key findings

• PEARL achieves 20-100X improvement in meta-training sample efficiency over prior meta-RL methods.
• PEARL attains substantial increases in asymptotic performance across six continuous-control meta-learning benchmarks.
• Posterior sampling over latent task contexts enables temporally extended exploration, aiding rapid adaptation in sparse-reward tasks.
• Decoupling context inference from the actor-critic enables effective off-policy meta-training with minimal distribution mismatch between meta-training and meta-testing.
• A probabilistic latent context is crucial for exploration under sparse rewards, outperforming deterministic-context variants and prior methods in sparse navigation tasks.