[Paper Review] Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

Jacob Buckman, Danijar Hafner+3 more|arXiv (Cornell University)|Jul 4, 2018

Reinforcement Learning in RoboticsComputer Science26 references97 citations

TL;DR

STEVE combines model-based rollouts with model-free TD learning using an ensemble to estimate uncertainty and adapt rollout horizons, achieving high sample efficiency without model bias degradation.

ABSTRACT

Integrating model-free and model-based approaches in reinforcement learning has the potential to achieve the high performance of model-free algorithms with low sample complexity. However, this is difficult because an imperfect dynamics model can degrade the performance of the learning algorithm, and in sufficiently complex environments, the dynamics model will almost always be imperfect. As a result, a key challenge is to combine model-based approaches with model-free learning in such a way that errors in the model do not degrade performance. We propose stochastic ensemble value expansion (STEVE), a novel model-based technique that addresses this issue. By dynamically interpolating between model rollouts of various horizon lengths for each individual example, STEVE ensures that the model is only utilized when doing so does not introduce significant errors. Our approach outperforms model-free baselines on challenging continuous control benchmarks with an order-of-magnitude increase in sample efficiency, and in contrast to previous model-based approaches, performance does not degrade in complex environments.

Motivation & Objective

• Motivate reducing sample complexity in reinforcement learning by integrating model-based and model-free approaches.
• Address model-bias problems when using imperfect dynamics by adaptively leveraging rollouts.
• Develop an uncertainty-aware method to select rollout horizons per example to minimize target error.

Proposed method

• Use an ensemble of Q-functions, reward models, and dynamics models to estimate uncertainty.
• Unroll the learned model for multiple horizons and compute multiple candidate TD targets.
• Compute STEVE target as an inverse-variance weighted blend of candidate targets across horizons (0 to H).
• Replace TD targets in TD-learning with the STEVE target in training the Q-function.
• Provide theoretical justification via bias-variance decomposition and approximate minimization of target variance.
• Demonstrate and compare performance on continuous control benchmarks with a DDPG backbone.

Experimental results

Research questions

• RQ1 Can stochastic ensembles and uncertainty-guided horizon selection improve the stability and efficiency of model-based value expansion?
• RQ2 Does STEVE outperform purely model-free methods and standard MVE under model inaccuracy across challenging continuous control tasks?
• RQ3 How does dynamic horizon weighting affect sample efficiency and robustness to model error?

Key findings

• STEVE significantly improves sample efficiency over model-free baselines on challenging continuous control tasks.
• STEVE remains robust to model imperfections, unlike vanilla MVE which can diverge with noisy models.
• Averaging targets with inverse-variance weighting reduces target error by leveraging uncertainty estimates.
• Ablations show uncertainty-aware reweighting, not just model-enlarged ensembles, drives performance gains.
• Wall-clock experiments indicate STEVE is competitive with model-free methods when parallelized, due to better sample efficiency.