[Paper Review] Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

Vladimir Feinberg, Alvin Wan+4 more|arXiv (Cornell University)|Feb 28, 2018

Reinforcement Learning in Robotics14 references168 citations

TL;DR

The paper introduces model-based value expansion (MVE), a hybrid approach that uses a learned dynamics model for short-horizon imagination to improve value estimates and accelerate learning in model-free RL, with a TD-k trick to mitigate distribution mismatch.

ABSTRACT

Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of continuous control tasks. Unfortunately, they rely on heuristics that limit usage of the dynamics model. We present model-based value expansion, which controls for uncertainty in the model by only allowing imagination to fixed depth. By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.

Motivation & Objective

• Reduce sample complexity of model-free RL by leveraging learned dynamics models.
• Improve the accuracy of value estimates through short-horizon model-based rollouts.
• Provide a practical, non-differentiable model-compatible framework for continuous control.
• Address distribution mismatch when using imagined data and propose fixes.
• Demonstrate that restricted, horizon-limited model use can outperform pure model-free and prior MB-MF hybrids.

Proposed method

• Define H-step Model Value Expansion (V̂_H) that sums imagined rewards for H steps and uses a value tail V̂ at step H.
• Assume an approximate dynamics model f̂ to simulate future states and rewards under policy π.
• Decompose value estimation into a near-term model-based component and a far-term model-free tail to avoid excessive model reliance.
• Introduce a distribution-mismatch remedy by constructing training distributions that approximate a fixed point of f^π and applying a TD-k strategy.
• Integrate MVE into a deep actor-critic framework (DDPG-like) where the critic targets use MVE-improved estimates and transitions are sampled from a fixed distribution ν.
• Allow non-differentiable dynamics and use forward predictions without requiring differentiability of the model.

Experimental results

Research questions

• RQ1 Does MVE improve the accuracy of Q^π estimates in continuous control tasks?
• RQ2 Can short-horizon model-based rollouts reduce sample complexity in model-free RL without relying on differentiable dynamics?
• RQ3 How does the TD-k trick affect training stability and performance when using imagined data?
• RQ4 What are the effects of training data distribution mismatch when using imagined states for critic learning?

Key findings

• MVE improves value estimation quality and speeds up learning in dense-reward continuous control tasks.
• The TD-k trick mitigates distribution-mismatch issues and enables larger model horizons to yield gains.
• Compared to imagination-buffer baselines, MVE with careful horizon and TD-k design outperforms them in experiments.
• MVE provides improved Q-values and faster mastery, with the model-based portion limited to a short horizon to balance accuracy and robustness.