[Paper Review] Temporal Difference Models: Model-Free Deep RL for Model-Based Control

Vitchyr H. Pong, Shixiang Gu+2 more|arXiv (Cornell University)|Feb 25, 2018

Reinforcement Learning in Robotics27 references44 citations

TL;DR

Temporal Difference Models (TDMs) are a family of goal-conditioned value functions trained with model-free learning that act as implicit models for planning, achieving model-based efficiency with model-free asymptotics.

ABSTRACT

Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for solving challenging real-world problems, even with off-policy algorithms such as Q-learning. A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples. Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias. We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control. TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods. Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free methods.

Motivation & Objective

• Motivate the need to combine model-based planning efficiency with model-free asymptotic performance.
• Introduce temporal difference models as a bridge between model-free and model-based RL.
• Show how relabeling and multi-step horizons enable efficient off-policy learning of TDMs.
• Demonstrate that TDMs achieve superior sample efficiency and final performance on continuous control tasks.

Proposed method

• Define TDMs as goal-conditioned Q-functions with an horizon parameter tau.
• Use distance-based rewards D(s, s_g) and a horizon-aware Q-learning recurrence Q(s, a, s_g, tau).
• Relabel experiences with different goals g and horizons tau to maximize data efficiency.
• Extract policies via MPC-like planning using the learned Q-function or via direct Q-based action selection.
• Optionally use vector-valued (per-dimension) distance rewards to improve supervision.
• Provide an algorithm (Algorithm 1) for off-policy training with replay and target networks.

Experimental results

Research questions

• RQ1 Can a goal-conditioned value function with a planning horizon tau interpolate between model-based and model-free learning?
• RQ2 Do temporal difference models achieve better sample efficiency than pure model-based or model-free methods on continuous control tasks?
• RQ3 Does relabeling with different goals and horizons improve data efficiency in off-policy learning?
• RQ4 How can TDMs be used for practical policy extraction in planning or direct control?
• RQ5 What is the impact of vectorized distance rewards and horizon parameter choices on performance?

Key findings

• TDMs provide substantially better sample efficiency than state-of-the-art model-free methods across several continuous control tasks.
• TDMs outperform pure model-based methods in final performance on harder tasks due to reduced model bias.
• Relabeling with different goals and horizons yields strong data efficiency gains, enabling rapid learning of short- and long-horizon behaviors.
• Vectorized (per-dimension) distance rewards significantly boost sample efficiency over scalar rewards.
• TDMs scale to real-world robotics, showing improved learning efficiency on a Sawyer 7-DoF arm compared to DDPG.
• Ablation shows horizon tau controls interpolation between model-based and model-free regimes, and vectorization improves learning over scalar rewards.