[Paper Review] The Predictron: End-To-End Learning and Planning

David Silver, Hado van Hasselt+9 more|arXiv (Cornell University)|Dec 28, 2016

Reinforcement Learning in Robotics23 references88 citations

TL;DR

The Predictron is an end-to-end differentiable deep learning architecture that integrates planning and learning by unrolling an abstract Markov reward process (MRP) over imagined trajectories to predict value functions. It outperforms conventional model-free deep networks in maze and pool simulation tasks by learning self-consistent value predictions across multiple planning depths, achieving significantly higher accuracy and robustness to architectural choices.

ABSTRACT

One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. The predictron consists of a fully abstract model, represented by a Markov reward process, that can be rolled forward multiple "imagined" planning steps. Each forward pass of the predictron accumulates internal rewards and values over multiple planning depths. The predictron is trained end-to-end so as to make these accumulated values accurately approximate the true value function. We applied the predictron to procedurally generated random mazes and a simulator for the game of pool. The predictron yielded significantly more accurate predictions than conventional deep neural network architectures.

Motivation & Objective

• To address the disconnect between separately trained models and planners in model-based reinforcement learning by integrating learning and planning into a single end-to-end framework.
• To develop a model that generalizes across multiple prediction tasks by training on diverse pseudo-reward functions and discount factors.
• To improve value prediction accuracy in uncontrolled environments by learning abstract, non-pixel-perfect internal representations that still align with true environment outcomes.
• To enable robust planning through a differentiable architecture that accumulates rewards and values across multiple imagined planning steps.
• To demonstrate that end-to-end training of abstract models yields superior performance compared to conventional deep networks in complex control tasks.

Proposed method

• The Predictron uses a deep neural network with a recurrent MRP core that unrolls over multiple imagined planning steps, accumulating rewards and value estimates.
• It computes value predictions at multiple planning depths within a single forward pass, combining them into an ensemble output.
• The model is trained end-to-end using a loss that encourages self-consistency across predictions at different planning depths.
• It supports vector-valued rewards and diagonal-matrix discounts, generalizing to multiple prediction tasks via pseudo-reward functions.
• The architecture uses a learnable attention-like weighting (λ) across planning steps to dynamically combine predictions, enabling adaptive depth per trajectory.
• The internal model is fully abstract—its state, action, reward, and discount spaces need not correspond to the real environment, as long as value predictions remain consistent with true returns.

Experimental results

Research questions

• RQ1 Can an end-to-end trained abstract model outperform conventional deep networks in value prediction tasks?
• RQ2 Does integrating planning and learning through a single differentiable architecture improve prediction accuracy and robustness?
• RQ3 Can a model trained on diverse pseudo-reward functions generalize across multiple prediction tasks without overfitting to a single objective?
• RQ4 How does the use of self-consistent value predictions across multiple planning depths affect performance?
• RQ5 Can abstract internal representations, even if non-human-interpretable, still yield accurate value estimates that align with real environment outcomes?

Key findings

• In procedurally generated mazes, the Predictron achieved significantly more accurate value predictions than state-of-the-art model-free deep networks.
• In the simulated pool domain, the Predictron selected initial conditions that led to 27 pocketed balls in 50 episodes, compared to only 10 with a similarly deep convolutional network.
• The Predictron demonstrated much greater robustness to architectural choices such as network depth, outperforming conventional architectures consistently across variations.
• The use of self-consistent value predictions across planning depths provided an additional learning signal that improved overall accuracy.
• The model's ability to plan over abstract, non-pixel-perfect representations enabled more effective value estimation than pixel-level reconstruction-based models.
• The video demonstration of rollouts selected by the Predictron confirmed its superior decision-making capability in complex environments.