[Paper Review] Learning model-based planning from scratch

Razvan Pascanu, Yujia Li+8 more|arXiv (Cornell University)|Jul 19, 2017

Artificial Intelligence in Games21 references78 citations

TL;DR

This work presents the Imagination-based Planner (IBP), a fully learnable model-based agent that constructs, evaluates, and executes plans through imagined rollouts, demonstrated on continuous control and discrete mazes.

ABSTRACT

Conventional wisdom holds that model-based planning is a powerful approach to sequential decision-making. It is often very challenging in practice, however, because while a model can be used to evaluate a plan, it does not prescribe how to construct a plan. Here we introduce the "Imagination-based Planner", the first model-based, sequential decision-making agent that can learn to construct, evaluate, and execute plans. Before any action, it can perform a variable number of imagination steps, which involve proposing an imagined action and evaluating it with its model-based imagination. All imagined actions and outcomes are aggregated, iteratively, into a "plan context" which conditions future real and imagined actions. The agent can even decide how to imagine: testing out alternative imagined actions, chaining sequences of actions together, or building a more complex "imagination tree" by navigating flexibly among the previously imagined states using a learned policy. And our agent can learn to plan economically, jointly optimizing for external rewards and computational costs associated with using its imagination. We show that our architecture can learn to solve a challenging continuous control problem, and also learn elaborate planning strategies in a discrete maze-solving task. Our work opens a new direction toward learning the components of a model-based planning system and how to use them.

Motivation & Objective

• Motivate model-based planning that learns how to plan, not just how to act, by integrating imagination into planning.
• Demonstrate a fully differentiable architecture that learns when to imagine, how to imagine, and how to aggregate imagined outcomes into plans.
• Showcase IBP on challenging continuous control and discrete maze tasks to learn planning strategies tailored to the task.
• Explore how imagination incurs computational costs and how agents balance external rewards with internal resource use.

Proposed method

• Define the IBP with four components: a manager that decides to act or imagine, a controller that proposes actions, an imagination model that predicts outcomes, and a memory that aggregates internal and external data.
• Represent planning as iterative cycles where each step either executes an action or imagines consequences, building a plan context from imagined and real experiences.
• Implement three imagination strategies (1-step, n-step, and imagination-tree) that determine from which state to imagine and how to chain imagined actions.
• Train the model end-to-end with two losses: external task loss (fuel cost + final distance to target) and internal resource cost (imagination cost), using gradient-based optimization and REINFORCE for discrete routing.
• Utilize an interaction-network as the world model for both imagining dynamics and predicting real-state transitions, and optimize using SVG-based gradients for continuous actions.

Experimental results

Research questions

• RQ1 Can a fully learnable model-based planner construct, evaluate, and execute plans using imagined rollouts?
• RQ2 How should an agent balance external task performance with internal computation costs when planning?
• RQ3 What planning strategies (1-step, multi-step, or tree-like imaginations) are most effective in continuous and discrete tasks?
• RQ4 Can learned imagination strategies generalize across tasks and handle state ambiguities in discrete mazes?

Key findings

• IBP learns to use model-based imagination to improve performance in a challenging continuous control task.
• Imagination helps the agent test alternatives, chain actions, and build complex imagination trees for planning.
• Increasing allowed imagination steps reduces task loss, showing the value of looking ahead in planning.
• In discrete mazes, the imagination-tree strategy outperforms 1-step and n-step strategies, approaching optimal rewards in multiple-goal scenarios.