[Paper Review] Dream to Control: Learning Behaviors by Latent Imagination

Danijar Hafner, Timothy Lillicrap+2 more|arXiv (Cornell University)|Dec 3, 2019

Reinforcement Learning in Robotics65 references137 citations

TL;DR

Dreamer learns long-horizon behaviors from images by latent imagination within a learned world model, backpropagating analytic gradients through imagined trajectories to optimize policy and values, achieving data-efficient visual control on 20 tasks.

ABSTRACT

Learned world models summarize an agent's experience to facilitate learning complex behaviors. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks from images purely by latent imagination. We efficiently learn behaviors by propagating analytic gradients of learned state values back through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.

Motivation & Objective

• Motivate and enable long-horizon control from high-dimensional visual observations without online planning for every step.
• Develop a world-model-based agent that learns both actions and state values in latent space to overcome horizon limitations.
• Leverage analytic gradients through latent dynamics to train a policy using imagined trajectories.
• Demonstrate data efficiency and performance gains on challenging visual control tasks.
• Compare representation learning objectives and horizon robustness to baseline methods.

Proposed method

• Learn a latent dynamics model from past experience to predict future rewards in a compact state space.
• Train an action model (policy) and a value model in the latent space to maximize imagined multi-step returns.
• Backpropagate analytic gradients through the latent dynamics to update the policy and value predictions.
• Use a reparameterized action sampler and Bellman-consistent value targets in imagination (V_lambda).
• Predict imagined trajectories starting from real latent states and optimize over them to improve real-world policy.
• Evaluate Dreamer on the DeepMind Control Suite with image inputs and compare to model-free and model-based baselines.

Experimental results

Research questions

• RQ1 Can long-horizon control be learned from pixels by latent imagination without online planning on every step?
• RQ2 Does predicting both actions and state values in latent space enable robust, farsighted policies beyond a fixed imagination horizon?
• RQ3 Which representation learning objective (reconstruction, reward prediction, or contrastive) best supports Dreamer’s performance?
• RQ4 How does Dreamer compare to online planning and model-free baselines in data efficiency and final performance on visual tasks?
• RQ5 Is Dreamer applicable to discrete actions and environments with early termination?

Key findings

• Dreamer achieves higher data efficiency, computation time efficiency, and final performance than prior model-based and model-free agents on 20 visual control tasks.
• After 5e6 environment steps, Dreamer averages 823 across tasks, surpassing PlaNet and approaching or exceeding top model-free D4PG performance given more steps.
• Learning a value model in imagination makes Dreamer robust to the imagination horizon and improves long-horizon credit assignment.
• Pixel reconstruction as a representation learning objective yields best performance on most tasks among the evaluated options.
• Dreamer solves long-horizon tasks (e.g., acrobot, hopper) that are difficult for horizon-limited imagination or reward-only approaches.
• Dreamer is applicable to both continuous and discrete actions and can cope with early termination in evaluation.