[Paper Review] Recurrent World Models Facilitate Policy Evolution

David Ha, Jürgen Schmidhuber|arXiv (Cornell University)|Sep 4, 2018

International Development and Aid407 citations

TL;DR

The paper trains a compact controller (C) via evolution to act within a learned world model composed of a VAE-based visual encoder (V) and an MDN-RNN dynamics model (M); this world model enables solving CarRacing-v0 and learning inside a generated VizDoom environment, with transfer back to the real environments.

ABSTRACT

A generative recurrent neural network is quickly trained in an unsupervised manner to model popular reinforcement learning environments through compressed spatio-temporal representations. The world model's extracted features are fed into compact and simple policies trained by evolution, achieving state of the art results in various environments. We also train our agent entirely inside of an environment generated by its own internal world model, and transfer this policy back into the actual environment. Interactive version of paper at https://worldmodels.github.io

Motivation & Objective

• Demonstrate unsupervised training of a world model (V, M) that compresses visual input and predicts futures.
• Show that a simple controller (C) can be evolved to maximize rewards using features from V and M.
• Evaluate the approach on CarRacing-v0 and VizDoom to establish performance benchmarks.
• Explore training policies inside generated environments and transferring them back to real environments.

Proposed method

• Use a convolutional VAE (V) to encode each frame to a latent z.
• Model the next latent z_{t+1} as a distribution P(z_{t+1}|a_t,z_t,h_t) with an MDN-RNN (M).
• Treat the controller C as a small linear model mapping [z_t, h_t] to action a_t and train it with CMA-ES.
• Adjust a temperature parameter τ of M to control environment uncertainty and prevent exploitative policies.
• Train V and M separately (unsupervised) and evolve C to maximize expected cumulative reward.
• Demonstrate a fully latent-space RL setup where C can be trained in a generated environment and transferred back to the real one.

Experimental results

Research questions

• RQ1 Can a compact controller trained via evolution maximize rewards when guided by a learned, predictive world model?
• RQ2 Does combining V (compression) and M (prediction) with a simple C enable solving high-dimensional control tasks from pixels?
• RQ3 Can agents trained inside a world-model-generated environment transfer policies back to the actual environment?
• RQ4 How does varying the uncertainty (τ) in the world model affect policy exploitation and transferability?

Key findings

• On CarRacing-v0, the Full World Model with V, M, and C achieves 906 ± 21, outperforming prior deep RL methods listed.
• Including both z_t and h_t as inputs to C yields more stable and capable driving than using z_t alone (632 ± 251 to 906+).
• In VizDoom, training inside a generated environment followed by transferring to the real game yields 1092 time steps, well above the 750 target, with the virtual environment scores around 918 ± 546.
• V and M enable navigation directly from raw pixels without hand-crafted features, achieving competitive results with a small linear controller.
• Varying the MDN-RNN temperature τ shows that higher uncertainty reduces exploitability and can improve transfer to the real environment (e.g., best transfer observed around τ = 1.15).
• The approach highlights a practical path for sim-to-real through iterative world-model-based training and uncertainty control.