[Paper Review] Genie: Generative Interactive Environments

Jake Bruce, Michael J. Dennis+23 more|arXiv (Cornell University)|Feb 23, 2024

Digital Games and Media11 citations

TL;DR

Genie is a foundational world model trained in an unsupervised manner from unlabelled Internet videos, enabling frame-by-frame, action-controllable interactive environments prompted by images, sketches, text, or prompts, with an 11B-parameter model.

ABSTRACT

We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

Motivation & Objective

• Motivate and enable generation of interactive, playable virtual worlds from prompts without ground-truth action labels.
• Develop a scalable, modular architecture that learns a latent action space from video data.
• Demonstrate frame-by-frame controllability and generalization to unseen prompts and domains.
• Explore the potential of latent actions learned from videos to support training generalist agents (Open Ended Learning).

Proposed method

• Three main components: a video tokenizer (VQ-VAE-based) to tokenise frames into discrete tokens; a latent action model (LAM) that learns a small discrete set of latent actions in an unsupervised manner; a dynamics model (MaskGIT-based) that autoregressively predicts future frame tokens conditioned on past tokens and latent actions.
• The architecture uses spatiotemporal (ST) transformers across components to handle video data efficiently; a causal mask enables processing entire sequences for latent action inference and future frame prediction.
• Training is performed in two phases: first train the video tokenizer, then jointly train the latent action model and the dynamics model on video tokens.
• The latent action space is discretized with a small VQ codebook (|A|=8) to ensure controllability and human-playability.
• Experiments are conducted on Platformers video data (≈30k hours) and robotics videos (RT1), with evaluation using Frechet Video Distance (FVD) and a controllability metric Delta_t-PSNR.

Experimental results

Research questions

• RQ1 Can a large-scale, unsupervised model learn a usable latent action space from unlabelled videos?
• RQ2 Can Genie generate diverse, controllable interactive environments from prompts such as images or sketches?
• RQ3 Do latent actions learned from internet videos transfer to unseen prompts and to robotics domains?
• RQ4 Is the approach scalable in model size and data, and can it support potential use as a foundation model for generalist agents?

Key findings

• Genie trains an 11B-parameter model (with tokenizer and latent action model bringing total to 11.0B parameters; larger website variant mentioned) that can generate interactive environments from prompts.
• The Platformers-trained model (11B) achieves qualitative and quantitative results including strong controllability across prompts, including out-of-distribution image prompts (e.g., hand-drawn sketches, real photos, Imagen2 prompts).
• The Robotics-trained model (2.5B parameters) learns consistent latent actions (e.g., down, up, left) without action labels and demonstrates object interactions and deformable objects handling.
• Quantitative metrics show convergence in scaling experiments; increasing model size and batch size yields lower training loss, with reported FVD and Delta_t-PSNR trends indicating improved fidelity and controllability as scale increases.
• Genie achieves a Frechet Video Distance (FVD) of 82.7 on the Robotics test set and demonstrates consistent latent-action behavior across multiple starting frames.
• The approach enables using latent actions learned from internet videos to imitate policies in unseen RL environments, with evidence that a small amount of expert data can map latent actions to real actions for policy cloning.