[Paper Review] GAIA-1: A Generative World Model for Autonomous Driving

Anthony Hu, Lloyd Russell+6 more|arXiv (Cornell University)|Sep 29, 2023

Human Motion and AnimationEngineering27 citations

TL;DR

GAIA-1 combines a world-model transformer with a video diffusion decoder to generate realistic driving scenarios from multimodal prompts, enabling future prediction, scene understanding, and fine-grained control of ego-vehicle behavior.

ABSTRACT

Autonomous driving promises transformative improvements to transportation, but building systems capable of safely navigating the unstructured complexity of real-world scenarios remains challenging. A critical problem lies in effectively predicting the various potential outcomes that may emerge in response to the vehicle's actions as the world evolves. To address this challenge, we introduce GAIA-1 ('Generative AI for Autonomy'), a generative world model that leverages video, text, and action inputs to generate realistic driving scenarios while offering fine-grained control over ego-vehicle behavior and scene features. Our approach casts world modeling as an unsupervised sequence modeling problem by mapping the inputs to discrete tokens, and predicting the next token in the sequence. Emerging properties from our model include learning high-level structures and scene dynamics, contextual awareness, generalization, and understanding of geometry. The power of GAIA-1's learned representation that captures expectations of future events, combined with its ability to generate realistic samples, provides new possibilities for innovation in the field of autonomy, enabling enhanced and accelerated training of autonomous driving technology.

Motivation & Objective

• Develop a scalable, unsupervised world model to predict future driving events under diverse conditions.
• Learn meaningful, high-level representations of road scenes and dynamics from real-world data.
• Enable controllable generation of ego-vehicle behavior and scene elements through actions and language prompts.
• Demonstrate emergent properties like long-horizon scene generation, generalization, and 3D geometric understanding.

Proposed method

• Partition the system into a world model and a video diffusion decoder to separate scene reasoning from high-quality video rendering.
• Represent each video frame with discrete image tokens via a learned image tokenizer; model the future as next-token prediction in a sequence.
• Use an autoregressive transformer as the world model to predict the next image token conditioned on past image, text, and action tokens.
• Train a multi-task video diffusion decoder that can render high-resolution video and perform temporal upsampling conditioned on world-model tokens.
• Train on a large real-world UK urban driving dataset with balanced sampling across geographies and weather to learn robust representations.
• Incorporate multimodal prompts (video, text, action) and employ classifier-free guidance during inference to align generated futures with text prompts.

Experimental results

Research questions

• RQ1 Can GAIA-1 reliably predict plausible future driving scenarios from multimodal prompts?
• RQ2 Do the learned tokens and emergent representations capture high-level scene structure, geometry, and dynamics relevant for autonomous driving?
• RQ3 Is the model able to generate multiple plausible futures from a single context?
• RQ4 To what extent can the ego-vehicle dynamics and scene elements be controlled via actions and text prompts?
• RQ5 How does scaling (data and compute) affect world-model performance and sample quality?

Key findings

• GAIA-1 learns high-level structures and scene dynamics, enabling coherent, plausible driving scenes.
• The model generalizes and shows creativity, producing novel futures beyond training instances.
• It demonstrates contextual awareness and an understanding of 3D geometry, including road-induced pitch/roll effects.
• GAIA-1 can generate long, stable driving videos from imagination and multiple plausible futures from the same context.
• Fine-grained control over ego-vehicle behavior and scene attributes is achievable via text prompts and actions.