[Paper Review] DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving

Xiaofeng Wang, Zheng Zhu+4 more|arXiv (Cornell University)|Sep 18, 2023

Autonomous Vehicle Technology and Safety15 citations

TL;DR

DriveDreamer builds a real-world driven diffusion-based world model for autonomous driving, enabling controllable driving video generation and future driving policy prediction, trained in two stages with structured traffic information. It demonstrates improved video quality and open-loop driving planning on nuScenes.

ABSTRACT

World models, especially in autonomous driving, are trending and drawing extensive attention due to their capacity for comprehending driving environments. The established world model holds immense potential for the generation of high-quality driving videos, and driving policies for safe maneuvering. However, a critical limitation in relevant research lies in its predominant focus on gaming environments or simulated settings, thereby lacking the representation of real-world driving scenarios. Therefore, we introduce DriveDreamer, a pioneering world model entirely derived from real-world driving scenarios. Regarding that modeling the world in intricate driving scenes entails an overwhelming search space, we propose harnessing the powerful diffusion model to construct a comprehensive representation of the complex environment. Furthermore, we introduce a two-stage training pipeline. In the initial phase, DriveDreamer acquires a deep understanding of structured traffic constraints, while the subsequent stage equips it with the ability to anticipate future states. The proposed DriveDreamer is the first world model established from real-world driving scenarios. We instantiate DriveDreamer on the challenging nuScenes benchmark, and extensive experiments verify that DriveDreamer empowers precise, controllable video generation that faithfully captures the structural constraints of real-world traffic scenarios. Additionally, DriveDreamer enables the generation of realistic and reasonable driving policies, opening avenues for interaction and practical applications.

Motivation & Objective

• Motivate the need for world models derived from real-world driving rather than simulated environments.
• Introduce a diffusion-based Autonomous-driving Diffusion Model (Auto-DM) to encode traffic structure for efficient sampling.
• Propose a two-stage training pipeline to learn structural constraints first and then predictive video and action modeling.
• Enable controllable driving video generation aligned with traffic constraints and text prompts.
• Demonstrate the utility of DriveDreamer for improving perception training and open-loop driving planning on nuScenes.

Proposed method

• Introduce Auto-DM, a diffusion-based model that conditions on HDMaps, 3D boxes, and text prompts to generate driving videos.
• Incorporate spatially aligned (HDMap) and position (3D boxes) conditions with gated self-attention and temporal attention to ensure frame coherence.
• Use cross-attention to integrate text prompts shaping style attributes like weather and time of day.
• Two-stage training: Stage 1 learns traffic structural constraints from single frames and then videos; Stage 2 trains a video-prediction world model with ActionFormer to forecast future traffic structures from driving actions.
• ActionFormer iteratively predicts future traffic structures from past actions, enabling Auto-DM to generate future driving videos and future driving actions.
• Model objective combines video prediction and action prediction terms via a variational lower bound, optimized with MSE for video and L1 for actions.

Experimental results

Research questions

• RQ1 Can a diffusion-based world model trained on real-world driving videos understand and utilize structured traffic information (HDMaps, 3D boxes) to generate realistic driving videos?
• RQ2 Can a two-stage training pipeline improve sampling efficiency and convergence when learning real-world driving world models?
• RQ3 Does integrating driving actions via ActionFormer enable accurate future state prediction and plausible driving policies aligned with real-world scenarios?
• RQ4 Do synthetic driving videos and policies improve downstream perception training and open-loop planning on real-world datasets like nuScenes?

Key findings

• DriveDreamer enables controllable driving video generation that adheres to structured traffic constraints and can be guided by text prompts for weather/time of day.
• First-stage Auto-DM training with traffic structure improves video generation quality and sampling efficiency; second-stage vision-action training yields predictive driving videos and actions.
• ActionFormer improves future traffic-structure prediction by updating latent future conditions from driving actions, enhancing video realism and policy plausibility.
• Synthetic data generated by DriveDreamer improves 3D object detection metrics (FCOS3D and BEVFusion) on nuScenes by up to 3.0 mAP and 1.9 NDS in the reported setup.
• Open-loop planning evaluation shows DriveDreamer achieving competitive L2 trajectory error (0.29 m) and lower collision rates compared with prior multimodal methods.
• Quantitative comparisons indicate DriveDreamer with full pipeline (Auto-DM + ActionFormer) achieves better FID (14.9) and FVD (340.8) than ablations and DriveGAN.