[Paper Review] DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model

Xiaofan Li, Yifu Zhang+1 more|arXiv (Cornell University)|Oct 11, 2023

Advanced Vision and Imaging4 citations

TL;DR

DrivingDiffusion proposes a cascaded latent diffusion framework for generating high-quality, temporally and spatially consistent multi-view driving videos from 3D layout control. By integrating cross-view attention, key-frame conditioning, local prompts, and sliding-window post-processing, it achieves state-of-the-art video consistency and instance quality, significantly improving BEV perception performance when used for data augmentation.

ABSTRACT

With the increasing popularity of autonomous driving based on the powerful and unified bird's-eye-view (BEV) representation, a demand for high-quality and large-scale multi-view video data with accurate annotation is urgently required. However, such large-scale multi-view data is hard to obtain due to expensive collection and annotation costs. To alleviate the problem, we propose a spatial-temporal consistent diffusion framework DrivingDiffusion, to generate realistic multi-view videos controlled by 3D layout. There are three challenges when synthesizing multi-view videos given a 3D layout: How to keep 1) cross-view consistency and 2) cross-frame consistency? 3) How to guarantee the quality of the generated instances? Our DrivingDiffusion solves the problem by cascading the multi-view single-frame image generation step, the single-view video generation step shared by multiple cameras, and post-processing that can handle long video generation. In the multi-view model, the consistency of multi-view images is ensured by information exchange between adjacent cameras. In the temporal model, we mainly query the information that needs attention in subsequent frame generation from the multi-view images of the first frame. We also introduce the local prompt to effectively improve the quality of generated instances. In post-processing, we further enhance the cross-view consistency of subsequent frames and extend the video length by employing temporal sliding window algorithm. Without any extra cost, our model can generate large-scale realistic multi-camera driving videos in complex urban scenes, fueling the downstream driving tasks. The code will be made publicly available.

Motivation & Objective

• To address the scarcity of large-scale, high-quality, multi-view, and temporally consistent video data for autonomous driving perception.
• To enable controllable generation of multi-view driving videos using 3D layouts as conditioning input.
• To maintain cross-view and cross-frame consistency while ensuring high-fidelity instance generation in complex urban scenes.
• To develop a scalable, end-to-end framework for synthetic data generation that supports downstream BEV perception tasks.
• To explore the feasibility of using diffusion models for multi-view video generation with precise geometric and semantic control.

Proposed method

• Cascaded architecture combining a multi-view single-frame generation model, a shared single-view video generation model, and a post-processing refinement stage.
• Cross-view attention module with geometric constraints to enforce spatial consistency across multiple camera views in the same frame.
• Key-frame controller that uses the first frame's multi-view features as context for subsequent frame generation, ensuring temporal coherence.
• Local prompt injection to guide detailed, high-quality generation of specific instances like vehicles and pedestrians.
• Temporal sliding window inference to extend video length beyond the model's native receptive field.
• Post-processing with a fine-tuned multi-view model that re-optimizes subsequent frames using the first frame as a reference, enhancing long-range consistency.

Experimental results

Research questions

• RQ1 Can a latent diffusion model generate multi-view, temporally consistent driving videos from a 3D layout while maintaining high visual fidelity?
• RQ2 How can cross-view consistency be preserved when generating synchronized video frames from multiple camera perspectives?
• RQ3 What mechanisms ensure long-term temporal coherence in generated video sequences without frame-level drift?
• RQ4 To what extent can local prompts improve the quality of generated instances such as vehicles and pedestrians?
• RQ5 How effective is the proposed framework as a data augmentation strategy for BEV-based perception models?

Key findings

• The combination of the consistency module and local prompt achieved the best generation quality, with FID of 15.83, FVD of 332, and Object NDS of 33.1.
• Removing the consistency module led to view-inconsistent perspectives, while removing the local prompt degraded instance quality, confirming their necessity.
• Synthetic data augmentation using DrivingDiffusion improved BEV perception: NDS increased from 0.412 to 0.434 and mAOE decreased from 0.5613 to 0.5130 when using 6,000 synthetic frames.
• The method successfully generated realistic collision and rare-scenario videos by perturbing the input 3D layout, demonstrating robust controllability.
• The framework significantly reduced FVD by 55 points and FID by 0.63 when consistency and local prompt components were included, indicating strong improvements in video and image quality.
• The model can generate long video sequences via sliding window inference and maintain high consistency across views and frames without additional training cost.