[Paper Review] Generating Driving Scenes with Diffusion

Ethan Pronovost, Kai Wang+1 more|arXiv (Cornell University)|May 29, 2023

Computer Graphics and Visualization Techniques4 citations

TL;DR

This paper proposes Scene Diffusion, a latent diffusion model that directly generates physically plausible, oriented bounding boxes for vehicles and pedestrians in driving scenes by combining object detection with diffusion in a differentiable autoencoder framework. The method enables high-quality, diverse scene generation conditioned on map data and generalizes effectively across different U.S. regions, outperforming heuristic and autoregressive baselines in realism and diversity.

ABSTRACT

In this paper we describe a learned method of traffic scene generation designed to simulate the output of the perception system of a self-driving car. In our "Scene Diffusion" system, inspired by latent diffusion, we use a novel combination of diffusion and object detection to directly create realistic and physically plausible arrangements of discrete bounding boxes for agents. We show that our scene generation model is able to adapt to different regions in the US, producing scenarios that capture the intricacies of each region.

Motivation & Objective

• To develop a scalable, differentiable method for generating diverse, physically plausible traffic scenes for autonomous vehicle simulation.
• To address the limitations of heuristic and autoregressive scene generation by enabling end-to-end learning of agent layouts from map-conditioned diffusion.
• To improve scenario diversity and realism by modeling the distribution of real-world driving scenes across multiple U.S. regions.
• To directly generate oriented bounding boxes for agents without post-processing, ensuring geometric and semantic consistency.
• To enable high-fidelity simulation for training and validating perception, prediction, and planning systems in autonomous driving.

Proposed method

• A conditional variational autoencoder (CVAE) is trained to encode bird's-eye view images of agents into a latent space, with the decoder outputting oriented bounding boxes.
• The autoencoder uses a reconstruction loss based on predicted bounding box parameters and a KL regularization term to stabilize the latent distribution.
• A latent diffusion model is trained on the autoencoder's latent embeddings, conditioned on map images, to generate novel scene configurations.
• The diffusion model uses a noise prediction U-Net architecture with cross-attention to attend to map features during denoising.
• During inference, the diffusion model generates latent codes which are decoded into oriented bounding boxes using the trained autoencoder decoder.
• The model is fine-tuned on region-specific datasets to enable generalization across different U.S. driving environments.

Experimental results

Research questions

• RQ1 Can a latent diffusion model be effectively adapted to generate structured, discrete agent layouts (oriented bounding boxes) in driving scenes?
• RQ2 How well does the model generalize to different geographical regions in the U.S. without retraining?
• RQ3 Does end-to-end differentiable generation of oriented bounding boxes outperform heuristic or autoregressive methods in realism and diversity?
• RQ4 Can the model generate complex, physically plausible traffic scenes with minimal post-processing?
• RQ5 How does the combination of object detection and diffusion improve scene generation quality compared to occupancy map-based GANs?

Key findings

• The Scene Diffusion model successfully generates diverse, realistic driving scenes with physically plausible agent arrangements across multiple U.S. regions.
• Qualitative evaluation shows that scenes generated by the model capture region-specific driving behaviors and infrastructure patterns.
• The model generalizes well to unseen regions without fine-tuning, indicating robust learning of underlying scene distributions.
• Quantitative evaluation demonstrates superior performance in scene diversity and realism compared to autoregressive and GAN-based baselines.
• The end-to-end differentiable architecture enables stable training and high-fidelity generation without post-processing steps.
• The use of sine and cosine encoding for orientation avoids angle discontinuities and improves detection accuracy.