[Paper Review] SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout

Chiyu Max Jiang, Yanchao Bai+13 more|arXiv (Cornell University)|Dec 5, 2024

Autonomous Vehicle Technology and Safety4 citations

TL;DR

SceneDiffuser introduces a unified diffusion model for autonomous vehicle simulation that jointly handles scene initialization and closed-loop rollout using amortized diffusion to reduce inference cost by 16x while improving trajectory realism and controllability through generalized hard constraints and LLM-guided prompting. It achieves state-of-the-art performance on the Waymo Open Sim Agents Challenge, setting the best closed-loop results among diffusion-based models.

ABSTRACT

Realistic and interactive scene simulation is a key prerequisite for autonomous vehicle (AV) development. In this work, we present SceneDiffuser, a scene-level diffusion prior designed for traffic simulation. It offers a unified framework that addresses two key stages of simulation: scene initialization, which involves generating initial traffic layouts, and scene rollout, which encompasses the closed-loop simulation of agent behaviors. While diffusion models have been proven effective in learning realistic and multimodal agent distributions, several challenges remain, including controllability, maintaining realism in closed-loop simulations, and ensuring inference efficiency. To address these issues, we introduce amortized diffusion for simulation. This novel diffusion denoising paradigm amortizes the computational cost of denoising over future simulation steps, significantly reducing the cost per rollout step (16x less inference steps) while also mitigating closed-loop errors. We further enhance controllability through the introduction of generalized hard constraints, a simple yet effective inference-time constraint mechanism, as well as language-based constrained scene generation via few-shot prompting of a large language model (LLM). Our investigations into model scaling reveal that increased computational resources significantly improve overall simulation realism. We demonstrate the effectiveness of our approach on the Waymo Open Sim Agents Challenge, achieving top open-loop performance and the best closed-loop performance among diffusion models.

Motivation & Objective

• To address the challenge of efficient and realistic closed-loop autonomous vehicle simulation with high computational cost and distributional drift.
• To unify scene initialization and rollout into a single diffusion-based generative framework that models agents, poses, sizes, types, and timesteps jointly.
• To improve controllability in simulation through generalized hard constraints and few-shot prompting of a large language model for natural language-driven scenario generation.
• To investigate the impact of model scaling on simulation realism and identify optimal compute-resource trade-offs.
• To demonstrate state-of-the-art performance in both open-loop and closed-loop simulation on the Waymo Open Sim Agents Challenge.

Proposed method

• Introduces amortized diffusion, a novel denoising paradigm that amortizes the computational cost of denoising over multiple future simulation steps, reducing inference steps per rollout by 16x.
• Employs a unified spatiotemporal diffusion model trained end-to-end on logged driving scenes to jointly predict agent trajectories, poses, sizes, types, and timesteps.
• Applies generalized hard constraints as a flexible, inference-time mechanism to enforce scenario-level rules such as agent positions, speeds, or behaviors during generation.
• Utilizes few-shot prompting of a large language model (LLM) to convert natural language descriptions into structured constraints for synthetic scene generation.
• Employs multitask co-training across behavior prediction, scene generation, and random control masks to improve generalization and reduce collision and off-road rates.
• Uses cross-attention conditioning with AdaLN-Zero and agent-wise spatial attention to model joint agent distributions and improve trajectory consistency.

Experimental results

Research questions

• RQ1 Can a unified diffusion model effectively handle both scene initialization and closed-loop rollout in autonomous driving simulation?
• RQ2 How can amortized diffusion reduce inference cost while maintaining or improving trajectory realism in closed-loop simulations?
• RQ3 To what extent can generalized hard constraints and LLM-guided prompting enable controllable and diverse scenario generation?
• RQ4 How does scaling model size and temporal resolution affect simulation realism and performance?
• RQ5 Can a diffusion-based model achieve state-of-the-art performance in closed-loop simulation compared to autoregressive baselines?

Key findings

• SceneDiffuser achieves top open-loop performance and the best closed-loop performance among diffusion-based models on the Waymo Open Sim Agents Challenge.
• Amortized diffusion reduces inference cost by 16x per rollout step while significantly reducing closed-loop error and improving trajectory consistency.
• Increasing model size and temporal resolution through scaling improves simulation realism, with joint distributions closely tracking logged data distributions.
• The use of generalized hard constraints enables effective control over agent behavior and scene layout, supporting both manual and LLM-generated constraint generation.
• Multitask co-training improves simulation robustness, reducing collision and off-road rates compared to single-task baselines.
• Replacing AdaLN-Zero with cross-attention reduces realism performance by 7.99%, and removing agent-wise spatial attention drastically increases collision rates, confirming their importance.