[Paper Review] VAD: Vectorized Scene Representation for Efficient Autonomous Driving

Bo Jiang, Shaoyu Chen+8 more|arXiv (Cornell University)|Mar 21, 2023

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper proposes VAD, a fully vectorized, end-to-end autonomous driving framework that models scenes using vectorized agent motions and map elements instead of dense rasterized representations. By leveraging explicit instance-level planning constraints and query-based feature interaction, VAD achieves state-of-the-art planning performance with 2.5× faster inference than prior methods, reducing collision rates by 29.0% and displacement error by 30.1% on nuScenes.

ABSTRACT

Autonomous driving requires a comprehensive understanding of the surrounding environment for reliable trajectory planning. Previous works rely on dense rasterized scene representation (e.g., agent occupancy and semantic map) to perform planning, which is computationally intensive and misses the instance-level structure information. In this paper, we propose VAD, an end-to-end vectorized paradigm for autonomous driving, which models the driving scene as a fully vectorized representation. The proposed vectorized paradigm has two significant advantages. On one hand, VAD exploits the vectorized agent motion and map elements as explicit instance-level planning constraints which effectively improves planning safety. On the other hand, VAD runs much faster than previous end-to-end planning methods by getting rid of computation-intensive rasterized representation and hand-designed post-processing steps. VAD achieves state-of-the-art end-to-end planning performance on the nuScenes dataset, outperforming the previous best method by a large margin. Our base model, VAD-Base, greatly reduces the average collision rate by 29.0% and runs 2.5x faster. Besides, a lightweight variant, VAD-Tiny, greatly improves the inference speed (up to 9.3x) while achieving comparable planning performance. We believe the excellent performance and the high efficiency of VAD are critical for the real-world deployment of an autonomous driving system. Code and models are available at https://github.com/hustvl/VAD for facilitating future research.

Motivation & Objective

• To address the computational inefficiency and loss of instance-level structure in rasterized scene representations used in prior end-to-end autonomous driving methods.
• To improve planning safety by explicitly encoding vectorized motion and map information as hard constraints.
• To enable real-time deployment of end-to-end autonomous driving systems through efficient vectorized computation and reduced post-processing.
• To demonstrate that vectorized scene representation outperforms rasterized counterparts in both accuracy and speed on benchmark datasets.

Proposed method

• VAD uses a BEV-based backbone and transformer-based encoder to process front-camera images and generate bird’s-eye-view features.
• It employs map queries and agent queries to implicitly attend to vectorized map elements (e.g., lane boundaries, road edges) and agent motion trajectories.
• The framework introduces three explicit vectorized planning constraints: ego-agent collision, ego-boundary overstepping, and ego-lane direction regularization.
• Vectorized map and motion predictions are generated via dedicated head modules, replacing dense semantic or occupancy maps.
• The planning head uses sparse vectorized queries to generate future trajectories, with inference accelerated by the compactness of vectorized representations.
• The model is trained end-to-end with multi-task supervision, including motion prediction, map prediction, and planning objectives.

Experimental results

Research questions

• RQ1 Can a fully vectorized scene representation outperform dense rasterized representations in end-to-end autonomous driving planning?
• RQ2 How do explicit vectorized planning constraints improve safety and accuracy without increasing computational cost?
• RQ3 To what extent does eliminating rasterized representation and post-processing steps enhance inference speed while maintaining or improving performance?
• RQ4 How does the integration of instance-level map and agent features via query interaction improve planning generalization and robustness?
• RQ5 Can lightweight variants of the vectorized paradigm achieve real-time inference with competitive planning accuracy?

Key findings

• VAD-Base reduces average displacement error by 30.1% (0.72m vs. 1.03m) and collision rate by 29.0% (0.22% vs. 0.31%) compared to the previous SOTA method UniAD on nuScenes.
• VAD-Base runs 2.5× faster than UniAD (4.5 FPS vs. 1.8 FPS), achieving the best balance of speed and accuracy.
• VAD-Tiny achieves 9.3× faster inference (16.8 FPS) than VAD-Base while maintaining comparable performance, with a displacement error of 0.78m and collision rate of 0.38%.
• Ablation studies confirm that ego-map and ego-agent interactions are critical for performance, reducing collision rates by up to 29.0% when combined.
• Replacing vectorized map prediction with rasterized BEV segmentation increases collision rates significantly, demonstrating the superiority of vectorized representation.
• The planning module accounts for only 3.4ms of inference time in VAD-Tiny, highlighting the efficiency of sparse vectorized computation.