[Paper Review] V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models

Junwei You, Haotian Shi+7 more|arXiv (Cornell University)|Aug 17, 2024

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes V2X-VLM, an end-to-end vehicle-infrastructure cooperative autonomous driving framework that leverages large vision-language models (VLMs) to fuse multimodal data from vehicle cameras, infrastructure sensors, and textual context for improved situational awareness and trajectory planning. It achieves state-of-the-art performance with a 1.40m average L2 error on the DAIR-V2X dataset, significantly outperforming prior methods.

ABSTRACT

Vehicle-to-everything (V2X) cooperation has emerged as a promising paradigm to overcome the perception limitations of classical autonomous driving by leveraging information from both ego-vehicle and infrastructure sensors. However, effectively fusing heterogeneous visual and semantic information while ensuring robust trajectory planning remains a significant challenge. This paper introduces V2X-VLM, a novel end-to-end (E2E) cooperative autonomous driving framework based on vision-language models (VLMs). V2X-VLM integrates multiperspective camera views from vehicles and infrastructure with text-based scene descriptions to enable a more comprehensive understanding of driving environments. Specifically, we propose a contrastive learning-based mechanism to reinforce the alignment of heterogeneous visual and textual characteristics, which enhances the semantic understanding of complex driving scenarios, and employ a knowledge distillation strategy to stabilize training. Experiments on a large real-world dataset demonstrate that V2X-VLM achieves state-of-the-art trajectory planning accuracy, significantly reducing L2 error and collision rate compared to existing cooperative autonomous driving baselines. Ablation studies validate the contributions of each component. Moreover, the evaluation of robustness and efficiency highlights the practicality of V2X-VLM for real-world deployment to enhance overall autonomous driving safety and decision-making.

Motivation & Objective

• To address the limitations of current end-to-end autonomous driving systems in handling complex, dynamic driving environments by integrating multimodal data from vehicles and infrastructure.
• To enhance situational awareness and decision-making in cooperative autonomous driving through the use of large vision-language models (VLMs) that process both visual and textual inputs.
• To improve trajectory planning accuracy and robustness in real-world scenarios by leveraging comprehensive multimodal data fusion.
• To reduce the long-tail failure rate in autonomous driving by enabling systems to generalize across rare and complex urban scenarios through common-sense reasoning and contextual understanding.

Proposed method

• The framework employs a large vision-language model (VLM) as the core backbone to process and fuse multimodal inputs, including images from vehicle-mounted cameras, infrastructure sensor data, and textual descriptions of the environment.
• It integrates visual features from vehicle and infrastructure cameras with textual context (e.g., traffic signs, road conditions) to generate a unified, context-aware representation of the driving scene.
• The VLM is fine-tuned end-to-end on the DAIR-V2X dataset to jointly optimize perception, reasoning, and trajectory planning in a single unified pipeline.
• The system uses a transformer-based decoder to generate waypoint sequences directly from the fused multimodal embeddings, treating trajectory planning as a language generation task.
• Data transmission is optimized by extracting and transmitting key features rather than full-resolution images, reducing bandwidth while preserving critical information.
• The framework supports real-time inference by leveraging efficient VLM architectures and hardware-aware design principles to balance performance and communication cost.

Experimental results

Research questions

• RQ1 Can large vision-language models effectively integrate multimodal inputs (visual, textual, and infrastructure data) to improve situational awareness in cooperative autonomous driving?
• RQ2 How does end-to-end V2X-VLM compare to existing state-of-the-art methods in terms of trajectory planning accuracy and robustness across diverse driving scenarios?
• RQ3 To what extent does the fusion of textual context with visual data enhance decision-making and reduce error in complex urban environments?
• RQ4 What is the trade-off between transmission cost and planning accuracy when using high-fidelity visual data versus compressed feature representations in V2X communication?

Key findings

• V2X-VLM achieves a 1.40m average L2 error in trajectory prediction on the DAIR-V2X dataset, significantly outperforming the next-best method (UniV2X) with 3.43m.
• The framework reduces L2 error by over 50% compared to the closest baseline (UniV2X) across all evaluated time intervals (2.5s, 3.5s, 4.5s).
• Despite a higher transmission cost of $1.24 imes 10^7$ BPS, V2X-VLM maintains a favorable trade-off by achieving superior accuracy through comprehensive multimodal fusion.
• The model demonstrates robust performance across diverse and complex driving scenarios, including intersections, pedestrian crossings, and adverse weather conditions.
• The integration of textual context via VLMs enhances reasoning and generalization, particularly in long-tail scenarios where visual cues alone are insufficient.
• The framework shows strong potential for scalability and real-world deployment, especially with further optimization of data transmission through feature extraction.