[Paper Review] AccidentGPT: Accident Analysis and Prevention from V2X Environmental Perception with Multi-modal Large Model

Lening Wang, Yilong Ren+10 more|arXiv (Cornell University)|Dec 20, 2023

Autonomous Vehicle Technology and Safety15 citations

TL;DR

AccidentGPT introduces a multi-modal large model that fuses V2X environmental perception with GPT-based reasoning to analyze accidents and aid prevention for autonomous, human-driven, and enforcement contexts.

ABSTRACT

Traffic accidents, being a significant contributor to both human casualties and property damage, have long been a focal point of research for many scholars in the field of traffic safety. However, previous studies, whether focusing on static environmental assessments or dynamic driving analyses, as well as pre-accident predictions or post-accident rule analyses, have typically been conducted in isolation. There has been a lack of an effective framework for developing a comprehensive understanding and application of traffic safety. To address this gap, this paper introduces AccidentGPT, a comprehensive accident analysis and prevention multi-modal large model. AccidentGPT establishes a multi-modal information interaction framework grounded in multi-sensor perception, thereby enabling a holistic approach to accident analysis and prevention in the field of traffic safety. Specifically, our capabilities can be categorized as follows: for autonomous driving vehicles, we provide comprehensive environmental perception and understanding to control the vehicle and avoid collisions. For human-driven vehicles, we offer proactive long-range safety warnings and blind-spot alerts while also providing safety driving recommendations and behavioral norms through human-machine dialogue and interaction. Additionally, for traffic police and management agencies, our framework supports intelligent and real-time analysis of traffic safety, encompassing pedestrian, vehicles, roads, and the environment through collaborative perception from multiple vehicles and road testing devices. The system is also capable of providing a thorough analysis of accident causes and liability after vehicle collisions. Our framework stands as the first large model to integrate comprehensive scene understanding into traffic safety studies. Project page: https://accidentgpt.github.io

Motivation & Objective

• Motivate holistic traffic safety analysis across static environments, dynamic states, and post/pre-accident tasks using integrated perception and reasoning.
• Develop a unified framework that merges multi-sensor V2X perception with a large language model for end-to-end accident analysis and prevention.
• Enable autonomous vehicles to perceive and predict for collision avoidance, while providing proactive warnings and safety guidance for human-driven vehicles.
• Support traffic police and management with real-time, collaborative perception and comprehensive accident causation analysis.
• Demonstrate the first large-model integration that provides comprehensive scene understanding for traffic safety research.

Proposed method

• Create a V2X-perception module that fuses panoramic imagery from multiple vehicles and road devices to produce 3D detections, BEV maps, and trajectory predictions.
• Implement a GPT-reasoning module (based on GPT-4V) with a multi-level priority sampling prompt system, active/passive task prompts, and specialized output modules for accident-related tasks.
• Align multi-time-sequence perception data via Ego-motion for BEV feature fusion and multi-task detection heads.
• Develop a corpus-backed prompting pipeline that dynamically builds temporal corpora, retrieves historical and contextual data, and supports proactive and reactive driver prompts.
• Incorporate a chain-of-thought style reasoning flow within LLM computation to improve safety decision-making and uncertainty handling.
• Enable automatic corpus updates from driving outcomes to progressively improve inference and risk assessment.

Figure 1: The overall framework of AccidentGPT.

Experimental results

Research questions

• RQ1 What is the impact of integrating multi-vehicle and vehicle-road collaborative perception with LLM-based reasoning on accident analysis and prevention?
• RQ2 How can V2X-perception outputs (3D detection, BEV perception, trajectories) effectively fuel GPT-4V reasoning for proactive warnings and post-accident causation analysis?
• RQ3 Can active/passive prompting and iterative corpus updates improve safety recommendations and liability assessments in real-world traffic scenarios?

Key findings

• V2X-perception (our) achieves higher metrics than listed baselines in Table 1, with Detection 41.07%, mIOU 57.3%, and VPQ 45.2%.
• Compared to other models, our approach shows improved BEV perception and trajectory prediction performance across multiple baselines.
• Table 2 indicates the model’s Car/Truck/Van/Pedestrian mATE, mASE, and mAOE metrics with the Pedestrian row providing detailed error measures (mATE 1.0017, mASE 0.9993, mAOE 1.0253).
• The GPT-reasoning module enables proactive long-range safety warnings, blind-spot alerts, and driver-specific safety recommendations via human-machine dialogue.
• The framework supports real-time accident causation analysis and liability assessment for traffic management agencies.

Figure 2: The V2X-perception architecture.