[Paper Review] DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving

Wencheng Han, Dongqian Guo+2 more|arXiv (Cornell University)|Jan 8, 2024

Advanced Neural Network Applications4 citations

TL;DR

DME-Driver proposes a novel autonomous driving system that integrates a Large Vision Language Model (LVLM) as a logic-driven Decision-Maker and a planning-oriented perception model as an Executor to enhance both interpretability and control accuracy. By leveraging human driver behavior logic and precise 3D scene perception, the system achieves state-of-the-art planning accuracy (L2: 0.98m) with full decision traceability.

ABSTRACT

In the field of autonomous driving, two important features of autonomous driving car systems are the explainability of decision logic and the accuracy of environmental perception. This paper introduces DME-Driver, a new autonomous driving system that enhances the performance and reliability of autonomous driving system. DME-Driver utilizes a powerful vision language model as the decision-maker and a planning-oriented perception model as the control signal generator. To ensure explainable and reliable driving decisions, the logical decision-maker is constructed based on a large vision language model. This model follows the logic employed by experienced human drivers and makes decisions in a similar manner. On the other hand, the generation of accurate control signals relies on precise and detailed environmental perception, which is where 3D scene perception models excel. Therefore, a planning oriented perception model is employed as the signal generator. It translates the logical decisions made by the decision-maker into accurate control signals for the self-driving cars. To effectively train the proposed model, a new dataset for autonomous driving was created. This dataset encompasses a diverse range of human driver behaviors and their underlying motivations. By leveraging this dataset, our model achieves high-precision planning accuracy through a logical thinking process.

Motivation & Objective

• To address the lack of interpretability in end-to-end planning-based autonomous driving systems.
• To overcome the limitation of LLM-based systems that do not optimize perception tasks through decision feedback.
• To integrate human-like decision logic with high-precision 3D scene perception for reliable autonomous navigation.
• To develop a new dataset (HBD) capturing human driver behavior, gaze, and decision logic for training.
• To enable traceable, accountable, and human-aligned decision-making in autonomous vehicles.

Proposed method

• The Decision-Maker is a Large Vision Language Model (LVLM) trained via imitation learning on real-world driving data to mimic human driver logic and reasoning.
• The Executor is a planning-oriented perception model that translates high-level decisions into precise vehicle control signals.
• The system uses a two-stage architecture: the Decision-Maker reasons about driving scenarios and generates logical instructions, while the Executor generates control signals based on these instructions.
• A consistency loss is applied between the Decision-Maker and Executor to align their outputs and improve decision reliability.
• The HBD dataset is introduced, containing diverse human driver behaviors, gaze data, and decision logic for training and evaluation.
• The framework enables full traceability of decisions by logging the reasoning process of the LVLM-based Decision-Maker.

Figure 1: Comparative Structures in Autonomous Driving Systems: (a) depicts an planning-oriented autonomous driving system that optimizes for overall performance using planning results, but lacks interpretability; (b) shows an LLM-based autonomous driving system capable of producing reasonable contr

Experimental results

Research questions

• RQ1 Can an LVLM-based decision-maker effectively emulate human driver logic in complex driving scenarios?
• RQ2 Can a planning-oriented perception model generate accurate control signals when guided by logical decision outputs?
• RQ3 Does integrating human decision logic with high-precision 3D perception improve overall planning accuracy and system reliability?
• RQ4 Can the system maintain interpretability while achieving state-of-the-art performance in autonomous driving?
• RQ5 How does the consistency loss between Decision-Maker and Executor improve decision alignment and system robustness?

Key findings

• DME-Driver achieves a planning accuracy of 0.98m L2 error, outperforming prior methods such as UniAD (1.03m) and ST-P3 (2.11m).
• The system demonstrates superior logic understanding, scoring 80.3 on the logic benchmark—significantly higher than LLaVA-13B (48.2) and GPT-4V (65.4).
• The ablation study confirms that both the Decision-Maker and Executor are essential, with the full system reducing L2 error by 0.05m compared to the baseline Executor-only setup.
• Incorporating ground-truth language cues (GT+Executor) improves performance slightly, indicating the value of accurate decision guidance.
• The consistency loss mechanism slightly increases L2 error (0.98m vs. 0.96m) but significantly improves alignment between decision and execution, enhancing reliability.
• The HBD dataset enables the model to learn diverse human behaviors, including gaze patterns and decision rationale, supporting explainable and human-aligned driving decisions.

Figure 2: HBD Dataset Annotation Pipeline: There are four key steps in our dataset creation - Data Collection, Prompt Design, Manual Corrections, and Dialogue Generation, streamlining the process from raw data gathering to structured dialogue formation.