[Paper Review] LMDrive: Closed-Loop End-to-End Driving with Large Language Models

Hao Shao, Yuxuan Hu+4 more|arXiv (Cornell University)|Dec 12, 2023

Topic Modeling4 citations

TL;DR

LMDrive is the first end-to-end, closed-loop autonomous driving framework that integrates natural language instructions with multi-modal sensor data (camera and LiDAR) via a frozen large language model (LLM), enabling real-time, language-guided control. It achieves state-of-the-art performance on the LangAuto benchmark, demonstrating robustness in complex, long-horizon driving scenarios with human-like reasoning and instruction following.

ABSTRACT

Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes, models, and datasets can be found at https://github.com/opendilab/LMDrive

Motivation & Objective

• To address the limitations of current autonomous driving systems in handling long-tail, unforeseen events and complex urban scenarios by enabling language-guided reasoning.
• To overcome the rigid, fixed-format input constraints of modular and end-to-end driving systems by integrating natural language instructions as first-class inputs.
• To enable real-time, closed-loop interaction between autonomous vehicles and human passengers or navigation software through natural language.
• To establish a benchmark and dataset for evaluating language-based, closed-loop autonomous driving systems in realistic, complex driving scenarios.

Proposed method

• Leverages a frozen, pre-trained large language model (LLM) as the reasoning backbone to maintain generalization and reasoning capabilities.
• Integrates multi-modal sensor inputs (camera and LiDARD) through dedicated vision encoders (ResNet-50 for 2D, PointPillars for 3D) and BEV (bird’s-eye view) feature extraction.
• Employs a Q-Former module to compress and adapt visual features into a learnable token sequence compatible with the LLM’s context window.
• Uses learnable input and output adapters to bridge the vision encoder outputs with the LLM, enabling end-to-end fine-tuning without updating the LLM weights.
• Applies a task-specific pre-training strategy on the vision encoder using a masked autoencoding objective on driving data to improve visual representation learning.
• Trains the full system in a closed-loop setting using a newly collected dataset of ~64K instruction-following clips in the CARLA simulator, with data augmentation to prevent overfitting to notice instructions.

Experimental results

Research questions

• RQ1 Can large language models be effectively leveraged for closed-loop, end-to-end autonomous driving with natural language instructions?
• RQ2 How does integrating natural language instructions with multi-modal sensor data improve reasoning and robustness in long-tail and complex urban driving scenarios?
• RQ3 To what extent can a frozen LLM with vision adapters generalize to diverse navigation tasks and adversarial instruction scenarios?
• RQ4 What is the impact of vision pre-training and architectural components (e.g., Q-Former, BEV token usage) on the performance of language-guided driving agents?
• RQ5 How does the system handle temporal consistency and instruction ambiguity when following sequential or conflicting navigation commands?

Key findings

• LLaVA-v1.5 outperforms other LLM backbones (including LLaMA, LLaMA2, Vicuna) on the LangAuto benchmarks, achieving the highest driving score of 36.2, demonstrating the benefit of multi-modal pretraining.
• Removing the Q-Former module reduces the driving score from 36.2 to 31.7, indicating that feature compression via Q-Former is critical for effective LLM input.
• Excluding BEV tokens from the LLM decoder drops the infraction score from 0.81 to 0.72, confirming that visual context is essential for safe decision-making.
• Eliminating vision encoder pre-training reduces the driving score to 16.9, proving that pre-training on driving data significantly improves visual representation quality.
• On the LangAuto-Notice benchmark, LMDrive reduces collisions and traffic violations when handling adversarial or unexpected instructions, showing improved robustness.
• On the LangAuto-Sequential benchmark, the model maintains strong performance despite complex, multi-step instructions, with only a moderate drop in route completion and driving score, indicating temporal awareness and instruction tracking capability.