[Paper Review] DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving

Tingkai Wang, Enze Xie+3 more|arXiv (Cornell University)|Mar 25, 2024

Semantic Web and Ontologies4 citations

TL;DR

This paper introduces DriveCoT, a new end-to-end driving dataset with chain-of-thought (CoT) annotations for reasoning across driving aspects and final decisions, collected using a rule-based expert policy in the CARLA simulator. It proposes DriveCoT-Agent, a multi-view video-based model that generates CoT predictions to improve interpretability and achieves state-of-the-art performance in both open-loop and closed-loop evaluations on challenging scenarios like high-speed driving and lane changes.

ABSTRACT

End-to-end driving has made significant progress in recent years, demonstrating benefits such as system simplicity and competitive driving performance under both open-loop and closed-loop settings. Nevertheless, the lack of interpretability and controllability in its driving decisions hinders real-world deployment for end-to-end driving systems. In this paper, we collect a comprehensive end-to-end driving dataset named DriveCoT, leveraging the CARLA simulator. It contains sensor data, control decisions, and chain-of-thought labels to indicate the reasoning process. We utilize the challenging driving scenarios from the CARLA leaderboard 2.0, which involve high-speed driving and lane-changing, and propose a rule-based expert policy to control the vehicle and generate ground truth labels for its reasoning process across different driving aspects and the final decisions. This dataset can serve as an open-loop end-to-end driving benchmark, enabling the evaluation of accuracy in various chain-of-thought aspects and the final decision. In addition, we propose a baseline model called DriveCoT-Agent, trained on our dataset, to generate chain-of-thought predictions and final decisions. The trained model exhibits strong performance in both open-loop and closed-loop evaluations, demonstrating the effectiveness of our proposed dataset.

Motivation & Objective

• To address the lack of interpretability and controllability in end-to-end autonomous driving systems.
• To create a comprehensive dataset with detailed chain-of-thought labels for driving decisions in safety-critical scenarios.
• To develop a baseline model that generates reasoning traces and final driving decisions using multi-view video input.
• To enable evaluation of model performance not only on final decisions but also on reasoning accuracy across multiple driving aspects.
• To improve generalization and robustness in high-speed and complex driving scenarios through CoT-integrated learning.

Proposed method

• The DriveCoT dataset is collected using the CARLA simulator with multi-view cameras and LiDAR, capturing sensor data during high-speed and lane-changing scenarios.
• A rule-based expert policy generates ground-truth chain-of-thought labels for each driving decision, covering aspects like collision risk, traffic light status, and pedestrian presence.
• The DriveCoT-Agent model takes multi-view video and a target point as input to predict CoT reasoning for different driving aspects.
• Final driving decisions (speed and waypoints) are derived from the CoT predictions through a reasoning aggregation mechanism.
• Data augmentation with motion-injected video sequences is applied to improve model generalization and motion understanding.
• The model is fine-tuned on a mixed dataset of real and augmented data, then evaluated in both open-loop and closed-loop settings.

Experimental results

Research questions

• RQ1 Can chain-of-thought reasoning improve interpretability and controllability in end-to-end autonomous driving?
• RQ2 How effective is a multi-view video-based model in generating accurate and coherent driving reasoning traces?
• RQ3 Can a CoT-integrated model generalize to unseen, complex driving scenarios such as high-speed driving and lane changes?
• RQ4 Does integrating reasoning traces lead to improved performance in closed-loop evaluations compared to prior end-to-end methods?
• RQ5 To what extent can a model fine-tuned on synthetic CoT data generalize to real-world driving scenarios?

Key findings

• DriveCoT-Agent achieves a driving score of 73.6, route completion of 96.8%, and infraction score of 0.76 on the Town05Long benchmark, outperforming prior methods.
• On the more challenging CARLA leaderboard 2.0 with unseen routes, DriveCoT-Agent achieves a driving score of 51.9, significantly surpassing previous methods.
• The model demonstrates strong zero-shot generalization to real-world nuScenes data, correctly identifying traffic lights and pedestrians and making appropriate braking or driving decisions.
• Qualitative results show that DriveCoT-Agent correctly predicts potential collisions with merging vehicles, pedestrians, and stopped objects by analyzing motion in video input.
• The integration of CoT reasoning enables the model to make safer and more interpretable decisions, particularly in dynamic and high-speed scenarios.
• Data augmentation with motion-injected video improved model performance, especially in predicting hazards from moving objects.