[Paper Review] End-to-end Autonomous Driving: Challenges and Frontiers

Li Chen, Penghao Wu+4 more|arXiv (Cornell University)|Jun 29, 2023

Autonomous Vehicle Technology and SafetyEngineering22 citations

TL;DR

A comprehensive survey of end-to-end autonomous driving, analyzing imitation and reinforcement learning approaches, benchmarks, challenges, and future trends, with emphasis on multi-modality, interpretability, world models, and foundation models.

ABSTRACT

The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. End-to-end systems, in comparison to modular pipelines, benefit from joint feature optimization for perception and planning. This field has flourished due to the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. In this survey, we provide a comprehensive analysis of more than 270 papers, covering the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. We delve into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, amongst others. Additionally, we discuss current advancements in foundation models and visual pre-training, as well as how to incorporate these techniques within the end-to-end driving framework. we maintain an active repository that contains up-to-date literature and open-source projects at https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving.

Motivation & Objective

• Motivate the shift from modular pipelines to end-to-end learning for driving safety and efficiency.
• Summarize the methodological landscape of imitation and reinforcement learning in end-to-end driving.
• Assess datasets, benchmarks, and evaluation paradigms for closed-loop and open-loop settings.
• Identify key challenges such as multi-modality fusion, interpretability, and generalization, and discuss potential solutions.
• Highlight future directions including foundation models, data engines, and sensor fusion strategies.

Proposed method

• Classifies end-to-end driving approaches into imitation learning and reinforcement learning.
• Describes behavior cloning and inverse optimal control as imitation learning avenues and their challenges (covariate shift and causal confusion).
• Reviews reinforcement learning in driving, including its limitations in real-world deployment and success when combined with supervised pretraining or privileged simulation data.
• Discusses evaluation benchmarks and simulation-based closed-loop vs. offline open-loop paradigms, including CARLA and nuPlan ecosystems.
• Examines world models, multi-task learning, and policy distillation as strategies to improve end-to-end systems.
• Considers the role of foundation models and visual pre-training in policy learning for driving.

Experimental results

Research questions

• RQ1 What are the major paradigms and methodologies used in end-to-end autonomous driving (IL vs RL)?
• RQ2 What benchmarks and evaluation setups best assess end-to-end driving in closed-loop and open-loop contexts?
• RQ3 What are the central challenges (multi-modality, interpretability, causal confusion, robustness, world models) and how can they be addressed?
• RQ4 How can foundation models and data-centric approaches influence end-to-end driving development?
• RQ5 What future directions are most promising for safe and efficient end-to-end autonomous driving?

Key findings

• End-to-end driving can leverage joint optimization across perception, prediction, and planning, potentially improving efficiency and safety.
• Imitation learning (behavior cloning and IOC) remains foundational but suffers from covariate shift and causal confusion, with on-policy strategies and cost learning as remedies.
• RL in real-world driving lags behind IL in end-to-end settings, though pretraining and privileged simulator data can yield strong results in combination with RL.
• Open-loop benchmarks may not reflect real-world performance; closed-loop simulation (e.g., CARLA, nuPlan) provides more reliable evaluation, though generalization remains challenging.
• Multi-modality fusion and Transformer-based architectures (e.g., TransFuser) show strong performance gains in end-to-end systems, particularly for integrating camera/LiDAR data.
• Foundation models and data engines are identified as key trends to advance end-to-end driving, with ongoing maintenance of open literature repositories.