[论文解读] End-to-end Autonomous Driving: Challenges and Frontiers

Li Chen, Penghao Wu+4 more|arXiv (Cornell University)|Jun 29, 2023

Autonomous Vehicle Technology and Safety被引用 22

一句话总结

对端到端自动驾驶的综合综述，分析模仿学习与强化学习方法、基准、挑战与未来趋势，着重多模态、可解释性、世界模型和基础模型。

ABSTRACT

The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. End-to-end systems, in comparison to modular pipelines, benefit from joint feature optimization for perception and planning. This field has flourished due to the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. In this survey, we provide a comprehensive analysis of more than 270 papers, covering the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. We delve into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, amongst others. Additionally, we discuss current advancements in foundation models and visual pre-training, as well as how to incorporate these techniques within the end-to-end driving framework. we maintain an active repository that contains up-to-date literature and open-source projects at https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving.

研究动机与目标

• 推动从模块化管线向端到端学习在驾驶安全与效率方面的转变。
• 总结端到端驾驶中模仿学习与强化学习的方法论格局。
• 评估闭环与开环设置的数据集、基准与评估范式。
• 识别多模态融合、可解释性和泛化等关键挑战，并讨论潜在解决方案。
• 强调未来方向，包括基础模型、数据引擎和传感融合策略。

提出的方法

• 将端到端驾驶方法分为模仿学习和强化学习。
• 描述行为克隆和逆最优控制作为模仿学习途径及其挑战（协变量偏移与因果混淆）。
• 回顾驾驶中的强化学习，包括在实际部署中的局限性以及与监督预训练或特权仿真数据结合时的成功。
• 讨论评估基准与基于仿真的闭环对比离线开环范式，包括CARLA和nuPlan生态。
• 考察世界模型、多任务学习和策略蒸馏作为提升端到端系统的策略。
• 考虑基础模型和视觉预训练在策略学习中的作用。

实验结果

研究问题

• RQ1 端到端自动驾驶中使用的主要范式与方法有哪些（IL 与 RL）？
• RQ2 哪些基准与评估设置最能在闭环和开环情境中评估端到端驾驶？
• RQ3 核心挑战（多模态、可解释性、因果混淆、鲁棒性、世界模型）是什么，以及如何应对？
• RQ4 基础模型和数据驱动方法如何影响端到端驾驶的发展？
• RQ5 哪些未来方向对安全高效的端到端自动驾驶最具前景？

主要发现

• 端到端驾驶可以在感知、预测与规划的联合优化中受益，潜在地提高效率与安全。
• 模仿学习（行为克隆与 IOC）仍然是基础，但受限于协变量偏移和因果混淆，现场策略和成本学习可作为补救。
• RL in real-world driving lags behind IL in end-to-end settings, though pretraining and privileged simulator data can yield strong results in combination with RL.
• Open-loop benchmarks may not reflect real-world performance; closed-loop simulation (e.g., CARLA, nuPlan) provides more reliable evaluation, though generalization remains challenging.
• 多模态融合和 Transformer-based 架构（如 TransFuser）在端到端系统中表现出强劲的性能提升，尤其是在整合摄像头/LiDAR 数据方面。
• Foundation models and data engines are identified as key trends to advance end-to-end driving, with ongoing maintenance of open literature repositories.