[论文解读] Towards Knowledge-driven Autonomous Driving

Xin Li, Yeqi Bai+15 more|arXiv (Cornell University)|Dec 7, 2023

Image Processing and 3D Reconstruction被引用 11

一句话总结

本文综述知識驱动的自动驾驶，指出数据驱动系统的局限性，并提出一个由数据集/基准、环境和驾驶员代理组成的框架，通过大语言模型、世界模型和神经渲染来实现认知、泛化和终身学习。

ABSTRACT

This paper explores the emerging knowledge-driven autonomous driving technologies. Our investigation highlights the limitations of current autonomous driving systems, in particular their sensitivity to data bias, difficulty in handling long-tail scenarios, and lack of interpretability. Conversely, knowledge-driven methods with the abilities of cognition, generalization and life-long learning emerge as a promising way to overcome these challenges. This paper delves into the essence of knowledge-driven autonomous driving and examines its core components: dataset \& benchmark, environment, and driver agent. By leveraging large language models, world models, neural rendering, and other advanced artificial intelligence techniques, these components collectively contribute to a more holistic, adaptive, and intelligent autonomous driving system. The paper systematically organizes and reviews previous research efforts in this area, and provides insights and guidance for future research and practical applications of autonomous driving. We will continually share the latest updates on cutting-edge developments in knowledge-driven autonomous driving along with the relevant valuable open-source resources at: \url{https://github.com/PJLab-ADG/awesome-knowledge-driven-AD}.

研究动机与目标

• 突出数据驱动自动驾驶的局限性（数据偏差、长尾效应、缺乏可解释性）。
• 定义知识驱动范式及其三大核心组成：数据集与基准、环境、驾驶员代理。
• 回顾现有研究趋势，提出利用先进人工智能技术（如大语言模型和世界模型）的知识驱动解决方案。
• 讨论机会、挑战以及未来在知识驱动自动驾驶中的应用与实践指南。

提出的方法

• 对知识驱动自动驾驶文献进行系统综述。
• 提出一个由三大核心组件组成的框架：数据集与基准、环境、驾驶员代理。
• 讨论使能技术，包括大语言模型、世界模型和神经渲染。
• 分析从感知向认知的转变及其对规划与控制的影响。
• 强调开源资源和社区驱动更新的作用（GitHub 链接）。

Figure 1: Comparison of three technical paradigms to autonomous driving. (1) The rule-based paradigm utilizes the understanding of driving scenarios that are summarized in the scenario semantic space to guide driving. (2) The data-based paradigm tends to model the driving scenarios into the represen

实验结果

研究问题

• RQ1 什么构成一个知识驱动的自动驾驶系统？
• RQ2 如何设计数据集、环境和驾驶员代理以实现知识驱动学习？
• RQ3 相比数据驱动方法，知识驱动驾驶在泛化和可解释性方面有哪些优势？
• RQ4 知识驱动自动驾驶的主要挑战及潜在解决方案是什么？

主要发现

• 知识驱动方法通过将人类知识和常识纳入驾驶表示，旨在实现跨领域的泛化。
• 大语言模型和世界模型实现对驾驶场景的推理、场景理解和未来内容预测。
• 神经渲染通过闭环仿真支持真实、丰富的环境和罕见边缘场景。
• 端到端与模块化设计之间存在权衡；知识驱动方法寻求可解释、以知识为基础的输出。
• 论文指出有价值的开源资源与基准，促进知识驱动自动驾驶的进展。

Figure 2: Comparison between the single-domain data-driven paradigm (left), cross-domain data-driven paradigm (center), and the knowledge-driven paradigm (right). The gray $\times$ in the driving scenario represents corner cases, while it transitions to green $\times$ , indicating that the method ca