[论文解读] Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles

Can Cui, Yunsheng Ma+3 more|arXiv (Cornell University)|Oct 12, 2023

Human-Automation Interaction and Safety被引用 8

一句话总结

该论文提出一个以人为本的框架，使用大型语言模型（LLMs）作为自治车辆的决策大脑，并与感知、定位和座舱内工具协同，解释指令、进行连锁思维推理并个性化驾驶行为。

ABSTRACT

The fusion of human-centric design and artificial intelligence (AI) capabilities has opened up new possibilities for next-generation autonomous vehicles that go beyond transportation. These vehicles can dynamically interact with passengers and adapt to their preferences. This paper proposes a novel framework that leverages Large Language Models (LLMs) to enhance the decision-making process in autonomous vehicles. By utilizing LLMs' linguistic and contextual understanding abilities with specialized tools, we aim to integrate the language and reasoning capabilities of LLMs into autonomous vehicles. Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks, to explore LLMs' interpretation, interaction, and reasoning in various scenarios. We also examine real-time personalization, demonstrating how LLMs can influence driving behaviors based on verbal commands. Our empirical results highlight the substantial advantages of utilizing chain-of-thought prompting, leading to improved driving decisions, and showing the potential for LLMs to enhance personalized driving experiences through ongoing verbal feedback. The proposed framework aims to transform autonomous vehicle operations, offering personalized support, transparent decision-making, and continuous learning to enhance safety and effectiveness. We achieve user-centric, transparent, and adaptive autonomous driving ecosystems supported by the integration of LLMs into autonomous vehicles.

研究动机与目标

• 展示一个以人为本、LLM 集成的自治车辆框架，该框架将 LLM 与感知、定位和座舱监控协同工作。
• 在 HighwayEnv 驾驶场景中研究 LLM 的解释、交互和推理能力。
• 评估连锁思维提示对驾驶决策与安全性的影响。
• 通过口头指令实现驾驶风格的实时个性化展示。
• 分析情境内学习与记忆在自适应自治驾驶行为中的作用。

提出的方法

• 将 LLM（GPT-4）作为车辆的决策大脑，查询感知、定位和记忆模块。
• 使用工具和记忆向 LLM 提供环境观测，便于推理与行动规划。
• 在高速公路超车与并线场景中比较连锁思维提示与标准提示的效果。
• 在 HighwayEnv 中通过口头指令（以更保守/更激进的方式驾驶）进行驾驶风格个性化实验。
• 将感知、定位、座舱监控整合作为车载感官输入，为基于 LLM 的决策提供信息。

实验结果

研究问题

• RQ1 当结合感知、定位和记忆工具时，LLMs 是否能作为自治车辆的有效决策大脑？
• RQ2 连锁思维提示是否提升复杂场景下的驾驶决策安全性和可行性？
• RQ3 驾驶员指令在多大程度上能够实时影响并个性化自治驾驶行为？
• RQ4 情境内学习与记忆在提升基于 LLM 的自治驾驶决策中的作用是什么？

主要发现

Commands	Mean Abs Acceleration	Mean Abs Steering	Max Abs Speed	Min Front Gap	Overall Time	# Chaging Lanes
Drive more Aggresively	3.10 m/s^2	0.03 rad	34.77 m/s	7.17 m	24.33 s	6
Drive more Conservatively	0.18 m/s^2	0.01 rad	20.00 m/s	39.14 m	46.20 s	1
No Extra Command	1.41 m/s^2	0.02 rad	27.43 m/s	24.01 m	34.20 s	2

• 连锁思维提示在高速场景下比标准提示能产生更安全、可行的超车决策。
• LLMs 能解读传感器观测与当地法律，生成包含变道、速度调整以及在安全情况下重新回到车道的行动计划。
• 口头指令（如更保守/更激进）在加速、转向、速度、前方间隙和路线时间等方面产生明显且可测量的变化。
• 若缺乏情境性提示，LLMs 可能在交通情景中做出鲁莽决策，凸显了连锁思维引导的价值。
• 情境内学习与记忆使系统在不完全重新训练的情况下适应新场景，降低计算和资金成本。