[论文解读] GPT-Driver: Learning to Drive with GPT

Jiageng Mao, Yuxi Qian+3 more|arXiv (Cornell University)|Oct 2, 2023

Natural Language Processing Techniques被引用 39

一句话总结

GPT-Driver 通过将规划视为语言建模，将 GPT-3.5 重新用于运动规划，使用 prompting-reasoning-finetuning 策略在 nuScenes 的自动驾驶中生成精确轨迹并具有可解释的推理链。

ABSTRACT

We present a simple yet effective approach that can transform the OpenAI GPT-3.5 model into a reliable motion planner for autonomous vehicles. Motion planning is a core challenge in autonomous driving, aiming to plan a driving trajectory that is safe and comfortable. Existing motion planners predominantly leverage heuristic methods to forecast driving trajectories, yet these approaches demonstrate insufficient generalization capabilities in the face of novel and unseen driving scenarios. In this paper, we propose a novel approach to motion planning that capitalizes on the strong reasoning capabilities and generalization potential inherent to Large Language Models (LLMs). The fundamental insight of our approach is the reformulation of motion planning as a language modeling problem, a perspective not previously explored. Specifically, we represent the planner inputs and outputs as language tokens, and leverage the LLM to generate driving trajectories through a language description of coordinate positions. Furthermore, we propose a novel prompting-reasoning-finetuning strategy to stimulate the numerical reasoning potential of the LLM. With this strategy, the LLM can describe highly precise trajectory coordinates and also its internal decision-making process in natural language. We evaluate our approach on the large-scale nuScenes dataset, and extensive experiments substantiate the effectiveness, generalization ability, and interpretability of our GPT-based motion planner. Code is now available at https://github.com/PointsCoder/GPT-Driver.

研究动机与目标

• 将运动规划转换为语言建模问题，通过将输入与输出编码为语言令牌。
• 利用 GPT-3.5 与 prompting-reasoning-finetuning 策略，以提升数值精度和决策透明度。
• 在 nuScenes 数据集上展示该方法，并与最先进的运动规划器进行比较。
• 评估基于 GPT 的规划器的泛化、少样本学习能力和可解释性。

提出的方法

• 通过 GPT-3.5 的分词编码，将观测与自车状态表示为语言提示。
• 将轨迹生成表述为预测对应于航路点坐标的一系列令牌。
• 使用 prompting-reasoning-finetuning 流水线，诱发推理链条式思维，然后用语言生成轨迹。
• 用人工驾驶轨迹对 LLM 输出进行微调，以对齐真实世界行为。
• 同时提供规划的轨迹和模型的推理过程，以增强可解释性。

Figure 1: Overview of GPT-Driver. We reformulate motion planning as a language modeling problem. We convert observations and ego-states into language prompts, guiding the LLM to produce a planned trajectory alongside its decision-making process in natural language. Subsequently, this planned traject

实验结果

研究问题

• RQ1 大型语言模型（LLMs）是否可用于自动驾驶中的高精度、低级别运动规划？
• RQ2 将规划转变为语言建模问题，是否能比传统规划器改善泛化和可解释性？
• RQ3 提示、推理和微调策略对轨迹精度和安全性的影响如何？

主要发现

方法	L2 (1s)	L2 (2s)	L2 (3s)	L2 平均	碰撞（1s）	碰撞（2s）	碰撞（3s）	碰撞 平均
ST-P3 Hu et al. (2022)	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71
VAD Jiang et al. (2023)	0.17	0.34	0.60	0.37	0.07	0.10	0.24	0.14
GPT-Driver (ours)	0.20	0.40	0.70	0.44	0.04	0.12	0.36	0.17
UniAD Hu et al. (2023)	5.37	1.80	1.42	1.03	6.86	1.31	0.49	0.31
GPT-Driver (in-context learning)	2.41	3.11	4.00	3.17	4.20	5.13	6.58	5.30
GPT-Driver (fine-tuning)	0.27	0.74	1.52	0.84	0.07	0.15	1.10	0.44

• GPT-Driver 在 nuScenes 的 1s、2s、3s 时距下显著提升了 L2 误差，相较于此前方法。
• 该方法达到具有竞争力的碰撞率，表明与最先进的规划器相比的安全性。
• 微调在此任务上优于 in-context learning，展示出强数据效率和泛化能力。
• GPT-Driver 展示了少样本泛化，在训练数据有限时相较于 UniAD 仍表现良好。
• 模型通过在轨迹旁边包含类似推理过程的链式思维来实现可解释性输出。

Figure 2: An example of input prompts provided to the LLM. The upper text box offers a universal context related to motion planning for every driving scenario. The lower text box provides a language description of the observations and ego-states specific to this particular frame. Parameterized input