[论文解读] Modeling Human Driving Behavior through Generative Adversarial Imitation Learning

Raunak Bhattacharyya, Blake Wulfe+5 more|arXiv (Cornell University)|Jun 10, 2020

Anomaly Detection Techniques and Applications被引用 12

一句话总结

该论文提出了一种增强型生成对抗模仿学习（GAIL）框架，利用真实世界的NGSIM数据对人类驾驶行为进行建模。通过引入PS-GAIL实现多智能体交互，RAIL实现领域特定的奖励设计，以及Burn-InfoGAIL实现潜在驾驶风格的解耦，该方法在轨迹模仿中表现出更低的误差，并涌现出逼真的交通动态。

ABSTRACT

An open problem in autonomous vehicle safety validation is building reliable models of human driving behavior in simulation. This work presents an approach to learn neural driving policies from real world driving demonstration data. We model human driving as a sequential decision making problem that is characterized by non-linearity and stochasticity, and unknown underlying cost functions. Imitation learning is an approach for generating intelligent behavior when the cost function is unknown or difficult to specify. Building upon work in inverse reinforcement learning (IRL), Generative Adversarial Imitation Learning (GAIL) aims to provide effective imitation even for problems with large or continuous state and action spaces, such as modeling human driving. This article describes the use of GAIL for learning-based driver modeling. Because driver modeling is inherently a multi-agent problem, where the interaction between agents needs to be modeled, this paper describes a parameter-sharing extension of GAIL called PS-GAIL to tackle multi-agent driver modeling. In addition, GAIL is domain agnostic, making it difficult to encode specific knowledge relevant to driving in the learning process. This paper describes Reward Augmented Imitation Learning (RAIL), which modifies the reward signal to provide domain-specific knowledge to the agent. Finally, human demonstrations are dependent upon latent factors that may not be captured by GAIL. This paper describes Burn-InfoGAIL, which allows for disentanglement of latent variability in demonstrations. Imitation learning experiments are performed using NGSIM, a real-world highway driving dataset. Experiments show that these modifications to GAIL can successfully model highway driving behavior, accurately replicating human demonstrations and generating realistic, emergent behavior in the traffic flow arising from the interaction between driving agents.

研究动机与目标

• 开发一种可靠且数据驱动的方法，用于在自动驾驶车辆仿真中建模人类驾驶行为。
• 解决标准GAIL在处理多智能体交互、领域特定驾驶规则以及人类示范中潜在可变性方面的局限性。
• 通过直接从真实世界驾驶数据中学习神经策略，实现在高速公路场景中准确、可扩展且逼真的交通仿真。
• 提升在高维、随机且非线性驾驶环境中模仿学习的性能，其中成本函数未知。

提出的方法

• 提出PS-GAIL，即GAIL的一种参数共享变体，可实现对交通场景中多个相互作用智能体的联合策略学习。
• 引入RAIL（奖励增强型模仿学习），通过修改奖励信号以注入领域知识，如车道保持和安全跟车距离。
• 开发Burn-InfoGAIL，利用互信息最大化方法从专家示范中解耦潜在驾驶风格（如激进型与谨慎型）。
• 采用条件策略网络并结合学习到的潜在代码，以烧入示范序列作为条件，建模个体驾驶风格。
• 使用信任区域策略优化（TRPO）通过对抗性模仿端到端训练策略，最小化专家行为与模仿行为之间的分布差距。
• 利用LIDAR和道路特征输入作为状态表示，采用两层MLP策略头和两层LSTM编码器用于潜在代码推理。

实验结果

研究问题

• RQ1 GAIL能否被有效扩展以建模高速公路驾驶场景中的多智能体交互？
• RQ2 如何在不显式进行奖励设计的情况下，将领域特定的驾驶规则（如车道保持、安全跟车距离）融入模仿学习？
• RQ3 通过表示学习，人类驾驶行为中的潜在可变性（如个体驾驶风格）在多大程度上可以被解耦并建模？
• RQ4 PS-GAIL、RAIL与Burn-InfoGAIL的组合是否能提升轨迹模仿的准确度并涌现出逼真的交通动态？

主要发现

• Burn-InfoGAIL在1000次滚动仿真中对速度和位置的预测均达到最低的均方根误差（RMSE），在长时程驾驶中优于所有基线方法。
• 仅使用GAIL时表现出模式聚焦行为，导致策略逐渐趋同于平均策略，并在约10秒后丢失特定风格的行为特征。
• VAE基线方法由于潜在代码的条件作用，保持了更好的短期风格保真度，但因误差累积导致轨迹偏离道路。
• RAIL通过将领域知识注入奖励信号，改善了策略行为，使车道保持和跟车行为更加真实。
• PS-GAIL成功建模了协调的多智能体行为，实现了密集交通中如变道和跟车等逼真的交互行为。
• 三者扩展方法——PS-GAIL、RAIL与Burn-InfoGAIL的结合，产生了涌现的、逼真的交通流模式，与真实世界的NGSIM数据高度一致。