[论文解读] Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

Zhiyu Huang, Jingda Wu+1 more|arXiv (Cornell University)|Mar 19, 2021

Reinforcement Learning in Robotics参考文献 29被引用 11

一句话总结

该论文提出了一种深度强化学习框架，通过行为克隆和不确定性估计，将人类专家演示提炼为模仿性专家策略，从而在自动驾驶中提升样本效率并减少奖励函数设计工作。该方法通过KL散度将DRL智能体的策略正则化为专家先验，相比SAC方法样本效率提高60%，并可在稀疏奖励下实现类人驾驶行为。

ABSTRACT

Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this paper proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions. Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios (unprotected left turn and roundabout) are designed. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60\% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse and human-like driving behaviors as demonstrated by the human expert.

研究动机与目标

• 解决深度强化学习在自动驾驶中样本效率低和奖励函数设计复杂的问题。
• 将人类先验知识从专家演示中迁移至DRL智能体，以提升训练效率和策略质量。
• 仅使用稀疏奖励信号即实现类人驾驶行为的涌现。
• 研究专家策略不确定性估计和训练数据规模对性能的影响。
• 开发一种可扩展的框架，在保持探索能力的同时引导策略趋向专家行为。

提出的方法

• 在模拟环境中收集人类专家执行驾驶任务时的状态-动作对作为专家演示。
• 通过行为克隆结合不确定性估计，推导出模仿性专家策略，以量化预测动作的置信度。
• 通过最小化智能体策略与模仿性专家策略之间的KL散度，对DRL智能体的策略进行正则化。
• 提出两种集成方法：价值惩罚和策略约束，二者均嵌入演员-评论家强化学习框架中。
• 利用集成方法构建不确定性感知的专家策略，以同时估计策略不确定性和模型不确定性。
• 在两种复杂的城市场景中验证该框架：无保护左转和环岛导航。

实验结果

研究问题

• RQ1 能否有效将专家演示提炼为模仿性专家策略，从而提升DRL的样本效率？
• RQ2 在专家策略中引入不确定性估计如何影响DRL训练的稳定性和性能？
• RQ3 使用模仿性专家先验在多大程度上可减少自动驾驶中复杂奖励函数设计的需求？
• RQ4 训练样本规模和不确定性估计方法对复杂驾驶任务性能有何影响？
• RQ5 所提方法能否仅通过稀疏奖励反馈实现类人驾驶行为？

主要发现

• 所提方法相比SAC（当前最先进基线）样本效率提高60%。
• 当训练样本规模为40组演示时，该方法在保守型无保护左转场景中成功率达到96%，而10组样本时仅为70%。
• 在稀疏奖励环境中，价值惩罚方法优于策略约束，尤其在复杂任务中表现更优。
• 采用基于集成的专家策略并联合估计策略与模型不确定性，可获得更优性能，尤其在困难场景中。
• 智能体成功学习到多样化、类人的驾驶行为（包括激进和保守风格），而无需显式奖励函数设计。
• 增加演示数据规模可同时提升训练速度和最终测试性能，且在挑战性任务中提升最为显著。