[论文解读] Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving

Haochen Liu, Zhiyu Huang+2 more|arXiv (Cornell University)|Aug 24, 2022

Autonomous Vehicle Technology and Safety被引用 8

一句话总结

本文提出Scene-Rep Transformer，一种新颖的框架，通过整合用于交互式和意图感知场景表征的多阶段Transformer，以及采用自监督学习将未来预测信息提炼至当前潜在状态的序列潜在Transformer（SLT），从而增强强化学习（RL）在自动驾驶中的应用。该方法显著提升了复杂城市驾驶场景下的样本效率与决策性能，相较于基线RL方法，实现了更高的成功率、更优的安全性以及更多样化的驾驶行为。

ABSTRACT

Decision-making for urban autonomous driving is challenging due to the stochastic nature of interactive traffic participants and the complexity of road structures. Although reinforcement learning (RL)-based decision-making scheme is promising to handle urban driving scenarios, it suffers from low sample efficiency and poor adaptability. In this paper, we propose Scene-Rep Transformer to improve the RL decision-making capabilities with better scene representation encoding and sequential predictive latent distillation. Specifically, a multi-stage Transformer (MST) encoder is constructed to model not only the interaction awareness between the ego vehicle and its neighbors but also intention awareness between the agents and their candidate routes. A sequential latent Transformer (SLT) with self-supervised learning objectives is employed to distill the future predictive information into the latent scene representation, in order to reduce the exploration space and speed up training. The final decision-making module based on soft actor-critic (SAC) takes as input the refined latent scene representation from the Scene-Rep Transformer and outputs driving actions. The framework is validated in five challenging simulated urban scenarios with dense traffic, and its performance is manifested quantitatively by the substantial improvements in data efficiency and performance in terms of success rate, safety, and efficiency. The qualitative results reveal that our framework is able to extract the intentions of neighbor agents to help make decisions and deliver more diversified driving behaviors.

研究动机与目标

• 解决基于强化学习的城市场景自动驾驶决策中样本效率低与适应性差的问题。
• 通过建模自车、邻近车辆与候选路径之间的交互，改进场景表征学习。
• 将未来动态的预测信息注入当前潜在状态，以缩小探索空间并加速训练。
• 通过意图感知的表征实现更可解释且多样的驾驶行为。
• 在密集交通的复杂模拟城市场景中验证该框架的有效性。

提出的方法

• 多阶段Transformer（MST）将异构场景元素（如智能体轨迹与候选路径航点）编码为统一的潜在表征，同时捕捉智能体交互与路径意图。
• 序列潜在Transformer（SLT）利用自监督学习，将未来潜在-动作对序列的特征提炼至当前潜在向量中，嵌入预测知识。
• SLT仅在强化学习训练阶段进行训练，以指导策略学习，不影响推理阶段。
• 软演员评论家（SAC）智能体使用优化后的潜在表征作为输入，生成高质量的驾驶决策。
• 整个框架端到端训练，以最大化累积奖励与任务成功率。
• SLT中的未来时域超参数$T_G$通过调优，以在不同任务长度下平衡预测准确率与训练稳定性。

Figure 1: An overview of our RL decision-making framework with Scene-Rep Transformer. Given perception-processed vectorized scene inputs, the multi-stage Transformer (MST) encodes the multi-modal information with interaction awareness. During training, a sequential latent Transformer (SLT) performs

实验结果

研究问题

• RQ1 基于Transformer的场景表征模型能否有效编码异构驾驶元素，包括智能体交互与路径意图？
• RQ2 将序列预测信息提炼至当前潜在状态是否能缩小探索空间并提升基于强化学习的决策训练效率？
• RQ3 在复杂城市场景任务中，该框架相较于基线强化学习方法，在成功率、安全性与行为多样性方面表现如何？
• RQ4 在长期驾驶场景中，预测提炼的最优未来时域$T_G$是多少？
• RQ5 该模型能否学习到可解释的驾驶行为，如变道、激进转弯与汇入操作？

主要发现

• Scene-Rep Transformer在数据效率方面实现显著提升，相较于PPO与DrQ基线方法，达到高性能所需训练样本数量大幅减少。
• 在五个模拟城市场景（包括无保护左转、双侧汇入、环岛等）中，该方法实现了更高的成功率，定性结果表明其驾驶操作更具多样性与安全性。
• 自车智能体学习到保守与激进的双重行为，如左转时变道、汇入时多次变道，表明策略具有强大的泛化能力。
• 当$T_G = 10$时，SLT在如Roundabout-C等长时任务中达到最佳性能，表明更长的预测时域有助于复杂场景下的学习。
• 注意力可视化结果证实，模型能自适应地聚焦于与意图相关的关键线索，如候选航点与交互动态，从而提升可解释性。
• 消融实验表明，多阶段Transformer与SLT均对性能有显著贡献，其中SLT有效缩小探索空间并加速收敛。

Figure 2: An overview of the MST Structure. The urban scenario on the left side can be layered to explicit representations $s_{t}$ as the historical motion states $\mathcal{M}_{t}$ and candidate route waypoints $\mathcal{K}_{t}$ . The implicit layer for agent interactions is modeled in the subsequen