[论文解读] Transformers in Reinforcement Learning: A Survey

Pranav Agarwal, Aamer Abdul Rahman+3 more|arXiv (Cornell University)|Jul 12, 2023

Reinforcement Learning in Robotics被引用 8

一句话总结

这份综述分析了变换器在表示学习、转移与奖励建模，以及策略优化中的应用，以应对强化学习中的不稳定性、信用分配和部分可观测性等挑战。

ABSTRACT

Transformers have significantly impacted domains like natural language processing, computer vision, and robotics, where they improve performance compared to other neural networks. This survey explores how transformers are used in reinforcement learning (RL), where they are seen as a promising solution for addressing challenges such as unstable training, credit assignment, lack of interpretability, and partial observability. We begin by providing a brief domain overview of RL, followed by a discussion on the challenges of classical RL algorithms. Next, we delve into the properties of the transformer and its variants and discuss the characteristics that make them well-suited to address the challenges inherent in RL. We examine the application of transformers to various aspects of RL, including representation learning, transition and reward function modeling, and policy optimization. We also discuss recent research that aims to enhance the interpretability and efficiency of transformers in RL, using visualization techniques and efficient training strategies. Often, the transformer architecture must be tailored to the specific needs of a given application. We present a broad overview of how transformers have been adapted for several applications, including robotics, medicine, language modeling, cloud computing, and combinatorial optimization. We conclude by discussing the limitations of using transformers in RL and assess their potential for catalyzing future breakthroughs in this field.

研究动机与目标

• 解释关键的强化学习挑战以及变换器如何应对它们。
• 综述变换器变体及其对强化学习任务的适用性。
• 按表示学习、转移/奖励建模和策略学习对变换器应用进行分类。
• 讨论强化学习中使用变换器的训练、可解释性和效率改进。
• 概述在强化学习中变换器的应用、局限性及未来发展方向。

提出的方法

• 对强化学习基础和变换器基础知识进行结构化概述。
• 描述如何将变换器整合到强化学习工作流中，用于表示学习、转移建模、奖励建模和策略优化。
• 总结架构变体（BERT、GPT、ViT、Transformer-XL）及其对强化学习的影响。
• 讨论基于变换器的强化学习的训练策略和可解释性技术。
• 调查多样化的应用领域及潜在局限性，以指导未来工作。

Figure 1 . This survey presents a comprehensive overview of the use of transformers in RL . Modeling an RL policy may involve representation learning, modeling the transition function, reward function learning, and policy learning. Transformers can be used across all of these tasks.

实验结果

研究问题

• RQ1 变换器如何缓解强化学习中的部分可观测性与长期信用分配问题？
• RQ2 哪种变换器架构及配置最适合强化学习任务和数据模态？
• RQ3 在强化学习的哪些阶段（表示、转移、奖励、策略）中，变换器能带来最大的收益？
• RQ4 哪些训练与可解释性策略可以提升基于变换器的强化学习方法？
• RQ5 当前变换器在强化学习中的局限性及未来研究方向是什么？

主要发现

• 变换器在建模长程依赖、多模态数据，以及可并行化训练方面对强化学习具有优势。
• 它们能够产生比卷积神经网络在某些泛化设置下更具表达力的表示，并在多任务强化学习中可替代某些图神经网络的角色。
• 基于变换器的方法支持元强化学习和具备记忆的策略学习，提升稳定性与适应性。
• 视觉变换器使对基于图像的输入在强化学习任务中得到有效处理。
• 变换器的可扩展性暗示在跨领域的通用、任务无关代理方面的潜力。

Figure 2 . a) The dot-product attention mechanism (for 4 embeddings of size 5 each). Input features ( $\mathbf{X}$ ) are processed using the key ( $\mathbf{K}$ ), query ( $\mathbf{Q}$ ), and value ( $\mathbf{V}$ ) tensors. Each query undergoes a dot product with every key, and the result is normaliz