[论文解读] From Motor Control to Team Play in Simulated Humanoid Football

Siqi Liu, Guy Lever+20 more|arXiv (Cornell University)|May 25, 2021

Sports Analytics and Performance被引用 12

一句话总结

本文提出了一种分层强化学习框架，通过整合模仿学习、多智能体强化学习和基于种群的训练方法，使模拟的人形智能体能够学习踢足球。该方法使智能体能够从低层次的运动控制（毫秒级）到高层次的团队协作（秒级）进行学习，在物理上逼真的环境中实现协调的、类人足球动作，涌现出团队战术，并具备可迁移的行为表征。

ABSTRACT

Intelligent behaviour in the physical world exhibits structure at multiple spatial and temporal scales. Although movements are ultimately executed at the level of instantaneous muscle tensions or joint torques, they must be selected to serve goals defined on much longer timescales, and in terms of relations that extend far beyond the body itself, ultimately involving coordination with other agents. Recent research in artificial intelligence has shown the promise of learning-based approaches to the respective problems of complex movement, longer-term planning and multi-agent coordination. However, there is limited research aimed at their integration. We study this problem by training teams of physically simulated humanoid avatars to play football in a realistic virtual environment. We develop a method that combines imitation learning, single- and multi-agent reinforcement learning and population-based training, and makes use of transferable representations of behaviour for decision making at different levels of abstraction. In a sequence of stages, players first learn to control a fully articulated body to perform realistic, human-like movements such as running and turning; they then acquire mid-level football skills such as dribbling and shooting; finally, they develop awareness of others and play as a team, bridging the gap between low-level motor control at a timescale of milliseconds, and coordinated goal-directed behaviour as a team at the timescale of tens of seconds. We investigate the emergence of behaviours at different levels of abstraction, as well as the representations that underlie these behaviours using several analysis techniques, including statistics from real-world sports analytics. Our work constitutes a complete demonstration of integrated decision-making at multiple scales in a physically embodied multi-agent setting. See project video at https://youtu.be/KHMwq9pv7mg.

研究动机与目标

• 弥合具身多智能体系统中低层次运动控制与高层次团队协作之间的差距。
• 研究模仿学习、单智能体与多智能体强化学习以及基于种群的训练在复杂、多尺度行为中的整合方法。
• 在模拟足球环境中实现物理上逼真、类人动作以及涌现的团队策略。
• 利用现实世界体育分析技术，分析跨多个时间与空间尺度的行为涌现现象。
• 展示在具有挑战性的多智能体、基于物理的环境中，端到端学习协调的、长时程行为的能力。

提出的方法

• 采用分阶段训练流程：首先通过从动作捕捉数据中学习基本行走技能。
• 应用单智能体强化学习，发展如运球和射门等中层技能。
• 采用多智能体强化学习结合自对弈，发展团队协作与战术意识。
• 引入可迁移的行为表征，以实现在不同技能层级与智能体之间的知识迁移。
• 利用基于种群的训练优化超参数，并提升在多样化团队配置下的泛化能力。
• 采用具有真实物理动力学的仿真环境，以支持复杂交互，包括身体接触。

实验结果

研究问题

• RQ1 在多智能体、基于物理的仿真环境中，如何有效整合低层次运动控制与高层次团队协作？
• RQ2 可迁移的行为表征在实现多层级抽象的分层学习中起到何种作用？
• RQ3 能否将模仿学习与强化学习结合，以产生自然的类人动作与有效的团队协作？
• RQ4 模拟足球中涌现的团队战术与现实世界体育分析模式有何异同？
• RQ5 端到端学习方法在复杂多智能体环境中，能在多大程度上支持协调的、长时程行为的涌现？

主要发现

• 智能体通过模仿学习成功学习到逼真的类人动作，如奔跑、转向和运球。
• 在模拟环境中，通过单智能体强化学习自然涌现出如射门和传球等中层足球技能。
• 通过多智能体强化学习结合自对弈，涌现出团队协作与战术意识，实现了有效的2v2足球对战。
• 分层技能学习与可迁移表征的整合，实现了从毫秒级到数十秒级的多时间尺度高效学习。
• 利用现实世界体育分析技术进行分析发现，涌现的团队行为与职业足球中观察到的模式高度相似。
• 该框架表明，基于课程的端到端学习方法可在多智能体系统中支持复杂、协调且物理上合理的动作行为。