[论文解读] Collaborative AI Teaming in Unknown Environments via Active Goal Deduction

Zuyuan Zhang, Hanhan Zhou+3 more|arXiv (Cornell University)|Mar 22, 2024

Human-Automation Interaction and Safety被引用 5

一句话总结

STUN proposes kernel density Bayesian inverse learning (KD-BIL) to infer latent rewards of unknown agents and enables zero-shot policy adaptation of goal-conditioned policies for synergistic teaming in unknown environments.

ABSTRACT

With the advancements of artificial intelligence (AI), we're seeing more scenarios that require AI to work closely with other agents, whose goals and strategies might not be known beforehand. However, existing approaches for training collaborative agents often require defined and known reward signals and cannot address the problem of teaming with unknown agents that often have latent objectives/rewards. In response to this challenge, we propose teaming with unknown agents framework, which leverages kernel density Bayesian inverse learning method for active goal deduction and utilizes pre-trained, goal-conditioned policies to enable zero-shot policy adaptation. We prove that unbiased reward estimates in our framework are sufficient for optimal teaming with unknown agents. We further evaluate the framework of redesigned multi-agent particle and StarCraft II micromanagement environments with diverse unknown agents of different behaviors/rewards. Empirical results demonstrate that our framework significantly advances the teaming performance of AI and unknown agents in a wide range of collaborative scenarios.

研究动机与目标

• 阐明与具有潜在目标或奖励的未知代理进行协同作战的必要性。
• 开发高样本效率的主动目标推断方法，从观测轨迹估计潜在奖励。
• 在代理模型上对目标条件策略进行预训练，以在与未知代理互动时支持零样本自适应。
• 证明无偏奖励估计足以实现最优策略学习，并在定制的多代理环境中展示改进的协同作战性能。

提出的方法

• 引入一个带有未知代理（潜在奖励）和协作代理（STUN）的分布式部分可观测马尔可夫决策过程框架。
• 提出 Kernel Density Bayesian Inverse Learning (KD-BIL)，利用核密度估计从观测轨迹中获得潜在奖励的后验分布。
• 证明奖励的最大后验估计（MAP）对于最优性并不充分，并证明无偏奖励估计可确保 Bellman 收敛。
• 使用带有随机采样奖励的代理模型对目标条件策略 pi(a|o,R) 进行预训练，以通过无偏奖励估计实现零样本自适应。
• 开发通过对无偏奖励估计进行条件化来实现零样本自适应规则 pi(a|o,âR)，以在无需重新训练的情况下实现接近最优的协同。
• 在重新设计的 MPE/SMAC 环境中通过集中预训练和分散执行来展示可扩展性。

Figure 1: We consider the problem of enabling synergistic teaming of AI agents with other unknown agents (e.g., human or autonomous agents that could have latent rewards/objectives) in collaborative task environments.

实验结果

研究问题

• RQ1 是否能够对具有潜在奖励的未知代理进行有效实时推理并与之协作？
• RQ2 KD-BIL 是否是从有限观测中推断潜在奖励的高效且准确的方法？
• RQ3 使用无偏奖励估计的零样本自适应是否能保证最优或近似最优的协同表现？
• RQ4 在不同环境（MPE/SMAC）中与多样化未知代理协作时，STUN 代理相比基线在性能上如何？

主要发现

• KD-BIL 提供了对潜在奖励的样本高效后验，并且适用于随时间变化的目标。
• 在策略学习下，无偏奖励估计是 Bellman 收敛和最优 Q 值的必要且充分条件。
• 对预训练的目标条件策略进行零样本自适应，在多样的未知代理上无重新训练即可实现近似最优的协同。
• STUN代理在重新设计的 MPE 和 SMAC 任务中超越基线，达到了接近最优的协同能力，包括困难地图和潜在奖励不同的代理。
• STUN 展现出对不断变化的未知代理的快速适应，在某些情况下在具有挑战性的地图上性能提升高达 50%。

Figure 2: An illustration of our proposed framework. STUN agents $\pi_{i}(\cdot|R)$ are pre-trained using surrogate agent models with sampled latent rewards $R$ . To collaborate with unknown agents, they use inverse learning (step b) on the observed trajectories $\{\tau^{u}_{i}\}$ of the unknown age