[Paper Review] Collaborative AI Teaming in Unknown Environments via Active Goal Deduction

Zuyuan Zhang, Hanhan Zhou+3 more|arXiv (Cornell University)|Mar 22, 2024

Human-Automation Interaction and Safety5 citations

TL;DR

STUN proposes kernel density Bayesian inverse learning (KD-BIL) to infer latent rewards of unknown agents and enables zero-shot policy adaptation of goal-conditioned policies for synergistic teaming in unknown environments.

ABSTRACT

With the advancements of artificial intelligence (AI), we're seeing more scenarios that require AI to work closely with other agents, whose goals and strategies might not be known beforehand. However, existing approaches for training collaborative agents often require defined and known reward signals and cannot address the problem of teaming with unknown agents that often have latent objectives/rewards. In response to this challenge, we propose teaming with unknown agents framework, which leverages kernel density Bayesian inverse learning method for active goal deduction and utilizes pre-trained, goal-conditioned policies to enable zero-shot policy adaptation. We prove that unbiased reward estimates in our framework are sufficient for optimal teaming with unknown agents. We further evaluate the framework of redesigned multi-agent particle and StarCraft II micromanagement environments with diverse unknown agents of different behaviors/rewards. Empirical results demonstrate that our framework significantly advances the teaming performance of AI and unknown agents in a wide range of collaborative scenarios.

Motivation & Objective

• Motivate the need for synergistic teaming with unknown agents that have latent objectives or rewards.
• Develop a sample-efficient active goal inference method to estimate latent rewards from observed trajectories.
• Pre-train goal-conditioned policies on surrogate models to support zero-shot adaptation when interfacing with unknown agents.
• Prove that unbiased reward estimates suffice for optimal policy learning and demonstrate improved teaming performance in customized multi-agent environments.

Proposed method

• Introduce a dec-POMDP framework with unknown agents (latent rewards) and collaborative agents (STUN).
• Propose Kernel Density Bayesian Inverse Learning (KD-BIL) to obtain a posterior over latent rewards from observed trajectories using kernel density estimates.
• Show that a maximum a posteriori (MAP) estimate of rewards is not sufficient for optimality, and prove that unbiased reward estimates ensure Bellman convergence.
• Pre-train goal-conditioned policies pi(a|o,R) using surrogate models with randomly sampled rewards to enable zero-shot adaptation via unbiased reward estimates.
• Develop a zero-shot adaptation rule pi(a|o,âR) by conditioning on the unbiased reward estimate to achieve near-optimal teaming without re-training.
• Demonstrate scalability with centralized pre-training and decentralized execution across redesigned MPE/SMAC environments.

Figure 1: We consider the problem of enabling synergistic teaming of AI agents with other unknown agents (e.g., human or autonomous agents that could have latent rewards/objectives) in collaborative task environments.

Experimental results

Research questions

• RQ1 Can unknown agents with latent rewards be effectively reasoned about and collaborated with in real-time?
• RQ2 Is KD-BIL an efficient and accurate method to infer latent rewards from limited observations?
• RQ3 Does zero-shot adaptation using unbiased reward estimates guarantee optimal or near-optimal teaming performance?
• RQ4 How do STUN agents perform when teamed with diverse unknown agents across different environments (MPE/SMAC) compared to baselines?

Key findings

• KD-BIL provides a sample-efficient posterior over latent rewards and works with time-varying objectives.
• Unbiased reward estimates are necessary and sufficient for Bellman convergence and optimal Q-values under policy learning.
• Zero-shot adaptation of pre-trained goal-conditioned policies yields near-optimal teaming without re-training across diverse unknown agents.
• STUN agents outperform baselines and achieve close-to-optimal teaming in redesigned MPE and SMAC tasks, including hard maps and agents with varied latent rewards.
• STUN demonstrates rapid adaptation to changing unknown agents, with performance gains up to 50% on challenging maps in some cases.

Figure 2: An illustration of our proposed framework. STUN agents $\pi_{i}(\cdot|R)$ are pre-trained using surrogate agent models with sampled latent rewards $R$ . To collaborate with unknown agents, they use inverse learning (step b) on the observed trajectories $\{\tau^{u}_{i}\}$ of the unknown age