[Paper Review] Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased

Chao Yu, Jiaxuan Gao+6 more|arXiv (Cornell University)|Feb 3, 2023

Mental Health Research Topics4 citations

TL;DR

This paper proposes Hidden-Utility Self-Play (HSP), a novel multi-agent reinforcement learning framework that explicitly models human behavioral biases as hidden reward functions during self-play training. By augmenting the policy pool with policies optimized under diverse biased objectives, HSP enables zero-shot cooperation with humans, achieving significantly higher rewards and superior human-assistive ratings compared to baselines across learned models, scripted policies, and real human participants in the Overcooked benchmark.

ABSTRACT

There is a recent trend of applying multi-agent reinforcement learning (MARL) to train an agent that can cooperate with humans in a zero-shot fashion without using any human data. The typical workflow is to first repeatedly run self-play (SP) to build a policy pool and then train the final adaptive policy against this pool. A crucial limitation of this framework is that every policy in the pool is optimized w.r.t. the environment reward function, which implicitly assumes that the testing partners of the adaptive policy will be precisely optimizing the same reward function as well. However, human objectives are often substantially biased according to their own preferences, which can differ greatly from the environment reward. We propose a more general framework, Hidden-Utility Self-Play (HSP), which explicitly models human biases as hidden reward functions in the self-play objective. By approximating the reward space as linear functions, HSP adopts an effective technique to generate an augmented policy pool with biased policies. We evaluate HSP on the Overcooked benchmark. Empirical results show that our HSP method produces higher rewards than baselines when cooperating with learned human models, manually scripted policies, and real humans. The HSP policy is also rated as the most assistive policy based on human feedback.

Motivation & Objective

• To address the limitation of existing self-play MARL methods that assume human partners optimize the same environment reward function.
• To improve zero-shot generalization in human-AI cooperation by explicitly modeling human behavioral biases.
• To develop a scalable method that generates diverse, biased policies during self-play to enhance adaptation to unseen human partners.
• To evaluate the framework on real human interactions and demonstrate superior assistive behavior beyond prior methods.

Proposed method

• HSP extends standard self-play by introducing a hidden reward function to model human biases, which are assumed to be unknown but consistent.
• The hidden reward space is approximated as a linear function of the environment reward, enabling efficient exploration of biased objectives.
• A random search over the approximated hidden reward space generates a diverse policy pool containing strategies aligned with various human preferences.
• The final adaptive policy is trained against this augmented policy pool to improve zero-shot generalization to unseen human behaviors.
• The method maintains the two-stage framework: (1) policy pool generation with biased objectives, (2) adaptive policy training via imitation or multi-agent RL.
• The approach is evaluated in the Overcooked benchmark with real human participants, learned human models, and scripted biased policies.

Figure 1: Layouts in Overcooked. From left to right are Asymmetric Advantages , Coordination Ring , Counter Circuit , Distant Tomato and Many Orders respectively, with orders shown below.

Experimental results

Research questions

• RQ1 Can modeling human biases as hidden reward functions in self-play improve zero-shot cooperation with humans?
• RQ2 How does HSP perform compared to baselines when cooperating with human behavior models trained on real data?
• RQ3 Does HSP generalize better to diverse human preferences, including non-optimizing strategies and skill-level variations?
• RQ4 How do real human participants rate HSP in terms of assistiveness compared to existing AI agents?
• RQ5 Can HSP discover and execute complex cooperative strategies, such as counter passing or mixed-soup avoidance, that other agents fail to learn?

Key findings

• HSP achieved the highest average episode reward in the Many Orders layout, with 384.3 and 380.7 points in the two human roles, significantly outperforming all baselines.
• In the Counter Circuit layout, HSP delivered 155.2 average reward, outperforming the next-best baseline MEP (134.5), a difference of 20.7 points equivalent to one additional soup delivery per episode.
• HSP was rated as the most assistive agent by human participants in all layouts except Coordination Ring, where it was still preferred over FCP and TrajDiv.
• In the Distant Tomato layout, HSP uniquely avoided invalid orders by correctly placing tomatoes when required, while FCP, MEP, and TrajDiv agents repeatedly placed onions in tomato-filled pots.
• HSP successfully executed the counter-passing strategy in Counter Circuit, with both top and bottom players contributing to the task, unlike other agents that failed to coordinate this complex pattern.
• Human preference analysis showed HSP had a preference score of +0.38 in Many Orders and +0.32 in Counter Circuit, indicating strong majority preference over baselines.

(b) FCP’s failure case when cooperating with a human player