[Paper Review] On the Utility of Learning about Humans for Human-AI Coordination

Micah Carroll, Rohin Shah+5 more|arXiv (Cornell University)|Oct 13, 2019

Reinforcement Learning in Robotics91 citations

TL;DR

Self-play agents coordinate well with other AIs but falter with humans; training with human data or models improves human-AI collaboration, demonstrated in an Overcooked-based environment and confirmed by human user studies.

ABSTRACT

While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves. Agents that assume their partner to be optimal or similar to them can converge to coordination protocols that fail to understand and be understood by humans. To demonstrate this, we introduce a simple environment that requires challenging coordination, based on the popular game Overcooked, and learn a simple model that mimics human play. We evaluate the performance of agents trained via self-play and population-based training. These agents perform very well when paired with themselves, but when paired with our human model, they are significantly worse than agents designed to play with the human model. An experiment with a planning algorithm yields the same conclusion, though only when the human-aware planner is given the exact human model that it is playing with. A user study with real humans shows this pattern as well, though less strongly. Qualitatively, we find that the gains come from having the agent adapt to the human's gameplay. Given this result, we suggest several approaches for designing agents that learn about humans in order to better coordinate with them. Code is available at https://github.com/HumanCompatibleAI/overcooked_ai.

Motivation & Objective

• Motivate the need for human-aware coordination in AI systems, addressing failures of self-play when partnering with humans.
• Introduce an Overcooked-inspired environment to test human-AI collaboration under challenging coordination.
• Evaluate self-play, population-based training, planning, and human-model-based training for collaboration with humans.
• Demonstrate that incorporating human models improves performance with both simulated and real human partners.

Proposed method

• Develop an Overcooked-like multi-agent environment with onions, dishes, and soups to create coordination challenges.
• Collect human-human trajectories and train a simple behavior cloning human model (BC) for each layout.
• Compare agents trained with self-play (SP), population-based training (PBT), and coupled planning (CP) to agents trained with the human model (PPO BC, planning with BC).
• Evaluate agents against a held-out proxy human model H_Proxy and in a user study with real humans.
• Use a gold-standard baseline by training agents with direct access to the proxy human model to bound achievable performance.

Experimental results

Research questions

• RQ1 Does self-play-trained coordination degrade when paired with a non-optimal human model or real humans?
• RQ2 Can incorporating a human model into training (via behavior cloning or planning) improve human-AI coordination compared to self-play alone?
• RQ3 How do planning-based and reinforcement-learning-based approaches compare when coordinating with human partners?
• RQ4 Do findings with simulated proxy humans generalize to real human users?

Key findings

• Self-play and PBT agents perform well with themselves but substantially worse when paired with a proxy human model or real humans.
• Agents trained with a behavior-cloned human model (PPO BC) outperform self-play agents when paired with humans and approach gold-standard performance when possible.
• Planning with access to the true human model helps, but planning with a BC model can still lead to loops or poor performance if the human model is inaccurate.
• Imitation-based human models yield better collaboration than assuming humans are optimal or like the agent; planning/RL with human models typically beats vanilla imitation.
• In user studies, PPO BC generally outperforms SP and PBT in several layouts, though effects vary by task layout and model quality.