[Paper Review] On the Critical Role of Conventions in Adaptive Human-AI Collaboration

Andy Shih, Arjun Sawhney+3 more|arXiv (Cornell University)|Apr 7, 2021

Reinforcement Learning in Robotics29 references33 citations

TL;DR

The paper presents a learning framework that separates convention-dependent representations from rule-dependent representations to enable fast adaptation to new partners and zero-shot coordination on new tasks, validated across contextual bandits, block placing, and Hanabi, plus a human study.

ABSTRACT

Humans can quickly adapt to new partners in collaborative tasks (e.g. playing basketball), because they understand which fundamental skills of the task (e.g. how to dribble, how to shoot) carry over across new partners. Humans can also quickly adapt to similar tasks with the same partners by carrying over conventions that they have developed (e.g. raising hand signals pass the ball), without learning to coordinate from scratch. To collaborate seamlessly with humans, AI agents should adapt quickly to new partners and new tasks as well. However, current approaches have not attempted to distinguish between the complexities intrinsic to a task and the conventions used by a partner, and more generally there has been little focus on leveraging conventions for adapting to new settings. In this work, we propose a learning framework that teases apart rule-dependent representation from convention-dependent representation in a principled way. We show that, under some assumptions, our rule-dependent representation is a sufficient statistic of the distribution over best-response strategies across partners. Using this separation of representations, our agents are able to adapt quickly to new partners, and to coordinate with old partners on new tasks in a zero-shot manner. We experimentally validate our approach on three collaborative tasks varying in complexity: a contextual multi-armed bandit, a block placing task, and the card game Hanabi.

Motivation & Objective

• Motivate the need for AI that can quickly adapt to new human partners by leveraging conventions.
• Propose a representation-learning framework that disentangles convention-dependent and rule-dependent components.
• Show that a rule representation can be a sufficient statistic of best-response distributions across partners.
• Demonstrate rapid adaptation to new partners and zero-shot coordination on new tasks through modular policy design.
• Validate the approach with three collaborative tasks and a human-human study.

Proposed method

• Define a two-player MDP with shared payoff and model distinct task and partner distributions.
• Introduce a modular policy architecture with a task module gt and per-partner modules gpi that multiply to form the ego agent’s policy.
• Regularize learning with a Wasserstein-distance term D(s) to push gt toward the marginal best-response under partner distributions.
• Prove (informally) that gt approximates a sufficient statistic of the distribution over best-response strategies for a deterministic ego agent.
• Train with multiple partners on the same task to learn task and partner representations separately.
• Evaluate adaptation to new partners and zero-shot coordination on new tasks across contextual bandit, block placing, and Hanabi; include a human-human study to investigate convention transfer.

Experimental results

Research questions

• RQ1 Can the marginal (task-driven) policy representation serve as a sufficient statistic for best-response distributions across partners?
• RQ2 Does separating convention and rule representations support faster adaptation to new partners on the same task?
• RQ3 Do conventions learned with partners transfer to new tasks involving the same or similar symmetries (zero-shot coordination)?
• RQ4 How do human partners carry over conventions across tasks, and under what conditions?
• RQ5 Do modular architectures with regularization outperform baselines in adaptation scenarios?

Key findings

• A modular architecture that separates task (convention-free) and partner (convention-specific) representations improves adaptation to new partners.
• Regularizing the task module to align with marginal best-response strategies reduces distance to oracle marginals and enhances zero-shot coordination on new tasks.
• Experiments across contextual bandits, block placing, and a Hanabi variant show improved adaptation vs baselines and First-Order MAML.
• A human study suggests humans carry conventions across tasks with similar symmetries, supporting the proposed separation of representations.
• The approach enables coordination with old partners on new tasks without relearning all task-specific details.