[Paper Review] Human-Agent Cooperation in Bridge Bidding

Edward Lockhart, Neil Burch+5 more|arXiv (Cornell University)|Nov 28, 2020

Reinforcement Learning in Robotics15 references4 citations

TL;DR

This paper presents a reinforcement learning approach for training AI agents to cooperate with humans in bridge bidding by combining imitation learning, search, and policy iteration. The resulting agent outperforms the strong hand-coded bot WBridge5 by 0.97 IMPs/deal when partnered with a human expert, while maintaining full compatibility with human bidding conventions.

ABSTRACT

We introduce a human-compatible reinforcement-learning approach to a cooperative game, making use of a third-party hand-coded human-compatible bot to generate initial training data and to perform initial evaluation. Our learning approach consists of imitation learning, search, and policy iteration. Our trained agents achieve a new state-of-the-art for bridge bidding in three settings: an agent playing in partnership with a copy of itself; an agent partnering a pre-existing bot; and an agent partnering a human player.

Motivation & Objective

• To develop an AI agent that can cooperate effectively with human players in the complex, communication-intensive domain of bridge bidding.
• To maintain compatibility with human bidding systems while improving performance over existing hand-coded bots.
• To enable agents to learn and refine cooperative strategies through imitation learning and iterative policy improvement.
• To evaluate the agent’s performance not only against bots but also in direct partnership with expert human players.
• To ensure the agent’s policy remains compatible with human conventions by using conservative, soft policy updates.

Proposed method

• The method begins with imitation learning from a dataset of hands played by the human-compatible WBridge5 bot to initialize a policy.
• A search algorithm is used to improve the policy by combining the prior policy with results from rollouts, simulating potential future actions.
• Policy iteration is applied over multiple rounds (up to 16), where each round generates a new dataset using the current policy and search, followed by retraining the policy.
• The agent uses the WBridge5 policy as a partner during training to ensure compatibility, and optionally applies larger search at test time for further improvement.
• The approach assumes the partner adheres to the initial policy, which discourages incompatible strategy innovations and promotes coordination.
• The final agent is evaluated in a human-in-the-loop setting, where a human expert partners the agent and a bot in randomized seat positions across 32 deals.

Experimental results

Research questions

• RQ1 Can an AI agent be trained to cooperate with humans in bridge bidding while adhering to standard human bidding systems?
• RQ2 How can imitation learning and policy iteration be combined to improve performance without sacrificing human compatibility?
• RQ3 Does an agent trained to cooperate with a strong hand-coded bot also perform better than the bot when partnered with a human expert?
• RQ4 To what extent does the agent’s improved performance stem from better strategy versus differences in the double-dummy evaluation or information disclosure?
• RQ5 Can the agent maintain compatibility with existing human conventions while still outperforming established bots?

Key findings

• The agent outperformed WBridge5 by 0.97 IMPs per deal when partnered with a human expert, with a standard error of the mean of 0.76.
• The human expert observed that the agent conformed to the Standard American Yellow Card system, differing from WBridge5 only in judgment, not in system or convention.
• The agent was slightly more aggressive in competitive auctions, aligning with modern expert human practice.
• The agent preferred simpler, more direct auctions, which are more robust to interpretation differences.
• The performance gain was not due to the double-dummy play assumption or information concealment during bidding, indicating genuine strategic improvement.
• The agent maintained compatibility with WBridge5 and even outperformed WBridge5 when partnered with the same human expert, demonstrating strong human-compatibility and cooperative capability.