[Paper Review] `Why not give this work to them?' Explaining AI-Moderated Task-Allocation Outcomes using Negotiation Trees

Zahra Zahedi, Sailik Sengupta+1 more|arXiv (Cornell University)|Feb 5, 2020

Explainable Artificial Intelligence (XAI)22 references4 citations

TL;DR

This paper proposes an AI Task Allocator (AITA) that provides explainable, fair task allocations in human-multiagent teams by generating negotiation trees to justify why alternative (counterfactual) allocations would underperform. It demonstrates that humans understand and are convinced by these explanations, especially when they underestimate others' costs, leading to longer but more informative justifications.

ABSTRACT

The problem of multi-agent task allocation arises in a variety of scenarios involving human teams. In many such settings, human teammates may act with selfish motives and try to minimize their cost metrics. In the absence of (1) complete knowledge about the reward of other agents and (2) the team's overall cost associated with a particular allocation outcome, distributed algorithms can only arrive at sub-optimal solutions within a reasonable amount of time. To address these challenges, we introduce the notion of an AI Task Allocator (AITA) that, with complete knowledge, comes up with fair allocations that strike a balance between the individual human costs and the team's performance cost. To ensure that AITA is explicable to the humans, we allow each human agent to question AITA's proposed allocation with counterfactual allocations. In response, we design AITA to provide a replay negotiation tree that acts as an explanation showing why the counterfactual allocation, with the correct costs, will eventually result in a sub-optimal allocation. This explanation also updates a human's incomplete knowledge about their teammate's and the team's actual costs. We then investigate whether humans are (1) able to understand the explanations provided and (2) convinced by it using human factor studies. Finally, we show the effect of various kinds of incompleteness on the length of explanations. We conclude that underestimation of other's costs often leads to the need for explanations and in turn, longer explanations on average.

Motivation & Objective

• To address sub-optimal task allocations in human teams due to incomplete knowledge of individual and team costs.
• To design an AI Task Allocator (AITA) with complete knowledge that produces fair, balanced allocations between individual and team costs.
• To enable AITA to explain its allocations using replayed negotiation trees that show why counterfactual allocations fail.
• To evaluate whether humans can understand and be convinced by these explanations in human factor studies.
• To analyze how different types of cost incompleteness affect explanation length and clarity.

Proposed method

• AITA uses complete knowledge of individual and team costs to compute fair task allocations that balance individual effort and team performance.
• When a human agent questions the allocation, AITA accepts a counterfactual allocation and simulates its negotiation process using a replayed negotiation tree.
• The negotiation tree explains why the counterfactual allocation would eventually lead to a sub-optimal outcome by tracing cost propagation and decision paths.
• The explanation dynamically updates the human’s understanding by revealing accurate cost information they previously lacked.
• The system is evaluated through human factor studies to assess comprehension and persuasion by the explanations.
• Explanation length is measured and correlated with types of incompleteness, especially underestimation of others’ costs.

Experimental results

Research questions

• RQ1 Can humans understand the negotiation tree explanations generated by AITA for AI-mediated task allocations?
• RQ2 Are humans convinced by the explanations when they question the AI’s allocation proposal?
• RQ3 How does the level of incompleteness in human knowledge—especially underestimation of others’ costs—affect explanation length?
• RQ4 Does the explanation mechanism successfully update human agents’ beliefs about teammates’ and the team’s actual costs?
• RQ5 What is the relationship between the type of cost incompleteness and the complexity of the explanation required?

Key findings

• Humans are able to understand the negotiation tree explanations provided by AITA, indicating the method supports effective human-AI communication.
• Humans are often convinced by the explanations, especially when the counterfactual allocation would lead to a clearly sub-optimal outcome.
• Underestimation of others’ costs is the primary driver of longer explanations, as it triggers more complex reasoning paths in the negotiation tree.
• The explanation mechanism successfully updates human agents’ beliefs about teammates’ and team-wide costs, reducing knowledge gaps.
• The length of explanations increases significantly when agents underestimate others’ costs, confirming that such incompleteness necessitates more detailed justification.
• The system demonstrates that explainable AI can improve trust and acceptance in collaborative task allocation by making AI decisions transparent and justifiable.