[Paper Review] Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence

Patrick Hemmer, Max Schemmer+3 more|arXiv (Cornell University)|Mar 21, 2024

Ethics and Social Impacts of AI12 citations

TL;DR

The paper develops a theoretical framework for human-AI complementarity, defining complementarity potential and its inherent/collaborative components, and validates it through two empirical studies on information and capability asymmetries.

ABSTRACT

Artificial intelligence (AI) has the potential to significantly enhance human performance across various domains. Ideally, collaboration between humans and AI should result in complementary team performance (CTP) -- a level of performance that neither of them can attain individually. So far, however, CTP has rarely been observed, suggesting an insufficient understanding of the principle and the application of complementarity. Therefore, we develop a general concept of complementarity and formalize its theoretical potential as well as the actual realized effect in decision-making situations. Moreover, we identify information and capability asymmetry as the two key sources of complementarity. Finally, we illustrate the impact of each source on complementarity potential and effect in two empirical studies. Our work provides researchers with a comprehensive theoretical foundation of human-AI complementarity in decision-making and demonstrates that leveraging these sources constitutes a viable pathway towards designing effective human-AI collaboration, i.e., the realization of CTP.

Motivation & Objective

• Define the concept of human-AI complementarity and formalize complementarity potential (CP).
• Identify and categorize sources of CP, including information and capability asymmetries.
• Differentiate inherent and collaborative CP and relate them to realized performance gains (CE).
• Empirically validate the framework in two studies: information asymmetry in real estate appraisal and capability asymmetry in image classification.

Proposed method

• Formalize decision-making with human (H) and AI (AI) decisions and a collaboration mechanism I(·).
• Define complementarity potential (CP) as the difference between the best single actor’s loss and 0-loss ground truth, decomposed into CPinh and CPcoll.
• Define realized complementarity effect (CE) as the realized loss reduction from collaboration, with CEinh and CEcoll components.
• Derive equations for CP, CPinh, CPcoll, CE, and CE components (Equations 3–10).
• Develop two experimental studies to isolate information asymmetry (real estate valuation) and capability asymmetry (image classification).
• Discuss implications for designing and evaluating human-AI collaboration in decision-making.

Experimental results

Research questions

• RQ1 RQ1: How can we model human-AI decision-making to enable a nuanced understanding of synergies in a human-AI team?
• RQ2 RQ2: What factors contribute to complementary team performance (CTP) in human-AI decision-making?

Key findings

• Complementarity potential (CP) is decomposed into inherent CP and collaborative CP, and together they explain how H and AI can outperform each on their own.
• Evidence from two studies shows that asymmetric information and asymmetric capabilities can increase CP and enable CTP in decision tasks.
• In the real estate study, humans leveraged unique contextual information alongside AI predictions to contribute to CP.
• In the image-classification study, heterogeneous capabilities between humans and AI contributed to CP and enabled CTP.
• The framework provides a way to measure realized collaboration (CE) and its inherent/collaborative components across tasks.