[Paper Review] Evaluating Human-AI Collaboration: A Review and Methodological Framework

George Fragiadakis, Christos Diou+2 more|arXiv (Cornell University)|Jul 9, 2024

Ethics and Social Impacts of AI31 citations

TL;DR

The paper reviews HAIC evaluation approaches and proposes a new mixed-methods framework with a decision tree to select metrics across AI-Centric, Human-Centric, and Symbiotic modes, applicable across domains.

ABSTRACT

The use of artificial intelligence (AI) in working environments with individuals, known as Human-AI Collaboration (HAIC), has become essential in a variety of domains, boosting decision-making, efficiency, and innovation. Despite HAIC's wide potential, evaluating its effectiveness remains challenging due to the complex interaction of components involved. This paper provides a detailed analysis of existing HAIC evaluation approaches and develops a fresh paradigm for more effectively evaluating these systems. Our framework includes a structured decision tree which assists to select relevant metrics based on distinct HAIC modes (AI-Centric, Human-Centric, and Symbiotic). By including both quantitative and qualitative metrics, the framework seeks to represent HAIC's dynamic and reciprocal nature, enabling the assessment of its impact and success. This framework's practicality can be examined by its application in an array of domains, including manufacturing, healthcare, finance, and education, each of which has unique challenges and requirements. Our hope is that this study will facilitate further research on the systematic evaluation of HAIC in real-world applications.

Motivation & Objective

• Survey existing HAIC evaluation approaches and identify gaps and opportunities.
• Propose a new, adaptable HAIC evaluation framework that combines quantitative and qualitative metrics.
• Introduce a structured decision tree to select metrics based on HAIC modes (AI-Centric, Human-Centric, Symbiotic).
• Demonstrate domain-specific applicability and discuss ethical considerations in HAIC evaluations.

Proposed method

• Critical literature review of HAIC evaluation methods across domains.
• Development of a structured evaluation framework with factors, subfactors, and metrics.
• Incorporation of both quantitative and qualitative measures to capture dynamic HAIC interactions.
• Discussion of domain-specific insights (healthcare, finance, education) and ethical considerations.
• Outline of a roadmap for applying the framework in real-world HAIC settings.

Experimental results

Research questions

• RQ1 What are the limitations of current HAIC evaluation methodologies across domains?
• RQ2 How can a unified, mixed-methods evaluation framework improve assessment of HAIC effectiveness?
• RQ3 What factors and metrics best capture the dynamics of AI-centric, human-centric, and symbiotic HAIC modes?
• RQ4 How can the proposed framework be tailored to domain-specific requirements and ethical considerations?

Key findings

• HAIC evaluation is fragmented across quantitative, qualitative, and mixed-methods approaches, with a need for a unified framework.
• A structured framework with goals, interaction, and task allocation as core factors can standardize HAIC assessment.
• A decision-tree approach enables selecting relevant metrics for AI-Centric, Human-Centric, and Symbiotic HAIC modes.
• Ethical considerations, transparency, and trust should be integrated into HAIC evaluation alongside performance metrics.
• Domain-specific insights show the framework’s applicability to healthcare, finance, and education.