[论文解读] Evaluating Human-AI Collaboration: A Review and Methodological Framework

George Fragiadakis, Christos Diou+2 more|arXiv (Cornell University)|Jul 9, 2024

Ethics and Social Impacts of AI被引用 31

一句话总结

本文评估 HAIC 的评估方法，并提出一种新的混合方法框架，结合一个决策树在 AI-Centric、Human-Centric 和 Symbiotic 模式之间选择指标，适用于跨领域。

ABSTRACT

The use of artificial intelligence (AI) in working environments with individuals, known as Human-AI Collaboration (HAIC), has become essential in a variety of domains, boosting decision-making, efficiency, and innovation. Despite HAIC's wide potential, evaluating its effectiveness remains challenging due to the complex interaction of components involved. This paper provides a detailed analysis of existing HAIC evaluation approaches and develops a fresh paradigm for more effectively evaluating these systems. Our framework includes a structured decision tree which assists to select relevant metrics based on distinct HAIC modes (AI-Centric, Human-Centric, and Symbiotic). By including both quantitative and qualitative metrics, the framework seeks to represent HAIC's dynamic and reciprocal nature, enabling the assessment of its impact and success. This framework's practicality can be examined by its application in an array of domains, including manufacturing, healthcare, finance, and education, each of which has unique challenges and requirements. Our hope is that this study will facilitate further research on the systematic evaluation of HAIC in real-world applications.

研究动机与目标

• 调查现有的 HAIC 评价方法，识别差距和机会。
• 提出一个新的、可适应的 HAIC 评估框架，结合定量与定性指标。
• 引入一个结构化的决策树，根据 HAIC 模式（AI-Centric、Human-Centric、Symbiotic）选择指标。
• 展示领域特定的适用性并讨论 HAIC 评估中的伦理考量。

提出的方法

• 跨领域的 HAIC 评估方法的关键文献综述。
• 开发一个包含因素、子因素和指标的结构化评估框架。
• 纳入定量与定性指标以捕捉动态的 HAIC 互动。
• 讨论领域特定的洞见（医疗、金融、教育）及伦理考量。
• 概述在现实世界 HAIC 场景中应用该框架的路线图。

实验结果

研究问题

• RQ1 当前跨领域 HAIC 评估方法的局限性是什么？
• RQ2 如何通过统一的混合方法评估框架改进对 HAIC 效力的评估？
• RQ3 哪些因素和指标最能捕捉 AI-Centric、Human-Centric 和 Symbiotic 模式的动态？
• RQ4 如何将所提出的框架调整以满足领域特定需求和伦理考量？

主要发现

• HAIC 评估在定量、定性和混合方法方面呈碎片化，需要一个统一的框架。
• 以目标、互动和任务分配为核心因素的结构化框架可以标准化 HAIC 评估。
• 决策树方法使选择 AI-Centric、Human-Centric 和 Symbiotic 模式的相关指标成为可能。
• 伦理考量、透明度和信任应与性能指标一起纳入 HAIC 评估。
• 领域特定洞见表明该框架可应用于医疗、金融和教育。