[Paper Review] Effective Human-AI Teams via Learned Natural Language Rules and Onboarding

Hussein Mozannar, Jimin J Lee+4 more|arXiv (Cornell University)|Nov 2, 2023

Topic Modeling4 citations

TL;DR

This paper proposes IntegrAI, a framework that learns human-AI collaboration rules as natural language descriptions of data regions in embedding space, where humans should rely on, ignore, or collaborate with AI. Through onboarding with these rules, human-AI team accuracy improved by 5.2% on an object detection task, while recommendations alone were ineffective on a question-answering task due to high human-AI performance gap and existing LLM explanations.

ABSTRACT

People are relying on AI agents to assist them with various tasks. The human must know when to rely on the agent, collaborate with the agent, or ignore its suggestions. In this work, we propose to learn rules, grounded in data regions and described in natural language, that illustrate how the human should collaborate with the AI. Our novel region discovery algorithm finds local regions in the data as neighborhoods in an embedding space where prior human behavior should be corrected. Each region is then described using a large language model in an iterative and contrastive procedure. We then teach these rules to the human via an onboarding stage. Through user studies on object detection and question-answering tasks, we show that our method can lead to more accurate human-AI teams. We also evaluate our region discovery and description algorithms separately.

Motivation & Objective

• To address the challenge of suboptimal human-AI collaboration, where humans often underperform due to miscalibrated trust in AI.
• To develop a method that learns task-specific, interpretable rules for when humans should rely on, ignore, or collaborate with AI.
• To improve human-AI team performance through an onboarding process that teaches these rules using natural language.
• To evaluate whether onboarding with learned rules enhances human-AI collaboration more than AI recommendations alone.
• To investigate the role of human-AI team performance cards in supporting decision-making during collaboration.

Proposed method

• The method uses a novel region discovery algorithm to identify local neighborhoods in a multi-modal embedding space (e.g., CLIP for images, text embeddings for NLP) where human decisions deviate from optimal AI integration.
• Each discovered region is described using a contrastive, iterative procedure: first, an LLM summarizes points inside the region; then, similar and dissimilar examples are retrieved via embedding search to refine the description.
• The final rule is a natural language description of the region, labeled with the correct AI-integration action (rely, ignore, collaborate).
• The onboarding phase teaches these rules to users through a structured interface, calibrating expectations and improving decision-making.
• AI-integration recommendations are surfaced in a dashboard post-onboarding to test their impact on performance.
• Performance is evaluated using user studies on object detection and MMLU question-answering tasks, with human-AI team accuracy as the primary metric.

Experimental results

Research questions

• RQ1 Can learned natural language rules describing data regions in embedding space improve human-AI team accuracy?
• RQ2 Does onboarding with these rules lead to better AI-integration decisions than relying on AI explanations or recommendations alone?
• RQ3 How effective is the region discovery algorithm in identifying meaningful, behaviorally relevant regions in real-world tasks?
• RQ4 To what extent do LLM-based contrastive descriptions improve the interpretability and utility of the learned rules?
• RQ5 Why does onboarding improve performance in some tasks (e.g., object detection) but not others (e.g., MMLU), despite high human and AI accuracy?

Key findings

• Onboarding with IntegrAI rules improved human-AI team accuracy by 5.2% on an object detection task involving noisy traffic light images, compared to no onboarding.
• In the MMLU question-answering task, onboarding had no significant effect on performance, likely due to the high gap between human and AI accuracy and the availability of GPT-3.5 explanations.
• Displaying AI-integration recommendations after onboarding had a negative effect on performance in the MMLU task, suggesting that recommendations may not be effective when explanations already exist.
• Participants in the onboarding condition relied on the AI 9% more than in the baseline, indicating that onboarding successfully altered user behavior and trust calibration.
• The region discovery algorithm successfully identified meaningful local regions in data space where human decisions were suboptimal, particularly in complex visual tasks.
• The contrastive LLM-based description procedure produced natural language rules that captured key decision factors, such as image clarity and uncertainty, improving interpretability.