[Paper Review] A Survey on Human-AI Collaboration with Large Foundation Models

Vanshika Vats, Marzia Binta Nizam+21 more|arXiv (Cornell University)|Mar 7, 2024

Scientific Computing and Data Management12 citations

TL;DR

This survey reviews how large pre-trained models (LPtMs) augment Human-AI teaming, covering model improvements, joint systems, safety, and applications across sectors. It synthesizes methods, challenges, and future directions for ethical, effective collaboration.

ABSTRACT

As the capabilities of artificial intelligence (AI) continue to expand rapidly, Human-AI (HAI) Collaboration, combining human intellect and AI systems, has become pivotal for advancing problem-solving and decision-making processes. The advent of Large Foundation Models (LFMs) has greatly expanded its potential, offering unprecedented capabilities by leveraging vast amounts of data to understand and predict complex patterns. At the same time, realizing this potential responsibly requires addressing persistent challenges related to safety, fairness, and control. This paper reviews the crucial integration of LFMs with HAI, highlighting both opportunities and risks. We structure our analysis around four areas: human-guided model development, collaborative design principles, ethical and governance frameworks, and applications in high-stakes domains. Our review shows that successful HAI systems are not the automatic result of stronger models but the product of careful, human-centered design. By identifying key open challenges, this survey aims to give insight into current and future research that turns the raw power of LFMs into partnerships that are reliable, trustworthy, and beneficial to society.

Motivation & Objective

• Assess how LPtMs transform Human-AI (HAI) collaboration across sectors.
• Identify methods for integrating human input into LPtM training and evaluation.
• Examine design, safety, and trust issues in HAI systems enabled by LPtMs.
• Map applications of LPtM-enabled HAI in healthcare, transportation, education, and more.

Proposed method

• Survey literature on Human-AI teaming with LPtMs through four focus areas: Model improvements, Effective HAI systems, Safe and Trustworthy AI, and Applications.
• Organize findings around traditional HITL approaches (Human in the Loop, Active Learning, RLHF, Human Evaluation).
• Categorize UI/UX and system design considerations for HAI with LPtMs.
• Reference a broad set of cited works to illustrate trends and challenges in LPtM-enabled HAI.
• Use a dual human-AI drafting process (authors plus AI assistance) to structure the review.

(a)

Experimental results

Research questions

• RQ1 How do LPtMs reshape model training, evaluation, and deployment in Human-AI teaming?
• RQ2 What are the key challenges (trust, safety, bias, privacy, governance) in LPtM-enabled HAI systems and how can they be mitigated?
• RQ3 What UI/UX and system architectures best support effective, efficient, and ethical HAI collaboration?
• RQ4 What sector-specific applications demonstrate the impact and limitations of LPtM-driven HAI teaming?

Key findings

• LPtMs substantially enhance adaptability, personalization, and conversational capability in HAI teams.
• RLHF, InstructRL, and related feedback methods are central to aligning LPtMs with human preferences and reducing biases, though challenges remain in scaling and consistency.
• Improved UI/UX, intersection of human guidance with AI, and robust evaluation frameworks are critical for trust and effectiveness in HAI systems.
• Applications span healthcare, autonomous vehicles, education, gaming, accessibility, and surveillance, illustrating broad societal implications and deployment considerations.
• Ethical, legal, and policy dimensions (privacy, accountability, labor impact, fairness) are integral to responsible adoption of LPtM-enabled HAI.

(b)