[Paper Review] Challenges in Human-Agent Communication

Gagan Bansal, Jennifer Wortman Vaughan+6 more|arXiv (Cornell University)|Nov 28, 2024

Robotics and Automated Systems4 citations

TL;DR

This paper identifies and analyzes 12 key communication challenges in human-agent interaction arising from advanced generative AI agents, categorizing them into agent-to-user, user-to-agent, and overarching challenges across pre-, during-, and post-interaction phases. It calls for new design principles to enhance transparency, shared understanding, and control in complex, high-stakes agent interactions.

ABSTRACT

Remarkable advancements in modern generative foundation models have enabled the development of sophisticated and highly capable autonomous agents that can observe their environment, invoke tools, and communicate with other agents to solve problems. Although such agents can communicate with users through natural language, their complexity and wide-ranging failure modes present novel challenges for human-AI interaction. Building on prior research and informed by a communication grounding perspective, we contribute to the study of \emph{human-agent communication} by identifying and analyzing twelve key communication challenges that these systems pose. These include challenges in conveying information from the agent to the user, challenges in enabling the user to convey information to the agent, and overarching challenges that need to be considered across all human-agent communication. We illustrate each challenge through concrete examples and identify open directions of research. Our findings provide insights into critical gaps in human-agent communication research and serve as an urgent call for new design patterns, principles, and guidelines to support transparency and control in these systems.

Motivation & Objective

• To identify and categorize emerging communication challenges in human-agent interaction due to the rise of advanced, tool-using AI agents.
• To analyze how these challenges affect the establishment and maintenance of common ground between users and agents across interaction phases.
• To highlight the risks of miscommunication in high-stakes agent applications, such as financial spending, data leakage, or system corruption.
• To call for new design patterns, guidelines, and research directions to improve transparency, verification, and user control in agent systems.
• To bridge gaps in human-AI collaboration by formalizing communication needs across goal setting, behavior monitoring, and feedback loops.

Proposed method

• Categorizes communication challenges into three domains: agent-to-user (A1–A5), user-to-agent (U1–U3), and general challenges (X1–X4).
• Uses a communication grounding framework (Clark & Brennan, 1991) to structure the analysis of mutual understanding in human-agent interaction.
• Illustrates challenges with concrete, real-world examples from domains like academic research, event planning, and financial transactions.
• Analyzes agent behavior across three interaction phases: before execution (goal setting), during execution (monitoring), and after execution (verification).
• Identifies key failure modes such as unintended actions, poor verification, and misaligned expectations due to agent opacity and stochasticity.
• Proposes that effective communication requires dynamic, context-aware information exchange—especially about tools used, decisions made, and environmental impacts.

Experimental results

Research questions

• RQ1 How do modern agentic systems fail to convey their intentions, actions, and reasoning clearly to users during or after execution?
• RQ2 What challenges do users face in expressing goals, preferences, and constraints effectively to autonomous agents?
• RQ3 How can agents maintain consistent, verifiable, and interpretable behavior across complex, multi-step workflows?
• RQ4 What role does context—such as past interactions or environmental changes—play in shaping effective human-agent communication?
• RQ5 How can users verify that an agent has achieved a goal correctly, especially when the agent uses multiple tools and sources?

Key findings

• Agents often fail to communicate their current actions, next steps, or environmental impacts clearly, leading to user confusion and lack of trust.
• Users struggle to verify whether agents have achieved goals, especially in complex tasks like literature reviews or event planning, due to opaque reasoning and tool use.
• Agents may cause unintended side effects—such as leaking sensitive data or overwriting files—because users lack visibility into their decision-making process.
• The stochastic and context-sensitive nature of foundation models makes it difficult for agents to provide consistent, predictable behavior across similar requests.
• Users frequently need to manually correct or refine agent outputs, indicating a lack of shared understanding and insufficient feedback mechanisms.
• Maintaining common ground requires explicit, iterative communication about goals, actions, and outcomes, especially in high-stakes scenarios where failure costs are significant.