[Paper Review] Enhancing Trust in LLM-Based AI Automation Agents: New Considerations and Future Challenges

Sivan Schwartz, Avi Yaeli+1 more|arXiv (Cornell University)|Aug 10, 2023

Robotic Process Automation Applications10 citations

TL;DR

The paper analyzes trust in nascent LLM-based AI automation agents, proposing a multidimensional trust framework and evaluating current products.

ABSTRACT

Trust in AI agents has been extensively studied in the literature, resulting in significant advancements in our understanding of this field. However, the rapid advancements in Large Language Models (LLMs) and the emergence of LLM-based AI agent frameworks pose new challenges and opportunities for further research. In the field of process automation, a new generation of AI-based agents has emerged, enabling the execution of complex tasks. At the same time, the process of building automation has become more accessible to business users via user-friendly no-code tools and training mechanisms. This paper explores these new challenges and opportunities, analyzes the main aspects of trust in AI agents discussed in existing literature, and identifies specific considerations and challenges relevant to this new generation of automation agents. We also evaluate how nascent products in this category address these considerations. Finally, we highlight several challenges that the research community should address in this evolving landscape.

Motivation & Objective

• Summarize how trust concepts from human-to-human interaction transfer to AI agents.
• Identify new trust considerations specific to LLM-based automation agents.
• Propose concrete dimensions and grounding mechanisms for reliability and openness.
• Assess current market products against the proposed trust considerations.

Proposed method

• Synthesize literature on trust (cognitive and emotional) and adapt to AI agents.
• Define trust dimensions: reliability, openness, tangibility, immediacy, task characteristics, and trust trajectory.
• Introduce concrete grounding/mediation mechanisms (prompt/content mediation, task/knowledge/application grounding).
• Propose safety guardrails and fail-safe strategies to preserve trust during failures.
• Provide a nascent-product assessment (ChatGPT, MS Copilot, Adept.AI, AgentGPT) against the framework.

Experimental results

Research questions

• RQ1 What new challenges and opportunities do LLM-based automation agents introduce for trust research?
• RQ2 How should trust be measured and validated in autonomously acting AI agents within business processes?
• RQ3 To what extent do nascent products address the proposed trust dimensions and guardrails?

Key findings

• Trust in AI agents comprises cognitive and emotional components and is shaped by reliability, openness, tangibility, immediacy, and task characteristics.
• The paper identifies concrete design dimensions and mediations—prompt mediation, content mediation, task grounding, knowledge grounding, application grounding, user feedback, and testing—to improve reliability.
• Transparency about goals, capabilities, data use, and algorithms positively influences openness and trust.
• Tangibility (avatars/visual cues) and immediacy behaviors (empathy, style adaptation) affect anthropomorphism and user trust.
• Task characteristics (human-in-the-loop vs autonomous actions, open-ended tasks) determine trust requirements and mitigation needs.
• A preliminary assessment of ChatGPT+plugins, MS Copilot, AgentGPT, and Adept.AI shows varying levels of alignment with the proposed trust dimensions.