Skip to main content
QUICK REVIEW

[论文解读] Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence

Sumanth Balaji, Piyush Mishra|arXiv (Cornell University)|Jan 2, 2026
AI in Service InteractionsComputer Science被引用 0
一句话总结

JourneyBench 引入基于图的 SOP 来评估在客服场景中遵守政策的 LLM 代理。动态提示编排提升遵守性,使较小模型在政策合规方面甚至超越更大模型。

ABSTRACT

Traditional customer support systems, such as Interactive Voice Response (IVR), rely on rigid scripts and lack the flexibility required for handling complex, policy-driven tasks. While large language model (LLM) agents offer a promising alternative, evaluating their ability to act in accordance with business rules and real-world support workflows remains an open challenge. Existing benchmarks primarily focus on tool usage or task completion, overlooking an agent's capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. In this work, we introduce JourneyBench, a benchmark designed to assess policy-aware agents in customer support. JourneyBench leverages graph representations to generate diverse, realistic support scenarios and proposes the User Journey Coverage Score, a novel metric to measure policy adherence. We evaluate multiple state-of-the-art LLMs using two agent designs: a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models policy control. Across 703 conversations in three domains, we show that DPA significantly boosts policy adherence, even allowing smaller models like GPT-4o-mini to outperform more capable ones like GPT-4o. Our findings demonstrate the importance of structured orchestration and establish JourneyBench as a critical resource to advance AI-driven customer support beyond IVR-era limitations.

研究动机与目标

  • 推动超越刚性 IVR 系统,使用遵循业务规则的自治 LLM 驱动代理。
  • 在真实支持工作流中捕捉多步骤的政策遵循和任务依赖。
  • 提供可扩展的基准,用于在不同场景下评估代理对 SOP 的忠实度。
  • 展示结构化工作流编排对生产就绪 AI 代理的益处。

提出的方法

  • 将 SOP 表示为有向无环图(DAG),节点为任务,边为有条件的连接。
  • 通过多阶段的 LLM 驱动管线并经人类验证,生成合成的 SOP 图和用户旅程。
  • 定义 User Journey Coverage Score(UJCS)以衡量对 SOP 指定的行动序列的遵循程度。
  • 比较 Static-Prompt Agent (SPA) 与 Dynamic-Prompt Agent (DPA) 设计,其中 DPA 使用编排器来管理工作流状态。
  • 在三个领域(E-commerce、Loan Application、Telecommunications)中评估 703 次对话。
  • 通过缺失输入和工具故障等情景评估鲁棒性;以接近生产的 QA 和实际部署为基准进行评估。
Figure 1: Example SOP graph for loan application processing, showing sequential tasks and decision points.
Figure 1: Example SOP graph for loan application processing, showing sequential tasks and decision points.

实验结果

研究问题

  • RQ1是否可以使用编码任务依赖和分支逻辑的基于图的 SOP 来可靠地评估具有政策意识的代理?
  • RQ2在实时客服任务中,Dynamic Prompt 编排(DPA)是否相对于 Static Prompt(SPA)提升政策遵循?
  • RQ3在处理常见干扰(缺失输入、工具故障)时,不同规模的 LLM 在结构化工作流控制下的表现如何?
  • RQ4JourneyBench 的评估框架是否与生产行为和 QA 标准保持一致?

主要发现

  • Dynamic-Prompt Agents (DPA) 显著高于 Static-Prompt Agents (SPA) 的政策遵循。
  • GPT-4o 配合 DPA 的 UJCS 平均为 0.717,SPA 为 0.564。
  • 较小模型(GPT-4o-mini)搭配 DPA(0.649)胜过较大模型搭配 SPA(0.564)。
  • JourneyBench 报告在三个领域共 703 次对话,平均 10.91 轮对话,平均每次对话 3.34 次工具调用,使用 41 种工具。
  • 在生产环境中,基于 DPA 的编排在客户联系中心日处理 6,000+ 通话方面表现稳定。
  • 合成对话在总体 QA 真实感方面达到 84.37%(会话能力 82.33%,目标实现 87.78%),与生产 QA 分布相当。
Figure 2: Components within a single node: the task description (prompt), available tools for execution, and conditional pathways (edges) that define transitions to the next node based on outcomes.
Figure 2: Components within a single node: the task description (prompt), available tools for execution, and conditional pathways (edges) that define transitions to the next node based on outcomes.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。