[论文解读] Evaluating Quality of Chatbots and Intelligent Conversational Agents

Nicole Radziwill, Morgan C. Benton|arXiv (Cornell University)|Apr 15, 2017

AI in Service Interactions参考文献 23被引用 184

一句话总结

本文综述了聊天机器人质量问题与属性，评估质量的方法，并提出一种基于 Analytic Hierarchy Process–based 方法的聊天机器人质量评估方法。

ABSTRACT

Chatbots are one class of intelligent, conversational software agents activated by natural language input (which can be in the form of text, voice, or both). They provide conversational output in response, and if commanded, can sometimes also execute tasks. Although chatbot technologies have existed since the 1960s and have influenced user interface development in games since the early 1980s, chatbots are now easier to train and implement. This is due to plentiful open source code, widely available development platforms, and implementation options via Software as a Service (SaaS). In addition to enhancing customer experiences and supporting learning, chatbots can also be used to engineer social harm - that is, to spread rumors and misinformation, or attack people for posting their thoughts and opinions online. This paper presents a literature review of quality issues and attributes as they relate to the contemporary issue of chatbot development and implementation. Finally, quality assessment approaches are reviewed, and a quality assessment method based on these attributes and the Analytic Hierarchy Process (AHP) is proposed and examined.

研究动机与目标

• 识别并归类与当代聊天机器人与对话代理相关的质量问题与属性。
• 评审现有的用于聊天机器人的质量评估方法与框架。
• 提出并初步检验一种基于已识别属性与 Analytic Hierarchy Process (AHP) 的质量评估方法。

提出的方法

• 对聊天机器人研究中的质量问题与属性进行文献综述。
• 综合前人工作中的质量属性与评估实践。
• 开发一种基于所识别属性与 Analytic Hierarchy Process (AHP) 的拟议质量评估方法。
• 在聊天机器人质量评估的情境下对拟议方法进行检验。

实验结果

研究问题

• RQ1 哪些质量问题和属性与现代聊天机器人与智能对话代理相关？
• RQ2 存在哪些用于评估聊天机器人的质量评估方法，它们的有效性如何？
• RQ3 如何使用基于 AHP 的方法基于已识别属性来评估聊天机器人质量？
• RQ4 对拟议方法的初步检验对其在质量评估中的实用性有何提示？

主要发现

• 文献综述确定了与聊天机器人开发和部署相关的具体质量问题与属性。
• 对用于聊天机器人的各种质量评估方法进行了综述和摘要。
• 提出并检验了一种基于所识别属性与 Analytic Hierarchy Process (AHP) 的质量评估方法。
• 研究将质量属性与对话代理的实际评估策略联系起来。