[论文解读] Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize

Chandra Khatri, Behnam Hedayatnia+20 more|arXiv (Cornell University)|Dec 27, 2018

Topic Modeling参考文献 45被引用 61

一句话总结

该论文综述了2018年Alexa Prize由大学团队取得的进展，详细介绍了如情境感知建模、知识图谱、分层对话管理器，以及CoBot工具包等方法，并取得显著的用户体验提升。

ABSTRACT

Building open domain conversational systems that allow users to have engaging conversations on topics of their choice is a challenging task. Alexa Prize was launched in 2016 to tackle the problem of achieving natural, sustained, coherent and engaging open-domain dialogs. In the second iteration of the competition in 2018, university teams advanced the state of the art by using context in dialog models, leveraging knowledge graphs for language understanding, handling complex utterances, building statistical and hierarchical dialog managers, and leveraging model-driven signals from user responses. The 2018 competition also included the provision of a suite of tools and models to the competitors including the CoBot (conversational bot) toolkit, topic and dialog act detection models, conversation evaluators, and a sensitive content detection model so that the competing teams could focus on building knowledge-rich, coherent and engaging multi-turn dialog systems. This paper outlines the advances developed by the university teams as well as the Alexa Prize team to achieve the common goal of advancing the science of Conversational AI. We address several key open-ended problems such as conversational speech recognition, open domain natural language understanding, commonsense reasoning, statistical dialog management, and dialog evaluation. These collaborative efforts have driven improved experiences by Alexa users to an average rating of 3.61, the median duration of 2 mins 18 seconds, and average turns to 14.6, increases of 14%, 92%, 54% respectively since the launch of the 2018 competition. For conversational speech recognition, we have improved our relative Word Error Rate by 55% and our relative Entity Error Rate by 34% since the launch of the Alexa Prize. Socialbots improved in quality significantly more rapidly in 2018, in part due to the release of the CoBot toolkit.

研究动机与目标

• 激励在Alexa风格系统中实现自然、持续、连贯的开放域对话的进展。
• 总结2018年大学团队取得的关键技术进展。
• 描述提供给参赛者的工具和模型及其对系统质量的影响。

提出的方法

• 描述在开放域交互中使用对话上下文、知识图谱和语言理解。
• 解释统计与分层对话管理器的开发。
• 概述来自用户回应的模型驱动信号以提升对话质量。
• 突出CoBot工具包及相关模型在话题与对话行为检测、评估器以及敏感内容检测方面。

实验结果

研究问题

• RQ1 2018年在Alexa Prize下，大学团队在开放域对话系统方面取得了哪些进展？
• RQ2 情境、知识图谱和现代对话管理如何提升了对话质量？
• RQ3 像CoBot这样的工具包对社交机器人开发与评估有何影响？
• RQ4 相较于之前的基准，会话语音识别和自然语言理解能力如何提高？

主要发现

• 平均用户评分达到3.61（Alexa用户）。
• 中位对话时长增加到2分18秒。
• 平均对话轮次上升至14.6。
• 自发布以来的相对改进：评分提升14%，时长提升92%，轮次数提升54%。
• 会话语音识别相对词错误率提升了55%。
• 实体错误率提升了34%。