[Paper Review] Conversational AI: The Science Behind the Alexa Prize

Ashwin Ram, Rohit Prasad+16 more|arXiv (Cornell University)|Jan 11, 2018

Topic ModelingComputer Science18 references201 citations

TL;DR

The paper describes the Alexa Prize, a 2.5-million-dollar university competition where 16 teams built socialbots to converse with humans for 20 minutes, enabling live data collection and real-time evaluation to advance conversational AI.

ABSTRACT

Conversational agents are exploding in popularity. However, much work remains in the area of social conversation as well as free-form conversation over a broad range of domains and topics. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million-dollar university competition where sixteen selected university teams were challenged to build conversational agents, known as socialbots, to converse coherently and engagingly with humans on popular topics such as Sports, Politics, Entertainment, Fashion and Technology for 20 minutes. The Alexa Prize offers the academic community a unique opportunity to perform research with a live system used by millions of users. The competition provided university teams with real user conversational data at scale, along with the user-provided ratings and feedback augmented with annotations by the Alexa team. This enabled teams to effectively iterate and make improvements throughout the competition while being evaluated in real-time through live user interactions. To build their socialbots, university teams combined state-of-the-art techniques with novel strategies in the areas of Natural Language Understanding, Context Modeling, Dialog Management, Response Generation, and Knowledge Acquisition. To support the efforts of participating teams, the Alexa Prize team made significant scientific and engineering investments to build and improve Conversational Speech Recognition, Topic Tracking, Dialog Evaluation, Voice User Experience, and tools for traffic management and scalability. This paper outlines the advances created by the university teams as well as the Alexa Prize team to achieve the common goal of solving the problem of Conversational AI.

Motivation & Objective

• Motivate the study of conversational AI through a large-scale, real-user evaluation setting.
• Describe the Alexa Prize architecture, data collection, and evaluation framework used by teams and organizers.
• Summarize the scientific and engineering investments made to advance speech recognition, topic tracking, dialog evaluation, and user experience in conversational systems.

Proposed method

• Describe the competition design and criteria for socialbots.
• Explain how teams combined state-of-the-art techniques with novel strategies across NLU, context modeling, dialog management, response generation, and knowledge acquisition.
• Highlight the data collection pipeline: live user conversations, ratings, feedback, and Alexa team annotations.
• Discuss the infrastructural investments for conversational speech recognition, topic tracking, and traffic management at scale.

Experimental results

Research questions

• RQ1 What approaches enable coherent and engaging long-context socialbot conversations with humans across diverse topics?
• RQ2 How does live user feedback at scale accelerate iterative improvements in socialbot capabilities?
• RQ3 What are the key research areas (NLU, context, dialog, knowledge) that drive improvements in conversational AI in a competitive, real-world setting?
• RQ4 What infrastructural and methodological challenges arise when running a large-scale, real-user AI competition?

Key findings

• The Alexa Prize enabled real-user interactions and annotated feedback that teams used to iterate socialbot design.
• The competition combined state-of-the-art techniques with novel strategies across core AI areas to advance conversational AI.
• Significant scientific and engineering investments were made to improve speech recognition, topic tracking, dialog evaluation, and user experience at scale.
• Teams and organizers leveraged live data to evaluate and refine conversational agents in real time.