[Paper Review] Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems

Xiujun Li, Yu Wang+4 more|arXiv (Cornell University)|Jul 29, 2018

Speech and dialogue systems60 citations

TL;DR

The paper proposes a Dialogue Challenge to develop and benchmark end-to-end task-completion dialogue systems using a unified platform with simulators and three labeled domains (movie-ticket, restaurant, taxi).

ABSTRACT

This proposal introduces a Dialogue Challenge for building end-to-end task-completion dialogue systems, with the goal of encouraging the dialogue research community to collaborate and benchmark on standard datasets and unified experimental environment. In this special session, we will release human-annotated conversational data in three domains (movie-ticket booking, restaurant reservation, and taxi booking), as well as an experiment platform with built-in simulators in each domain, for training and evaluation purposes. The final submitted systems will be evaluated both in simulated setting and by human judges.

Motivation & Objective

• Promote collaboration and benchmarking in end-to-end task-completion dialogue research.
• Provide standard, human-annotated datasets across multiple domains for end-to-end systems.
• Offer an integrated experimentation platform with domain-specific simulators for training and evaluation.
• Facilitate both simulated and human evaluation of submitted dialogue systems.

Proposed method

• Introduce a user-simulator–dialogue system platform with an agenda-based user model and modular NLU/NLG components.
• Release annotated conversational data across three domains to support end-to-end system development.
• Provide knowledge bases built from labeled dialogues for task-specific reasoning and response generation.
• Allow participants to plug in any NLU/NLG modules and train/evaluate against predefined task goals.
• Evaluate submitted systems using simulated metrics and human judgments in a unified framework.

Experimental results

Research questions

• RQ1 How effectively can end-to-end dialogue systems achieve task completion in three distinct domains (movie-ticket, restaurant, taxi) within a unified platform?
• RQ2 What is the impact of simulation-based evaluation versus human evaluation on system assessment and progress?
• RQ3 How do different modules (NLU, state tracking, dialog-acts, NLG) contribute to end-to-end task completion performance across domains?
• RQ4 Can a standardized data and platform framework accelerate reinforcement learning and benchmarking in dialogue research?

Key findings

• Three annotated task domains are released: movie-ticket booking, restaurant reservation, and taxi ordering.
• Movie-ticket data comprises 2890 dialogues with approximately 7.5 turns per session; each domain includes 11 intents and a set of slots (e.g., 29 for movie-ticket).
• Dataset examples and knowledge-base structures are provided to support end-to-end dialogue development and evaluation.
• A platform architecture pairs a user simulator with a dialogue system, enabling multi-turn interactions and task-specific data access for evaluation.
• Evaluation plans include both simulation-based metrics (success rate, average turns, average reward) and human judgments for naturalness and coherence.