[論文レビュー] Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems

Xiujun Li, Yu Wang+4 more|arXiv (Cornell University)|Jul 29, 2018

Speech and dialogue systems被引用数 60

ひとこと要約

本論文は、統一プラットフォームとシミュレータ、3 つのラベル付きドメイン（movie-ticket、restaurant、taxi）を用いて、エンドツーエンドのタスク完遂対話システムを開発・ベンチマークするための Dialogue Challenge を提案する。

ABSTRACT

This proposal introduces a Dialogue Challenge for building end-to-end task-completion dialogue systems, with the goal of encouraging the dialogue research community to collaborate and benchmark on standard datasets and unified experimental environment. In this special session, we will release human-annotated conversational data in three domains (movie-ticket booking, restaurant reservation, and taxi booking), as well as an experiment platform with built-in simulators in each domain, for training and evaluation purposes. The final submitted systems will be evaluated both in simulated setting and by human judges.

研究の動機と目的

• Promote collaboration and benchmarking in end-to-end task-completion dialogue research.
• Provide standard, human-annotated datasets across multiple domains for end-to-end systems.
• Offer an integrated experimentation platform with domain-specific simulators for training and evaluation.
• Facilitate both simulated and human evaluation of submitted dialogue systems.

提案手法

• Introduce a user-simulator–dialogue system platform with an agenda-based user model and modular NLU/NLG components.
• Release annotated conversational data across three domains to support end-to-end system development.
• Provide knowledge bases built from labeled dialogues for task-specific reasoning and response generation.
• Allow participants to plug in any NLU/NLG modules and train/evaluate against predefined task goals.
• Evaluate submitted systems using simulated metrics and human judgments in a unified framework.

実験結果

リサーチクエスチョン

• RQ1 How effectively can end-to-end dialogue systems achieve task completion in three distinct domains (movie-ticket, restaurant, taxi) within a unified platform?
• RQ2 What is the impact of simulation-based evaluation versus human evaluation on system assessment and progress?
• RQ3 How do different modules (NLU, state tracking, dialog-acts, NLG) contribute to end-to-end task completion performance across domains?
• RQ4 Can a standardized data and platform framework accelerate reinforcement learning and benchmarking in dialogue research?

主な発見

• Three annotated task domains are released: movie-ticket booking, restaurant reservation, and taxi ordering.
• Movie-ticket data comprises 2890 dialogues with approximately 7.5 turns per session; each domain includes 11 intents and a set of slots (e.g., 29 for movie-ticket).
• Dataset examples and knowledge-base structures are provided to support end-to-end dialogue development and evaluation.
• A platform architecture pairs a user simulator with a dialogue system, enabling multi-turn interactions and task-specific data access for evaluation.
• Evaluation plans include both simulation-based metrics (success rate, average turns, average reward) and human judgments for naturalness and coherence.