[Paper Review] Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

Bill Byrne, K. S. Krishnamoorthi+8 more|arXiv (Cornell University)|Sep 1, 2019

Topic ModelingComputer Science26 references75 citations

TL;DR

Taskmaster-1 introduces a dataset of 13,215 goal-oriented dialogs across six domains collected via two methods (WOz two-person spoken and self-dialog) with API-style annotations and baseline neural models.

ABSTRACT

A significant barrier to progress in data-driven approaches to building dialog systems is the lack of high quality, goal-oriented conversational data. To help satisfy this elementary requirement, we introduce the initial release of the Taskmaster-1 dataset which includes 13,215 task-based dialogs comprising six domains. Two procedures were used to create this collection, each with unique advantages. The first involves a two-person, spoken "Wizard of Oz" (WOz) approach in which trained agents and crowdsourced workers interact to complete the task while the second is "self-dialog" in which crowdsourced workers write the entire dialog themselves. We do not restrict the workers to detailed scripts or to a small knowledge base and hence we observe that our dataset contains more realistic and diverse conversations in comparison to existing datasets. We offer several baseline models including state of the art neural seq2seq architectures with benchmark performance as well as qualitative human evaluations. Dialogs are labeled with API calls and arguments, a simple and cost effective approach which avoids the requirement of complex annotation schema. The layer of abstraction between the dialog model and the service provider API allows for a given model to interact with multiple services that provide similar functionally. Finally, the dataset will evoke interest in written vs. spoken language, discourse patterns, error handling and other linguistic phenomena related to dialog system research, development and design.

Motivation & Objective

• Address the lack of high-quality, goal-oriented dialog data for data-driven dialog systems.
• Provide a realistically diverse corpus by combining spoken WOz and self-dialog approaches.
• Annotate with simple API calls and arguments to ease annotation and enable multi-service grounding.
• Benchmark baseline seq2seq and Transformer models and compare automatic vs human judgments.

Proposed method

• Create 13,215 dialogs across six domains (pizza, auto repair, ride service, movie tickets, coffee, restaurant reservations).
• Collect two-person spoken dialogs using a Wizard-of-Oz setup with crowd workers as users and trained agents as assistants.
• Collect self-dialogs where crowd workers write complete user-assistant conversations.
• Annotate dialogs with API calls and arguments (no complex dialog state) to enable grounding and ease reuse across services.
• Train and evaluate multiple seq2seq baselines (3-gram, 4-gram, LSTM, LSTM with attention, Convolution, Transformer) and a GPT-2 baseline on the self-dialog corpus.
• Compare automatic metrics (Perplexity, BLEU) and human judgments (rating and ranking) for model outputs.

Experimental results

Research questions

• RQ1 How can we construct a realistic, diverse goal-oriented dialog dataset without restricting to small knowledge bases?
• RQ2 Do WOz two-person conversations and self-dialogs yield different linguistic properties and model challenges?
• RQ3 Are API-call based annotations sufficient for grounding dialogs across multiple services?
• RQ4 How do standard seq2seq architectures perform on Taskmaster-1 compared to recent baselines (e.g., Transformer, GPT-2) in terms of automatic metrics and human judgments?
• RQ5 What is the relationship between automatic metrics and human judgments for this dataset?

Key findings

• Taskmaster-1 contains 13,215 dialogs across six domains, with 5,507 spoken and 7,708 written dialogs.
• Two-person WOz dialogs vs. self-dialogs show distinct language characteristics and difficulties for modeling; self-dialogs are more diverse and harder to model automatically.
• The dataset has more unique words and real-world named entities than MultiWOZ, and self-dialogs yield higher perplexity and lower BLEU than MultiWOZ, indicating greater modeling difficulty.
• Transformer-based models achieve the best automatic evaluation among the reported architectures on self-dialogs (PPL and BLEU), with strong alignment to human judgments in ranking.
• API-argument annotation (with accept/reject markers) provides a simple grounding scheme suitable for multi-service scenarios and efficient annotation.
• Copy-enabled Transformer variants improve API-argument prediction accuracy (Transformer 51.79% vs Transformer 48.73%).