[Paper Review] Schema-Guided Dialogue State Tracking Task at DSTC8

Abhinav Rastogi, Xiaoxue Zang+3 more|arXiv (Cornell University)|Feb 2, 2020

Topic Modeling29 references32 citations

TL;DR

The paper presents the Schema-Guided Dialogue State Tracking (SGDST) task at DSTC8, introducing a large-scale SGD dataset spanning 16 domains and multiple APIs, a schema-guided modeling approach, and a variety of submitted models with emphasis on zero-shot generalization to unseen APIs.

ABSTRACT

This paper gives an overview of the Schema-Guided Dialogue State Tracking task of the 8th Dialogue System Technology Challenge. The goal of this task is to develop dialogue state tracking models suitable for large-scale virtual assistants, with a focus on data-efficient joint modeling across domains and zero-shot generalization to new APIs. This task provided a new dataset consisting of over 16000 dialogues in the training set spanning 16 domains to highlight these challenges, and a baseline model capable of zero-shot generalization to new APIs. Twenty-five teams participated, developing a range of neural network models, exceeding the performance of the baseline model by a very high margin. The submissions incorporated a variety of pre-trained encoders and data augmentation techniques. This paper describes the task definition, dataset and evaluation methodology. We also summarize the approach and results of the submitted systems to highlight the overall trends in the state-of-the-art.

Motivation & Objective

• Motivate building data-efficient, cross-domain DST models for large-scale virtual assistants.
• Provide a schema-guided paradigm to handle many APIs with overlapping functionality without a single master schema.
• Introduce the SGD dataset with 16 domains and 45 synthetic services to test cross-domain and zero-shot generalization.
• Evaluate multiple submissions and identify effective techniques for zero-shot DST with unseen APIs.

Proposed method

• Propose a schema-guided approach where each service provides a schema of intents and slots with natural language descriptions.
• Train a single unified model conditioned on semantic schema elements to predict dialogue state across services without domain-specific parameters.
• Use pre-trained encoders (e.g., BERT) and data augmentation to enable zero-shot generalization to unseen APIs.
• Represent dialogue turns as frames per service and utilize schema descriptions to encode intents/slots.
• Evaluate on a dataset with unseen services and domains to test zero-shot and cross-domain capabilities.

Experimental results

Research questions

• RQ1 Can a schema-guided model achieve effective dialogue state tracking across many domains and overlapping APIs without a fixed master schema?
• RQ2 How well do models generalize to unseen services and domains (zero-shot) when conditioned on schema descriptions?
• RQ3 What dataset and evaluation design best reveal cross-domain and zero-shot DST capabilities in large-scale virtual assistants?
• RQ4 What modeling and data augmentation strategies best leverage schema semantics for DST across unseen APIs?

Key findings

• SGDST enables strong zero-shot generalization to unseen APIs across 16 domains.
• Top submissions using pre-trained encoders and data augmentation (back-translation) achieve high joint goal accuracy on unseen services.
• The winning team achieved 86.53% joint goal accuracy on the test set, with notable gains on unseen services and domains.
• Evaluations show that unseen-domain performance for categorical slots can be close to seen-domain performance, with larger gaps for other teams.
• Many approaches rely on encoding service/slot/intent descriptions via schema descriptions, and treat the problem with classifiers, QA-style spans, or multi-task BERT-based models.