[Paper Review] SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Services

Yang Zhang, Vahid Noroozi+2 more|arXiv (Cornell University)|May 17, 2021

Topic Modeling16 references4 citations

TL;DR

This paper proposes SGD-QA, a fast, memory-efficient, multi-pass dialogue state tracking model for schema-guided dialogues that uses a single BERT encoder to jointly process domain descriptions (as queries) and dialogue turns (as context) via a question-answering framework. It achieves state-of-the-art performance on unseen services—improving zero-shot generalization by at least 1.6x over single-pass models—while maintaining training and inference efficiency comparable to single-pass baselines.

ABSTRACT

Dialogue state tracking is an essential part of goal-oriented dialogue systems, while most of these state tracking models often fail to handle unseen services. In this paper, we propose SGD-QA, a simple and extensible model for schema-guided dialogue state tracking based on a question answering approach. The proposed multi-pass model shares a single encoder between the domain information and dialogue utterance. The domain's description represents the query and the dialogue utterance serves as the context. The model improves performance on unseen services by at least 1.6x compared to single-pass baseline models on the SGD dataset. SGD-QA shows competitive performance compared to state-of-the-art multi-pass models while being significantly more efficient in terms of memory consumption and training performance. We provide a thorough discussion on the model with ablation study and error analysis.

Motivation & Objective

• To address the poor zero-shot generalization of existing single-pass dialogue state tracking models on unseen services.
• To develop a multi-pass model that maintains high performance while significantly reducing memory and computational costs compared to existing multi-pass approaches.
• To enable ontology-agnostic, extensible dialogue state tracking by decoupling slot prediction from domain-specific architecture.
• To improve inference efficiency through parallelizable multi-pass inference and shared encoder architecture.
• To demonstrate competitive performance on the SGD benchmark, especially on unseen services, with minimal architectural complexity.

Proposed method

• The model uses a single BERT encoder shared across all tasks, with domain descriptions as queries and dialogue turns as context.
• Each prediction task (intent, slot request, slot status, value prediction) is treated as a separate QA-style task, with input formatted as (query, context) pairs.
• The model employs multi-task learning with separate classification and span-prediction heads for different slot types, with task-specific loss masking during training.
• For value prediction, categorical slots use classification heads, while non-categorical slots use span extraction heads to locate values in the dialogue context.
• Inference is parallelized across all slots and values, enabling low-latency deployment.
• The architecture is implemented within the NVIDIA NeMo toolkit, enabling modular and extensible integration.

Experimental results

Research questions

• RQ1 Can a single, shared BERT encoder achieve strong zero-shot generalization on unseen services in schema-guided dialogue state tracking?
• RQ2 Does a multi-pass QA-based approach with a single encoder outperform multi-pass models using multiple encoders in terms of memory and training efficiency?
• RQ3 How does the proposed model compare to single-pass models in performance on unseen services?
• RQ4 Can joint multi-task learning across intent, slot request, status, and value prediction improve overall dialogue state tracking accuracy?
• RQ5 What is the impact of architectural choices such as shared encoding and parallel inference on inference latency and training convergence?

Key findings

• SGD-QA improves zero-shot performance on unseen services by at least 1.6x compared to single-pass models like SGD baseline and FastSGT on the SGD dataset.
• On the single-domain dev set, SGD-QA achieves 97.5% intent accuracy and 98.3% slot request F1, with 65.0% joint goal accuracy on unseen services.
• On the all-domain dev set, SGD-QA achieves 52.3% joint goal accuracy, outperforming SPPD (41.9%) and FastSGT (42.0%) on unseen services.
• With state tracker integration, SGD-QA achieves 60.0% joint goal accuracy on all-domain dev set, significantly outperforming FastSGT (51.6%) and SPPD (84.0% with ST).
• The model converges faster than multi-pass models and enables parallelized inference, achieving training speed comparable to single-pass models despite being multi-pass.
• Ablation studies confirm that sharing a single BERT encoder across tasks is sufficient and that combining value prediction with span extraction harms performance, favoring separate treatment.