[Paper Review] Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering

Li Zhou, Kevin Small|arXiv (Cornell University)|Nov 7, 2019

Topic ModelingComputer Science29 references58 citations

TL;DR

The paper models multi-domain dialogue state tracking as a question answering task (DSTQA) with a bidirectional attention network and a dynamically evolving knowledge graph to capture domain–slot relationships, achieving state-of-the-art results on MultiWOZ 2.0/2.1 and demonstrating strong domain adaptation.

ABSTRACT

Multi-domain dialogue state tracking (DST) is a critical component for conversational AI systems. The domain ontology (i.e., specification of domains, slots, and values) of a conversational AI system is generally incomplete, making the capability for DST models to generalize to new slots, values, and domains during inference imperative. In this paper, we propose to model multi-domain DST as a question answering problem, referred to as Dialogue State Tracking via Question Answering (DSTQA). Within DSTQA, each turn generates a question asking for the value of a (domain, slot) pair, thus making it naturally extensible to unseen domains, slots, and values. Additionally, we use a dynamically-evolving knowledge graph to explicitly learn relationships between (domain, slot) pairs. Our model has a 5.80% and 12.21% relative improvement over the current state-of-the-art model on MultiWOZ 2.0 and MultiWOZ 2.1 datasets, respectively. Additionally, our model consistently outperforms the state-of-the-art model in domain adaptation settings. (Code is released at https://github.com/alexa/dstqa )

Motivation & Objective

• Motivate the need for robust multi-domain DST that generalizes to unseen domains, slots, and values due to incomplete ontologies.
• Propose a DST model that treats DST as answering questions about domain-slot-value triples.
• Introduce a dynamically-evolving knowledge graph to encode relationships among domain-slot pairs and values.
• Show that DSTQA improves state tracking performance over prior methods on standard benchmarks and in domain adaptation settings.

Proposed method

• Construct a set of questions Q_{d,s} for each (domain, slot) pair, incorporating possible values when available.
• Use a bidirectional attention flow to relate the dialogue context to each question, producing context- and value-specific embeddings.
• Predict values via a bilinear scoring mechanism over the context- and question-aware embeddings, or predict spans for slots with large/infinite value sets.
• Introduce a dynamic knowledge graph with domain-slot nodes and value nodes, where graph attention propagates information across turns and domains.
• Inject graph embeddings into the prediction via a gating mechanism to control graph information flow depending on dialogue context.
• Preprocess datasets to obtain span labels for time-related slots where necessary, enabling span-based predictions when appropriate.

Experimental results

Research questions

• RQ1 Can DST be effectively modeled as a QA task to generalize to unseen domains, slots, and values?
• RQ2 Does incorporating a dynamic knowledge graph that captures relations among domain-slot/value pairs improve DST performance?
• RQ3 How does bidirectional attention between dialogue context and slot/value questions impact accuracy?
• RQ4 What is the impact of span-based versus value-based predictions for slots with large or unknown value sets?
• RQ5 How well does the proposed approach transfer to domain adaptation scenarios with limited target-domain data?

Key findings

• DSTQA w/span achieves 51.36 (Joint) and 97.22 (Slot) on MultiWOZ 2.0, outperforming several baselines including TRADE.
• DSTQA w/o span achieves 51.44 (Joint) and 97.24 (Slot) on MultiWOZ 2.0, illustrating strong performance when enumerating time-related values.
• On MultiWOZ 2.1, DSTQA w/span attains 49.67 (Joint) and 97.10 (Slot), surpassing TRADE by a notable margin; DSTQA w/o span achieves 51.17 (Joint) and 97.21 (Slot).
• In ablation studies, removing the graph component or the gating mechanism reduces joint accuracy, underscoring the benefit of the dynamic knowledge graph and gated graph integration.
• The model shows robust domain adaptation, consistently outperforming baselines when fine-tuning with limited target-domain data across multiple domains.
• Ablations indicate bidirectional attention and contextual embeddings (ELMo) contribute meaningfully to performance.