[Paper Review] Improving Dialogue State Tracking by Discerning the Relevant Context

Sanuj Sharma, Prafulla Kumar Choubey+1 more|arXiv (Cornell University)|Apr 4, 2019

Topic Modeling15 references4 citations

TL;DR

This paper proposes a novel dialogue state tracking (DST) framework that improves accuracy by identifying non-adjacent, relevant historical utterances and gating relevant system utterances using a context-aware mechanism. By leveraging past slot-value changes and a gating mechanism on system utterances, the model achieves 2.75% and 2.36% absolute gains in joint goal accuracy over GLAD on WoZ 2.0 and MultiWoZ 2.0, respectively, while using only one-third the parameters of GLAD.

ABSTRACT

A typical conversation comprises of multiple turns between participants where they go back-and-forth between different topics. At each user turn, dialogue state tracking (DST) aims to estimate user's goal by processing the current utterance. However, in many turns, users implicitly refer to the previous goal, necessitating the use of relevant dialogue history. Nonetheless, distinguishing relevant history is challenging and a popular method of using dialogue recency for that is inefficient. We, therefore, propose a novel framework for DST that identifies relevant historical context by referring to the past utterances where a particular slot-value changes and uses that together with weighted system utterance to identify the relevant context. Specifically, we use the current user utterance and the most recent system utterance to determine the relevance of a system utterance. Empirical analyses show that our method improves joint goal accuracy by 2.75% and 2.36% on WoZ 2.0 and MultiWoZ 2.0 restaurant domain datasets respectively over the previous state-of-the-art GLAD model.

Motivation & Objective

• To address the challenge of tracking implicit references in dialogue state tracking, where users refer to past goals not explicitly stated in current utterances.
• To overcome the limitations of relying on dialogue recency, which often fails to capture non-adjacent, relevant historical context.
• To improve DST performance by identifying and incorporating only the most relevant past utterances and system responses, reducing noise and parameter burden.
• To demonstrate that a lightweight model using selective context can outperform larger state-of-the-art models like GLAD.

Proposed method

• The model identifies relevant historical context by tracking the last turn where each slot-value was modified, enabling retrieval of non-adjacent utterances that may contain implicit references.
• A gating mechanism computes relevance between the current user utterance and the most recent system utterance to selectively incorporate informative system responses.
• The framework uses a biLSTM encoder with self-attention to represent user and system utterances, enhanced by a context-aware mechanism that retrieves and integrates key historical turns.
• The model employs a fused representation that combines the current user utterance, the most relevant historical user utterance, and the gated system utterance to improve state tracking.
• The approach is modular and can be integrated into existing DST systems, such as GLAD, by replacing recency-based encoding with relevance-aware context selection.
• The method uses a learnable scorer to determine the relevance of past utterances and slot-values, focusing only on those that may contain cues for current slot predictions.

Experimental results

Research questions

• RQ1 Can identifying non-adjacent, relevant historical utterances improve dialogue state tracking performance beyond relying on recency?
• RQ2 How effective is a gating mechanism that dynamically selects relevant system utterances based on current user input?
• RQ3 Can a lightweight model with selective context integration outperform larger, more complex state-of-the-art models like GLAD?
• RQ4 To what extent does incorporating relevant context reduce the number of required model parameters while maintaining or improving accuracy?

Key findings

• The proposed method improves joint goal accuracy by 2.75% on the WoZ 2.0 dataset and 2.36% on the MultiWoZ 2.0 restaurant domain compared to the GLAD model.
• The model achieves superior performance using only 1.2 million parameters in its base form, and 6 million with context and gating—just one-third the parameters of GLAD (17 million).
• Adding referential context improves joint goal accuracy by 2.4% and the gating mechanism adds an additional 1.0% improvement on the global biLSTM baseline.
• The performance gains are consistent across both datasets, with the model outperforming GLAD on all evaluation metrics, including turn request and inform accuracy.
• The model shows robustness in handling complex, long-form dialogues with rich slot-value spaces, as evidenced by strong results on the more challenging MultiWoZ 2.0 dataset.
• Despite high variance in joint goal accuracy due to error propagation, the model maintains stable and significant improvements, demonstrating the value of context discernment.