[Paper Review] Dialog state tracking, a machine reading approach using Memory Network

Julien Pérez, Fei Liu|arXiv (Cornell University)|Jun 13, 2016

Speech and dialogue systems31 references14 citations

TL;DR

This paper proposes a novel dialog state tracking approach by framing it as a machine reading task using MemN2N, a memory-enhanced neural network. It achieves competitive performance on DSTC-2, with joint accuracy of 74%, and extends the benchmark to include reasoning tasks like counting and yes-no questions, demonstrating the model's capability for approximate reasoning beyond standard slot filling.

ABSTRACT

In an end-to-end dialog system, the aim of dialog state tracking is to accurately estimate a compact representation of the current dialog status from a sequence of noisy observations produced by the speech recognition and the natural language understanding modules. This paper introduces a novel method of dialog state tracking based on the general paradigm of machine reading and proposes to solve it using an End-to-End Memory Network, MemN2N, a memory-enhanced neural network architecture. We evaluate the proposed approach on the second Dialog State Tracking Challenge (DSTC-2) dataset. The corpus has been converted for the occasion in order to frame the hidden state variable inference as a question-answering task based on a sequence of utterances extracted from a dialog. We show that the proposed tracker gives encouraging results. Then, we propose to extend the DSTC-2 dataset with specific reasoning capabilities requirement like counting, list maintenance, yes-no question answering and indefinite knowledge management. Finally, we present encouraging results using our proposed MemN2N based tracking model.

Motivation & Objective

• To address limitations in existing dialog state trackers that rely on fixed utterance windows and lack correlation modeling between slots.
• To reframe dialog state tracking as a machine reading problem, enabling attention to relevant dialog history at the utterance level.
• To extend the DSTC-2 dataset with reasoning capabilities such as counting, list maintenance, yes-no questions, and indefinite knowledge management.
• To evaluate the effectiveness of MemN2N in handling both standard slot-filling and complex reasoning tasks in dialog state tracking.
• To demonstrate that memory-augmented networks can support scalable, reasoning-enhanced dialog state tracking with minimal annotation overhead.

Proposed method

• The dialog state tracking task is reformulated as a question-answering problem, where each slot becomes a question over the dialog history.
• The MemN2N architecture is used to process the dialog history and attend to relevant utterances across multiple hops to infer slot values.
• Each utterance is embedded into a vector space, and memory blocks store representations of past dialog turns for attention-based reasoning.
• The model uses a multi-hop attention mechanism to iteratively refine its focus on relevant parts of the dialog, with attention weights computed at each hop.
• The network is trained end-to-end using gradient descent with L2 regularization and Xavier initialization for all weights.
• The model is evaluated on both standard DSTC-2 slot-filling and extended reasoning tasks, including yes-no questions, counting, list maintenance, and indefinite knowledge.

Experimental results

Research questions

• RQ1 Can dialog state tracking be effectively reformulated as a machine reading task using a memory-augmented neural network?
• RQ2 Does the MemN2N architecture outperform standard discriminative models in dialog state tracking while handling complex reasoning?
• RQ3 Can the proposed framework support reasoning beyond slot filling, such as counting and list maintenance, within a dialog context?
• RQ4 How does the model perform when trained on dialog-level annotations instead of utterance-level annotations?
• RQ5 Can the extended DSTC-2 dataset with reasoning tasks serve as a benchmark for evaluating reasoning-aware dialog state trackers?

Key findings

• The MemN2N model achieved a joint accuracy of 74% on the standard DSTC-2 test set, outperforming baseline RNN and NBT models.
• For the reasoning tasks, the model achieved 96% accuracy on yes-no questions and 75% on list maintenance for the Area slot when using 40-dimensional embeddings.
• The model showed strong performance on counting tasks, achieving 80% accuracy on the PriceRange slot with 40-dimensional embeddings.
• The model’s performance on indefinite knowledge (I.K.) was 89% for Food and 90% for Area, indicating robust handling of 'dontcare' and similar concepts.
• The model demonstrated improved generalization on reasoning tasks compared to non-memory-enhanced models, especially in handling complex dependencies across slots.
• The results suggest that memory-augmented networks are well-suited for dialog state tracking with reasoning, offering a scalable alternative to rule-based or hand-crafted systems.