[Paper Review] An End-to-End Dialogue State Tracking System with Machine Reading Comprehension and Wide & Deep Classification

Yue Ma, Zengfeng Zeng+6 more|arXiv (Cornell University)|Dec 19, 2019

Topic Modeling21 references18 citations

TL;DR

This paper presents an end-to-end dialogue state tracking system for the DSTC 8 Track 4 challenge, combining a machine reading comprehension (MRC) model for span-based and numerical slots and a Wide & Deep model with hand-crafted features for boolean and text-based slots. The system achieves state-of-the-art performance with a joint goal accuracy of 0.8652 and a slot tagging F1-score of 0.9835, excelling in zero-shot generalization to unseen services.

ABSTRACT

This paper describes our approach in DSTC 8 Track 4: Schema-Guided Dialogue State Tracking. The goal of this task is to predict the intents and slots in each user turn to complete the dialogue state tracking (DST) based on the information provided by the task's schema. Different from traditional stage-wise DST, we propose an end-to-end DST system to avoid error accumulation between the dialogue turns. The DST system consists of a machine reading comprehension (MRC) model for non-categorical slots and a Wide & Deep model for categorical slots. As far as we know, this is the first time that MRC and Wide & Deep model are applied to DST problem in a fully end-to-end way. Experimental results show that our framework achieves an excellent performance on the test dataset including 50% zero-shot services with a joint goal accuracy of 0.8652 and a slot tagging F1-Score of 0.9835.

Motivation & Objective

• To address error accumulation in traditional pipeline-based dialogue state tracking by eliminating reliance on previous state estimates.
• To enable zero-shot generalization to unseen services by modeling intent and slot semantics using natural language descriptions.
• To improve performance on both categorical (boolean/text-based) and non-categorical (span-based/numerical) slots through specialized end-to-end architectures.
• To enhance model robustness and generalization via data augmentation and hybrid feature engineering in the Wide & Deep component.

Proposed method

• The MRC-DST module treats each slot as a question over the full dialogue history, using RoBERTa to extract answer spans directly from context.
• The WD-DST model combines pre-trained contextual embeddings with hand-crafted linguistic features (e.g., slot presence, position, similarity) to improve classification for boolean and text-based slots.
• All intent and slot names are replaced with their natural language descriptions as input, enabling zero-shot generalization to new schemas.
• Data augmentation is applied to the WD-DST training data to improve generalization, especially for rare or unseen slot types.
• A special rule clears dialogue state when switching between conflicting intents (e.g., 'MakePayment' and 'RequestPayment' in 'Payment_1'), preventing state inconsistency.
• The final system ensembles MRC-DST and WD-DST with data augmentation, achieving optimal joint performance.

Experimental results

Research questions

• RQ1 Can an end-to-end dialogue state tracking system eliminate error propagation from previous turns without relying on prior state estimates?
• RQ2 Can modeling intent and slot semantics using natural language descriptions enable effective zero-shot generalization to unseen services?
• RQ3 Does combining deep contextual representations with hand-crafted features improve performance on categorical slot tracking compared to pure neural baselines?
• RQ4 To what extent does data augmentation enhance zero-shot generalization in dialogue state tracking?
• RQ5 How effective is a unified MRC-based approach for non-categorical slots versus a hybrid model for categorical slots in a multi-domain, schema-guided setting?

Key findings

• The proposed end-to-end system achieved a joint goal accuracy of 0.8652 and a slot tagging F1-score of 0.9835 on the DSTC 8 test set, securing the top rank among all teams.
• The MRC-DST model demonstrated strong semantic understanding, effectively extracting span-based and numerical slot values directly from dialogue history.
• The WD-DST model outperformed a RoBERTa-only baseline, achieving 0.9751 average accuracy on the development set compared to 0.9355, due to effective integration of hand-crafted features.
• Data augmentation significantly improved zero-shot performance, particularly for unseen services, as shown by ablation studies.
• The ensemble of MRC-DST and WD-DST with data augmentation achieved a 10%+ improvement over baseline methods, confirming the effectiveness of the hybrid architecture.
• The system successfully handled conflicting slot values across intents by applying a state-clearing rule upon intent switching, preventing inconsistency.