[Paper Review] A dataset for resolving referring expressions in spoken dialogue via contextual query rewrites (CQR)

Michael Regan, Pushpendre Rastogi+2 more|arXiv (Cornell University)|Mar 28, 2019

Natural Language Processing Techniques20 references4 citations

TL;DR

This paper introduces Contextual Query Rewrite (CQR), a novel approach to resolving referring expressions in multi-turn spoken dialogues by reformulating ambiguous user utterances into unambiguous, single-shot natural language queries using dialogue context. The method achieves higher slot-filling accuracy (F1: 0.91 vs. 0.89) on a newly released public dataset, demonstrating that query reformulation simplifies downstream NLU and improves performance without requiring explicit state tracking or domain-specific NLU updates.

ABSTRACT

We present Contextual Query Rewrite (CQR) a dataset for multi-domain task-oriented spoken dialogue systems that is an extension of the Stanford dialog corpus (Eric et al., 2017a). While previous approaches have addressed the issue of diverse schemas by learning candidate transformations (Naik et al., 2018), we instead model the reference resolution task as a user query reformulation task, where the dialog state is serialized into a natural language query that can be executed by the downstream spoken language understanding system. In this paper, we describe our methodology for creating the query reformulation extension to the dialog corpus, and present an initial set of experiments to establish a baseline for the CQR task. We have released the corpus to the public [1] to support further research in this area.

Motivation & Objective

• To address the challenge of resolving referring expressions in multi-turn task-oriented dialogues, especially implicit references like zero anaphora.
• To eliminate the need for explicit state trackers and domain-specific NLU retraining by framing reference resolution as query reformulation.
• To create a scalable, schema-agnostic method for improving spoken language understanding in dialogue systems.
• To release a publicly available dataset of gold-standard and crowd-sourced query reformulations to support future research.

Proposed method

• The CQR task reformulates ambiguous user utterances into unambiguous, contextually enriched single-turn queries by incorporating relevant slot values from prior dialogue turns.
• The method leverages dialogue history to carry over relevant slot values (e.g., location, entity names) into a natural language query, resolving implicit references.
• The dataset is constructed by extending the Stanford Dialog Corpus with gold-standard and crowd-sourced query rewrites using Amazon Mechanical Turk.
• A sequence-to-sequence model based on OpenNMT is trained to generate reformulated queries, with hyperparameters tuned to improve accuracy.
• The approach is evaluated on two tasks: domain classification and slot-filling, comparing performance on original dialogues versus reformulated queries.
• Delexicalization is applied to preserve generalization, replacing specific entities with canonical types (e.g., poi_type) during training.

Experimental results

Research questions

• RQ1 Can contextual query rewriting effectively resolve referring expressions in multi-turn spoken dialogues without explicit state tracking?
• RQ2 Does reformulating user utterances into unambiguous, single-shot queries improve downstream NLU performance for domain classification and slot-filling?
• RQ3 How does the performance of NLU systems compare when using original dialogues versus CQR-reformulated queries?
• RQ4 To what extent can crowd-sourced query rewrites generalize, and how do they compare to gold-standard reformulations in terms of lexical diversity and model accuracy?

Key findings

• The CQR approach achieves a slot-filling F1 score of 0.91 on gold reformulations, a significant improvement over the original dialogue's F1 of 0.89.
• Domain classification accuracy remains high (0.98) for both original and reformulated inputs, indicating that the reformulation process preserves intent understanding.
• The model trained on gold-only reformulations achieves an F1 of 0.838 and BLEU score of 0.711 on the dev set, indicating strong performance on the core task.
• When trained on both gold and crowd-sourced reformulations, the model achieves an F1 of 0.897 and BLEU score of 0.397, showing that diverse human-generated reformulations improve generalization despite higher error rates.
• The task is non-trivial, as 67% of slots are carried over from dialogue history to reformulation, indicating substantial complexity in capturing contextually relevant information.
• The results suggest that query reformulation simplifies NLU by transforming multi-turn ambiguity into single-turn clarity, enabling higher accuracy with minimal architectural changes.