[Paper Review] Where is the context? -- A critique of recent dialogue datasets

Johannes E. M. Mosig, В. И. Власов+1 more|arXiv (Cornell University)|Apr 22, 2020

Topic Modeling3 references4 citations

TL;DR

This paper critiques MultiWOZ 2.1 and Taskmaster-1, two widely used dialogue datasets, revealing that they suffer from ambiguous system actions and near-total history independence, making them unsuitable for training deterministic, supervised dialogue systems. The authors demonstrate through modular and end-to-end models that system responses depend only on the most recent user input and prior system action, not deeper dialogue history, and advocate for datasets with delexicalized, distribution-based actions to better train robust conversational AI.

ABSTRACT

Recent dialogue datasets like MultiWOZ 2.1 and Taskmaster-1 constitute some of the most challenging tasks for present-day dialogue models and, therefore, are widely used for system evaluation. We identify several issues with the above-mentioned datasets, such as history independence, strong knowledge base dependence, and ambiguous system responses. Finally, we outline key desiderata for future datasets that we believe would be more suitable for the construction of conversational artificial intelligence.

Motivation & Objective

• To identify fundamental flaws in widely used dialogue datasets like MultiWOZ 2.1 and Taskmaster-1 that hinder the training of deterministic, supervised dialogue systems.
• To investigate whether dialogue models truly benefit from longer dialogue histories, challenging the assumption that history is essential for accurate next-action prediction.
• To analyze the root causes of ambiguous system actions, including knowledge base dependence and unobservable wizard personality/mood factors.
• To propose dataset design improvements—such as enforcing consistent responses for repeated dialogue states and using delexicalized annotations—to enable more reliable training of dialogue policies.
• To advocate for a shift from single-best-response prediction to distributional action prediction in future datasets, aligning with the inherent 'one-to-many' nature of human dialogue.

Proposed method

• Trained a memorization model on dialogue history (up to 10 turns) using user dialogue acts and system actions as input/output, evaluating F1 score to detect ambiguity in action selection.
• Defined user intents as 'inform' or 'bye' based on utterance content and inferred slots via belief state changes, using generic tags (e.g., 'specific', 'do-not-care') to reduce knowledge base dependence.
• Applied delexicalization to both datasets by replacing entity names with placeholders and retaining only domain, intent, and slot information to isolate model behavior from KB dependence.
• Trained and compared modular models (TED, LSTM) and an end-to-end retrieval model on both original and delexicalized versions of the datasets to assess history dependence.
• Reduced the maximum history length (max_history) from 10 to 2 turns to test whether performance drops, indicating true history dependence.
• Analyzed dialogues manually and examined prediction errors to identify cases of ambiguous or context-insensitive system responses.

Experimental results

Research questions

• RQ1 To what extent do MultiWOZ 2.1 and Taskmaster-1 exhibit ambiguous system actions, where the same dialogue history leads to multiple distinct system actions?
• RQ2 Does the performance of dialogue models on these datasets truly depend on longer dialogue histories, or is it nearly independent of context beyond the last user and system turn?
• RQ3 What are the primary causes of action ambiguity and history independence in human-annotated dialogue datasets?
• RQ4 How does delexicalization affect model performance and the perception of history dependence in these datasets?
• RQ5 What dataset design principles would make future dialogue datasets more suitable for training robust, deterministic dialogue systems?

Key findings

• The memorization model achieved an F1 score below 1.0 on both MultiWOZ 2.1 and Taskmaster-1, indicating that the same dialogue history leads to multiple system actions, violating determinism.
• Reducing max_history from 10 to 2 turns caused no significant drop in performance for any model, demonstrating that dialogue systems are nearly history independent on both datasets.
• Even with delexicalization to remove knowledge base dependence, models still failed to benefit from longer histories, confirming that the issue lies in the data, not the model architecture.
• Manual analysis revealed that many system responses are ambiguous and context-insensitive, often relying on the most recent exchange rather than the full dialogue history.
• The history independence is likely due to crowd workers being incentivized to complete dialogues quickly rather than engage meaningfully, leading to repetitive and shallow interactions.
• The authors conclude that future datasets should enforce consistent responses for repeated dialogue states and support distributional action prediction to reflect the true 'one-to-many' nature of human dialogue.