[Paper Review] SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents

Shuzheng Si, Wentao Ma+8 more|arXiv (Cornell University)|May 22, 2023

Topic Modeling4 citations

TL;DR

SpokenWOZ introduces a large-scale, human-to-human speech-text dataset with 203K turns, 5.7K dialogues, and 249 hours of audio across 8 domains, designed to bridge the gap between written-text TOD benchmarks and real spoken dialogue. It identifies novel challenges like cross-turn slot detection and reasoning slot detection, and shows current models still underperform, with the best dialogue state tracker achieving only 25.65% joint goal accuracy and end-to-end models succeeding in just 52.1% of requests.

ABSTRACT

Task-oriented dialogue (TOD) models have made significant progress in recent years. However, previous studies primarily focus on datasets written by annotators, which has resulted in a gap between academic research and real-world spoken conversation scenarios. While several small-scale spoken TOD datasets are proposed to address robustness issues such as ASR errors, they ignore the unique challenges in spoken conversation. To tackle the limitations, we introduce SpokenWOZ, a large-scale speech-text dataset for spoken TOD, containing 8 domains, 203k turns, 5.7k dialogues and 249 hours of audios from human-to-human spoken conversations. SpokenWOZ further incorporates common spoken characteristics such as word-by-word processing and reasoning in spoken language. Based on these characteristics, we present cross-turn slot and reasoning slot detection as new challenges. We conduct experiments on various baselines, including text-modal models, newly proposed dual-modal models, and LLMs, e.g., ChatGPT. The results show that the current models still have substantial room for improvement in spoken conversation, where the most advanced dialogue state tracker only achieves 25.65% in joint goal accuracy and the SOTA end-to-end model only correctly completes the user request in 52.1% of dialogues. The dataset, code, and leaderboard are available: https://spokenwoz.github.io/.

Motivation & Objective

• To close the gap between academic research on written-text TOD datasets and real-world spoken dialogue systems.
• To address the limitations of existing spoken TOD datasets, including small scale, lack of authentic human-to-human audio, and absence of spoken-specific challenges.
• To introduce new evaluation challenges—cross-turn slot detection and reasoning slot detection—reflecting real spoken conversation dynamics like incomplete utterances and indirect reasoning.
• To provide a large-scale, high-quality, speech-text dataset with detailed annotations for dialogue state and dialogue acts to support robust model training and evaluation.

Proposed method

• Extended the Wizard-of-Oz data collection pipeline to collect real-time voice conversations between human agents and users.
• Collected 249 hours of audio from 5,700 human-to-human dialogues across 8 domains, ensuring natural spoken language patterns.
• Annotated each utterance with dialogue state and dialogue act, achieving over 97% turn-level annotation accuracy.
• Introduced two new tasks: cross-turn slot detection to model state tracking across incomplete or fragmented utterances, and reasoning slot detection to capture indirect or implicit user intentions.
• Built a comprehensive benchmark with evaluation protocols for text-only, dual-modal (speech+text), and LLM-based models.
• Released the dataset, code, and leaderboard via a public GitHub repository under CC BY-NC 4.0 license.

Experimental results

Research questions

• RQ1 How does model performance on spoken task-oriented dialogue vary when evaluated on a large-scale, authentic human-to-human speech-text dataset compared to written-text benchmarks?
• RQ2 To what extent do current state-of-the-art models handle cross-turn slot detection, where slot information is distributed across multiple utterances in spoken dialogue?
• RQ3 Can models effectively detect reasoning slots that require indirect inference rather than direct slot extraction in spoken language?
• RQ4 What are the performance limits of current dialogue state trackers and end-to-end models on a realistic spoken dialogue benchmark with natural speech artifacts and incomplete utterances?
• RQ5 How do dual-modal models (speech + text) compare to text-only models in capturing spoken nuances and improving robustness in real-world scenarios?

Key findings

• The most advanced dialogue state tracker achieved only 25.65% joint goal accuracy on the SpokenWOZ benchmark, indicating significant room for improvement in state tracking for spoken dialogue.
• The best end-to-end model correctly completed user requests in only 52.1% of dialogues, highlighting the gap between current models and human-level performance in spoken task-oriented settings.
• Even with access to both speech and text modalities, existing dual-modal models still struggle with handling incomplete or fragmented utterances common in spoken conversation.
• The introduction of cross-turn slot detection as a task revealed that current models fail to effectively track slots that span multiple turns, especially when utterances are interrupted or partially spoken.
• Reasoning slot detection proved particularly challenging, as models often misinterpret indirect or implicit user requests, underscoring the need for better reasoning and contextual understanding in spoken dialogue systems.
• The dataset's high-quality annotations and realistic speech patterns make it a strong foundation for future research, with the potential to drive progress in robust, human-like spoken dialogue agents.