[Paper Review] Can Prosody Aid the Automatic Classification of Dialog Acts in Conversational Speech?

E. Shriberg, Rebecca Bates+8 more|ArXiv.org|Jun 11, 2000

Speech and dialogue systems34 references4 citations

TL;DR

This study investigates whether prosodic features—such as pitch, duration, energy, and speaking rate—can improve automatic classification of dialog acts (DAs) in spontaneous conversational speech. Using the Switchboard corpus, the authors trained decision trees on prosodic features and combined them with word-level information (from manual or ASR outputs), showing that prosody significantly boosts DA classification accuracy, especially when combined with language models, and that multiple prosodic features provide redundancy in marking dialog acts.

ABSTRACT

Identifying whether an utterance is a statement, question, greeting, and so forth is integral to effective automatic understanding of natural dialog. Little is known, however, about how such dialog acts (DAs) can be automatically classified in truly natural conversation. This study asks whether current approaches, which use mainly word information, could be improved by adding prosodic information. The study is based on more than 1000 conversations from the Switchboard corpus. DAs were hand-annotated, and prosodic features (duration, pause, F0, energy, and speaking rate) were automatically extracted for each DA. In training, decision trees based on these features were inferred; trees were then applied to unseen test data to evaluate performance. Performance was evaluated for prosody models alone, and after combining the prosody models with word information -- either from true words or from the output of an automatic speech recognizer. For an overall classification task, as well as three subtasks, prosody made significant contributions to classification. Feature-specific analyses further revealed that although canonical features (such as F0 for questions) were important, less obvious features could compensate if canonical features were removed. Finally, in each task, integrating the prosodic model with a DA-specific statistical language model improved performance over that of the language model alone, especially for the case of recognized words. Results suggest that DAs are redundantly marked in natural conversation, and that a variety of automatically extractable prosodic features could aid dialog processing in speech applications.

Motivation & Objective

• To investigate whether prosodic features can improve automatic classification of dialog acts in spontaneous, natural conversation.
• To evaluate the contribution of prosody relative to word-level information in dialog act classification tasks.
• To determine whether prosodic features can compensate for missing canonical features (e.g., F0 for questions) in DA classification.
• To assess the impact of integrating prosodic models with DA-specific statistical language models on classification performance.
• To explore the feasibility of using prosody to improve automatic speech recognition through DA-aware language modeling.

Proposed method

• The study used over 1,000 conversations from the Switchboard corpus, with dialog acts hand-annotated using the SWBD-DAMSL tagset.
• Prosodic features—including fundamental frequency (F0), duration, pause, energy, and speaking rate—were automatically extracted for each dialog act.
• Decision trees were trained on prosodic features alone and in combination with word information (from ground-truth transcriptions or automatic speech recognizer outputs).
• Performance was evaluated on overall dialog act classification and three subtasks, with comparisons to language model baselines.
• The prosodic model was integrated with DA-specific statistical language models to assess joint performance improvements.
• Feature importance and redundancy were analyzed by removing canonical features and observing performance changes.

Experimental results

Research questions

• RQ1 Can prosodic features improve the automatic classification of dialog acts in spontaneous conversational speech beyond word-level information?
• RQ2 Which prosodic features are most predictive for specific dialog act types, and how do they compare to canonical features like F0 for questions?
• RQ3 To what extent can non-canonical prosodic features compensate when canonical features are unavailable?
• RQ4 Does integrating prosodic models with DA-specific language models improve classification accuracy, particularly when using automatically recognized words?
• RQ5 How does prosody contribute to reducing word error rates in automatic speech recognition when used as a constraint in language modeling?

Key findings

• Prosody significantly improved dialog act classification performance across all tasks, with the greatest gains observed when combined with language models.
• The integration of prosodic decision trees with DA-specific statistical language models improved performance over language models alone, especially when using recognized words from automatic speech recognition.
• Even when canonical features like F0 were removed, other prosodic features such as duration and energy could compensate, indicating redundancy in prosodic marking of dialog acts.
• Word error rate was reduced by 18% for No-Answers and 7% for Backchannels, though overall word error rate dropped only 0.9% due to the dominance of Statements in the Switchboard corpus.
• The study demonstrated that dialog acts are redundantly marked in natural conversation, with multiple prosodic features contributing to robust classification.
• The results suggest that prosody is a reliable and complementary source of information for dialog act classification and speech understanding systems.