[Paper Review] Cross-Attention End-to-End ASR for Two-Party Conversations

Suyoun Kim, Siddharth Dalmia+1 more|arXiv (Cornell University)|Jul 24, 2019

Speech Recognition and Synthesis39 references4 citations

TL;DR

This paper proposes a cross-attention end-to-end ASR model that leverages speaker-specific conversational context from both speakers across multiple turns to improve long conversation recognition. By integrating speaker-specific cross-attention mechanisms that attend to both current and other speaker's utterance histories, the model achieves a 16.4% WER on Switchboard and 29.8% WER on CallHome, outperforming standard end-to-end models.

ABSTRACT

We present an end-to-end speech recognition model that learns interaction between two speakers based on the turn-changing information. Unlike conventional speech recognition models, our model exploits two speakers' history of conversational-context information that spans across multiple turns within an end-to-end framework. Specifically, we propose a speaker-specific cross-attention mechanism that can look at the output of the other speaker side as well as the one of the current speaker for better at recognizing long conversations. We evaluated the models on the Switchboard conversational speech corpus and show that our model outperforms standard end-to-end speech recognition models.

Motivation & Objective

• To improve end-to-end speech recognition in long two-party conversations by modeling cross-speaker contextual dependencies.
• To address the limitation of conventional ASR models that ignore inter-speaker context and rely only on isolated utterances.
• To develop a unified, end-to-end framework that jointly optimizes acoustic and conversational context representations.
• To evaluate the effectiveness of speaker-specific cross-attention mechanisms in capturing long-range dependencies across turns.
• To demonstrate performance gains on standard conversational speech benchmarks like Switchboard and CallHome.

Proposed method

• The model uses a CNN-BLSTM encoder and an attention-based decoder with joint CTC/attention training.
• It introduces speaker-specific conversational-context embeddings derived from BERT representations of multi-turn utterance histories.
• A cross-attention mechanism enables each speaker's decoder to attend to both their own and the other speaker's historical utterance representations.
• The model employs a matchLSTM-based attention mechanism to dynamically track interactions between speakers across turns.
• Training uses serialized dialog batches to preserve context across mini-batches, with AdaDelta optimization and gradient clipping.
• The framework is implemented using PyTorch and ESPnet, with left-right beam search decoding and length normalization.

Experimental results

Research questions

• RQ1 Can modeling inter-speaker conversational context improve end-to-end ASR performance on long two-party conversations?
• RQ2 How does speaker-specific cross-attention comparing current and other speaker’s history affect recognition accuracy?
• RQ3 Does incorporating multi-turn utterance history via attention mechanisms reduce WER compared to standard end-to-end models?
• RQ4 How do different attention mechanisms (e.g., matchLSTM vs. standard attention) compare in capturing conversational flow?
• RQ5 Can a unified end-to-end model jointly optimize acoustic and conversational context without external language models?

Key findings

• The proposed model with matchLSTM-based cross-attention achieved a 16.4% WER on the Switchboard evaluation set, outperforming the baseline end-to-end model.
• On the CallHome set, the model achieved a 29.8% WER, representing a 1.1% relative reduction compared to the baseline.
• The matchLSTM variant slightly outperformed the standard attention mechanism across all history lengths, suggesting better tracking of conversational flow.
• Attention visualization confirmed that the model focuses on semantically informative utterances and ignores short, non-lexical fillers like 'oh' or 'heck yeah'.
• Pre-training from a vanilla conversational model was essential for stable convergence due to the added attention parameters.
• The model achieved consistent WER improvements across both Switchboard and CallHome, demonstrating robustness to diverse conversational patterns.