[Paper Review] LSTM-LM with Long-Term History for First-Pass Decoding in Conversational Speech Recognition

Xie Chen, Sarangarajan Parthasarathy+3 more|arXiv (Cornell University)|Oct 21, 2020

Speech Recognition and Synthesis28 references4 citations

TL;DR

This paper proposes integrating long-term context from prior utterances into LSTM language models (LSTM-LMs) during first-pass decoding in conversational speech recognition using a WFST decoder. By leveraging the LSTM-LM's ability to maintain hidden states across utterances, the method significantly improves word error rate (WER) over second-pass rescoring, achieving a 4.3% relative WER reduction with full history context compared to utterance-level modeling.

ABSTRACT

LSTM language models (LSTM-LMs) have been proven to be powerful and yielded significant performance improvements over count based n-gram LMs in modern speech recognition systems. Due to its infinite history states and computational load, most previous studies focus on applying LSTM-LMs in the second-pass for rescoring purpose. Recent work shows that it is feasible and computationally affordable to adopt the LSTM-LMs in the first-pass decoding within a dynamic (or tree based) decoder framework. In this work, the LSTM-LM is composed with a WFST decoder on-the-fly for the first-pass decoding. Furthermore, motivated by the long-term history nature of LSTM-LMs, the use of context beyond the current utterance is explored for the first-pass decoding in conversational speech recognition. The context information is captured by the hidden states of LSTM-LMs across utterance and can be used to guide the first-pass search effectively. The experimental results in our internal meeting transcription system show that significant performance improvements can be obtained by incorporating the contextual information with LSTM-LMs in the first-pass decoding, compared to applying the contextual information in the second-pass rescoring.

Motivation & Objective

• To investigate the feasibility and effectiveness of using long-term context from prior utterances in first-pass decoding for conversational speech recognition.
• To overcome the limitations of n-gram LMs, which only model short-term history and restrict search space in first-pass decoding.
• To explore whether LSTM-LMs can effectively utilize cross-utterance context during first-pass decoding, rather than only in second-pass rescoring.
• To compare the performance of context integration in first-pass decoding versus second-pass rescoring, focusing on WER and search space quality.
• To evaluate the impact of reference vs. hypothesis-based context and the role of sentence boundary tokens in context modeling.

Proposed method

• The study employs a WFST decoder to integrate LSTM-LMs in real-time first-pass decoding, enabling dynamic lattice generation with long-term context.
• LSTM-LMs are trained on utterance-level transcriptions but maintain hidden states across utterances to model long-term dependencies.
• Context from prior utterances is incorporated by feeding the hidden states of the LSTM-LM from previous utterances into the current decoding process.
• The method uses both reference transcriptions and hypothesis outputs from prior utterances as context, enabling ablation studies on context quality.
• Sentence boundary tokens (<s>) are optionally excluded from context to assess their impact on modeling coherence.
• The approach compares first-pass decoding with LSTM-LM context to second-pass rescoring using the same n-best list, ensuring fair performance evaluation.

Experimental results

Research questions

• RQ1 Can LSTM-LMs effectively utilize long-term context from prior utterances during first-pass decoding in conversational speech recognition?
• RQ2 Does incorporating cross-utterance context in first-pass decoding yield better WER than applying the same context in second-pass rescoring?
• RQ3 How does the quality of context (reference vs. hypothesis) affect the performance of first-pass decoding with LSTM-LMs?
• RQ4 What is the optimal number of prior utterances to include in the context for maximum WER improvement?
• RQ5 Does the presence of sentence boundary tokens in the context improve or degrade the performance of the LSTM-LM in first-pass decoding?

Key findings

• Using full reference history in first-pass decoding reduced WER to 15.51%, a 4.3% relative improvement over the baseline with no context.
• The WER improvement from context was consistent across different numbers of prior utterances, with marginal gains beyond two prior utterances.
• First-pass decoding with context outperformed second-pass rescoring by 0.3–0.4% absolute WER, indicating that n-best lists from n-gram LMs prune correct paths early.
• The use of reference-based context outperformed hypothesis-based context, though the gap was negligible (less than 1% relative), suggesting robustness to recognition errors.
• Excluding sentence boundary tokens (<s>) from the context led to a slight but consistent WER improvement, indicating that boundary tokens may interfere with long-term coherence modeling.
• The perplexity (PPL) of the LSTM-LM decreased from 96.9 (no history) to 81.4 (4 prior utterances), confirming better language modeling with context.