[Paper Review] Transformer Language Models with LSTM-based Cross-utterance Information Representation

Guangzhi Sun, C. Zhang+1 more|arXiv (Cornell University)|Feb 12, 2021

Speech Recognition and Synthesis33 references4 citations

TL;DR

This paper proposes R-TLM, a Transformer language model enhanced with an LSTM module to improve cross-utterance context representation for automatic speech recognition. By integrating LSTM hidden states with segment-wise recurrence in selected Transformer blocks and using a fusion layer for residual connection, R-TLM achieves 0.9–0.8% absolute WER reductions over single-utterance TLM and 0.5–0.2% over strong cross-utterance baselines on AMI, Eval2000, and RT03 sets, with significance tests confirming improvements.

ABSTRACT

The effective incorporation of cross-utterance information has the potential to improve language models (LMs) for automatic speech recognition (ASR). To extract more powerful and robust cross-utterance representations for the Transformer LM (TLM), this paper proposes the R-TLM which uses hidden states in a long short-term memory (LSTM) LM. To encode the cross-utterance information, the R-TLM incorporates an LSTM module together with a segment-wise recurrence in some of the Transformer blocks. In addition to the LSTM module output, a shortcut connection using a fusion layer that bypasses the LSTM module is also investigated. The proposed system was evaluated on the AMI meeting corpus, the Eval2000 and the RT03 telephone conversation evaluation sets. The best R-TLM achieved 0.9%, 0.6%, and 0.8% absolute WER reductions over the single-utterance TLM baseline, and 0.5%, 0.3%, 0.2% absolute WER reductions over a strong cross-utterance TLM baseline on the AMI evaluation set, Eval2000 and RT03 respectively. Improvements on Eval2000 and RT03 were further supported by significance tests. R-TLMs were found to have better LM scores on words where recognition errors are more likely to occur. The R-TLM WER can be further reduced by interpolation with an LSTM-LM.

Motivation & Objective

• To improve cross-utterance context modeling in Transformer language models for better automatic speech recognition (ASR) performance.
• To address the limitation of standard TLMs in robustly representing long-range dependencies across utterances, especially for rare or error-prone words.
• To explore the complementary strengths of LSTM and Transformer architectures by integrating LSTM hidden states into selective Transformer blocks.
• To evaluate the effectiveness of the proposed R-TLM structure on standard ASR benchmarks, including AMI, Eval2000, and RT03.
• To investigate whether combining R-TLM with an LSTM-LM further reduces WER through model interpolation.

Proposed method

• The R-TLM architecture integrates a single-layer LSTM module into a subset of Transformer blocks, enabling recurrence across utterance segments.
• The LSTM module maintains hidden states across utterance boundaries, capturing long-range dependencies not fully modeled by self-attention alone.
• A fusion layer is introduced to combine the LSTM output with the Transformer block input via a shortcut connection, preserving gradient flow and enhancing representation learning.
• Segment-wise recurrence from Transformer-XL is used to extend context beyond a single utterance, enabling the model to attend to prior segments.
• The model is trained end-to-end on large-scale transcribed conversations, with cross-utterance context explicitly encoded via both LSTM states and extended attention spans.
• Final WER improvements are achieved through interpolation with an external LSTM-LM, combining the strengths of both architectures.

Experimental results

Research questions

• RQ1 Can integrating an LSTM module into selected Transformer blocks improve cross-utterance context representation in language models for ASR?
• RQ2 Does the combination of segment-wise recurrence (from Transformer-XL) and LSTM-based hidden state recurrence lead to more robust modeling of long-range dependencies?
• RQ3 Are the improvements in WER statistically significant when using R-TLM compared to strong TLM baselines on standard evaluation sets?
• RQ4 Does the R-TLM better model rare or error-prone words, as indicated by lower LM scores on such words?
• RQ5 Can further WER reduction be achieved by interpolating R-TLM with a standalone LSTM-LM?

Key findings

• R-TLM achieved 0.9%, 0.6%, and 0.8% absolute WER reductions over the single-utterance TLM baseline on the AMI, Eval2000, and RT03 evaluation sets, respectively.
• On the same sets, R-TLM achieved 0.5%, 0.3%, and 0.2% absolute WER reductions over a strong cross-utterance TLM baseline, with improvements on Eval2000 and RT03 confirmed by significance tests (p < 0.05).
• The R-TLM model demonstrated lower LM scores (better language modeling) on error-prone words—defined as words incorrectly recognized in transcriptions—indicating better robustness to acoustic errors.
• The TLM with extended history (TLM XL) had a lower perplexity (PPL) than R-TLM but higher WER, suggesting that R-TLM better models rare or error-prone words despite similar PPL.
• The R-TLM with fusion layer and segment-wise recurrence (d-R-TLM XL) achieved a statistically significant WER improvement on RT03 (p = 0.001), confirming robustness of the method.
• Interpolating R-TLM with an LSTM-LM further reduced WER, achieving the lowest WER of 11.8% on RT03, indicating complementary benefits from both architectures.