[Paper Review] Improving accuracy of rare words for RNN-Transducer through unigram shallow fusion

Vijay Ravi, Yile Gu+6 more|arXiv (Cornell University)|Nov 30, 2020

Natural Language Processing Techniques23 references7 citations

TL;DR

This paper proposes unigram shallow fusion (USF), a simple yet effective method to improve rare word recognition in RNN-Transducer (RNN-T) ASR systems by applying a fixed score boost to rare words during decoding. USF improves rare word WER by 3.7% relative without degrading general test set performance, and its gains are additive to second-pass rescoring, working by correcting Viterbi search-induced probability estimation errors in subword-based RNN-T models.

ABSTRACT

End-to-end automatic speech recognition (ASR) systems, such as recurrent neural network transducer (RNN-T), have become popular, but rare word remains a challenge. In this paper, we propose a simple, yet effective method called unigram shallow fusion (USF) to improve rare words for RNN-T. In USF, we extract rare words from RNN-T training data based on unigram count, and apply a fixed reward when the word is encountered during decoding. We show that this simple method can improve performance on rare words by 3.7% WER relative without degradation on general test set, and the improvement from USF is additive to any additional language model based rescoring. Then, we show that the same USF does not work on conventional hybrid system. Finally, we reason that USF works by fixing errors in probability estimates of words due to Viterbi search used during decoding with subword-based RNN-T.

Motivation & Objective

• To address the persistent challenge of low accuracy in recognizing rare words within end-to-end RNN-Transducer (RNN-T) automatic speech recognition (ASR) systems.
• To develop a lightweight, low-latency method that improves rare word recognition without requiring retraining or significant computational overhead.
• To investigate whether shallow fusion with unigram word counts can correct inherent probability estimation errors in RNN-T due to Viterbi decoding.
• To evaluate the effectiveness of USF in isolation and in combination with second-pass rescoring techniques.
• To compare the performance of USF across RNN-T and conventional hybrid ASR systems to understand its model-specific behavior.

Proposed method

• USF constructs a small unigram Finite-State Transducer (FST) using unigram counts from RNN-T training data, selecting words with counts between 2 and a threshold $n_{\mathrm{thresh}}$.
• Each selected rare word is assigned a fixed score of $-1$ in the FST, while a failure arc $\rho$ with score 0 handles all other words.
• During on-the-fly decoding, the RNN-T score is combined with the USF FST score via interpolation using a learnable weight $\alpha$.
• The combined score is further rescored using a neural language model (NLM) and a coverage term $\gamma|w|$ in a second-pass rescoring stage.
• The method applies shallow fusion at inference time only, requiring no model retraining and adding minimal memory footprint (a few MB).
• The approach is contrasted with contextual biasing by using a large, non-contextual unigram list instead of n-gram-based context signals.

Experimental results

Research questions

• RQ1 Can a simple, unigram-based shallow fusion method significantly improve rare word recognition in RNN-T without degrading general performance?
• RQ2 Does unigram shallow fusion correct probability estimation errors introduced by Viterbi decoding in subword-based RNN-T models?
• RQ3 Is the improvement from USF additive to second-pass rescoring with neural language models?
• RQ4 Why does USF work effectively for RNN-T but not for conventional hybrid ASR systems?
• RQ5 Can unigram shallow fusion be applied efficiently in production environments due to low memory and latency overhead?

Key findings

• USF improves rare word WER by 3.7% relative on the test set without any degradation on the general test set, demonstrating strong robustness.
• The improvement from USF is additive to second-pass rescoring with a neural language model, indicating complementary gains.
• USF does not improve performance on hybrid ASR systems, with even small interpolation weights causing significant degradation on the general test set.
• The failure of USF on hybrid models suggests it specifically corrects a unique error source in RNN-T: Viterbi search-induced underestimation of rare word probabilities.
• The method is highly efficient, requiring only a few megabytes of memory and adding no latency, making it suitable for production deployment.
• Empirical analysis supports the hypothesis that USF mitigates search errors from Viterbi decoding by compensating for the omission of subword segmentation sums in probability estimation.