[Paper Review] Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition

Ye Bai, Jiangyan Yi+3 more|arXiv (Cornell University)|Jul 13, 2019

Speech Recognition and Synthesis20 references4 citations

TL;DR

This paper proposes a knowledge distillation approach called 'Learn Spelling from Teachers' (LST) that transfers knowledge from an external RNN language model (teacher) to a sequence-to-sequence (Seq2Seq) speech recognition model (student) by using soft labels during training. The method improves character error rate (CER) by 18.42% relative to the vanilla Seq2Seq model, achieving 9.3% CER on AISHELL-1 and 8.3% with shallow fusion, without adding external components at inference time.

ABSTRACT

Integrating an external language model into a sequence-to-sequence speech recognition system is non-trivial. Previous works utilize linear interpolation or a fusion network to integrate external language models. However, these approaches introduce external components, and increase decoding computation. In this paper, we instead propose a knowledge distillation based training approach to integrating external language models into a sequence-to-sequence model. A recurrent neural network language model, which is trained on large scale external text, generates soft labels to guide the sequence-to-sequence model training. Thus, the language model plays the role of the teacher. This approach does not add any external component to the sequence-to-sequence model during testing. And this approach is flexible to be combined with shallow fusion technique together for decoding. The experiments are conducted on public Chinese datasets AISHELL-1 and CLMAD. Our approach achieves a character error rate of 9.3%, which is relatively reduced by 18.42% compared with the vanilla sequence-to-sequence model.

Motivation & Objective

• To address the challenge of integrating external language models into end-to-end sequence-to-sequence speech recognition systems without increasing inference complexity.
• To improve automatic speech recognition performance by leveraging large-scale external text data through a data-driven, regularization-based distillation approach.
• To enable seamless integration of external language models during training while removing them entirely at test time, maintaining model simplicity.
• To demonstrate that knowledge distillation from a pre-trained RNNLM can outperform traditional regularization techniques like label smoothing and unigram smoothing.
• To show that LST can be effectively combined with shallow fusion to further improve recognition accuracy.

Proposed method

• A pre-trained RNN language model is used as a 'teacher' to generate soft labels for training transcriptions.
• The Seq2Seq model (student) is trained using cross-entropy loss with these soft labels, which encode rich linguistic priors from the external text.
• The distillation process does not modify the model architecture or introduce new parameters during inference.
• The temperature scaling technique is applied to the teacher's softmax output to produce smoother, more informative soft labels.
• The method is compatible with shallow fusion, allowing joint use with external language models during decoding.
• Hyperparameters such as temperature and interpolation weights are tuned on development sets to optimize performance.

Experimental results

Research questions

• RQ1 Can knowledge distillation effectively transfer linguistic knowledge from an external language model to a sequence-to-sequence ASR model without modifying the model structure?
• RQ2 Does using soft labels from a pre-trained RNNLM as supervision improve character error rate compared to standard training and regularization methods?
• RQ3 Can the proposed LST method be combined with shallow fusion to further enhance ASR performance?
• RQ4 How does the performance of LST compare to label smoothing and unigram smoothing in terms of CER reduction?
• RQ5 Does the distillation process lead to better generalization, as indicated by training and validation loss behavior?

Key findings

• The proposed LST method reduces character error rate (CER) by 18.42% relative to the vanilla Seq2Seq model, achieving 9.3% CER on the AISHELL-1 dataset.
• LST outperforms both label smoothing (4.4% relative CER reduction) and smoothed unigram smoothing (10.0% CER) without shallow fusion.
• When combined with shallow fusion, the LST-based model achieves the best performance of 8.3% CER on the test set.
• The training loss of the LST model is higher than its validation loss, indicating a regularization effect that improves generalization.
• The loss curves show that the validation loss of the LST model is slightly lower than that of the baseline Seq2Seq model, confirming improved generalization.
• The optimal temperature for soft label generation was found to be 5.0, and the optimal interpolation weight for shallow fusion was 0.9, both determined via hyperparameter tuning on the development set.