[Paper Review] Distilling the Knowledge of BERT for Sequence-to-Sequence ASR

Hayato Futami, Hirofumi Inaguma+4 more|arXiv (Cornell University)|Aug 9, 2020

Speech Recognition and Synthesis34 references4 citations

TL;DR

This paper proposes distilling bidirectional contextual knowledge from BERT into sequence-to-sequence ASR models via knowledge distillation, using both left and right context—including cross-utterance context—during training. The method significantly improves ASR performance over unidirectional language models and existing LM integration techniques like n-best rescoring and shallow fusion, achieving a 9.19% WER on CSJ without extra inference cost.

ABSTRACT

Attention-based sequence-to-sequence (seq2seq) models have achieved promising results in automatic speech recognition (ASR). However, as these models decode in a left-to-right way, they do not have access to context on the right. We leverage both left and right context by applying BERT as an external language model to seq2seq ASR through knowledge distillation. In our proposed method, BERT generates soft labels to guide the training of seq2seq ASR. Furthermore, we leverage context beyond the current utterance as input to BERT. Experimental evaluations show that our method significantly improves the ASR performance from the seq2seq baseline on the Corpus of Spontaneous Japanese (CSJ). Knowledge distillation from BERT outperforms that from a transformer LM that only looks at left context. We also show the effectiveness of leveraging context beyond the current utterance. Our method outperforms other LM application approaches such as n-best rescoring and shallow fusion, while it does not require extra inference cost.

Motivation & Objective

• To address the left-to-right decoding limitation in seq2seq ASR models, which lack access to future (right) context during training and inference.
• To improve ASR performance by transferring contextual knowledge from BERT, a deep bidirectional language model, to a seq2seq ASR model through knowledge distillation.
• To investigate the effectiveness of using context beyond the current utterance—such as preceding and following utterances—during BERT-based soft label generation.
• To compare the proposed method with conventional LM integration techniques like n-best rescoring and shallow fusion, particularly in terms of performance and inference efficiency.

Proposed method

• BERT is used as a teacher model to generate soft labels for the ASR student model during training, leveraging both left and right context of the current utterance.
• The method extends context beyond the current utterance by concatenating previous, current, and future utterances up to a fixed token length before feeding into BERT.
• Knowledge distillation is applied by minimizing a soft label cross-entropy loss between BERT’s output probabilities and the ASR model’s predictions, with a temperature scaling and distillation weight hyperparameter.
• The ASR model is trained end-to-end using the distillation loss, while BERT is frozen and only used during training to provide soft supervision.
• The method does not require the language model during inference, eliminating additional decoding cost.
• Hyperparameters such as temperature and distillation weight are tuned on the development set.

Experimental results

Research questions

• RQ1 Can knowledge distillation from BERT improve sequence-to-sequence ASR performance compared to unidirectional language models?
• RQ2 Does incorporating right context and cross-utterance context in BERT’s input lead to better ASR performance than using only intra-utterance context?
• RQ3 How does the proposed distillation method compare to conventional LM integration techniques like n-best rescoring and shallow fusion in terms of accuracy and inference cost?
• RQ4 Does the performance gain from BERT distillation persist when training data is increased?

Key findings

• Knowledge distillation from BERT reduced the WER on the CSJ-APS (240h) dataset to 9.19%, a 10.86% relative improvement over the baseline seq2seq ASR model.
• BERT-based distillation outperformed distillation from a unidirectional transformer language model (TrfLM(uni)), which achieved 9.89% WER, demonstrating the advantage of bidirectional context.
• Using context spanning across utterances (context size 256) improved performance over intra-utterance context (utterance), reducing WER from 9.53% to 9.19%.
• The method outperformed both n-best rescoring and shallow fusion across all beam widths, even when applied to the baseline model, indicating that distillation already captures much of the LM’s benefit.
• When trained on 520h of data (CSJ-APS + CSJ-SPS), the method still achieved a 7.85% WER, a 7.5% relative improvement over the improved baseline, confirming its effectiveness on larger data.
• Ablation studies showed that pre-training BERT on longer context (256 tokens) and using the same context length during distillation yielded the best performance, with 9.19% WER.