[Paper Review] Data Techniques For Online End-to-end Speech Recognition

Yang Chen, Weiran Wang+2 more|arXiv (Cornell University)|Jan 24, 2020

Speech Recognition and Synthesis34 references4 citations

TL;DR

This paper proposes a suite of data-centric techniques—domain adaptation, data augmentation, and knowledge distillation using non-transcribed data—to improve online end-to-end ASR performance with limited in-domain data. By combining these methods on a unidirectional CTC model, the authors achieve over 50% relative WER reduction on WSJ and 25.4% relative improvement on a challenging conversational speech dataset, outperforming both strong baselines and bi-directional models trained on more data.

ABSTRACT

Practitioners often need to build ASR systems for new use cases in a short amount of time, given limited in-domain data. While recently developed end-to-end methods largely simplify the modeling pipelines, they still suffer from the data sparsity issue. In this work, we explore a few simple-to-implement techniques for building online ASR systems in an end-to-end fashion, with a small amount of transcribed data in the target domain. These techniques include data augmentation in the target domain, domain adaptation using models previously trained on a large source domain, and knowledge distillation on non-transcribed target domain data, using an adapted bi-directional model as the teacher; they are applicable in real scenarios with different types of resources. Our experiments demonstrate that each technique is independently useful in the improvement of the online ASR performance in the target domain.

Motivation & Objective

• To address the challenge of building accurate online end-to-end ASR systems with limited in-domain transcribed data.
• To improve performance on low-resource, real-world ASR scenarios where data sparsity limits model accuracy.
• To develop practical, easy-to-implement data techniques suitable for deployment in real-world systems with varying resource availability.
• To demonstrate that data manipulation—rather than model complexity—can yield significant performance gains in low-data regimes.

Proposed method

• Apply data augmentation techniques directly on in-domain transcribed data to increase training diversity and robustness.
• Use a source-domain pre-trained unidirectional CTC model for domain adaptation, fine-tuning it on limited in-domain data.
• Implement a novel teacher-student distillation framework where a bi-directional model (trained on in-domain data) generates pseudo-labels for knowledge distillation into a unidirectional student model using non-transcribed target data.
• Use a 5-layer unidirectional LSTM architecture for the student model and a 5-layer bi-directional LSTM as the teacher, both trained on the same source domain data.
• Perform frame stacking (3x) and per-speaker mean normalization to improve training efficiency and robustness.
• Train with ADAM optimizer and beam search decoding using language models and lexicons from the Kaldi recipe, with tuned priors for <blank> symbols.

Experimental results

Research questions

• RQ1 Can data augmentation on in-domain transcribed data significantly improve online end-to-end ASR performance with minimal data?
• RQ2 To what extent can domain adaptation from a large-source-domain pre-trained model improve performance on a low-resource target domain?
• RQ3 Can knowledge distillation using non-transcribed data and a bi-directional teacher model effectively transfer modeling capacity to a unidirectional student model?
• RQ4 How do these techniques combine, and what is the cumulative gain when applied together?

Key findings

• On the WSJ corpus, combining domain adaptation, data augmentation, and knowledge distillation reduced WER from 17.72% (baseline on 15h of in-domain data) to 7.58%, achieving over 50% relative improvement.
• The proposed method outperformed a bi-directional CTC model trained on the same amount of unsupervised data (13.50% WER) and a unidirectional model trained on more in-domain data (11.98% WER).
• On a challenging conversational speech dataset, the method achieved a 25.4% relative WER improvement, reducing error from 27.68% to 20.64% on the test set.
• Each individual technique—data augmentation, domain adaptation, and unsupervised knowledge distillation—produced measurable and additive performance gains.
• The teacher-student distillation framework using non-transcribed data proved particularly effective and novel for this low-data, online ASR setting.
• The results were consistent across datasets with different speaking styles, confirming the generalizability of the proposed techniques.