[Paper Review] Improving Transformer-based Speech Recognition Using Unsupervised Pre-training

Dongwei Jiang, Xiaoning Lei+5 more|arXiv (Cornell University)|Oct 22, 2019

Speech Recognition and SynthesisComputer Science30 references103 citations

TL;DR

The paper introduces Masked Predictive Coding (MPC) for unsupervised pre-training of Transformer-based ASR models, achieving notable CER reductions on HKUST and AISHELL-1 with varying pre-training data sizes and speaking styles.

ABSTRACT

Speech recognition technologies are gaining enormous popularity in various industrial applications. However, building a good speech recognition system usually requires large amounts of transcribed data, which is expensive to collect. To tackle this problem, an unsupervised pre-training method called Masked Predictive Coding is proposed, which can be applied for unsupervised pre-training with Transformer based model. Experiments on HKUST show that using the same training data, we can achieve CER 23.3%, exceeding the best end-to-end model by over 0.2% absolute CER. With more pre-training data, we can further reduce the CER to 21.0%, or a 11.8% relative CER reduction over baseline.

Motivation & Objective

• Motivate reducing dependence on large labeled datasets for ASR by leveraging abundant unlabeled audio data.
• Propose a Transformer-compatible unsupervised pre-training objective inspired by BERT’s MLM.
• Demonstrate that MPC improves downstream ASR performance across Mandarin corpora with varying pre-training data sizes and speaking styles.
• Assess the impact of pre-training data size and speaking style on fine-tuning performance.

Proposed method

• Apply Masked Predictive Coding to the encoder with a Transformer-based ASR setup.
• Mask 15% of input frames and reconstruct encoder outputs using an L1 loss, with 80/10/10 replacement scheme for masked frames.
• Downsample input features in pre-training to 8x reduction and 8kHz data, with downsampling applied in fine-tuning.
• After pre-training, remove MPC-specific components and add a Transformer decoder for supervised fine-tuning.
• Experiment with HKUST and AISHELL-1, plus Open Mandarin and internal datasets (Didi Callcenter, Didi Dictation) to study data size and style effects.
• Compare against APC and supervised adapters to contextualize gains from unsupervised pre-training.

Experimental results

Research questions

• RQ1 Does MPC pre-training improve Transformer-based ASR performance when labeled data is limited?
• RQ2 How does the size and speaking style of pre-training data affect downstream CER after fine-tuning?
• RQ3 How does MPC compare with APC and supervised adaption in terms of performance and data requirements?

Key findings

• MPC with pre-training on Open Mandarin (~1500h) reduces HKUST CER to 22.9–23.3% and AISHELL-1 CER to 7.8–8.1% in various setups, outperforming a strong baseline.
• With about 1500 hours of open Mandarin data, HKUST CER drops from 23.5% to 23.3%, and further improvements are seen with more pre-training data.
• Pre-training with 5000h of Didi Callcenter yields HKUST CER of 21.0% and AISHELL-1 CER of 7.7–7.8% on 8kHz downsampled data, indicating strong benefits from matching speaking style and larger data.
• Using 10000h of Didi Dictation or Didi Callcenter for pre-training can reduce HKUST AISHELL-1 CER by substantial margins (up to ~11.8% relative for HKUST and ~22.1% for AISHELL-1) versus baselines.
• Unsupervised MPC approaches can approach supervised adaptation performance, but supervised adaption remains stronger when transcribed data is available.