[Paper Review] Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction

Weiran Wang, Qingming Tang+1 more|arXiv (Cornell University)|Jan 28, 2020

Speech Recognition and SynthesisComputer Science22 citations

TL;DR

This paper proposes a masked reconstruction pre-training method for bidirectional speech encoders using continuous spectrogram inputs, where random time and frequency segments are masked and the model reconstructs them. The approach achieves consistent improvements in speech recognition accuracy across LibriSpeech and Wall Street Journal datasets, especially when pre-training on large-scale unlabeled data and using domain adaptation during fine-tuning.

ABSTRACT

We propose an approach for pre-training speech representations via a masked reconstruction loss. Our pre-trained encoder networks are bidirectional and can therefore be used directly in typical bidirectional speech recognition models. The pre-trained networks can then be fine-tuned on a smaller amount of supervised data for speech recognition. Experiments with this approach on the LibriSpeech and Wall Street Journal corpora show promising results. We find that the main factors that lead to speech recognition improvements are: masking segments of sufficient width in both time and frequency, pre-training on a much larger amount of unlabeled data than the labeled data, and domain adaptation when the unlabeled and labeled data come from different domains. The gain from pre-training is additive to that of supervised data augmentation.

Motivation & Objective

• To improve speech recognition performance through unsupervised pre-training of bidirectional speech encoders.
• To adapt BERT-style masked reconstruction to the continuous, high-granularity nature of speech signals.
• To investigate the impact of masking strategy, pre-training data scale, and domain mismatch on recognition performance.
• To evaluate the method on both phone-based and character-based ASR systems with and without data augmentation.
• To demonstrate the effectiveness of domain adaptation via a linear input network (LIN) when pre-training and fine-tuning data differ in domain.

Proposed method

• Pre-train bidirectional RNNs using a masked reconstruction loss on large-scale unlabeled speech data.
• Apply random masking in both time and frequency domains: contiguous segments in time and random channels across time steps.
• Use spectrogram features as input, enabling joint time-frequency masking for better modeling of speech continuity.
• Fine-tune the pre-trained models on smaller labeled datasets using standard ASR training procedures.
• Apply domain adaptation via a learnable linear input network (LIN) that is updated during early fine-tuning epochs.
• Optimize hyperparameters such as mask width and number of masks via development set tuning.

Fig. 1 : Illustration of our masked reconstruction approach.

Experimental results

Research questions

• RQ1 Can BERT-style masked reconstruction be effectively adapted to continuous speech signals for unsupervised pre-training?
• RQ2 How do masking strategy (width and number of time/frequency segments) affect downstream ASR performance?
• RQ3 What is the impact of pre-training on large-scale unlabeled data versus limited labeled data?
• RQ4 How does domain mismatch between pre-training and fine-tuning data affect performance, and can it be mitigated?
• RQ5 Are the gains from pre-training additive to those from supervised data augmentation like SpecAugment?

Key findings

• Pre-training on 960 hours of unlabeled LibriSpeech data significantly improves ASR performance on both si284 and WSJ test sets.
• On the si284 set, pre-training reduces phone-based WER from 18.52% (baseline) to 15.56% with SpecAugment, and to 14.92% with both pre-training and SpecAugment.
• For character-based systems, pre-training reduces CER from 15.23% (baseline) to 11.70% with SpecAugment and LIN adaptation.
• The combination of pre-training and SpecAugment yields additive gains, with test set WER dropping to 6.33% when both are applied.
• Domain adaptation via LIN consistently improves performance when pre-training and fine-tuning data come from different domains, especially on WSJ.
• Learning curves show that pre-trained models converge faster and generalize better, with lower CER and CTC loss, indicating reduced overfitting.

Fig. 2 : Dev set learning curves (%CER and CTC loss) of different systems pre-trained on LibriSpeech . The first 5 epochs of fine-tuning update only the LIN and softmax layers.