[Paper Review] Injecting Text in Self-Supervised Speech Pretraining

Zhehuai Chen, Yu Zhang+4 more|arXiv (Cornell University)|Aug 27, 2021

Speech Recognition and Synthesis55 references4 citations

TL;DR

This paper proposes tts4pretrain, a method that enhances self-supervised speech representation learning by jointly pretraining on untranscribed speech and synthesized speech from unspoken text. By introducing auxiliary sequence losses (e.g., CTC, phoneme, word-piece) on TTS-generated speech, the model injects lexical and phonetic information, reducing WER by 10% relative on LibriSpeech and matching 5,000 hours of transcribed data with only 100 hours.

ABSTRACT

Self-supervised pretraining for Automated Speech Recognition (ASR) has shown varied degrees of success. In this paper, we propose to jointly learn representations during pretraining from two different modalities: speech and text. The proposed method, tts4pretrain complements the power of contrastive learning in self-supervision with linguistic/lexical representations derived from synthesized speech, effectively learning from untranscribed speech and unspoken text. Lexical learning in the speech encoder is enforced through an additional sequence loss term that is coupled with contrastive loss during pretraining. We demonstrate that this novel pretraining method yields Word Error Rate (WER) reductions of 10% relative on the well-benchmarked, Librispeech task over a state-of-the-art baseline pretrained with wav2vec2.0 only. The proposed method also serves as an effective strategy to compensate for the lack of transcribed speech, effectively matching the performance of 5000 hours of transcribed speech with just 100 hours of transcribed speech on the AMI meeting transcription task. Finally, we demonstrate WER reductions of up to 15% on an in-house Voice Search task over traditional pretraining. Incorporating text into encoder pretraining is complimentary to rescoring with a larger or in-domain language model, resulting in additional 6% relative reduction in WER.

Motivation & Objective

• To address the limitation of self-supervised speech pretraining in capturing linguistic and phonetic structure without transcribed data.
• To explore whether unspoken text, via text-to-speech (TTS) synthesis, can enhance speech representation learning.
• To reduce dependency on large-scale transcribed speech data for downstream ASR fine-tuning.
• To evaluate the effectiveness of combining contrastive learning with auxiliary ASR-style losses on synthesized speech.
• To investigate whether language model fusion remains complementary when textual information is injected during pretraining.

Proposed method

• The method uses a TTS system to generate speech from unspoken text, creating a source of supervised-like signal for pretraining.
• An auxiliary decoder head is trained with sequence-level objectives (e.g., CTC, RNN-T) on the synthesized speech to enforce alignment with text tokens (word-pieces, phonemes).
• The speech encoder is jointly optimized via contrastive loss on real and synthesized speech, and sequence loss on the auxiliary decoder.
• Data augmentation (20% time/frequency masking + frequency warping) is applied only to the synthesized speech for the auxiliary loss, not the contrastive loss.
• The approach is compatible with various ASR architectures and training objectives, including CTC, RNN-T, and HAT.
• Domain adaptation is enabled by using in-domain text (e.g., search queries) to generate TTS data for pretraining.

Fig. 1 : Proposed joint speech and text self-supervised pretraining architecture.

Experimental results

Research questions

• RQ1 Can injecting textual information via TTS-generated speech improve self-supervised speech representation learning?
• RQ2 How does the amount of unspoken text affect performance gains in low-resource ASR settings?
• RQ3 Does combining TTS-based pretraining with language model fusion yield further WER improvements?
• RQ4 How does the choice of auxiliary decoder objective (e.g., CTC vs. RNN-T) impact learning efficiency?
• RQ5 Can tts4pretrain reduce the need for large-scale transcribed data in downstream ASR fine-tuning?

Key findings

• tts4pretrain reduces WER by 10% relative on the LibriSpeech test set compared to wav2vec2.0 pretraining with only real speech.
• The method achieves performance equivalent to 5,000 hours of transcribed data using only 100 hours of transcribed data on the AMI meeting transcription task.
• Using 1 million TTS-generated utterances reduces WER by 6.8% relative, with diminishing returns beyond 100 million utterances.
• The best performance is achieved with a CTC-based decoder head, outperforming RNN-T, due to better alignment properties.
• Using domain-matched text (e.g., search queries) for TTS pretraining yields a 6.2% WER on an in-house Voice Search task, outperforming generic text sources.
• Language model fusion provides an additional 6% relative WER reduction when combined with tts4pretrain, confirming its complementarity.