[Paper Review] Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages

Yi Cheng, Wang Jian-zhong+3 more|arXiv (Cornell University)|Dec 22, 2020

Speech Recognition and Synthesis24 references57 citations

TL;DR

This paper applies pre-trained wav2vec2.0 models to low-resource ASR across six real spoken languages, showing substantial gains over prior work and that coarse-grained units plus self-supervised pre-training are advantageous.

ABSTRACT

There are several domains that own corresponding widely used feature extractors, such as ResNet, BERT, and GPT-x. These models are usually pre-trained on large amounts of unlabeled data by self-supervision and can be effectively applied to downstream tasks. In the speech domain, wav2vec2.0 starts to show its powerful representation ability and feasibility of ultra-low resource speech recognition on the Librispeech corpus, which belongs to the audiobook domain. However, wav2vec2.0 has not been examined on real spoken scenarios and languages other than English. To verify its universality over languages, we apply pre-trained models to solve low-resource speech recognition tasks in various spoken languages. We achieve more than 20% relative improvements in six languages compared with previous work. Among these languages, English achieves a gain of 52.4%. Moreover, using coarse-grained modeling units, such as subword or character, achieves better results than fine-grained modeling units, such as phone or letter.

Motivation & Objective

• Demonstrate the effectiveness of wav2vec2.0 in real low-resource ASR across multiple languages.
• Assess how output modeling units (subword/char vs. letters/phonemes) affect performance.
• Evaluate encoder-only vs encoder-decoder architectures for low-resource ASR.
• Compare self-supervised pre-training to supervised pre-training in low-resource settings.

Proposed method

• Use pre-trained wav2vec2.0-base and wav2vec2.0-large as encoders.
• Fine-tune with either CTC or cross-entropy loss, optionally with an LM during decoding.
• Experiment with coarse-grained units (subword/character) and fine-grained units (letter/phoneme).
• Optionally add an extra projection layer or an attention-based decoder for ASR output.
• Compare self-supervised pre-training vs supervised pre-training on target and non-target language data.

Experimental results

Research questions

• RQ1 Can wav2vec2.0 pre-trained on English effectively solve ASR in six low-resource languages with real spoken data?
• RQ2 Do coarse-grained units (subword/character) outperform fine-grained units (letters/phonemes/phones) in low-resource ASR?
• RQ3 What is the impact of encoder-only vs encoder-decoder architectures for low-resource ASR?
• RQ4 How does self-supervised pre-training compare with supervised pre-training for low-resource languages?

Key findings

• wav2vec2.0 models outperform prior work across all six languages on CALLHOME; wav2vec2.0-large yields the largest gains.
• English shows a 52.4% relative improvement with wav2vec2.0 compared to prior methods, with substantial gains also for German (42.9%).
• Coarse-grained modeling units (subword/character) generally perform better than fine-grained units (letters/phones) in these low-resource settings.
• An encoder-only setup with CTC/LM decoding outperforms encoder-decoder variants in low-resource data scenarios; adding a decoder can hurt performance when data are scarce.
• Self-supervised pre-training on large multilingual data can outperform supervised pre-training on target-similar data, suggesting robust cross-language acoustic representations.