[论文解读] Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages

Yi Cheng, Wang Jian-zhong+3 more|arXiv (Cornell University)|Dec 22, 2020

Speech Recognition and Synthesis参考文献 24被引用 57

一句话总结

本研究将预训练的 wav2vec2.0 模型应用于六种真实口语数据的低资源 ASR，显著优于以往工作，并且粗粒度单位结合自监督预训练具有优势。

ABSTRACT

There are several domains that own corresponding widely used feature extractors, such as ResNet, BERT, and GPT-x. These models are usually pre-trained on large amounts of unlabeled data by self-supervision and can be effectively applied to downstream tasks. In the speech domain, wav2vec2.0 starts to show its powerful representation ability and feasibility of ultra-low resource speech recognition on the Librispeech corpus, which belongs to the audiobook domain. However, wav2vec2.0 has not been examined on real spoken scenarios and languages other than English. To verify its universality over languages, we apply pre-trained models to solve low-resource speech recognition tasks in various spoken languages. We achieve more than 20% relative improvements in six languages compared with previous work. Among these languages, English achieves a gain of 52.4%. Moreover, using coarse-grained modeling units, such as subword or character, achieves better results than fine-grained modeling units, such as phone or letter.

研究动机与目标

• Demonstrate the effectiveness of wav2vec2.0 in real low-resource ASR across multiple languages.
• Assess how output modeling units (subword/char vs. letters/phonemes) affect performance.
• Evaluate encoder-only vs encoder-decoder architectures for low-resource ASR.
• Compare self-supervised pre-training to supervised pre-training in low-resource settings.

提出的方法

• Use pre-trained wav2vec2.0-base and wav2vec2.0-large as encoders.
• Fine-tune with either CTC or cross-entropy loss, optionally with an LM during decoding.
• Experiment with coarse-grained units (subword/character) and fine-grained units (letter/phoneme).
• Optionally add an extra projection layer or an attention-based decoder for ASR output.
• Compare self-supervised pre-training vs supervised pre-training on target and non-target language data.

实验结果

研究问题

• RQ1 Can wav2vec2.0 pre-trained on English effectively solve ASR in six low-resource languages with real spoken data?
• RQ2 Do coarse-grained units (subword/character) outperform fine-grained units (letters/phonemes/phones) in low-resource ASR?
• RQ3 What is the impact of encoder-only vs encoder-decoder architectures for low-resource ASR?
• RQ4 How does self-supervised pre-training compare with supervised pre-training for low-resource languages?

主要发现

• wav2vec2.0 模型在 CALLHOME 的六种语言上超越了以往工作；wav2vec2.0-large 取得最大的提升。
• 与以往方法相比，英语的相对提升为 52.4%，德语也有显著提升（42.9%）。
• 粗粒度建模单位（subword/character）在这些低资源设置中通常比细粒度单位（letters/phones）表现更好。
• 在低资源数据情形下，使用 CTC/LM 解码的编码器仅架构优于编码器-解码器变体；在数据稀缺时再添加解码器可能会降低性能。
• 在大规模多语言数据上的自监督预训练可以优于在目标相似数据上的监督预训练，表明跨语言声学表示具有鲁棒性。