[Paper Review] Adaptation of Whisper models to child speech recognition

Rishabh Jain, Andrei Barcovschi+3 more|arXiv (Cornell University)|Jul 24, 2023

Speech Recognition and SynthesisComputer Science3 citations

TL;DR

This paper investigates fine-tuning the multilingual Whisper ASR model on child speech to improve recognition performance. It demonstrates that fine-tuning Whisper on child speech yields significant ASR improvements over the original model, though self-supervised wav2vec2 models fine-tuned on child data outperform both Whisper variants.

ABSTRACT

Automatic Speech Recognition (ASR) systems often struggle with transcribing child speech due to the lack of large child speech datasets required to accurately train child-friendly ASR models. However, there are huge amounts of annotated adult speech datasets which were used to create multilingual ASR models, such as Whisper. Our work aims to explore whether such models can be adapted to child speech to improve ASR for children. In addition, we compare Whisper child-adaptations with finetuned self-supervised models, such as wav2vec2. We demonstrate that finetuning Whisper on child speech yields significant improvements in ASR performance on child speech, compared to non finetuned Whisper models. Additionally, utilizing self-supervised Wav2vec2 models that have been finetuned on child speech outperforms Whisper finetuning.

Motivation & Objective

• Address the challenge of poor ASR performance on child speech due to limited annotated child speech datasets.
• Investigate whether pre-trained multilingual ASR models like Whisper can be effectively adapted to child speech using limited fine-tuning data.
• Compare the performance of fine-tuned Whisper models against fine-tuned self-supervised models such as wav2vec2 for child speech recognition.
• Evaluate the effectiveness of transfer learning from adult speech pre-training to child speech adaptation in low-resource settings.

Proposed method

• Fine-tune the pre-trained Whisper base and large models on a curated child speech dataset to adapt them to child voice characteristics.
• Utilize existing large-scale adult speech datasets used to pre-train Whisper for initial model weights.
• Fine-tune self-supervised wav2vec2 models on the same child speech data for comparative performance evaluation.
• Apply standard ASR training procedures including spec-augmentation, learning rate scheduling, and early stopping during fine-tuning.
• Evaluate all models using standard ASR metrics such as word error rate (WER) on held-out child speech test sets.
• Ensure fair comparison by training all models with the same hyperparameters and data splits where possible.

Experimental results

Research questions

• RQ1 Can fine-tuning the Whisper model on child speech significantly improve its ASR performance compared to the original pre-trained model?
• RQ2 How does the performance of fine-tuned Whisper compare to that of a self-supervised wav2vec2 model fine-tuned on the same child speech data?
• RQ3 To what extent does transfer learning from adult speech pre-training benefit child speech recognition when fine-tuned on limited child data?
• RQ4 What is the relative impact of model architecture and pre-training data distribution on child speech recognition accuracy?

Key findings

• Fine-tuning Whisper on child speech leads to a substantial reduction in word error rate (WER) compared to the non-fine-tuned base model.
• The fine-tuned Whisper model achieves significantly better performance on child speech than the original Whisper model, demonstrating effective adaptation to child voice characteristics.
• Self-supervised wav2vec2 models fine-tuned on child speech outperform both the original and fine-tuned Whisper models in terms of WER.
• The performance gap between Whisper and wav2vec2 highlights the advantage of self-supervised pre-training on child-specific data for downstream ASR tasks.