[Paper Review] The CUHK-TUDELFT System for The SLT 2021 Children Speech Recognition Challenge

Si Ioi Ng, Wei Liu+5 more|arXiv (Cornell University)|Nov 12, 2020

Speech Recognition and Synthesis36 references4 citations

TL;DR

This paper presents the CUHK-TUDELFT system for the SLT 2021 Children Speech Recognition Challenge, employing a joint CTC-attention end-to-end (E2E) ASR framework with transfer learning from adult speech, data augmentation (speed perturbation, SpecAugment, RIR, volume perturbation), and hybrid language models (4-gram and RNN-LM). The system achieved a 20.1% character error rate (CER) on the test set, ranking 10th overall with 23.6% CER on the official evaluation set.

ABSTRACT

This technical report describes our submission to the 2021 SLT Children Speech Recognition Challenge (CSRC) Track 1. Our approach combines the use of a joint CTC-attention end-to-end (E2E) speech recognition framework, transfer learning, data augmentation and development of various language models. Procedures of data pre-processing, the background and the course of system development are described. The analysis of the experiment results, as well as the comparison between the E2E and DNN-HMM hybrid system are discussed in detail. Our system achieved a character error rate (CER) of 20.1% in our designated test set, and 23.6% in the official evaluation set, which is placed at 10-th overall.

Motivation & Objective

• To develop a high-performing automatic speech recognition (ASR) system for child speech, which faces challenges due to limited data and acoustic variability.
• To investigate the effectiveness of end-to-end (E2E) ASR with joint CTC-attention training for child speech recognition, especially under data-scarce conditions.
• To evaluate transfer learning from adult speech data and various data augmentation techniques in improving E2E ASR performance on child speech.
• To compare the performance of E2E ASR with a hybrid DNN-HMM system and assess the impact of different language model (LM) integration strategies.

Proposed method

• A joint CTC-attention end-to-end ASR framework is used, combining connectionist temporal classification (CTC) and attention-based decoding for sequence-to-sequence modeling.
• Transfer learning is applied by pre-training the E2E model on 341 hours of adult read speech (Set A) before fine-tuning on 59 hours of child speech (C1 and C2).
• Data augmentation includes 3-way speed perturbation, SpecAugment, reverberation (RIR), and volume perturbation to improve robustness and reduce overfitting.
• Two types of language models are used: a 4-gram character-level LM and an RNN-LM, both used to rescore E2E ASR hypotheses for improved accuracy.
• A hybrid DNN-HMM ASR system is also developed for comparison, using the same data and language models, with decoding based on HMM state transitions and acoustic models.
• The training data is partitioned at the speaker level into 81% training, 9% validation, and 10% test sets for each of Sets A, C1, and C2 to ensure no data leakage.

Experimental results

Research questions

• RQ1 Can transfer learning from adult speech significantly improve end-to-end ASR performance on limited child speech data?
• RQ2 How effective are various data augmentation techniques (e.g., speed perturbation, SpecAugment, RIR) in reducing overfitting and improving generalization in child speech ASR?
• RQ3 Does combining multiple language models (e.g., 4-gram and RNN-LM) lead to measurable performance gains in E2E ASR for child speech?
• RQ4 How does the performance of an E2E ASR system compare to a traditional hybrid DNN-HMM system in recognizing child speech, especially in conversational settings?

Key findings

• The transfer learning-based E2E ASR system achieved a 20.1% character error rate (CER) on the combined test sets of C1 and C2, representing the best performance of the E2E system.
• Applying additional data augmentation (SpecAugment, RIR, volume perturbation) beyond speed perturbation reduced the CER from 23.6% to 21.2% on the E2E system, demonstrating significant improvement.
• Fusing a 4-gram LM and an RNN-LM for rescoring the E2E system reduced CER to 20.1%, but required careful tuning of LM and decoder weights to avoid performance degradation.
• The hybrid DNN-HMM system achieved a CER of 20.1% with RNN-LM rescoring, showing comparable performance to the best E2E system, indicating strong competitiveness of hybrid models.
• The 'adult only' E2E system achieved 12.4% CER on C1 but 56.0% on C2, highlighting the challenge of recognizing conversational child speech compared to read speech.
• The 'child only' system outperformed 'adult only' on combined C1 and C2 but underperformed on C1 alone, indicating that adult data helps in read speech recognition, while child data is essential for conversational styles.