[Paper Review] Syllable based DNN-HMM Cantonese Speech to Text System

Timothy Wong, Claire W.Y. Li+7 more|arXiv (Cornell University)|Feb 13, 2024

Speech Recognition and Synthesis19 references3 citations

TL;DR

This paper presents a syllable-based DNN-HMM acoustic model for Cantonese speech-to-text recognition, comparing Initial-Final (IF) and Onset-Nucleus-Coda (ONC) syllable units. Using the Kaldi toolkit with GPU-accelerated training and I-vector speaker adaptation, the ONC-based model achieves a 9.66% word error rate and real-time factor of 1.388, outperforming other configurations.

ABSTRACT

This paper reports our work on building up a Cantonese Speech-to-Text (STT) system with a syllable based acoustic model. This is a part of an effort in building a STT system to aid dyslexic students who have cognitive deficiency in writing skills but have no problem expressing their ideas through speech. For Cantonese speech recognition, the basic unit of acoustic models can either be the conventional Initial-Final (IF) syllables, or the Onset-Nucleus-Coda (ONC) syllables where finals are further split into nucleus and coda to reflect the intra-syllable variations in Cantonese. By using the Kaldi toolkit, our system is trained using the stochastic gradient descent optimization model with the aid of GPUs for the hybrid Deep Neural Network and Hidden Markov Model (DNN-HMM) with and without I-vector based speaker adaptive training technique. The input features of the same Gaussian Mixture Model with speaker adaptive training (GMM-SAT) to DNN are used in all cases. Experiments show that the ONC-based syllable acoustic modeling with I-vector based DNN-HMM achieves the best performance with the word error rate (WER) of 9.66% and the real time factor (RTF) of 1.38812.

Motivation & Objective

• To develop a speech-to-text system tailored for dyslexic students with writing difficulties but intact spoken expression.
• To evaluate syllable-level acoustic modeling in Cantonese using Initial-Final (IF) and Onset-Nucleus-Coda (ONC) units.
• To improve recognition accuracy through I-vector based speaker adaptation in a DNN-HMM framework.
• To optimize real-time performance and word error rate (WER) using deep neural networks and Kaldi toolkit.
• To establish a robust, low-latency STT system suitable for educational applications in Cantonese.

Proposed method

• The system employs a hybrid DNN-HMM architecture trained with stochastic gradient descent using GPU acceleration.
• Acoustic modeling is performed using syllable units, comparing Initial-Final (IF) and Onset-Nucleus-Coda (ONC) structures.
• I-vector based speaker adaptive training (SAT) is applied to improve generalization across speakers.
• Input features are shared between GMM-SAT and DNN models to ensure consistency in training.
• The Kaldi toolkit is used for end-to-end system training and decoding with standard recipe-based configurations.
• Word error rate (WER) and real-time factor (RTF) are used to evaluate system performance.

Experimental results

Research questions

• RQ1 Does using ONC-based syllable units improve recognition accuracy over traditional IF units in Cantonese STT?
• RQ2 To what extent does I-vector based speaker adaptation reduce word error rate in a DNN-HMM system for Cantonese?
• RQ3 What is the real-time performance (RTF) of a syllable-based DNN-HMM system with and without speaker adaptation?
• RQ4 How does the choice of syllable unit (IF vs. ONC) affect model complexity and inference efficiency?
• RQ5 Can a syllable-based DNN-HMM system achieve low WER and acceptable RTF for real-time educational applications?

Key findings

• The ONC-based syllable model with I-vector adaptation achieves the lowest word error rate of 9.66%.
• The system achieves a real-time factor of 1.388, indicating efficient inference suitable for real-time deployment.
• The I-vector based DNN-HMM configuration outperforms both IF-based and non-adapted models in terms of WER.
• The use of ONC units captures intra-syllable phonetic variations more effectively than IF units in Cantonese.
• The combination of DNNs and I-vector adaptation significantly improves robustness across diverse speakers.
• The system demonstrates strong potential for assistive technology applications targeting dyslexic learners.