[Paper Review] A Full Text-Dependent End to End Mispronunciation Detection and Diagnosis with Easy Data Augmentation Techniques

Kaiqi Fu, Jones Lin+4 more|arXiv (Cornell University)|Apr 17, 2021

Natural Language Processing Techniques16 references26 citations

TL;DR

The paper proposes a text-dependent end-to-end MD&D model that uses prior phoneme sequences with attention, plus three simple data augmentation methods to address class imbalance, achieving improved F-measure and PER over a CNN-RNN-CTC baseline on TIMIT and L2-ARCTIC.

ABSTRACT

Recently, end-to-end mispronunciation detection and diagnosis (MD&D) systems has become a popular alternative to greatly simplify the model-building process of conventional hybrid DNN-HMM systems by representing complicated modules with a single deep network architecture. In this paper, in order to utilize the prior text in the end-to-end structure, we present a novel text-dependent model which is difference with sed-mdd, the model achieves a fully end-to-end system by aligning the audio with the phoneme sequences of the prior text inside the model through the attention mechanism. Moreover, the prior text as input will be a problem of imbalance between positive and negative samples in the phoneme sequence. To alleviate this problem, we propose three simple data augmentation methods, which effectively improve the ability of model to capture mispronounced phonemes. We conduct experiments on L2-ARCTIC, and our best performance improved from 49.29% to 56.08% in F-measure metric compared to the CNN-RNN-CTC model.

Motivation & Objective

• Motivate and enable end-to-end mispronunciation detection and diagnosis (MD&D) by leveraging prior text information in an end-to-end framework.
• Propose a text-dependent model that aligns audio with prior phoneme sequences via attention without requiring forced alignment.
• Address positive/negative sample imbalance in phoneme sequences with simple data augmentation techniques.
• Demonstrate performance gains over a CNN-RNN-CTC baseline on public MD&D benchmarks.

Proposed method

• Three-module architecture: sentence encoder for prior phoneme sequence, audio encoder (CNN-RNN) for acoustic features, and decoder with attention to align audio and text.
• Sentence encoder uses a Bi-LSTM to produce keys/values from phoneme embeddings of the prior text.
• Audio encoder processes 243-dim features via CNN-RNN (two CNN, four Bi-LSTM layers) to produce queries.
• Attention computes context vectors by aligning audio queries with text keys/values; final framewise probabilities are produced with a concatenation of context and acoustic features, followed by a softmax to yield phoneme predictions.
• CTC-time alignment is not required; training uses attention-based cross-entropy without explicit time labeling.
• Three data augmentation techniques to balance positive (mispronunciations) and negative samples: Phoneme Set based (PS), Vowels/Consonants set based (VC), and Confusing pairs based (CP).

Experimental results

Research questions

• RQ1 Can a text-attended end-to-end MD&D model effectively utilize prior text information to detect mispronunciations at the phoneme level?
• RQ2 Do simple data augmentation strategies improve the model’s ability to detect mispronunciations under phoneme-level imbalance?
• RQ3 What are the comparative gains of phoneme-attention vs. character-attention and baseline CNN-RNN-CTC in MD&D tasks?
• RQ4 How does the proposed model perform on standard MD&D benchmarks (TIMIT and L2-ARCTIC) in terms of TA, diagnosis accuracy, and F-measure?

Key findings

• Replacing baseline with text-dependent attention improves F-measure from 49.29% to 52.51%.
• Phoneme-attention with data augmentation achieves the best F-measure (56.08% with VC=10%).
• The best configuration yields a mean F-measure improvement and maintains high true accept (TA) rates (e.g., TA around 93.06% for the best setup).
• Data augmentation increases positive samples and leads to substantial PER reductions (best PER around 15.58%–16.13% depending on augmentation level).
• All data-augmented phoneme-attention variants maintain high TA and F-measure, with notable reductions in error diagnoses for substitution and deletion types when prior text is used.
• Compared to the CNN-RNN-CTC baseline, the phoneme-attention model with augmentation significantly improves MD&D performance on the evaluated metrics.