[Paper Review] Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

Kaiqi Fu, Shaojun Gao+4 more|arXiv (Cornell University)|Mar 1, 2022

Speech Recognition and Synthesis5 citations

TL;DR

This paper proposes phone-level mixup data augmentation to improve non-native word-level pronunciation scoring in low-resource settings. By combining phone-level features and GOP scores from training data, it generates synthetic words with diverse pronunciation scores, boosting model generalization. Experiments on Speechocean762 show a 0.043 increase in Pearson correlation (PCC) to 0.610, outperforming baselines using 1/10 of the real unlabeled data.

ABSTRACT

Deep learning-based pronunciation scoring models highly rely on the availability of the annotated non-native data, which is costly and has scalability issues. To deal with the data scarcity problem, data augmentation is commonly used for model pretraining. In this paper, we propose a phone-level mixup, a simple yet effective data augmentation method, to improve the performance of word-level pronunciation scoring. Specifically, given a phoneme sequence from lexicon, the artificial augmented word sample can be generated by randomly sampling from the corresponding phone-level features in training data, while the word score is the average of their GOP scores. Benefit from the arbitrary phone-level combination, the mixup is able to generate any word with various pronunciation scores. Moreover, we utilize multi-source information (e.g., MFCC and deep features) to further improve the scoring system performance. The experiments conducted on the Speechocean762 show that the proposed system outperforms the baseline by adding the mixup data for pretraining, with Pearson correlation coefficients (PCC) increasing from 0.567 to 0.61. The results also indicate that proposed method achieves similar performance by using 1/10 unlabeled data of baseline. In addition, the experimental results also demonstrate the efficiency of our proposed multi-source approach.

Motivation & Objective

• Address the challenge of limited annotated non-native speech data for word-level pronunciation scoring.
• Improve model generalization in low-resource scenarios where labeled data is scarce and expensive to collect.
• Develop a data augmentation method that generates diverse, realistic pronunciation variations at the word level.
• Investigate the effectiveness of multi-source features (MFCC and deep features) in enhancing pronunciation scoring performance.
• Demonstrate that synthetic data via phone-level mixup can match or exceed performance of real data when labeled data is limited.

Proposed method

• Propose phone-level mixup: for each word, randomly sample phone-level features and GOP scores from training data to synthesize new word-level samples.
• Construct a phone-level feature and GOP score pool from existing training data, enabling arbitrary combinations of phones to form new words.
• Use the average GOP score of constituent phones as the pronunciation score for each augmented word.
• Combine MFCC and deep acoustic features as multi-source input to the pronunciation scorer, enhancing discriminative capability.
• Pre-train the pronunciation scorer on a mix of real unlabeled data and synthetic mixup-generated data to improve generalization.
• Optimize the model using Pearson correlation coefficient (PCC) between predicted and human-annotated scores as the evaluation metric.

Experimental results

Research questions

• RQ1 Can phone-level mixup data augmentation improve word-level pronunciation scoring performance in low-resource settings?
• RQ2 How does the performance of the proposed method compare to baselines using real unlabeled data when data size is constrained?
• RQ3 Does combining MFCC and deep features (multi-source input) lead to better pronunciation scoring than using either feature alone?
• RQ4 What is the optimal amount of augmented data needed to maximize performance without overfitting?
• RQ5 Can synthetic data generated via mixup achieve performance comparable to real data when the real data size is increased tenfold?

Key findings

• The proposed mixup method increases the Pearson correlation coefficient (PCC) from 0.567 (baseline) to 0.610, a significant improvement of 0.043.
• Using only 50k real unlabeled words and 450k mixup-generated words (total 500k), the model achieves a PCC of 0.610, outperforming a real-pretrain baseline with 500k real words.
• The system with mixup-pretraining achieves similar performance to a real-pretrain baseline using 10 times more real data, demonstrating data efficiency.
• Multi-source input combining MFCC and deep features yields the highest performance, indicating complementary information gain.
• Performance plateaus beyond 500k augmented samples, suggesting diminishing returns beyond this data size.
• Deep features alone outperform MFCC, and the combination of both further improves performance, confirming their complementary nature.