[Paper Review] Foreign English Accent Adjustment by Learning Phonetic Patterns

Fedor Kitashov, Elizaveta Svitanko+1 more|arXiv (Cornell University)|Jul 9, 2018

Speech Recognition and Synthesis12 references9 citations

TL;DR

This paper proposes a statistical model that automatically learns phonological generalizations from small accented speech datasets and uses them to generate millions of accented word variants from the CMU Pronouncing Dictionary. By training a sequence-to-sequence RNN on these synthetic accented samples, the model achieves 59% accuracy in recognizing non-native English pronunciations, effectively mimicking manually derived phonological rules with no human annotation of accent patterns.

ABSTRACT

State-of-the-art automatic speech recognition (ASR) systems struggle with the lack of data for rare accents. For sufficiently large datasets, neural engines tend to outshine statistical models in most natural language processing problems. However, a speech accent remains a challenge for both approaches. Phonologists manually create general rules describing a speaker's accent, but their results remain underutilized. In this paper, we propose a model that automatically retrieves phonological generalizations from a small dataset. This method leverages the difference in pronunciation between a particular dialect and General American English (GAE) and creates new accented samples of words. The proposed model is able to learn all generalizations that previously were manually obtained by phonologists. We use this statistical method to generate a million phonological variations of words from the CMU Pronouncing Dictionary and train a sequence-to-sequence RNN to recognize accented words with 59% accuracy.

Motivation & Objective

• To automatically extract phonological generalizations from limited accented speech data without manual rule creation.
• To generate a large-scale, diverse dataset of accented English words by applying learned phonetic patterns to native pronunciations.
• To improve automatic speech recognition (ASR) performance for non-native accents using synthetic accented training data.
• To demonstrate that a sequence-to-sequence RNN can effectively correct accented speech by learning from statistically derived accent patterns.

Proposed method

• The model uses a statistical approach to compare accented phonetic transcriptions against General American English (GAE) references, identifying insertions, deletions, and substitutions at the phoneme level.
• It constructs a sound-level statistics table tracking replacement, insertion, deletion, and occurrence frequencies across accent variations.
• A reduction dictionary maps 169 IPA characters from the GMU dataset to 39 phonemes in the CMU dictionary, enabling compatibility and scalability.
• The method generates one million accented word variants by applying learned phonological rules to CMU dictionary entries.
• A sequence-to-sequence RNN with bidirectional LSTM encoders and autoregressive decoders is trained to convert accented transcriptions into standard GAE forms.
• The model is trained on 800,000 synthetic accented samples (primarily Russian-influenced) with a batch size of 4096, using RMSprop and 100 training epochs.

Experimental results

Research questions

• RQ1 Can a statistical model automatically learn phonological generalizations from a small number of accented speech samples?
• RQ2 To what extent can automatically learned accent patterns generate diverse, realistic accented word variants?
• RQ3 Can a sequence-to-sequence RNN trained on synthetic accented data achieve high accuracy in recognizing non-native pronunciations?
• RQ4 How does the performance of the model compare to manually created phonological rules in terms of generalization and coverage?

Key findings

• The statistical model successfully recovered 13 out of 20 manually identified phonological generalizations when trained on the simplified CMU 39-phoneme representation.
• When trained on the full IPA GMU dataset with 169 characters, the model recovered all 20 manually identified generalizations, including complex patterns like palatalization and retroflexing.
• The sequence-to-sequence RNN achieved a test accuracy of 59.3% in converting accented phonetic transcriptions to standard GAE forms.
• Overfitting was observed after 25 training epochs, as validation accuracy plateaued while training accuracy continued to rise, suggesting noise from overabundant synthetic data.
• The model’s performance was sensitive to data simplification: loss of phonetic detail in the CMU dictionary reduced the model’s ability to capture rare or specific accent features.