[Paper Review] Variation and Synthetic Speech

Corey Miller, Orhan Karaali+1 more|arXiv (Cornell University)|Nov 17, 1997

Speech Recognition and Synthesis15 references3 citations

TL;DR

This paper presents a neural network-based approach to modeling linguistic variation in synthetic speech using a pan-dialectal pronunciation dictionary and a trainable postlexical module. By learning individual speaker-specific pronunciation variations through retraining, the system improves naturalness in synthesized speech without requiring extensive manual phonetic labeling for each speaker.

ABSTRACT

We describe the approach to linguistic variation taken by the Motorola speech synthesizer. A pan-dialectal pronunciation dictionary is described, which serves as the training data for a neural network based letter-to-sound converter. Subsequent to dictionary retrieval or letter-to-sound generation, pronunciations are submitted a neural network based postlexical module. The postlexical module has been trained on aligned dictionary pronunciations and hand-labeled narrow phonetic transcriptions. This architecture permits the learning of individual postlexical variation, and can be retrained for each speaker whose voice is being modeled for synthesis. Learning variation in this way can result in greater naturalness for the synthetic speech that is produced by the system.

Motivation & Objective

• To address the challenge of producing natural-sounding synthetic speech that reflects regional and individual pronunciation variations.
• To develop a scalable, trainable system that captures postlexical phonetic variation without requiring full phonetic transcriptions for every speaker.
• To enable speaker-specific adaptation of synthetic voices by retraining a postlexical neural network on individual speaker data.
• To integrate dialectal diversity into a single pronunciation dictionary while preserving phonetic accuracy and naturalness in synthesis.

Proposed method

• A pan-dialectal pronunciation dictionary is constructed as training data for a letter-to-sound (L2S) neural network.
• The L2S model generates initial pronunciations from text input, which are then passed to a postlexical neural network.
• The postlexical module is trained on aligned dictionary pronunciations and hand-labeled narrow phonetic transcriptions to model speaker-specific phonetic variation.
• The postlexical network is retrained for each target speaker using their voice data, enabling personalized pronunciation modeling.
• The system leverages neural networks to learn complex, non-linear mappings between input text and output phonetic sequences with speaker-specific characteristics.
• The architecture allows for end-to-end learning of variation patterns across dialects and individuals, enhancing naturalness in synthesized speech.

Experimental results

Research questions

• RQ1 How can a single speech synthesis system effectively model multiple dialectal variations in pronunciation?
• RQ2 What role does a trainable postlexical module play in capturing individual speaker-specific phonetic variation?
• RQ3 Can neural networks be effectively used to learn and apply postlexical pronunciation rules without full phonetic annotation for each speaker?
• RQ4 How does retraining the postlexical module per speaker improve the naturalness of synthetic speech?

Key findings

• The system successfully models linguistic variation across multiple dialects using a single, unified pronunciation dictionary.
• The postlexical neural network learns speaker-specific pronunciation patterns effectively through retraining on individual voice data.
• The use of a neural network for postlexical processing enables the system to generalize across dialectal and individual variation with high accuracy.
• Retraining the postlexical module for each speaker results in significantly more natural-sounding synthetic speech compared to fixed-rule systems.